Harvest for Alaskan Arthropods Created 09 Apr 08:43

Stage: completed
Fetched: 09 Apr 08:43
Validated: 09 Apr 08:43
Deltas Created 09 Apr 08:43
Units Normalized: 09 Apr 08:53
Ancestry Built: 09 Apr 08:45
Nodes Matched: 09 Apr 08:53
Names Parsed: 09 Apr 08:45
New Models Stored: 09 Apr 08:44
Indexed: 09 Apr 08:53
Completed: 09 Apr 08:57
Time to Harvest: less than a minute

Expected File Format Definitions

Harvesting Log (most recent first)

# Logfile created on 2020-04-09 08:43:31 -0400 by logger.rb/v1.4.2
[INFO] [2020-04-09 08:43:31] ## HARVEST: type = -harvest
[START] [2020-04-09 08:43:32] logged process
[START] [2020-04-09 08:43:32] create_harvest_instance
[STOP] [2020-04-09 08:43:41] create_harvest_instance
[START] [2020-04-09 08:43:41] fetch_files
[STOP] [2020-04-09 08:43:41] fetch_files
[START] [2020-04-09 08:43:41] validate_each_file
[STOP] [2020-04-09 08:43:43] validate_each_file
[START] [2020-04-09 08:43:43] convert_to_csv
[CMD] [2020-04-09 08:43:43] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_agents_20699.csv > /app/public/converted_csv/Alaskan_Arthropo_agents_20699.csv_sorted
[CMD] [2020-04-09 08:43:43] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_refs_20700.csv > /app/public/converted_csv/Alaskan_Arthropo_refs_20700.csv_sorted
[CMD] [2020-04-09 08:43:43] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_nodes_20701.csv > /app/public/converted_csv/Alaskan_Arthropo_nodes_20701.csv_sorted
[CMD] [2020-04-09 08:43:43] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_media_20702.csv > /app/public/converted_csv/Alaskan_Arthropo_media_20702.csv_sorted
[CMD] [2020-04-09 08:43:43] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_vernaculars_20703.csv > /app/public/converted_csv/Alaskan_Arthropo_vernaculars_20703.csv_sorted
[CMD] [2020-04-09 08:43:43] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_occurrences_20704.csv > /app/public/converted_csv/Alaskan_Arthropo_occurrences_20704.csv_sorted
[CMD] [2020-04-09 08:43:43] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_assocs_20705.csv > /app/public/converted_csv/Alaskan_Arthropo_assocs_20705.csv_sorted
[CMD] [2020-04-09 08:43:43] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_measurements_20706.csv > /app/public/converted_csv/Alaskan_Arthropo_measurements_20706.csv_sorted
[STOP] [2020-04-09 08:43:43] convert_to_csv
[START] [2020-04-09 08:43:43] calculate_delta
[CMD] [2020-04-09 08:43:43] echo "0a" > /app/public/diff/Alaskan_Arthropo_agents_20699.diff
[CMD] [2020-04-09 08:43:43] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_agents_20699.csv >> /app/public/diff/Alaskan_Arthropo_agents_20699.diff
[CMD] [2020-04-09 08:43:43] echo "." >> /app/public/diff/Alaskan_Arthropo_agents_20699.diff
[CMD] [2020-04-09 08:43:43] echo "0a" > /app/public/diff/Alaskan_Arthropo_refs_20700.diff
[CMD] [2020-04-09 08:43:43] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_refs_20700.csv >> /app/public/diff/Alaskan_Arthropo_refs_20700.diff
[CMD] [2020-04-09 08:43:43] echo "." >> /app/public/diff/Alaskan_Arthropo_refs_20700.diff
[CMD] [2020-04-09 08:43:43] echo "0a" > /app/public/diff/Alaskan_Arthropo_nodes_20701.diff
[CMD] [2020-04-09 08:43:43] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_nodes_20701.csv >> /app/public/diff/Alaskan_Arthropo_nodes_20701.diff
[CMD] [2020-04-09 08:43:43] echo "." >> /app/public/diff/Alaskan_Arthropo_nodes_20701.diff
[CMD] [2020-04-09 08:43:43] echo "0a" > /app/public/diff/Alaskan_Arthropo_media_20702.diff
[CMD] [2020-04-09 08:43:43] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_media_20702.csv >> /app/public/diff/Alaskan_Arthropo_media_20702.diff
[CMD] [2020-04-09 08:43:43] echo "." >> /app/public/diff/Alaskan_Arthropo_media_20702.diff
[CMD] [2020-04-09 08:43:43] echo "0a" > /app/public/diff/Alaskan_Arthropo_vernaculars_20703.diff
[CMD] [2020-04-09 08:43:43] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_vernaculars_20703.csv >> /app/public/diff/Alaskan_Arthropo_vernaculars_20703.diff
[CMD] [2020-04-09 08:43:43] echo "." >> /app/public/diff/Alaskan_Arthropo_vernaculars_20703.diff
[CMD] [2020-04-09 08:43:43] echo "0a" > /app/public/diff/Alaskan_Arthropo_occurrences_20704.diff
[CMD] [2020-04-09 08:43:43] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_occurrences_20704.csv >> /app/public/diff/Alaskan_Arthropo_occurrences_20704.diff
[CMD] [2020-04-09 08:43:43] echo "." >> /app/public/diff/Alaskan_Arthropo_occurrences_20704.diff
[CMD] [2020-04-09 08:43:43] echo "0a" > /app/public/diff/Alaskan_Arthropo_assocs_20705.diff
[CMD] [2020-04-09 08:43:43] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_assocs_20705.csv >> /app/public/diff/Alaskan_Arthropo_assocs_20705.diff
[CMD] [2020-04-09 08:43:43] echo "." >> /app/public/diff/Alaskan_Arthropo_assocs_20705.diff
[CMD] [2020-04-09 08:43:43] echo "0a" > /app/public/diff/Alaskan_Arthropo_measurements_20706.diff
[CMD] [2020-04-09 08:43:43] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_measurements_20706.csv >> /app/public/diff/Alaskan_Arthropo_measurements_20706.diff
[CMD] [2020-04-09 08:43:43] echo "." >> /app/public/diff/Alaskan_Arthropo_measurements_20706.diff
[STOP] [2020-04-09 08:43:43] calculate_delta
[START] [2020-04-09 08:43:43] parse_diff_and_store
[INFO] [2020-04-09 08:43:43] Loading agents diff file into memory (true lines)...
[INFO] [2020-04-09 08:43:43] Loading refs diff file into memory (true lines)...
[INFO] [2020-04-09 08:43:43] Loading nodes diff file into memory (true lines)...
[INFO] [2020-04-09 08:43:46] Loading media diff file into memory (true lines)...
[INFO] [2020-04-09 08:43:46] Loading vernaculars diff file into memory (true lines)...
[INFO] [2020-04-09 08:43:46] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-04-09 08:43:47] Loading assocs diff file into memory (true lines)...
[INFO] [2020-04-09 08:43:47] Loading measurements diff file into memory (true lines)...
[INFO] [2020-04-09 08:44:24] Storing 443 References
[INFO] [2020-04-09 08:44:24] Processing group of 443 in 1 groups of 1000
[INFO] [2020-04-09 08:44:25] Average Time: 0.08
[INFO] [2020-04-09 08:44:25] Total Time: 1s
[INFO] [2020-04-09 08:44:25] Storing 11116 ScientificNames
[INFO] [2020-04-09 08:44:25] Processing group of 11116 in 12 groups of 1000
[INFO] [2020-04-09 08:44:28] Average Time: 0.296
[INFO] [2020-04-09 08:44:28] Total Time: 4s
[INFO] [2020-04-09 08:44:28] last 3 / first 3: 0.67
[INFO] [2020-04-09 08:44:28] Std.Dev: 0.08366600265340755; Max: 0.44
[INFO] [2020-04-09 08:44:28] Storing 11116 Nodes
[INFO] [2020-04-09 08:44:28] Processing group of 11116 in 12 groups of 1000
[INFO] [2020-04-09 08:44:32] Average Time: 0.282
[INFO] [2020-04-09 08:44:32] Total Time: 4s
[INFO] [2020-04-09 08:44:32] last 3 / first 3: 0.51
[INFO] [2020-04-09 08:44:32] Std.Dev: 0.09486832980505137; Max: 0.43
[INFO] [2020-04-09 08:44:32] Storing 7682 Occurrences
[INFO] [2020-04-09 08:44:32] Processing group of 7682 in 8 groups of 1000
[INFO] [2020-04-09 08:44:33] Average Time: 0.11
[INFO] [2020-04-09 08:44:33] Total Time: 1s
[INFO] [2020-04-09 08:44:33] last 3 / first 3: 0.73
[INFO] [2020-04-09 08:44:33] Std.Dev: 0.0; Max: 0.14
[INFO] [2020-04-09 08:44:33] Storing 8258 TraitsReferences
[INFO] [2020-04-09 08:44:33] Processing group of 8258 in 9 groups of 1000
[INFO] [2020-04-09 08:44:33] Average Time: 0.072
[INFO] [2020-04-09 08:44:33] Total Time: 1s
[INFO] [2020-04-09 08:44:33] last 3 / first 3: 0.48
[INFO] [2020-04-09 08:44:33] Std.Dev: 0.044721359549995794; Max: 0.17
[INFO] [2020-04-09 08:44:33] Storing 7880 Traits
[INFO] [2020-04-09 08:44:33] Processing group of 7880 in 8 groups of 1000
[INFO] [2020-04-09 08:44:36] Average Time: 0.336
[INFO] [2020-04-09 08:44:36] Total Time: 3s
[INFO] [2020-04-09 08:44:36] last 3 / first 3: 0.67
[INFO] [2020-04-09 08:44:36] Std.Dev: 0.07745966692414834; Max: 0.44
[INFO] [2020-04-09 08:44:36] Storing 14991 MetaTraits
[INFO] [2020-04-09 08:44:36] Processing group of 14991 in 15 groups of 1000
[INFO] [2020-04-09 08:44:37] Average Time: 0.097
[INFO] [2020-04-09 08:44:37] Total Time: 2s
[INFO] [2020-04-09 08:44:37] last 3 / first 3: 0.79
[INFO] [2020-04-09 08:44:37] Std.Dev: 0.0; Max: 0.16
[STOP] [2020-04-09 08:44:37] parse_diff_and_store
[START] [2020-04-09 08:44:37] resolve_keys
[INFO] [2020-04-09 08:44:55] Occurrences to nodes (through scientific_names)...
[INFO] [2020-04-09 08:44:55] traits to occurrences...
[INFO] [2020-04-09 08:44:56] traits to nodes (through occurrences)...
[INFO] [2020-04-09 08:44:56] Traits to sex term...
[INFO] [2020-04-09 08:44:56] Traits to lifestage term...
[INFO] [2020-04-09 08:44:56] MetaTraits to traits...
[INFO] [2020-04-09 08:44:57] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-04-09 08:44:58] Assocs to occurrences...
[INFO] [2020-04-09 08:44:58] Assocs to nodes...
[INFO] [2020-04-09 08:44:58] Assoc to sex term...
[INFO] [2020-04-09 08:44:58] Assoc to lifestage term...
[STOP] [2020-04-09 08:44:58] resolve_keys
[START] [2020-04-09 08:44:58] hold_for_later_1
[STOP] [2020-04-09 08:44:58] hold_for_later_1
[START] [2020-04-09 08:44:58] hold_for_later_2
[STOP] [2020-04-09 08:44:58] hold_for_later_2
[START] [2020-04-09 08:44:58] resolve_missing_parents
[STOP] [2020-04-09 08:44:59] resolve_missing_parents
[START] [2020-04-09 08:44:59] rebuild_nodes
[START] [2020-04-09 08:44:59] Flattener#flatten
[START] [2020-04-09 08:44:59] Flattener#study_resource
[START] [2020-04-09 08:44:59] Flattener#build_ancestry
[STOP] [2020-04-09 08:45:00] Flattener#build_ancestry
[INFO] [2020-04-09 08:45:00] 11116 ancestry keys
[START] [2020-04-09 08:45:00] build_node_ancestors
[INFO] [2020-04-09 08:45:00] old ancestors deleted.
[STOP] [2020-04-09 08:45:02] build_node_ancestors
[START] [2020-04-09 08:45:05] Flattener#propagate_ancestor_ids
[STOP] [2020-04-09 08:45:06] Flattener#propagate_ancestor_ids
[STOP] [2020-04-09 08:45:06] Flattener#flatten
[STOP] [2020-04-09 08:45:06] rebuild_nodes
[START] [2020-04-09 08:45:06] resolve_missing_media_owners
[STOP] [2020-04-09 08:45:06] resolve_missing_media_owners
[START] [2020-04-09 08:45:06] sanitize_media_verbatims
[STOP] [2020-04-09 08:45:06] sanitize_media_verbatims
[START] [2020-04-09 08:45:06] queue_downloads
[STOP] [2020-04-09 08:45:06] queue_downloads
[START] [2020-04-09 08:45:06] parse_names
[WARN] [2020-04-09 08:45:06] I see 11116 names which still need to be parsed.
[WARN] [2020-04-09 08:45:15] I see 54 names which still need to be parsed.
[STOP] [2020-04-09 08:45:16] parse_names
[START] [2020-04-09 08:45:16] denormalize_canonical_names_to_nodes
[STOP] [2020-04-09 08:45:16] denormalize_canonical_names_to_nodes
[START] [2020-04-09 08:45:16] match_nodes
[START] [2020-04-09 08:45:16] map_all_nodes_to_pages
[STOP] [2020-04-09 08:53:25] map_all_nodes_to_pages
[INFO] [2020-04-09 08:53:25] 1063 Unmatched nodes (of 11116)! That's too many to output. First 10: Acronicta lupini (#68219881); Acronicta vulpina (#68219883); Actebia balanitis (#68219909); Alypia langtoni (#68220299); Anarta nigrolunata (#68220448); Anarta trifolii (#68220449); Androloma (#68220500); Androloma maccullochii (#68220501); Apamea amputatrix (#68220645); Apamea cogitata (#68220647)
[START] [2020-04-09 08:53:25] update_nodes
[STOP] [2020-04-09 08:53:29] update_nodes
[STOP] [2020-04-09 08:53:29] match_nodes
[START] [2020-04-09 08:53:29] reindex_search
[STOP] [2020-04-09 08:53:43] reindex_search
[START] [2020-04-09 08:53:43] normalize_units
[STOP] [2020-04-09 08:53:49] normalize_units
[START] [2020-04-09 08:53:49] calculate_statistics
[STOP] [2020-04-09 08:53:50] calculate_statistics
[START] [2020-04-09 08:53:50] complete_harvest_instance
[START] [2020-04-09 08:53:50] overall_tsv_creation
[INFO] [2020-04-09 08:53:50] Processing group of 11116 in 2 batches of 10000
[INFO] [2020-04-09 08:55:13] 7145 Traits (unfiltered)...
[INFO] [2020-04-09 08:55:26] 7145 Traits (filtered)...
[INFO] [2020-04-09 08:55:26] 0 Associations (filtered)...
[INFO] [2020-04-09 08:56:09] 20721 metadata added.
[INFO] [2020-04-09 08:56:09] 0 metadata added.
[INFO] [2020-04-09 08:57:04] 735 Traits (unfiltered)...
[INFO] [2020-04-09 08:57:17] 735 Traits (filtered)...
[INFO] [2020-04-09 08:57:17] 0 Associations (filtered)...
[INFO] [2020-04-09 08:57:54] 1853 metadata added.
[INFO] [2020-04-09 08:57:54] 0 metadata added.
[INFO] [2020-04-09 08:57:54] Average Time: 89.715
[INFO] [2020-04-09 08:57:54] Total Time: 4m5s
[STOP] [2020-04-09 08:57:54] overall_tsv_creation
[INFO] [2020-04-09 08:57:54] Done. Check your files:
[INFO] [2020-04-09 08:57:54] (11063 lines) /app/public/data/Alaskan_Arthropo/publish_nodes.tsv
[INFO] [2020-04-09 08:57:54] (50687 lines) /app/public/data/Alaskan_Arthropo/publish_node_ancestors.tsv
[INFO] [2020-04-09 08:57:54] (11116 lines) /app/public/data/Alaskan_Arthropo/publish_scientific_names.tsv
[INFO] [2020-04-09 08:57:54] (7881 lines) /app/public/data/Alaskan_Arthropo/publish_traits.tsv
[INFO] [2020-04-09 08:57:54] (22575 lines) /app/public/data/Alaskan_Arthropo/publish_metadata.tsv
[STOP] [2020-04-09 08:57:54] complete_harvest_instance
[START] [2020-04-09 08:57:54] completed
[STOP] [2020-04-09 08:57:54] completed
[STOP] [2020-04-09 08:57:54] logged process, took 862.07

Latest Process