Harvest for Banda Sea Species List Created 22 Dec 20:05

Stage: completed
Fetched: 22 Dec 20:05
Validated: 22 Dec 20:05
Deltas Created 22 Dec 20:05
Units Normalized: 22 Dec 20:55
Ancestry Built: 22 Dec 20:07
Nodes Matched: 22 Dec 20:54
Names Parsed: 22 Dec 20:07
New Models Stored: 22 Dec 20:06
Indexed: 22 Dec 20:55
Completed: 22 Dec 20:57
Time to Harvest: 1 minute
Harvesting Log

(149 lines)
# Logfile created on 2019-12-22 20:05:47 -0500 by logger.rb/56815
[START] [2019-12-22 20:05:47] logged process
[START] [2019-12-22 20:05:47] create_harvest_instance
[STOP] [2019-12-22 20:05:48] create_harvest_instance
[START] [2019-12-22 20:05:48] fetch_files
[STOP] [2019-12-22 20:05:48] fetch_files
[START] [2019-12-22 20:05:48] validate_each_file
[STOP] [2019-12-22 20:05:49] validate_each_file
[START] [2019-12-22 20:05:49] convert_to_csv
[CMD] [2019-12-22 20:05:49] /usr/bin/sort /app/public/converted_csv/banda_sea_sp_lis_refs_19058.csv > /app/public/converted_csv/banda_sea_sp_lis_refs_19058.csv_sorted
[CMD] [2019-12-22 20:05:49] /usr/bin/sort /app/public/converted_csv/banda_sea_sp_lis_nodes_19059.csv > /app/public/converted_csv/banda_sea_sp_lis_nodes_19059.csv_sorted
[CMD] [2019-12-22 20:05:49] /usr/bin/sort /app/public/converted_csv/banda_sea_sp_lis_occurrences_19060.csv > /app/public/converted_csv/banda_sea_sp_lis_occurrences_19060.csv_sorted
[CMD] [2019-12-22 20:05:49] /usr/bin/sort /app/public/converted_csv/banda_sea_sp_lis_measurements_19061.csv > /app/public/converted_csv/banda_sea_sp_lis_measurements_19061.csv_sorted
[STOP] [2019-12-22 20:05:49] convert_to_csv
[START] [2019-12-22 20:05:49] calculate_delta
[CMD] [2019-12-22 20:05:49] echo "0a" > /app/public/diff/banda_sea_sp_lis_refs_19058.diff
[CMD] [2019-12-22 20:05:49] tail -n +1 /app/public/converted_csv/banda_sea_sp_lis_refs_19058.csv >> /app/public/diff/banda_sea_sp_lis_refs_19058.diff
[CMD] [2019-12-22 20:05:49] echo "." >> /app/public/diff/banda_sea_sp_lis_refs_19058.diff
[CMD] [2019-12-22 20:05:49] echo "0a" > /app/public/diff/banda_sea_sp_lis_nodes_19059.diff
[CMD] [2019-12-22 20:05:49] tail -n +1 /app/public/converted_csv/banda_sea_sp_lis_nodes_19059.csv >> /app/public/diff/banda_sea_sp_lis_nodes_19059.diff
[CMD] [2019-12-22 20:05:49] echo "." >> /app/public/diff/banda_sea_sp_lis_nodes_19059.diff
[CMD] [2019-12-22 20:05:49] echo "0a" > /app/public/diff/banda_sea_sp_lis_occurrences_19060.diff
[CMD] [2019-12-22 20:05:49] tail -n +1 /app/public/converted_csv/banda_sea_sp_lis_occurrences_19060.csv >> /app/public/diff/banda_sea_sp_lis_occurrences_19060.diff
[CMD] [2019-12-22 20:05:49] echo "." >> /app/public/diff/banda_sea_sp_lis_occurrences_19060.diff
[CMD] [2019-12-22 20:05:49] echo "0a" > /app/public/diff/banda_sea_sp_lis_measurements_19061.diff
[CMD] [2019-12-22 20:05:49] tail -n +1 /app/public/converted_csv/banda_sea_sp_lis_measurements_19061.csv >> /app/public/diff/banda_sea_sp_lis_measurements_19061.diff
[CMD] [2019-12-22 20:05:49] echo "." >> /app/public/diff/banda_sea_sp_lis_measurements_19061.diff
[STOP] [2019-12-22 20:05:50] calculate_delta
[START] [2019-12-22 20:05:50] parse_diff_and_store
[INFO] [2019-12-22 20:05:50] Loading refs diff file into memory (true lines)...
[INFO] [2019-12-22 20:05:50] Loading nodes diff file into memory (true lines)...
[INFO] [2019-12-22 20:05:53] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-12-22 20:05:54] Loading measurements diff file into memory (true lines)...
[INFO] [2019-12-22 20:06:23] Storing 2 References
[INFO] [2019-12-22 20:06:23] Processing group of 2 in 1 groups of 1000
[INFO] [2019-12-22 20:06:23] Average Time: 0.0
[INFO] [2019-12-22 20:06:23] Total Time: 1s
[INFO] [2019-12-22 20:06:23] Storing 8909 ScientificNames
[INFO] [2019-12-22 20:06:23] Processing group of 8909 in 9 groups of 1000
[INFO] [2019-12-22 20:06:26] Average Time: 0.388
[INFO] [2019-12-22 20:06:26] Total Time: 4s
[INFO] [2019-12-22 20:06:26] last 3 / first 3: 0.81
[INFO] [2019-12-22 20:06:26] Std.Dev: 0.05477225575051661; Max: 0.48
[INFO] [2019-12-22 20:06:26] Storing 8909 Nodes
[INFO] [2019-12-22 20:06:26] Processing group of 8909 in 9 groups of 1000
[INFO] [2019-12-22 20:06:29] Average Time: 0.286
[INFO] [2019-12-22 20:06:29] Total Time: 3s
[INFO] [2019-12-22 20:06:29] last 3 / first 3: 1.0
[INFO] [2019-12-22 20:06:29] Std.Dev: 0.03162277660168379; Max: 0.33
[INFO] [2019-12-22 20:06:29] Storing 5186 Occurrences
[INFO] [2019-12-22 20:06:29] Processing group of 5186 in 6 groups of 1000
[INFO] [2019-12-22 20:06:29] Average Time: 0.087
[INFO] [2019-12-22 20:06:29] Total Time: 1s
[INFO] [2019-12-22 20:06:29] Storing 10372 TraitsReferences
[INFO] [2019-12-22 20:06:29] Processing group of 10372 in 11 groups of 1000
[INFO] [2019-12-22 20:06:30] Average Time: 0.075
[INFO] [2019-12-22 20:06:30] Total Time: 1s
[INFO] [2019-12-22 20:06:30] last 3 / first 3: 0.61
[INFO] [2019-12-22 20:06:30] Std.Dev: 0.03162277660168379; Max: 0.15
[INFO] [2019-12-22 20:06:30] Storing 10372 Traits
[INFO] [2019-12-22 20:06:30] Processing group of 10372 in 11 groups of 1000
[INFO] [2019-12-22 20:06:34] Average Time: 0.309
[INFO] [2019-12-22 20:06:34] Total Time: 4s
[INFO] [2019-12-22 20:06:34] last 3 / first 3: 0.66
[INFO] [2019-12-22 20:06:34] Std.Dev: 0.08366600265340755; Max: 0.47
[INFO] [2019-12-22 20:06:34] Storing 10371 MetaTraits
[INFO] [2019-12-22 20:06:34] Processing group of 10371 in 11 groups of 1000
[INFO] [2019-12-22 20:06:35] Average Time: 0.135
[INFO] [2019-12-22 20:06:35] Total Time: 2s
[INFO] [2019-12-22 20:06:35] last 3 / first 3: 0.71
[INFO] [2019-12-22 20:06:35] Std.Dev: 0.03162277660168379; Max: 0.18
[STOP] [2019-12-22 20:06:35] parse_diff_and_store
[START] [2019-12-22 20:06:35] resolve_keys
[INFO] [2019-12-22 20:07:11] Occurrences to nodes (through scientific_names)...
[INFO] [2019-12-22 20:07:14] traits to occurrences...
[INFO] [2019-12-22 20:07:17] traits to nodes (through occurrences)...
[INFO] [2019-12-22 20:07:17] Traits to sex term...
[INFO] [2019-12-22 20:07:19] Traits to lifestage term...
[INFO] [2019-12-22 20:07:21] MetaTraits to traits...
[INFO] [2019-12-22 20:07:22] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-12-22 20:07:23] Assocs to occurrences...
[INFO] [2019-12-22 20:07:23] Assocs to nodes...
[INFO] [2019-12-22 20:07:23] Assoc to sex term...
[INFO] [2019-12-22 20:07:23] Assoc to lifestage term...
[STOP] [2019-12-22 20:07:23] resolve_keys
[START] [2019-12-22 20:07:23] hold_for_later_1
[STOP] [2019-12-22 20:07:23] hold_for_later_1
[START] [2019-12-22 20:07:23] hold_for_later_2
[STOP] [2019-12-22 20:07:23] hold_for_later_2
[START] [2019-12-22 20:07:23] resolve_missing_parents
[STOP] [2019-12-22 20:07:42] resolve_missing_parents
[START] [2019-12-22 20:07:42] rebuild_nodes
[START] [2019-12-22 20:07:42] Flattener#flatten
[START] [2019-12-22 20:07:42] Flattener#study_resource
[START] [2019-12-22 20:07:42] Flattener#build_ancestry
[STOP] [2019-12-22 20:07:43] Flattener#build_ancestry
[INFO] [2019-12-22 20:07:43] 8909 ancestry keys
[START] [2019-12-22 20:07:43] build_node_ancestors
[INFO] [2019-12-22 20:07:43] old ancestors deleted.
[STOP] [2019-12-22 20:07:46] build_node_ancestors
[START] [2019-12-22 20:07:50] Flattener#propagate_ancestor_ids
[STOP] [2019-12-22 20:07:51] Flattener#propagate_ancestor_ids
[STOP] [2019-12-22 20:07:51] Flattener#flatten
[STOP] [2019-12-22 20:07:51] rebuild_nodes
[START] [2019-12-22 20:07:51] resolve_missing_media_owners
[STOP] [2019-12-22 20:07:51] resolve_missing_media_owners
[START] [2019-12-22 20:07:51] sanitize_media_verbatims
[STOP] [2019-12-22 20:07:51] sanitize_media_verbatims
[START] [2019-12-22 20:07:51] queue_downloads
[STOP] [2019-12-22 20:07:51] queue_downloads
[START] [2019-12-22 20:07:51] parse_names
[WARN] [2019-12-22 20:07:51] I see 8909 names which still need to be parsed.
[STOP] [2019-12-22 20:07:58] parse_names
[START] [2019-12-22 20:07:58] denormalize_canonical_names_to_nodes
[STOP] [2019-12-22 20:07:58] denormalize_canonical_names_to_nodes
[START] [2019-12-22 20:07:58] match_nodes
[START] [2019-12-22 20:07:58] map_all_nodes_to_pages
[STOP] [2019-12-22 20:54:41] map_all_nodes_to_pages
[INFO] [2019-12-22 20:54:41] 334 Unmatched nodes (of 8909)! That's too many to output. First 10: Giuris aporos (#61708626); Ptereleotridae (#61710620); Trachinocephalus (#61709671); Benthalbella lowei (#61710867); Scorpaeniformes (#61708241); Caracanthidae (#61710860); Stephanoberyciformes (#61708701); Cetomimiformes (#61710561); Zenionidae (#61714859); Anas querquedula (#61716356)
[START] [2019-12-22 20:54:41] update_nodes
[STOP] [2019-12-22 20:54:45] update_nodes
[STOP] [2019-12-22 20:54:45] match_nodes
[START] [2019-12-22 20:54:45] reindex_search
[STOP] [2019-12-22 20:55:31] reindex_search
[START] [2019-12-22 20:55:31] normalize_units
[STOP] [2019-12-22 20:55:31] normalize_units
[START] [2019-12-22 20:55:31] calculate_statistics
[STOP] [2019-12-22 20:55:31] calculate_statistics
[START] [2019-12-22 20:55:31] complete_harvest_instance
[START] [2019-12-22 20:55:31] overall_tsv_creation
[INFO] [2019-12-22 20:55:31] Processing group of 8909 in 1 batches of 10000
[INFO] [2019-12-22 20:56:56] 5186 Traits (unfiltered)...
[INFO] [2019-12-22 20:57:09] 5186 Traits (filtered)...
[INFO] [2019-12-22 20:57:09] 0 Associations (filtered)...
[INFO] [2019-12-22 20:57:58] 25929 metadata added.
[INFO] [2019-12-22 20:57:58] 0 metadata added.
[INFO] [2019-12-22 20:57:58] Average Time: 119.02
[INFO] [2019-12-22 20:57:58] Total Time: 2m28s
[STOP] [2019-12-22 20:57:58] overall_tsv_creation
[INFO] [2019-12-22 20:57:58] Done. Check your files:
[INFO] [2019-12-22 20:57:59] (8909 lines) /app/public/data/banda_sea_sp_lis/publish_nodes.tsv
[INFO] [2019-12-22 20:57:59] (47311 lines) /app/public/data/banda_sea_sp_lis/publish_node_ancestors.tsv
[INFO] [2019-12-22 20:57:59] (8909 lines) /app/public/data/banda_sea_sp_lis/publish_scientific_names.tsv
[INFO] [2019-12-22 20:57:59] (5187 lines) /app/public/data/banda_sea_sp_lis/publish_traits.tsv
[INFO] [2019-12-22 20:57:59] (25930 lines) /app/public/data/banda_sea_sp_lis/publish_metadata.tsv
[STOP] [2019-12-22 20:57:59] complete_harvest_instance
[START] [2019-12-22 20:57:59] completed
[STOP] [2019-12-22 20:57:59] completed
[STOP] [2019-12-22 20:57:59] logged process, took 3131.64
Latest Process

22 Dec 20:05 EXITED currently processing 0