Harvest for Molukka Sea Species List Created 24 Dec 09:02

Stage: completed
Fetched: 24 Dec 09:02
Validated: 24 Dec 09:02
Deltas Created 24 Dec 09:03
Units Normalized: 24 Dec 09:26
Ancestry Built: 24 Dec 09:04
Nodes Matched: 24 Dec 09:26
Names Parsed: 24 Dec 09:04
New Models Stored: 24 Dec 09:03
Indexed: 24 Dec 09:26
Completed: 24 Dec 09:28
Time to Harvest: less than a minute
Harvesting Log

(139 lines)
# Logfile created on 2019-12-24 09:02:56 -0500 by logger.rb/56815
[START] [2019-12-24 09:02:56] logged process
[START] [2019-12-24 09:02:56] create_harvest_instance
[STOP] [2019-12-24 09:02:57] create_harvest_instance
[START] [2019-12-24 09:02:57] fetch_files
[STOP] [2019-12-24 09:02:57] fetch_files
[START] [2019-12-24 09:02:57] validate_each_file
[STOP] [2019-12-24 09:02:57] validate_each_file
[START] [2019-12-24 09:02:57] convert_to_csv
[CMD] [2019-12-24 09:02:57] /usr/bin/sort /app/public/converted_csv/molukka_sea_sp_2_refs_19546.csv > /app/public/converted_csv/molukka_sea_sp_2_refs_19546.csv_sorted
[CMD] [2019-12-24 09:02:58] /usr/bin/sort /app/public/converted_csv/molukka_sea_sp_2_nodes_19547.csv > /app/public/converted_csv/molukka_sea_sp_2_nodes_19547.csv_sorted
[CMD] [2019-12-24 09:02:59] /usr/bin/sort /app/public/converted_csv/molukka_sea_sp_2_occurrences_19548.csv > /app/public/converted_csv/molukka_sea_sp_2_occurrences_19548.csv_sorted
[CMD] [2019-12-24 09:02:59] /usr/bin/sort /app/public/converted_csv/molukka_sea_sp_2_measurements_19549.csv > /app/public/converted_csv/molukka_sea_sp_2_measurements_19549.csv_sorted
[STOP] [2019-12-24 09:03:00] convert_to_csv
[START] [2019-12-24 09:03:00] calculate_delta
[CMD] [2019-12-24 09:03:00] echo "0a" > /app/public/diff/molukka_sea_sp_2_refs_19546.diff
[CMD] [2019-12-24 09:03:01] tail -n +1 /app/public/converted_csv/molukka_sea_sp_2_refs_19546.csv >> /app/public/diff/molukka_sea_sp_2_refs_19546.diff
[CMD] [2019-12-24 09:03:01] echo "." >> /app/public/diff/molukka_sea_sp_2_refs_19546.diff
[CMD] [2019-12-24 09:03:02] echo "0a" > /app/public/diff/molukka_sea_sp_2_nodes_19547.diff
[CMD] [2019-12-24 09:03:03] tail -n +1 /app/public/converted_csv/molukka_sea_sp_2_nodes_19547.csv >> /app/public/diff/molukka_sea_sp_2_nodes_19547.diff
[CMD] [2019-12-24 09:03:04] echo "." >> /app/public/diff/molukka_sea_sp_2_nodes_19547.diff
[CMD] [2019-12-24 09:03:04] echo "0a" > /app/public/diff/molukka_sea_sp_2_occurrences_19548.diff
[CMD] [2019-12-24 09:03:05] tail -n +1 /app/public/converted_csv/molukka_sea_sp_2_occurrences_19548.csv >> /app/public/diff/molukka_sea_sp_2_occurrences_19548.diff
[CMD] [2019-12-24 09:03:06] echo "." >> /app/public/diff/molukka_sea_sp_2_occurrences_19548.diff
[CMD] [2019-12-24 09:03:06] echo "0a" > /app/public/diff/molukka_sea_sp_2_measurements_19549.diff
[CMD] [2019-12-24 09:03:07] tail -n +1 /app/public/converted_csv/molukka_sea_sp_2_measurements_19549.csv >> /app/public/diff/molukka_sea_sp_2_measurements_19549.diff
[CMD] [2019-12-24 09:03:08] echo "." >> /app/public/diff/molukka_sea_sp_2_measurements_19549.diff
[STOP] [2019-12-24 09:03:08] calculate_delta
[START] [2019-12-24 09:03:08] parse_diff_and_store
[INFO] [2019-12-24 09:03:09] Loading refs diff file into memory (true lines)...
[INFO] [2019-12-24 09:03:10] Loading nodes diff file into memory (true lines)...
[INFO] [2019-12-24 09:03:12] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-12-24 09:03:13] Loading measurements diff file into memory (true lines)...
[INFO] [2019-12-24 09:03:26] Storing 2 References
[INFO] [2019-12-24 09:03:26] Processing group of 2 in 1 groups of 1000
[INFO] [2019-12-24 09:03:26] Average Time: 0.0
[INFO] [2019-12-24 09:03:26] Total Time: 1s
[INFO] [2019-12-24 09:03:26] Storing 4197 ScientificNames
[INFO] [2019-12-24 09:03:26] Processing group of 4197 in 5 groups of 1000
[INFO] [2019-12-24 09:03:28] Average Time: 0.342
[INFO] [2019-12-24 09:03:28] Total Time: 2s
[INFO] [2019-12-24 09:03:28] Storing 4197 Nodes
[INFO] [2019-12-24 09:03:28] Processing group of 4197 in 5 groups of 1000
[INFO] [2019-12-24 09:03:30] Average Time: 0.312
[INFO] [2019-12-24 09:03:30] Total Time: 2s
[INFO] [2019-12-24 09:03:30] Storing 2193 Occurrences
[INFO] [2019-12-24 09:03:30] Processing group of 2193 in 3 groups of 1000
[INFO] [2019-12-24 09:03:30] Average Time: 0.1
[INFO] [2019-12-24 09:03:30] Total Time: 1s
[INFO] [2019-12-24 09:03:30] Storing 4386 TraitsReferences
[INFO] [2019-12-24 09:03:30] Processing group of 4386 in 5 groups of 1000
[INFO] [2019-12-24 09:03:31] Average Time: 0.114
[INFO] [2019-12-24 09:03:31] Total Time: 1s
[INFO] [2019-12-24 09:03:31] Storing 4386 Traits
[INFO] [2019-12-24 09:03:31] Processing group of 4386 in 5 groups of 1000
[INFO] [2019-12-24 09:03:32] Average Time: 0.314
[INFO] [2019-12-24 09:03:32] Total Time: 2s
[INFO] [2019-12-24 09:03:32] Storing 4386 MetaTraits
[INFO] [2019-12-24 09:03:32] Processing group of 4386 in 5 groups of 1000
[INFO] [2019-12-24 09:03:33] Average Time: 0.17
[INFO] [2019-12-24 09:03:33] Total Time: 1s
[STOP] [2019-12-24 09:03:33] parse_diff_and_store
[START] [2019-12-24 09:03:33] resolve_keys
[INFO] [2019-12-24 09:03:53] Occurrences to nodes (through scientific_names)...
[INFO] [2019-12-24 09:03:55] traits to occurrences...
[INFO] [2019-12-24 09:03:57] traits to nodes (through occurrences)...
[INFO] [2019-12-24 09:03:57] Traits to sex term...
[INFO] [2019-12-24 09:03:59] Traits to lifestage term...
[INFO] [2019-12-24 09:04:01] MetaTraits to traits...
[INFO] [2019-12-24 09:04:02] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-12-24 09:04:02] Assocs to occurrences...
[INFO] [2019-12-24 09:04:02] Assocs to nodes...
[INFO] [2019-12-24 09:04:02] Assoc to sex term...
[INFO] [2019-12-24 09:04:02] Assoc to lifestage term...
[STOP] [2019-12-24 09:04:02] resolve_keys
[START] [2019-12-24 09:04:02] hold_for_later_1
[STOP] [2019-12-24 09:04:02] hold_for_later_1
[START] [2019-12-24 09:04:02] hold_for_later_2
[STOP] [2019-12-24 09:04:02] hold_for_later_2
[START] [2019-12-24 09:04:02] resolve_missing_parents
[STOP] [2019-12-24 09:04:10] resolve_missing_parents
[START] [2019-12-24 09:04:10] rebuild_nodes
[START] [2019-12-24 09:04:10] Flattener#flatten
[START] [2019-12-24 09:04:10] Flattener#study_resource
[START] [2019-12-24 09:04:10] Flattener#build_ancestry
[STOP] [2019-12-24 09:04:11] Flattener#build_ancestry
[INFO] [2019-12-24 09:04:11] 4197 ancestry keys
[START] [2019-12-24 09:04:11] build_node_ancestors
[INFO] [2019-12-24 09:04:11] old ancestors deleted.
[STOP] [2019-12-24 09:04:12] build_node_ancestors
[START] [2019-12-24 09:04:14] Flattener#propagate_ancestor_ids
[STOP] [2019-12-24 09:04:15] Flattener#propagate_ancestor_ids
[STOP] [2019-12-24 09:04:15] Flattener#flatten
[STOP] [2019-12-24 09:04:15] rebuild_nodes
[START] [2019-12-24 09:04:15] resolve_missing_media_owners
[STOP] [2019-12-24 09:04:15] resolve_missing_media_owners
[START] [2019-12-24 09:04:15] sanitize_media_verbatims
[STOP] [2019-12-24 09:04:15] sanitize_media_verbatims
[START] [2019-12-24 09:04:15] queue_downloads
[STOP] [2019-12-24 09:04:15] queue_downloads
[START] [2019-12-24 09:04:15] parse_names
[WARN] [2019-12-24 09:04:15] I see 4197 names which still need to be parsed.
[STOP] [2019-12-24 09:04:19] parse_names
[START] [2019-12-24 09:04:19] denormalize_canonical_names_to_nodes
[STOP] [2019-12-24 09:04:19] denormalize_canonical_names_to_nodes
[START] [2019-12-24 09:04:19] match_nodes
[START] [2019-12-24 09:04:19] map_all_nodes_to_pages
[STOP] [2019-12-24 09:26:28] map_all_nodes_to_pages
[INFO] [2019-12-24 09:26:28] 155 Unmatched nodes (of 4197)! That's too many to output. First 10: Acropora lutkeni (#62376274); Montipora hirsuta (#62377259); Euphyllidae (#62376373); Acrhelia (#62378190); Pectiniidae (#62377444); Mussidae (#62378995); Symphyllia (#62379395); Nephthea (#62377850); Actinodendronidae (#62376313); Abylopsis eschscholtzi (#62378371)
[START] [2019-12-24 09:26:28] update_nodes
[STOP] [2019-12-24 09:26:29] update_nodes
[STOP] [2019-12-24 09:26:29] match_nodes
[START] [2019-12-24 09:26:29] reindex_search
[STOP] [2019-12-24 09:26:44] reindex_search
[START] [2019-12-24 09:26:44] normalize_units
[STOP] [2019-12-24 09:26:44] normalize_units
[START] [2019-12-24 09:26:44] calculate_statistics
[STOP] [2019-12-24 09:26:44] calculate_statistics
[START] [2019-12-24 09:26:44] complete_harvest_instance
[START] [2019-12-24 09:26:44] overall_tsv_creation
[INFO] [2019-12-24 09:26:44] Processing group of 4197 in 1 batches of 10000
[INFO] [2019-12-24 09:27:49] 2193 Traits (unfiltered)...
[INFO] [2019-12-24 09:28:02] 2193 Traits (filtered)...
[INFO] [2019-12-24 09:28:02] 0 Associations (filtered)...
[INFO] [2019-12-24 09:28:44] 10965 metadata added.
[INFO] [2019-12-24 09:28:44] 0 metadata added.
[INFO] [2019-12-24 09:28:44] Average Time: 94.6
[INFO] [2019-12-24 09:28:44] Total Time: 2m1s
[STOP] [2019-12-24 09:28:44] overall_tsv_creation
[INFO] [2019-12-24 09:28:44] Done. Check your files:
[INFO] [2019-12-24 09:28:45] (4197 lines) /app/public/data/molukka_sea_sp_2/publish_nodes.tsv
[INFO] [2019-12-24 09:28:46] (21786 lines) /app/public/data/molukka_sea_sp_2/publish_node_ancestors.tsv
[INFO] [2019-12-24 09:28:46] (4197 lines) /app/public/data/molukka_sea_sp_2/publish_scientific_names.tsv
[INFO] [2019-12-24 09:28:47] (2194 lines) /app/public/data/molukka_sea_sp_2/publish_traits.tsv
[INFO] [2019-12-24 09:28:48] (10966 lines) /app/public/data/molukka_sea_sp_2/publish_metadata.tsv
[STOP] [2019-12-24 09:28:48] complete_harvest_instance
[START] [2019-12-24 09:28:48] completed
[STOP] [2019-12-24 09:28:48] completed
[STOP] [2019-12-24 09:28:48] logged process, took 1551.75
Latest Process

24 Dec 09:02 EXITED currently processing 0