Harvest for Kara Sea Species List Created 23 Dec 15:59

Stage: completed
Fetched: 23 Dec 15:59
Validated: 23 Dec 15:59
Deltas Created 23 Dec 15:59
Units Normalized: 23 Dec 16:11
Ancestry Built: 23 Dec 16:00
Nodes Matched: 23 Dec 16:11
Names Parsed: 23 Dec 16:00
New Models Stored: 23 Dec 16:00
Indexed: 23 Dec 16:11
Completed: 23 Dec 16:13
Time to Harvest: less than a minute
Harvesting Log

(139 lines)
# Logfile created on 2019-12-23 15:59:41 -0500 by logger.rb/56815
[START] [2019-12-23 15:59:41] logged process
[START] [2019-12-23 15:59:41] create_harvest_instance
[STOP] [2019-12-23 15:59:41] create_harvest_instance
[START] [2019-12-23 15:59:41] fetch_files
[STOP] [2019-12-23 15:59:41] fetch_files
[START] [2019-12-23 15:59:41] validate_each_file
[STOP] [2019-12-23 15:59:42] validate_each_file
[START] [2019-12-23 15:59:42] convert_to_csv
[CMD] [2019-12-23 15:59:42] /usr/bin/sort /app/public/converted_csv/kara_sea_sp_list_refs_19418.csv > /app/public/converted_csv/kara_sea_sp_list_refs_19418.csv_sorted
[CMD] [2019-12-23 15:59:42] /usr/bin/sort /app/public/converted_csv/kara_sea_sp_list_nodes_19419.csv > /app/public/converted_csv/kara_sea_sp_list_nodes_19419.csv_sorted
[CMD] [2019-12-23 15:59:43] /usr/bin/sort /app/public/converted_csv/kara_sea_sp_list_occurrences_19420.csv > /app/public/converted_csv/kara_sea_sp_list_occurrences_19420.csv_sorted
[CMD] [2019-12-23 15:59:44] /usr/bin/sort /app/public/converted_csv/kara_sea_sp_list_measurements_19421.csv > /app/public/converted_csv/kara_sea_sp_list_measurements_19421.csv_sorted
[STOP] [2019-12-23 15:59:44] convert_to_csv
[START] [2019-12-23 15:59:44] calculate_delta
[CMD] [2019-12-23 15:59:44] echo "0a" > /app/public/diff/kara_sea_sp_list_refs_19418.diff
[CMD] [2019-12-23 15:59:45] tail -n +1 /app/public/converted_csv/kara_sea_sp_list_refs_19418.csv >> /app/public/diff/kara_sea_sp_list_refs_19418.diff
[CMD] [2019-12-23 15:59:46] echo "." >> /app/public/diff/kara_sea_sp_list_refs_19418.diff
[CMD] [2019-12-23 15:59:46] echo "0a" > /app/public/diff/kara_sea_sp_list_nodes_19419.diff
[CMD] [2019-12-23 15:59:47] tail -n +1 /app/public/converted_csv/kara_sea_sp_list_nodes_19419.csv >> /app/public/diff/kara_sea_sp_list_nodes_19419.diff
[CMD] [2019-12-23 15:59:48] echo "." >> /app/public/diff/kara_sea_sp_list_nodes_19419.diff
[CMD] [2019-12-23 15:59:48] echo "0a" > /app/public/diff/kara_sea_sp_list_occurrences_19420.diff
[CMD] [2019-12-23 15:59:49] tail -n +1 /app/public/converted_csv/kara_sea_sp_list_occurrences_19420.csv >> /app/public/diff/kara_sea_sp_list_occurrences_19420.diff
[CMD] [2019-12-23 15:59:50] echo "." >> /app/public/diff/kara_sea_sp_list_occurrences_19420.diff
[CMD] [2019-12-23 15:59:50] echo "0a" > /app/public/diff/kara_sea_sp_list_measurements_19421.diff
[CMD] [2019-12-23 15:59:51] tail -n +1 /app/public/converted_csv/kara_sea_sp_list_measurements_19421.csv >> /app/public/diff/kara_sea_sp_list_measurements_19421.diff
[CMD] [2019-12-23 15:59:52] echo "." >> /app/public/diff/kara_sea_sp_list_measurements_19421.diff
[STOP] [2019-12-23 15:59:52] calculate_delta
[START] [2019-12-23 15:59:52] parse_diff_and_store
[INFO] [2019-12-23 15:59:53] Loading refs diff file into memory (true lines)...
[INFO] [2019-12-23 15:59:54] Loading nodes diff file into memory (true lines)...
[INFO] [2019-12-23 15:59:55] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-12-23 15:59:56] Loading measurements diff file into memory (true lines)...
[INFO] [2019-12-23 16:00:01] Storing 2 References
[INFO] [2019-12-23 16:00:01] Processing group of 2 in 1 groups of 1000
[INFO] [2019-12-23 16:00:01] Average Time: 0.0
[INFO] [2019-12-23 16:00:01] Total Time: 1s
[INFO] [2019-12-23 16:00:01] Storing 1782 ScientificNames
[INFO] [2019-12-23 16:00:01] Processing group of 1782 in 2 groups of 1000
[INFO] [2019-12-23 16:00:02] Average Time: 0.42
[INFO] [2019-12-23 16:00:02] Total Time: 1s
[INFO] [2019-12-23 16:00:02] Storing 1782 Nodes
[INFO] [2019-12-23 16:00:02] Processing group of 1782 in 2 groups of 1000
[INFO] [2019-12-23 16:00:02] Average Time: 0.315
[INFO] [2019-12-23 16:00:02] Total Time: 1s
[INFO] [2019-12-23 16:00:02] Storing 778 Occurrences
[INFO] [2019-12-23 16:00:02] Processing group of 778 in 1 groups of 1000
[INFO] [2019-12-23 16:00:02] Average Time: 0.12
[INFO] [2019-12-23 16:00:02] Total Time: 1s
[INFO] [2019-12-23 16:00:02] Storing 1556 TraitsReferences
[INFO] [2019-12-23 16:00:02] Processing group of 1556 in 2 groups of 1000
[INFO] [2019-12-23 16:00:03] Average Time: 0.105
[INFO] [2019-12-23 16:00:03] Total Time: 1s
[INFO] [2019-12-23 16:00:03] Storing 1556 Traits
[INFO] [2019-12-23 16:00:03] Processing group of 1556 in 2 groups of 1000
[INFO] [2019-12-23 16:00:03] Average Time: 0.37
[INFO] [2019-12-23 16:00:03] Total Time: 1s
[INFO] [2019-12-23 16:00:03] Storing 1553 MetaTraits
[INFO] [2019-12-23 16:00:03] Processing group of 1553 in 2 groups of 1000
[INFO] [2019-12-23 16:00:04] Average Time: 0.125
[INFO] [2019-12-23 16:00:04] Total Time: 1s
[STOP] [2019-12-23 16:00:04] parse_diff_and_store
[START] [2019-12-23 16:00:04] resolve_keys
[INFO] [2019-12-23 16:00:13] Occurrences to nodes (through scientific_names)...
[INFO] [2019-12-23 16:00:14] traits to occurrences...
[INFO] [2019-12-23 16:00:14] traits to nodes (through occurrences)...
[INFO] [2019-12-23 16:00:14] Traits to sex term...
[INFO] [2019-12-23 16:00:14] Traits to lifestage term...
[INFO] [2019-12-23 16:00:14] MetaTraits to traits...
[INFO] [2019-12-23 16:00:15] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-12-23 16:00:15] Assocs to occurrences...
[INFO] [2019-12-23 16:00:15] Assocs to nodes...
[INFO] [2019-12-23 16:00:15] Assoc to sex term...
[INFO] [2019-12-23 16:00:15] Assoc to lifestage term...
[STOP] [2019-12-23 16:00:15] resolve_keys
[START] [2019-12-23 16:00:15] hold_for_later_1
[STOP] [2019-12-23 16:00:15] hold_for_later_1
[START] [2019-12-23 16:00:15] hold_for_later_2
[STOP] [2019-12-23 16:00:15] hold_for_later_2
[START] [2019-12-23 16:00:15] resolve_missing_parents
[STOP] [2019-12-23 16:00:17] resolve_missing_parents
[START] [2019-12-23 16:00:17] rebuild_nodes
[START] [2019-12-23 16:00:17] Flattener#flatten
[START] [2019-12-23 16:00:17] Flattener#study_resource
[START] [2019-12-23 16:00:17] Flattener#build_ancestry
[STOP] [2019-12-23 16:00:17] Flattener#build_ancestry
[INFO] [2019-12-23 16:00:17] 1782 ancestry keys
[START] [2019-12-23 16:00:17] build_node_ancestors
[INFO] [2019-12-23 16:00:17] old ancestors deleted.
[STOP] [2019-12-23 16:00:18] build_node_ancestors
[START] [2019-12-23 16:00:19] Flattener#propagate_ancestor_ids
[STOP] [2019-12-23 16:00:19] Flattener#propagate_ancestor_ids
[STOP] [2019-12-23 16:00:19] Flattener#flatten
[STOP] [2019-12-23 16:00:19] rebuild_nodes
[START] [2019-12-23 16:00:19] resolve_missing_media_owners
[STOP] [2019-12-23 16:00:19] resolve_missing_media_owners
[START] [2019-12-23 16:00:19] sanitize_media_verbatims
[STOP] [2019-12-23 16:00:19] sanitize_media_verbatims
[START] [2019-12-23 16:00:19] queue_downloads
[STOP] [2019-12-23 16:00:19] queue_downloads
[START] [2019-12-23 16:00:19] parse_names
[WARN] [2019-12-23 16:00:19] I see 1782 names which still need to be parsed.
[STOP] [2019-12-23 16:00:21] parse_names
[START] [2019-12-23 16:00:21] denormalize_canonical_names_to_nodes
[STOP] [2019-12-23 16:00:21] denormalize_canonical_names_to_nodes
[START] [2019-12-23 16:00:21] match_nodes
[START] [2019-12-23 16:00:21] map_all_nodes_to_pages
[STOP] [2019-12-23 16:11:10] map_all_nodes_to_pages
[INFO] [2019-12-23 16:11:10] 138 Unmatched nodes (of 1782)! That's too many to output. First 10: Maxillopoda (#62084081); Metridia lucens (#62085144); Limnocalanus grimaldii (#62084441); Harpacticus uniremis (#62085466); Poecilostomatoida (#62084115); Pedunculata (#62085445); Scalpellum angustum (#62085664); Cytheropteron suzdalskyi (#62084342); Cytheropteron arcuatum (#62084492); Cytheropteron biconvexa (#62084658)
[START] [2019-12-23 16:11:10] update_nodes
[STOP] [2019-12-23 16:11:10] update_nodes
[STOP] [2019-12-23 16:11:10] match_nodes
[START] [2019-12-23 16:11:10] reindex_search
[STOP] [2019-12-23 16:11:18] reindex_search
[START] [2019-12-23 16:11:18] normalize_units
[STOP] [2019-12-23 16:11:18] normalize_units
[START] [2019-12-23 16:11:18] calculate_statistics
[STOP] [2019-12-23 16:11:18] calculate_statistics
[START] [2019-12-23 16:11:18] complete_harvest_instance
[START] [2019-12-23 16:11:18] overall_tsv_creation
[INFO] [2019-12-23 16:11:18] Processing group of 1782 in 1 batches of 10000
[INFO] [2019-12-23 16:12:11] 778 Traits (unfiltered)...
[INFO] [2019-12-23 16:12:24] 778 Traits (filtered)...
[INFO] [2019-12-23 16:12:24] 0 Associations (filtered)...
[INFO] [2019-12-23 16:13:03] 3887 metadata added.
[INFO] [2019-12-23 16:13:03] 0 metadata added.
[INFO] [2019-12-23 16:13:03] Average Time: 82.03
[INFO] [2019-12-23 16:13:03] Total Time: 1m46s
[STOP] [2019-12-23 16:13:03] overall_tsv_creation
[INFO] [2019-12-23 16:13:03] Done. Check your files:
[INFO] [2019-12-23 16:13:04] (1782 lines) /app/public/data/kara_sea_sp_list/publish_nodes.tsv
[INFO] [2019-12-23 16:13:04] (8344 lines) /app/public/data/kara_sea_sp_list/publish_node_ancestors.tsv
[INFO] [2019-12-23 16:13:05] (1782 lines) /app/public/data/kara_sea_sp_list/publish_scientific_names.tsv
[INFO] [2019-12-23 16:13:06] (779 lines) /app/public/data/kara_sea_sp_list/publish_traits.tsv
[INFO] [2019-12-23 16:13:06] (3888 lines) /app/public/data/kara_sea_sp_list/publish_metadata.tsv
[STOP] [2019-12-23 16:13:06] complete_harvest_instance
[START] [2019-12-23 16:13:06] completed
[STOP] [2019-12-23 16:13:06] completed
[STOP] [2019-12-23 16:13:06] logged process, took 805.74
Latest Process

23 Dec 15:59 EXITED currently processing 0