Harvest for Lincoln Sea Species List Created 23 Dec 18:52

Stage: completed
Fetched: 23 Dec 18:52
Validated: 23 Dec 18:52
Deltas Created 23 Dec 18:52
Units Normalized: 23 Dec 18:53
Ancestry Built: 23 Dec 18:52
Nodes Matched: 23 Dec 18:53
Names Parsed: 23 Dec 18:53
New Models Stored: 23 Dec 18:52
Indexed: 23 Dec 18:53
Completed: 23 Dec 18:55
Time to Harvest: less than a minute
Harvesting Log

(139 lines)
# Logfile created on 2019-12-23 18:52:49 -0500 by logger.rb/56815
[START] [2019-12-23 18:52:49] logged process
[START] [2019-12-23 18:52:49] create_harvest_instance
[STOP] [2019-12-23 18:52:49] create_harvest_instance
[START] [2019-12-23 18:52:49] fetch_files
[STOP] [2019-12-23 18:52:49] fetch_files
[START] [2019-12-23 18:52:49] validate_each_file
[STOP] [2019-12-23 18:52:50] validate_each_file
[START] [2019-12-23 18:52:50] convert_to_csv
[CMD] [2019-12-23 18:52:50] /usr/bin/sort /app/public/converted_csv/lincoln_sea_sp_l_refs_19498.csv > /app/public/converted_csv/lincoln_sea_sp_l_refs_19498.csv_sorted
[CMD] [2019-12-23 18:52:50] /usr/bin/sort /app/public/converted_csv/lincoln_sea_sp_l_nodes_19499.csv > /app/public/converted_csv/lincoln_sea_sp_l_nodes_19499.csv_sorted
[CMD] [2019-12-23 18:52:50] /usr/bin/sort /app/public/converted_csv/lincoln_sea_sp_l_occurrences_19500.csv > /app/public/converted_csv/lincoln_sea_sp_l_occurrences_19500.csv_sorted
[CMD] [2019-12-23 18:52:50] /usr/bin/sort /app/public/converted_csv/lincoln_sea_sp_l_measurements_19501.csv > /app/public/converted_csv/lincoln_sea_sp_l_measurements_19501.csv_sorted
[STOP] [2019-12-23 18:52:50] convert_to_csv
[START] [2019-12-23 18:52:50] calculate_delta
[CMD] [2019-12-23 18:52:50] echo "0a" > /app/public/diff/lincoln_sea_sp_l_refs_19498.diff
[CMD] [2019-12-23 18:52:50] tail -n +1 /app/public/converted_csv/lincoln_sea_sp_l_refs_19498.csv >> /app/public/diff/lincoln_sea_sp_l_refs_19498.diff
[CMD] [2019-12-23 18:52:50] echo "." >> /app/public/diff/lincoln_sea_sp_l_refs_19498.diff
[CMD] [2019-12-23 18:52:51] echo "0a" > /app/public/diff/lincoln_sea_sp_l_nodes_19499.diff
[CMD] [2019-12-23 18:52:51] tail -n +1 /app/public/converted_csv/lincoln_sea_sp_l_nodes_19499.csv >> /app/public/diff/lincoln_sea_sp_l_nodes_19499.diff
[CMD] [2019-12-23 18:52:51] echo "." >> /app/public/diff/lincoln_sea_sp_l_nodes_19499.diff
[CMD] [2019-12-23 18:52:51] echo "0a" > /app/public/diff/lincoln_sea_sp_l_occurrences_19500.diff
[CMD] [2019-12-23 18:52:51] tail -n +1 /app/public/converted_csv/lincoln_sea_sp_l_occurrences_19500.csv >> /app/public/diff/lincoln_sea_sp_l_occurrences_19500.diff
[CMD] [2019-12-23 18:52:51] echo "." >> /app/public/diff/lincoln_sea_sp_l_occurrences_19500.diff
[CMD] [2019-12-23 18:52:51] echo "0a" > /app/public/diff/lincoln_sea_sp_l_measurements_19501.diff
[CMD] [2019-12-23 18:52:51] tail -n +1 /app/public/converted_csv/lincoln_sea_sp_l_measurements_19501.csv >> /app/public/diff/lincoln_sea_sp_l_measurements_19501.diff
[CMD] [2019-12-23 18:52:52] echo "." >> /app/public/diff/lincoln_sea_sp_l_measurements_19501.diff
[STOP] [2019-12-23 18:52:52] calculate_delta
[START] [2019-12-23 18:52:52] parse_diff_and_store
[INFO] [2019-12-23 18:52:52] Loading refs diff file into memory (true lines)...
[INFO] [2019-12-23 18:52:52] Loading nodes diff file into memory (true lines)...
[INFO] [2019-12-23 18:52:52] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-12-23 18:52:52] Loading measurements diff file into memory (true lines)...
[INFO] [2019-12-23 18:52:53] Storing 2 References
[INFO] [2019-12-23 18:52:53] Processing group of 2 in 1 groups of 1000
[INFO] [2019-12-23 18:52:53] Average Time: 0.0
[INFO] [2019-12-23 18:52:53] Total Time: 1s
[INFO] [2019-12-23 18:52:53] Storing 103 ScientificNames
[INFO] [2019-12-23 18:52:53] Processing group of 103 in 1 groups of 1000
[INFO] [2019-12-23 18:52:53] Average Time: 0.07
[INFO] [2019-12-23 18:52:53] Total Time: 1s
[INFO] [2019-12-23 18:52:53] Storing 103 Nodes
[INFO] [2019-12-23 18:52:53] Processing group of 103 in 1 groups of 1000
[INFO] [2019-12-23 18:52:53] Average Time: 0.04
[INFO] [2019-12-23 18:52:53] Total Time: 1s
[INFO] [2019-12-23 18:52:53] Storing 32 Occurrences
[INFO] [2019-12-23 18:52:53] Processing group of 32 in 1 groups of 1000
[INFO] [2019-12-23 18:52:53] Average Time: 0.01
[INFO] [2019-12-23 18:52:53] Total Time: 1s
[INFO] [2019-12-23 18:52:53] Storing 64 TraitsReferences
[INFO] [2019-12-23 18:52:53] Processing group of 64 in 1 groups of 1000
[INFO] [2019-12-23 18:52:53] Average Time: 0.02
[INFO] [2019-12-23 18:52:53] Total Time: 1s
[INFO] [2019-12-23 18:52:53] Storing 64 Traits
[INFO] [2019-12-23 18:52:53] Processing group of 64 in 1 groups of 1000
[INFO] [2019-12-23 18:52:53] Average Time: 0.04
[INFO] [2019-12-23 18:52:53] Total Time: 1s
[INFO] [2019-12-23 18:52:53] Storing 64 MetaTraits
[INFO] [2019-12-23 18:52:53] Processing group of 64 in 1 groups of 1000
[INFO] [2019-12-23 18:52:53] Average Time: 0.02
[INFO] [2019-12-23 18:52:53] Total Time: 1s
[STOP] [2019-12-23 18:52:53] parse_diff_and_store
[START] [2019-12-23 18:52:53] resolve_keys
[INFO] [2019-12-23 18:52:58] Occurrences to nodes (through scientific_names)...
[INFO] [2019-12-23 18:52:58] traits to occurrences...
[INFO] [2019-12-23 18:52:58] traits to nodes (through occurrences)...
[INFO] [2019-12-23 18:52:58] Traits to sex term...
[INFO] [2019-12-23 18:52:58] Traits to lifestage term...
[INFO] [2019-12-23 18:52:58] MetaTraits to traits...
[INFO] [2019-12-23 18:52:58] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-12-23 18:52:58] Assocs to occurrences...
[INFO] [2019-12-23 18:52:58] Assocs to nodes...
[INFO] [2019-12-23 18:52:58] Assoc to sex term...
[INFO] [2019-12-23 18:52:58] Assoc to lifestage term...
[STOP] [2019-12-23 18:52:58] resolve_keys
[START] [2019-12-23 18:52:58] hold_for_later_1
[STOP] [2019-12-23 18:52:58] hold_for_later_1
[START] [2019-12-23 18:52:58] hold_for_later_2
[STOP] [2019-12-23 18:52:58] hold_for_later_2
[START] [2019-12-23 18:52:58] resolve_missing_parents
[STOP] [2019-12-23 18:52:58] resolve_missing_parents
[START] [2019-12-23 18:52:58] rebuild_nodes
[START] [2019-12-23 18:52:58] Flattener#flatten
[START] [2019-12-23 18:52:58] Flattener#study_resource
[START] [2019-12-23 18:52:58] Flattener#build_ancestry
[STOP] [2019-12-23 18:52:58] Flattener#build_ancestry
[INFO] [2019-12-23 18:52:58] 103 ancestry keys
[START] [2019-12-23 18:52:58] build_node_ancestors
[INFO] [2019-12-23 18:52:58] old ancestors deleted.
[STOP] [2019-12-23 18:52:58] build_node_ancestors
[START] [2019-12-23 18:52:58] Flattener#propagate_ancestor_ids
[STOP] [2019-12-23 18:52:58] Flattener#propagate_ancestor_ids
[STOP] [2019-12-23 18:52:58] Flattener#flatten
[STOP] [2019-12-23 18:52:58] rebuild_nodes
[START] [2019-12-23 18:52:58] resolve_missing_media_owners
[STOP] [2019-12-23 18:52:58] resolve_missing_media_owners
[START] [2019-12-23 18:52:58] sanitize_media_verbatims
[STOP] [2019-12-23 18:52:58] sanitize_media_verbatims
[START] [2019-12-23 18:52:58] queue_downloads
[STOP] [2019-12-23 18:52:58] queue_downloads
[START] [2019-12-23 18:52:58] parse_names
[WARN] [2019-12-23 18:52:58] I see 103 names which still need to be parsed.
[STOP] [2019-12-23 18:53:00] parse_names
[START] [2019-12-23 18:53:00] denormalize_canonical_names_to_nodes
[STOP] [2019-12-23 18:53:00] denormalize_canonical_names_to_nodes
[START] [2019-12-23 18:53:00] match_nodes
[START] [2019-12-23 18:53:00] map_all_nodes_to_pages
[STOP] [2019-12-23 18:53:19] map_all_nodes_to_pages
[INFO] [2019-12-23 18:53:19] Unmatched nodes (6 of 103): Larus thayeri (#62115934); Maxillopoda (#62115899); Nuculanoida (#62115974); Veneroida (#62115980); Magnoliopsida (#62115944); Magnoliophyta (#62115945)
[START] [2019-12-23 18:53:19] update_nodes
[STOP] [2019-12-23 18:53:19] update_nodes
[STOP] [2019-12-23 18:53:19] match_nodes
[START] [2019-12-23 18:53:19] reindex_search
[STOP] [2019-12-23 18:53:19] reindex_search
[START] [2019-12-23 18:53:19] normalize_units
[STOP] [2019-12-23 18:53:19] normalize_units
[START] [2019-12-23 18:53:19] calculate_statistics
[STOP] [2019-12-23 18:53:19] calculate_statistics
[START] [2019-12-23 18:53:19] complete_harvest_instance
[START] [2019-12-23 18:53:19] overall_tsv_creation
[INFO] [2019-12-23 18:53:19] Processing group of 103 in 1 batches of 10000
[INFO] [2019-12-23 18:54:11] 32 Traits (unfiltered)...
[INFO] [2019-12-23 18:54:31] 32 Traits (filtered)...
[INFO] [2019-12-23 18:54:31] 0 Associations (filtered)...
[INFO] [2019-12-23 18:55:19] 160 metadata added.
[INFO] [2019-12-23 18:55:19] 0 metadata added.
[INFO] [2019-12-23 18:55:19] Average Time: 95.79
[INFO] [2019-12-23 18:55:19] Total Time: 1m60s
[STOP] [2019-12-23 18:55:19] overall_tsv_creation
[INFO] [2019-12-23 18:55:19] Done. Check your files:
[INFO] [2019-12-23 18:55:19] (103 lines) /app/public/data/lincoln_sea_sp_l/publish_nodes.tsv
[INFO] [2019-12-23 18:55:19] (423 lines) /app/public/data/lincoln_sea_sp_l/publish_node_ancestors.tsv
[INFO] [2019-12-23 18:55:19] (103 lines) /app/public/data/lincoln_sea_sp_l/publish_scientific_names.tsv
[INFO] [2019-12-23 18:55:20] (33 lines) /app/public/data/lincoln_sea_sp_l/publish_traits.tsv
[INFO] [2019-12-23 18:55:20] (161 lines) /app/public/data/lincoln_sea_sp_l/publish_metadata.tsv
[STOP] [2019-12-23 18:55:20] complete_harvest_instance
[START] [2019-12-23 18:55:20] completed
[STOP] [2019-12-23 18:55:20] completed
[STOP] [2019-12-23 18:55:20] logged process, took 150.98
Latest Process

23 Dec 18:52 EXITED currently processing 0