Harvest for Black Sea Species List Created 22 Dec 22:51

Stage: completed
Fetched: 22 Dec 22:51
Validated: 22 Dec 22:51
Deltas Created 22 Dec 22:51
Units Normalized: 22 Dec 22:58
Ancestry Built: 22 Dec 22:51
Nodes Matched: 22 Dec 22:58
Names Parsed: 22 Dec 22:51
New Models Stored: 22 Dec 22:51
Indexed: 22 Dec 22:58
Completed: 22 Dec 22:59
Time to Harvest: less than a minute
Harvesting Log

(139 lines)
# Logfile created on 2019-12-22 22:51:30 -0500 by logger.rb/56815
[START] [2019-12-22 22:51:30] logged process
[START] [2019-12-22 22:51:30] create_harvest_instance
[STOP] [2019-12-22 22:51:30] create_harvest_instance
[START] [2019-12-22 22:51:30] fetch_files
[STOP] [2019-12-22 22:51:30] fetch_files
[START] [2019-12-22 22:51:30] validate_each_file
[STOP] [2019-12-22 22:51:31] validate_each_file
[START] [2019-12-22 22:51:31] convert_to_csv
[CMD] [2019-12-22 22:51:31] /usr/bin/sort /app/public/converted_csv/black_sea_sp_lis_refs_19138.csv > /app/public/converted_csv/black_sea_sp_lis_refs_19138.csv_sorted
[CMD] [2019-12-22 22:51:31] /usr/bin/sort /app/public/converted_csv/black_sea_sp_lis_nodes_19139.csv > /app/public/converted_csv/black_sea_sp_lis_nodes_19139.csv_sorted
[CMD] [2019-12-22 22:51:31] /usr/bin/sort /app/public/converted_csv/black_sea_sp_lis_occurrences_19140.csv > /app/public/converted_csv/black_sea_sp_lis_occurrences_19140.csv_sorted
[CMD] [2019-12-22 22:51:31] /usr/bin/sort /app/public/converted_csv/black_sea_sp_lis_measurements_19141.csv > /app/public/converted_csv/black_sea_sp_lis_measurements_19141.csv_sorted
[STOP] [2019-12-22 22:51:31] convert_to_csv
[START] [2019-12-22 22:51:31] calculate_delta
[CMD] [2019-12-22 22:51:31] echo "0a" > /app/public/diff/black_sea_sp_lis_refs_19138.diff
[CMD] [2019-12-22 22:51:31] tail -n +1 /app/public/converted_csv/black_sea_sp_lis_refs_19138.csv >> /app/public/diff/black_sea_sp_lis_refs_19138.diff
[CMD] [2019-12-22 22:51:31] echo "." >> /app/public/diff/black_sea_sp_lis_refs_19138.diff
[CMD] [2019-12-22 22:51:31] echo "0a" > /app/public/diff/black_sea_sp_lis_nodes_19139.diff
[CMD] [2019-12-22 22:51:31] tail -n +1 /app/public/converted_csv/black_sea_sp_lis_nodes_19139.csv >> /app/public/diff/black_sea_sp_lis_nodes_19139.diff
[CMD] [2019-12-22 22:51:31] echo "." >> /app/public/diff/black_sea_sp_lis_nodes_19139.diff
[CMD] [2019-12-22 22:51:31] echo "0a" > /app/public/diff/black_sea_sp_lis_occurrences_19140.diff
[CMD] [2019-12-22 22:51:31] tail -n +1 /app/public/converted_csv/black_sea_sp_lis_occurrences_19140.csv >> /app/public/diff/black_sea_sp_lis_occurrences_19140.diff
[CMD] [2019-12-22 22:51:31] echo "." >> /app/public/diff/black_sea_sp_lis_occurrences_19140.diff
[CMD] [2019-12-22 22:51:31] echo "0a" > /app/public/diff/black_sea_sp_lis_measurements_19141.diff
[CMD] [2019-12-22 22:51:31] tail -n +1 /app/public/converted_csv/black_sea_sp_lis_measurements_19141.csv >> /app/public/diff/black_sea_sp_lis_measurements_19141.diff
[CMD] [2019-12-22 22:51:31] echo "." >> /app/public/diff/black_sea_sp_lis_measurements_19141.diff
[STOP] [2019-12-22 22:51:31] calculate_delta
[START] [2019-12-22 22:51:31] parse_diff_and_store
[INFO] [2019-12-22 22:51:31] Loading refs diff file into memory (true lines)...
[INFO] [2019-12-22 22:51:32] Loading nodes diff file into memory (true lines)...
[INFO] [2019-12-22 22:51:32] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-12-22 22:51:32] Loading measurements diff file into memory (true lines)...
[INFO] [2019-12-22 22:51:37] Storing 2 References
[INFO] [2019-12-22 22:51:37] Processing group of 2 in 1 groups of 1000
[INFO] [2019-12-22 22:51:37] Average Time: 0.0
[INFO] [2019-12-22 22:51:37] Total Time: 1s
[INFO] [2019-12-22 22:51:37] Storing 1726 ScientificNames
[INFO] [2019-12-22 22:51:37] Processing group of 1726 in 2 groups of 1000
[INFO] [2019-12-22 22:51:38] Average Time: 0.4
[INFO] [2019-12-22 22:51:38] Total Time: 1s
[INFO] [2019-12-22 22:51:38] Storing 1726 Nodes
[INFO] [2019-12-22 22:51:38] Processing group of 1726 in 2 groups of 1000
[INFO] [2019-12-22 22:51:38] Average Time: 0.36
[INFO] [2019-12-22 22:51:38] Total Time: 1s
[INFO] [2019-12-22 22:51:38] Storing 736 Occurrences
[INFO] [2019-12-22 22:51:38] Processing group of 736 in 1 groups of 1000
[INFO] [2019-12-22 22:51:39] Average Time: 0.09
[INFO] [2019-12-22 22:51:39] Total Time: 1s
[INFO] [2019-12-22 22:51:39] Storing 1472 TraitsReferences
[INFO] [2019-12-22 22:51:39] Processing group of 1472 in 2 groups of 1000
[INFO] [2019-12-22 22:51:39] Average Time: 0.1
[INFO] [2019-12-22 22:51:39] Total Time: 1s
[INFO] [2019-12-22 22:51:39] Storing 1472 Traits
[INFO] [2019-12-22 22:51:39] Processing group of 1472 in 2 groups of 1000
[INFO] [2019-12-22 22:51:39] Average Time: 0.285
[INFO] [2019-12-22 22:51:39] Total Time: 1s
[INFO] [2019-12-22 22:51:39] Storing 1472 MetaTraits
[INFO] [2019-12-22 22:51:39] Processing group of 1472 in 2 groups of 1000
[INFO] [2019-12-22 22:51:41] Average Time: 0.1
[INFO] [2019-12-22 22:51:41] Total Time: 2s
[STOP] [2019-12-22 22:51:41] parse_diff_and_store
[START] [2019-12-22 22:51:41] resolve_keys
[INFO] [2019-12-22 22:51:51] Occurrences to nodes (through scientific_names)...
[INFO] [2019-12-22 22:51:51] traits to occurrences...
[INFO] [2019-12-22 22:51:52] traits to nodes (through occurrences)...
[INFO] [2019-12-22 22:51:52] Traits to sex term...
[INFO] [2019-12-22 22:51:52] Traits to lifestage term...
[INFO] [2019-12-22 22:51:52] MetaTraits to traits...
[INFO] [2019-12-22 22:51:52] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-12-22 22:51:53] Assocs to occurrences...
[INFO] [2019-12-22 22:51:53] Assocs to nodes...
[INFO] [2019-12-22 22:51:53] Assoc to sex term...
[INFO] [2019-12-22 22:51:53] Assoc to lifestage term...
[STOP] [2019-12-22 22:51:53] resolve_keys
[START] [2019-12-22 22:51:53] hold_for_later_1
[STOP] [2019-12-22 22:51:53] hold_for_later_1
[START] [2019-12-22 22:51:53] hold_for_later_2
[STOP] [2019-12-22 22:51:53] hold_for_later_2
[START] [2019-12-22 22:51:53] resolve_missing_parents
[STOP] [2019-12-22 22:51:55] resolve_missing_parents
[START] [2019-12-22 22:51:55] rebuild_nodes
[START] [2019-12-22 22:51:55] Flattener#flatten
[START] [2019-12-22 22:51:55] Flattener#study_resource
[START] [2019-12-22 22:51:55] Flattener#build_ancestry
[STOP] [2019-12-22 22:51:55] Flattener#build_ancestry
[INFO] [2019-12-22 22:51:55] 1726 ancestry keys
[START] [2019-12-22 22:51:55] build_node_ancestors
[INFO] [2019-12-22 22:51:55] old ancestors deleted.
[STOP] [2019-12-22 22:51:55] build_node_ancestors
[START] [2019-12-22 22:51:56] Flattener#propagate_ancestor_ids
[STOP] [2019-12-22 22:51:56] Flattener#propagate_ancestor_ids
[STOP] [2019-12-22 22:51:56] Flattener#flatten
[STOP] [2019-12-22 22:51:56] rebuild_nodes
[START] [2019-12-22 22:51:56] resolve_missing_media_owners
[STOP] [2019-12-22 22:51:56] resolve_missing_media_owners
[START] [2019-12-22 22:51:56] sanitize_media_verbatims
[STOP] [2019-12-22 22:51:56] sanitize_media_verbatims
[START] [2019-12-22 22:51:56] queue_downloads
[STOP] [2019-12-22 22:51:56] queue_downloads
[START] [2019-12-22 22:51:56] parse_names
[WARN] [2019-12-22 22:51:56] I see 1726 names which still need to be parsed.
[STOP] [2019-12-22 22:51:59] parse_names
[START] [2019-12-22 22:51:59] denormalize_canonical_names_to_nodes
[STOP] [2019-12-22 22:51:59] denormalize_canonical_names_to_nodes
[START] [2019-12-22 22:51:59] match_nodes
[START] [2019-12-22 22:51:59] map_all_nodes_to_pages
[STOP] [2019-12-22 22:57:59] map_all_nodes_to_pages
[INFO] [2019-12-22 22:57:59] 118 Unmatched nodes (of 1726)! That's too many to output. First 10: Larus melanocephalus (#61776953); Larus ichthyaetus (#61777498); Philomachus (#61776806); Philomachus pugnax (#61776805); Limicola (#61776974); Limicola falcinellus (#61776973); Anas querquedula (#61776761); Anas strepera (#61776772); Anas clypeata (#61776822); Anas penelope (#61776942)
[START] [2019-12-22 22:57:59] update_nodes
[STOP] [2019-12-22 22:58:00] update_nodes
[STOP] [2019-12-22 22:58:00] match_nodes
[START] [2019-12-22 22:58:00] reindex_search
[STOP] [2019-12-22 22:58:03] reindex_search
[START] [2019-12-22 22:58:03] normalize_units
[STOP] [2019-12-22 22:58:03] normalize_units
[START] [2019-12-22 22:58:03] calculate_statistics
[STOP] [2019-12-22 22:58:03] calculate_statistics
[START] [2019-12-22 22:58:03] complete_harvest_instance
[START] [2019-12-22 22:58:03] overall_tsv_creation
[INFO] [2019-12-22 22:58:03] Processing group of 1726 in 1 batches of 10000
[INFO] [2019-12-22 22:58:57] 736 Traits (unfiltered)...
[INFO] [2019-12-22 22:59:10] 736 Traits (filtered)...
[INFO] [2019-12-22 22:59:10] 0 Associations (filtered)...
[INFO] [2019-12-22 22:59:49] 3680 metadata added.
[INFO] [2019-12-22 22:59:49] 0 metadata added.
[INFO] [2019-12-22 22:59:49] Average Time: 81.66
[INFO] [2019-12-22 22:59:49] Total Time: 1m46s
[STOP] [2019-12-22 22:59:49] overall_tsv_creation
[INFO] [2019-12-22 22:59:49] Done. Check your files:
[INFO] [2019-12-22 22:59:49] (1726 lines) /app/public/data/black_sea_sp_lis/publish_nodes.tsv
[INFO] [2019-12-22 22:59:49] (8272 lines) /app/public/data/black_sea_sp_lis/publish_node_ancestors.tsv
[INFO] [2019-12-22 22:59:49] (1726 lines) /app/public/data/black_sea_sp_lis/publish_scientific_names.tsv
[INFO] [2019-12-22 22:59:49] (737 lines) /app/public/data/black_sea_sp_lis/publish_traits.tsv
[INFO] [2019-12-22 22:59:49] (3681 lines) /app/public/data/black_sea_sp_lis/publish_metadata.tsv
[STOP] [2019-12-22 22:59:49] complete_harvest_instance
[START] [2019-12-22 22:59:49] completed
[STOP] [2019-12-22 22:59:49] completed
[STOP] [2019-12-22 22:59:49] logged process, took 499.63
Latest Process

22 Dec 22:51 EXITED currently processing 0