Harvest for Latvia Species List Created 14 Oct 01:28

Stage: completed
Fetched: 14 Oct 01:28
Validated: 14 Oct 01:28
Deltas Created 14 Oct 01:28
Units Normalized: 14 Oct 01:33
Ancestry Built: 14 Oct 01:29
Nodes Matched: 14 Oct 01:33
Names Parsed: 14 Oct 01:29
New Models Stored: 14 Oct 01:29
Indexed: 14 Oct 01:33
Completed: 14 Oct 01:35
Time to Harvest: less than a minute
Harvesting Log

(139 lines)
# Logfile created on 2019-10-14 01:28:51 -0400 by logger.rb/56815
[START] [2019-10-14 01:28:51] logged process
[START] [2019-10-14 01:28:51] create_harvest_instance
[STOP] [2019-10-14 01:28:51] create_harvest_instance
[START] [2019-10-14 01:28:51] fetch_files
[STOP] [2019-10-14 01:28:51] fetch_files
[START] [2019-10-14 01:28:51] validate_each_file
[STOP] [2019-10-14 01:28:52] validate_each_file
[START] [2019-10-14 01:28:52] convert_to_csv
[CMD] [2019-10-14 01:28:52] /usr/bin/sort /app/public/converted_csv/latvia_sp_list_refs_16371.csv > /app/public/converted_csv/latvia_sp_list_refs_16371.csv_sorted
[CMD] [2019-10-14 01:28:52] /usr/bin/sort /app/public/converted_csv/latvia_sp_list_nodes_16372.csv > /app/public/converted_csv/latvia_sp_list_nodes_16372.csv_sorted
[CMD] [2019-10-14 01:28:52] /usr/bin/sort /app/public/converted_csv/latvia_sp_list_occurrences_16373.csv > /app/public/converted_csv/latvia_sp_list_occurrences_16373.csv_sorted
[CMD] [2019-10-14 01:28:52] /usr/bin/sort /app/public/converted_csv/latvia_sp_list_measurements_16374.csv > /app/public/converted_csv/latvia_sp_list_measurements_16374.csv_sorted
[STOP] [2019-10-14 01:28:52] convert_to_csv
[START] [2019-10-14 01:28:52] calculate_delta
[CMD] [2019-10-14 01:28:52] echo "0a" > /app/public/diff/latvia_sp_list_refs_16371.diff
[CMD] [2019-10-14 01:28:52] tail -n +1 /app/public/converted_csv/latvia_sp_list_refs_16371.csv >> /app/public/diff/latvia_sp_list_refs_16371.diff
[CMD] [2019-10-14 01:28:53] echo "." >> /app/public/diff/latvia_sp_list_refs_16371.diff
[CMD] [2019-10-14 01:28:53] echo "0a" > /app/public/diff/latvia_sp_list_nodes_16372.diff
[CMD] [2019-10-14 01:28:53] tail -n +1 /app/public/converted_csv/latvia_sp_list_nodes_16372.csv >> /app/public/diff/latvia_sp_list_nodes_16372.diff
[CMD] [2019-10-14 01:28:53] echo "." >> /app/public/diff/latvia_sp_list_nodes_16372.diff
[CMD] [2019-10-14 01:28:53] echo "0a" > /app/public/diff/latvia_sp_list_occurrences_16373.diff
[CMD] [2019-10-14 01:28:53] tail -n +1 /app/public/converted_csv/latvia_sp_list_occurrences_16373.csv >> /app/public/diff/latvia_sp_list_occurrences_16373.diff
[CMD] [2019-10-14 01:28:53] echo "." >> /app/public/diff/latvia_sp_list_occurrences_16373.diff
[CMD] [2019-10-14 01:28:53] echo "0a" > /app/public/diff/latvia_sp_list_measurements_16374.diff
[CMD] [2019-10-14 01:28:53] tail -n +1 /app/public/converted_csv/latvia_sp_list_measurements_16374.csv >> /app/public/diff/latvia_sp_list_measurements_16374.diff
[CMD] [2019-10-14 01:28:53] echo "." >> /app/public/diff/latvia_sp_list_measurements_16374.diff
[STOP] [2019-10-14 01:28:53] calculate_delta
[START] [2019-10-14 01:28:53] parse_diff_and_store
[INFO] [2019-10-14 01:28:54] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-14 01:28:54] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-14 01:28:55] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-14 01:28:56] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-14 01:29:08] Storing 2 References
[INFO] [2019-10-14 01:29:08] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-14 01:29:08] Average Time: 0.0
[INFO] [2019-10-14 01:29:08] Total Time: 1s
[INFO] [2019-10-14 01:29:08] Storing 3925 ScientificNames
[INFO] [2019-10-14 01:29:08] Processing group of 3925 in 4 groups of 1000
[INFO] [2019-10-14 01:29:09] Average Time: 0.413
[INFO] [2019-10-14 01:29:09] Total Time: 2s
[INFO] [2019-10-14 01:29:09] Storing 3925 Nodes
[INFO] [2019-10-14 01:29:09] Processing group of 3925 in 4 groups of 1000
[INFO] [2019-10-14 01:29:11] Average Time: 0.295
[INFO] [2019-10-14 01:29:11] Total Time: 2s
[INFO] [2019-10-14 01:29:11] Storing 1928 Occurrences
[INFO] [2019-10-14 01:29:11] Processing group of 1928 in 2 groups of 1000
[INFO] [2019-10-14 01:29:11] Average Time: 0.135
[INFO] [2019-10-14 01:29:11] Total Time: 1s
[INFO] [2019-10-14 01:29:11] Storing 4140 TraitsReferences
[INFO] [2019-10-14 01:29:11] Processing group of 4140 in 5 groups of 1000
[INFO] [2019-10-14 01:29:11] Average Time: 0.072
[INFO] [2019-10-14 01:29:11] Total Time: 1s
[INFO] [2019-10-14 01:29:11] Storing 4139 Traits
[INFO] [2019-10-14 01:29:11] Processing group of 4139 in 5 groups of 1000
[INFO] [2019-10-14 01:29:13] Average Time: 0.266
[INFO] [2019-10-14 01:29:13] Total Time: 2s
[INFO] [2019-10-14 01:29:13] Storing 4133 MetaTraits
[INFO] [2019-10-14 01:29:13] Processing group of 4133 in 5 groups of 1000
[INFO] [2019-10-14 01:29:13] Average Time: 0.12
[INFO] [2019-10-14 01:29:13] Total Time: 1s
[STOP] [2019-10-14 01:29:13] parse_diff_and_store
[START] [2019-10-14 01:29:13] resolve_keys
[INFO] [2019-10-14 01:29:32] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-14 01:29:33] traits to occurrences...
[INFO] [2019-10-14 01:29:34] traits to nodes (through occurrences)...
[INFO] [2019-10-14 01:29:34] Traits to sex term...
[INFO] [2019-10-14 01:29:35] Traits to lifestage term...
[INFO] [2019-10-14 01:29:36] MetaTraits to traits...
[INFO] [2019-10-14 01:29:36] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-14 01:29:37] Assocs to occurrences...
[INFO] [2019-10-14 01:29:37] Assocs to nodes...
[INFO] [2019-10-14 01:29:37] Assoc to sex term...
[INFO] [2019-10-14 01:29:37] Assoc to lifestage term...
[STOP] [2019-10-14 01:29:37] resolve_keys
[START] [2019-10-14 01:29:37] hold_for_later_1
[STOP] [2019-10-14 01:29:37] hold_for_later_1
[START] [2019-10-14 01:29:37] hold_for_later_2
[STOP] [2019-10-14 01:29:37] hold_for_later_2
[START] [2019-10-14 01:29:37] resolve_missing_parents
[STOP] [2019-10-14 01:29:45] resolve_missing_parents
[START] [2019-10-14 01:29:45] rebuild_nodes
[START] [2019-10-14 01:29:45] Flattener#flatten
[START] [2019-10-14 01:29:45] Flattener#study_resource
[START] [2019-10-14 01:29:45] Flattener#build_ancestry
[STOP] [2019-10-14 01:29:45] Flattener#build_ancestry
[INFO] [2019-10-14 01:29:45] 3925 ancestry keys
[START] [2019-10-14 01:29:45] build_node_ancestors
[INFO] [2019-10-14 01:29:45] old ancestors deleted.
[STOP] [2019-10-14 01:29:46] build_node_ancestors
[START] [2019-10-14 01:29:46] Flattener#propagate_ancestor_ids
[STOP] [2019-10-14 01:29:46] Flattener#propagate_ancestor_ids
[STOP] [2019-10-14 01:29:46] Flattener#flatten
[STOP] [2019-10-14 01:29:46] rebuild_nodes
[START] [2019-10-14 01:29:46] resolve_missing_media_owners
[STOP] [2019-10-14 01:29:46] resolve_missing_media_owners
[START] [2019-10-14 01:29:46] sanitize_media_verbatims
[STOP] [2019-10-14 01:29:46] sanitize_media_verbatims
[START] [2019-10-14 01:29:46] queue_downloads
[STOP] [2019-10-14 01:29:46] queue_downloads
[START] [2019-10-14 01:29:46] parse_names
[WARN] [2019-10-14 01:29:46] I see 3925 names which still need to be parsed.
[STOP] [2019-10-14 01:29:50] parse_names
[START] [2019-10-14 01:29:50] denormalize_canonical_names_to_nodes
[STOP] [2019-10-14 01:29:50] denormalize_canonical_names_to_nodes
[START] [2019-10-14 01:29:50] match_nodes
[START] [2019-10-14 01:29:50] map_all_nodes_to_pages
[STOP] [2019-10-14 01:33:45] map_all_nodes_to_pages
[INFO] [2019-10-14 01:33:45] 401 Unmatched nodes (of 3925)! That's too many to output. First 10: Carduelis chloris (#50529178); Carduelis spinus (#50529193); Carduelis cannabina (#50529220); Carduelis flammea (#50530139); Dendrocopos minor (#50528994); Dendrocopos medius (#50529062); Anas querquedula (#50528993); Anas clypeata (#50529060); Anas penelope (#50529157); Anas strepera (#50529293)
[START] [2019-10-14 01:33:45] update_nodes
[STOP] [2019-10-14 01:33:46] update_nodes
[STOP] [2019-10-14 01:33:46] match_nodes
[START] [2019-10-14 01:33:46] reindex_search
[STOP] [2019-10-14 01:33:56] reindex_search
[START] [2019-10-14 01:33:56] normalize_units
[STOP] [2019-10-14 01:33:56] normalize_units
[START] [2019-10-14 01:33:56] calculate_statistics
[STOP] [2019-10-14 01:33:56] calculate_statistics
[START] [2019-10-14 01:33:56] complete_harvest_instance
[START] [2019-10-14 01:33:56] overall_tsv_creation
[INFO] [2019-10-14 01:33:56] Processing group of 3925 in 1 batches of 10000
[INFO] [2019-10-14 01:35:00] 1928 Traits (unfiltered)...
[INFO] [2019-10-14 01:35:14] 1928 Traits (filtered)...
[INFO] [2019-10-14 01:35:14] 0 Associations (filtered)...
[INFO] [2019-10-14 01:35:58] 9633 metadata added.
[INFO] [2019-10-14 01:35:58] 0 metadata added.
[INFO] [2019-10-14 01:35:58] Average Time: 98.69
[INFO] [2019-10-14 01:35:58] Total Time: 2m2s
[STOP] [2019-10-14 01:35:58] overall_tsv_creation
[INFO] [2019-10-14 01:35:58] Done. Check your files:
[INFO] [2019-10-14 01:35:58] (3925 lines) /app/public/data/latvia_sp_list/publish_nodes.tsv
[INFO] [2019-10-14 01:35:58] (5384 lines) /app/public/data/latvia_sp_list/publish_node_ancestors.tsv
[INFO] [2019-10-14 01:35:58] (3925 lines) /app/public/data/latvia_sp_list/publish_scientific_names.tsv
[INFO] [2019-10-14 01:35:59] (1929 lines) /app/public/data/latvia_sp_list/publish_traits.tsv
[INFO] [2019-10-14 01:35:59] (9634 lines) /app/public/data/latvia_sp_list/publish_metadata.tsv
[STOP] [2019-10-14 01:35:59] complete_harvest_instance
[START] [2019-10-14 01:35:59] completed
[STOP] [2019-10-14 01:35:59] completed
[STOP] [2019-10-14 01:35:59] logged process, took 428.16
Latest Process

14 Oct 01:28 EXITED currently processing 0