Harvest for Brunei Species List Created 11 Oct 23:09

Stage: completed
Fetched: 11 Oct 23:09
Validated: 11 Oct 23:09
Deltas Created 11 Oct 23:09
Units Normalized: 11 Oct 23:12
Ancestry Built: 11 Oct 23:10
Nodes Matched: 11 Oct 23:12
Names Parsed: 11 Oct 23:10
New Models Stored: 11 Oct 23:10
Indexed: 11 Oct 23:12
Completed: 11 Oct 23:14
Time to Harvest: less than a minute
Harvesting Log

(139 lines)
# Logfile created on 2019-10-11 23:09:33 -0400 by logger.rb/56815
[START] [2019-10-11 23:09:33] logged process
[START] [2019-10-11 23:09:33] create_harvest_instance
[STOP] [2019-10-11 23:09:33] create_harvest_instance
[START] [2019-10-11 23:09:33] fetch_files
[STOP] [2019-10-11 23:09:33] fetch_files
[START] [2019-10-11 23:09:33] validate_each_file
[STOP] [2019-10-11 23:09:34] validate_each_file
[START] [2019-10-11 23:09:34] convert_to_csv
[CMD] [2019-10-11 23:09:34] /usr/bin/sort /app/public/converted_csv/brunei_sp_list_refs_15323.csv > /app/public/converted_csv/brunei_sp_list_refs_15323.csv_sorted
[CMD] [2019-10-11 23:09:34] /usr/bin/sort /app/public/converted_csv/brunei_sp_list_nodes_15324.csv > /app/public/converted_csv/brunei_sp_list_nodes_15324.csv_sorted
[CMD] [2019-10-11 23:09:34] /usr/bin/sort /app/public/converted_csv/brunei_sp_list_occurrences_15325.csv > /app/public/converted_csv/brunei_sp_list_occurrences_15325.csv_sorted
[CMD] [2019-10-11 23:09:34] /usr/bin/sort /app/public/converted_csv/brunei_sp_list_measurements_15326.csv > /app/public/converted_csv/brunei_sp_list_measurements_15326.csv_sorted
[STOP] [2019-10-11 23:09:34] convert_to_csv
[START] [2019-10-11 23:09:34] calculate_delta
[CMD] [2019-10-11 23:09:34] echo "0a" > /app/public/diff/brunei_sp_list_refs_15323.diff
[CMD] [2019-10-11 23:09:34] tail -n +1 /app/public/converted_csv/brunei_sp_list_refs_15323.csv >> /app/public/diff/brunei_sp_list_refs_15323.diff
[CMD] [2019-10-11 23:09:35] echo "." >> /app/public/diff/brunei_sp_list_refs_15323.diff
[CMD] [2019-10-11 23:09:35] echo "0a" > /app/public/diff/brunei_sp_list_nodes_15324.diff
[CMD] [2019-10-11 23:09:35] tail -n +1 /app/public/converted_csv/brunei_sp_list_nodes_15324.csv >> /app/public/diff/brunei_sp_list_nodes_15324.diff
[CMD] [2019-10-11 23:09:35] echo "." >> /app/public/diff/brunei_sp_list_nodes_15324.diff
[CMD] [2019-10-11 23:09:35] echo "0a" > /app/public/diff/brunei_sp_list_occurrences_15325.diff
[CMD] [2019-10-11 23:09:35] tail -n +1 /app/public/converted_csv/brunei_sp_list_occurrences_15325.csv >> /app/public/diff/brunei_sp_list_occurrences_15325.diff
[CMD] [2019-10-11 23:09:35] echo "." >> /app/public/diff/brunei_sp_list_occurrences_15325.diff
[CMD] [2019-10-11 23:09:35] echo "0a" > /app/public/diff/brunei_sp_list_measurements_15326.diff
[CMD] [2019-10-11 23:09:35] tail -n +1 /app/public/converted_csv/brunei_sp_list_measurements_15326.csv >> /app/public/diff/brunei_sp_list_measurements_15326.diff
[CMD] [2019-10-11 23:09:35] echo "." >> /app/public/diff/brunei_sp_list_measurements_15326.diff
[STOP] [2019-10-11 23:09:36] calculate_delta
[START] [2019-10-11 23:09:36] parse_diff_and_store
[INFO] [2019-10-11 23:09:36] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-11 23:09:36] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-11 23:09:37] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-11 23:09:38] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-11 23:09:52] Storing 2 References
[INFO] [2019-10-11 23:09:52] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-11 23:09:52] Average Time: 0.0
[INFO] [2019-10-11 23:09:52] Total Time: 1s
[INFO] [2019-10-11 23:09:52] Storing 3690 ScientificNames
[INFO] [2019-10-11 23:09:52] Processing group of 3690 in 4 groups of 1000
[INFO] [2019-10-11 23:09:53] Average Time: 0.395
[INFO] [2019-10-11 23:09:53] Total Time: 2s
[INFO] [2019-10-11 23:09:53] Storing 3690 Nodes
[INFO] [2019-10-11 23:09:53] Processing group of 3690 in 4 groups of 1000
[INFO] [2019-10-11 23:09:55] Average Time: 0.275
[INFO] [2019-10-11 23:09:55] Total Time: 2s
[INFO] [2019-10-11 23:09:55] Storing 1928 Occurrences
[INFO] [2019-10-11 23:09:55] Processing group of 1928 in 2 groups of 1000
[INFO] [2019-10-11 23:09:55] Average Time: 0.105
[INFO] [2019-10-11 23:09:55] Total Time: 1s
[INFO] [2019-10-11 23:09:55] Storing 4752 TraitsReferences
[INFO] [2019-10-11 23:09:55] Processing group of 4752 in 5 groups of 1000
[INFO] [2019-10-11 23:09:55] Average Time: 0.08
[INFO] [2019-10-11 23:09:55] Total Time: 1s
[INFO] [2019-10-11 23:09:55] Storing 4751 Traits
[INFO] [2019-10-11 23:09:55] Processing group of 4751 in 5 groups of 1000
[INFO] [2019-10-11 23:09:57] Average Time: 0.362
[INFO] [2019-10-11 23:09:57] Total Time: 2s
[INFO] [2019-10-11 23:09:57] Storing 4746 MetaTraits
[INFO] [2019-10-11 23:09:57] Processing group of 4746 in 5 groups of 1000
[INFO] [2019-10-11 23:10:00] Average Time: 0.56
[INFO] [2019-10-11 23:10:00] Total Time: 3s
[STOP] [2019-10-11 23:10:00] parse_diff_and_store
[START] [2019-10-11 23:10:00] resolve_keys
[INFO] [2019-10-11 23:10:18] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-11 23:10:19] traits to occurrences...
[INFO] [2019-10-11 23:10:21] traits to nodes (through occurrences)...
[INFO] [2019-10-11 23:10:21] Traits to sex term...
[INFO] [2019-10-11 23:10:22] Traits to lifestage term...
[INFO] [2019-10-11 23:10:24] MetaTraits to traits...
[INFO] [2019-10-11 23:10:24] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-11 23:10:25] Assocs to occurrences...
[INFO] [2019-10-11 23:10:25] Assocs to nodes...
[INFO] [2019-10-11 23:10:25] Assoc to sex term...
[INFO] [2019-10-11 23:10:25] Assoc to lifestage term...
[STOP] [2019-10-11 23:10:25] resolve_keys
[START] [2019-10-11 23:10:25] hold_for_later_1
[STOP] [2019-10-11 23:10:25] hold_for_later_1
[START] [2019-10-11 23:10:25] hold_for_later_2
[STOP] [2019-10-11 23:10:25] hold_for_later_2
[START] [2019-10-11 23:10:25] resolve_missing_parents
[STOP] [2019-10-11 23:10:33] resolve_missing_parents
[START] [2019-10-11 23:10:33] rebuild_nodes
[START] [2019-10-11 23:10:33] Flattener#flatten
[START] [2019-10-11 23:10:33] Flattener#study_resource
[START] [2019-10-11 23:10:33] Flattener#build_ancestry
[STOP] [2019-10-11 23:10:33] Flattener#build_ancestry
[INFO] [2019-10-11 23:10:33] 3690 ancestry keys
[START] [2019-10-11 23:10:33] build_node_ancestors
[INFO] [2019-10-11 23:10:33] old ancestors deleted.
[STOP] [2019-10-11 23:10:33] build_node_ancestors
[START] [2019-10-11 23:10:34] Flattener#propagate_ancestor_ids
[STOP] [2019-10-11 23:10:34] Flattener#propagate_ancestor_ids
[STOP] [2019-10-11 23:10:34] Flattener#flatten
[STOP] [2019-10-11 23:10:34] rebuild_nodes
[START] [2019-10-11 23:10:34] resolve_missing_media_owners
[STOP] [2019-10-11 23:10:34] resolve_missing_media_owners
[START] [2019-10-11 23:10:34] sanitize_media_verbatims
[STOP] [2019-10-11 23:10:34] sanitize_media_verbatims
[START] [2019-10-11 23:10:34] queue_downloads
[STOP] [2019-10-11 23:10:34] queue_downloads
[START] [2019-10-11 23:10:34] parse_names
[WARN] [2019-10-11 23:10:34] I see 3690 names which still need to be parsed.
[STOP] [2019-10-11 23:10:37] parse_names
[START] [2019-10-11 23:10:37] denormalize_canonical_names_to_nodes
[STOP] [2019-10-11 23:10:37] denormalize_canonical_names_to_nodes
[START] [2019-10-11 23:10:37] match_nodes
[START] [2019-10-11 23:10:37] map_all_nodes_to_pages
[STOP] [2019-10-11 23:12:17] map_all_nodes_to_pages
[INFO] [2019-10-11 23:12:17] 170 Unmatched nodes (of 3690)! That's too many to output. First 10: Ctenopterella lobbiana (#48969801); Ctenopterella quinquefurcata (#48973011); Phymatosorus membranifolium (#48969832); Microsorum pteropus (#48971859); Drynaria (#48969846); Lepisorus mucronata (#48970225); Colysis macrophyllus (#48973454); Belvisia (#48970212); Leptochilus macrophylla (#48970242); Microgramma percussum (#48971083)
[START] [2019-10-11 23:12:17] update_nodes
[STOP] [2019-10-11 23:12:19] update_nodes
[STOP] [2019-10-11 23:12:19] match_nodes
[START] [2019-10-11 23:12:19] reindex_search
[STOP] [2019-10-11 23:12:25] reindex_search
[START] [2019-10-11 23:12:25] normalize_units
[STOP] [2019-10-11 23:12:25] normalize_units
[START] [2019-10-11 23:12:25] calculate_statistics
[STOP] [2019-10-11 23:12:25] calculate_statistics
[START] [2019-10-11 23:12:25] complete_harvest_instance
[START] [2019-10-11 23:12:25] overall_tsv_creation
[INFO] [2019-10-11 23:12:25] Processing group of 3690 in 1 batches of 10000
[INFO] [2019-10-11 23:13:27] 1928 Traits (unfiltered)...
[INFO] [2019-10-11 23:13:41] 1928 Traits (filtered)...
[INFO] [2019-10-11 23:13:41] 0 Associations (filtered)...
[INFO] [2019-10-11 23:14:22] 9635 metadata added.
[INFO] [2019-10-11 23:14:22] 0 metadata added.
[INFO] [2019-10-11 23:14:22] Average Time: 92.94
[INFO] [2019-10-11 23:14:22] Total Time: 1m57s
[STOP] [2019-10-11 23:14:22] overall_tsv_creation
[INFO] [2019-10-11 23:14:22] Done. Check your files:
[INFO] [2019-10-11 23:14:22] (3690 lines) /app/public/data/brunei_sp_list/publish_nodes.tsv
[INFO] [2019-10-11 23:14:22] (5628 lines) /app/public/data/brunei_sp_list/publish_node_ancestors.tsv
[INFO] [2019-10-11 23:14:22] (3690 lines) /app/public/data/brunei_sp_list/publish_scientific_names.tsv
[INFO] [2019-10-11 23:14:22] (1929 lines) /app/public/data/brunei_sp_list/publish_traits.tsv
[INFO] [2019-10-11 23:14:22] (9636 lines) /app/public/data/brunei_sp_list/publish_metadata.tsv
[STOP] [2019-10-11 23:14:22] complete_harvest_instance
[START] [2019-10-11 23:14:22] completed
[STOP] [2019-10-11 23:14:22] completed
[STOP] [2019-10-11 23:14:22] logged process, took 289.44
Latest Process

11 Oct 23:09 EXITED currently processing 0