Harvest for Bass Strait Species List Created 22 Dec 22:09

Stage: completed
Fetched: 22 Dec 22:09
Validated: 22 Dec 22:09
Deltas Created 22 Dec 22:09
Units Normalized: 22 Dec 22:47
Ancestry Built: 22 Dec 22:12
Nodes Matched: 22 Dec 22:46
Names Parsed: 22 Dec 22:12
New Models Stored: 22 Dec 22:10
Indexed: 22 Dec 22:47
Completed: 22 Dec 22:51
Time to Harvest: 1 minute
Harvesting Log

(156 lines)
# Logfile created on 2019-12-22 22:09:14 -0500 by logger.rb/56815
[START] [2019-12-22 22:09:14] logged process
[START] [2019-12-22 22:09:14] create_harvest_instance
[STOP] [2019-12-22 22:09:14] create_harvest_instance
[START] [2019-12-22 22:09:14] fetch_files
[STOP] [2019-12-22 22:09:14] fetch_files
[START] [2019-12-22 22:09:14] validate_each_file
[STOP] [2019-12-22 22:09:16] validate_each_file
[START] [2019-12-22 22:09:16] convert_to_csv
[CMD] [2019-12-22 22:09:16] /usr/bin/sort /app/public/converted_csv/bass_strait_sp_l_refs_19122.csv > /app/public/converted_csv/bass_strait_sp_l_refs_19122.csv_sorted
[CMD] [2019-12-22 22:09:16] /usr/bin/sort /app/public/converted_csv/bass_strait_sp_l_nodes_19123.csv > /app/public/converted_csv/bass_strait_sp_l_nodes_19123.csv_sorted
[CMD] [2019-12-22 22:09:16] /usr/bin/sort /app/public/converted_csv/bass_strait_sp_l_occurrences_19124.csv > /app/public/converted_csv/bass_strait_sp_l_occurrences_19124.csv_sorted
[CMD] [2019-12-22 22:09:16] /usr/bin/sort /app/public/converted_csv/bass_strait_sp_l_measurements_19125.csv > /app/public/converted_csv/bass_strait_sp_l_measurements_19125.csv_sorted
[STOP] [2019-12-22 22:09:16] convert_to_csv
[START] [2019-12-22 22:09:16] calculate_delta
[CMD] [2019-12-22 22:09:16] echo "0a" > /app/public/diff/bass_strait_sp_l_refs_19122.diff
[CMD] [2019-12-22 22:09:16] tail -n +1 /app/public/converted_csv/bass_strait_sp_l_refs_19122.csv >> /app/public/diff/bass_strait_sp_l_refs_19122.diff
[CMD] [2019-12-22 22:09:16] echo "." >> /app/public/diff/bass_strait_sp_l_refs_19122.diff
[CMD] [2019-12-22 22:09:16] echo "0a" > /app/public/diff/bass_strait_sp_l_nodes_19123.diff
[CMD] [2019-12-22 22:09:16] tail -n +1 /app/public/converted_csv/bass_strait_sp_l_nodes_19123.csv >> /app/public/diff/bass_strait_sp_l_nodes_19123.diff
[CMD] [2019-12-22 22:09:16] echo "." >> /app/public/diff/bass_strait_sp_l_nodes_19123.diff
[CMD] [2019-12-22 22:09:16] echo "0a" > /app/public/diff/bass_strait_sp_l_occurrences_19124.diff
[CMD] [2019-12-22 22:09:16] tail -n +1 /app/public/converted_csv/bass_strait_sp_l_occurrences_19124.csv >> /app/public/diff/bass_strait_sp_l_occurrences_19124.diff
[CMD] [2019-12-22 22:09:16] echo "." >> /app/public/diff/bass_strait_sp_l_occurrences_19124.diff
[CMD] [2019-12-22 22:09:16] echo "0a" > /app/public/diff/bass_strait_sp_l_measurements_19125.diff
[CMD] [2019-12-22 22:09:16] tail -n +1 /app/public/converted_csv/bass_strait_sp_l_measurements_19125.csv >> /app/public/diff/bass_strait_sp_l_measurements_19125.diff
[CMD] [2019-12-22 22:09:16] echo "." >> /app/public/diff/bass_strait_sp_l_measurements_19125.diff
[STOP] [2019-12-22 22:09:16] calculate_delta
[START] [2019-12-22 22:09:16] parse_diff_and_store
[INFO] [2019-12-22 22:09:16] Loading refs diff file into memory (true lines)...
[INFO] [2019-12-22 22:09:16] Loading nodes diff file into memory (true lines)...
[INFO] [2019-12-22 22:09:21] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-12-22 22:09:22] Loading measurements diff file into memory (true lines)...
[INFO] [2019-12-22 22:10:00] Storing 2 References
[INFO] [2019-12-22 22:10:00] Processing group of 2 in 1 groups of 1000
[INFO] [2019-12-22 22:10:00] Average Time: 0.0
[INFO] [2019-12-22 22:10:00] Total Time: 1s
[INFO] [2019-12-22 22:10:00] Storing 11870 ScientificNames
[INFO] [2019-12-22 22:10:00] Processing group of 11870 in 12 groups of 1000
[INFO] [2019-12-22 22:10:05] Average Time: 0.413
[INFO] [2019-12-22 22:10:05] Total Time: 6s
[INFO] [2019-12-22 22:10:05] last 3 / first 3: 0.83
[INFO] [2019-12-22 22:10:05] Std.Dev: 0.044721359549995794; Max: 0.48
[INFO] [2019-12-22 22:10:05] Storing 11870 Nodes
[INFO] [2019-12-22 22:10:05] Processing group of 11870 in 12 groups of 1000
[INFO] [2019-12-22 22:10:10] Average Time: 0.335
[INFO] [2019-12-22 22:10:10] Total Time: 5s
[INFO] [2019-12-22 22:10:10] last 3 / first 3: 0.96
[INFO] [2019-12-22 22:10:10] Std.Dev: 0.0; Max: 0.35
[INFO] [2019-12-22 22:10:10] Storing 6720 Occurrences
[INFO] [2019-12-22 22:10:10] Processing group of 6720 in 7 groups of 1000
[INFO] [2019-12-22 22:10:11] Average Time: 0.184
[INFO] [2019-12-22 22:10:11] Total Time: 2s
[INFO] [2019-12-22 22:10:11] last 3 / first 3: 0.46
[INFO] [2019-12-22 22:10:11] Std.Dev: 0.08944271909999159; Max: 0.31
[INFO] [2019-12-22 22:10:11] Storing 13440 TraitsReferences
[INFO] [2019-12-22 22:10:11] Processing group of 13440 in 14 groups of 1000
[INFO] [2019-12-22 22:10:12] Average Time: 0.086
[INFO] [2019-12-22 22:10:12] Total Time: 2s
[INFO] [2019-12-22 22:10:12] last 3 / first 3: 0.7
[INFO] [2019-12-22 22:10:12] Std.Dev: 0.0; Max: 0.12
[INFO] [2019-12-22 22:10:12] Storing 13440 Traits
[INFO] [2019-12-22 22:10:12] Processing group of 13440 in 14 groups of 1000
[INFO] [2019-12-22 22:10:17] Average Time: 0.353
[INFO] [2019-12-22 22:10:17] Total Time: 6s
[INFO] [2019-12-22 22:10:17] last 3 / first 3: 0.65
[INFO] [2019-12-22 22:10:17] Std.Dev: 0.07745966692414834; Max: 0.48
[INFO] [2019-12-22 22:10:17] Storing 13405 MetaTraits
[INFO] [2019-12-22 22:10:17] Processing group of 13405 in 14 groups of 1000
[INFO] [2019-12-22 22:10:19] Average Time: 0.158
[INFO] [2019-12-22 22:10:19] Total Time: 3s
[INFO] [2019-12-22 22:10:19] last 3 / first 3: 0.61
[INFO] [2019-12-22 22:10:19] Std.Dev: 0.03162277660168379; Max: 0.24
[STOP] [2019-12-22 22:10:19] parse_diff_and_store
[START] [2019-12-22 22:10:19] resolve_keys
[INFO] [2019-12-22 22:11:05] Occurrences to nodes (through scientific_names)...
[INFO] [2019-12-22 22:11:09] traits to occurrences...
[INFO] [2019-12-22 22:11:14] traits to nodes (through occurrences)...
[INFO] [2019-12-22 22:11:14] Traits to sex term...
[INFO] [2019-12-22 22:11:19] Traits to lifestage term...
[INFO] [2019-12-22 22:11:24] MetaTraits to traits...
[INFO] [2019-12-22 22:11:24] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-12-22 22:11:26] Assocs to occurrences...
[INFO] [2019-12-22 22:11:26] Assocs to nodes...
[INFO] [2019-12-22 22:11:26] Assoc to sex term...
[INFO] [2019-12-22 22:11:26] Assoc to lifestage term...
[STOP] [2019-12-22 22:11:26] resolve_keys
[START] [2019-12-22 22:11:26] hold_for_later_1
[STOP] [2019-12-22 22:11:26] hold_for_later_1
[START] [2019-12-22 22:11:26] hold_for_later_2
[STOP] [2019-12-22 22:11:26] hold_for_later_2
[START] [2019-12-22 22:11:26] resolve_missing_parents
[STOP] [2019-12-22 22:11:50] resolve_missing_parents
[START] [2019-12-22 22:11:50] rebuild_nodes
[START] [2019-12-22 22:11:50] Flattener#flatten
[START] [2019-12-22 22:11:50] Flattener#study_resource
[START] [2019-12-22 22:11:50] Flattener#build_ancestry
[STOP] [2019-12-22 22:11:52] Flattener#build_ancestry
[INFO] [2019-12-22 22:11:52] 11870 ancestry keys
[START] [2019-12-22 22:11:52] build_node_ancestors
[INFO] [2019-12-22 22:11:52] old ancestors deleted.
[STOP] [2019-12-22 22:11:57] build_node_ancestors
[START] [2019-12-22 22:12:01] Flattener#propagate_ancestor_ids
[STOP] [2019-12-22 22:12:03] Flattener#propagate_ancestor_ids
[STOP] [2019-12-22 22:12:03] Flattener#flatten
[STOP] [2019-12-22 22:12:03] rebuild_nodes
[START] [2019-12-22 22:12:03] resolve_missing_media_owners
[STOP] [2019-12-22 22:12:03] resolve_missing_media_owners
[START] [2019-12-22 22:12:03] sanitize_media_verbatims
[STOP] [2019-12-22 22:12:03] sanitize_media_verbatims
[START] [2019-12-22 22:12:03] queue_downloads
[STOP] [2019-12-22 22:12:03] queue_downloads
[START] [2019-12-22 22:12:03] parse_names
[WARN] [2019-12-22 22:12:03] I see 11870 names which still need to be parsed.
[STOP] [2019-12-22 22:12:13] parse_names
[START] [2019-12-22 22:12:13] denormalize_canonical_names_to_nodes
[STOP] [2019-12-22 22:12:13] denormalize_canonical_names_to_nodes
[START] [2019-12-22 22:12:13] match_nodes
[START] [2019-12-22 22:12:13] map_all_nodes_to_pages
[STOP] [2019-12-22 22:46:30] map_all_nodes_to_pages
[INFO] [2019-12-22 22:46:30] 619 Unmatched nodes (of 11870)! That's too many to output. First 10: Limicola (#61760156); Limicola falcinellus (#61760155); Philomachus (#61760694); Philomachus pugnax (#61760693); Procelsterna (#61767492); Procelsterna cerulea (#61767491); Puffinus tenuirostris (#61756235); Puffinus pacifica (#61756375); Puffinus griseus (#61758536); Puffinus pacificus (#61760438)
[START] [2019-12-22 22:46:30] update_nodes
[STOP] [2019-12-22 22:46:35] update_nodes
[STOP] [2019-12-22 22:46:35] match_nodes
[START] [2019-12-22 22:46:35] reindex_search
[STOP] [2019-12-22 22:47:03] reindex_search
[START] [2019-12-22 22:47:03] normalize_units
[STOP] [2019-12-22 22:47:03] normalize_units
[START] [2019-12-22 22:47:03] calculate_statistics
[STOP] [2019-12-22 22:47:03] calculate_statistics
[START] [2019-12-22 22:47:03] complete_harvest_instance
[START] [2019-12-22 22:47:03] overall_tsv_creation
[INFO] [2019-12-22 22:47:03] Processing group of 11870 in 2 batches of 10000
[INFO] [2019-12-22 22:48:36] 5476 Traits (unfiltered)...
[INFO] [2019-12-22 22:48:49] 5476 Traits (filtered)...
[INFO] [2019-12-22 22:48:49] 0 Associations (filtered)...
[INFO] [2019-12-22 22:49:40] 27349 metadata added.
[INFO] [2019-12-22 22:49:40] 0 metadata added.
[INFO] [2019-12-22 22:50:34] 1244 Traits (unfiltered)...
[INFO] [2019-12-22 22:50:47] 1244 Traits (filtered)...
[INFO] [2019-12-22 22:50:47] 0 Associations (filtered)...
[INFO] [2019-12-22 22:51:27] 6216 metadata added.
[INFO] [2019-12-22 22:51:27] 0 metadata added.
[INFO] [2019-12-22 22:51:27] Average Time: 105.445
[INFO] [2019-12-22 22:51:27] Total Time: 4m24s
[STOP] [2019-12-22 22:51:27] overall_tsv_creation
[INFO] [2019-12-22 22:51:27] Done. Check your files:
[INFO] [2019-12-22 22:51:27] (11870 lines) /app/public/data/bass_strait_sp_l/publish_nodes.tsv
[INFO] [2019-12-22 22:51:27] (62180 lines) /app/public/data/bass_strait_sp_l/publish_node_ancestors.tsv
[INFO] [2019-12-22 22:51:27] (11870 lines) /app/public/data/bass_strait_sp_l/publish_scientific_names.tsv
[INFO] [2019-12-22 22:51:27] (6721 lines) /app/public/data/bass_strait_sp_l/publish_traits.tsv
[INFO] [2019-12-22 22:51:27] (33566 lines) /app/public/data/bass_strait_sp_l/publish_metadata.tsv
[STOP] [2019-12-22 22:51:27] complete_harvest_instance
[START] [2019-12-22 22:51:27] completed
[STOP] [2019-12-22 22:51:27] completed
[STOP] [2019-12-22 22:51:27] logged process, took 2533.39
Latest Process

22 Dec 22:09 EXITED currently processing 0