Harvest for Iran Species List Created 13 Oct 17:55

Stage: completed
Fetched: 13 Oct 17:55
Validated: 13 Oct 17:55
Deltas Created 13 Oct 17:55
Units Normalized: 13 Oct 18:08
Ancestry Built: 13 Oct 17:58
Nodes Matched: 13 Oct 18:08
Names Parsed: 13 Oct 17:58
New Models Stored: 13 Oct 17:56
Indexed: 13 Oct 18:08
Completed: 13 Oct 18:12
Time to Harvest: less than a minute
Harvesting Log

(156 lines)
# Logfile created on 2019-10-13 17:55:26 -0400 by logger.rb/56815
[START] [2019-10-13 17:55:26] logged process
[START] [2019-10-13 17:55:26] create_harvest_instance
[STOP] [2019-10-13 17:55:26] create_harvest_instance
[START] [2019-10-13 17:55:26] fetch_files
[STOP] [2019-10-13 17:55:26] fetch_files
[START] [2019-10-13 17:55:26] validate_each_file
[STOP] [2019-10-13 17:55:28] validate_each_file
[START] [2019-10-13 17:55:28] convert_to_csv
[CMD] [2019-10-13 17:55:28] /usr/bin/sort /app/public/converted_csv/iran_sp_list_refs_16179.csv > /app/public/converted_csv/iran_sp_list_refs_16179.csv_sorted
[CMD] [2019-10-13 17:55:28] /usr/bin/sort /app/public/converted_csv/iran_sp_list_nodes_16180.csv > /app/public/converted_csv/iran_sp_list_nodes_16180.csv_sorted
[CMD] [2019-10-13 17:55:28] /usr/bin/sort /app/public/converted_csv/iran_sp_list_occurrences_16181.csv > /app/public/converted_csv/iran_sp_list_occurrences_16181.csv_sorted
[CMD] [2019-10-13 17:55:29] /usr/bin/sort /app/public/converted_csv/iran_sp_list_measurements_16182.csv > /app/public/converted_csv/iran_sp_list_measurements_16182.csv_sorted
[STOP] [2019-10-13 17:55:29] convert_to_csv
[START] [2019-10-13 17:55:29] calculate_delta
[CMD] [2019-10-13 17:55:29] echo "0a" > /app/public/diff/iran_sp_list_refs_16179.diff
[CMD] [2019-10-13 17:55:29] tail -n +1 /app/public/converted_csv/iran_sp_list_refs_16179.csv >> /app/public/diff/iran_sp_list_refs_16179.diff
[CMD] [2019-10-13 17:55:29] echo "." >> /app/public/diff/iran_sp_list_refs_16179.diff
[CMD] [2019-10-13 17:55:29] echo "0a" > /app/public/diff/iran_sp_list_nodes_16180.diff
[CMD] [2019-10-13 17:55:29] tail -n +1 /app/public/converted_csv/iran_sp_list_nodes_16180.csv >> /app/public/diff/iran_sp_list_nodes_16180.diff
[CMD] [2019-10-13 17:55:29] echo "." >> /app/public/diff/iran_sp_list_nodes_16180.diff
[CMD] [2019-10-13 17:55:29] echo "0a" > /app/public/diff/iran_sp_list_occurrences_16181.diff
[CMD] [2019-10-13 17:55:29] tail -n +1 /app/public/converted_csv/iran_sp_list_occurrences_16181.csv >> /app/public/diff/iran_sp_list_occurrences_16181.diff
[CMD] [2019-10-13 17:55:29] echo "." >> /app/public/diff/iran_sp_list_occurrences_16181.diff
[CMD] [2019-10-13 17:55:30] echo "0a" > /app/public/diff/iran_sp_list_measurements_16182.diff
[CMD] [2019-10-13 17:55:30] tail -n +1 /app/public/converted_csv/iran_sp_list_measurements_16182.csv >> /app/public/diff/iran_sp_list_measurements_16182.diff
[CMD] [2019-10-13 17:55:30] echo "." >> /app/public/diff/iran_sp_list_measurements_16182.diff
[STOP] [2019-10-13 17:55:30] calculate_delta
[START] [2019-10-13 17:55:30] parse_diff_and_store
[INFO] [2019-10-13 17:55:30] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-13 17:55:30] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-13 17:55:34] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-13 17:55:35] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-13 17:56:20] Storing 2 References
[INFO] [2019-10-13 17:56:20] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-13 17:56:20] Average Time: 0.0
[INFO] [2019-10-13 17:56:20] Total Time: 1s
[INFO] [2019-10-13 17:56:20] Storing 11048 ScientificNames
[INFO] [2019-10-13 17:56:20] Processing group of 11048 in 12 groups of 1000
[INFO] [2019-10-13 17:56:24] Average Time: 0.335
[INFO] [2019-10-13 17:56:24] Total Time: 5s
[INFO] [2019-10-13 17:56:24] last 3 / first 3: 0.62
[INFO] [2019-10-13 17:56:24] Std.Dev: 0.10488088481701516; Max: 0.44
[INFO] [2019-10-13 17:56:24] Storing 11048 Nodes
[INFO] [2019-10-13 17:56:24] Processing group of 11048 in 12 groups of 1000
[INFO] [2019-10-13 17:56:28] Average Time: 0.316
[INFO] [2019-10-13 17:56:28] Total Time: 4s
[INFO] [2019-10-13 17:56:28] last 3 / first 3: 0.69
[INFO] [2019-10-13 17:56:28] Std.Dev: 0.15491933384829668; Max: 0.72
[INFO] [2019-10-13 17:56:28] Storing 7230 Occurrences
[INFO] [2019-10-13 17:56:28] Processing group of 7230 in 8 groups of 1000
[INFO] [2019-10-13 17:56:28] Average Time: 0.086
[INFO] [2019-10-13 17:56:28] Total Time: 1s
[INFO] [2019-10-13 17:56:28] last 3 / first 3: 0.79
[INFO] [2019-10-13 17:56:28] Std.Dev: 0.03162277660168379; Max: 0.1
[INFO] [2019-10-13 17:56:28] Storing 15016 TraitsReferences
[INFO] [2019-10-13 17:56:28] Processing group of 15016 in 16 groups of 1000
[INFO] [2019-10-13 17:56:30] Average Time: 0.071
[INFO] [2019-10-13 17:56:30] Total Time: 2s
[INFO] [2019-10-13 17:56:30] last 3 / first 3: 0.45
[INFO] [2019-10-13 17:56:30] Std.Dev: 0.03162277660168379; Max: 0.15
[INFO] [2019-10-13 17:56:30] Storing 15015 Traits
[INFO] [2019-10-13 17:56:30] Processing group of 15015 in 16 groups of 1000
[INFO] [2019-10-13 17:56:35] Average Time: 0.321
[INFO] [2019-10-13 17:56:35] Total Time: 6s
[INFO] [2019-10-13 17:56:35] last 3 / first 3: 0.67
[INFO] [2019-10-13 17:56:35] Std.Dev: 0.09486832980505137; Max: 0.47
[INFO] [2019-10-13 17:56:35] Storing 15007 MetaTraits
[INFO] [2019-10-13 17:56:35] Processing group of 15007 in 16 groups of 1000
[INFO] [2019-10-13 17:56:38] Average Time: 0.173
[INFO] [2019-10-13 17:56:38] Total Time: 3s
[INFO] [2019-10-13 17:56:38] last 3 / first 3: 0.28
[INFO] [2019-10-13 17:56:38] Std.Dev: 0.11832159566199232; Max: 0.58
[STOP] [2019-10-13 17:56:38] parse_diff_and_store
[START] [2019-10-13 17:56:38] resolve_keys
[INFO] [2019-10-13 17:57:26] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-13 17:57:29] traits to occurrences...
[INFO] [2019-10-13 17:57:35] traits to nodes (through occurrences)...
[INFO] [2019-10-13 17:57:35] Traits to sex term...
[INFO] [2019-10-13 17:57:40] Traits to lifestage term...
[INFO] [2019-10-13 17:57:45] MetaTraits to traits...
[INFO] [2019-10-13 17:57:46] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-13 17:57:48] Assocs to occurrences...
[INFO] [2019-10-13 17:57:48] Assocs to nodes...
[INFO] [2019-10-13 17:57:48] Assoc to sex term...
[INFO] [2019-10-13 17:57:48] Assoc to lifestage term...
[STOP] [2019-10-13 17:57:48] resolve_keys
[START] [2019-10-13 17:57:48] hold_for_later_1
[STOP] [2019-10-13 17:57:48] hold_for_later_1
[START] [2019-10-13 17:57:48] hold_for_later_2
[STOP] [2019-10-13 17:57:48] hold_for_later_2
[START] [2019-10-13 17:57:48] resolve_missing_parents
[STOP] [2019-10-13 17:58:10] resolve_missing_parents
[START] [2019-10-13 17:58:10] rebuild_nodes
[START] [2019-10-13 17:58:10] Flattener#flatten
[START] [2019-10-13 17:58:10] Flattener#study_resource
[START] [2019-10-13 17:58:10] Flattener#build_ancestry
[STOP] [2019-10-13 17:58:11] Flattener#build_ancestry
[INFO] [2019-10-13 17:58:11] 11048 ancestry keys
[START] [2019-10-13 17:58:11] build_node_ancestors
[INFO] [2019-10-13 17:58:11] old ancestors deleted.
[STOP] [2019-10-13 17:58:14] build_node_ancestors
[START] [2019-10-13 17:58:17] Flattener#propagate_ancestor_ids
[STOP] [2019-10-13 17:58:18] Flattener#propagate_ancestor_ids
[STOP] [2019-10-13 17:58:18] Flattener#flatten
[STOP] [2019-10-13 17:58:18] rebuild_nodes
[START] [2019-10-13 17:58:18] resolve_missing_media_owners
[STOP] [2019-10-13 17:58:18] resolve_missing_media_owners
[START] [2019-10-13 17:58:18] sanitize_media_verbatims
[STOP] [2019-10-13 17:58:18] sanitize_media_verbatims
[START] [2019-10-13 17:58:18] queue_downloads
[STOP] [2019-10-13 17:58:18] queue_downloads
[START] [2019-10-13 17:58:18] parse_names
[WARN] [2019-10-13 17:58:18] I see 11048 names which still need to be parsed.
[STOP] [2019-10-13 17:58:27] parse_names
[START] [2019-10-13 17:58:27] denormalize_canonical_names_to_nodes
[STOP] [2019-10-13 17:58:27] denormalize_canonical_names_to_nodes
[START] [2019-10-13 17:58:27] match_nodes
[START] [2019-10-13 17:58:27] map_all_nodes_to_pages
[STOP] [2019-10-13 18:08:00] map_all_nodes_to_pages
[INFO] [2019-10-13 18:08:00] 1129 Unmatched nodes (of 11048)! That's too many to output. First 10: Myoxus (#50296357); Vulpes rueppelli (#50298332); Erinaceomorpha (#50289149); Hordeum violaceum (#50296767); Aegilops squarrosum (#50289622); Aegilops crassum (#50292299); Festuca pratensis (#50295705); Eremopyrum longiglume (#50293474); Poa asperrima (#50298904); Bromus gracilis (#50295364)
[START] [2019-10-13 18:08:00] update_nodes
[STOP] [2019-10-13 18:08:03] update_nodes
[STOP] [2019-10-13 18:08:03] match_nodes
[START] [2019-10-13 18:08:03] reindex_search
[STOP] [2019-10-13 18:08:29] reindex_search
[START] [2019-10-13 18:08:29] normalize_units
[STOP] [2019-10-13 18:08:29] normalize_units
[START] [2019-10-13 18:08:29] calculate_statistics
[STOP] [2019-10-13 18:08:29] calculate_statistics
[START] [2019-10-13 18:08:29] complete_harvest_instance
[START] [2019-10-13 18:08:29] overall_tsv_creation
[INFO] [2019-10-13 18:08:29] Processing group of 11048 in 2 batches of 10000
[INFO] [2019-10-13 18:09:58] 6731 Traits (unfiltered)...
[INFO] [2019-10-13 18:10:12] 6731 Traits (filtered)...
[INFO] [2019-10-13 18:10:12] 0 Associations (filtered)...
[INFO] [2019-10-13 18:11:04] 33651 metadata added.
[INFO] [2019-10-13 18:11:04] 0 metadata added.
[INFO] [2019-10-13 18:11:53] 499 Traits (unfiltered)...
[INFO] [2019-10-13 18:12:06] 499 Traits (filtered)...
[INFO] [2019-10-13 18:12:06] 0 Associations (filtered)...
[INFO] [2019-10-13 18:12:45] 2490 metadata added.
[INFO] [2019-10-13 18:12:45] 0 metadata added.
[INFO] [2019-10-13 18:12:45] Average Time: 103.625
[INFO] [2019-10-13 18:12:45] Total Time: 4m16s
[STOP] [2019-10-13 18:12:45] overall_tsv_creation
[INFO] [2019-10-13 18:12:45] Done. Check your files:
[INFO] [2019-10-13 18:12:45] (11048 lines) /app/public/data/iran_sp_list/publish_nodes.tsv
[INFO] [2019-10-13 18:12:45] (38002 lines) /app/public/data/iran_sp_list/publish_node_ancestors.tsv
[INFO] [2019-10-13 18:12:45] (11048 lines) /app/public/data/iran_sp_list/publish_scientific_names.tsv
[INFO] [2019-10-13 18:12:45] (7231 lines) /app/public/data/iran_sp_list/publish_traits.tsv
[INFO] [2019-10-13 18:12:45] (36142 lines) /app/public/data/iran_sp_list/publish_metadata.tsv
[STOP] [2019-10-13 18:12:45] complete_harvest_instance
[START] [2019-10-13 18:12:45] completed
[STOP] [2019-10-13 18:12:45] completed
[STOP] [2019-10-13 18:12:45] logged process, took 1039.78
Latest Process

13 Oct 17:55 EXITED currently processing 0