Harvest for NMNH type specimens Created 17 Dec 10:20

Harvesting Log

[INFO] [2021-12-17 10:20:42] Created harvest instance #4083
[STOP] [2021-12-17 10:20:42] create_harvest_instance
[START] [2021-12-17 10:20:42] fetch_files
[STOP] [2021-12-17 10:20:42] fetch_files
[START] [2021-12-17 10:20:42] validate_each_file
[INFO] [2021-12-17 10:20:42] Looping over 3 formats...
[INFO] [2021-12-17 10:20:42] ...nodes (/app/public/data/NMNHtypes/taxon.tab)
[INFO] [2021-12-17 10:20:52] Valid: /app/public/converted_csv/NMNHtypes_nodes_4083.csv (259695 lines)
[INFO] [2021-12-17 10:20:52] ...occurrences (/app/public/data/NMNHtypes/occurrence.tab)
[INFO] [2021-12-17 10:21:20] Valid: /app/public/converted_csv/NMNHtypes_occurrences_4083.csv (359251 lines)
[INFO] [2021-12-17 10:21:20] ...measurements (/app/public/data/NMNHtypes/measurement_or_fact.tab)
[INFO] [2021-12-17 10:24:22] Valid: /app/public/converted_csv/NMNHtypes_measurements_4083.csv (4330055 lines)
[STOP] [2021-12-17 10:24:22] validate_each_file
[START] [2021-12-17 10:24:23] convert_to_csv
[INFO] [2021-12-17 10:24:23] Looping over 3 formats...
[INFO] [2021-12-17 10:24:23] ...nodes (/app/public/data/NMNHtypes/taxon.tab)
[CMD] [2021-12-17 10:24:23] /usr/bin/sort /app/public/converted_csv/NMNHtypes_nodes_4083.csv > /app/public/converted_csv/NMNHtypes_nodes_4083.csv_sorted
[INFO] [2021-12-17 10:24:23] Converted: /app/public/converted_csv/NMNHtypes_nodes_4083.csv (259695 lines)
[INFO] [2021-12-17 10:24:23] ...occurrences (/app/public/data/NMNHtypes/occurrence.tab)
[CMD] [2021-12-17 10:24:23] /usr/bin/sort /app/public/converted_csv/NMNHtypes_occurrences_4083.csv > /app/public/converted_csv/NMNHtypes_occurrences_4083.csv_sorted
[INFO] [2021-12-17 10:24:23] Converted: /app/public/converted_csv/NMNHtypes_occurrences_4083.csv (359251 lines)
[INFO] [2021-12-17 10:24:23] ...measurements (/app/public/data/NMNHtypes/measurement_or_fact.tab)
[CMD] [2021-12-17 10:24:23] /usr/bin/sort /app/public/converted_csv/NMNHtypes_measurements_4083.csv > /app/public/converted_csv/NMNHtypes_measurements_4083.csv_sorted
[INFO] [2021-12-17 10:24:29] Converted: /app/public/converted_csv/NMNHtypes_measurements_4083.csv (4330055 lines)
[STOP] [2021-12-17 10:24:29] convert_to_csv
[START] [2021-12-17 10:24:29] calculate_delta
[INFO] [2021-12-17 10:24:29] Looping over 3 formats...
[INFO] [2021-12-17 10:24:29] ...nodes (/app/public/data/NMNHtypes/taxon.tab)
[CMD] [2021-12-17 10:24:29] echo "0a" > /app/public/diff/NMNHtypes_nodes_4083.diff
[CMD] [2021-12-17 10:24:29] tail -n +1 /app/public/converted_csv/NMNHtypes_nodes_4083.csv >> /app/public/diff/NMNHtypes_nodes_4083.diff
[CMD] [2021-12-17 10:24:29] echo "." >> /app/public/diff/NMNHtypes_nodes_4083.diff
[INFO] [2021-12-17 10:24:29] Created diff: /app/public/diff/NMNHtypes_nodes_4083.diff (259697 lines)
[INFO] [2021-12-17 10:24:29] ...occurrences (/app/public/data/NMNHtypes/occurrence.tab)
[CMD] [2021-12-17 10:24:29] echo "0a" > /app/public/diff/NMNHtypes_occurrences_4083.diff
[CMD] [2021-12-17 10:24:29] tail -n +1 /app/public/converted_csv/NMNHtypes_occurrences_4083.csv >> /app/public/diff/NMNHtypes_occurrences_4083.diff
[CMD] [2021-12-17 10:24:29] echo "." >> /app/public/diff/NMNHtypes_occurrences_4083.diff
[INFO] [2021-12-17 10:24:30] Created diff: /app/public/diff/NMNHtypes_occurrences_4083.diff (359253 lines)
[INFO] [2021-12-17 10:24:30] ...measurements (/app/public/data/NMNHtypes/measurement_or_fact.tab)
[CMD] [2021-12-17 10:24:30] echo "0a" > /app/public/diff/NMNHtypes_measurements_4083.diff
[CMD] [2021-12-17 10:24:31] tail -n +1 /app/public/converted_csv/NMNHtypes_measurements_4083.csv >> /app/public/diff/NMNHtypes_measurements_4083.diff
[CMD] [2021-12-17 10:24:31] echo "." >> /app/public/diff/NMNHtypes_measurements_4083.diff
[INFO] [2021-12-17 10:24:34] Created diff: /app/public/diff/NMNHtypes_measurements_4083.diff (4330057 lines)
[STOP] [2021-12-17 10:24:34] calculate_delta
[START] [2021-12-17 10:24:34] parse_diff_and_store
[INFO] [2021-12-17 10:24:35] Handling diff: /app/public/diff/NMNHtypes_nodes_4083.diff (259697 lines)
[INFO] [2021-12-17 10:24:35] Loading nodes diff file into memory (259697 /app/public/diff/NMNHtypes_nodes_4083.diff lines)...
[WARN] [2021-12-17 10:24:45] Filtered Scientific Name `Carpodectes antoniae "Zeledon (Ms.)" Ridgway` to `Carpodectes antoniae Zeledon (Ms.) Ridgway`
[WARN] [2021-12-17 10:25:01] Filtered Scientific Name `Cotinga ridgwayi "Zeledon (Ms.)" Ridgway` to `Cotinga ridgwayi Zeledon (Ms.) Ridgway`
[WARN] [2021-12-17 10:25:11] Filtered Scientific Name `Signiphora blacella / basilica Girault` to `Signiphora blacella basilica Girault`
[WARN] [2021-12-17 10:25:15] Filtered Scientific Name `Pheucticus tibialis "Baird (Ms.)" Lawrence` to `Pheucticus tibialis Baird (Ms.) Lawrence`
[WARN] [2021-12-17 10:25:32] Filtered Scientific Name `Rallus elegans tenuirostris "Lawrence (Ms.)" Ridgway` to `Rallus elegans tenuirostris Lawrence (Ms.) Ridgway`
[WARN] [2021-12-17 10:25:33] Filtered Scientific Name `Rhodinocichla rosea schistacea "Ridgway (Ms.)" Ridgway` to `Rhodinocichla rosea schistacea Ridgway (Ms.) Ridgway`
[WARN] [2021-12-17 10:25:34] Filtered Scientific Name `Thryothorus felix lawrencii "Ridgway (Ms.)" Ridgway` to `Thryothorus felix lawrencii Ridgway (Ms.) Ridgway`
[WARN] [2021-12-17 10:25:43] Filtered Scientific Name `Puffinus creatopus "Cooper (Ms.)" (Ms.)" & Coues` to `Puffinus creatopus Cooper (Ms.) (Ms.) & Coues`
[WARN] [2021-12-17 10:25:56] Filtered Scientific Name `Spizella atrogularis cana "Coues (Ms.)" Grinnell & Swarth` to `Spizella atrogularis cana Coues (Ms.) Grinnell & Swarth`
[WARN] [2021-12-17 10:25:58] Filtered Scientific Name `Dendroica dominica albilora "Baird (Ms.)" Ridgway` to `Dendroica dominica albilora Baird (Ms.) Ridgway`
[WARN] [2021-12-17 10:25:58] Filtered Scientific Name `Perisoreus canadensis capitalis "Baird (Ms.)" Ridgway` to `Perisoreus canadensis capitalis Baird (Ms.) Ridgway`
[WARN] [2021-12-17 10:26:13] Filtered Scientific Name `Aceros corrugatus "megistus"` to `Aceros corrugatus megistus`
[WARN] [2021-12-17 10:26:16] Filtered Scientific Name `Pseudorhabdosynochus "cupatum"group` to `Pseudorhabdosynochus cupatumgroup`
[WARN] [2021-12-17 10:26:27] Filtered Scientific Name `Icterus pectoralis espinachi "Nutting (Ms.)" Ridgway` to `Icterus pectoralis espinachi Nutting (Ms.) Ridgway`
[WARN] [2021-12-17 10:26:30] Filtered Scientific Name `Neorhopus australicus/aureus Girault` to `Neorhopus australicusaureus Girault`
[WARN] [2021-12-17 10:26:32] Filtered Scientific Name `Tinamus guttatus "Natterer (Ms.)" Von Pelzeln` to `Tinamus guttatus Natterer (Ms.) Von Pelzeln`
[INFO] [2021-12-17 10:26:37] Handling diff: /app/public/diff/NMNHtypes_occurrences_4083.diff (359253 lines)
[INFO] [2021-12-17 10:26:38] Loading occurrences diff file into memory (359253 /app/public/diff/NMNHtypes_occurrences_4083.diff lines)...
[INFO] [2021-12-17 10:37:22] Handling diff: /app/public/diff/NMNHtypes_measurements_4083.diff (4330057 lines)
[INFO] [2021-12-17 10:37:22] Loading measurements diff file into memory (4330057 /app/public/diff/NMNHtypes_measurements_4083.diff lines)...
[INFO] [2021-12-17 12:40:50] Storing 313115 ScientificNames
[INFO] [2021-12-17 12:40:50] Processing group of 313115 in 314 groups of 1000
[INFO] [2021-12-17 12:43:29] Average Time: 0.5
[INFO] [2021-12-17 12:43:29] Total Time: 2m39s
[INFO] [2021-12-17 12:43:29] last 3 / first 3: 0.54
[INFO] [2021-12-17 12:43:29] Std.Dev: 1.1878552100319297; Max: 12.3
[INFO] [2021-12-17 12:43:29] Storing 313115 Nodes
[INFO] [2021-12-17 12:43:29] Processing group of 313115 in 314 groups of 1000
[INFO] [2021-12-17 12:50:00] Average Time: 1.24
[INFO] [2021-12-17 12:50:00] Total Time: 6m32s
[INFO] [2021-12-17 12:50:00] last 3 / first 3: 0.28
[INFO] [2021-12-17 12:50:00] Std.Dev: 5.1536394906900505; Max: 47.44
[INFO] [2021-12-17 12:50:00] Storing 359251 Occurrences
[INFO] [2021-12-17 12:50:00] Processing group of 359251 in 360 groups of 1000
[INFO] [2021-12-17 12:52:31] Average Time: 0.414
[INFO] [2021-12-17 12:52:31] Total Time: 2m32s
[INFO] [2021-12-17 12:52:31] last 3 / first 3: 0.29
[INFO] [2021-12-17 12:52:31] Std.Dev: 1.396424004376894; Max: 9.09
[INFO] [2021-12-17 12:52:31] Storing 1826471 OccurrenceMetadata
[INFO] [2021-12-17 12:52:31] Processing group of 1826471 in 1827 groups of 1000
[INFO] [2021-12-17 12:59:25] Average Time: 0.22
[INFO] [2021-12-17 12:59:25] Total Time: 6m54s
[INFO] [2021-12-17 12:59:25] last 3 / first 3: 0.64
[INFO] [2021-12-17 12:59:25] Std.Dev: 0.74565407529229; Max: 9.08
[INFO] [2021-12-17 12:59:25] Storing 4330055 Traits
[INFO] [2021-12-17 12:59:25] Processing group of 4330055 in 4331 groups of 1000
[INFO] [2021-12-17 13:50:10] Average Time: 0.696
[INFO] [2021-12-17 13:50:10] Total Time: 50m45s
[INFO] [2021-12-17 13:50:10] last 3 / first 3: 0.41
[INFO] [2021-12-17 13:50:10] Std.Dev: 2.204313952230943; Max: 19.53
[INFO] [2021-12-17 13:50:10] Storing 25443 MetaTraits
[INFO] [2021-12-17 13:50:10] Processing group of 25443 in 26 groups of 1000
[INFO] [2021-12-17 13:50:14] Average Time: 0.152
[INFO] [2021-12-17 13:50:14] Total Time: 5s
[INFO] [2021-12-17 13:50:14] last 3 / first 3: 0.32
[INFO] [2021-12-17 13:50:14] Std.Dev: 0.13784048752090222; Max: 0.62
[STOP] [2021-12-17 13:50:14] parse_diff_and_store
[START] [2021-12-17 13:50:14] resolve_keys
[INFO] [2021-12-17 13:51:42] Occurrences to nodes (through scientific_names)...
[INFO] [2021-12-17 13:52:23] traits to occurrences...
[INFO] [2021-12-17 13:52:43] traits to nodes (through occurrences)...
[INFO] [2021-12-17 13:52:58] Traits to sex term...
[INFO] [2021-12-17 13:53:10] Traits to lifestage term...
[INFO] [2021-12-17 13:53:21] MetaTraits to traits...
[INFO] [2021-12-17 13:53:22] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-12-17 13:55:40] Assocs to occurrences...
[INFO] [2021-12-17 13:55:40] Assocs to nodes...
[INFO] [2021-12-17 13:55:40] Assoc to sex term...
[INFO] [2021-12-17 13:55:40] Assoc to lifestage term...
[INFO] [2021-12-17 13:55:40] MetaAssoc to assocs...
[STOP] [2021-12-17 13:55:40] resolve_keys
[START] [2021-12-17 13:55:40] hold_for_later_1
[STOP] [2021-12-17 13:55:40] hold_for_later_1
[START] [2021-12-17 13:55:40] hold_for_later_2
[STOP] [2021-12-17 13:55:40] hold_for_later_2
[START] [2021-12-17 13:55:40] resolve_missing_parents
[STOP] [2021-12-17 13:55:51] resolve_missing_parents
[START] [2021-12-17 13:55:51] rebuild_nodes
[START] [2021-12-17 13:55:51] Flattener#flatten
[START] [2021-12-17 13:55:51] Flattener#study_resource
[START] [2021-12-17 13:56:15] Flattener#build_ancestry
[STOP] [2021-12-17 14:02:43] Flattener#build_ancestry
[INFO] [2021-12-17 14:02:43] 313115 ancestry keys
[START] [2021-12-17 14:02:43] build_node_ancestors
[INFO] [2021-12-17 14:02:43] old ancestors deleted.
[STOP] [2021-12-17 14:05:28] build_node_ancestors
[START] [2021-12-17 14:05:34] Flattener#propagate_ancestor_ids
[STOP] [2021-12-17 14:06:17] Flattener#propagate_ancestor_ids
[STOP] [2021-12-17 14:06:17] Flattener#flatten
[STOP] [2021-12-17 14:06:17] rebuild_nodes
[START] [2021-12-17 14:06:17] resolve_missing_media_owners
[STOP] [2021-12-17 14:06:17] resolve_missing_media_owners
[START] [2021-12-17 14:06:17] sanitize_media_verbatims
[STOP] [2021-12-17 14:06:17] sanitize_media_verbatims
[START] [2021-12-17 14:06:17] queue_downloads
[STOP] [2021-12-17 14:06:17] queue_downloads
[START] [2021-12-17 14:06:17] parse_names
[WARN] [2021-12-17 14:06:18] I see 313115 names which still need to be parsed.
[WARN] [2021-12-17 14:06:19] Names to parse: 10000 formatted: 10000 learned: 9972 parsed: 10000
[WARN] [2021-12-17 14:06:29] Names to parse: 10000 formatted: 10000 learned: 9984 parsed: 10000
[WARN] [2021-12-17 14:06:37] Names to parse: 10000 formatted: 10000 learned: 9996 parsed: 10000
[WARN] [2021-12-17 14:06:45] Names to parse: 10000 formatted: 10000 learned: 9993 parsed: 10000
[WARN] [2021-12-17 14:06:54] Names to parse: 10000 formatted: 10000 learned: 9993 parsed: 10000
[WARN] [2021-12-17 14:07:02] Names to parse: 10000 formatted: 10000 learned: 9997 parsed: 10000
[WARN] [2021-12-17 14:07:10] Names to parse: 10000 formatted: 10000 learned: 9994 parsed: 10000
[WARN] [2021-12-17 14:07:19] Names to parse: 10000 formatted: 10000 learned: 9998 parsed: 10000
[WARN] [2021-12-17 14:07:27] Names to parse: 10000 formatted: 10000 learned: 9989 parsed: 10000
[WARN] [2021-12-17 14:07:35] Names to parse: 10000 formatted: 10000 learned: 9999 parsed: 10000
[WARN] [2021-12-17 14:07:43] Names to parse: 10000 formatted: 10000 learned: 9999 parsed: 10000
[WARN] [2021-12-17 14:07:51] Names to parse: 10000 formatted: 10000 learned: 9995 parsed: 10000
[WARN] [2021-12-17 14:07:59] Names to parse: 10000 formatted: 10000 learned: 9997 parsed: 10000
[WARN] [2021-12-17 14:08:07] Names to parse: 10000 formatted: 10000 learned: 9998 parsed: 10000
[WARN] [2021-12-17 14:08:16] Names to parse: 10000 formatted: 10000 learned: 9998 parsed: 10000
[WARN] [2021-12-17 14:08:24] Names to parse: 10000 formatted: 10000 learned: 9998 parsed: 10000
[WARN] [2021-12-17 14:08:31] Names to parse: 10000 formatted: 10000 learned: 9999 parsed: 10000
[WARN] [2021-12-17 14:08:40] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2021-12-17 14:08:48] Names to parse: 10000 formatted: 10000 learned: 9999 parsed: 10000
[WARN] [2021-12-17 14:08:56] Names to parse: 10000 formatted: 10000 learned: 9999 parsed: 10000
[WARN] [2021-12-17 14:09:04] Names to parse: 10000 formatted: 10000 learned: 9999 parsed: 10000
[WARN] [2021-12-17 14:09:12] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2021-12-17 14:09:19] Names to parse: 10000 formatted: 10000 learned: 9999 parsed: 10000
[WARN] [2021-12-17 14:09:27] Names to parse: 10000 formatted: 10000 learned: 9999 parsed: 10000
[WARN] [2021-12-17 14:09:35] Names to parse: 10000 formatted: 10000 learned: 9999 parsed: 10000
[WARN] [2021-12-17 14:09:43] Names to parse: 10000 formatted: 10000 learned: 9998 parsed: 10000
[WARN] [2021-12-17 14:09:51] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2021-12-17 14:09:59] Names to parse: 10000 formatted: 10000 learned: 9999 parsed: 10000
[WARN] [2021-12-17 14:10:07] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2021-12-17 14:10:15] Names to parse: 10000 formatted: 10000 learned: 9997 parsed: 10000
[WARN] [2021-12-17 14:10:23] Names to parse: 10000 formatted: 10000 learned: 9997 parsed: 10000
[WARN] [2021-12-17 14:10:30] Names to parse: 3115 formatted: 3115 learned: 3114 parsed: 3115
[STOP] [2021-12-17 14:10:33] parse_names
[START] [2021-12-17 14:10:33] denormalize_canonical_names_to_nodes
[STOP] [2021-12-17 14:10:45] denormalize_canonical_names_to_nodes
[START] [2021-12-17 14:10:45] match_nodes
[START] [2021-12-17 14:10:45] map_all_nodes_to_pages
[STOP] [2021-12-17 17:55:00] map_all_nodes_to_pages
[INFO] [2021-12-17 17:55:00] 62685 Unmatched nodes (of 313115)! That's too many to output. Full list in /app/public/data/NMNHtypes/unmatched_nodes.txt ; First 10: Canonical: Heteropogon crabroides; Node#101348443; ResourceID: 0f91320f8b4e5333689badb9c2457ded; Canonical: Heteropogon secundus; Node#101413765; ResourceID: 40e448b01a0416682324067d60d83cb8; Canonical: Heteropogon villosus apogynus; Node#101476312; ResourceID: 74af57ddac4fbc6cb9346aa09e7c201f; Canonical: Heteropogon stipoides; Node#101477514; ResourceID: 75b9c26a99c7861e5ba5b356da1daed6; Canonical: Heteropogon villosus dactyloides; Node#101482618; ResourceID: 7a0763bfb6e181ba179183968654a740; Canonical: Heteropogon villosus selloanus; Node#101484051; ResourceID: 7b53c5cfabb47cb0510ca576ff79bef9; Canonical: Heteropogon villosus; Node#101542764; ResourceID: ae256bb4e0b337a0b7d693c851a7d8ef; Canonical: Heteropogon fischerianus; Node#101599338; ResourceID: e06505e5939db91f6b70b971005d5707; Canonical: Heteropogon villosus gardneri; Node#101632008; ResourceID: fdcedf6c67ed300fabe895b15da0da28; Canonical: Ospriocerus; Node#101328681; ResourceID: Ospriocerus
[START] [2021-12-17 17:55:00] update_nodes
[STOP] [2021-12-17 17:55:05] update_nodes
[STOP] [2021-12-17 17:55:05] match_nodes
[START] [2021-12-17 17:55:05] reindex_search
[STOP] [2021-12-17 18:00:21] reindex_search
[START] [2021-12-17 18:00:21] normalize_units
[STOP] [2021-12-17 18:00:59] normalize_units
[START] [2021-12-17 18:00:59] calculate_statistics
[STOP] [2021-12-17 18:01:16] calculate_statistics
[START] [2021-12-17 18:01:16] complete_harvest_instance
[START] [2021-12-17 18:01:16] overall_tsv_creation
[INFO] [2021-12-17 18:01:16] Processing group of 313115 in 32 batches of 10000
[INFO] [2021-12-17 18:03:36] 6674 Traits (unfiltered)...
[INFO] [2021-12-17 18:05:29] 6674 Traits (filtered)...
[INFO] [2021-12-17 18:05:34] 0 Associations (filtered)...
[INFO] [2021-12-17 18:05:45] 105658 metadata added.
[INFO] [2021-12-17 18:05:45] 0 metadata added.
[INFO] [2021-12-17 18:07:50] 8539 Traits (unfiltered)...
[INFO] [2021-12-17 18:10:05] 8539 Traits (filtered)...
[INFO] [2021-12-17 18:10:09] 0 Associations (filtered)...
[INFO] [2021-12-17 18:10:21] 134385 metadata added.
[INFO] [2021-12-17 18:10:21] 0 metadata added.
[INFO] [2021-12-17 18:12:18] 9119 Traits (unfiltered)...
[INFO] [2021-12-17 18:14:42] 9119 Traits (filtered)...
[INFO] [2021-12-17 18:14:46] 0 Associations (filtered)...
[INFO] [2021-12-17 18:15:01] 143108 metadata added.
[INFO] [2021-12-17 18:15:01] 0 metadata added.
[INFO] [2021-12-17 18:17:00] 9880 Traits (unfiltered)...
[INFO] [2021-12-17 18:19:26] 9880 Traits (filtered)...
[INFO] [2021-12-17 18:19:29] 0 Associations (filtered)...
[INFO] [2021-12-17 18:19:48] 154711 metadata added.
[INFO] [2021-12-17 18:19:48] 0 metadata added.
[INFO] [2021-12-17 18:21:45] 10188 Traits (unfiltered)...
[INFO] [2021-12-17 18:24:20] 10188 Traits (filtered)...
[INFO] [2021-12-17 18:24:24] 0 Associations (filtered)...
[INFO] [2021-12-17 18:24:42] 158956 metadata added.
[INFO] [2021-12-17 18:24:42] 0 metadata added.
[INFO] [2021-12-17 18:26:41] 10787 Traits (unfiltered)...
[INFO] [2021-12-17 18:29:18] 10787 Traits (filtered)...
[INFO] [2021-12-17 18:29:21] 0 Associations (filtered)...
[INFO] [2021-12-17 18:29:41] 168963 metadata added.
[INFO] [2021-12-17 18:29:41] 0 metadata added.
[INFO] [2021-12-17 18:31:44] 10866 Traits (unfiltered)...
[INFO] [2021-12-17 18:34:21] 10866 Traits (filtered)...
[INFO] [2021-12-17 18:34:25] 0 Associations (filtered)...
[INFO] [2021-12-17 18:34:45] 168653 metadata added.
[INFO] [2021-12-17 18:34:45] 0 metadata added.
[INFO] [2021-12-17 18:36:44] 11075 Traits (unfiltered)...
[INFO] [2021-12-17 18:39:30] 11075 Traits (filtered)...
[INFO] [2021-12-17 18:39:34] 0 Associations (filtered)...
[INFO] [2021-12-17 18:39:55] 175435 metadata added.
[INFO] [2021-12-17 18:39:55] 0 metadata added.
[INFO] [2021-12-17 18:41:56] 11425 Traits (unfiltered)...
[INFO] [2021-12-17 18:44:46] 11425 Traits (filtered)...
[INFO] [2021-12-17 18:44:50] 0 Associations (filtered)...
[INFO] [2021-12-17 18:45:08] 179732 metadata added.
[INFO] [2021-12-17 18:45:08] 0 metadata added.
[INFO] [2021-12-17 18:47:11] 11033 Traits (unfiltered)...
[INFO] [2021-12-17 18:50:00] 11033 Traits (filtered)...
[INFO] [2021-12-17 18:50:03] 0 Associations (filtered)...
[INFO] [2021-12-17 18:50:24] 172217 metadata added.
[INFO] [2021-12-17 18:50:24] 0 metadata added.
[INFO] [2021-12-17 18:52:28] 10955 Traits (unfiltered)...
[INFO] [2021-12-17 18:55:08] 10955 Traits (filtered)...
[INFO] [2021-12-17 18:55:11] 0 Associations (filtered)...
[INFO] [2021-12-17 18:55:31] 170402 metadata added.
[INFO] [2021-12-17 18:55:31] 0 metadata added.
[INFO] [2021-12-17 18:57:31] 11661 Traits (unfiltered)...
[INFO] [2021-12-17 19:00:21] 11661 Traits (filtered)...
[INFO] [2021-12-17 19:00:25] 0 Associations (filtered)...
[INFO] [2021-12-17 19:00:46] 181955 metadata added.
[INFO] [2021-12-17 19:00:46] 0 metadata added.
[INFO] [2021-12-17 19:07:10] 11539 Traits (unfiltered)...
[INFO] [2021-12-17 19:10:00] 11539 Traits (filtered)...
[INFO] [2021-12-17 19:10:05] 0 Associations (filtered)...
[INFO] [2021-12-17 19:10:26] 180615 metadata added.
[INFO] [2021-12-17 19:10:26] 0 metadata added.
[INFO] [2021-12-17 19:13:24] 11998 Traits (unfiltered)...
[INFO] [2021-12-17 19:16:18] 11998 Traits (filtered)...
[INFO] [2021-12-17 19:16:21] 0 Associations (filtered)...
[INFO] [2021-12-17 19:16:43] 188840 metadata added.
[INFO] [2021-12-17 19:16:43] 0 metadata added.
[INFO] [2021-12-17 19:18:48] 11986 Traits (unfiltered)...
[INFO] [2021-12-17 19:21:37] 11986 Traits (filtered)...
[INFO] [2021-12-17 19:21:40] 0 Associations (filtered)...
[INFO] [2021-12-17 19:22:02] 188735 metadata added.
[INFO] [2021-12-17 19:22:02] 0 metadata added.
[INFO] [2021-12-17 19:24:09] 11920 Traits (unfiltered)...
[INFO] [2021-12-17 19:26:57] 11920 Traits (filtered)...
[INFO] [2021-12-17 19:27:00] 0 Associations (filtered)...
[INFO] [2021-12-17 19:27:23] 186924 metadata added.
[INFO] [2021-12-17 19:27:23] 0 metadata added.
[INFO] [2021-12-17 19:29:28] 12126 Traits (unfiltered)...
[INFO] [2021-12-17 19:32:25] 12126 Traits (filtered)...
[INFO] [2021-12-17 19:32:29] 0 Associations (filtered)...
[INFO] [2021-12-17 19:32:51] 191275 metadata added.
[INFO] [2021-12-17 19:32:51] 0 metadata added.
[INFO] [2021-12-17 19:34:57] 12431 Traits (unfiltered)...
[INFO] [2021-12-17 19:37:56] 12431 Traits (filtered)...
[INFO] [2021-12-17 19:37:59] 0 Associations (filtered)...
[INFO] [2021-12-17 19:38:22] 196419 metadata added.
[INFO] [2021-12-17 19:38:22] 0 metadata added.
[INFO] [2021-12-17 19:40:29] 11791 Traits (unfiltered)...
[INFO] [2021-12-17 19:43:17] 11791 Traits (filtered)...
[INFO] [2021-12-17 19:43:21] 0 Associations (filtered)...
[INFO] [2021-12-17 19:43:43] 184192 metadata added.
[INFO] [2021-12-17 19:43:43] 0 metadata added.
[INFO] [2021-12-17 19:45:46] 12163 Traits (unfiltered)...
[INFO] [2021-12-17 19:48:45] 12163 Traits (filtered)...
[INFO] [2021-12-17 19:48:48] 0 Associations (filtered)...
[INFO] [2021-12-17 19:49:10] 191300 metadata added.
[INFO] [2021-12-17 19:49:10] 0 metadata added.
[INFO] [2021-12-17 19:51:13] 11981 Traits (unfiltered)...
[INFO] [2021-12-17 19:54:03] 11981 Traits (filtered)...
[INFO] [2021-12-17 19:54:07] 0 Associations (filtered)...
[INFO] [2021-12-17 19:54:29] 187183 metadata added.
[INFO] [2021-12-17 19:54:29] 0 metadata added.
[INFO] [2021-12-17 19:56:33] 12286 Traits (unfiltered)...
[INFO] [2021-12-17 19:59:31] 12286 Traits (filtered)...
[INFO] [2021-12-17 19:59:34] 0 Associations (filtered)...
[INFO] [2021-12-17 19:59:56] 192895 metadata added.
[INFO] [2021-12-17 19:59:56] 0 metadata added.
[INFO] [2021-12-17 20:02:02] 12383 Traits (unfiltered)...
[INFO] [2021-12-17 20:05:00] 12383 Traits (filtered)...
[INFO] [2021-12-17 20:05:03] 0 Associations (filtered)...
[INFO] [2021-12-17 20:05:26] 194576 metadata added.
[INFO] [2021-12-17 20:05:26] 0 metadata added.
[INFO] [2021-12-17 20:07:34] 12198 Traits (unfiltered)...
[INFO] [2021-12-17 20:10:30] 12198 Traits (filtered)...
[INFO] [2021-12-17 20:10:34] 0 Associations (filtered)...
[INFO] [2021-12-17 20:10:56] 189765 metadata added.
[INFO] [2021-12-17 20:10:56] 0 metadata added.
[INFO] [2021-12-17 20:13:02] 12590 Traits (unfiltered)...
[INFO] [2021-12-17 20:16:00] 12590 Traits (filtered)...
[INFO] [2021-12-17 20:16:03] 0 Associations (filtered)...
[INFO] [2021-12-17 20:16:30] 198660 metadata added.
[INFO] [2021-12-17 20:16:30] 0 metadata added.
[INFO] [2021-12-17 20:18:34] 12290 Traits (unfiltered)...
[INFO] [2021-12-17 20:21:34] 12290 Traits (filtered)...
[INFO] [2021-12-17 20:21:37] 0 Associations (filtered)...
[INFO] [2021-12-17 20:22:00] 191391 metadata added.
[INFO] [2021-12-17 20:22:00] 0 metadata added.
[INFO] [2021-12-17 20:24:03] 12224 Traits (unfiltered)...
[INFO] [2021-12-17 20:27:01] 12224 Traits (filtered)...
[INFO] [2021-12-17 20:27:04] 0 Associations (filtered)...
[INFO] [2021-12-17 20:27:27] 192054 metadata added.
[INFO] [2021-12-17 20:27:27] 0 metadata added.
[INFO] [2021-12-17 20:29:33] 12860 Traits (unfiltered)...
[INFO] [2021-12-17 20:32:34] 12860 Traits (filtered)...
[INFO] [2021-12-17 20:32:38] 0 Associations (filtered)...
[INFO] [2021-12-17 20:33:01] 201890 metadata added.
[INFO] [2021-12-17 20:33:01] 0 metadata added.
[INFO] [2021-12-17 20:35:09] 12573 Traits (unfiltered)...
[INFO] [2021-12-17 20:38:13] 12573 Traits (filtered)...
[INFO] [2021-12-17 20:38:16] 0 Associations (filtered)...
[INFO] [2021-12-17 20:38:40] 197581 metadata added.
[INFO] [2021-12-17 20:38:40] 0 metadata added.
[INFO] [2021-12-17 20:40:44] 12961 Traits (unfiltered)...
[INFO] [2021-12-17 20:43:48] 12961 Traits (filtered)...
[INFO] [2021-12-17 20:43:51] 0 Associations (filtered)...
[INFO] [2021-12-17 20:44:15] 204844 metadata added.
[INFO] [2021-12-17 20:44:15] 0 metadata added.
[INFO] [2021-12-17 20:46:21] 12746 Traits (unfiltered)...
[INFO] [2021-12-17 20:49:24] 12746 Traits (filtered)...
[INFO] [2021-12-17 20:49:27] 0 Associations (filtered)...
[INFO] [2021-12-17 20:49:51] 201140 metadata added.
[INFO] [2021-12-17 20:49:51] 0 metadata added.
[INFO] [2021-12-17 20:51:26] 4003 Traits (unfiltered)...
[INFO] [2021-12-17 20:52:58] 4003 Traits (filtered)...
[INFO] [2021-12-17 20:52:58] 0 Associations (filtered)...
[INFO] [2021-12-17 20:53:04] 62921 metadata added.
[INFO] [2021-12-17 20:53:04] 0 metadata added.
[INFO] [2021-12-17 20:53:41] Average Time: 276.988
[INFO] [2021-12-17 20:53:41] Total Time: 2h52m26s
[INFO] [2021-12-17 20:53:41] last 3 / first 3: 1.12
[INFO] [2021-12-17 20:53:41] Std.Dev: 30.156442097833754; Max: 305.59
[STOP] [2021-12-17 20:53:41] overall_tsv_creation
[INFO] [2021-12-17 20:53:41] Done. Check your files:
[INFO] [2021-12-17 20:53:43] (313000 lines) /app/public/data/NMNHtypes/publish_nodes.tsv
[INFO] [2021-12-17 20:53:44] (1694826 lines) /app/public/data/NMNHtypes/publish_node_ancestors.tsv
[INFO] [2021-12-17 20:53:45] (313115 lines) /app/public/data/NMNHtypes/publish_scientific_names.tsv
[INFO] [2021-12-17 20:53:47] (359252 lines) /app/public/data/NMNHtypes/publish_traits.tsv
[INFO] [2021-12-17 20:53:48] (5637376 lines) /app/public/data/NMNHtypes/publish_metadata.tsv
[STOP] [2021-12-17 20:53:49] complete_harvest_instance
[START] [2021-12-17 20:53:49] completed
[STOP] [2021-12-17 20:53:49] completed
[STOP] [2021-12-17 20:53:49] logged process, took 37987.06

Latest Process

17 Dec 10:20 EXITED currently processing 0