Harvest for Wikipedia (inferred records) Created 03 Jun 23:25

Stage: completed
Fetched: 03 Jun 23:25
Validated: 03 Jun 23:25
Deltas Created 03 Jun 23:25
Units Normalized: 04 Jun 00:31
Ancestry Built: 03 Jun 23:46
Nodes Matched: 04 Jun 00:25
Names Parsed: 03 Jun 23:48
New Models Stored: 03 Jun 23:33
Indexed: 04 Jun 00:31
Completed: 04 Jun 02:14
Time to Harvest: 3 minutes

Harvesting Log

(447 lines)
[INFO] [2022-06-03 23:25:04] Created harvest instance #4130
[STOP] [2022-06-03 23:25:04] create_harvest_instance
[START] [2022-06-03 23:25:04] fetch_files
[STOP] [2022-06-03 23:25:04] fetch_files
[START] [2022-06-03 23:25:04] validate_each_file
[INFO] [2022-06-03 23:25:04] Looping over 3 formats...
[INFO] [2022-06-03 23:25:04] ...nodes (/app/public/data/environments_eol/taxon.tab)
[INFO] [2022-06-03 23:25:09] Valid: /app/public/data/environments_eol/converted_csv/environments_eol_nodes_29353.csv (169090 lines)
[INFO] [2022-06-03 23:25:09] ...occurrences (/app/public/data/environments_eol/occurrence.tab)
[INFO] [2022-06-03 23:25:12] Valid: /app/public/data/environments_eol/converted_csv/environments_eol_occurrences_29351.csv (253641 lines)
[INFO] [2022-06-03 23:25:12] ...measurements (/app/public/data/environments_eol/measurement_or_fact.tab)
[INFO] [2022-06-03 23:25:21] Valid: /app/public/data/environments_eol/converted_csv/environments_eol_measurements_29352.csv (253641 lines)
[STOP] [2022-06-03 23:25:21] validate_each_file
[START] [2022-06-03 23:25:21] convert_to_csv
[INFO] [2022-06-03 23:25:21] Looping over 3 formats...
[INFO] [2022-06-03 23:25:21] ...nodes (/app/public/data/environments_eol/taxon.tab)
[CMD] [2022-06-03 23:25:21] /usr/bin/sort /app/public/data/environments_eol/converted_csv/environments_eol_nodes_29353.csv > /app/public/data/environments_eol/converted_csv/environments_eol_nodes_29353.csv_sorted
[INFO] [2022-06-03 23:25:22] Converted: /app/public/data/environments_eol/converted_csv/environments_eol_nodes_29353.csv (169090 lines)
[INFO] [2022-06-03 23:25:22] ...occurrences (/app/public/data/environments_eol/occurrence.tab)
[CMD] [2022-06-03 23:25:22] /usr/bin/sort /app/public/data/environments_eol/converted_csv/environments_eol_occurrences_29351.csv > /app/public/data/environments_eol/converted_csv/environments_eol_occurrences_29351.csv_sorted
[INFO] [2022-06-03 23:25:23] Converted: /app/public/data/environments_eol/converted_csv/environments_eol_occurrences_29351.csv (253641 lines)
[INFO] [2022-06-03 23:25:23] ...measurements (/app/public/data/environments_eol/measurement_or_fact.tab)
[CMD] [2022-06-03 23:25:23] /usr/bin/sort /app/public/data/environments_eol/converted_csv/environments_eol_measurements_29352.csv > /app/public/data/environments_eol/converted_csv/environments_eol_measurements_29352.csv_sorted
[INFO] [2022-06-03 23:25:24] Converted: /app/public/data/environments_eol/converted_csv/environments_eol_measurements_29352.csv (253641 lines)
[STOP] [2022-06-03 23:25:24] convert_to_csv
[START] [2022-06-03 23:25:24] calculate_delta
[INFO] [2022-06-03 23:25:24] Looping over 3 formats...
[INFO] [2022-06-03 23:25:24] ...nodes (/app/public/data/environments_eol/taxon.tab)
[CMD] [2022-06-03 23:25:24] echo "0a" > /app/public/data/environments_eol/diff/environments_eol_nodes_29353.diff
[CMD] [2022-06-03 23:25:24] tail -n +1 /app/public/data/environments_eol/converted_csv/environments_eol_nodes_29353.csv >> /app/public/data/environments_eol/diff/environments_eol_nodes_29353.diff
[CMD] [2022-06-03 23:25:24] echo "." >> /app/public/data/environments_eol/diff/environments_eol_nodes_29353.diff
[INFO] [2022-06-03 23:25:24] Created diff: /app/public/data/environments_eol/diff/environments_eol_nodes_29353.diff (169092 lines)
[INFO] [2022-06-03 23:25:24] ...occurrences (/app/public/data/environments_eol/occurrence.tab)
[CMD] [2022-06-03 23:25:24] echo "0a" > /app/public/data/environments_eol/diff/environments_eol_occurrences_29351.diff
[CMD] [2022-06-03 23:25:24] tail -n +1 /app/public/data/environments_eol/converted_csv/environments_eol_occurrences_29351.csv >> /app/public/data/environments_eol/diff/environments_eol_occurrences_29351.diff
[CMD] [2022-06-03 23:25:25] echo "." >> /app/public/data/environments_eol/diff/environments_eol_occurrences_29351.diff
[INFO] [2022-06-03 23:25:25] Created diff: /app/public/data/environments_eol/diff/environments_eol_occurrences_29351.diff (253643 lines)
[INFO] [2022-06-03 23:25:25] ...measurements (/app/public/data/environments_eol/measurement_or_fact.tab)
[CMD] [2022-06-03 23:25:25] echo "0a" > /app/public/data/environments_eol/diff/environments_eol_measurements_29352.diff
[CMD] [2022-06-03 23:25:25] tail -n +1 /app/public/data/environments_eol/converted_csv/environments_eol_measurements_29352.csv >> /app/public/data/environments_eol/diff/environments_eol_measurements_29352.diff
[CMD] [2022-06-03 23:25:25] echo "." >> /app/public/data/environments_eol/diff/environments_eol_measurements_29352.diff
[INFO] [2022-06-03 23:25:25] Created diff: /app/public/data/environments_eol/diff/environments_eol_measurements_29352.diff (253643 lines)
[STOP] [2022-06-03 23:25:26] calculate_delta
[START] [2022-06-03 23:25:26] parse_diff_and_store
[INFO] [2022-06-03 23:25:26] Handling diff: /app/public/data/environments_eol/diff/environments_eol_nodes_29353.diff (169092 lines)
[INFO] [2022-06-03 23:25:26] Loading nodes diff file into memory (169092 lines)...
[INFO] [2022-06-03 23:25:29] Storing 9999 ScientificNames (29997/10000/169092)
[INFO] [2022-06-03 23:25:32] Storing 9999 Identifiers (29997/10000/169092)
[INFO] [2022-06-03 23:25:36] Storing 9999 Nodes (29997/10000/169092)
[WARN] [2022-06-03 23:25:45] Filtered Scientific Name `Cuon alpinus fumosus/javanicus` to `Cuon alpinus fumosusjavanicus`
[INFO] [2022-06-03 23:25:47] Storing 10000 ScientificNames (59997/20000/169092)
[INFO] [2022-06-03 23:25:50] Storing 10000 Identifiers (59997/20000/169092)
[INFO] [2022-06-03 23:25:54] Storing 10000 Nodes (59997/20000/169092)
[INFO] [2022-06-03 23:26:02] Storing 10000 ScientificNames (89997/30000/169092)
[INFO] [2022-06-03 23:26:05] Storing 10000 Identifiers (89997/30000/169092)
[INFO] [2022-06-03 23:26:07] Storing 10000 Nodes (89997/30000/169092)
[INFO] [2022-06-03 23:26:15] Storing 10000 ScientificNames (119997/40000/169092)
[INFO] [2022-06-03 23:26:18] Storing 10000 Identifiers (119997/40000/169092)
[INFO] [2022-06-03 23:26:20] Storing 10000 Nodes (119997/40000/169092)
[INFO] [2022-06-03 23:26:27] Storing 10000 ScientificNames (149997/50000/169092)
[INFO] [2022-06-03 23:26:30] Storing 10000 Identifiers (149997/50000/169092)
[INFO] [2022-06-03 23:26:32] Storing 10000 Nodes (149997/50000/169092)
[INFO] [2022-06-03 23:26:39] Storing 10000 ScientificNames (179997/60000/169092)
[INFO] [2022-06-03 23:26:42] Storing 10000 Identifiers (179997/60000/169092)
[INFO] [2022-06-03 23:26:43] Storing 10000 Nodes (179997/60000/169092)
[INFO] [2022-06-03 23:26:50] Storing 10000 ScientificNames (209997/70000/169092)
[INFO] [2022-06-03 23:26:53] Storing 10000 Identifiers (209997/70000/169092)
[INFO] [2022-06-03 23:26:54] Storing 10000 Nodes (209997/70000/169092)
[INFO] [2022-06-03 23:27:01] Storing 10000 ScientificNames (239997/80000/169092)
[INFO] [2022-06-03 23:27:04] Storing 10000 Identifiers (239997/80000/169092)
[INFO] [2022-06-03 23:27:05] Storing 10000 Nodes (239997/80000/169092)
[INFO] [2022-06-03 23:27:12] Storing 10000 ScientificNames (269997/90000/169092)
[INFO] [2022-06-03 23:27:14] Storing 10000 Identifiers (269997/90000/169092)
[INFO] [2022-06-03 23:27:15] Storing 10000 Nodes (269997/90000/169092)
[INFO] [2022-06-03 23:27:22] Storing 10000 ScientificNames (299997/100000/169092)
[INFO] [2022-06-03 23:27:25] Storing 10000 Identifiers (299997/100000/169092)
[INFO] [2022-06-03 23:27:26] Storing 10000 Nodes (299997/100000/169092)
[INFO] [2022-06-03 23:27:33] Storing 10000 ScientificNames (329997/110000/169092)
[INFO] [2022-06-03 23:27:35] Storing 10000 Identifiers (329997/110000/169092)
[INFO] [2022-06-03 23:27:36] Storing 10000 Nodes (329997/110000/169092)
[INFO] [2022-06-03 23:27:43] Storing 10000 ScientificNames (359997/120000/169092)
[INFO] [2022-06-03 23:27:46] Storing 10000 Identifiers (359997/120000/169092)
[INFO] [2022-06-03 23:27:47] Storing 10000 Nodes (359997/120000/169092)
[INFO] [2022-06-03 23:27:53] Storing 10000 ScientificNames (389997/130000/169092)
[INFO] [2022-06-03 23:27:56] Storing 10000 Identifiers (389997/130000/169092)
[INFO] [2022-06-03 23:27:57] Storing 10000 Nodes (389997/130000/169092)
[INFO] [2022-06-03 23:28:04] Storing 10000 ScientificNames (419997/140000/169092)
[INFO] [2022-06-03 23:28:07] Storing 10000 Identifiers (419997/140000/169092)
[INFO] [2022-06-03 23:28:08] Storing 10000 Nodes (419997/140000/169092)
[WARN] [2022-06-03 23:28:11] Filtered Scientific Name `Homalocephala  polycephala` to `Homalocephala polycephala`
[INFO] [2022-06-03 23:28:15] Storing 10000 ScientificNames (449997/150000/169092)
[INFO] [2022-06-03 23:28:18] Storing 10000 Identifiers (449997/150000/169092)
[INFO] [2022-06-03 23:28:18] Storing 10000 Nodes (449997/150000/169092)
[INFO] [2022-06-03 23:28:25] Storing 10000 ScientificNames (479997/160000/169092)
[INFO] [2022-06-03 23:28:28] Storing 10000 Identifiers (479997/160000/169092)
[INFO] [2022-06-03 23:28:29] Storing 10000 Nodes (479997/160000/169092)
[INFO] [2022-06-03 23:28:35] Storing 9091 ScientificNames (507270/169090/169092)
[INFO] [2022-06-03 23:28:38] Storing 9091 Identifiers (507270/169090/169092)
[INFO] [2022-06-03 23:28:39] Storing 9091 Nodes (507270/169090/169092)
[INFO] [2022-06-03 23:28:41] Handling diff: /app/public/data/environments_eol/diff/environments_eol_occurrences_29351.diff (253643 lines)
[INFO] [2022-06-03 23:28:41] Loading occurrences diff file into memory (253643 lines)...
[INFO] [2022-06-03 23:28:42] Storing 9999 Occurrences (9999/10000/253643)
[INFO] [2022-06-03 23:28:45] Storing 10000 Occurrences (19999/20000/253643)
[INFO] [2022-06-03 23:28:47] Storing 10000 Occurrences (29999/30000/253643)
[INFO] [2022-06-03 23:28:49] Storing 10000 Occurrences (39999/40000/253643)
[INFO] [2022-06-03 23:28:52] Storing 10000 Occurrences (49999/50000/253643)
[INFO] [2022-06-03 23:28:55] Storing 10000 Occurrences (59999/60000/253643)
[INFO] [2022-06-03 23:28:57] Storing 10000 Occurrences (69999/70000/253643)
[INFO] [2022-06-03 23:29:00] Storing 10000 Occurrences (79999/80000/253643)
[INFO] [2022-06-03 23:29:02] Storing 10000 Occurrences (89999/90000/253643)
[INFO] [2022-06-03 23:29:05] Storing 10000 Occurrences (99999/100000/253643)
[INFO] [2022-06-03 23:29:08] Storing 10000 Occurrences (109999/110000/253643)
[INFO] [2022-06-03 23:29:11] Storing 10000 Occurrences (119999/120000/253643)
[INFO] [2022-06-03 23:29:15] Storing 10000 Occurrences (129999/130000/253643)
[INFO] [2022-06-03 23:29:17] Storing 10000 Occurrences (139999/140000/253643)
[INFO] [2022-06-03 23:29:20] Storing 10000 Occurrences (149999/150000/253643)
[INFO] [2022-06-03 23:29:24] Storing 10000 Occurrences (159999/160000/253643)
[INFO] [2022-06-03 23:29:26] Storing 10000 Occurrences (169999/170000/253643)
[INFO] [2022-06-03 23:29:29] Storing 10000 Occurrences (179999/180000/253643)
[INFO] [2022-06-03 23:29:32] Storing 10000 Occurrences (189999/190000/253643)
[INFO] [2022-06-03 23:29:35] Storing 10000 Occurrences (199999/200000/253643)
[INFO] [2022-06-03 23:29:38] Storing 10000 Occurrences (209999/210000/253643)
[INFO] [2022-06-03 23:29:41] Storing 10000 Occurrences (219999/220000/253643)
[INFO] [2022-06-03 23:29:44] Storing 10000 Occurrences (229999/230000/253643)
[INFO] [2022-06-03 23:29:48] Storing 10000 Occurrences (239999/240000/253643)
[INFO] [2022-06-03 23:29:51] Storing 10000 Occurrences (249999/250000/253643)
[INFO] [2022-06-03 23:29:53] Storing 3642 Occurrences (253641/253641/253643)
[INFO] [2022-06-03 23:29:54] Handling diff: /app/public/data/environments_eol/diff/environments_eol_measurements_29352.diff (253643 lines)
[INFO] [2022-06-03 23:29:54] Loading measurements diff file into memory (253643 lines)...
[INFO] [2022-06-03 23:29:59] Storing 9999 Traits (19998/10000/253643)
[INFO] [2022-06-03 23:30:03] Storing 9999 MetaTraits (19998/10000/253643)
[INFO] [2022-06-03 23:30:08] Storing 10000 Traits (39998/20000/253643)
[INFO] [2022-06-03 23:30:11] Storing 10000 MetaTraits (39998/20000/253643)
[INFO] [2022-06-03 23:30:16] Storing 10000 Traits (59998/30000/253643)
[INFO] [2022-06-03 23:30:19] Storing 10000 MetaTraits (59998/30000/253643)
[INFO] [2022-06-03 23:30:24] Storing 10000 Traits (79998/40000/253643)
[INFO] [2022-06-03 23:30:28] Storing 10000 MetaTraits (79998/40000/253643)
[INFO] [2022-06-03 23:30:33] Storing 10000 Traits (99998/50000/253643)
[INFO] [2022-06-03 23:30:36] Storing 10000 MetaTraits (99998/50000/253643)
[INFO] [2022-06-03 23:30:41] Storing 10000 Traits (119998/60000/253643)
[INFO] [2022-06-03 23:30:44] Storing 10000 MetaTraits (119998/60000/253643)
[INFO] [2022-06-03 23:30:49] Storing 10000 Traits (139998/70000/253643)
[INFO] [2022-06-03 23:30:52] Storing 10000 MetaTraits (139998/70000/253643)
[INFO] [2022-06-03 23:30:58] Storing 10000 Traits (159998/80000/253643)
[INFO] [2022-06-03 23:31:02] Storing 10000 MetaTraits (159998/80000/253643)
[INFO] [2022-06-03 23:31:08] Storing 10000 Traits (179998/90000/253643)
[INFO] [2022-06-03 23:31:11] Storing 10000 MetaTraits (179998/90000/253643)
[INFO] [2022-06-03 23:31:16] Storing 10000 Traits (199998/100000/253643)
[INFO] [2022-06-03 23:31:19] Storing 10000 MetaTraits (199998/100000/253643)
[INFO] [2022-06-03 23:31:25] Storing 10000 Traits (219998/110000/253643)
[INFO] [2022-06-03 23:31:28] Storing 10000 MetaTraits (219998/110000/253643)
[INFO] [2022-06-03 23:31:33] Storing 10000 Traits (239998/120000/253643)
[INFO] [2022-06-03 23:31:36] Storing 10000 MetaTraits (239998/120000/253643)
[INFO] [2022-06-03 23:31:42] Storing 10000 Traits (259998/130000/253643)
[INFO] [2022-06-03 23:31:45] Storing 10000 MetaTraits (259998/130000/253643)
[INFO] [2022-06-03 23:31:50] Storing 10000 Traits (279998/140000/253643)
[INFO] [2022-06-03 23:31:54] Storing 10000 MetaTraits (279998/140000/253643)
[INFO] [2022-06-03 23:31:59] Storing 10000 Traits (299998/150000/253643)
[INFO] [2022-06-03 23:32:02] Storing 10000 MetaTraits (299998/150000/253643)
[INFO] [2022-06-03 23:32:08] Storing 10000 Traits (319998/160000/253643)
[INFO] [2022-06-03 23:32:11] Storing 10000 MetaTraits (319998/160000/253643)
[INFO] [2022-06-03 23:32:16] Storing 10000 Traits (339998/170000/253643)
[INFO] [2022-06-03 23:32:20] Storing 10000 MetaTraits (339998/170000/253643)
[INFO] [2022-06-03 23:32:25] Storing 10000 Traits (359998/180000/253643)
[INFO] [2022-06-03 23:32:28] Storing 10000 MetaTraits (359998/180000/253643)
[INFO] [2022-06-03 23:32:34] Storing 10000 Traits (379998/190000/253643)
[INFO] [2022-06-03 23:32:37] Storing 10000 MetaTraits (379998/190000/253643)
[INFO] [2022-06-03 23:32:43] Storing 10000 Traits (399998/200000/253643)
[INFO] [2022-06-03 23:32:46] Storing 10000 MetaTraits (399998/200000/253643)
[INFO] [2022-06-03 23:32:52] Storing 10000 Traits (419998/210000/253643)
[INFO] [2022-06-03 23:32:55] Storing 10000 MetaTraits (419998/210000/253643)
[INFO] [2022-06-03 23:33:01] Storing 10000 Traits (439998/220000/253643)
[INFO] [2022-06-03 23:33:04] Storing 10000 MetaTraits (439998/220000/253643)
[INFO] [2022-06-03 23:33:09] Storing 10000 Traits (459998/230000/253643)
[INFO] [2022-06-03 23:33:13] Storing 10000 MetaTraits (459998/230000/253643)
[INFO] [2022-06-03 23:33:18] Storing 10000 Traits (479998/240000/253643)
[INFO] [2022-06-03 23:33:21] Storing 10000 MetaTraits (479998/240000/253643)
[INFO] [2022-06-03 23:33:27] Storing 10000 Traits (499998/250000/253643)
[INFO] [2022-06-03 23:33:30] Storing 10000 MetaTraits (499998/250000/253643)
[INFO] [2022-06-03 23:33:34] Storing 3642 Traits (507282/253641/253643)
[INFO] [2022-06-03 23:33:35] Storing 3642 MetaTraits (507282/253641/253643)
[STOP] [2022-06-03 23:33:35] parse_diff_and_store
[START] [2022-06-03 23:33:35] resolve_keys
[2022-06-03 23:35:39] Resolving downloaded urls (this is not actually downloading them yet)
[INFO] [2022-06-03 23:35:47] Occurrences to nodes (through scientific_names)...
[INFO] [2022-06-03 23:36:00] traits to occurrences...
[INFO] [2022-06-03 23:36:10] traits to nodes (through occurrences)...
[INFO] [2022-06-03 23:36:15] Traits to sex term...
[INFO] [2022-06-03 23:36:20] Traits to lifestage term...
[INFO] [2022-06-03 23:36:24] MetaTraits to traits...
[INFO] [2022-06-03 23:36:31] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2022-06-03 23:36:31] Assocs to occurrences...
[INFO] [2022-06-03 23:36:31] Assocs to nodes...
[INFO] [2022-06-03 23:36:31] Assoc to sex term...
[INFO] [2022-06-03 23:36:31] Assoc to lifestage term...
[INFO] [2022-06-03 23:36:31] MetaAssoc to assocs...
[STOP] [2022-06-03 23:36:31] resolve_keys
[START] [2022-06-03 23:36:31] hold_for_later_1
[STOP] [2022-06-03 23:36:31] hold_for_later_1
[START] [2022-06-03 23:36:31] hold_for_later_2
[STOP] [2022-06-03 23:36:31] hold_for_later_2
[START] [2022-06-03 23:36:31] resolve_missing_parents
[STOP] [2022-06-03 23:36:40] resolve_missing_parents
[START] [2022-06-03 23:36:40] rebuild_nodes
[START] [2022-06-03 23:36:40] Flattener#flatten
[START] [2022-06-03 23:36:40] Flattener#study_resource
[START] [2022-06-03 23:37:19] Flattener#build_ancestry
[STOP] [2022-06-03 23:38:30] Flattener#build_ancestry
[INFO] [2022-06-03 23:38:30] 169090 ancestry keys
[START] [2022-06-03 23:38:30] build_node_ancestors
[INFO] [2022-06-03 23:38:30] old ancestors deleted.
[STOP] [2022-06-03 23:44:46] build_node_ancestors
[START] [2022-06-03 23:44:49] Flattener#propagate_ancestor_ids
[STOP] [2022-06-03 23:46:32] Flattener#propagate_ancestor_ids
[STOP] [2022-06-03 23:46:32] Flattener#flatten
[STOP] [2022-06-03 23:46:32] rebuild_nodes
[START] [2022-06-03 23:46:32] resolve_missing_media_owners
[STOP] [2022-06-03 23:46:32] resolve_missing_media_owners
[START] [2022-06-03 23:46:32] sanitize_media_verbatims
[STOP] [2022-06-03 23:46:32] sanitize_media_verbatims
[START] [2022-06-03 23:46:32] queue_downloads
[STOP] [2022-06-03 23:46:32] queue_downloads
[START] [2022-06-03 23:46:32] parse_names
[WARN] [2022-06-03 23:46:32] I see 169090 names which still need to be parsed.
[WARN] [2022-06-03 23:46:33] Names to parse: 10000 formatted: 10000 learned: 9998 parsed: 10000
[WARN] [2022-06-03 23:46:41] Names to parse: 10000 formatted: 10000 learned: 9997 parsed: 10000
[WARN] [2022-06-03 23:46:48] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2022-06-03 23:46:56] Names to parse: 10000 formatted: 10000 learned: 9998 parsed: 10000
[WARN] [2022-06-03 23:47:03] Names to parse: 10000 formatted: 10000 learned: 9998 parsed: 10000
[WARN] [2022-06-03 23:47:14] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2022-06-03 23:47:21] Names to parse: 10000 formatted: 10000 learned: 9999 parsed: 10000
[WARN] [2022-06-03 23:47:29] Names to parse: 10000 formatted: 10000 learned: 9997 parsed: 10000
[WARN] [2022-06-03 23:47:37] Names to parse: 10000 formatted: 10000 learned: 9998 parsed: 10000
[WARN] [2022-06-03 23:47:44] Names to parse: 10000 formatted: 10000 learned: 9996 parsed: 10000
[WARN] [2022-06-03 23:47:52] Names to parse: 10000 formatted: 10000 learned: 9998 parsed: 10000
[WARN] [2022-06-03 23:47:59] Names to parse: 10000 formatted: 10000 learned: 9993 parsed: 10000
[WARN] [2022-06-03 23:48:06] Names to parse: 10000 formatted: 10000 learned: 9996 parsed: 10000
[WARN] [2022-06-03 23:48:14] Names to parse: 10000 formatted: 10000 learned: 9997 parsed: 10000
[WARN] [2022-06-03 23:48:21] Names to parse: 10000 formatted: 10000 learned: 9996 parsed: 10000
[WARN] [2022-06-03 23:48:28] Names to parse: 10000 formatted: 10000 learned: 9997 parsed: 10000
[WARN] [2022-06-03 23:48:36] Names to parse: 9090 formatted: 9090 learned: 9087 parsed: 9090
[STOP] [2022-06-03 23:48:42] parse_names
[START] [2022-06-03 23:48:42] denormalize_canonical_names_to_nodes
[STOP] [2022-06-03 23:48:50] denormalize_canonical_names_to_nodes
[START] [2022-06-03 23:48:50] match_nodes
[START] [2022-06-03 23:48:51] map_all_nodes_to_pages
[STOP] [2022-06-04 00:24:36] map_all_nodes_to_pages
[INFO] [2022-06-04 00:24:36] 5188 Unmatched nodes (of 169090)! That's too many to output. Full list in /app/public/data/environments_eol/unmatched_nodes.txt ; First 10: Canonical: Titanodula; Node#115564870; ResourceID: Q105226767; Canonical: Pseudagkistrodon rudis; Node#115566881; ResourceID: Q106595923; Canonical: Weizmannia; Node#115567696; ResourceID: Q107210946; Canonical: Dinteracanthus; Node#115570060; ResourceID: Q108913297; Canonical: Craspedocephalus malabaricus; Node#115570221; ResourceID: Q109232820; Canonical: Hemiarmidae; Node#115570614; ResourceID: Q110209206; Canonical: Parakaryon myojinensis; Node#115633588; ResourceID: Q22329203; Canonical: Biota; Node#115637314; ResourceID: Q2382443; Canonical: Prokaryota; Node#115619382; ResourceID: Q19081; Canonical: Nitrososphaera gargensis; Node#115639650; ResourceID: Q24976640
[START] [2022-06-04 00:24:36] update_nodes
[STOP] [2022-06-04 00:25:17] update_nodes
[STOP] [2022-06-04 00:25:17] match_nodes
[START] [2022-06-04 00:25:17] reindex_search
[STOP] [2022-06-04 00:31:06] reindex_search
[START] [2022-06-04 00:31:06] normalize_units
[STOP] [2022-06-04 00:31:06] normalize_units
[START] [2022-06-04 00:31:06] calculate_statistics
[INFO] [2022-06-04 00:31:18] Duplicate page_id count: 0
[STOP] [2022-06-04 00:31:18] calculate_statistics
[START] [2022-06-04 00:31:18] complete_harvest_instance
[START] [2022-06-04 00:31:18] overall_tsv_creation
[INFO] [2022-06-04 00:31:18] Processing group of 169090 in 17 batches of 10000
[INFO] [2022-06-04 00:34:36] 11370 Traits (unfiltered)...
[INFO] [2022-06-04 00:34:36] Building Traits map (this can take a while)...
[INFO] [2022-06-04 00:37:16] 10000 traits mapped (10000 meta)...
[INFO] [2022-06-04 00:37:38] Done. 11370 traits mapped (11370 meta).
[INFO] [2022-06-04 00:37:38] Building Associations map (this can take a while)...
[INFO] [2022-06-04 00:37:49] Done. 0 assocs mapped (0 meta).
[INFO] [2022-06-04 00:37:49] Adding 11370 traits...
[INFO] [2022-06-04 00:37:50] 0 metadata added.
[INFO] [2022-06-04 00:37:50] Adding 0 assocs...
[INFO] [2022-06-04 00:37:50] 0 metadata added.
[INFO] [2022-06-04 00:41:29] 17856 Traits (unfiltered)...
[INFO] [2022-06-04 00:41:29] Building Traits map (this can take a while)...
[INFO] [2022-06-04 00:44:05] 10000 traits mapped (10000 meta)...
[INFO] [2022-06-04 00:45:32] Done. 17856 traits mapped (17856 meta).
[INFO] [2022-06-04 00:45:32] Building Associations map (this can take a while)...
[INFO] [2022-06-04 00:45:42] Done. 0 assocs mapped (0 meta).
[INFO] [2022-06-04 00:45:42] Adding 17856 traits...
[INFO] [2022-06-04 00:45:44] 0 metadata added.
[INFO] [2022-06-04 00:45:44] Adding 0 assocs...
[INFO] [2022-06-04 00:45:44] 0 metadata added.
[INFO] [2022-06-04 00:48:18] 10880 Traits (unfiltered)...
[INFO] [2022-06-04 00:48:18] Building Traits map (this can take a while)...
[INFO] [2022-06-04 00:50:53] 10000 traits mapped (10000 meta)...
[INFO] [2022-06-04 00:51:04] Done. 10880 traits mapped (10880 meta).
[INFO] [2022-06-04 00:51:04] Building Associations map (this can take a while)...
[INFO] [2022-06-04 00:51:11] Done. 0 assocs mapped (0 meta).
[INFO] [2022-06-04 00:51:11] Adding 10880 traits...
[INFO] [2022-06-04 00:51:11] 0 metadata added.
[INFO] [2022-06-04 00:51:11] Adding 0 assocs...
[INFO] [2022-06-04 00:51:11] 0 metadata added.
[INFO] [2022-06-04 00:53:30] 16147 Traits (unfiltered)...
[INFO] [2022-06-04 00:53:30] Building Traits map (this can take a while)...
[INFO] [2022-06-04 00:56:04] 10000 traits mapped (10000 meta)...
[INFO] [2022-06-04 00:57:20] Done. 16147 traits mapped (16147 meta).
[INFO] [2022-06-04 00:57:20] Building Associations map (this can take a while)...
[INFO] [2022-06-04 00:57:27] Done. 0 assocs mapped (0 meta).
[INFO] [2022-06-04 00:57:27] Adding 16147 traits...
[INFO] [2022-06-04 00:57:28] 0 metadata added.
[INFO] [2022-06-04 00:57:28] Adding 0 assocs...
[INFO] [2022-06-04 00:57:28] 0 metadata added.
[INFO] [2022-06-04 00:59:50] 15552 Traits (unfiltered)...
[INFO] [2022-06-04 00:59:50] Building Traits map (this can take a while)...
[INFO] [2022-06-04 01:02:22] 10000 traits mapped (10000 meta)...
[INFO] [2022-06-04 01:03:25] Done. 15552 traits mapped (15552 meta).
[INFO] [2022-06-04 01:03:25] Building Associations map (this can take a while)...
[INFO] [2022-06-04 01:03:32] Done. 0 assocs mapped (0 meta).
[INFO] [2022-06-04 01:03:32] Adding 15552 traits...
[INFO] [2022-06-04 01:03:36] 0 metadata added.
[INFO] [2022-06-04 01:03:36] Adding 0 assocs...
[INFO] [2022-06-04 01:03:36] 0 metadata added.
[INFO] [2022-06-04 01:05:54] 13316 Traits (unfiltered)...
[INFO] [2022-06-04 01:05:54] Building Traits map (this can take a while)...
[INFO] [2022-06-04 01:08:23] 10000 traits mapped (10000 meta)...
[INFO] [2022-06-04 01:09:05] Done. 13316 traits mapped (13316 meta).
[INFO] [2022-06-04 01:09:05] Building Associations map (this can take a while)...
[INFO] [2022-06-04 01:09:13] Done. 0 assocs mapped (0 meta).
[INFO] [2022-06-04 01:09:13] Adding 13316 traits...
[INFO] [2022-06-04 01:09:14] 0 metadata added.
[INFO] [2022-06-04 01:09:14] Adding 0 assocs...
[INFO] [2022-06-04 01:09:14] 0 metadata added.
[INFO] [2022-06-04 01:11:23] 11356 Traits (unfiltered)...
[INFO] [2022-06-04 01:11:23] Building Traits map (this can take a while)...
[INFO] [2022-06-04 01:13:57] 10000 traits mapped (10000 meta)...
[INFO] [2022-06-04 01:14:18] Done. 11356 traits mapped (11356 meta).
[INFO] [2022-06-04 01:14:18] Building Associations map (this can take a while)...
[INFO] [2022-06-04 01:14:25] Done. 0 assocs mapped (0 meta).
[INFO] [2022-06-04 01:14:25] Adding 11356 traits...
[INFO] [2022-06-04 01:14:25] 0 metadata added.
[INFO] [2022-06-04 01:14:25] Adding 0 assocs...
[INFO] [2022-06-04 01:14:25] 0 metadata added.
[INFO] [2022-06-04 01:16:36] 19127 Traits (unfiltered)...
[INFO] [2022-06-04 01:16:36] Building Traits map (this can take a while)...
[INFO] [2022-06-04 01:19:05] 10000 traits mapped (10000 meta)...
[INFO] [2022-06-04 01:20:54] Done. 19127 traits mapped (19127 meta).
[INFO] [2022-06-04 01:20:54] Building Associations map (this can take a while)...
[INFO] [2022-06-04 01:21:01] Done. 0 assocs mapped (0 meta).
[INFO] [2022-06-04 01:21:01] Adding 19127 traits...
[INFO] [2022-06-04 01:21:03] 0 metadata added.
[INFO] [2022-06-04 01:21:03] Adding 0 assocs...
[INFO] [2022-06-04 01:21:03] 0 metadata added.
[INFO] [2022-06-04 01:23:11] 18386 Traits (unfiltered)...
[INFO] [2022-06-04 01:23:11] Building Traits map (this can take a while)...
[INFO] [2022-06-04 01:25:40] 10000 traits mapped (10000 meta)...
[INFO] [2022-06-04 01:27:20] Done. 18386 traits mapped (18386 meta).
[INFO] [2022-06-04 01:27:20] Building Associations map (this can take a while)...
[INFO] [2022-06-04 01:27:27] Done. 0 assocs mapped (0 meta).
[INFO] [2022-06-04 01:27:27] Adding 18386 traits...
[INFO] [2022-06-04 01:27:28] 0 metadata added.
[INFO] [2022-06-04 01:27:28] Adding 0 assocs...
[INFO] [2022-06-04 01:27:28] 0 metadata added.
[INFO] [2022-06-04 01:29:39] 16705 Traits (unfiltered)...
[INFO] [2022-06-04 01:29:39] Building Traits map (this can take a while)...
[INFO] [2022-06-04 01:32:08] 10000 traits mapped (10000 meta)...
[INFO] [2022-06-04 01:33:28] Done. 16705 traits mapped (16705 meta).
[INFO] [2022-06-04 01:33:28] Building Associations map (this can take a while)...
[INFO] [2022-06-04 01:33:35] Done. 0 assocs mapped (0 meta).
[INFO] [2022-06-04 01:33:35] Adding 16705 traits...
[INFO] [2022-06-04 01:33:36] 0 metadata added.
[INFO] [2022-06-04 01:33:36] Adding 0 assocs...
[INFO] [2022-06-04 01:33:36] 0 metadata added.
[INFO] [2022-06-04 01:35:47] 15347 Traits (unfiltered)...
[INFO] [2022-06-04 01:35:47] Building Traits map (this can take a while)...
[INFO] [2022-06-04 01:38:15] 10000 traits mapped (10000 meta)...
[INFO] [2022-06-04 01:39:21] Done. 15347 traits mapped (15347 meta).
[INFO] [2022-06-04 01:39:21] Building Associations map (this can take a while)...
[INFO] [2022-06-04 01:39:28] Done. 0 assocs mapped (0 meta).
[INFO] [2022-06-04 01:39:28] Adding 15347 traits...
[INFO] [2022-06-04 01:39:29] 0 metadata added.
[INFO] [2022-06-04 01:39:29] Adding 0 assocs...
[INFO] [2022-06-04 01:39:29] 0 metadata added.
[INFO] [2022-06-04 01:41:50] 13770 Traits (unfiltered)...
[INFO] [2022-06-04 01:41:50] Building Traits map (this can take a while)...
[INFO] [2022-06-04 01:44:19] 10000 traits mapped (10000 meta)...
[INFO] [2022-06-04 01:45:01] Done. 13770 traits mapped (13770 meta).
[INFO] [2022-06-04 01:45:01] Building Associations map (this can take a while)...
[INFO] [2022-06-04 01:45:08] Done. 0 assocs mapped (0 meta).
[INFO] [2022-06-04 01:45:08] Adding 13770 traits...
[INFO] [2022-06-04 01:45:09] 0 metadata added.
[INFO] [2022-06-04 01:45:09] Adding 0 assocs...
[INFO] [2022-06-04 01:45:09] 0 metadata added.
[INFO] [2022-06-04 01:47:23] 13372 Traits (unfiltered)...
[INFO] [2022-06-04 01:47:23] Building Traits map (this can take a while)...
[INFO] [2022-06-04 01:49:52] 10000 traits mapped (10000 meta)...
[INFO] [2022-06-04 01:50:34] Done. 13372 traits mapped (13372 meta).
[INFO] [2022-06-04 01:50:34] Building Associations map (this can take a while)...
[INFO] [2022-06-04 01:50:41] Done. 0 assocs mapped (0 meta).
[INFO] [2022-06-04 01:50:41] Adding 13372 traits...
[INFO] [2022-06-04 01:50:42] 0 metadata added.
[INFO] [2022-06-04 01:50:42] Adding 0 assocs...
[INFO] [2022-06-04 01:50:42] 0 metadata added.
[INFO] [2022-06-04 01:52:50] 15396 Traits (unfiltered)...
[INFO] [2022-06-04 01:52:50] Building Traits map (this can take a while)...
[INFO] [2022-06-04 01:55:21] 10000 traits mapped (10000 meta)...
[INFO] [2022-06-04 01:56:26] Done. 15396 traits mapped (15396 meta).
[INFO] [2022-06-04 01:56:26] Building Associations map (this can take a while)...
[INFO] [2022-06-04 01:56:33] Done. 0 assocs mapped (0 meta).
[INFO] [2022-06-04 01:56:33] Adding 15396 traits...
[INFO] [2022-06-04 01:56:34] 0 metadata added.
[INFO] [2022-06-04 01:56:34] Adding 0 assocs...
[INFO] [2022-06-04 01:56:34] 0 metadata added.
[INFO] [2022-06-04 01:58:43] 14708 Traits (unfiltered)...
[INFO] [2022-06-04 01:58:43] Building Traits map (this can take a while)...
[INFO] [2022-06-04 02:01:13] 10000 traits mapped (10000 meta)...
[INFO] [2022-06-04 02:02:08] Done. 14708 traits mapped (14708 meta).
[INFO] [2022-06-04 02:02:08] Building Associations map (this can take a while)...
[INFO] [2022-06-04 02:02:14] Done. 0 assocs mapped (0 meta).
[INFO] [2022-06-04 02:02:14] Adding 14708 traits...
[INFO] [2022-06-04 02:02:15] 0 metadata added.
[INFO] [2022-06-04 02:02:15] Adding 0 assocs...
[INFO] [2022-06-04 02:02:15] 0 metadata added.
[INFO] [2022-06-04 02:04:27] 13760 Traits (unfiltered)...
[INFO] [2022-06-04 02:04:27] Building Traits map (this can take a while)...
[INFO] [2022-06-04 02:06:56] 10000 traits mapped (10000 meta)...
[INFO] [2022-06-04 02:07:38] Done. 13760 traits mapped (13760 meta).
[INFO] [2022-06-04 02:07:38] Building Associations map (this can take a while)...
[INFO] [2022-06-04 02:07:45] Done. 0 assocs mapped (0 meta).
[INFO] [2022-06-04 02:07:45] Adding 13760 traits...
[INFO] [2022-06-04 02:07:46] 0 metadata added.
[INFO] [2022-06-04 02:07:46] Adding 0 assocs...
[INFO] [2022-06-04 02:07:46] 0 metadata added.
[INFO] [2022-06-04 02:09:53] 16593 Traits (unfiltered)...
[INFO] [2022-06-04 02:09:53] Building Traits map (this can take a while)...
[INFO] [2022-06-04 02:12:22] 10000 traits mapped (10000 meta)...
[INFO] [2022-06-04 02:13:41] Done. 16593 traits mapped (16593 meta).
[INFO] [2022-06-04 02:13:41] Building Associations map (this can take a while)...
[INFO] [2022-06-04 02:13:47] Done. 0 assocs mapped (0 meta).
[INFO] [2022-06-04 02:13:47] Adding 16593 traits...
[INFO] [2022-06-04 02:13:48] 0 metadata added.
[INFO] [2022-06-04 02:13:48] Adding 0 assocs...
[INFO] [2022-06-04 02:13:48] 0 metadata added.
[INFO] [2022-06-04 02:14:33] Average Time: 311.732
[INFO] [2022-06-04 02:14:33] Total Time: 1h43m15s
[INFO] [2022-06-04 02:14:33] last 3 / first 3: 0.95
[INFO] [2022-06-04 02:14:33] Std.Dev: 27.459; Max: 355.73
[STOP] [2022-06-04 02:14:33] overall_tsv_creation
[INFO] [2022-06-04 02:14:33] Done. Check your files:
[INFO] [2022-06-04 02:14:33] (169090 lines) /app/public/data/environments_eol/publish_nodes.tsv
[INFO] [2022-06-04 02:14:33] (169090 lines) /app/public/data/environments_eol/publish_identifiers.tsv
[INFO] [2022-06-04 02:14:33] (3743138 lines) /app/public/data/environments_eol/publish_node_ancestors.tsv
[INFO] [2022-06-04 02:14:33] (169090 lines) /app/public/data/environments_eol/publish_scientific_names.tsv
[INFO] [2022-06-04 02:14:33] (253642 lines) /app/public/data/environments_eol/publish_traits.tsv
[INFO] [2022-06-04 02:14:33] (1 lines) /app/public/data/environments_eol/publish_metadata.tsv
[STOP] [2022-06-04 02:14:33] complete_harvest_instance
[START] [2022-06-04 02:14:33] completed
[STOP] [2022-06-04 02:14:33] completed
[STOP] [2022-06-04 02:14:33] logged process, took 10169.55

Latest Process