Harvest for Wikipedia (inferred records) Created 25 May 01:37

Harvesting Log

[INFO] [2024-05-25 01:37:35] Created harvest instance #4540
[STOP] [2024-05-25 01:37:35] create_harvest_instance
[START] [2024-05-25 01:37:35] fetch_files
[STOP] [2024-05-25 01:37:35] fetch_files
[START] [2024-05-25 01:37:35] validate_each_file
[INFO] [2024-05-25 01:37:35] Looping over 3 formats...
[INFO] [2024-05-25 01:37:35] ...nodes (/app/public/data/environments_eol/taxon.tab)
[INFO] [2024-05-25 01:37:40] Valid: /app/public/data/environments_eol/converted_csv/environments_eol_nodes_31130.csv (162291 lines)
[INFO] [2024-05-25 01:37:40] ...occurrences (/app/public/data/environments_eol/occurrence.tab)
[INFO] [2024-05-25 01:37:42] Valid: /app/public/data/environments_eol/converted_csv/environments_eol_occurrences_31128.csv (185767 lines)
[INFO] [2024-05-25 01:37:42] ...measurements (/app/public/data/environments_eol/measurement_or_fact.tab)
[INFO] [2024-05-25 01:37:49] Valid: /app/public/data/environments_eol/converted_csv/environments_eol_measurements_31129.csv (185767 lines)
[STOP] [2024-05-25 01:37:49] validate_each_file
[START] [2024-05-25 01:37:49] convert_to_csv
[INFO] [2024-05-25 01:37:49] Looping over 3 formats...
[INFO] [2024-05-25 01:37:49] ...nodes (/app/public/data/environments_eol/taxon.tab)
[CMD] [2024-05-25 01:37:49] /usr/bin/sort /app/public/data/environments_eol/converted_csv/environments_eol_nodes_31130.csv > /app/public/data/environments_eol/converted_csv/environments_eol_nodes_31130.csv_sorted
[INFO] [2024-05-25 01:37:49] Converted: /app/public/data/environments_eol/converted_csv/environments_eol_nodes_31130.csv (162291 lines)
[INFO] [2024-05-25 01:37:49] ...occurrences (/app/public/data/environments_eol/occurrence.tab)
[CMD] [2024-05-25 01:37:49] /usr/bin/sort /app/public/data/environments_eol/converted_csv/environments_eol_occurrences_31128.csv > /app/public/data/environments_eol/converted_csv/environments_eol_occurrences_31128.csv_sorted
[INFO] [2024-05-25 01:37:50] Converted: /app/public/data/environments_eol/converted_csv/environments_eol_occurrences_31128.csv (185767 lines)
[INFO] [2024-05-25 01:37:50] ...measurements (/app/public/data/environments_eol/measurement_or_fact.tab)
[CMD] [2024-05-25 01:37:50] /usr/bin/sort /app/public/data/environments_eol/converted_csv/environments_eol_measurements_31129.csv > /app/public/data/environments_eol/converted_csv/environments_eol_measurements_31129.csv_sorted
[INFO] [2024-05-25 01:37:51] Converted: /app/public/data/environments_eol/converted_csv/environments_eol_measurements_31129.csv (185767 lines)
[STOP] [2024-05-25 01:37:51] convert_to_csv
[START] [2024-05-25 01:37:51] calculate_delta
[INFO] [2024-05-25 01:37:51] Looping over 3 formats...
[INFO] [2024-05-25 01:37:51] ...nodes (/app/public/data/environments_eol/taxon.tab)
[CMD] [2024-05-25 01:37:51] echo "0a" > /app/public/data/environments_eol/diff/environments_eol_nodes_31130.diff
[CMD] [2024-05-25 01:37:51] tail -n +1 /app/public/data/environments_eol/converted_csv/environments_eol_nodes_31130.csv >> /app/public/data/environments_eol/diff/environments_eol_nodes_31130.diff
[CMD] [2024-05-25 01:37:51] echo "." >> /app/public/data/environments_eol/diff/environments_eol_nodes_31130.diff
[INFO] [2024-05-25 01:37:52] Created diff: /app/public/data/environments_eol/diff/environments_eol_nodes_31130.diff (162293 lines)
[INFO] [2024-05-25 01:37:52] ...occurrences (/app/public/data/environments_eol/occurrence.tab)
[CMD] [2024-05-25 01:37:52] echo "0a" > /app/public/data/environments_eol/diff/environments_eol_occurrences_31128.diff
[CMD] [2024-05-25 01:37:52] tail -n +1 /app/public/data/environments_eol/converted_csv/environments_eol_occurrences_31128.csv >> /app/public/data/environments_eol/diff/environments_eol_occurrences_31128.diff
[CMD] [2024-05-25 01:37:52] echo "." >> /app/public/data/environments_eol/diff/environments_eol_occurrences_31128.diff
[INFO] [2024-05-25 01:37:52] Created diff: /app/public/data/environments_eol/diff/environments_eol_occurrences_31128.diff (185769 lines)
[INFO] [2024-05-25 01:37:52] ...measurements (/app/public/data/environments_eol/measurement_or_fact.tab)
[CMD] [2024-05-25 01:37:52] echo "0a" > /app/public/data/environments_eol/diff/environments_eol_measurements_31129.diff
[CMD] [2024-05-25 01:37:52] tail -n +1 /app/public/data/environments_eol/converted_csv/environments_eol_measurements_31129.csv >> /app/public/data/environments_eol/diff/environments_eol_measurements_31129.diff
[CMD] [2024-05-25 01:37:53] echo "." >> /app/public/data/environments_eol/diff/environments_eol_measurements_31129.diff
[INFO] [2024-05-25 01:37:53] Created diff: /app/public/data/environments_eol/diff/environments_eol_measurements_31129.diff (185769 lines)
[STOP] [2024-05-25 01:37:53] calculate_delta
[START] [2024-05-25 01:37:53] parse_diff_and_store
[INFO] [2024-05-25 01:37:54] Handling diff: /app/public/data/environments_eol/diff/environments_eol_nodes_31130.diff (162293 lines)
[INFO] [2024-05-25 01:37:54] Loading nodes diff file into memory (162293 lines)...
[WARN] [2024-05-25 01:37:57] Filtered Scientific Name `Neotrombicula  fujigmo` to `Neotrombicula fujigmo`
[INFO] [2024-05-25 01:37:57] Storing 9999 ScientificNames (29997/10000/162293)
[INFO] [2024-05-25 01:38:00] Storing 9999 Identifiers (29997/10000/162293)
[INFO] [2024-05-25 01:38:02] Storing 9999 Nodes (29997/10000/162293)
[INFO] [2024-05-25 01:38:08] Storing 10000 ScientificNames (59997/20000/162293)
[INFO] [2024-05-25 01:38:11] Storing 10000 Identifiers (59997/20000/162293)
[INFO] [2024-05-25 01:38:12] Storing 10000 Nodes (59997/20000/162293)
[INFO] [2024-05-25 01:38:19] Storing 10000 ScientificNames (89997/30000/162293)
[INFO] [2024-05-25 01:38:22] Storing 10000 Identifiers (89997/30000/162293)
[INFO] [2024-05-25 01:38:23] Storing 10000 Nodes (89997/30000/162293)
[INFO] [2024-05-25 01:38:30] Storing 10000 ScientificNames (119997/40000/162293)
[INFO] [2024-05-25 01:38:33] Storing 10000 Identifiers (119997/40000/162293)
[INFO] [2024-05-25 01:38:34] Storing 10000 Nodes (119997/40000/162293)
[INFO] [2024-05-25 01:38:41] Storing 10000 ScientificNames (149997/50000/162293)
[INFO] [2024-05-25 01:38:44] Storing 10000 Identifiers (149997/50000/162293)
[INFO] [2024-05-25 01:38:45] Storing 10000 Nodes (149997/50000/162293)
[INFO] [2024-05-25 01:38:52] Storing 10000 ScientificNames (179997/60000/162293)
[INFO] [2024-05-25 01:38:55] Storing 10000 Identifiers (179997/60000/162293)
[INFO] [2024-05-25 01:38:56] Storing 10000 Nodes (179997/60000/162293)
[INFO] [2024-05-25 01:39:02] Storing 10000 ScientificNames (209997/70000/162293)
[INFO] [2024-05-25 01:39:06] Storing 10000 Identifiers (209997/70000/162293)
[INFO] [2024-05-25 01:39:07] Storing 10000 Nodes (209997/70000/162293)
[INFO] [2024-05-25 01:39:14] Storing 10000 ScientificNames (239997/80000/162293)
[INFO] [2024-05-25 01:39:17] Storing 10000 Identifiers (239997/80000/162293)
[INFO] [2024-05-25 01:39:18] Storing 10000 Nodes (239997/80000/162293)
[WARN] [2024-05-25 01:39:23] Filtered Scientific Name `/Gunneridae` to `Gunneridae`
[INFO] [2024-05-25 01:39:25] Storing 10000 ScientificNames (269997/90000/162293)
[INFO] [2024-05-25 01:39:28] Storing 10000 Identifiers (269997/90000/162293)
[INFO] [2024-05-25 01:39:29] Storing 10000 Nodes (269997/90000/162293)
[INFO] [2024-05-25 01:39:36] Storing 10000 ScientificNames (299997/100000/162293)
[INFO] [2024-05-25 01:39:39] Storing 10000 Identifiers (299997/100000/162293)
[INFO] [2024-05-25 01:39:40] Storing 10000 Nodes (299997/100000/162293)
[INFO] [2024-05-25 01:39:47] Storing 10000 ScientificNames (329997/110000/162293)
[INFO] [2024-05-25 01:39:50] Storing 10000 Identifiers (329997/110000/162293)
[INFO] [2024-05-25 01:39:51] Storing 10000 Nodes (329997/110000/162293)
[WARN] [2024-05-25 01:39:55] Filtered Scientific Name `/Eudicotyledoneae` to `Eudicotyledoneae`
[WARN] [2024-05-25 01:39:55] Filtered Scientific Name `/Mesangiospermae` to `Mesangiospermae`
[WARN] [2024-05-25 01:39:55] Filtered Scientific Name `/Pan-Angiospermae` to `Pan-Angiospermae`
[INFO] [2024-05-25 01:39:59] Storing 10000 ScientificNames (359997/120000/162293)
[INFO] [2024-05-25 01:40:02] Storing 10000 Identifiers (359997/120000/162293)
[INFO] [2024-05-25 01:40:03] Storing 10000 Nodes (359997/120000/162293)
[INFO] [2024-05-25 01:40:10] Storing 10000 ScientificNames (389997/130000/162293)
[INFO] [2024-05-25 01:40:13] Storing 10000 Identifiers (389997/130000/162293)
[INFO] [2024-05-25 01:40:14] Storing 10000 Nodes (389997/130000/162293)
[WARN] [2024-05-25 01:40:19] Filtered Scientific Name `/Pentapetalae` to `Pentapetalae`
[WARN] [2024-05-25 01:40:19] Filtered Scientific Name `Homalocephala  polycephala` to `Homalocephala polycephala`
[INFO] [2024-05-25 01:40:21] Storing 10000 ScientificNames (419997/140000/162293)
[INFO] [2024-05-25 01:40:24] Storing 10000 Identifiers (419997/140000/162293)
[INFO] [2024-05-25 01:40:25] Storing 10000 Nodes (419997/140000/162293)
[INFO] [2024-05-25 01:40:33] Storing 10000 ScientificNames (449997/150000/162293)
[INFO] [2024-05-25 01:40:36] Storing 10000 Identifiers (449997/150000/162293)
[INFO] [2024-05-25 01:40:37] Storing 10000 Nodes (449997/150000/162293)
[INFO] [2024-05-25 01:40:44] Storing 10000 ScientificNames (479997/160000/162293)
[INFO] [2024-05-25 01:40:47] Storing 10000 Identifiers (479997/160000/162293)
[INFO] [2024-05-25 01:40:48] Storing 10000 Nodes (479997/160000/162293)
[INFO] [2024-05-25 01:40:53] Storing 2292 ScientificNames (486873/162291/162293)
[INFO] [2024-05-25 01:40:54] Storing 2292 Identifiers (486873/162291/162293)
[INFO] [2024-05-25 01:40:54] Storing 2292 Nodes (486873/162291/162293)
[INFO] [2024-05-25 01:40:55] Handling diff: /app/public/data/environments_eol/diff/environments_eol_occurrences_31128.diff (185769 lines)
[INFO] [2024-05-25 01:40:55] Loading occurrences diff file into memory (185769 lines)...
[INFO] [2024-05-25 01:40:56] Storing 9999 Occurrences (9999/10000/185769)
[INFO] [2024-05-25 01:40:58] Storing 10000 Occurrences (19999/20000/185769)
[INFO] [2024-05-25 01:41:00] Storing 10000 Occurrences (29999/30000/185769)
[INFO] [2024-05-25 01:41:03] Storing 10000 Occurrences (39999/40000/185769)
[INFO] [2024-05-25 01:41:05] Storing 10000 Occurrences (49999/50000/185769)
[INFO] [2024-05-25 01:41:07] Storing 10000 Occurrences (59999/60000/185769)
[INFO] [2024-05-25 01:41:09] Storing 10000 Occurrences (69999/70000/185769)
[INFO] [2024-05-25 01:41:12] Storing 10000 Occurrences (79999/80000/185769)
[INFO] [2024-05-25 01:41:15] Storing 10000 Occurrences (89999/90000/185769)
[INFO] [2024-05-25 01:41:17] Storing 10000 Occurrences (99999/100000/185769)
[INFO] [2024-05-25 01:41:20] Storing 10000 Occurrences (109999/110000/185769)
[INFO] [2024-05-25 01:41:22] Storing 10000 Occurrences (119999/120000/185769)
[INFO] [2024-05-25 01:41:25] Storing 10000 Occurrences (129999/130000/185769)
[INFO] [2024-05-25 01:41:27] Storing 10000 Occurrences (139999/140000/185769)
[INFO] [2024-05-25 01:41:30] Storing 10000 Occurrences (149999/150000/185769)
[INFO] [2024-05-25 01:41:33] Storing 10000 Occurrences (159999/160000/185769)
[INFO] [2024-05-25 01:41:36] Storing 10000 Occurrences (169999/170000/185769)
[INFO] [2024-05-25 01:41:39] Storing 10000 Occurrences (179999/180000/185769)
[INFO] [2024-05-25 01:41:41] Storing 5768 Occurrences (185767/185767/185769)
[INFO] [2024-05-25 01:41:42] Handling diff: /app/public/data/environments_eol/diff/environments_eol_measurements_31129.diff (185769 lines)
[INFO] [2024-05-25 01:41:42] Loading measurements diff file into memory (185769 lines)...
[INFO] [2024-05-25 01:41:46] Storing 9999 Traits (19998/10000/185769)
[INFO] [2024-05-25 01:41:50] Storing 9999 MetaTraits (19998/10000/185769)
[INFO] [2024-05-25 01:41:55] Storing 10000 Traits (39998/20000/185769)
[INFO] [2024-05-25 01:41:59] Storing 10000 MetaTraits (39998/20000/185769)
[INFO] [2024-05-25 01:42:04] Storing 10000 Traits (59998/30000/185769)
[INFO] [2024-05-25 01:42:08] Storing 10000 MetaTraits (59998/30000/185769)
[INFO] [2024-05-25 01:42:13] Storing 10000 Traits (79998/40000/185769)
[INFO] [2024-05-25 01:42:16] Storing 10000 MetaTraits (79998/40000/185769)
[INFO] [2024-05-25 01:42:22] Storing 10000 Traits (99998/50000/185769)
[INFO] [2024-05-25 01:42:25] Storing 10000 MetaTraits (99998/50000/185769)
[INFO] [2024-05-25 01:42:31] Storing 10000 Traits (119998/60000/185769)
[INFO] [2024-05-25 01:42:34] Storing 10000 MetaTraits (119998/60000/185769)
[INFO] [2024-05-25 01:42:40] Storing 10000 Traits (139998/70000/185769)
[INFO] [2024-05-25 01:42:43] Storing 10000 MetaTraits (139998/70000/185769)
[INFO] [2024-05-25 01:42:49] Storing 10000 Traits (159998/80000/185769)
[INFO] [2024-05-25 01:42:53] Storing 10000 MetaTraits (159998/80000/185769)
[INFO] [2024-05-25 01:42:58] Storing 10000 Traits (179998/90000/185769)
[INFO] [2024-05-25 01:43:02] Storing 10000 MetaTraits (179998/90000/185769)
[INFO] [2024-05-25 01:43:07] Storing 10000 Traits (199998/100000/185769)
[INFO] [2024-05-25 01:43:11] Storing 10000 MetaTraits (199998/100000/185769)
[INFO] [2024-05-25 01:43:17] Storing 10000 Traits (219998/110000/185769)
[INFO] [2024-05-25 01:43:20] Storing 10000 MetaTraits (219998/110000/185769)
[INFO] [2024-05-25 01:43:25] Storing 10000 Traits (239998/120000/185769)
[INFO] [2024-05-25 01:43:29] Storing 10000 MetaTraits (239998/120000/185769)
[INFO] [2024-05-25 01:43:35] Storing 10000 Traits (259998/130000/185769)
[INFO] [2024-05-25 01:43:39] Storing 10000 MetaTraits (259998/130000/185769)
[INFO] [2024-05-25 01:43:45] Storing 10000 Traits (279998/140000/185769)
[INFO] [2024-05-25 01:43:48] Storing 10000 MetaTraits (279998/140000/185769)
[INFO] [2024-05-25 01:43:54] Storing 10000 Traits (299998/150000/185769)
[INFO] [2024-05-25 01:43:57] Storing 10000 MetaTraits (299998/150000/185769)
[INFO] [2024-05-25 01:44:03] Storing 10000 Traits (319998/160000/185769)
[INFO] [2024-05-25 01:44:06] Storing 10000 MetaTraits (319998/160000/185769)
[INFO] [2024-05-25 01:44:12] Storing 10000 Traits (339998/170000/185769)
[INFO] [2024-05-25 01:44:16] Storing 10000 MetaTraits (339998/170000/185769)
[INFO] [2024-05-25 01:44:22] Storing 10000 Traits (359998/180000/185769)
[INFO] [2024-05-25 01:44:25] Storing 10000 MetaTraits (359998/180000/185769)
[INFO] [2024-05-25 01:44:30] Storing 5768 Traits (371534/185767/185769)
[INFO] [2024-05-25 01:44:32] Storing 5768 MetaTraits (371534/185767/185769)
[STOP] [2024-05-25 01:44:32] parse_diff_and_store
[START] [2024-05-25 01:44:32] resolve_keys
[2024-05-25 01:46:43] Resolving downloaded urls (this is not actually downloading them yet)
[INFO] [2024-05-25 01:46:55] Occurrences to nodes (through scientific_names)...
[INFO] [2024-05-25 01:47:11] traits to occurrences...
[INFO] [2024-05-25 01:47:23] traits to nodes (through occurrences)...
[INFO] [2024-05-25 01:47:34] Traits to sex term...
[INFO] [2024-05-25 01:47:37] Traits to lifestage term...
[INFO] [2024-05-25 01:47:39] MetaTraits to traits...
[INFO] [2024-05-25 01:47:48] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2024-05-25 01:47:48] Assocs to occurrences...
[INFO] [2024-05-25 01:47:48] Assocs to nodes...
[INFO] [2024-05-25 01:47:48] Assoc to sex term...
[INFO] [2024-05-25 01:47:48] Assoc to lifestage term...
[INFO] [2024-05-25 01:47:48] MetaAssoc to assocs...
[STOP] [2024-05-25 01:47:48] resolve_keys
[START] [2024-05-25 01:47:48] hold_for_later_1
[STOP] [2024-05-25 01:47:48] hold_for_later_1
[START] [2024-05-25 01:47:48] hold_for_later_2
[STOP] [2024-05-25 01:47:48] hold_for_later_2
[START] [2024-05-25 01:47:48] resolve_missing_parents
[STOP] [2024-05-25 01:47:58] resolve_missing_parents
[START] [2024-05-25 01:47:58] rebuild_nodes
[START] [2024-05-25 01:47:58] Flattener#flatten
[START] [2024-05-25 01:47:58] Flattener#study_resource
[START] [2024-05-25 01:47:59] Flattener#build_ancestry
[STOP] [2024-05-25 01:49:16] Flattener#build_ancestry
[INFO] [2024-05-25 01:49:16] 162291 ancestry keys
[START] [2024-05-25 01:49:16] build_node_ancestors
[INFO] [2024-05-25 01:49:16] old ancestors deleted.
[STOP] [2024-05-25 01:56:01] build_node_ancestors
[START] [2024-05-25 01:56:08] Flattener#propagate_ancestor_ids
[STOP] [2024-05-25 01:58:49] Flattener#propagate_ancestor_ids
[STOP] [2024-05-25 01:58:49] Flattener#flatten
[STOP] [2024-05-25 01:58:49] rebuild_nodes
[START] [2024-05-25 01:58:49] resolve_missing_media_owners
[STOP] [2024-05-25 01:58:49] resolve_missing_media_owners
[START] [2024-05-25 01:58:49] sanitize_media_verbatims
[STOP] [2024-05-25 01:58:49] sanitize_media_verbatims
[START] [2024-05-25 01:58:49] queue_downloads
[STOP] [2024-05-25 01:58:49] queue_downloads
[START] [2024-05-25 01:58:49] parse_names
[WARN] [2024-05-25 01:58:50] I see 162291 names which still need to be parsed.
[WARN] [2024-05-25 01:58:55] Names to parse: 10000 formatted: 10000 learned: 9998 parsed: 10000
[WARN] [2024-05-25 01:59:03] Names to parse: 10000 formatted: 10000 learned: 9997 parsed: 10000
[WARN] [2024-05-25 01:59:10] Names to parse: 10000 formatted: 10000 learned: 9999 parsed: 10000
[WARN] [2024-05-25 01:59:17] Names to parse: 10000 formatted: 10000 learned: 9999 parsed: 10000
[WARN] [2024-05-25 01:59:24] Names to parse: 10000 formatted: 10000 learned: 9998 parsed: 10000
[WARN] [2024-05-25 01:59:30] Names to parse: 10000 formatted: 10000 learned: 9999 parsed: 10000
[WARN] [2024-05-25 01:59:37] Names to parse: 10000 formatted: 10000 learned: 9995 parsed: 10000
[WARN] [2024-05-25 01:59:44] Names to parse: 10000 formatted: 10000 learned: 9996 parsed: 10000
[WARN] [2024-05-25 01:59:51] Names to parse: 10000 formatted: 10000 learned: 9997 parsed: 10000
[WARN] [2024-05-25 01:59:58] Names to parse: 10000 formatted: 10000 learned: 9998 parsed: 10000
[WARN] [2024-05-25 02:00:05] Names to parse: 10000 formatted: 10000 learned: 9998 parsed: 10000
[WARN] [2024-05-25 02:00:12] Names to parse: 10000 formatted: 10000 learned: 9992 parsed: 10000
[WARN] [2024-05-25 02:00:19] Names to parse: 10000 formatted: 10000 learned: 9997 parsed: 10000
[WARN] [2024-05-25 02:00:26] Names to parse: 10000 formatted: 10000 learned: 9995 parsed: 10000
[WARN] [2024-05-25 02:00:33] Names to parse: 10000 formatted: 10000 learned: 9998 parsed: 10000
[WARN] [2024-05-25 02:00:40] Names to parse: 10000 formatted: 10000 learned: 9996 parsed: 10000
[WARN] [2024-05-25 02:00:46] Names to parse: 2291 formatted: 2291 learned: 2290 parsed: 2291
[STOP] [2024-05-25 02:00:49] parse_names
[START] [2024-05-25 02:00:49] denormalize_canonical_names_to_nodes
[STOP] [2024-05-25 02:00:53] denormalize_canonical_names_to_nodes
[START] [2024-05-25 02:00:53] match_nodes
[START] [2024-05-25 02:00:53] map_all_nodes_to_pages
[STOP] [2024-05-25 02:41:02] map_all_nodes_to_pages
[INFO] [2024-05-25 02:41:02] 9913 Unmatched nodes (of 162291)! That's too many to output. Full list in /app/public/data/environments_eol/unmatched_nodes.txt ; First 10: Canonical: Cryptocalicium; Node#152337391; ResourceID: Q118873929; Canonical: Paralophia; Node#152366861; ResourceID: Q15898960; Canonical: Cacota; Node#152369437; ResourceID: Q1655073; Canonical: Aurelia ayla; Node#152337451; ResourceID: Q119010151; Canonical: Aurelia cebimarensis; Node#152337452; ResourceID: Q119010178; Canonical: Wernerius; Node#152392381; ResourceID: Q21367139; Canonical: Parakaryon myojinensis; Node#152395741; ResourceID: Q22329203; Canonical: incertae sedis; Node#152398774; ResourceID: Q235536; Canonical: Biota; Node#152399284; ResourceID: Q2382443; Canonical: Prokaryota; Node#152382320; ResourceID: Q19081
[START] [2024-05-25 02:41:02] update_nodes
[STOP] [2024-05-25 02:41:39] update_nodes
[STOP] [2024-05-25 02:41:39] match_nodes
[START] [2024-05-25 02:41:39] reindex_search
[STOP] [2024-05-25 02:47:12] reindex_search
[START] [2024-05-25 02:47:12] normalize_units
[STOP] [2024-05-25 02:48:28] normalize_units
[START] [2024-05-25 02:48:28] calculate_statistics
[INFO] [2024-05-25 02:48:53] Duplicate page_id count: 0
[STOP] [2024-05-25 02:48:54] calculate_statistics
[START] [2024-05-25 02:48:54] complete_harvest_instance
[START] [2024-05-25 02:48:54] overall_tsv_creation
[INFO] [2024-05-25 02:48:54] Exporting 162291 nodes as TSV in batches of 10000...
[INFO] [2024-05-25 02:48:54] Processing group of 162291 in 17 batches of 10000
[INFO] [2024-05-25 02:51:04] 8494 Traits (unfiltered) and 0 associations...
[INFO] [2024-05-25 02:51:04] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-05-25 02:51:08] Mapped 8494 traits (8494 meta) for 10000 nodes.
[INFO] [2024-05-25 02:51:08] Building Associations map (this can take a while)...
[INFO] [2024-05-25 02:51:14] Done. 0 assocs mapped (0 meta).
[INFO] [2024-05-25 02:51:15] Adding 8494 traits...
[INFO] [2024-05-25 02:51:16] 0 metadata added.
[INFO] [2024-05-25 02:51:16] Adding 0 assocs...
[INFO] [2024-05-25 02:51:16] 0 metadata added.
[INFO] [2024-05-25 02:52:31] Processed 10000/162291 nodes
[INFO] [2024-05-25 02:54:46] 13840 Traits (unfiltered) and 0 associations...
[INFO] [2024-05-25 02:54:46] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-05-25 02:54:55] Mapped 13840 traits (13840 meta) for 10000 nodes.
[INFO] [2024-05-25 02:54:55] Building Associations map (this can take a while)...
[INFO] [2024-05-25 02:54:59] Done. 0 assocs mapped (0 meta).
[INFO] [2024-05-25 02:54:59] Adding 13840 traits...
[INFO] [2024-05-25 02:55:01] 0 metadata added.
[INFO] [2024-05-25 02:55:01] Adding 0 assocs...
[INFO] [2024-05-25 02:55:01] 0 metadata added.
[INFO] [2024-05-25 02:56:15] Processed 20000/162291 nodes
[INFO] [2024-05-25 02:58:33] 7680 Traits (unfiltered) and 0 associations...
[INFO] [2024-05-25 02:58:33] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-05-25 02:58:37] Mapped 7680 traits (7680 meta) for 10000 nodes.
[INFO] [2024-05-25 02:58:37] Building Associations map (this can take a while)...
[INFO] [2024-05-25 02:58:41] Done. 0 assocs mapped (0 meta).
[INFO] [2024-05-25 02:58:41] Adding 7680 traits...
[INFO] [2024-05-25 02:58:42] 0 metadata added.
[INFO] [2024-05-25 02:58:42] Adding 0 assocs...
[INFO] [2024-05-25 02:58:42] 0 metadata added.
[INFO] [2024-05-25 02:59:56] Processed 30000/162291 nodes
[INFO] [2024-05-25 03:02:08] 13834 Traits (unfiltered) and 0 associations...
[INFO] [2024-05-25 03:02:08] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-05-25 03:02:15] Mapped 13834 traits (13834 meta) for 10000 nodes.
[INFO] [2024-05-25 03:02:15] Building Associations map (this can take a while)...
[INFO] [2024-05-25 03:02:19] Done. 0 assocs mapped (0 meta).
[INFO] [2024-05-25 03:02:19] Adding 13834 traits...
[INFO] [2024-05-25 03:02:20] 0 metadata added.
[INFO] [2024-05-25 03:02:20] Adding 0 assocs...
[INFO] [2024-05-25 03:02:20] 0 metadata added.
[INFO] [2024-05-25 03:03:37] Processed 40000/162291 nodes
[INFO] [2024-05-25 03:05:51] 12182 Traits (unfiltered) and 0 associations...
[INFO] [2024-05-25 03:05:51] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-05-25 03:06:02] Mapped 12182 traits (12182 meta) for 10000 nodes.
[INFO] [2024-05-25 03:06:02] Building Associations map (this can take a while)...
[INFO] [2024-05-25 03:06:06] Done. 0 assocs mapped (0 meta).
[INFO] [2024-05-25 03:06:06] Adding 12182 traits...
[INFO] [2024-05-25 03:06:07] 0 metadata added.
[INFO] [2024-05-25 03:06:07] Adding 0 assocs...
[INFO] [2024-05-25 03:06:07] 0 metadata added.
[INFO] [2024-05-25 03:07:21] Processed 50000/162291 nodes
[INFO] [2024-05-25 03:09:38] 8869 Traits (unfiltered) and 0 associations...
[INFO] [2024-05-25 03:09:38] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-05-25 03:09:42] Mapped 8869 traits (8869 meta) for 10000 nodes.
[INFO] [2024-05-25 03:09:42] Building Associations map (this can take a while)...
[INFO] [2024-05-25 03:09:46] Done. 0 assocs mapped (0 meta).
[INFO] [2024-05-25 03:09:46] Adding 8869 traits...
[INFO] [2024-05-25 03:09:47] 0 metadata added.
[INFO] [2024-05-25 03:09:47] Adding 0 assocs...
[INFO] [2024-05-25 03:09:47] 0 metadata added.
[INFO] [2024-05-25 03:11:01] Processed 60000/162291 nodes
[INFO] [2024-05-25 03:13:17] 9596 Traits (unfiltered) and 0 associations...
[INFO] [2024-05-25 03:13:17] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-05-25 03:13:23] Mapped 9596 traits (9596 meta) for 10000 nodes.
[INFO] [2024-05-25 03:13:23] Building Associations map (this can take a while)...
[INFO] [2024-05-25 03:13:28] Done. 0 assocs mapped (0 meta).
[INFO] [2024-05-25 03:13:28] Adding 9596 traits...
[INFO] [2024-05-25 03:13:28] 0 metadata added.
[INFO] [2024-05-25 03:13:28] Adding 0 assocs...
[INFO] [2024-05-25 03:13:28] 0 metadata added.
[INFO] [2024-05-25 03:14:45] Processed 70000/162291 nodes
[INFO] [2024-05-25 03:16:59] 13008 Traits (unfiltered) and 0 associations...
[INFO] [2024-05-25 03:16:59] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-05-25 03:17:05] Mapped 13008 traits (13008 meta) for 10000 nodes.
[INFO] [2024-05-25 03:17:05] Building Associations map (this can take a while)...
[INFO] [2024-05-25 03:17:10] Done. 0 assocs mapped (0 meta).
[INFO] [2024-05-25 03:17:10] Adding 13008 traits...
[INFO] [2024-05-25 03:17:11] 0 metadata added.
[INFO] [2024-05-25 03:17:11] Adding 0 assocs...
[INFO] [2024-05-25 03:17:11] 0 metadata added.
[INFO] [2024-05-25 03:18:27] Processed 80000/162291 nodes
[INFO] [2024-05-25 03:20:45] 14172 Traits (unfiltered) and 0 associations...
[INFO] [2024-05-25 03:20:45] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-05-25 03:20:52] Mapped 14172 traits (14172 meta) for 10000 nodes.
[INFO] [2024-05-25 03:20:52] Building Associations map (this can take a while)...
[INFO] [2024-05-25 03:20:56] Done. 0 assocs mapped (0 meta).
[INFO] [2024-05-25 03:20:56] Adding 14172 traits...
[INFO] [2024-05-25 03:20:57] 0 metadata added.
[INFO] [2024-05-25 03:20:57] Adding 0 assocs...
[INFO] [2024-05-25 03:20:57] 0 metadata added.
[INFO] [2024-05-25 03:22:14] Processed 90000/162291 nodes
[INFO] [2024-05-25 03:24:33] 12875 Traits (unfiltered) and 0 associations...
[INFO] [2024-05-25 03:24:33] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-05-25 03:24:39] Mapped 12875 traits (12875 meta) for 10000 nodes.
[INFO] [2024-05-25 03:24:39] Building Associations map (this can take a while)...
[INFO] [2024-05-25 03:24:43] Done. 0 assocs mapped (0 meta).
[INFO] [2024-05-25 03:24:43] Adding 12875 traits...
[INFO] [2024-05-25 03:24:44] 0 metadata added.
[INFO] [2024-05-25 03:24:44] Adding 0 assocs...
[INFO] [2024-05-25 03:24:44] 0 metadata added.
[INFO] [2024-05-25 03:25:57] Processed 100000/162291 nodes
[INFO] [2024-05-25 03:28:12] 10717 Traits (unfiltered) and 0 associations...
[INFO] [2024-05-25 03:28:12] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-05-25 03:28:22] Mapped 10717 traits (10717 meta) for 10000 nodes.
[INFO] [2024-05-25 03:28:22] Building Associations map (this can take a while)...
[INFO] [2024-05-25 03:28:26] Done. 0 assocs mapped (0 meta).
[INFO] [2024-05-25 03:28:26] Adding 10717 traits...
[INFO] [2024-05-25 03:28:27] 0 metadata added.
[INFO] [2024-05-25 03:28:27] Adding 0 assocs...
[INFO] [2024-05-25 03:28:27] 0 metadata added.
[INFO] [2024-05-25 03:29:40] Processed 110000/162291 nodes
[INFO] [2024-05-25 03:31:55] 11399 Traits (unfiltered) and 0 associations...
[INFO] [2024-05-25 03:31:55] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-05-25 03:32:01] Mapped 11399 traits (11399 meta) for 10000 nodes.
[INFO] [2024-05-25 03:32:01] Building Associations map (this can take a while)...
[INFO] [2024-05-25 03:32:05] Done. 0 assocs mapped (0 meta).
[INFO] [2024-05-25 03:32:05] Adding 11399 traits...
[INFO] [2024-05-25 03:32:06] 0 metadata added.
[INFO] [2024-05-25 03:32:06] Adding 0 assocs...
[INFO] [2024-05-25 03:32:06] 0 metadata added.
[INFO] [2024-05-25 03:33:18] Processed 120000/162291 nodes
[INFO] [2024-05-25 03:35:31] 12072 Traits (unfiltered) and 0 associations...
[INFO] [2024-05-25 03:35:31] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-05-25 03:35:41] Mapped 12072 traits (12072 meta) for 10000 nodes.
[INFO] [2024-05-25 03:35:41] Building Associations map (this can take a while)...
[INFO] [2024-05-25 03:35:45] Done. 0 assocs mapped (0 meta).
[INFO] [2024-05-25 03:35:45] Adding 12072 traits...
[INFO] [2024-05-25 03:35:46] 0 metadata added.
[INFO] [2024-05-25 03:35:46] Adding 0 assocs...
[INFO] [2024-05-25 03:35:46] 0 metadata added.
[INFO] [2024-05-25 03:36:58] Processed 130000/162291 nodes
[INFO] [2024-05-25 03:39:14] 11098 Traits (unfiltered) and 0 associations...
[INFO] [2024-05-25 03:39:14] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-05-25 03:39:19] Mapped 11098 traits (11098 meta) for 10000 nodes.
[INFO] [2024-05-25 03:39:19] Building Associations map (this can take a while)...
[INFO] [2024-05-25 03:39:23] Done. 0 assocs mapped (0 meta).
[INFO] [2024-05-25 03:39:23] Adding 11098 traits...
[INFO] [2024-05-25 03:39:24] 0 metadata added.
[INFO] [2024-05-25 03:39:24] Adding 0 assocs...
[INFO] [2024-05-25 03:39:24] 0 metadata added.
[INFO] [2024-05-25 03:40:37] Processed 140000/162291 nodes
[INFO] [2024-05-25 03:42:51] 10341 Traits (unfiltered) and 0 associations...
[INFO] [2024-05-25 03:42:51] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-05-25 03:42:58] Mapped 10341 traits (10341 meta) for 10000 nodes.
[INFO] [2024-05-25 03:42:58] Building Associations map (this can take a while)...
[INFO] [2024-05-25 03:43:03] Done. 0 assocs mapped (0 meta).
[INFO] [2024-05-25 03:43:03] Adding 10341 traits...
[INFO] [2024-05-25 03:43:04] 0 metadata added.
[INFO] [2024-05-25 03:43:04] Adding 0 assocs...
[INFO] [2024-05-25 03:43:04] 0 metadata added.
[INFO] [2024-05-25 03:44:18] Processed 150000/162291 nodes
[INFO] [2024-05-25 03:46:33] 12373 Traits (unfiltered) and 0 associations...
[INFO] [2024-05-25 03:46:33] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-05-25 03:46:39] Mapped 12373 traits (12373 meta) for 10000 nodes.
[INFO] [2024-05-25 03:46:39] Building Associations map (this can take a while)...
[INFO] [2024-05-25 03:46:43] Done. 0 assocs mapped (0 meta).
[INFO] [2024-05-25 03:46:43] Adding 12373 traits...
[INFO] [2024-05-25 03:46:44] 0 metadata added.
[INFO] [2024-05-25 03:46:44] Adding 0 assocs...
[INFO] [2024-05-25 03:46:44] 0 metadata added.
[INFO] [2024-05-25 03:47:57] Processed 160000/162291 nodes
[INFO] [2024-05-25 03:48:09] 3217 Traits (unfiltered) and 0 associations...
[INFO] [2024-05-25 03:48:09] Building Traits map for 2291 nodes (this can take a while)...
[INFO] [2024-05-25 03:48:10] Mapped 3217 traits (3217 meta) for 2291 nodes.
[INFO] [2024-05-25 03:48:10] Building Associations map (this can take a while)...
[INFO] [2024-05-25 03:48:10] Done. 0 assocs mapped (0 meta).
[INFO] [2024-05-25 03:48:10] Adding 3217 traits...
[INFO] [2024-05-25 03:48:10] 0 metadata added.
[INFO] [2024-05-25 03:48:10] Adding 0 assocs...
[INFO] [2024-05-25 03:48:10] 0 metadata added.
[INFO] [2024-05-25 03:49:13] Processed 162291/162291 nodes
[INFO] [2024-05-25 03:49:13] Average Time: 107.906
[INFO] [2024-05-25 03:49:13] Total Time: 1h19s
[INFO] [2024-05-25 03:49:13] last 3 / first 3: 0.88
[INFO] [2024-05-25 03:49:13] Std.Dev: 10.222; Max: 115.56
[STOP] [2024-05-25 03:49:13] overall_tsv_creation
[INFO] [2024-05-25 03:49:13] Done. Check your files:
[INFO] [2024-05-25 03:49:13] (162291 lines) /app/public/data/environments_eol/publish_nodes.tsv
[INFO] [2024-05-25 03:49:13] (162291 lines) /app/public/data/environments_eol/publish_identifiers.tsv
[INFO] [2024-05-25 03:49:14] (3585946 lines) /app/public/data/environments_eol/publish_node_ancestors.tsv
[INFO] [2024-05-25 03:49:14] (162291 lines) /app/public/data/environments_eol/publish_scientific_names.tsv
[INFO] [2024-05-25 03:49:14] (185768 lines) /app/public/data/environments_eol/publish_traits.tsv
[INFO] [2024-05-25 03:49:14] (1 lines) /app/public/data/environments_eol/publish_metadata.tsv
[STOP] [2024-05-25 03:49:15] complete_harvest_instance
[START] [2024-05-25 03:49:15] completed
[STOP] [2024-05-25 03:49:15] completed
[STOP] [2024-05-25 03:49:15] logged process, took 7900.51

Latest Process

25 May 01:37 EXITED currently processing 0