Harvest for MADtraits Created 26 Oct 08:59

Harvesting Log

[INFO] [2023-10-26 08:59:27] Created harvest instance #4493
[STOP] [2023-10-26 08:59:27] create_harvest_instance
[START] [2023-10-26 08:59:27] fetch_files
[STOP] [2023-10-26 08:59:27] fetch_files
[START] [2023-10-26 08:59:27] validate_each_file
[INFO] [2023-10-26 08:59:27] Looping over 4 formats...
[INFO] [2023-10-26 08:59:27] ...refs (/app/public/data/natdb_dwca_dw_na/reference.tab)
[INFO] [2023-10-26 08:59:27] Valid: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_refs_30968.csv (11 lines)
[INFO] [2023-10-26 08:59:27] ...nodes (/app/public/data/natdb_dwca_dw_na/taxon.tab)
[INFO] [2023-10-26 08:59:27] Valid: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_nodes_30967.csv (2778 lines)
[INFO] [2023-10-26 08:59:27] ...occurrences (/app/public/data/natdb_dwca_dw_na/occurrence_specific.tab)
[INFO] [2023-10-26 08:59:29] Valid: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_occurrences_30969.csv (96894 lines)
[INFO] [2023-10-26 08:59:29] ...measurements (/app/public/data/natdb_dwca_dw_na/measurement_or_fact_specific.tab)
[INFO] [2023-10-26 08:59:40] Valid: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_measurements_30970.csv (211530 lines)
[STOP] [2023-10-26 08:59:40] validate_each_file
[START] [2023-10-26 08:59:40] convert_to_csv
[INFO] [2023-10-26 08:59:40] Looping over 4 formats...
[INFO] [2023-10-26 08:59:40] ...refs (/app/public/data/natdb_dwca_dw_na/reference.tab)
[CMD] [2023-10-26 08:59:40] /usr/bin/sort /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_refs_30968.csv > /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_refs_30968.csv_sorted
[INFO] [2023-10-26 08:59:40] Converted: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_refs_30968.csv (11 lines)
[INFO] [2023-10-26 08:59:40] ...nodes (/app/public/data/natdb_dwca_dw_na/taxon.tab)
[CMD] [2023-10-26 08:59:40] /usr/bin/sort /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_nodes_30967.csv > /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_nodes_30967.csv_sorted
[INFO] [2023-10-26 08:59:40] Converted: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_nodes_30967.csv (2778 lines)
[INFO] [2023-10-26 08:59:40] ...occurrences (/app/public/data/natdb_dwca_dw_na/occurrence_specific.tab)
[CMD] [2023-10-26 08:59:40] /usr/bin/sort /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_occurrences_30969.csv > /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_occurrences_30969.csv_sorted
[INFO] [2023-10-26 08:59:41] Converted: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_occurrences_30969.csv (96894 lines)
[INFO] [2023-10-26 08:59:41] ...measurements (/app/public/data/natdb_dwca_dw_na/measurement_or_fact_specific.tab)
[CMD] [2023-10-26 08:59:41] /usr/bin/sort /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_measurements_30970.csv > /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_measurements_30970.csv_sorted
[INFO] [2023-10-26 08:59:42] Converted: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_measurements_30970.csv (211530 lines)
[STOP] [2023-10-26 08:59:42] convert_to_csv
[START] [2023-10-26 08:59:42] calculate_delta
[INFO] [2023-10-26 08:59:42] Looping over 4 formats...
[INFO] [2023-10-26 08:59:42] ...refs (/app/public/data/natdb_dwca_dw_na/reference.tab)
[CMD] [2023-10-26 08:59:42] echo "0a" > /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_refs_30968.diff
[CMD] [2023-10-26 08:59:42] tail -n +1 /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_refs_30968.csv >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_refs_30968.diff
[CMD] [2023-10-26 08:59:42] echo "." >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_refs_30968.diff
[INFO] [2023-10-26 08:59:42] Created diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_refs_30968.diff (13 lines)
[INFO] [2023-10-26 08:59:42] ...nodes (/app/public/data/natdb_dwca_dw_na/taxon.tab)
[CMD] [2023-10-26 08:59:42] echo "0a" > /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_nodes_30967.diff
[CMD] [2023-10-26 08:59:43] tail -n +1 /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_nodes_30967.csv >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_nodes_30967.diff
[CMD] [2023-10-26 08:59:43] echo "." >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_nodes_30967.diff
[INFO] [2023-10-26 08:59:43] Created diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_nodes_30967.diff (2780 lines)
[INFO] [2023-10-26 08:59:43] ...occurrences (/app/public/data/natdb_dwca_dw_na/occurrence_specific.tab)
[CMD] [2023-10-26 08:59:43] echo "0a" > /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_occurrences_30969.diff
[CMD] [2023-10-26 08:59:43] tail -n +1 /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_occurrences_30969.csv >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_occurrences_30969.diff
[CMD] [2023-10-26 08:59:43] echo "." >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_occurrences_30969.diff
[INFO] [2023-10-26 08:59:43] Created diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_occurrences_30969.diff (96896 lines)
[INFO] [2023-10-26 08:59:43] ...measurements (/app/public/data/natdb_dwca_dw_na/measurement_or_fact_specific.tab)
[CMD] [2023-10-26 08:59:43] echo "0a" > /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_measurements_30970.diff
[CMD] [2023-10-26 08:59:43] tail -n +1 /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_measurements_30970.csv >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_measurements_30970.diff
[CMD] [2023-10-26 08:59:44] echo "." >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_measurements_30970.diff
[INFO] [2023-10-26 08:59:46] Created diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_measurements_30970.diff (211532 lines)
[STOP] [2023-10-26 08:59:46] calculate_delta
[START] [2023-10-26 08:59:46] parse_diff_and_store
[INFO] [2023-10-26 08:59:46] Handling diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_refs_30968.diff (13 lines)
[INFO] [2023-10-26 08:59:46] Loading refs diff file into memory (13 lines)...
[INFO] [2023-10-26 08:59:46] Storing 11 References (11/11/13)
[INFO] [2023-10-26 08:59:46] Handling diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_nodes_30967.diff (2780 lines)
[INFO] [2023-10-26 08:59:46] Loading nodes diff file into memory (2780 lines)...
[WARN] [2023-10-26 08:59:46] Filtered Scientific Name `Entelurus  aequoreus` to `Entelurus aequoreus`
[WARN] [2023-10-26 08:59:46] Filtered Scientific Name `Mesocestoides corti/variabilis/vogae` to `Mesocestoides cortivariabilisvogae`
[WARN] [2023-10-26 08:59:46] Filtered Scientific Name `Mesocestoides lineatus/litteratus` to `Mesocestoides lineatuslitteratus`
[WARN] [2023-10-26 08:59:46] Filtered Scientific Name `Parkesia  motacilla` to `Parkesia motacilla`
[WARN] [2023-10-26 08:59:46] Filtered Scientific Name `Protostrongylus pulmonalis/kamenskyi` to `Protostrongylus pulmonaliskamenskyi`
[WARN] [2023-10-26 08:59:46] Filtered Scientific Name `Protostrongylus stilesi/rushi` to `Protostrongylus stilesirushi`
[WARN] [2023-10-26 08:59:47] Filtered Scientific Name `Taenia ovis/krabbei` to `Taenia oviskrabbei`
[INFO] [2023-10-26 08:59:47] Storing 2814 ScientificNames (5628/2778/2780)
[INFO] [2023-10-26 08:59:48] Storing 2814 Nodes (5628/2778/2780)
[INFO] [2023-10-26 08:59:48] Handling diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_occurrences_30969.diff (96896 lines)
[INFO] [2023-10-26 08:59:48] Loading occurrences diff file into memory (96896 lines)...
[INFO] [2023-10-26 08:59:49] Storing 9999 Occurrences (10120/10000/96896)
[INFO] [2023-10-26 08:59:51] Storing 121 OccurrenceMetadata (10120/10000/96896)
[WARN] [2023-10-26 08:59:51] SKIPPED 125 Occurrence metadata (20245/20000/96896) with resource_pks already be in the database!
[INFO] [2023-10-26 08:59:51] Storing 10000 Occurrences (20245/20000/96896)
[INFO] [2023-10-26 08:59:53] Storing 0 OccurrenceMetadata (20245/20000/96896)
[WARN] [2023-10-26 08:59:53] No models to import, skipping!
[WARN] [2023-10-26 08:59:54] SKIPPED 116 Occurrence metadata (30361/30000/96896) with resource_pks already be in the database!
[INFO] [2023-10-26 08:59:54] Storing 10000 Occurrences (30361/30000/96896)
[INFO] [2023-10-26 08:59:55] Storing 0 OccurrenceMetadata (30361/30000/96896)
[WARN] [2023-10-26 08:59:55] No models to import, skipping!
[WARN] [2023-10-26 08:59:56] SKIPPED 126 Occurrence metadata (40487/40000/96896) with resource_pks already be in the database!
[INFO] [2023-10-26 08:59:56] Storing 10000 Occurrences (40487/40000/96896)
[INFO] [2023-10-26 08:59:58] Storing 0 OccurrenceMetadata (40487/40000/96896)
[WARN] [2023-10-26 08:59:58] No models to import, skipping!
[WARN] [2023-10-26 08:59:59] SKIPPED 105 Occurrence metadata (50592/50000/96896) with resource_pks already be in the database!
[INFO] [2023-10-26 08:59:59] Storing 10000 Occurrences (50592/50000/96896)
[INFO] [2023-10-26 09:00:00] Storing 0 OccurrenceMetadata (50592/50000/96896)
[WARN] [2023-10-26 09:00:00] No models to import, skipping!
[WARN] [2023-10-26 09:00:01] SKIPPED 141 Occurrence metadata (60733/60000/96896) with resource_pks already be in the database!
[INFO] [2023-10-26 09:00:01] Storing 10000 Occurrences (60733/60000/96896)
[INFO] [2023-10-26 09:00:02] Storing 0 OccurrenceMetadata (60733/60000/96896)
[WARN] [2023-10-26 09:00:02] No models to import, skipping!
[WARN] [2023-10-26 09:00:03] SKIPPED 117 Occurrence metadata (70850/70000/96896) with resource_pks already be in the database!
[INFO] [2023-10-26 09:00:03] Storing 10000 Occurrences (70850/70000/96896)
[INFO] [2023-10-26 09:00:05] Storing 0 OccurrenceMetadata (70850/70000/96896)
[WARN] [2023-10-26 09:00:05] No models to import, skipping!
[WARN] [2023-10-26 09:00:06] SKIPPED 117 Occurrence metadata (80967/80000/96896) with resource_pks already be in the database!
[INFO] [2023-10-26 09:00:06] Storing 10000 Occurrences (80967/80000/96896)
[INFO] [2023-10-26 09:00:07] Storing 0 OccurrenceMetadata (80967/80000/96896)
[WARN] [2023-10-26 09:00:07] No models to import, skipping!
[WARN] [2023-10-26 09:00:09] SKIPPED 103 Occurrence metadata (91070/90000/96896) with resource_pks already be in the database!
[INFO] [2023-10-26 09:00:09] Storing 10000 Occurrences (91070/90000/96896)
[INFO] [2023-10-26 09:00:10] Storing 0 OccurrenceMetadata (91070/90000/96896)
[WARN] [2023-10-26 09:00:10] No models to import, skipping!
[WARN] [2023-10-26 09:00:11] SKIPPED 78 Occurrence metadata (98043/96894/96896) with resource_pks already be in the database!
[INFO] [2023-10-26 09:00:11] Storing 6895 Occurrences (98043/96894/96896)
[INFO] [2023-10-26 09:00:12] Storing 0 OccurrenceMetadata (98043/96894/96896)
[WARN] [2023-10-26 09:00:12] No models to import, skipping!
[INFO] [2023-10-26 09:00:12] Handling diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_measurements_30970.diff (211532 lines)
[INFO] [2023-10-26 09:00:12] Loading measurements diff file into memory (211532 lines)...
[INFO] [2023-10-26 09:00:16] Storing 9999 Traits (22188/10000/211532)
[INFO] [2023-10-26 09:00:19] Storing 7620 MetaTraits (22188/10000/211532)
[INFO] [2023-10-26 09:00:20] Storing 4569 TraitsReferences (22188/10000/211532)
[INFO] [2023-10-26 09:00:25] Storing 10000 Traits (44415/20000/211532)
[INFO] [2023-10-26 09:00:28] Storing 4580 TraitsReferences (44415/20000/211532)
[INFO] [2023-10-26 09:00:28] Storing 7647 MetaTraits (44415/20000/211532)
[INFO] [2023-10-26 09:00:33] Storing 4562 TraitsReferences (66628/30000/211532)
[INFO] [2023-10-26 09:00:34] Storing 10000 Traits (66628/30000/211532)
[INFO] [2023-10-26 09:00:37] Storing 7651 MetaTraits (66628/30000/211532)
[INFO] [2023-10-26 09:00:42] Storing 4553 TraitsReferences (88813/40000/211532)
[INFO] [2023-10-26 09:00:42] Storing 10000 Traits (88813/40000/211532)
[INFO] [2023-10-26 09:00:45] Storing 7632 MetaTraits (88813/40000/211532)
[INFO] [2023-10-26 09:00:50] Storing 10000 Traits (110950/50000/211532)
[INFO] [2023-10-26 09:00:53] Storing 4564 TraitsReferences (110950/50000/211532)
[INFO] [2023-10-26 09:00:54] Storing 7573 MetaTraits (110950/50000/211532)
[INFO] [2023-10-26 09:00:59] Storing 4604 TraitsReferences (133185/60000/211532)
[INFO] [2023-10-26 09:00:59] Storing 10000 Traits (133185/60000/211532)
[INFO] [2023-10-26 09:01:02] Storing 7631 MetaTraits (133185/60000/211532)
[INFO] [2023-10-26 09:01:09] Storing 4429 TraitsReferences (155127/70000/211532)
[INFO] [2023-10-26 09:01:09] Storing 10000 Traits (155127/70000/211532)
[INFO] [2023-10-26 09:01:13] Storing 7513 MetaTraits (155127/70000/211532)
[INFO] [2023-10-26 09:01:19] Storing 10000 Traits (177351/80000/211532)
[INFO] [2023-10-26 09:01:23] Storing 4599 TraitsReferences (177351/80000/211532)
[INFO] [2023-10-26 09:01:24] Storing 7625 MetaTraits (177351/80000/211532)
[INFO] [2023-10-26 09:01:30] Storing 10000 Traits (199513/90000/211532)
[INFO] [2023-10-26 09:01:34] Storing 7591 MetaTraits (199513/90000/211532)
[INFO] [2023-10-26 09:01:35] Storing 4571 TraitsReferences (199513/90000/211532)
[INFO] [2023-10-26 09:01:40] Storing 4545 TraitsReferences (221665/100000/211532)
[INFO] [2023-10-26 09:01:41] Storing 10000 Traits (221665/100000/211532)
[INFO] [2023-10-26 09:01:45] Storing 7607 MetaTraits (221665/100000/211532)
[INFO] [2023-10-26 09:01:51] Storing 10000 Traits (243863/110000/211532)
[INFO] [2023-10-26 09:01:55] Storing 7607 MetaTraits (243863/110000/211532)
[INFO] [2023-10-26 09:01:56] Storing 4591 TraitsReferences (243863/110000/211532)
[INFO] [2023-10-26 09:02:01] Storing 4604 TraitsReferences (266115/120000/211532)
[INFO] [2023-10-26 09:02:01] Storing 10000 Traits (266115/120000/211532)
[INFO] [2023-10-26 09:02:05] Storing 7648 MetaTraits (266115/120000/211532)
[INFO] [2023-10-26 09:02:11] Storing 10000 Traits (288425/130000/211532)
[INFO] [2023-10-26 09:02:15] Storing 7691 MetaTraits (288425/130000/211532)
[INFO] [2023-10-26 09:02:16] Storing 4619 TraitsReferences (288425/130000/211532)
[INFO] [2023-10-26 09:02:21] Storing 4565 TraitsReferences (310574/140000/211532)
[INFO] [2023-10-26 09:02:21] Storing 10000 Traits (310574/140000/211532)
[INFO] [2023-10-26 09:02:25] Storing 7584 MetaTraits (310574/140000/211532)
[INFO] [2023-10-26 09:02:31] Storing 4658 TraitsReferences (332941/150000/211532)
[INFO] [2023-10-26 09:02:31] Storing 10000 Traits (332941/150000/211532)
[INFO] [2023-10-26 09:02:36] Storing 7709 MetaTraits (332941/150000/211532)
[INFO] [2023-10-26 09:02:42] Storing 4532 TraitsReferences (355031/160000/211532)
[INFO] [2023-10-26 09:02:43] Storing 10000 Traits (355031/160000/211532)
[INFO] [2023-10-26 09:02:47] Storing 7558 MetaTraits (355031/160000/211532)
[INFO] [2023-10-26 09:02:54] Storing 4584 TraitsReferences (377273/170000/211532)
[INFO] [2023-10-26 09:02:54] Storing 10000 Traits (377273/170000/211532)
[INFO] [2023-10-26 09:02:58] Storing 7658 MetaTraits (377273/170000/211532)
[INFO] [2023-10-26 09:03:04] Storing 10000 Traits (399569/180000/211532)
[INFO] [2023-10-26 09:03:07] Storing 7626 MetaTraits (399569/180000/211532)
[INFO] [2023-10-26 09:03:08] Storing 4670 TraitsReferences (399569/180000/211532)
[INFO] [2023-10-26 09:03:14] Storing 10000 Traits (421705/190000/211532)
[INFO] [2023-10-26 09:03:17] Storing 7583 MetaTraits (421705/190000/211532)
[INFO] [2023-10-26 09:03:18] Storing 4553 TraitsReferences (421705/190000/211532)
[INFO] [2023-10-26 09:03:23] Storing 4621 TraitsReferences (443972/200000/211532)
[INFO] [2023-10-26 09:03:24] Storing 10000 Traits (443972/200000/211532)
[INFO] [2023-10-26 09:03:28] Storing 7646 MetaTraits (443972/200000/211532)
[INFO] [2023-10-26 09:03:34] Storing 10000 Traits (466218/210000/211532)
[INFO] [2023-10-26 09:03:38] Storing 7636 MetaTraits (466218/210000/211532)
[INFO] [2023-10-26 09:03:39] Storing 4610 TraitsReferences (466218/210000/211532)
[INFO] [2023-10-26 09:03:41] Storing 1531 Traits (469615/211530/211532)
[INFO] [2023-10-26 09:03:41] Storing 1155 MetaTraits (469615/211530/211532)
[INFO] [2023-10-26 09:03:41] Storing 711 TraitsReferences (469615/211530/211532)
[STOP] [2023-10-26 09:03:41] parse_diff_and_store
[START] [2023-10-26 09:03:41] resolve_keys
[2023-10-26 09:03:45] Resolving downloaded urls (this is not actually downloading them yet)
[INFO] [2023-10-26 09:03:53] Occurrences to nodes (through scientific_names)...
[INFO] [2023-10-26 09:03:58] traits to occurrences...
[INFO] [2023-10-26 09:04:11] traits to nodes (through occurrences)...
[INFO] [2023-10-26 09:04:14] Traits to sex term...
[INFO] [2023-10-26 09:04:16] Traits to lifestage term...
[INFO] [2023-10-26 09:04:18] MetaTraits to traits...
[INFO] [2023-10-26 09:04:22] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2023-10-26 09:04:32] Assocs to occurrences...
[INFO] [2023-10-26 09:04:32] Assocs to nodes...
[INFO] [2023-10-26 09:04:32] Assoc to sex term...
[INFO] [2023-10-26 09:04:32] Assoc to lifestage term...
[INFO] [2023-10-26 09:04:32] MetaAssoc to assocs...
[STOP] [2023-10-26 09:04:32] resolve_keys
[START] [2023-10-26 09:04:32] hold_for_later_1
[STOP] [2023-10-26 09:04:32] hold_for_later_1
[START] [2023-10-26 09:04:32] hold_for_later_2
[STOP] [2023-10-26 09:04:32] hold_for_later_2
[START] [2023-10-26 09:04:32] resolve_missing_parents
[STOP] [2023-10-26 09:04:32] resolve_missing_parents
[START] [2023-10-26 09:04:32] rebuild_nodes
[START] [2023-10-26 09:04:32] Flattener#flatten
[START] [2023-10-26 09:04:32] Flattener#study_resource
[START] [2023-10-26 09:04:32] Flattener#build_ancestry
[STOP] [2023-10-26 09:04:32] Flattener#build_ancestry
[INFO] [2023-10-26 09:04:32] 2814 ancestry keys
[START] [2023-10-26 09:04:32] build_node_ancestors
[INFO] [2023-10-26 09:04:32] old ancestors deleted.
[STOP] [2023-10-26 09:04:32] build_node_ancestors
[START] [2023-10-26 09:04:32] Flattener#propagate_ancestor_ids
[STOP] [2023-10-26 09:04:32] Flattener#propagate_ancestor_ids
[STOP] [2023-10-26 09:04:32] Flattener#flatten
[STOP] [2023-10-26 09:04:32] rebuild_nodes
[START] [2023-10-26 09:04:32] resolve_missing_media_owners
[STOP] [2023-10-26 09:04:32] resolve_missing_media_owners
[START] [2023-10-26 09:04:32] sanitize_media_verbatims
[STOP] [2023-10-26 09:04:32] sanitize_media_verbatims
[START] [2023-10-26 09:04:32] queue_downloads
[STOP] [2023-10-26 09:04:32] queue_downloads
[START] [2023-10-26 09:04:32] parse_names
[WARN] [2023-10-26 09:04:32] I see 2814 names which still need to be parsed.
[WARN] [2023-10-26 09:04:32] Names to parse: 2814 formatted: 2814 learned: 2814 parsed: 2814
[STOP] [2023-10-26 09:04:35] parse_names
[START] [2023-10-26 09:04:35] denormalize_canonical_names_to_nodes
[STOP] [2023-10-26 09:04:35] denormalize_canonical_names_to_nodes
[START] [2023-10-26 09:04:35] match_nodes
[START] [2023-10-26 09:04:35] map_all_nodes_to_pages
[STOP] [2023-10-26 09:05:32] map_all_nodes_to_pages
[INFO] [2023-10-26 09:05:32] 714 Unmatched nodes (of 2814)! That's too many to output. Full list in /app/public/data/natdb_dwca_dw_na/unmatched_nodes.txt ; First 10: Canonical: Abbreviata antarctica; Node#137243519; ResourceID: abbreviata_antarctica; Canonical: Abbreviata caucasica; Node#137243520; ResourceID: abbreviata_caucasica; Canonical: Abbreviata kazachstanica; Node#137243521; ResourceID: abbreviata_kazachstanica; Canonical: Abbreviata turkomanica; Node#137243522; ResourceID: abbreviata_turkomanica; Canonical: Acanthocephalus; Node#137243550; ResourceID: acanthocephalus_sp.; Canonical: Acanthocirrus retrirostris; Node#137243552; ResourceID: acanthocirrus_retrirostris; Canonical: Cunionaceae; Node#137246312; ResourceID: Angiosperms/Cunionaceae; Canonical: Acer; Node#137243576; ResourceID: acer_spp.; Canonical: Acuaria anthuris; Node#137243586; ResourceID: acuaria_anthuris; Canonical: Acuaria depressa; Node#137243587; ResourceID: acuaria_depressa
[START] [2023-10-26 09:05:32] update_nodes
[STOP] [2023-10-26 09:05:33] update_nodes
[STOP] [2023-10-26 09:05:33] match_nodes
[START] [2023-10-26 09:05:33] reindex_search
[STOP] [2023-10-26 09:05:35] reindex_search
[START] [2023-10-26 09:05:35] normalize_units
[STOP] [2023-10-26 09:16:53] normalize_units
[START] [2023-10-26 09:16:53] calculate_statistics
[INFO] [2023-10-26 09:18:10] Duplicate page_id count: 0
[STOP] [2023-10-26 09:18:11] calculate_statistics
[START] [2023-10-26 09:18:11] complete_harvest_instance
[START] [2023-10-26 09:18:11] overall_tsv_creation
[INFO] [2023-10-26 09:18:11] Exporting 2814 nodes as TSV in batches of 10000...
[INFO] [2023-10-26 09:18:11] Processing group of 2814 in 1 batches of 10000
[INFO] [2023-10-26 09:18:15] 96894 Traits (unfiltered) and 0 associations...
[INFO] [2023-10-26 09:18:15] Building Traits map for 2814 nodes (this can take a while)...
[INFO] [2023-10-26 09:19:18] Mapped 96894 traits (94965 meta) for 2814 nodes.
[INFO] [2023-10-26 09:19:18] Building Associations map (this can take a while)...
[INFO] [2023-10-26 09:19:18] Done. 0 assocs mapped (0 meta).
[INFO] [2023-10-26 09:19:18] Adding 96894 traits...
[INFO] [2023-10-26 09:19:23] Trait #291994239 in key 291994239 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:24] Trait #291994954 in key 291994954 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:24] Trait #291995018 in key 291995018 has 22 metadata... that seems high?
[INFO] [2023-10-26 09:19:24] Trait #291996207 in key 291996207 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:24] Trait #292002652 in key 292002652 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:24] Trait #292003923 in key 292003923 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:24] Trait #292004060 in key 292004060 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:26] Trait #292017064 in key 292017064 has 22 metadata... that seems high?
[INFO] [2023-10-26 09:19:26] Trait #292019023 in key 292019023 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:26] Trait #292020756 in key 292020756 has 22 metadata... that seems high?
[INFO] [2023-10-26 09:19:26] Trait #292020846 in key 292020846 has 22 metadata... that seems high?
[INFO] [2023-10-26 09:19:27] Trait #292022094 in key 292022094 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:28] Trait #292036782 in key 292036782 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:29] Trait #292037449 in key 292037449 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:30] Trait #292053314 in key 292053314 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:31] Trait #292059601 in key 292059601 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:31] Trait #292059617 in key 292059617 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:32] Trait #292065498 in key 292065498 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:33] Trait #292071536 in key 292071536 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:33] Trait #292071709 in key 292071709 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:33] Trait #292078662 in key 292078662 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:34] Trait #292084718 in key 292084718 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:34] Trait #292084891 in key 292084891 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:34] Trait #292086772 in key 292086772 has 22 metadata... that seems high?
[INFO] [2023-10-26 09:19:35] Trait #292089532 in key 292089532 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:35] Trait #292091895 in key 292091895 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:35] Trait #292093925 in key 292093925 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:36] Trait #292100844 in key 292100844 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:36] Trait #292104073 in key 292104073 has 22 metadata... that seems high?
[INFO] [2023-10-26 09:19:37] Trait #292106418 in key 292106418 has 22 metadata... that seems high?
[INFO] [2023-10-26 09:19:37] Trait #292106987 in key 292106987 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:38] Trait #292111134 in key 292111134 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:38] Trait #292113935 in key 292113935 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:39] Trait #292124614 in key 292124614 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:40] Trait #292135496 in key 292135496 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:41] Trait #292137057 in key 292137057 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:41] Trait #292137840 in key 292137840 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:41] Trait #292143461 in key 292143461 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:42] Trait #292149742 in key 292149742 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:43] Trait #292152986 in key 292152986 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:43] Trait #292155321 in key 292155321 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:43] Trait #292158499 in key 292158499 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:43] Trait #292160444 in key 292160444 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:44] Trait #292164761 in key 292164761 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:46] Trait #292174512 in key 292174512 has 21 metadata... that seems high?
[INFO] [2023-10-26 09:19:47] Trait #292177318 in key 292177318 has 22 metadata... that seems high?
[INFO] [2023-10-26 09:19:47] Trait #292178018 in key 292178018 has 22 metadata... that seems high?
[INFO] [2023-10-26 09:19:48] 195606 metadata added.
[INFO] [2023-10-26 09:19:48] Adding 0 assocs...
[INFO] [2023-10-26 09:19:48] 0 metadata added.
[INFO] [2023-10-26 09:20:49] Processed 2814/2814 nodes
[INFO] [2023-10-26 09:20:49] Average Time: 157.03
[INFO] [2023-10-26 09:20:49] Total Time: 2m38s
[STOP] [2023-10-26 09:20:49] overall_tsv_creation
[INFO] [2023-10-26 09:20:49] Done. Check your files:
[INFO] [2023-10-26 09:20:49] (2814 lines) /app/public/data/natdb_dwca_dw_na/publish_nodes.tsv
[INFO] [2023-10-26 09:20:49] (199 lines) /app/public/data/natdb_dwca_dw_na/publish_node_ancestors.tsv
[INFO] [2023-10-26 09:20:49] (2814 lines) /app/public/data/natdb_dwca_dw_na/publish_scientific_names.tsv
[INFO] [2023-10-26 09:20:49] (96895 lines) /app/public/data/natdb_dwca_dw_na/publish_traits.tsv
[INFO] [2023-10-26 09:20:49] (195607 lines) /app/public/data/natdb_dwca_dw_na/publish_metadata.tsv
[STOP] [2023-10-26 09:20:49] complete_harvest_instance
[START] [2023-10-26 09:20:49] completed
[STOP] [2023-10-26 09:20:49] completed
[STOP] [2023-10-26 09:20:49] logged process, took 1282.69

Latest Process

26 Oct 08:59 EXITED currently processing 0