Harvest for MADtraits Created 04 Oct 09:06

Harvesting Log

[INFO] [2022-10-04 09:06:33] Created harvest instance #4220
[STOP] [2022-10-04 09:06:33] create_harvest_instance
[START] [2022-10-04 09:06:33] fetch_files
[STOP] [2022-10-04 09:06:33] fetch_files
[START] [2022-10-04 09:06:33] validate_each_file
[INFO] [2022-10-04 09:06:33] Looping over 4 formats...
[INFO] [2022-10-04 09:06:33] ...refs (/app/public/data/natdb_dwca_dw_na/reference.tab)
[INFO] [2022-10-04 09:06:33] Valid: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_refs_29769.csv (11 lines)
[INFO] [2022-10-04 09:06:33] ...nodes (/app/public/data/natdb_dwca_dw_na/taxon.tab)
[INFO] [2022-10-04 09:06:33] Valid: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_nodes_29768.csv (2778 lines)
[INFO] [2022-10-04 09:06:33] ...occurrences (/app/public/data/natdb_dwca_dw_na/occurrence_specific.tab)
[INFO] [2022-10-04 09:06:35] Valid: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_occurrences_29770.csv (96894 lines)
[INFO] [2022-10-04 09:06:35] ...measurements (/app/public/data/natdb_dwca_dw_na/measurement_or_fact_specific.tab)
[INFO] [2022-10-04 09:06:47] Valid: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_measurements_29771.csv (211530 lines)
[STOP] [2022-10-04 09:06:47] validate_each_file
[START] [2022-10-04 09:06:47] convert_to_csv
[INFO] [2022-10-04 09:06:47] Looping over 4 formats...
[INFO] [2022-10-04 09:06:47] ...refs (/app/public/data/natdb_dwca_dw_na/reference.tab)
[CMD] [2022-10-04 09:06:47] /usr/bin/sort /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_refs_29769.csv > /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_refs_29769.csv_sorted
[INFO] [2022-10-04 09:06:48] Converted: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_refs_29769.csv (11 lines)
[INFO] [2022-10-04 09:06:48] ...nodes (/app/public/data/natdb_dwca_dw_na/taxon.tab)
[CMD] [2022-10-04 09:06:48] /usr/bin/sort /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_nodes_29768.csv > /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_nodes_29768.csv_sorted
[INFO] [2022-10-04 09:06:48] Converted: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_nodes_29768.csv (2778 lines)
[INFO] [2022-10-04 09:06:48] ...occurrences (/app/public/data/natdb_dwca_dw_na/occurrence_specific.tab)
[CMD] [2022-10-04 09:06:48] /usr/bin/sort /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_occurrences_29770.csv > /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_occurrences_29770.csv_sorted
[INFO] [2022-10-04 09:06:48] Converted: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_occurrences_29770.csv (96894 lines)
[INFO] [2022-10-04 09:06:48] ...measurements (/app/public/data/natdb_dwca_dw_na/measurement_or_fact_specific.tab)
[CMD] [2022-10-04 09:06:48] /usr/bin/sort /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_measurements_29771.csv > /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_measurements_29771.csv_sorted
[INFO] [2022-10-04 09:06:50] Converted: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_measurements_29771.csv (211530 lines)
[STOP] [2022-10-04 09:06:50] convert_to_csv
[START] [2022-10-04 09:06:50] calculate_delta
[INFO] [2022-10-04 09:06:50] Looping over 4 formats...
[INFO] [2022-10-04 09:06:50] ...refs (/app/public/data/natdb_dwca_dw_na/reference.tab)
[CMD] [2022-10-04 09:06:50] echo "0a" > /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_refs_29769.diff
[CMD] [2022-10-04 09:06:50] tail -n +1 /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_refs_29769.csv >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_refs_29769.diff
[CMD] [2022-10-04 09:06:50] echo "." >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_refs_29769.diff
[INFO] [2022-10-04 09:06:50] Created diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_refs_29769.diff (13 lines)
[INFO] [2022-10-04 09:06:50] ...nodes (/app/public/data/natdb_dwca_dw_na/taxon.tab)
[CMD] [2022-10-04 09:06:50] echo "0a" > /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_nodes_29768.diff
[CMD] [2022-10-04 09:06:50] tail -n +1 /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_nodes_29768.csv >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_nodes_29768.diff
[CMD] [2022-10-04 09:06:50] echo "." >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_nodes_29768.diff
[INFO] [2022-10-04 09:06:50] Created diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_nodes_29768.diff (2780 lines)
[INFO] [2022-10-04 09:06:50] ...occurrences (/app/public/data/natdb_dwca_dw_na/occurrence_specific.tab)
[CMD] [2022-10-04 09:06:50] echo "0a" > /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_occurrences_29770.diff
[CMD] [2022-10-04 09:06:51] tail -n +1 /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_occurrences_29770.csv >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_occurrences_29770.diff
[CMD] [2022-10-04 09:06:51] echo "." >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_occurrences_29770.diff
[INFO] [2022-10-04 09:06:51] Created diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_occurrences_29770.diff (96896 lines)
[INFO] [2022-10-04 09:06:51] ...measurements (/app/public/data/natdb_dwca_dw_na/measurement_or_fact_specific.tab)
[CMD] [2022-10-04 09:06:51] echo "0a" > /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_measurements_29771.diff
[CMD] [2022-10-04 09:06:51] tail -n +1 /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_measurements_29771.csv >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_measurements_29771.diff
[CMD] [2022-10-04 09:06:52] echo "." >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_measurements_29771.diff
[INFO] [2022-10-04 09:06:52] Created diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_measurements_29771.diff (211532 lines)
[STOP] [2022-10-04 09:06:52] calculate_delta
[START] [2022-10-04 09:06:52] parse_diff_and_store
[INFO] [2022-10-04 09:06:52] Handling diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_refs_29769.diff (13 lines)
[INFO] [2022-10-04 09:06:52] Loading refs diff file into memory (13 lines)...
[INFO] [2022-10-04 09:06:52] Storing 11 References (11/11/13)
[INFO] [2022-10-04 09:06:52] Handling diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_nodes_29768.diff (2780 lines)
[INFO] [2022-10-04 09:06:53] Loading nodes diff file into memory (2780 lines)...
[WARN] [2022-10-04 09:06:53] Filtered Scientific Name `Entelurus  aequoreus` to `Entelurus aequoreus`
[WARN] [2022-10-04 09:06:53] Filtered Scientific Name `Mesocestoides corti/variabilis/vogae` to `Mesocestoides cortivariabilisvogae`
[WARN] [2022-10-04 09:06:53] Filtered Scientific Name `Mesocestoides lineatus/litteratus` to `Mesocestoides lineatuslitteratus`
[WARN] [2022-10-04 09:06:53] Filtered Scientific Name `Parkesia  motacilla` to `Parkesia motacilla`
[WARN] [2022-10-04 09:06:53] Filtered Scientific Name `Protostrongylus pulmonalis/kamenskyi` to `Protostrongylus pulmonaliskamenskyi`
[WARN] [2022-10-04 09:06:53] Filtered Scientific Name `Protostrongylus stilesi/rushi` to `Protostrongylus stilesirushi`
[WARN] [2022-10-04 09:06:53] Filtered Scientific Name `Taenia ovis/krabbei` to `Taenia oviskrabbei`
[INFO] [2022-10-04 09:06:53] Storing 2814 ScientificNames (5628/2778/2780)
[INFO] [2022-10-04 09:06:54] Storing 2814 Nodes (5628/2778/2780)
[INFO] [2022-10-04 09:06:55] Handling diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_occurrences_29770.diff (96896 lines)
[INFO] [2022-10-04 09:06:55] Loading occurrences diff file into memory (96896 lines)...
[INFO] [2022-10-04 09:06:56] Storing 9999 Occurrences (10120/10000/96896)
[INFO] [2022-10-04 09:06:57] Storing 121 OccurrenceMetadata (10120/10000/96896)
[WARN] [2022-10-04 09:06:58] SKIPPED 125 Occurrence metadata (20245/20000/96896) with resource_pks already be in the database!
[INFO] [2022-10-04 09:06:58] Storing 10000 Occurrences (20245/20000/96896)
[INFO] [2022-10-04 09:07:00] Storing 0 OccurrenceMetadata (20245/20000/96896)
[WARN] [2022-10-04 09:07:00] No models to import, skipping!
[WARN] [2022-10-04 09:07:01] SKIPPED 116 Occurrence metadata (30361/30000/96896) with resource_pks already be in the database!
[INFO] [2022-10-04 09:07:01] Storing 10000 Occurrences (30361/30000/96896)
[INFO] [2022-10-04 09:07:02] Storing 0 OccurrenceMetadata (30361/30000/96896)
[WARN] [2022-10-04 09:07:02] No models to import, skipping!
[WARN] [2022-10-04 09:07:03] SKIPPED 126 Occurrence metadata (40487/40000/96896) with resource_pks already be in the database!
[INFO] [2022-10-04 09:07:03] Storing 10000 Occurrences (40487/40000/96896)
[INFO] [2022-10-04 09:07:04] Storing 0 OccurrenceMetadata (40487/40000/96896)
[WARN] [2022-10-04 09:07:04] No models to import, skipping!
[WARN] [2022-10-04 09:07:05] SKIPPED 105 Occurrence metadata (50592/50000/96896) with resource_pks already be in the database!
[INFO] [2022-10-04 09:07:05] Storing 10000 Occurrences (50592/50000/96896)
[INFO] [2022-10-04 09:07:07] Storing 0 OccurrenceMetadata (50592/50000/96896)
[WARN] [2022-10-04 09:07:07] No models to import, skipping!
[WARN] [2022-10-04 09:07:08] SKIPPED 141 Occurrence metadata (60733/60000/96896) with resource_pks already be in the database!
[INFO] [2022-10-04 09:07:08] Storing 10000 Occurrences (60733/60000/96896)
[INFO] [2022-10-04 09:07:09] Storing 0 OccurrenceMetadata (60733/60000/96896)
[WARN] [2022-10-04 09:07:09] No models to import, skipping!
[WARN] [2022-10-04 09:07:10] SKIPPED 117 Occurrence metadata (70850/70000/96896) with resource_pks already be in the database!
[INFO] [2022-10-04 09:07:10] Storing 10000 Occurrences (70850/70000/96896)
[INFO] [2022-10-04 09:07:11] Storing 0 OccurrenceMetadata (70850/70000/96896)
[WARN] [2022-10-04 09:07:11] No models to import, skipping!
[WARN] [2022-10-04 09:07:13] SKIPPED 117 Occurrence metadata (80967/80000/96896) with resource_pks already be in the database!
[INFO] [2022-10-04 09:07:13] Storing 10000 Occurrences (80967/80000/96896)
[INFO] [2022-10-04 09:07:14] Storing 0 OccurrenceMetadata (80967/80000/96896)
[WARN] [2022-10-04 09:07:14] No models to import, skipping!
[WARN] [2022-10-04 09:07:15] SKIPPED 103 Occurrence metadata (91070/90000/96896) with resource_pks already be in the database!
[INFO] [2022-10-04 09:07:15] Storing 10000 Occurrences (91070/90000/96896)
[INFO] [2022-10-04 09:07:17] Storing 0 OccurrenceMetadata (91070/90000/96896)
[WARN] [2022-10-04 09:07:17] No models to import, skipping!
[WARN] [2022-10-04 09:07:18] SKIPPED 78 Occurrence metadata (98043/96894/96896) with resource_pks already be in the database!
[INFO] [2022-10-04 09:07:18] Storing 6895 Occurrences (98043/96894/96896)
[INFO] [2022-10-04 09:07:19] Storing 0 OccurrenceMetadata (98043/96894/96896)
[WARN] [2022-10-04 09:07:19] No models to import, skipping!
[INFO] [2022-10-04 09:07:19] Handling diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_measurements_29771.diff (211532 lines)
[INFO] [2022-10-04 09:07:19] Loading measurements diff file into memory (211532 lines)...
[INFO] [2022-10-04 09:07:23] Storing 9999 Traits (22196/10000/211532)
[INFO] [2022-10-04 09:07:26] Storing 7621 MetaTraits (22196/10000/211532)
[INFO] [2022-10-04 09:07:28] Storing 4576 TraitsReferences (22196/10000/211532)
[INFO] [2022-10-04 09:07:32] Storing 4589 TraitsReferences (44436/20000/211532)
[INFO] [2022-10-04 09:07:33] Storing 10000 Traits (44436/20000/211532)
[INFO] [2022-10-04 09:07:36] Storing 7651 MetaTraits (44436/20000/211532)
[INFO] [2022-10-04 09:07:41] Storing 10000 Traits (66658/30000/211532)
[INFO] [2022-10-04 09:07:44] Storing 7655 MetaTraits (66658/30000/211532)
[INFO] [2022-10-04 09:07:45] Storing 4567 TraitsReferences (66658/30000/211532)
[INFO] [2022-10-04 09:07:49] Storing 4548 TraitsReferences (88839/40000/211532)
[INFO] [2022-10-04 09:07:49] Storing 10000 Traits (88839/40000/211532)
[INFO] [2022-10-04 09:07:52] Storing 7633 MetaTraits (88839/40000/211532)
[INFO] [2022-10-04 09:07:57] Storing 10000 Traits (110977/50000/211532)
[INFO] [2022-10-04 09:08:00] Storing 7577 MetaTraits (110977/50000/211532)
[INFO] [2022-10-04 09:08:01] Storing 4561 TraitsReferences (110977/50000/211532)
[INFO] [2022-10-04 09:08:06] Storing 10000 Traits (133201/60000/211532)
[INFO] [2022-10-04 09:08:09] Storing 7622 MetaTraits (133201/60000/211532)
[INFO] [2022-10-04 09:08:10] Storing 4602 TraitsReferences (133201/60000/211532)
[INFO] [2022-10-04 09:08:14] Storing 10000 Traits (155145/70000/211532)
[INFO] [2022-10-04 09:08:17] Storing 7516 MetaTraits (155145/70000/211532)
[INFO] [2022-10-04 09:08:19] Storing 4428 TraitsReferences (155145/70000/211532)
[INFO] [2022-10-04 09:08:23] Storing 10000 Traits (177370/80000/211532)
[INFO] [2022-10-04 09:08:30] Storing 7625 MetaTraits (177370/80000/211532)
[INFO] [2022-10-04 09:08:31] Storing 4600 TraitsReferences (177370/80000/211532)
[INFO] [2022-10-04 09:08:36] Storing 10000 Traits (199539/90000/211532)
[INFO] [2022-10-04 09:08:40] Storing 7592 MetaTraits (199539/90000/211532)
[INFO] [2022-10-04 09:08:41] Storing 4577 TraitsReferences (199539/90000/211532)
[INFO] [2022-10-04 09:08:46] Storing 10000 Traits (221691/100000/211532)
[INFO] [2022-10-04 09:08:50] Storing 4544 TraitsReferences (221691/100000/211532)
[INFO] [2022-10-04 09:08:50] Storing 7608 MetaTraits (221691/100000/211532)
[INFO] [2022-10-04 09:08:55] Storing 10000 Traits (243888/110000/211532)
[INFO] [2022-10-04 09:08:59] Storing 7601 MetaTraits (243888/110000/211532)
[INFO] [2022-10-04 09:09:00] Storing 4596 TraitsReferences (243888/110000/211532)
[INFO] [2022-10-04 09:09:05] Storing 4594 TraitsReferences (266126/120000/211532)
[INFO] [2022-10-04 09:09:05] Storing 10000 Traits (266126/120000/211532)
[INFO] [2022-10-04 09:09:09] Storing 7644 MetaTraits (266126/120000/211532)
[INFO] [2022-10-04 09:09:14] Storing 4622 TraitsReferences (288440/130000/211532)
[INFO] [2022-10-04 09:09:15] Storing 10000 Traits (288440/130000/211532)
[INFO] [2022-10-04 09:09:20] Storing 7692 MetaTraits (288440/130000/211532)
[INFO] [2022-10-04 09:09:26] Storing 4572 TraitsReferences (310597/140000/211532)
[INFO] [2022-10-04 09:09:26] Storing 10000 Traits (310597/140000/211532)
[INFO] [2022-10-04 09:09:29] Storing 7585 MetaTraits (310597/140000/211532)
[INFO] [2022-10-04 09:09:34] Storing 4658 TraitsReferences (332963/150000/211532)
[INFO] [2022-10-04 09:09:35] Storing 10000 Traits (332963/150000/211532)
[INFO] [2022-10-04 09:09:39] Storing 7708 MetaTraits (332963/150000/211532)
[INFO] [2022-10-04 09:09:44] Storing 4525 TraitsReferences (355046/160000/211532)
[INFO] [2022-10-04 09:09:45] Storing 10000 Traits (355046/160000/211532)
[INFO] [2022-10-04 09:09:49] Storing 7558 MetaTraits (355046/160000/211532)
[INFO] [2022-10-04 09:09:54] Storing 4581 TraitsReferences (377282/170000/211532)
[INFO] [2022-10-04 09:09:55] Storing 10000 Traits (377282/170000/211532)
[INFO] [2022-10-04 09:09:59] Storing 7655 MetaTraits (377282/170000/211532)
[INFO] [2022-10-04 09:10:05] Storing 4670 TraitsReferences (399582/180000/211532)
[INFO] [2022-10-04 09:10:06] Storing 10000 Traits (399582/180000/211532)
[INFO] [2022-10-04 09:10:09] Storing 7630 MetaTraits (399582/180000/211532)
[INFO] [2022-10-04 09:10:15] Storing 10000 Traits (421700/190000/211532)
[INFO] [2022-10-04 09:10:18] Storing 7577 MetaTraits (421700/190000/211532)
[INFO] [2022-10-04 09:10:19] Storing 4541 TraitsReferences (421700/190000/211532)
[INFO] [2022-10-04 09:10:24] Storing 4624 TraitsReferences (443972/200000/211532)
[INFO] [2022-10-04 09:10:25] Storing 10000 Traits (443972/200000/211532)
[INFO] [2022-10-04 09:10:27] Storing 7648 MetaTraits (443972/200000/211532)
[INFO] [2022-10-04 09:10:34] Storing 10000 Traits (466216/210000/211532)
[INFO] [2022-10-04 09:10:37] Storing 7637 MetaTraits (466216/210000/211532)
[INFO] [2022-10-04 09:10:38] Storing 4607 TraitsReferences (466216/210000/211532)
[INFO] [2022-10-04 09:10:40] Storing 1531 Traits (469615/211530/211532)
[INFO] [2022-10-04 09:10:41] Storing 1156 MetaTraits (469615/211530/211532)
[INFO] [2022-10-04 09:10:41] Storing 712 TraitsReferences (469615/211530/211532)
[STOP] [2022-10-04 09:10:41] parse_diff_and_store
[START] [2022-10-04 09:10:41] resolve_keys
[2022-10-04 09:10:44] Resolving downloaded urls (this is not actually downloading them yet)
[INFO] [2022-10-04 09:10:58] Occurrences to nodes (through scientific_names)...
[INFO] [2022-10-04 09:11:00] traits to occurrences...
[INFO] [2022-10-04 09:11:14] traits to nodes (through occurrences)...
[INFO] [2022-10-04 09:11:16] Traits to sex term...
[INFO] [2022-10-04 09:11:17] Traits to lifestage term...
[INFO] [2022-10-04 09:11:19] MetaTraits to traits...
[INFO] [2022-10-04 09:11:23] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2022-10-04 09:11:32] Assocs to occurrences...
[INFO] [2022-10-04 09:11:32] Assocs to nodes...
[INFO] [2022-10-04 09:11:32] Assoc to sex term...
[INFO] [2022-10-04 09:11:32] Assoc to lifestage term...
[INFO] [2022-10-04 09:11:32] MetaAssoc to assocs...
[STOP] [2022-10-04 09:11:32] resolve_keys
[START] [2022-10-04 09:11:32] hold_for_later_1
[STOP] [2022-10-04 09:11:32] hold_for_later_1
[START] [2022-10-04 09:11:32] hold_for_later_2
[STOP] [2022-10-04 09:11:32] hold_for_later_2
[START] [2022-10-04 09:11:32] resolve_missing_parents
[STOP] [2022-10-04 09:11:32] resolve_missing_parents
[START] [2022-10-04 09:11:32] rebuild_nodes
[START] [2022-10-04 09:11:32] Flattener#flatten
[START] [2022-10-04 09:11:32] Flattener#study_resource
[START] [2022-10-04 09:11:32] Flattener#build_ancestry
[STOP] [2022-10-04 09:11:32] Flattener#build_ancestry
[INFO] [2022-10-04 09:11:32] 2814 ancestry keys
[START] [2022-10-04 09:11:32] build_node_ancestors
[INFO] [2022-10-04 09:11:32] old ancestors deleted.
[STOP] [2022-10-04 09:11:32] build_node_ancestors
[START] [2022-10-04 09:11:33] Flattener#propagate_ancestor_ids
[STOP] [2022-10-04 09:11:33] Flattener#propagate_ancestor_ids
[STOP] [2022-10-04 09:11:33] Flattener#flatten
[STOP] [2022-10-04 09:11:33] rebuild_nodes
[START] [2022-10-04 09:11:33] resolve_missing_media_owners
[STOP] [2022-10-04 09:11:33] resolve_missing_media_owners
[START] [2022-10-04 09:11:33] sanitize_media_verbatims
[STOP] [2022-10-04 09:11:33] sanitize_media_verbatims
[START] [2022-10-04 09:11:33] queue_downloads
[STOP] [2022-10-04 09:11:33] queue_downloads
[START] [2022-10-04 09:11:33] parse_names
[WARN] [2022-10-04 09:11:33] I see 2814 names which still need to be parsed.
[WARN] [2022-10-04 09:11:33] Names to parse: 2814 formatted: 2814 learned: 2814 parsed: 2814
[STOP] [2022-10-04 09:11:35] parse_names
[START] [2022-10-04 09:11:35] denormalize_canonical_names_to_nodes
[STOP] [2022-10-04 09:11:35] denormalize_canonical_names_to_nodes
[START] [2022-10-04 09:11:35] match_nodes
[START] [2022-10-04 09:11:36] map_all_nodes_to_pages
[STOP] [2022-10-04 09:12:22] map_all_nodes_to_pages
[INFO] [2022-10-04 09:12:22] 711 Unmatched nodes (of 2814)! That's too many to output. Full list in /app/public/data/natdb_dwca_dw_na/unmatched_nodes.txt ; First 10: Canonical: Abbreviata antarctica; Node#118763744; ResourceID: abbreviata_antarctica; Canonical: Abbreviata caucasica; Node#118763745; ResourceID: abbreviata_caucasica; Canonical: Abbreviata kazachstanica; Node#118763746; ResourceID: abbreviata_kazachstanica; Canonical: Abbreviata turkomanica; Node#118763747; ResourceID: abbreviata_turkomanica; Canonical: Acanthocephalus; Node#118763775; ResourceID: acanthocephalus_sp.; Canonical: Acanthocirrus retrirostris; Node#118763777; ResourceID: acanthocirrus_retrirostris; Canonical: Cunionaceae; Node#118766537; ResourceID: Angiosperms/Cunionaceae; Canonical: Acer; Node#118763801; ResourceID: acer_spp.; Canonical: Acuaria anthuris; Node#118763811; ResourceID: acuaria_anthuris; Canonical: Acuaria depressa; Node#118763812; ResourceID: acuaria_depressa
[START] [2022-10-04 09:12:22] update_nodes
[STOP] [2022-10-04 09:12:23] update_nodes
[STOP] [2022-10-04 09:12:23] match_nodes
[START] [2022-10-04 09:12:23] reindex_search
[STOP] [2022-10-04 09:12:24] reindex_search
[START] [2022-10-04 09:12:24] normalize_units
[STOP] [2022-10-04 09:22:56] normalize_units
[START] [2022-10-04 09:22:56] calculate_statistics
[INFO] [2022-10-04 09:24:34] Duplicate page_id count: 0
[STOP] [2022-10-04 09:24:35] calculate_statistics
[START] [2022-10-04 09:24:35] complete_harvest_instance
[START] [2022-10-04 09:24:35] overall_tsv_creation
[INFO] [2022-10-04 09:24:35] Processing group of 2814 in 1 batches of 10000
[INFO] [2022-10-04 09:27:01] 96894 Traits (unfiltered)...
[INFO] [2022-10-04 09:27:01] Building Traits map (this can take a while)...
[INFO] [2022-10-04 09:29:36] 10000 traits mapped (9781 meta)...
[INFO] [2022-10-04 09:31:21] 20000 traits mapped (19588 meta)...
[INFO] [2022-10-04 09:33:06] 30000 traits mapped (29399 meta)...
[INFO] [2022-10-04 09:34:53] 40000 traits mapped (39177 meta)...
[INFO] [2022-10-04 09:36:40] 50000 traits mapped (48969 meta)...
[INFO] [2022-10-04 09:38:25] 60000 traits mapped (58773 meta)...
[INFO] [2022-10-04 09:40:14] 70000 traits mapped (68586 meta)...
[INFO] [2022-10-04 09:42:03] 80000 traits mapped (78381 meta)...
[INFO] [2022-10-04 09:43:51] 90000 traits mapped (88209 meta)...
[INFO] [2022-10-04 09:45:07] Done. 96894 traits mapped (94965 meta).
[INFO] [2022-10-04 09:45:07] Building Associations map (this can take a while)...
[INFO] [2022-10-04 09:45:22] Done. 0 assocs mapped (0 meta).
[INFO] [2022-10-04 09:45:22] Adding 96894 traits...
[INFO] [2022-10-04 09:45:25] Trait #279312021 in key 279312021 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:25] Trait #279312735 in key 279312735 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:25] Trait #279312799 in key 279312799 has 22 metadata... that seems high?
[INFO] [2022-10-04 09:45:25] Trait #279313987 in key 279313987 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:26] Trait #279320432 in key 279320432 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:26] Trait #279321706 in key 279321706 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:26] Trait #279321843 in key 279321843 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:28] Trait #279334867 in key 279334867 has 22 metadata... that seems high?
[INFO] [2022-10-04 09:45:28] Trait #279336821 in key 279336821 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:28] Trait #279338559 in key 279338559 has 22 metadata... that seems high?
[INFO] [2022-10-04 09:45:28] Trait #279338649 in key 279338649 has 22 metadata... that seems high?
[INFO] [2022-10-04 09:45:28] Trait #279339895 in key 279339895 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:30] Trait #279354567 in key 279354567 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:30] Trait #279355236 in key 279355236 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:32] Trait #279371103 in key 279371103 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:33] Trait #279377381 in key 279377381 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:33] Trait #279377397 in key 279377397 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:34] Trait #279383289 in key 279383289 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:36] Trait #279389317 in key 279389317 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:36] Trait #279389489 in key 279389489 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:37] Trait #279396421 in key 279396421 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:39] Trait #279402485 in key 279402485 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:39] Trait #279402658 in key 279402658 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:39] Trait #279404544 in key 279404544 has 22 metadata... that seems high?
[INFO] [2022-10-04 09:45:40] Trait #279407307 in key 279407307 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:41] Trait #279409671 in key 279409671 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:41] Trait #279411697 in key 279411697 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:42] Trait #279418617 in key 279418617 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:42] Trait #279421838 in key 279421838 has 22 metadata... that seems high?
[INFO] [2022-10-04 09:45:42] Trait #279424180 in key 279424180 has 22 metadata... that seems high?
[INFO] [2022-10-04 09:45:42] Trait #279424750 in key 279424750 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:43] Trait #279428903 in key 279428903 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:43] Trait #279431719 in key 279431719 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:44] Trait #279442399 in key 279442399 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:46] Trait #279453291 in key 279453291 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:46] Trait #279454858 in key 279454858 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:46] Trait #279455642 in key 279455642 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:47] Trait #279461276 in key 279461276 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:47] Trait #279467547 in key 279467547 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:48] Trait #279470787 in key 279470787 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:48] Trait #279473117 in key 279473117 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:48] Trait #279476294 in key 279476294 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:49] Trait #279478229 in key 279478229 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:49] Trait #279482545 in key 279482545 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:50] Trait #279492305 in key 279492305 has 21 metadata... that seems high?
[INFO] [2022-10-04 09:45:50] Trait #279495106 in key 279495106 has 22 metadata... that seems high?
[INFO] [2022-10-04 09:45:51] Trait #279495806 in key 279495806 has 22 metadata... that seems high?
[INFO] [2022-10-04 09:45:51] 195606 metadata added.
[INFO] [2022-10-04 09:45:51] Adding 0 assocs...
[INFO] [2022-10-04 09:45:51] 0 metadata added.
[INFO] [2022-10-04 09:46:55] Average Time: 1214.84
[INFO] [2022-10-04 09:46:55] Total Time: 22m20s
[STOP] [2022-10-04 09:46:55] overall_tsv_creation
[INFO] [2022-10-04 09:46:55] Done. Check your files:
[INFO] [2022-10-04 09:46:55] (2814 lines) /app/public/data/natdb_dwca_dw_na/publish_nodes.tsv
[INFO] [2022-10-04 09:46:55] (199 lines) /app/public/data/natdb_dwca_dw_na/publish_node_ancestors.tsv
[INFO] [2022-10-04 09:46:55] (2814 lines) /app/public/data/natdb_dwca_dw_na/publish_scientific_names.tsv
[INFO] [2022-10-04 09:46:55] (96895 lines) /app/public/data/natdb_dwca_dw_na/publish_traits.tsv
[INFO] [2022-10-04 09:46:55] (195607 lines) /app/public/data/natdb_dwca_dw_na/publish_metadata.tsv
[STOP] [2022-10-04 09:46:55] complete_harvest_instance
[START] [2022-10-04 09:46:55] completed
[STOP] [2022-10-04 09:46:55] completed
[STOP] [2022-10-04 09:46:55] logged process, took 2422.73

Latest Process

04 Oct 09:06 EXITED currently processing 0