Harvest for CITES Created 26 Aug 15:10

Harvesting Log

[INFO] [2022-08-26 15:10:14] Created harvest instance #4207
[STOP] [2022-08-26 15:10:14] create_harvest_instance
[START] [2022-08-26 15:10:14] fetch_files
[STOP] [2022-08-26 15:10:14] fetch_files
[START] [2022-08-26 15:10:14] validate_each_file
[INFO] [2022-08-26 15:10:14] Looping over 4 formats...
[INFO] [2022-08-26 15:10:14] ...nodes (/app/public/data/cites_taxa_tar_g/taxon.tab)
[INFO] [2022-08-26 15:10:19] Valid: /app/public/data/cites_taxa_tar_g/converted_csv/cites_taxa_tar_g_nodes_28817.csv (80558 lines)
[INFO] [2022-08-26 15:10:19] ...vernaculars (/app/public/data/cites_taxa_tar_g/vernacular_name.tab)
[INFO] [2022-08-26 15:10:20] Valid: /app/public/data/cites_taxa_tar_g/converted_csv/cites_taxa_tar_g_vernaculars_28820.csv (30405 lines)
[INFO] [2022-08-26 15:10:20] ...occurrences (/app/public/data/cites_taxa_tar_g/occurrence.tab)
[INFO] [2022-08-26 15:10:23] Valid: /app/public/data/cites_taxa_tar_g/converted_csv/cites_taxa_tar_g_occurrences_28818.csv (154576 lines)
[INFO] [2022-08-26 15:10:23] ...measurements (/app/public/data/cites_taxa_tar_g/measurement_or_fact.tab)
[INFO] [2022-08-26 15:10:36] Valid: /app/public/data/cites_taxa_tar_g/converted_csv/cites_taxa_tar_g_measurements_28819.csv (154576 lines)
[STOP] [2022-08-26 15:10:36] validate_each_file
[START] [2022-08-26 15:10:36] convert_to_csv
[INFO] [2022-08-26 15:10:36] Looping over 4 formats...
[INFO] [2022-08-26 15:10:36] ...nodes (/app/public/data/cites_taxa_tar_g/taxon.tab)
[CMD] [2022-08-26 15:10:36] /usr/bin/sort /app/public/data/cites_taxa_tar_g/converted_csv/cites_taxa_tar_g_nodes_28817.csv > /app/public/data/cites_taxa_tar_g/converted_csv/cites_taxa_tar_g_nodes_28817.csv_sorted
[INFO] [2022-08-26 15:10:36] Converted: /app/public/data/cites_taxa_tar_g/converted_csv/cites_taxa_tar_g_nodes_28817.csv (80558 lines)
[INFO] [2022-08-26 15:10:36] ...vernaculars (/app/public/data/cites_taxa_tar_g/vernacular_name.tab)
[CMD] [2022-08-26 15:10:36] /usr/bin/sort /app/public/data/cites_taxa_tar_g/converted_csv/cites_taxa_tar_g_vernaculars_28820.csv > /app/public/data/cites_taxa_tar_g/converted_csv/cites_taxa_tar_g_vernaculars_28820.csv_sorted
[INFO] [2022-08-26 15:10:37] Converted: /app/public/data/cites_taxa_tar_g/converted_csv/cites_taxa_tar_g_vernaculars_28820.csv (30405 lines)
[INFO] [2022-08-26 15:10:37] ...occurrences (/app/public/data/cites_taxa_tar_g/occurrence.tab)
[CMD] [2022-08-26 15:10:37] /usr/bin/sort /app/public/data/cites_taxa_tar_g/converted_csv/cites_taxa_tar_g_occurrences_28818.csv > /app/public/data/cites_taxa_tar_g/converted_csv/cites_taxa_tar_g_occurrences_28818.csv_sorted
[INFO] [2022-08-26 15:10:37] Converted: /app/public/data/cites_taxa_tar_g/converted_csv/cites_taxa_tar_g_occurrences_28818.csv (154576 lines)
[INFO] [2022-08-26 15:10:37] ...measurements (/app/public/data/cites_taxa_tar_g/measurement_or_fact.tab)
[CMD] [2022-08-26 15:10:37] /usr/bin/sort /app/public/data/cites_taxa_tar_g/converted_csv/cites_taxa_tar_g_measurements_28819.csv > /app/public/data/cites_taxa_tar_g/converted_csv/cites_taxa_tar_g_measurements_28819.csv_sorted
[INFO] [2022-08-26 15:10:39] Converted: /app/public/data/cites_taxa_tar_g/converted_csv/cites_taxa_tar_g_measurements_28819.csv (154576 lines)
[STOP] [2022-08-26 15:10:39] convert_to_csv
[START] [2022-08-26 15:10:39] calculate_delta
[INFO] [2022-08-26 15:10:39] Looping over 4 formats...
[INFO] [2022-08-26 15:10:39] ...nodes (/app/public/data/cites_taxa_tar_g/taxon.tab)
[CMD] [2022-08-26 15:10:39] echo "0a" > /app/public/data/cites_taxa_tar_g/diff/cites_taxa_tar_g_nodes_28817.diff
[CMD] [2022-08-26 15:10:39] tail -n +1 /app/public/data/cites_taxa_tar_g/converted_csv/cites_taxa_tar_g_nodes_28817.csv >> /app/public/data/cites_taxa_tar_g/diff/cites_taxa_tar_g_nodes_28817.diff
[CMD] [2022-08-26 15:10:39] echo "." >> /app/public/data/cites_taxa_tar_g/diff/cites_taxa_tar_g_nodes_28817.diff
[INFO] [2022-08-26 15:10:39] Created diff: /app/public/data/cites_taxa_tar_g/diff/cites_taxa_tar_g_nodes_28817.diff (80560 lines)
[INFO] [2022-08-26 15:10:39] ...vernaculars (/app/public/data/cites_taxa_tar_g/vernacular_name.tab)
[CMD] [2022-08-26 15:10:39] echo "0a" > /app/public/data/cites_taxa_tar_g/diff/cites_taxa_tar_g_vernaculars_28820.diff
[CMD] [2022-08-26 15:10:39] tail -n +1 /app/public/data/cites_taxa_tar_g/converted_csv/cites_taxa_tar_g_vernaculars_28820.csv >> /app/public/data/cites_taxa_tar_g/diff/cites_taxa_tar_g_vernaculars_28820.diff
[CMD] [2022-08-26 15:10:40] echo "." >> /app/public/data/cites_taxa_tar_g/diff/cites_taxa_tar_g_vernaculars_28820.diff
[INFO] [2022-08-26 15:10:40] Created diff: /app/public/data/cites_taxa_tar_g/diff/cites_taxa_tar_g_vernaculars_28820.diff (30407 lines)
[INFO] [2022-08-26 15:10:40] ...occurrences (/app/public/data/cites_taxa_tar_g/occurrence.tab)
[CMD] [2022-08-26 15:10:40] echo "0a" > /app/public/data/cites_taxa_tar_g/diff/cites_taxa_tar_g_occurrences_28818.diff
[CMD] [2022-08-26 15:10:40] tail -n +1 /app/public/data/cites_taxa_tar_g/converted_csv/cites_taxa_tar_g_occurrences_28818.csv >> /app/public/data/cites_taxa_tar_g/diff/cites_taxa_tar_g_occurrences_28818.diff
[CMD] [2022-08-26 15:10:40] echo "." >> /app/public/data/cites_taxa_tar_g/diff/cites_taxa_tar_g_occurrences_28818.diff
[INFO] [2022-08-26 15:10:40] Created diff: /app/public/data/cites_taxa_tar_g/diff/cites_taxa_tar_g_occurrences_28818.diff (154578 lines)
[INFO] [2022-08-26 15:10:40] ...measurements (/app/public/data/cites_taxa_tar_g/measurement_or_fact.tab)
[CMD] [2022-08-26 15:10:40] echo "0a" > /app/public/data/cites_taxa_tar_g/diff/cites_taxa_tar_g_measurements_28819.diff
[CMD] [2022-08-26 15:10:41] tail -n +1 /app/public/data/cites_taxa_tar_g/converted_csv/cites_taxa_tar_g_measurements_28819.csv >> /app/public/data/cites_taxa_tar_g/diff/cites_taxa_tar_g_measurements_28819.diff
[CMD] [2022-08-26 15:10:42] echo "." >> /app/public/data/cites_taxa_tar_g/diff/cites_taxa_tar_g_measurements_28819.diff
[INFO] [2022-08-26 15:10:43] Created diff: /app/public/data/cites_taxa_tar_g/diff/cites_taxa_tar_g_measurements_28819.diff (154578 lines)
[STOP] [2022-08-26 15:10:43] calculate_delta
[START] [2022-08-26 15:10:43] parse_diff_and_store
[INFO] [2022-08-26 15:10:43] Handling diff: /app/public/data/cites_taxa_tar_g/diff/cites_taxa_tar_g_nodes_28817.diff (80560 lines)
[INFO] [2022-08-26 15:10:43] Loading nodes diff file into memory (80560 lines)...
[WARN] [2022-08-26 15:10:45] Filtered Scientific Name `Galanthus nivalis "flore pleno"` to `Galanthus nivalis flore pleno`
[INFO] [2022-08-26 15:10:47] Storing 10559 ScientificNames (20058/10000/80560)
[INFO] [2022-08-26 15:10:51] Storing 9499 Nodes (20058/10000/80560)
[WARN] [2022-08-26 15:11:03] SKIPPED 371 Scientific names (39379/20000/80560) with resource_pks already be in the database!
[WARN] [2022-08-26 15:11:03] SKIPPED 371 Nodes (39379/20000/80560) with resource_pks already be in the database!
[INFO] [2022-08-26 15:11:03] Storing 10067 ScientificNames (39379/20000/80560)
[INFO] [2022-08-26 15:11:07] Storing 8512 Nodes (39379/20000/80560)
[WARN] [2022-08-26 15:11:12] Filtered Scientific Name `Naultinus "north cape"` to `Naultinus north cape`
[WARN] [2022-08-26 15:11:13] SKIPPED 255 Scientific names (50907/30000/80560) with resource_pks already be in the database!
[WARN] [2022-08-26 15:11:13] SKIPPED 255 Nodes (50907/30000/80560) with resource_pks already be in the database!
[INFO] [2022-08-26 15:11:13] Storing 10005 ScientificNames (50907/30000/80560)
[INFO] [2022-08-26 15:11:17] Storing 1013 Nodes (50907/30000/80560)
[WARN] [2022-08-26 15:11:21] SKIPPED 229 Scientific names (62297/40000/80560) with resource_pks already be in the database!
[WARN] [2022-08-26 15:11:21] SKIPPED 229 Nodes (62297/40000/80560) with resource_pks already be in the database!
[INFO] [2022-08-26 15:11:21] Storing 10003 ScientificNames (62297/40000/80560)
[INFO] [2022-08-26 15:11:26] Storing 929 Nodes (62297/40000/80560)
[WARN] [2022-08-26 15:11:27] Filtered Scientific Name `"dendrobates" mysteriosus` to `dendrobates mysteriosus`
[WARN] [2022-08-26 15:11:29] Filtered Scientific Name `Pachypodium ruthenbergianum var.  perrieri` to `Pachypodium ruthenbergianum var. perrieri`
[WARN] [2022-08-26 15:11:30] SKIPPED 320 Scientific names (79026/50000/80560) with resource_pks already be in the database!
[WARN] [2022-08-26 15:11:30] SKIPPED 320 Nodes (79026/50000/80560) with resource_pks already be in the database!
[INFO] [2022-08-26 15:11:30] Storing 10011 ScientificNames (79026/50000/80560)
[INFO] [2022-08-26 15:11:34] Storing 6078 Nodes (79026/50000/80560)
[WARN] [2022-08-26 15:11:36] Filtered Scientific Name `Dactylocnemis cape"` to `Dactylocnemis cape`
[WARN] [2022-08-26 15:11:36] Filtered Scientific Name `Dactylocnemis cape"` to `Dactylocnemis cape`
[WARN] [2022-08-26 15:11:36] Filtered Scientific Name `Dactylocnemis "mokohinaus"` to `Dactylocnemis mokohinaus`
[WARN] [2022-08-26 15:11:36] Filtered Scientific Name `Woodworthia "Otago/Southland"` to `Woodworthia OtagoSouthland`
[WARN] [2022-08-26 15:11:36] Filtered Scientific Name `Dactylocnemis "Matapia"` to `Dactylocnemis Matapia`
[WARN] [2022-08-26 15:11:36] Filtered Scientific Name `Woodworthia "Cromwell Gorge"` to `Woodworthia Cromwell Gorge`
[WARN] [2022-08-26 15:11:36] Filtered Scientific Name `Woodworthia "Pygmy"` to `Woodworthia Pygmy`
[WARN] [2022-08-26 15:11:36] Filtered Scientific Name `Dactylocnemis "Poor Knights"` to `Dactylocnemis Poor Knights`
[WARN] [2022-08-26 15:11:36] Filtered Scientific Name `Woodworthia "Kaikouras"` to `Woodworthia Kaikouras`
[WARN] [2022-08-26 15:11:36] Filtered Scientific Name `Woodworthia "Central Otago"` to `Woodworthia Central Otago`
[WARN] [2022-08-26 15:11:36] Filtered Scientific Name `Woodworthia "Marlborough Mini"` to `Woodworthia Marlborough Mini`
[WARN] [2022-08-26 15:11:36] Filtered Scientific Name `Dactylocnemis "Three Kings"` to `Dactylocnemis Three Kings`
[WARN] [2022-08-26 15:11:36] Filtered Scientific Name `Woodworthia "Southern Alps"` to `Woodworthia Southern Alps`
[WARN] [2022-08-26 15:11:36] Filtered Scientific Name `Dactylocnemis "North Cape"` to `Dactylocnemis North Cape`
[WARN] [2022-08-26 15:11:36] Filtered Scientific Name `Woodworthia "Southern Mini"` to `Woodworthia Southern Mini`
[WARN] [2022-08-26 15:11:36] Filtered Scientific Name `Woodworthia "Mt. Arthur"` to `Woodworthia Mt. Arthur`
[WARN] [2022-08-26 15:11:36] Filtered Scientific Name `Dactylocnemis "Mokohinaus"` to `Dactylocnemis Mokohinaus`
[WARN] [2022-08-26 15:11:40] SKIPPED 254 Scientific names (99277/60000/80560) with resource_pks already be in the database!
[WARN] [2022-08-26 15:11:40] SKIPPED 254 Nodes (99277/60000/80560) with resource_pks already be in the database!
[INFO] [2022-08-26 15:11:40] Storing 10006 ScientificNames (99277/60000/80560)
[INFO] [2022-08-26 15:11:44] Storing 9737 Nodes (99277/60000/80560)
[WARN] [2022-08-26 15:11:52] SKIPPED 223 Scientific names (115789/70000/80560) with resource_pks already be in the database!
[WARN] [2022-08-26 15:11:52] SKIPPED 223 Nodes (115789/70000/80560) with resource_pks already be in the database!
[INFO] [2022-08-26 15:11:52] Storing 10002 ScientificNames (115789/70000/80560)
[INFO] [2022-08-26 15:11:55] Storing 6064 Nodes (115789/70000/80560)
[WARN] [2022-08-26 15:12:02] SKIPPED 276 Scientific names (128928/80000/80560) with resource_pks already be in the database!
[WARN] [2022-08-26 15:12:02] SKIPPED 276 Nodes (128928/80000/80560) with resource_pks already be in the database!
[INFO] [2022-08-26 15:12:02] Storing 10007 ScientificNames (128928/80000/80560)
[INFO] [2022-08-26 15:12:06] Storing 2580 Nodes (128928/80000/80560)
[WARN] [2022-08-26 15:12:08] SKIPPED 138 Scientific names (130239/80558/80560) with resource_pks already be in the database!
[WARN] [2022-08-26 15:12:08] SKIPPED 138 Nodes (130239/80558/80560) with resource_pks already be in the database!
[INFO] [2022-08-26 15:12:08] Storing 566 ScientificNames (130239/80558/80560)
[INFO] [2022-08-26 15:12:08] Storing 469 Nodes (130239/80558/80560)
[INFO] [2022-08-26 15:12:08] Handling diff: /app/public/data/cites_taxa_tar_g/diff/cites_taxa_tar_g_vernaculars_28820.diff (30407 lines)
[INFO] [2022-08-26 15:12:08] Loading vernaculars diff file into memory (30407 lines)...
[INFO] [2022-08-26 15:12:10] Storing 9999 Vernaculars (9999/10000/30407)
[INFO] [2022-08-26 15:12:14] Storing 10000 Vernaculars (19999/20000/30407)
[INFO] [2022-08-26 15:12:18] Storing 10000 Vernaculars (29999/30000/30407)
[INFO] [2022-08-26 15:12:20] Storing 406 Vernaculars (30405/30405/30407)
[INFO] [2022-08-26 15:12:20] Handling diff: /app/public/data/cites_taxa_tar_g/diff/cites_taxa_tar_g_occurrences_28818.diff (154578 lines)
[INFO] [2022-08-26 15:12:20] Loading occurrences diff file into memory (154578 lines)...
[INFO] [2022-08-26 15:12:21] Storing 9999 Occurrences (9999/10000/154578)
[INFO] [2022-08-26 15:12:25] Storing 10000 Occurrences (19999/20000/154578)
[INFO] [2022-08-26 15:12:28] Storing 10000 Occurrences (29999/30000/154578)
[INFO] [2022-08-26 15:12:30] Storing 10000 Occurrences (39999/40000/154578)
[INFO] [2022-08-26 15:12:33] Storing 10000 Occurrences (49999/50000/154578)
[INFO] [2022-08-26 15:12:36] Storing 10000 Occurrences (59999/60000/154578)
[INFO] [2022-08-26 15:12:40] Storing 10000 Occurrences (69999/70000/154578)
[INFO] [2022-08-26 15:12:43] Storing 10000 Occurrences (79999/80000/154578)
[INFO] [2022-08-26 15:12:46] Storing 10000 Occurrences (89999/90000/154578)
[INFO] [2022-08-26 15:12:49] Storing 10000 Occurrences (99999/100000/154578)
[INFO] [2022-08-26 15:12:52] Storing 10000 Occurrences (109999/110000/154578)
[INFO] [2022-08-26 15:12:56] Storing 10000 Occurrences (119999/120000/154578)
[INFO] [2022-08-26 15:12:59] Storing 10000 Occurrences (129999/130000/154578)
[INFO] [2022-08-26 15:13:04] Storing 10000 Occurrences (139999/140000/154578)
[INFO] [2022-08-26 15:13:09] Storing 10000 Occurrences (149999/150000/154578)
[INFO] [2022-08-26 15:13:13] Storing 4577 Occurrences (154576/154576/154578)
[INFO] [2022-08-26 15:13:15] Handling diff: /app/public/data/cites_taxa_tar_g/diff/cites_taxa_tar_g_measurements_28819.diff (154578 lines)
[INFO] [2022-08-26 15:13:15] Loading measurements diff file into memory (154578 lines)...
[INFO] [2022-08-26 15:13:20] Storing 9999 Traits (15141/10000/154578)
[INFO] [2022-08-26 15:13:25] Storing 5142 MetaTraits (15141/10000/154578)
[INFO] [2022-08-26 15:13:31] Storing 10000 Traits (30413/20000/154578)
[INFO] [2022-08-26 15:13:35] Storing 5272 MetaTraits (30413/20000/154578)
[INFO] [2022-08-26 15:13:41] Storing 10000 Traits (45519/30000/154578)
[INFO] [2022-08-26 15:13:47] Storing 5106 MetaTraits (45519/30000/154578)
[INFO] [2022-08-26 15:13:53] Storing 10000 Traits (60643/40000/154578)
[INFO] [2022-08-26 15:13:56] Storing 5124 MetaTraits (60643/40000/154578)
[INFO] [2022-08-26 15:14:02] Storing 10000 Traits (75804/50000/154578)
[INFO] [2022-08-26 15:14:06] Storing 5161 MetaTraits (75804/50000/154578)
[INFO] [2022-08-26 15:14:12] Storing 10000 Traits (91064/60000/154578)
[INFO] [2022-08-26 15:14:16] Storing 5260 MetaTraits (91064/60000/154578)
[INFO] [2022-08-26 15:14:22] Storing 10000 Traits (106237/70000/154578)
[INFO] [2022-08-26 15:14:26] Storing 5173 MetaTraits (106237/70000/154578)
[INFO] [2022-08-26 15:14:32] Storing 10000 Traits (121580/80000/154578)
[INFO] [2022-08-26 15:14:36] Storing 5343 MetaTraits (121580/80000/154578)
[INFO] [2022-08-26 15:14:42] Storing 10000 Traits (136818/90000/154578)
[INFO] [2022-08-26 15:14:46] Storing 5238 MetaTraits (136818/90000/154578)
[INFO] [2022-08-26 15:14:52] Storing 10000 Traits (151977/100000/154578)
[INFO] [2022-08-26 15:14:57] Storing 5159 MetaTraits (151977/100000/154578)
[INFO] [2022-08-26 15:15:02] Storing 10000 Traits (167146/110000/154578)
[INFO] [2022-08-26 15:15:07] Storing 5169 MetaTraits (167146/110000/154578)
[INFO] [2022-08-26 15:15:12] Storing 10000 Traits (182450/120000/154578)
[INFO] [2022-08-26 15:15:17] Storing 5304 MetaTraits (182450/120000/154578)
[INFO] [2022-08-26 15:15:23] Storing 10000 Traits (197894/130000/154578)
[INFO] [2022-08-26 15:15:27] Storing 5444 MetaTraits (197894/130000/154578)
[INFO] [2022-08-26 15:15:33] Storing 10000 Traits (213247/140000/154578)
[INFO] [2022-08-26 15:15:38] Storing 5353 MetaTraits (213247/140000/154578)
[INFO] [2022-08-26 15:15:44] Storing 10000 Traits (228475/150000/154578)
[INFO] [2022-08-26 15:15:48] Storing 5228 MetaTraits (228475/150000/154578)
[INFO] [2022-08-26 15:15:52] Storing 4577 Traits (235447/154576/154578)
[INFO] [2022-08-26 15:15:54] Storing 2395 MetaTraits (235447/154576/154578)
[STOP] [2022-08-26 15:15:54] parse_diff_and_store
[START] [2022-08-26 15:15:54] resolve_keys
[2022-08-26 15:16:45] Resolving downloaded urls (this is not actually downloading them yet)
[INFO] [2022-08-26 15:16:55] Occurrences to nodes (through scientific_names)...
[INFO] [2022-08-26 15:17:49] traits to occurrences...
[INFO] [2022-08-26 15:17:55] traits to nodes (through occurrences)...
[INFO] [2022-08-26 15:17:59] Traits to sex term...
[INFO] [2022-08-26 15:18:02] Traits to lifestage term...
[INFO] [2022-08-26 15:18:05] MetaTraits to traits...
[INFO] [2022-08-26 15:18:07] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2022-08-26 15:18:08] Assocs to occurrences...
[INFO] [2022-08-26 15:18:08] Assocs to nodes...
[INFO] [2022-08-26 15:18:08] Assoc to sex term...
[INFO] [2022-08-26 15:18:08] Assoc to lifestage term...
[INFO] [2022-08-26 15:18:08] MetaAssoc to assocs...
[STOP] [2022-08-26 15:18:08] resolve_keys
[START] [2022-08-26 15:18:08] hold_for_later_1
[STOP] [2022-08-26 15:18:08] hold_for_later_1
[START] [2022-08-26 15:18:08] hold_for_later_2
[STOP] [2022-08-26 15:18:08] hold_for_later_2
[START] [2022-08-26 15:18:08] resolve_missing_parents
[STOP] [2022-08-26 15:18:08] resolve_missing_parents
[START] [2022-08-26 15:18:08] rebuild_nodes
[START] [2022-08-26 15:18:08] Flattener#flatten
[START] [2022-08-26 15:18:08] Flattener#study_resource
[START] [2022-08-26 15:18:09] Flattener#build_ancestry
[STOP] [2022-08-26 15:18:16] Flattener#build_ancestry
[INFO] [2022-08-26 15:18:16] 44881 ancestry keys
[START] [2022-08-26 15:18:16] build_node_ancestors
[INFO] [2022-08-26 15:18:16] old ancestors deleted.
[STOP] [2022-08-26 15:18:31] build_node_ancestors
[START] [2022-08-26 15:18:35] Flattener#propagate_ancestor_ids
[STOP] [2022-08-26 15:18:39] Flattener#propagate_ancestor_ids
[STOP] [2022-08-26 15:18:39] Flattener#flatten
[STOP] [2022-08-26 15:18:39] rebuild_nodes
[START] [2022-08-26 15:18:39] resolve_missing_media_owners
[STOP] [2022-08-26 15:18:39] resolve_missing_media_owners
[START] [2022-08-26 15:18:39] sanitize_media_verbatims
[STOP] [2022-08-26 15:18:39] sanitize_media_verbatims
[START] [2022-08-26 15:18:39] queue_downloads
[STOP] [2022-08-26 15:18:39] queue_downloads
[START] [2022-08-26 15:18:39] parse_names
[WARN] [2022-08-26 15:18:39] I see 81226 names which still need to be parsed.
[WARN] [2022-08-26 15:18:40] Names to parse: 10000 formatted: 10000 learned: 9797 parsed: 10000
[WARN] [2022-08-26 15:18:48] Names to parse: 10000 formatted: 10000 learned: 9982 parsed: 10000
[WARN] [2022-08-26 15:18:56] Names to parse: 10000 formatted: 10000 learned: 9946 parsed: 10000
[WARN] [2022-08-26 15:19:05] Names to parse: 10000 formatted: 10000 learned: 9776 parsed: 10000
[WARN] [2022-08-26 15:19:13] Names to parse: 10000 formatted: 10000 learned: 9876 parsed: 10000
[WARN] [2022-08-26 15:19:22] Names to parse: 10000 formatted: 10000 learned: 9988 parsed: 10000
[WARN] [2022-08-26 15:19:30] Names to parse: 10000 formatted: 10000 learned: 9995 parsed: 10000
[WARN] [2022-08-26 15:19:38] Names to parse: 10000 formatted: 10000 learned: 9904 parsed: 10000
[WARN] [2022-08-26 15:19:45] Names to parse: 1226 formatted: 1226 learned: 1213 parsed: 1226
[STOP] [2022-08-26 15:19:47] parse_names
[START] [2022-08-26 15:19:47] denormalize_canonical_names_to_nodes
[STOP] [2022-08-26 15:19:47] denormalize_canonical_names_to_nodes
[START] [2022-08-26 15:19:47] match_nodes
[START] [2022-08-26 15:19:47] map_all_nodes_to_pages
[STOP] [2022-08-26 15:31:41] map_all_nodes_to_pages
[INFO] [2022-08-26 15:31:41] 3133 Unmatched nodes (of 44881)! That's too many to output. Full list in /app/public/data/cites_taxa_tar_g/unmatched_nodes.txt ; First 10: Canonical: Perodicticus; Node#118725583; ResourceID: 2653; Canonical: Pseudopotto martini; Node#118752932; ResourceID: 9369; Canonical: Semnopithecus hypoleucos; Node#118729560; ResourceID: 4349; Canonical: Trachypithecus villosus; Node#118734660; ResourceID: 6437; Canonical: Cebus flavius; Node#118729217; ResourceID: 4025; Canonical: Callithrix manicorensis; Node#118730080; ResourceID: 4882; Canonical: Lagothrix; Node#118720239; ResourceID: 2085; Canonical: Cheirogaleus medius; Node#118729813; ResourceID: 4609; Canonical: Cheirogaleus minusculus; Node#118752264; ResourceID: 8720; Canonical: Galago nyasae; Node#118734882; ResourceID: 6538
[START] [2022-08-26 15:31:41] update_nodes
[STOP] [2022-08-26 15:32:05] update_nodes
[STOP] [2022-08-26 15:32:05] match_nodes
[START] [2022-08-26 15:32:05] reindex_search
[STOP] [2022-08-26 15:32:48] reindex_search
[START] [2022-08-26 15:32:48] normalize_units
[STOP] [2022-08-26 15:33:42] normalize_units
[START] [2022-08-26 15:33:42] calculate_statistics
[INFO] [2022-08-26 15:34:30] Duplicate page_id count: 0
[STOP] [2022-08-26 15:34:31] calculate_statistics
[START] [2022-08-26 15:34:31] complete_harvest_instance
[START] [2022-08-26 15:34:31] overall_tsv_creation
[INFO] [2022-08-26 15:34:31] Processing group of 44881 in 5 batches of 10000
[INFO] [2022-08-26 15:35:48] 32185 Traits (unfiltered)...
[INFO] [2022-08-26 15:35:48] Building Traits map (this can take a while)...
[INFO] [2022-08-26 15:38:28] 10000 traits mapped (4998 meta)...
[INFO] [2022-08-26 15:40:21] 20000 traits mapped (9843 meta)...
[INFO] [2022-08-26 15:42:11] 30000 traits mapped (14807 meta)...
[INFO] [2022-08-26 15:42:43] Done. 32185 traits mapped (15915 meta).
[INFO] [2022-08-26 15:42:43] Building Associations map (this can take a while)...
[INFO] [2022-08-26 15:43:00] Done. 0 assocs mapped (0 meta).
[INFO] [2022-08-26 15:43:00] Adding 32185 traits...
[INFO] [2022-08-26 15:43:06] 8092 metadata added.
[INFO] [2022-08-26 15:43:06] Adding 0 assocs...
[INFO] [2022-08-26 15:43:06] 0 metadata added.
[INFO] [2022-08-26 15:45:58] 40536 Traits (unfiltered)...
[INFO] [2022-08-26 15:45:58] Building Traits map (this can take a while)...
[INFO] [2022-08-26 15:48:37] 10000 traits mapped (4221 meta)...
[INFO] [2022-08-26 15:50:29] 20000 traits mapped (8648 meta)...
[INFO] [2022-08-26 15:52:22] 30000 traits mapped (12879 meta)...
[INFO] [2022-08-26 15:54:13] 40000 traits mapped (17183 meta)...
[INFO] [2022-08-26 15:54:24] Done. 40536 traits mapped (17410 meta).
[INFO] [2022-08-26 15:54:24] Building Associations map (this can take a while)...
[INFO] [2022-08-26 15:54:34] Done. 0 assocs mapped (0 meta).
[INFO] [2022-08-26 15:54:34] Adding 40536 traits...
[INFO] [2022-08-26 15:54:40] 8841 metadata added.
[INFO] [2022-08-26 15:54:40] Adding 0 assocs...
[INFO] [2022-08-26 15:54:40] 0 metadata added.
[INFO] [2022-08-26 15:56:50] 33801 Traits (unfiltered)...
[INFO] [2022-08-26 15:56:50] Building Traits map (this can take a while)...
[INFO] [2022-08-26 15:59:28] 10000 traits mapped (5568 meta)...
[INFO] [2022-08-26 16:01:23] 20000 traits mapped (11243 meta)...
[INFO] [2022-08-26 16:03:14] 30000 traits mapped (16919 meta)...
[INFO] [2022-08-26 16:03:58] Done. 33801 traits mapped (19051 meta).
[INFO] [2022-08-26 16:03:58] Building Associations map (this can take a while)...
[INFO] [2022-08-26 16:04:08] Done. 0 assocs mapped (0 meta).
[INFO] [2022-08-26 16:04:08] Adding 33801 traits...
[INFO] [2022-08-26 16:04:11] 9629 metadata added.
[INFO] [2022-08-26 16:04:11] Adding 0 assocs...
[INFO] [2022-08-26 16:04:11] 0 metadata added.
[INFO] [2022-08-26 16:06:24] 28979 Traits (unfiltered)...
[INFO] [2022-08-26 16:06:24] Building Traits map (this can take a while)...
[INFO] [2022-08-26 16:09:02] 10000 traits mapped (6539 meta)...
[INFO] [2022-08-26 16:10:56] 20000 traits mapped (13401 meta)...
[INFO] [2022-08-26 16:12:36] Done. 28979 traits mapped (19669 meta).
[INFO] [2022-08-26 16:12:36] Building Associations map (this can take a while)...
[INFO] [2022-08-26 16:12:46] Done. 0 assocs mapped (0 meta).
[INFO] [2022-08-26 16:12:46] Adding 28979 traits...
[INFO] [2022-08-26 16:12:49] 9869 metadata added.
[INFO] [2022-08-26 16:12:49] Adding 0 assocs...
[INFO] [2022-08-26 16:12:49] 0 metadata added.
[INFO] [2022-08-26 16:14:44] 19075 Traits (unfiltered)...
[INFO] [2022-08-26 16:14:44] Building Traits map (this can take a while)...
[INFO] [2022-08-26 16:17:22] 10000 traits mapped (4699 meta)...
[INFO] [2022-08-26 16:19:14] Done. 19075 traits mapped (8826 meta).
[INFO] [2022-08-26 16:19:14] Building Associations map (this can take a while)...
[INFO] [2022-08-26 16:19:23] Done. 0 assocs mapped (0 meta).
[INFO] [2022-08-26 16:19:23] Adding 19075 traits...
[INFO] [2022-08-26 16:19:25] 4479 metadata added.
[INFO] [2022-08-26 16:19:25] Adding 0 assocs...
[INFO] [2022-08-26 16:19:25] 0 metadata added.
[INFO] [2022-08-26 16:20:13] Average Time: 506.93
[INFO] [2022-08-26 16:20:13] Total Time: 45m43s
[STOP] [2022-08-26 16:20:13] overall_tsv_creation
[INFO] [2022-08-26 16:20:13] Done. Check your files:
[INFO] [2022-08-26 16:20:14] (44881 lines) /app/public/data/cites_taxa_tar_g/publish_nodes.tsv
[INFO] [2022-08-26 16:20:14] (151817 lines) /app/public/data/cites_taxa_tar_g/publish_node_ancestors.tsv
[INFO] [2022-08-26 16:20:14] (81226 lines) /app/public/data/cites_taxa_tar_g/publish_scientific_names.tsv
[INFO] [2022-08-26 16:20:14] (30405 lines) /app/public/data/cites_taxa_tar_g/publish_vernaculars.tsv
[INFO] [2022-08-26 16:20:14] (154577 lines) /app/public/data/cites_taxa_tar_g/publish_traits.tsv
[INFO] [2022-08-26 16:20:14] (40911 lines) /app/public/data/cites_taxa_tar_g/publish_metadata.tsv
[STOP] [2022-08-26 16:20:14] complete_harvest_instance
[START] [2022-08-26 16:20:14] completed
[STOP] [2022-08-26 16:20:14] completed
[STOP] [2022-08-26 16:20:14] logged process, took 4200.89

Latest Process

26 Aug 15:10 EXITED currently processing 0