Harvest for USDA Plants data Created 28 May 14:44

Harvesting Log

[INFO] [2023-05-28 14:44:55] Created harvest instance #4355
[STOP] [2023-05-28 14:44:55] create_harvest_instance
[START] [2023-05-28 14:44:55] fetch_files
[STOP] [2023-05-28 14:44:55] fetch_files
[START] [2023-05-28 14:44:55] validate_each_file
[INFO] [2023-05-28 14:44:55] Looping over 6 formats...
[INFO] [2023-05-28 14:44:55] ...agents (/app/public/data/usda_plants/agent.tab)
[INFO] [2023-05-28 14:44:55] Valid: /app/public/data/usda_plants/converted_csv/usda_plants_agents_30377.csv (1 lines)
[INFO] [2023-05-28 14:44:55] ...refs (/app/public/data/usda_plants/reference.tab)
[INFO] [2023-05-28 14:44:55] Valid: /app/public/data/usda_plants/converted_csv/usda_plants_refs_30376.csv (2 lines)
[INFO] [2023-05-28 14:44:55] ...nodes (/app/public/data/usda_plants/taxon.tab)
[INFO] [2023-05-28 14:44:56] Valid: /app/public/data/usda_plants/converted_csv/usda_plants_nodes_30378.csv (35186 lines)
[INFO] [2023-05-28 14:44:56] ...media (/app/public/data/usda_plants/media_resource.tab)
[INFO] [2023-05-28 14:44:56] Valid: /app/public/data/usda_plants/converted_csv/usda_plants_media_30375.csv (5 lines)
[INFO] [2023-05-28 14:44:56] ...occurrences (/app/public/data/usda_plants/occurrence_specific.tab)
[INFO] [2023-05-28 14:45:11] Valid: /app/public/data/usda_plants/converted_csv/usda_plants_occurrences_30379.csv (634875 lines)
[INFO] [2023-05-28 14:45:11] ...measurements (/app/public/data/usda_plants/measurement_or_fact_specific.tab)
[INFO] [2023-05-28 14:45:56] Valid: /app/public/data/usda_plants/converted_csv/usda_plants_measurements_30380.csv (580161 lines)
[STOP] [2023-05-28 14:45:56] validate_each_file
[START] [2023-05-28 14:45:56] convert_to_csv
[INFO] [2023-05-28 14:45:56] Looping over 6 formats...
[INFO] [2023-05-28 14:45:56] ...agents (/app/public/data/usda_plants/agent.tab)
[CMD] [2023-05-28 14:45:56] /usr/bin/sort /app/public/data/usda_plants/converted_csv/usda_plants_agents_30377.csv > /app/public/data/usda_plants/converted_csv/usda_plants_agents_30377.csv_sorted
[INFO] [2023-05-28 14:45:56] Converted: /app/public/data/usda_plants/converted_csv/usda_plants_agents_30377.csv (1 lines)
[INFO] [2023-05-28 14:45:56] ...refs (/app/public/data/usda_plants/reference.tab)
[CMD] [2023-05-28 14:45:56] /usr/bin/sort /app/public/data/usda_plants/converted_csv/usda_plants_refs_30376.csv > /app/public/data/usda_plants/converted_csv/usda_plants_refs_30376.csv_sorted
[INFO] [2023-05-28 14:45:57] Converted: /app/public/data/usda_plants/converted_csv/usda_plants_refs_30376.csv (2 lines)
[INFO] [2023-05-28 14:45:57] ...nodes (/app/public/data/usda_plants/taxon.tab)
[CMD] [2023-05-28 14:45:57] /usr/bin/sort /app/public/data/usda_plants/converted_csv/usda_plants_nodes_30378.csv > /app/public/data/usda_plants/converted_csv/usda_plants_nodes_30378.csv_sorted
[INFO] [2023-05-28 14:45:57] Converted: /app/public/data/usda_plants/converted_csv/usda_plants_nodes_30378.csv (35186 lines)
[INFO] [2023-05-28 14:45:57] ...media (/app/public/data/usda_plants/media_resource.tab)
[CMD] [2023-05-28 14:45:57] /usr/bin/sort /app/public/data/usda_plants/converted_csv/usda_plants_media_30375.csv > /app/public/data/usda_plants/converted_csv/usda_plants_media_30375.csv_sorted
[INFO] [2023-05-28 14:45:57] Converted: /app/public/data/usda_plants/converted_csv/usda_plants_media_30375.csv (5 lines)
[INFO] [2023-05-28 14:45:57] ...occurrences (/app/public/data/usda_plants/occurrence_specific.tab)
[CMD] [2023-05-28 14:45:57] /usr/bin/sort /app/public/data/usda_plants/converted_csv/usda_plants_occurrences_30379.csv > /app/public/data/usda_plants/converted_csv/usda_plants_occurrences_30379.csv_sorted
[INFO] [2023-05-28 14:45:58] Converted: /app/public/data/usda_plants/converted_csv/usda_plants_occurrences_30379.csv (634875 lines)
[INFO] [2023-05-28 14:45:58] ...measurements (/app/public/data/usda_plants/measurement_or_fact_specific.tab)
[CMD] [2023-05-28 14:45:58] /usr/bin/sort /app/public/data/usda_plants/converted_csv/usda_plants_measurements_30380.csv > /app/public/data/usda_plants/converted_csv/usda_plants_measurements_30380.csv_sorted
[INFO] [2023-05-28 14:46:04] Converted: /app/public/data/usda_plants/converted_csv/usda_plants_measurements_30380.csv (580161 lines)
[STOP] [2023-05-28 14:46:04] convert_to_csv
[START] [2023-05-28 14:46:04] calculate_delta
[INFO] [2023-05-28 14:46:04] Looping over 6 formats...
[INFO] [2023-05-28 14:46:04] ...agents (/app/public/data/usda_plants/agent.tab)
[CMD] [2023-05-28 14:46:04] echo "0a" > /app/public/data/usda_plants/diff/usda_plants_agents_30377.diff
[CMD] [2023-05-28 14:46:04] tail -n +1 /app/public/data/usda_plants/converted_csv/usda_plants_agents_30377.csv >> /app/public/data/usda_plants/diff/usda_plants_agents_30377.diff
[CMD] [2023-05-28 14:46:04] echo "." >> /app/public/data/usda_plants/diff/usda_plants_agents_30377.diff
[INFO] [2023-05-28 14:46:04] Created diff: /app/public/data/usda_plants/diff/usda_plants_agents_30377.diff (3 lines)
[INFO] [2023-05-28 14:46:04] ...refs (/app/public/data/usda_plants/reference.tab)
[CMD] [2023-05-28 14:46:04] echo "0a" > /app/public/data/usda_plants/diff/usda_plants_refs_30376.diff
[CMD] [2023-05-28 14:46:04] tail -n +1 /app/public/data/usda_plants/converted_csv/usda_plants_refs_30376.csv >> /app/public/data/usda_plants/diff/usda_plants_refs_30376.diff
[CMD] [2023-05-28 14:46:04] echo "." >> /app/public/data/usda_plants/diff/usda_plants_refs_30376.diff
[INFO] [2023-05-28 14:46:04] Created diff: /app/public/data/usda_plants/diff/usda_plants_refs_30376.diff (4 lines)
[INFO] [2023-05-28 14:46:04] ...nodes (/app/public/data/usda_plants/taxon.tab)
[CMD] [2023-05-28 14:46:04] echo "0a" > /app/public/data/usda_plants/diff/usda_plants_nodes_30378.diff
[CMD] [2023-05-28 14:46:04] tail -n +1 /app/public/data/usda_plants/converted_csv/usda_plants_nodes_30378.csv >> /app/public/data/usda_plants/diff/usda_plants_nodes_30378.diff
[CMD] [2023-05-28 14:46:04] echo "." >> /app/public/data/usda_plants/diff/usda_plants_nodes_30378.diff
[INFO] [2023-05-28 14:46:04] Created diff: /app/public/data/usda_plants/diff/usda_plants_nodes_30378.diff (35188 lines)
[INFO] [2023-05-28 14:46:04] ...media (/app/public/data/usda_plants/media_resource.tab)
[CMD] [2023-05-28 14:46:04] echo "0a" > /app/public/data/usda_plants/diff/usda_plants_media_30375.diff
[CMD] [2023-05-28 14:46:04] tail -n +1 /app/public/data/usda_plants/converted_csv/usda_plants_media_30375.csv >> /app/public/data/usda_plants/diff/usda_plants_media_30375.diff
[CMD] [2023-05-28 14:46:04] echo "." >> /app/public/data/usda_plants/diff/usda_plants_media_30375.diff
[INFO] [2023-05-28 14:46:04] Created diff: /app/public/data/usda_plants/diff/usda_plants_media_30375.diff (7 lines)
[INFO] [2023-05-28 14:46:04] ...occurrences (/app/public/data/usda_plants/occurrence_specific.tab)
[CMD] [2023-05-28 14:46:04] echo "0a" > /app/public/data/usda_plants/diff/usda_plants_occurrences_30379.diff
[CMD] [2023-05-28 14:46:04] tail -n +1 /app/public/data/usda_plants/converted_csv/usda_plants_occurrences_30379.csv >> /app/public/data/usda_plants/diff/usda_plants_occurrences_30379.diff
[CMD] [2023-05-28 14:46:05] echo "." >> /app/public/data/usda_plants/diff/usda_plants_occurrences_30379.diff
[INFO] [2023-05-28 14:46:05] Created diff: /app/public/data/usda_plants/diff/usda_plants_occurrences_30379.diff (634877 lines)
[INFO] [2023-05-28 14:46:05] ...measurements (/app/public/data/usda_plants/measurement_or_fact_specific.tab)
[CMD] [2023-05-28 14:46:05] echo "0a" > /app/public/data/usda_plants/diff/usda_plants_measurements_30380.diff
[CMD] [2023-05-28 14:46:05] tail -n +1 /app/public/data/usda_plants/converted_csv/usda_plants_measurements_30380.csv >> /app/public/data/usda_plants/diff/usda_plants_measurements_30380.diff
[CMD] [2023-05-28 14:46:08] echo "." >> /app/public/data/usda_plants/diff/usda_plants_measurements_30380.diff
[INFO] [2023-05-28 14:46:09] Created diff: /app/public/data/usda_plants/diff/usda_plants_measurements_30380.diff (580163 lines)
[STOP] [2023-05-28 14:46:09] calculate_delta
[START] [2023-05-28 14:46:09] parse_diff_and_store
[INFO] [2023-05-28 14:46:09] Handling diff: /app/public/data/usda_plants/diff/usda_plants_agents_30377.diff (3 lines)
[INFO] [2023-05-28 14:46:09] Loading agents diff file into memory (3 lines)...
[INFO] [2023-05-28 14:46:09] Storing 1 Attributions (1/1/3)
[INFO] [2023-05-28 14:46:09] Handling diff: /app/public/data/usda_plants/diff/usda_plants_refs_30376.diff (4 lines)
[INFO] [2023-05-28 14:46:09] Loading refs diff file into memory (4 lines)...
[INFO] [2023-05-28 14:46:09] Storing 2 References (2/2/4)
[INFO] [2023-05-28 14:46:09] Handling diff: /app/public/data/usda_plants/diff/usda_plants_nodes_30378.diff (35188 lines)
[INFO] [2023-05-28 14:46:09] Loading nodes diff file into memory (35188 lines)...
[INFO] [2023-05-28 14:46:13] Storing 10200 ScientificNames (30399/10000/35188)
[INFO] [2023-05-28 14:46:16] Storing 10200 Nodes (30399/10000/35188)
[INFO] [2023-05-28 14:46:19] Storing 9999 Identifiers (30399/10000/35188)
[WARN] [2023-05-28 14:46:23] SKIPPED 130 Scientific names (60811/20000/35188) with resource_pks already be in the database!
[WARN] [2023-05-28 14:46:23] SKIPPED 130 Nodes (60811/20000/35188) with resource_pks already be in the database!
[INFO] [2023-05-28 14:46:23] Storing 10076 ScientificNames (60811/20000/35188)
[INFO] [2023-05-28 14:46:26] Storing 10076 Nodes (60811/20000/35188)
[INFO] [2023-05-28 14:46:29] Storing 10000 Identifiers (60811/20000/35188)
[WARN] [2023-05-28 14:46:34] SKIPPED 149 Scientific names (91207/30000/35188) with resource_pks already be in the database!
[WARN] [2023-05-28 14:46:34] SKIPPED 149 Nodes (91207/30000/35188) with resource_pks already be in the database!
[INFO] [2023-05-28 14:46:34] Storing 10049 ScientificNames (91207/30000/35188)
[INFO] [2023-05-28 14:46:37] Storing 10049 Nodes (91207/30000/35188)
[INFO] [2023-05-28 14:46:39] Storing 10000 Identifiers (91207/30000/35188)
[WARN] [2023-05-28 14:46:42] SKIPPED 128 Scientific names (107074/35186/35188) with resource_pks already be in the database!
[WARN] [2023-05-28 14:46:42] SKIPPED 128 Nodes (107074/35186/35188) with resource_pks already be in the database!
[INFO] [2023-05-28 14:46:42] Storing 5212 ScientificNames (107074/35186/35188)
[INFO] [2023-05-28 14:46:44] Storing 5212 Nodes (107074/35186/35188)
[INFO] [2023-05-28 14:46:45] Storing 5187 Identifiers (107074/35186/35188)
[INFO] [2023-05-28 14:46:45] Handling diff: /app/public/data/usda_plants/diff/usda_plants_media_30375.diff (7 lines)
[INFO] [2023-05-28 14:46:46] Loading media diff file into memory (7 lines)...
[INFO] [2023-05-28 14:46:46] Storing 2 BibliographicCitations (10/5/7)
[INFO] [2023-05-28 14:46:46] Storing 2 ArticlesSections (10/5/7)
[INFO] [2023-05-28 14:46:46] Storing 2 Articles (10/5/7)
[INFO] [2023-05-28 14:46:46] Storing 1 ContentAttributions (10/5/7)
[INFO] [2023-05-28 14:46:46] Storing 3 Media (10/5/7)
[INFO] [2023-05-28 14:46:46] Handling diff: /app/public/data/usda_plants/diff/usda_plants_occurrences_30379.diff (634877 lines)
[INFO] [2023-05-28 14:46:46] Loading occurrences diff file into memory (634877 lines)...
[INFO] [2023-05-28 14:46:47] Storing 9999 Occurrences (9999/10000/634877)
[INFO] [2023-05-28 14:46:49] Storing 10000 Occurrences (19999/20000/634877)
[INFO] [2023-05-28 14:46:52] Storing 10000 Occurrences (29999/30000/634877)
[INFO] [2023-05-28 14:46:54] Storing 10000 Occurrences (39999/40000/634877)
[INFO] [2023-05-28 14:46:56] Storing 10000 Occurrences (49999/50000/634877)
[INFO] [2023-05-28 14:46:59] Storing 10000 Occurrences (59999/60000/634877)
[INFO] [2023-05-28 14:47:01] Storing 10000 Occurrences (69999/70000/634877)
[INFO] [2023-05-28 14:47:03] Storing 10000 Occurrences (79999/80000/634877)
[INFO] [2023-05-28 14:47:05] Storing 10000 Occurrences (89999/90000/634877)
[INFO] [2023-05-28 14:47:08] Storing 10000 Occurrences (99999/100000/634877)
[INFO] [2023-05-28 14:47:10] Storing 10000 Occurrences (109999/110000/634877)
[INFO] [2023-05-28 14:47:13] Storing 10000 Occurrences (119999/120000/634877)
[INFO] [2023-05-28 14:47:16] Storing 10000 Occurrences (129999/130000/634877)
[INFO] [2023-05-28 14:47:18] Storing 10000 Occurrences (139999/140000/634877)
[INFO] [2023-05-28 14:47:21] Storing 10000 Occurrences (149999/150000/634877)
[INFO] [2023-05-28 14:47:23] Storing 10000 Occurrences (159999/160000/634877)
[INFO] [2023-05-28 14:47:26] Storing 10000 Occurrences (169999/170000/634877)
[INFO] [2023-05-28 14:47:29] Storing 10000 Occurrences (179999/180000/634877)
[INFO] [2023-05-28 14:47:32] Storing 10000 Occurrences (189999/190000/634877)
[INFO] [2023-05-28 14:47:34] Storing 10000 Occurrences (199999/200000/634877)
[INFO] [2023-05-28 14:47:37] Storing 10000 Occurrences (209999/210000/634877)
[INFO] [2023-05-28 14:47:40] Storing 10000 Occurrences (219999/220000/634877)
[INFO] [2023-05-28 14:47:43] Storing 10000 Occurrences (229999/230000/634877)
[INFO] [2023-05-28 14:47:46] Storing 10000 Occurrences (239999/240000/634877)
[INFO] [2023-05-28 14:47:49] Storing 10000 Occurrences (249999/250000/634877)
[INFO] [2023-05-28 14:47:52] Storing 10000 Occurrences (259999/260000/634877)
[INFO] [2023-05-28 14:47:55] Storing 10000 Occurrences (269999/270000/634877)
[INFO] [2023-05-28 14:47:58] Storing 10000 Occurrences (279999/280000/634877)
[INFO] [2023-05-28 14:48:01] Storing 10000 Occurrences (289999/290000/634877)
[INFO] [2023-05-28 14:48:04] Storing 10000 Occurrences (299999/300000/634877)
[INFO] [2023-05-28 14:48:10] Storing 10000 Occurrences (316694/310000/634877)
[INFO] [2023-05-28 14:48:11] Storing 6695 OccurrenceMetadata (316694/310000/634877)
[WARN] [2023-05-28 14:48:14] SKIPPED 1083 Occurrence metadata (327777/320000/634877) with resource_pks already be in the database!
[INFO] [2023-05-28 14:48:14] Storing 10000 Occurrences (327777/320000/634877)
[INFO] [2023-05-28 14:48:15] Storing 0 OccurrenceMetadata (327777/320000/634877)
[WARN] [2023-05-28 14:48:15] No models to import, skipping!
[INFO] [2023-05-28 14:48:17] Storing 10000 Occurrences (337777/330000/634877)
[INFO] [2023-05-28 14:48:20] Storing 10000 Occurrences (347777/340000/634877)
[INFO] [2023-05-28 14:48:23] Storing 10000 Occurrences (357777/350000/634877)
[INFO] [2023-05-28 14:48:27] Storing 10000 Occurrences (367777/360000/634877)
[INFO] [2023-05-28 14:48:31] Storing 10000 Occurrences (377777/370000/634877)
[INFO] [2023-05-28 14:48:34] Storing 10000 Occurrences (387777/380000/634877)
[INFO] [2023-05-28 14:48:37] Storing 10000 Occurrences (397777/390000/634877)
[INFO] [2023-05-28 14:48:41] Storing 10000 Occurrences (407777/400000/634877)
[WARN] [2023-05-28 14:48:44] SKIPPED 1814 Occurrence metadata (419591/410000/634877) with resource_pks already be in the database!
[INFO] [2023-05-28 14:48:44] Storing 10000 Occurrences (419591/410000/634877)
[INFO] [2023-05-28 14:48:45] Storing 0 OccurrenceMetadata (419591/410000/634877)
[WARN] [2023-05-28 14:48:45] No models to import, skipping!
[WARN] [2023-05-28 14:48:48] SKIPPED 61 Occurrence metadata (429652/420000/634877) with resource_pks already be in the database!
[INFO] [2023-05-28 14:48:48] Storing 10000 Occurrences (429652/420000/634877)
[INFO] [2023-05-28 14:48:49] Storing 0 OccurrenceMetadata (429652/420000/634877)
[WARN] [2023-05-28 14:48:49] No models to import, skipping!
[INFO] [2023-05-28 14:48:52] Storing 10000 Occurrences (439652/430000/634877)
[INFO] [2023-05-28 14:48:55] Storing 10000 Occurrences (449652/440000/634877)
[INFO] [2023-05-28 14:48:58] Storing 10000 Occurrences (459652/450000/634877)
[INFO] [2023-05-28 14:49:03] Storing 10000 Occurrences (469652/460000/634877)
[INFO] [2023-05-28 14:49:06] Storing 10000 Occurrences (479652/470000/634877)
[INFO] [2023-05-28 14:49:10] Storing 10000 Occurrences (489652/480000/634877)
[INFO] [2023-05-28 14:49:14] Storing 10000 Occurrences (499652/490000/634877)
[INFO] [2023-05-28 14:49:18] Storing 10000 Occurrences (509652/500000/634877)
[INFO] [2023-05-28 14:49:21] Storing 10000 Occurrences (519652/510000/634877)
[INFO] [2023-05-28 14:49:25] Storing 10000 Occurrences (529652/520000/634877)
[INFO] [2023-05-28 14:49:29] Storing 10000 Occurrences (539652/530000/634877)
[INFO] [2023-05-28 14:49:33] Storing 10000 Occurrences (549652/540000/634877)
[INFO] [2023-05-28 14:49:37] Storing 10000 Occurrences (559652/550000/634877)
[INFO] [2023-05-28 14:49:42] Storing 10000 Occurrences (569652/560000/634877)
[INFO] [2023-05-28 14:49:46] Storing 10000 Occurrences (579652/570000/634877)
[INFO] [2023-05-28 14:49:50] Storing 10000 Occurrences (589652/580000/634877)
[INFO] [2023-05-28 14:49:55] Storing 10000 Occurrences (599652/590000/634877)
[INFO] [2023-05-28 14:49:59] Storing 10000 Occurrences (609652/600000/634877)
[INFO] [2023-05-28 14:50:03] Storing 10000 Occurrences (619652/610000/634877)
[INFO] [2023-05-28 14:50:07] Storing 10000 Occurrences (629652/620000/634877)
[INFO] [2023-05-28 14:50:12] Storing 10000 Occurrences (639652/630000/634877)
[INFO] [2023-05-28 14:50:16] Storing 4876 Occurrences (644528/634875/634877)
[INFO] [2023-05-28 14:50:17] Handling diff: /app/public/data/usda_plants/diff/usda_plants_measurements_30380.diff (580163 lines)
[INFO] [2023-05-28 14:50:17] Loading measurements diff file into memory (580163 lines)...
[INFO] [2023-05-28 14:50:22] Storing 9999 Traits (19998/10000/580163)
[INFO] [2023-05-28 14:50:25] Storing 9999 MetaTraits (19998/10000/580163)
[INFO] [2023-05-28 14:50:31] Storing 10000 Traits (39998/20000/580163)
[INFO] [2023-05-28 14:50:34] Storing 10000 MetaTraits (39998/20000/580163)
[INFO] [2023-05-28 14:50:39] Storing 10000 Traits (59998/30000/580163)
[INFO] [2023-05-28 14:50:42] Storing 10000 MetaTraits (59998/30000/580163)
[INFO] [2023-05-28 14:50:48] Storing 10000 Traits (79998/40000/580163)
[INFO] [2023-05-28 14:50:51] Storing 10000 MetaTraits (79998/40000/580163)
[INFO] [2023-05-28 14:50:57] Storing 10000 Traits (99998/50000/580163)
[INFO] [2023-05-28 14:51:00] Storing 10000 MetaTraits (99998/50000/580163)
[INFO] [2023-05-28 14:51:06] Storing 10000 Traits (119998/60000/580163)
[INFO] [2023-05-28 14:51:09] Storing 10000 MetaTraits (119998/60000/580163)
[INFO] [2023-05-28 14:51:15] Storing 10000 Traits (139998/70000/580163)
[INFO] [2023-05-28 14:51:18] Storing 10000 MetaTraits (139998/70000/580163)
[INFO] [2023-05-28 14:51:24] Storing 10000 Traits (159998/80000/580163)
[INFO] [2023-05-28 14:51:27] Storing 10000 MetaTraits (159998/80000/580163)
[INFO] [2023-05-28 14:51:33] Storing 10000 Traits (179998/90000/580163)
[INFO] [2023-05-28 14:51:37] Storing 10000 MetaTraits (179998/90000/580163)
[INFO] [2023-05-28 14:51:43] Storing 10000 Traits (199998/100000/580163)
[INFO] [2023-05-28 14:51:46] Storing 10000 MetaTraits (199998/100000/580163)
[INFO] [2023-05-28 14:51:52] Storing 10000 Traits (219998/110000/580163)
[INFO] [2023-05-28 14:51:55] Storing 10000 MetaTraits (219998/110000/580163)
[INFO] [2023-05-28 14:52:01] Storing 10000 Traits (239998/120000/580163)
[INFO] [2023-05-28 14:52:05] Storing 10000 MetaTraits (239998/120000/580163)
[INFO] [2023-05-28 14:52:11] Storing 10000 Traits (259998/130000/580163)
[INFO] [2023-05-28 14:52:14] Storing 10000 MetaTraits (259998/130000/580163)
[INFO] [2023-05-28 14:52:20] Storing 10000 Traits (279998/140000/580163)
[INFO] [2023-05-28 14:52:23] Storing 10000 MetaTraits (279998/140000/580163)
[INFO] [2023-05-28 14:52:29] Storing 10000 Traits (299998/150000/580163)
[INFO] [2023-05-28 14:52:33] Storing 10000 MetaTraits (299998/150000/580163)
[INFO] [2023-05-28 14:52:39] Storing 10000 Traits (319998/160000/580163)
[INFO] [2023-05-28 14:52:42] Storing 10000 MetaTraits (319998/160000/580163)
[INFO] [2023-05-28 14:52:48] Storing 10000 Traits (339998/170000/580163)
[INFO] [2023-05-28 14:52:52] Storing 10000 MetaTraits (339998/170000/580163)
[INFO] [2023-05-28 14:52:58] Storing 10000 Traits (359998/180000/580163)
[INFO] [2023-05-28 14:53:01] Storing 10000 MetaTraits (359998/180000/580163)
[INFO] [2023-05-28 14:53:08] Storing 10000 Traits (379998/190000/580163)
[INFO] [2023-05-28 14:53:11] Storing 10000 MetaTraits (379998/190000/580163)
[INFO] [2023-05-28 14:53:17] Storing 10000 Traits (399998/200000/580163)
[INFO] [2023-05-28 14:53:20] Storing 10000 MetaTraits (399998/200000/580163)
[INFO] [2023-05-28 14:53:27] Storing 10000 Traits (419998/210000/580163)
[INFO] [2023-05-28 14:53:30] Storing 10000 MetaTraits (419998/210000/580163)
[INFO] [2023-05-28 14:53:36] Storing 10000 Traits (439998/220000/580163)
[INFO] [2023-05-28 14:53:40] Storing 10000 MetaTraits (439998/220000/580163)
[INFO] [2023-05-28 14:53:47] Storing 10000 Traits (463906/230000/580163)
[INFO] [2023-05-28 14:53:50] Storing 13908 MetaTraits (463906/230000/580163)
[INFO] [2023-05-28 14:53:58] Storing 10000 Traits (493246/240000/580163)
[INFO] [2023-05-28 14:54:01] Storing 19340 MetaTraits (493246/240000/580163)
[INFO] [2023-05-28 14:54:11] Storing 10000 Traits (523246/250000/580163)
[INFO] [2023-05-28 14:54:14] Storing 20000 MetaTraits (523246/250000/580163)
[INFO] [2023-05-28 14:54:24] Storing 10000 Traits (553246/260000/580163)
[INFO] [2023-05-28 14:54:27] Storing 20000 MetaTraits (553246/260000/580163)
[INFO] [2023-05-28 14:54:37] Storing 10000 Traits (585677/270000/580163)
[INFO] [2023-05-28 14:54:40] Storing 22431 MetaTraits (585677/270000/580163)
[INFO] [2023-05-28 14:54:50] Storing 10000 Traits (616110/280000/580163)
[INFO] [2023-05-28 14:54:53] Storing 20433 MetaTraits (616110/280000/580163)
[INFO] [2023-05-28 14:55:02] Storing 10000 Traits (646110/290000/580163)
[INFO] [2023-05-28 14:55:06] Storing 20000 MetaTraits (646110/290000/580163)
[INFO] [2023-05-28 14:55:16] Storing 10000 Traits (676171/300000/580163)
[INFO] [2023-05-28 14:55:19] Storing 20061 MetaTraits (676171/300000/580163)
[INFO] [2023-05-28 14:55:28] Storing 10000 Traits (708120/310000/580163)
[INFO] [2023-05-28 14:55:31] Storing 21949 MetaTraits (708120/310000/580163)
[INFO] [2023-05-28 14:55:41] Storing 10000 Traits (738120/320000/580163)
[INFO] [2023-05-28 14:55:44] Storing 20000 MetaTraits (738120/320000/580163)
[INFO] [2023-05-28 14:55:53] Storing 10000 Traits (768120/330000/580163)
[INFO] [2023-05-28 14:55:56] Storing 20000 MetaTraits (768120/330000/580163)
[INFO] [2023-05-28 14:56:06] Storing 10000 Traits (798120/340000/580163)
[INFO] [2023-05-28 14:56:09] Storing 20000 MetaTraits (798120/340000/580163)
[INFO] [2023-05-28 14:56:18] Storing 10000 Traits (828120/350000/580163)
[INFO] [2023-05-28 14:56:21] Storing 20000 MetaTraits (828120/350000/580163)
[INFO] [2023-05-28 14:56:31] Storing 10000 Traits (858120/360000/580163)
[INFO] [2023-05-28 14:56:34] Storing 20000 MetaTraits (858120/360000/580163)
[INFO] [2023-05-28 14:56:44] Storing 10000 Traits (888120/370000/580163)
[INFO] [2023-05-28 14:56:47] Storing 20000 MetaTraits (888120/370000/580163)
[INFO] [2023-05-28 14:56:57] Storing 10000 Traits (918120/380000/580163)
[INFO] [2023-05-28 14:57:00] Storing 20000 MetaTraits (918120/380000/580163)
[INFO] [2023-05-28 14:57:10] Storing 10000 Traits (948120/390000/580163)
[INFO] [2023-05-28 14:57:13] Storing 20000 MetaTraits (948120/390000/580163)
[INFO] [2023-05-28 14:57:23] Storing 10000 Traits (978120/400000/580163)
[INFO] [2023-05-28 14:57:26] Storing 20000 MetaTraits (978120/400000/580163)
[INFO] [2023-05-28 14:57:36] Storing 10000 Traits (1008120/410000/580163)
[INFO] [2023-05-28 14:57:40] Storing 20000 MetaTraits (1008120/410000/580163)
[INFO] [2023-05-28 14:57:49] Storing 10000 Traits (1038120/420000/580163)
[INFO] [2023-05-28 14:57:53] Storing 20000 MetaTraits (1038120/420000/580163)
[INFO] [2023-05-28 14:58:03] Storing 10000 Traits (1068120/430000/580163)
[INFO] [2023-05-28 14:58:06] Storing 20000 MetaTraits (1068120/430000/580163)
[INFO] [2023-05-28 14:58:16] Storing 10000 Traits (1098120/440000/580163)
[INFO] [2023-05-28 14:58:19] Storing 20000 MetaTraits (1098120/440000/580163)
[INFO] [2023-05-28 14:58:28] Storing 10000 Traits (1122302/450000/580163)
[INFO] [2023-05-28 14:58:32] Storing 14182 MetaTraits (1122302/450000/580163)
[INFO] [2023-05-28 14:58:40] Storing 10000 Traits (1142302/460000/580163)
[INFO] [2023-05-28 14:58:43] Storing 10000 MetaTraits (1142302/460000/580163)
[INFO] [2023-05-28 14:58:50] Storing 10000 Traits (1162302/470000/580163)
[INFO] [2023-05-28 14:58:54] Storing 10000 MetaTraits (1162302/470000/580163)
[INFO] [2023-05-28 14:59:01] Storing 10000 Traits (1182302/480000/580163)
[INFO] [2023-05-28 14:59:04] Storing 10000 MetaTraits (1182302/480000/580163)
[INFO] [2023-05-28 14:59:11] Storing 10000 Traits (1202302/490000/580163)
[INFO] [2023-05-28 14:59:15] Storing 10000 MetaTraits (1202302/490000/580163)
[INFO] [2023-05-28 14:59:22] Storing 10000 Traits (1222302/500000/580163)
[INFO] [2023-05-28 14:59:25] Storing 10000 MetaTraits (1222302/500000/580163)
[INFO] [2023-05-28 14:59:32] Storing 10000 Traits (1242302/510000/580163)
[INFO] [2023-05-28 14:59:36] Storing 10000 MetaTraits (1242302/510000/580163)
[INFO] [2023-05-28 14:59:43] Storing 10000 Traits (1262302/520000/580163)
[INFO] [2023-05-28 14:59:46] Storing 10000 MetaTraits (1262302/520000/580163)
[INFO] [2023-05-28 14:59:53] Storing 10000 Traits (1282302/530000/580163)
[INFO] [2023-05-28 14:59:57] Storing 10000 MetaTraits (1282302/530000/580163)
[INFO] [2023-05-28 15:00:04] Storing 10000 Traits (1302302/540000/580163)
[INFO] [2023-05-28 15:00:08] Storing 10000 MetaTraits (1302302/540000/580163)
[INFO] [2023-05-28 15:00:15] Storing 10000 Traits (1322302/550000/580163)
[INFO] [2023-05-28 15:00:18] Storing 10000 MetaTraits (1322302/550000/580163)
[INFO] [2023-05-28 15:00:26] Storing 10000 Traits (1342302/560000/580163)
[INFO] [2023-05-28 15:00:29] Storing 10000 MetaTraits (1342302/560000/580163)
[INFO] [2023-05-28 15:00:36] Storing 10000 Traits (1362302/570000/580163)
[INFO] [2023-05-28 15:00:40] Storing 10000 MetaTraits (1362302/570000/580163)
[INFO] [2023-05-28 15:00:47] Storing 10000 Traits (1382302/580000/580163)
[INFO] [2023-05-28 15:00:51] Storing 10000 MetaTraits (1382302/580000/580163)
[INFO] [2023-05-28 15:00:54] Storing 162 Traits (1382626/580161/580163)
[INFO] [2023-05-28 15:00:54] Storing 162 MetaTraits (1382626/580161/580163)
[STOP] [2023-05-28 15:00:54] parse_diff_and_store
[START] [2023-05-28 15:00:54] resolve_keys
[2023-05-28 15:01:09] Resolving downloaded urls (this is not actually downloading them yet)
[INFO] [2023-05-28 15:01:43] Occurrences to nodes (through scientific_names)...
[INFO] [2023-05-28 15:02:00] traits to occurrences...
[INFO] [2023-05-28 15:02:21] traits to nodes (through occurrences)...
[INFO] [2023-05-28 15:02:38] Traits to sex term...
[INFO] [2023-05-28 15:02:48] Traits to lifestage term...
[INFO] [2023-05-28 15:02:58] MetaTraits to traits...
[INFO] [2023-05-28 15:03:18] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2023-05-28 15:03:18] Assocs to occurrences...
[INFO] [2023-05-28 15:03:18] Assocs to nodes...
[INFO] [2023-05-28 15:03:18] Assoc to sex term...
[INFO] [2023-05-28 15:03:18] Assoc to lifestage term...
[INFO] [2023-05-28 15:03:18] MetaAssoc to assocs...
[STOP] [2023-05-28 15:03:18] resolve_keys
[START] [2023-05-28 15:03:18] hold_for_later_1
[STOP] [2023-05-28 15:03:18] hold_for_later_1
[START] [2023-05-28 15:03:18] hold_for_later_2
[STOP] [2023-05-28 15:03:18] hold_for_later_2
[START] [2023-05-28 15:03:18] resolve_missing_parents
[STOP] [2023-05-28 15:03:21] resolve_missing_parents
[START] [2023-05-28 15:03:21] rebuild_nodes
[START] [2023-05-28 15:03:21] Flattener#flatten
[START] [2023-05-28 15:03:21] Flattener#study_resource
[START] [2023-05-28 15:03:21] Flattener#build_ancestry
[STOP] [2023-05-28 15:03:25] Flattener#build_ancestry
[INFO] [2023-05-28 15:03:25] 35537 ancestry keys
[START] [2023-05-28 15:03:25] build_node_ancestors
[INFO] [2023-05-28 15:03:25] old ancestors deleted.
[STOP] [2023-05-28 15:03:26] build_node_ancestors
[START] [2023-05-28 15:03:28] Flattener#propagate_ancestor_ids
[STOP] [2023-05-28 15:03:29] Flattener#propagate_ancestor_ids
[STOP] [2023-05-28 15:03:29] Flattener#flatten
[STOP] [2023-05-28 15:03:29] rebuild_nodes
[START] [2023-05-28 15:03:29] resolve_missing_media_owners
[STOP] [2023-05-28 15:03:29] resolve_missing_media_owners
[START] [2023-05-28 15:03:29] sanitize_media_verbatims
[STOP] [2023-05-28 15:03:29] sanitize_media_verbatims
[START] [2023-05-28 15:03:29] queue_downloads
[STOP] [2023-05-28 15:03:29] queue_downloads
[START] [2023-05-28 15:03:29] parse_names
[WARN] [2023-05-28 15:03:29] I see 35537 names which still need to be parsed.
[INFO] [2023-05-28 15:03:30] 0% of media downloaded
[ERR] [2023-05-28 15:03:30][hdls] NO additional images were found to download
[INFO] [2023-05-28 15:03:30] 100% of media downloaded
[ERR] [2023-05-28 15:03:30][hdls] NO additional images were found to download
[INFO] [2023-05-28 15:03:30] 100% of media downloaded
[ERR] [2023-05-28 15:03:30][hdls] NO additional images were found to download
[INFO] [2023-05-28 15:03:30] 100% of media downloaded
[ERR] [2023-05-28 15:03:30][hdls] NO additional images were found to download
[INFO] [2023-05-28 15:03:30] 100% of media downloaded
[ERR] [2023-05-28 15:03:30][hdls] NO additional images were found to download
[INFO] [2023-05-28 15:03:30] 100% of media downloaded
[ERR] [2023-05-28 15:03:30][hdls] NO additional images were found to download
[INFO] [2023-05-28 15:03:30] 100% of media downloaded
[ERR] [2023-05-28 15:03:30][hdls] NO additional images were found to download
[INFO] [2023-05-28 15:03:30] 100% of media downloaded
[ERR] [2023-05-28 15:03:30][hdls] NO additional images were found to download
[INFO] [2023-05-28 15:03:30] 100% of media downloaded
[ERR] [2023-05-28 15:03:30][hdls] NO additional images were found to download
[INFO] [2023-05-28 15:03:30] 100% of media downloaded
[ERR] [2023-05-28 15:03:30][hdls] NO additional images were found to download
[INFO] [2023-05-28 15:03:30] 100% of media downloaded
[ERR] [2023-05-28 15:03:30][hdls] NO additional images were found to download
[INFO] [2023-05-28 15:03:30] 100% of media downloaded
[ERR] [2023-05-28 15:03:30][hdls] NO additional images were found to download
[WARN] [2023-05-28 15:03:31] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2023-05-28 15:03:38] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2023-05-28 15:03:45] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2023-05-28 15:03:52] Names to parse: 5537 formatted: 5537 learned: 5537 parsed: 5537
[STOP] [2023-05-28 15:03:56] parse_names
[START] [2023-05-28 15:03:56] denormalize_canonical_names_to_nodes
[STOP] [2023-05-28 15:03:57] denormalize_canonical_names_to_nodes
[START] [2023-05-28 15:03:57] match_nodes
[START] [2023-05-28 15:03:57] map_all_nodes_to_pages
[STOP] [2023-05-28 15:12:46] map_all_nodes_to_pages
[INFO] [2023-05-28 15:12:46] 3062 Unmatched nodes (of 35537)! That's too many to output. Full list in /app/public/data/usda_plants/unmatched_nodes.txt ; First 10: Canonical: Bastardia; Node#134925367; ResourceID: BASTA; Canonical: Bastardia viscosa sanctae-crucis; Node#134925391; ResourceID: BAVIS2; Canonical: Gossypium hirsutum hirsutum; Node#134936674; ResourceID: GOHIH2; Canonical: Gossypium hirsutum marie-galante; Node#134936675; ResourceID: GOHIM; Canonical: Hibiscus elatus; Node#134937815; ResourceID: HIEL; Canonical: Hibiscus macrophyllus; Node#134937878; ResourceID: HIMA5; Canonical: Iliamna rivularis diversa; Node#134938602; ResourceID: ILRID; Canonical: Lavatera olbia; Node#134939772; ResourceID: LAOL2; Canonical: Lagunaria patersonia; Node#134939775; ResourceID: LAPA; Canonical: Lavatera thuringiaca; Node#134939855; ResourceID: LATH
[START] [2023-05-28 15:12:46] update_nodes
[STOP] [2023-05-28 15:13:03] update_nodes
[STOP] [2023-05-28 15:13:03] match_nodes
[START] [2023-05-28 15:13:03] reindex_search
[STOP] [2023-05-28 15:13:27] reindex_search
[START] [2023-05-28 15:13:27] normalize_units
[STOP] [2023-05-28 15:15:28] normalize_units
[START] [2023-05-28 15:15:28] calculate_statistics
[INFO] [2023-05-28 15:15:32] Duplicate page_id count: 0
[STOP] [2023-05-28 15:15:32] calculate_statistics
[START] [2023-05-28 15:15:32] complete_harvest_instance
[START] [2023-05-28 15:15:32] overall_tsv_creation
[INFO] [2023-05-28 15:15:32] Exporting 35537 nodes as TSV in batches of 10000...
[INFO] [2023-05-28 15:15:32] Processing group of 35537 in 4 batches of 10000
[INFO] [2023-05-28 15:15:50] 162828 Traits (unfiltered) and 0 associations...
[INFO] [2023-05-28 15:15:50] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2023-05-28 15:17:47] 100000 traits mapped (137877 meta)...
[INFO] [2023-05-28 15:19:12] Mapped 162828 traits (225615 meta) for 10000 nodes.
[INFO] [2023-05-28 15:19:12] Building Associations map (this can take a while)...
[INFO] [2023-05-28 15:19:17] Done. 0 assocs mapped (0 meta).
[INFO] [2023-05-28 15:19:17] Adding 162828 traits...
[INFO] [2023-05-28 15:19:37] 1461 metadata added.
[INFO] [2023-05-28 15:19:37] Adding 0 assocs...
[INFO] [2023-05-28 15:19:37] 0 metadata added.
[INFO] [2023-05-28 15:21:03] Processed 10000/35537 nodes
[INFO] [2023-05-28 15:21:32] 150011 Traits (unfiltered) and 0 associations...
[INFO] [2023-05-28 15:21:32] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2023-05-28 15:23:43] 100000 traits mapped (139989 meta)...
[INFO] [2023-05-28 15:24:47] Mapped 150011 traits (203282 meta) for 10000 nodes.
[INFO] [2023-05-28 15:24:47] Building Associations map (this can take a while)...
[INFO] [2023-05-28 15:24:49] Done. 0 assocs mapped (0 meta).
[INFO] [2023-05-28 15:24:49] Adding 150011 traits...
[INFO] [2023-05-28 15:25:12] 792 metadata added.
[INFO] [2023-05-28 15:25:12] Adding 0 assocs...
[INFO] [2023-05-28 15:25:12] 0 metadata added.
[INFO] [2023-05-28 15:26:31] Processed 20000/35537 nodes
[INFO] [2023-05-28 15:26:56] 162955 Traits (unfiltered) and 0 associations...
[INFO] [2023-05-28 15:26:56] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2023-05-28 15:28:55] 100000 traits mapped (138406 meta)...
[INFO] [2023-05-28 15:30:23] Mapped 162955 traits (226431 meta) for 10000 nodes.
[INFO] [2023-05-28 15:30:23] Building Associations map (this can take a while)...
[INFO] [2023-05-28 15:30:25] Done. 0 assocs mapped (0 meta).
[INFO] [2023-05-28 15:30:25] Adding 162955 traits...
[INFO] [2023-05-28 15:30:46] 1037 metadata added.
[INFO] [2023-05-28 15:30:46] Adding 0 assocs...
[INFO] [2023-05-28 15:30:46] 0 metadata added.
[INFO] [2023-05-28 15:32:08] Processed 30000/35537 nodes
[INFO] [2023-05-28 15:32:29] 93222 Traits (unfiltered) and 0 associations...
[INFO] [2023-05-28 15:32:29] Building Traits map for 5537 nodes (this can take a while)...
[INFO] [2023-05-28 15:33:44] Mapped 93222 traits (124768 meta) for 5537 nodes.
[INFO] [2023-05-28 15:33:44] Building Associations map (this can take a while)...
[INFO] [2023-05-28 15:33:46] Done. 0 assocs mapped (0 meta).
[INFO] [2023-05-28 15:33:46] Adding 93222 traits...
[INFO] [2023-05-28 15:34:00] 464 metadata added.
[INFO] [2023-05-28 15:34:00] Adding 0 assocs...
[INFO] [2023-05-28 15:34:00] 0 metadata added.
[INFO] [2023-05-28 15:35:07] Processed 35537/35537 nodes
[INFO] [2023-05-28 15:35:07] Average Time: 285.195
[INFO] [2023-05-28 15:35:07] Total Time: 19m35s
[STOP] [2023-05-28 15:35:07] overall_tsv_creation
[INFO] [2023-05-28 15:35:07] Done. Check your files:
[INFO] [2023-05-28 15:35:07] (35537 lines) /app/public/data/usda_plants/publish_nodes.tsv
[INFO] [2023-05-28 15:35:07] (35186 lines) /app/public/data/usda_plants/publish_identifiers.tsv
[INFO] [2023-05-28 15:35:07] (35186 lines) /app/public/data/usda_plants/publish_node_ancestors.tsv
[INFO] [2023-05-28 15:35:08] (35537 lines) /app/public/data/usda_plants/publish_scientific_names.tsv
[INFO] [2023-05-28 15:35:08] (569017 lines) /app/public/data/usda_plants/publish_traits.tsv
[INFO] [2023-05-28 15:35:08] (3755 lines) /app/public/data/usda_plants/publish_metadata.tsv
[STOP] [2023-05-28 15:35:08] complete_harvest_instance
[START] [2023-05-28 15:35:08] completed
[STOP] [2023-05-28 15:35:08] completed
[STOP] [2023-05-28 15:35:08] logged process, took 3013.45

Latest Process

28 May 14:44 EXITED currently processing 0