Harvest for USDA Plants data Created 18 May 12:19

Stage: completed
Fetched: 18 May 12:19
Validated: 18 May 12:20
Deltas Created 18 May 12:20
Units Normalized: 18 May 12:51
Ancestry Built: 18 May 12:38
Nodes Matched: 18 May 12:49
Names Parsed: 18 May 12:38
New Models Stored: 18 May 12:35
Indexed: 18 May 12:49
Completed: 19 May 16:03
Time to Harvest: 2 minutes

Harvesting Log

(570 lines)
[INFO] [2022-05-18 12:19:48] Created harvest instance #4114
[STOP] [2022-05-18 12:19:48] create_harvest_instance
[START] [2022-05-18 12:19:48] fetch_files
[STOP] [2022-05-18 12:19:48] fetch_files
[START] [2022-05-18 12:19:48] validate_each_file
[INFO] [2022-05-18 12:19:48] Looping over 7 formats...
[INFO] [2022-05-18 12:19:48] ...agents (/app/public/data/usda_plants/agent.tab)
[INFO] [2022-05-18 12:19:48] Valid: /app/public/data/usda_plants/converted_csv/usda_plants_agents_29292.csv (1 lines)
[INFO] [2022-05-18 12:19:48] ...refs (/app/public/data/usda_plants/reference.tab)
[INFO] [2022-05-18 12:19:48] Valid: /app/public/data/usda_plants/converted_csv/usda_plants_refs_29291.csv (2 lines)
[INFO] [2022-05-18 12:19:48] ...nodes (/app/public/data/usda_plants/taxon.tab)
[INFO] [2022-05-18 12:19:49] Valid: /app/public/data/usda_plants/converted_csv/usda_plants_nodes_29293.csv (35186 lines)
[INFO] [2022-05-18 12:19:49] ...media (/app/public/data/usda_plants/media_resource.tab)
[INFO] [2022-05-18 12:19:49] Valid: /app/public/data/usda_plants/converted_csv/usda_plants_media_29290.csv (5 lines)
[INFO] [2022-05-18 12:19:49] ...vernaculars (/app/public/data/usda_plants/vernacular_name.tab)
[INFO] [2022-05-18 12:19:50] Valid: /app/public/data/usda_plants/converted_csv/usda_plants_vernaculars_29294.csv (47481 lines)
[INFO] [2022-05-18 12:19:50] ...occurrences (/app/public/data/usda_plants/occurrence_specific.tab)
[INFO] [2022-05-18 12:20:02] Valid: /app/public/data/usda_plants/converted_csv/usda_plants_occurrences_29295.csv (634875 lines)
[INFO] [2022-05-18 12:20:02] ...measurements (/app/public/data/usda_plants/measurement_or_fact_specific.tab)
[INFO] [2022-05-18 12:20:47] Valid: /app/public/data/usda_plants/converted_csv/usda_plants_measurements_29296.csv (580161 lines)
[STOP] [2022-05-18 12:20:47] validate_each_file
[START] [2022-05-18 12:20:47] convert_to_csv
[INFO] [2022-05-18 12:20:47] Looping over 7 formats...
[INFO] [2022-05-18 12:20:47] ...agents (/app/public/data/usda_plants/agent.tab)
[CMD] [2022-05-18 12:20:47] /usr/bin/sort /app/public/data/usda_plants/converted_csv/usda_plants_agents_29292.csv > /app/public/data/usda_plants/converted_csv/usda_plants_agents_29292.csv_sorted
[INFO] [2022-05-18 12:20:47] Converted: /app/public/data/usda_plants/converted_csv/usda_plants_agents_29292.csv (1 lines)
[INFO] [2022-05-18 12:20:47] ...refs (/app/public/data/usda_plants/reference.tab)
[CMD] [2022-05-18 12:20:47] /usr/bin/sort /app/public/data/usda_plants/converted_csv/usda_plants_refs_29291.csv > /app/public/data/usda_plants/converted_csv/usda_plants_refs_29291.csv_sorted
[INFO] [2022-05-18 12:20:47] Converted: /app/public/data/usda_plants/converted_csv/usda_plants_refs_29291.csv (2 lines)
[INFO] [2022-05-18 12:20:47] ...nodes (/app/public/data/usda_plants/taxon.tab)
[CMD] [2022-05-18 12:20:47] /usr/bin/sort /app/public/data/usda_plants/converted_csv/usda_plants_nodes_29293.csv > /app/public/data/usda_plants/converted_csv/usda_plants_nodes_29293.csv_sorted
[INFO] [2022-05-18 12:20:47] Converted: /app/public/data/usda_plants/converted_csv/usda_plants_nodes_29293.csv (35186 lines)
[INFO] [2022-05-18 12:20:47] ...media (/app/public/data/usda_plants/media_resource.tab)
[CMD] [2022-05-18 12:20:47] /usr/bin/sort /app/public/data/usda_plants/converted_csv/usda_plants_media_29290.csv > /app/public/data/usda_plants/converted_csv/usda_plants_media_29290.csv_sorted
[INFO] [2022-05-18 12:20:47] Converted: /app/public/data/usda_plants/converted_csv/usda_plants_media_29290.csv (5 lines)
[INFO] [2022-05-18 12:20:47] ...vernaculars (/app/public/data/usda_plants/vernacular_name.tab)
[CMD] [2022-05-18 12:20:47] /usr/bin/sort /app/public/data/usda_plants/converted_csv/usda_plants_vernaculars_29294.csv > /app/public/data/usda_plants/converted_csv/usda_plants_vernaculars_29294.csv_sorted
[INFO] [2022-05-18 12:20:47] Converted: /app/public/data/usda_plants/converted_csv/usda_plants_vernaculars_29294.csv (47481 lines)
[INFO] [2022-05-18 12:20:47] ...occurrences (/app/public/data/usda_plants/occurrence_specific.tab)
[CMD] [2022-05-18 12:20:47] /usr/bin/sort /app/public/data/usda_plants/converted_csv/usda_plants_occurrences_29295.csv > /app/public/data/usda_plants/converted_csv/usda_plants_occurrences_29295.csv_sorted
[INFO] [2022-05-18 12:20:47] Converted: /app/public/data/usda_plants/converted_csv/usda_plants_occurrences_29295.csv (634875 lines)
[INFO] [2022-05-18 12:20:47] ...measurements (/app/public/data/usda_plants/measurement_or_fact_specific.tab)
[CMD] [2022-05-18 12:20:47] /usr/bin/sort /app/public/data/usda_plants/converted_csv/usda_plants_measurements_29296.csv > /app/public/data/usda_plants/converted_csv/usda_plants_measurements_29296.csv_sorted
[INFO] [2022-05-18 12:20:53] Converted: /app/public/data/usda_plants/converted_csv/usda_plants_measurements_29296.csv (580161 lines)
[STOP] [2022-05-18 12:20:53] convert_to_csv
[START] [2022-05-18 12:20:53] calculate_delta
[INFO] [2022-05-18 12:20:53] Looping over 7 formats...
[INFO] [2022-05-18 12:20:53] ...agents (/app/public/data/usda_plants/agent.tab)
[CMD] [2022-05-18 12:20:53] echo "0a" > /app/public/data/usda_plants/diff/usda_plants_agents_29292.diff
[CMD] [2022-05-18 12:20:54] tail -n +1 /app/public/data/usda_plants/converted_csv/usda_plants_agents_29292.csv >> /app/public/data/usda_plants/diff/usda_plants_agents_29292.diff
[CMD] [2022-05-18 12:20:54] echo "." >> /app/public/data/usda_plants/diff/usda_plants_agents_29292.diff
[INFO] [2022-05-18 12:20:54] Created diff: /app/public/data/usda_plants/diff/usda_plants_agents_29292.diff (3 lines)
[INFO] [2022-05-18 12:20:54] ...refs (/app/public/data/usda_plants/reference.tab)
[CMD] [2022-05-18 12:20:54] echo "0a" > /app/public/data/usda_plants/diff/usda_plants_refs_29291.diff
[CMD] [2022-05-18 12:20:54] tail -n +1 /app/public/data/usda_plants/converted_csv/usda_plants_refs_29291.csv >> /app/public/data/usda_plants/diff/usda_plants_refs_29291.diff
[CMD] [2022-05-18 12:20:54] echo "." >> /app/public/data/usda_plants/diff/usda_plants_refs_29291.diff
[INFO] [2022-05-18 12:20:54] Created diff: /app/public/data/usda_plants/diff/usda_plants_refs_29291.diff (4 lines)
[INFO] [2022-05-18 12:20:54] ...nodes (/app/public/data/usda_plants/taxon.tab)
[CMD] [2022-05-18 12:20:54] echo "0a" > /app/public/data/usda_plants/diff/usda_plants_nodes_29293.diff
[CMD] [2022-05-18 12:20:54] tail -n +1 /app/public/data/usda_plants/converted_csv/usda_plants_nodes_29293.csv >> /app/public/data/usda_plants/diff/usda_plants_nodes_29293.diff
[CMD] [2022-05-18 12:20:54] echo "." >> /app/public/data/usda_plants/diff/usda_plants_nodes_29293.diff
[INFO] [2022-05-18 12:20:54] Created diff: /app/public/data/usda_plants/diff/usda_plants_nodes_29293.diff (35188 lines)
[INFO] [2022-05-18 12:20:54] ...media (/app/public/data/usda_plants/media_resource.tab)
[CMD] [2022-05-18 12:20:54] echo "0a" > /app/public/data/usda_plants/diff/usda_plants_media_29290.diff
[CMD] [2022-05-18 12:20:54] tail -n +1 /app/public/data/usda_plants/converted_csv/usda_plants_media_29290.csv >> /app/public/data/usda_plants/diff/usda_plants_media_29290.diff
[CMD] [2022-05-18 12:20:54] echo "." >> /app/public/data/usda_plants/diff/usda_plants_media_29290.diff
[INFO] [2022-05-18 12:20:54] Created diff: /app/public/data/usda_plants/diff/usda_plants_media_29290.diff (7 lines)
[INFO] [2022-05-18 12:20:54] ...vernaculars (/app/public/data/usda_plants/vernacular_name.tab)
[CMD] [2022-05-18 12:20:54] echo "0a" > /app/public/data/usda_plants/diff/usda_plants_vernaculars_29294.diff
[CMD] [2022-05-18 12:20:54] tail -n +1 /app/public/data/usda_plants/converted_csv/usda_plants_vernaculars_29294.csv >> /app/public/data/usda_plants/diff/usda_plants_vernaculars_29294.diff
[CMD] [2022-05-18 12:20:54] echo "." >> /app/public/data/usda_plants/diff/usda_plants_vernaculars_29294.diff
[INFO] [2022-05-18 12:20:54] Created diff: /app/public/data/usda_plants/diff/usda_plants_vernaculars_29294.diff (47483 lines)
[INFO] [2022-05-18 12:20:54] ...occurrences (/app/public/data/usda_plants/occurrence_specific.tab)
[CMD] [2022-05-18 12:20:54] echo "0a" > /app/public/data/usda_plants/diff/usda_plants_occurrences_29295.diff
[CMD] [2022-05-18 12:20:54] tail -n +1 /app/public/data/usda_plants/converted_csv/usda_plants_occurrences_29295.csv >> /app/public/data/usda_plants/diff/usda_plants_occurrences_29295.diff
[CMD] [2022-05-18 12:20:54] echo "." >> /app/public/data/usda_plants/diff/usda_plants_occurrences_29295.diff
[INFO] [2022-05-18 12:20:54] Created diff: /app/public/data/usda_plants/diff/usda_plants_occurrences_29295.diff (634877 lines)
[INFO] [2022-05-18 12:20:54] ...measurements (/app/public/data/usda_plants/measurement_or_fact_specific.tab)
[CMD] [2022-05-18 12:20:54] echo "0a" > /app/public/data/usda_plants/diff/usda_plants_measurements_29296.diff
[CMD] [2022-05-18 12:20:54] tail -n +1 /app/public/data/usda_plants/converted_csv/usda_plants_measurements_29296.csv >> /app/public/data/usda_plants/diff/usda_plants_measurements_29296.diff
[CMD] [2022-05-18 12:20:57] echo "." >> /app/public/data/usda_plants/diff/usda_plants_measurements_29296.diff
[INFO] [2022-05-18 12:20:58] Created diff: /app/public/data/usda_plants/diff/usda_plants_measurements_29296.diff (580163 lines)
[STOP] [2022-05-18 12:20:58] calculate_delta
[START] [2022-05-18 12:20:58] parse_diff_and_store
[INFO] [2022-05-18 12:20:58] Handling diff: /app/public/data/usda_plants/diff/usda_plants_agents_29292.diff (3 lines)
[INFO] [2022-05-18 12:20:58] Loading agents diff file into memory (3 lines)...
[INFO] [2022-05-18 12:20:58] Storing 1 Attributions (1/1/3)
[INFO] [2022-05-18 12:20:58] Handling diff: /app/public/data/usda_plants/diff/usda_plants_refs_29291.diff (4 lines)
[INFO] [2022-05-18 12:20:58] Loading refs diff file into memory (4 lines)...
[INFO] [2022-05-18 12:20:58] Storing 2 References (2/2/4)
[INFO] [2022-05-18 12:20:58] Handling diff: /app/public/data/usda_plants/diff/usda_plants_nodes_29293.diff (35188 lines)
[INFO] [2022-05-18 12:20:58] Loading nodes diff file into memory (35188 lines)...
[INFO] [2022-05-18 12:21:02] Storing 10200 ScientificNames (30399/10000/35188)
[INFO] [2022-05-18 12:21:05] Storing 10200 Nodes (30399/10000/35188)
[INFO] [2022-05-18 12:21:08] Storing 9999 Identifiers (30399/10000/35188)
[WARN] [2022-05-18 12:21:13] SKIPPED 130 Scientific names (60811/20000/35188) with resource_pks already be in the database!
[WARN] [2022-05-18 12:21:13] SKIPPED 130 Nodes (60811/20000/35188) with resource_pks already be in the database!
[INFO] [2022-05-18 12:21:13] Storing 10076 ScientificNames (60811/20000/35188)
[INFO] [2022-05-18 12:21:16] Storing 10076 Nodes (60811/20000/35188)
[INFO] [2022-05-18 12:21:19] Storing 10000 Identifiers (60811/20000/35188)
[WARN] [2022-05-18 12:21:23] SKIPPED 149 Scientific names (91207/30000/35188) with resource_pks already be in the database!
[WARN] [2022-05-18 12:21:23] SKIPPED 149 Nodes (91207/30000/35188) with resource_pks already be in the database!
[INFO] [2022-05-18 12:21:23] Storing 10049 ScientificNames (91207/30000/35188)
[INFO] [2022-05-18 12:21:26] Storing 10049 Nodes (91207/30000/35188)
[INFO] [2022-05-18 12:21:29] Storing 10000 Identifiers (91207/30000/35188)
[WARN] [2022-05-18 12:21:32] SKIPPED 128 Scientific names (107074/35186/35188) with resource_pks already be in the database!
[WARN] [2022-05-18 12:21:32] SKIPPED 128 Nodes (107074/35186/35188) with resource_pks already be in the database!
[INFO] [2022-05-18 12:21:32] Storing 5212 ScientificNames (107074/35186/35188)
[INFO] [2022-05-18 12:21:34] Storing 5212 Nodes (107074/35186/35188)
[INFO] [2022-05-18 12:21:35] Storing 5187 Identifiers (107074/35186/35188)
[INFO] [2022-05-18 12:21:36] Handling diff: /app/public/data/usda_plants/diff/usda_plants_media_29290.diff (7 lines)
[INFO] [2022-05-18 12:21:36] Loading media diff file into memory (7 lines)...
[INFO] [2022-05-18 12:21:36] Storing 2 BibliographicCitations (10/5/7)
[INFO] [2022-05-18 12:21:36] Storing 2 ArticlesSections (10/5/7)
[INFO] [2022-05-18 12:21:36] Storing 2 Articles (10/5/7)
[INFO] [2022-05-18 12:21:36] Storing 1 ContentAttributions (10/5/7)
[INFO] [2022-05-18 12:21:36] Storing 3 Media (10/5/7)
[INFO] [2022-05-18 12:21:37] Handling diff: /app/public/data/usda_plants/diff/usda_plants_vernaculars_29294.diff (47483 lines)
[INFO] [2022-05-18 12:21:37] Loading vernaculars diff file into memory (47483 lines)...
[INFO] [2022-05-18 12:21:38] Storing 9999 Vernaculars (9999/10000/47483)
[INFO] [2022-05-18 12:21:41] Storing 10000 Vernaculars (19999/20000/47483)
[INFO] [2022-05-18 12:21:44] Storing 10000 Vernaculars (29999/30000/47483)
[INFO] [2022-05-18 12:21:46] Storing 10000 Vernaculars (39999/40000/47483)
[INFO] [2022-05-18 12:21:49] Storing 7482 Vernaculars (47481/47481/47483)
[INFO] [2022-05-18 12:21:51] Handling diff: /app/public/data/usda_plants/diff/usda_plants_occurrences_29295.diff (634877 lines)
[INFO] [2022-05-18 12:21:51] Loading occurrences diff file into memory (634877 lines)...
[INFO] [2022-05-18 12:21:51] Storing 9999 Occurrences (9999/10000/634877)
[INFO] [2022-05-18 12:21:53] Storing 10000 Occurrences (19999/20000/634877)
[INFO] [2022-05-18 12:21:56] Storing 10000 Occurrences (29999/30000/634877)
[INFO] [2022-05-18 12:21:58] Storing 10000 Occurrences (39999/40000/634877)
[INFO] [2022-05-18 12:22:00] Storing 10000 Occurrences (49999/50000/634877)
[INFO] [2022-05-18 12:22:02] Storing 10000 Occurrences (59999/60000/634877)
[INFO] [2022-05-18 12:22:04] Storing 10000 Occurrences (69999/70000/634877)
[INFO] [2022-05-18 12:22:07] Storing 10000 Occurrences (79999/80000/634877)
[INFO] [2022-05-18 12:22:09] Storing 10000 Occurrences (89999/90000/634877)
[INFO] [2022-05-18 12:22:11] Storing 10000 Occurrences (99999/100000/634877)
[INFO] [2022-05-18 12:22:13] Storing 10000 Occurrences (109999/110000/634877)
[INFO] [2022-05-18 12:22:16] Storing 10000 Occurrences (119999/120000/634877)
[INFO] [2022-05-18 12:22:18] Storing 10000 Occurrences (129999/130000/634877)
[INFO] [2022-05-18 12:22:21] Storing 10000 Occurrences (139999/140000/634877)
[INFO] [2022-05-18 12:22:23] Storing 10000 Occurrences (149999/150000/634877)
[INFO] [2022-05-18 12:22:26] Storing 10000 Occurrences (159999/160000/634877)
[INFO] [2022-05-18 12:22:29] Storing 10000 Occurrences (169999/170000/634877)
[INFO] [2022-05-18 12:22:31] Storing 10000 Occurrences (179999/180000/634877)
[INFO] [2022-05-18 12:22:34] Storing 10000 Occurrences (189999/190000/634877)
[INFO] [2022-05-18 12:22:37] Storing 10000 Occurrences (199999/200000/634877)
[INFO] [2022-05-18 12:22:40] Storing 10000 Occurrences (209999/210000/634877)
[INFO] [2022-05-18 12:22:43] Storing 10000 Occurrences (219999/220000/634877)
[INFO] [2022-05-18 12:22:46] Storing 10000 Occurrences (229999/230000/634877)
[INFO] [2022-05-18 12:22:48] Storing 10000 Occurrences (239999/240000/634877)
[INFO] [2022-05-18 12:22:51] Storing 10000 Occurrences (249999/250000/634877)
[INFO] [2022-05-18 12:22:54] Storing 10000 Occurrences (259999/260000/634877)
[INFO] [2022-05-18 12:22:57] Storing 10000 Occurrences (269999/270000/634877)
[INFO] [2022-05-18 12:23:00] Storing 10000 Occurrences (279999/280000/634877)
[INFO] [2022-05-18 12:23:03] Storing 10000 Occurrences (289999/290000/634877)
[INFO] [2022-05-18 12:23:06] Storing 10000 Occurrences (299999/300000/634877)
[INFO] [2022-05-18 12:23:11] Storing 10000 Occurrences (316694/310000/634877)
[INFO] [2022-05-18 12:23:12] Storing 6695 OccurrenceMetadata (316694/310000/634877)
[WARN] [2022-05-18 12:23:15] SKIPPED 1083 Occurrence metadata (327777/320000/634877) with resource_pks already be in the database!
[INFO] [2022-05-18 12:23:15] Storing 10000 Occurrences (327777/320000/634877)
[INFO] [2022-05-18 12:23:16] Storing 0 OccurrenceMetadata (327777/320000/634877)
[WARN] [2022-05-18 12:23:16] No models to import, skipping!
[INFO] [2022-05-18 12:23:18] Storing 10000 Occurrences (337777/330000/634877)
[INFO] [2022-05-18 12:23:21] Storing 10000 Occurrences (347777/340000/634877)
[INFO] [2022-05-18 12:23:25] Storing 10000 Occurrences (357777/350000/634877)
[INFO] [2022-05-18 12:23:28] Storing 10000 Occurrences (367777/360000/634877)
[INFO] [2022-05-18 12:23:31] Storing 10000 Occurrences (377777/370000/634877)
[INFO] [2022-05-18 12:23:34] Storing 10000 Occurrences (387777/380000/634877)
[INFO] [2022-05-18 12:23:37] Storing 10000 Occurrences (397777/390000/634877)
[INFO] [2022-05-18 12:23:40] Storing 10000 Occurrences (407777/400000/634877)
[WARN] [2022-05-18 12:23:44] SKIPPED 1814 Occurrence metadata (419591/410000/634877) with resource_pks already be in the database!
[INFO] [2022-05-18 12:23:44] Storing 10000 Occurrences (419591/410000/634877)
[INFO] [2022-05-18 12:23:45] Storing 0 OccurrenceMetadata (419591/410000/634877)
[WARN] [2022-05-18 12:23:45] No models to import, skipping!
[WARN] [2022-05-18 12:23:48] SKIPPED 61 Occurrence metadata (429652/420000/634877) with resource_pks already be in the database!
[INFO] [2022-05-18 12:23:48] Storing 10000 Occurrences (429652/420000/634877)
[INFO] [2022-05-18 12:23:49] Storing 0 OccurrenceMetadata (429652/420000/634877)
[WARN] [2022-05-18 12:23:49] No models to import, skipping!
[INFO] [2022-05-18 12:23:51] Storing 10000 Occurrences (439652/430000/634877)
[INFO] [2022-05-18 12:23:54] Storing 10000 Occurrences (449652/440000/634877)
[INFO] [2022-05-18 12:23:58] Storing 10000 Occurrences (459652/450000/634877)
[INFO] [2022-05-18 12:24:01] Storing 10000 Occurrences (469652/460000/634877)
[INFO] [2022-05-18 12:24:04] Storing 10000 Occurrences (479652/470000/634877)
[INFO] [2022-05-18 12:24:08] Storing 10000 Occurrences (489652/480000/634877)
[INFO] [2022-05-18 12:24:12] Storing 10000 Occurrences (499652/490000/634877)
[INFO] [2022-05-18 12:24:15] Storing 10000 Occurrences (509652/500000/634877)
[INFO] [2022-05-18 12:24:19] Storing 10000 Occurrences (519652/510000/634877)
[INFO] [2022-05-18 12:24:23] Storing 10000 Occurrences (529652/520000/634877)
[INFO] [2022-05-18 12:24:27] Storing 10000 Occurrences (539652/530000/634877)
[INFO] [2022-05-18 12:24:31] Storing 10000 Occurrences (549652/540000/634877)
[INFO] [2022-05-18 12:24:35] Storing 10000 Occurrences (559652/550000/634877)
[INFO] [2022-05-18 12:24:39] Storing 10000 Occurrences (569652/560000/634877)
[INFO] [2022-05-18 12:24:43] Storing 10000 Occurrences (579652/570000/634877)
[INFO] [2022-05-18 12:24:47] Storing 10000 Occurrences (589652/580000/634877)
[INFO] [2022-05-18 12:24:52] Storing 10000 Occurrences (599652/590000/634877)
[INFO] [2022-05-18 12:24:56] Storing 10000 Occurrences (609652/600000/634877)
[INFO] [2022-05-18 12:25:00] Storing 10000 Occurrences (619652/610000/634877)
[INFO] [2022-05-18 12:25:05] Storing 10000 Occurrences (629652/620000/634877)
[INFO] [2022-05-18 12:25:09] Storing 10000 Occurrences (639652/630000/634877)
[INFO] [2022-05-18 12:25:13] Storing 4876 Occurrences (644528/634875/634877)
[INFO] [2022-05-18 12:25:15] Handling diff: /app/public/data/usda_plants/diff/usda_plants_measurements_29296.diff (580163 lines)
[INFO] [2022-05-18 12:25:15] Loading measurements diff file into memory (580163 lines)...
[INFO] [2022-05-18 12:25:19] Storing 9999 Traits (19998/10000/580163)
[INFO] [2022-05-18 12:25:22] Storing 9999 MetaTraits (19998/10000/580163)
[INFO] [2022-05-18 12:25:28] Storing 10000 Traits (39998/20000/580163)
[INFO] [2022-05-18 12:25:31] Storing 10000 MetaTraits (39998/20000/580163)
[INFO] [2022-05-18 12:25:36] Storing 10000 Traits (59998/30000/580163)
[INFO] [2022-05-18 12:25:39] Storing 10000 MetaTraits (59998/30000/580163)
[INFO] [2022-05-18 12:25:45] Storing 10000 Traits (79998/40000/580163)
[INFO] [2022-05-18 12:25:48] Storing 10000 MetaTraits (79998/40000/580163)
[INFO] [2022-05-18 12:25:53] Storing 10000 Traits (99998/50000/580163)
[INFO] [2022-05-18 12:25:56] Storing 10000 MetaTraits (99998/50000/580163)
[INFO] [2022-05-18 12:26:02] Storing 10000 Traits (119998/60000/580163)
[INFO] [2022-05-18 12:26:05] Storing 10000 MetaTraits (119998/60000/580163)
[INFO] [2022-05-18 12:26:10] Storing 10000 Traits (139998/70000/580163)
[INFO] [2022-05-18 12:26:13] Storing 10000 MetaTraits (139998/70000/580163)
[INFO] [2022-05-18 12:26:19] Storing 10000 Traits (159998/80000/580163)
[INFO] [2022-05-18 12:26:22] Storing 10000 MetaTraits (159998/80000/580163)
[INFO] [2022-05-18 12:26:28] Storing 10000 Traits (179998/90000/580163)
[INFO] [2022-05-18 12:26:32] Storing 10000 MetaTraits (179998/90000/580163)
[INFO] [2022-05-18 12:26:37] Storing 10000 Traits (199998/100000/580163)
[INFO] [2022-05-18 12:26:41] Storing 10000 MetaTraits (199998/100000/580163)
[INFO] [2022-05-18 12:26:47] Storing 10000 Traits (219998/110000/580163)
[INFO] [2022-05-18 12:26:50] Storing 10000 MetaTraits (219998/110000/580163)
[INFO] [2022-05-18 12:26:56] Storing 10000 Traits (239998/120000/580163)
[INFO] [2022-05-18 12:26:59] Storing 10000 MetaTraits (239998/120000/580163)
[INFO] [2022-05-18 12:27:04] Storing 10000 Traits (259998/130000/580163)
[INFO] [2022-05-18 12:27:07] Storing 10000 MetaTraits (259998/130000/580163)
[INFO] [2022-05-18 12:27:13] Storing 10000 Traits (279998/140000/580163)
[INFO] [2022-05-18 12:27:16] Storing 10000 MetaTraits (279998/140000/580163)
[INFO] [2022-05-18 12:27:22] Storing 10000 Traits (299998/150000/580163)
[INFO] [2022-05-18 12:27:25] Storing 10000 MetaTraits (299998/150000/580163)
[INFO] [2022-05-18 12:27:31] Storing 10000 Traits (319998/160000/580163)
[INFO] [2022-05-18 12:27:34] Storing 10000 MetaTraits (319998/160000/580163)
[INFO] [2022-05-18 12:27:40] Storing 10000 Traits (339998/170000/580163)
[INFO] [2022-05-18 12:27:44] Storing 10000 MetaTraits (339998/170000/580163)
[INFO] [2022-05-18 12:27:50] Storing 10000 Traits (359998/180000/580163)
[INFO] [2022-05-18 12:27:53] Storing 10000 MetaTraits (359998/180000/580163)
[INFO] [2022-05-18 12:27:59] Storing 10000 Traits (379998/190000/580163)
[INFO] [2022-05-18 12:28:02] Storing 10000 MetaTraits (379998/190000/580163)
[INFO] [2022-05-18 12:28:08] Storing 10000 Traits (399998/200000/580163)
[INFO] [2022-05-18 12:28:12] Storing 10000 MetaTraits (399998/200000/580163)
[INFO] [2022-05-18 12:28:18] Storing 10000 Traits (419998/210000/580163)
[INFO] [2022-05-18 12:28:21] Storing 10000 MetaTraits (419998/210000/580163)
[INFO] [2022-05-18 12:28:27] Storing 10000 Traits (439998/220000/580163)
[INFO] [2022-05-18 12:28:30] Storing 10000 MetaTraits (439998/220000/580163)
[INFO] [2022-05-18 12:28:37] Storing 10000 Traits (463745/230000/580163)
[INFO] [2022-05-18 12:28:40] Storing 13747 MetaTraits (463745/230000/580163)
[INFO] [2022-05-18 12:28:49] Storing 10000 Traits (493059/240000/580163)
[INFO] [2022-05-18 12:28:52] Storing 19314 MetaTraits (493059/240000/580163)
[INFO] [2022-05-18 12:29:01] Storing 10000 Traits (523059/250000/580163)
[INFO] [2022-05-18 12:29:04] Storing 20000 MetaTraits (523059/250000/580163)
[INFO] [2022-05-18 12:29:14] Storing 10000 Traits (553059/260000/580163)
[INFO] [2022-05-18 12:29:17] Storing 20000 MetaTraits (553059/260000/580163)
[INFO] [2022-05-18 12:29:27] Storing 10000 Traits (585322/270000/580163)
[INFO] [2022-05-18 12:29:30] Storing 22263 MetaTraits (585322/270000/580163)
[INFO] [2022-05-18 12:29:40] Storing 10000 Traits (615923/280000/580163)
[INFO] [2022-05-18 12:29:43] Storing 20601 MetaTraits (615923/280000/580163)
[INFO] [2022-05-18 12:29:52] Storing 10000 Traits (645923/290000/580163)
[INFO] [2022-05-18 12:29:55] Storing 20000 MetaTraits (645923/290000/580163)
[INFO] [2022-05-18 12:30:05] Storing 10000 Traits (675923/300000/580163)
[INFO] [2022-05-18 12:30:08] Storing 20000 MetaTraits (675923/300000/580163)
[INFO] [2022-05-18 12:30:18] Storing 10000 Traits (707933/310000/580163)
[INFO] [2022-05-18 12:30:21] Storing 22010 MetaTraits (707933/310000/580163)
[INFO] [2022-05-18 12:30:30] Storing 10000 Traits (737933/320000/580163)
[INFO] [2022-05-18 12:30:34] Storing 20000 MetaTraits (737933/320000/580163)
[INFO] [2022-05-18 12:30:43] Storing 10000 Traits (767933/330000/580163)
[INFO] [2022-05-18 12:30:46] Storing 20000 MetaTraits (767933/330000/580163)
[INFO] [2022-05-18 12:30:55] Storing 10000 Traits (797933/340000/580163)
[INFO] [2022-05-18 12:30:58] Storing 20000 MetaTraits (797933/340000/580163)
[INFO] [2022-05-18 12:31:08] Storing 10000 Traits (827933/350000/580163)
[INFO] [2022-05-18 12:31:11] Storing 20000 MetaTraits (827933/350000/580163)
[INFO] [2022-05-18 12:31:21] Storing 10000 Traits (857933/360000/580163)
[INFO] [2022-05-18 12:31:24] Storing 20000 MetaTraits (857933/360000/580163)
[INFO] [2022-05-18 12:31:33] Storing 10000 Traits (887933/370000/580163)
[INFO] [2022-05-18 12:31:36] Storing 20000 MetaTraits (887933/370000/580163)
[INFO] [2022-05-18 12:31:46] Storing 10000 Traits (917933/380000/580163)
[INFO] [2022-05-18 12:31:49] Storing 20000 MetaTraits (917933/380000/580163)
[INFO] [2022-05-18 12:31:59] Storing 10000 Traits (947933/390000/580163)
[INFO] [2022-05-18 12:32:02] Storing 20000 MetaTraits (947933/390000/580163)
[INFO] [2022-05-18 12:32:12] Storing 10000 Traits (977933/400000/580163)
[INFO] [2022-05-18 12:32:15] Storing 20000 MetaTraits (977933/400000/580163)
[INFO] [2022-05-18 12:32:24] Storing 10000 Traits (1007933/410000/580163)
[INFO] [2022-05-18 12:32:27] Storing 20000 MetaTraits (1007933/410000/580163)
[INFO] [2022-05-18 12:32:37] Storing 10000 Traits (1037933/420000/580163)
[INFO] [2022-05-18 12:32:40] Storing 20000 MetaTraits (1037933/420000/580163)
[INFO] [2022-05-18 12:32:50] Storing 10000 Traits (1067933/430000/580163)
[INFO] [2022-05-18 12:32:53] Storing 20000 MetaTraits (1067933/430000/580163)
[INFO] [2022-05-18 12:33:03] Storing 10000 Traits (1097933/440000/580163)
[INFO] [2022-05-18 12:33:06] Storing 20000 MetaTraits (1097933/440000/580163)
[INFO] [2022-05-18 12:33:15] Storing 10000 Traits (1122302/450000/580163)
[INFO] [2022-05-18 12:33:18] Storing 14369 MetaTraits (1122302/450000/580163)
[INFO] [2022-05-18 12:33:26] Storing 10000 Traits (1142302/460000/580163)
[INFO] [2022-05-18 12:33:29] Storing 10000 MetaTraits (1142302/460000/580163)
[INFO] [2022-05-18 12:33:36] Storing 10000 Traits (1162302/470000/580163)
[INFO] [2022-05-18 12:33:39] Storing 10000 MetaTraits (1162302/470000/580163)
[INFO] [2022-05-18 12:33:46] Storing 10000 Traits (1182302/480000/580163)
[INFO] [2022-05-18 12:33:49] Storing 10000 MetaTraits (1182302/480000/580163)
[INFO] [2022-05-18 12:33:56] Storing 10000 Traits (1202302/490000/580163)
[INFO] [2022-05-18 12:33:59] Storing 10000 MetaTraits (1202302/490000/580163)
[INFO] [2022-05-18 12:34:07] Storing 10000 Traits (1222302/500000/580163)
[INFO] [2022-05-18 12:34:10] Storing 10000 MetaTraits (1222302/500000/580163)
[INFO] [2022-05-18 12:34:17] Storing 10000 Traits (1242302/510000/580163)
[INFO] [2022-05-18 12:34:20] Storing 10000 MetaTraits (1242302/510000/580163)
[INFO] [2022-05-18 12:34:27] Storing 10000 Traits (1262302/520000/580163)
[INFO] [2022-05-18 12:34:30] Storing 10000 MetaTraits (1262302/520000/580163)
[INFO] [2022-05-18 12:34:37] Storing 10000 Traits (1282302/530000/580163)
[INFO] [2022-05-18 12:34:40] Storing 10000 MetaTraits (1282302/530000/580163)
[INFO] [2022-05-18 12:34:47] Storing 10000 Traits (1302302/540000/580163)
[INFO] [2022-05-18 12:34:50] Storing 10000 MetaTraits (1302302/540000/580163)
[INFO] [2022-05-18 12:34:58] Storing 10000 Traits (1322302/550000/580163)
[INFO] [2022-05-18 12:35:01] Storing 10000 MetaTraits (1322302/550000/580163)
[INFO] [2022-05-18 12:35:08] Storing 10000 Traits (1342302/560000/580163)
[INFO] [2022-05-18 12:35:12] Storing 10000 MetaTraits (1342302/560000/580163)
[INFO] [2022-05-18 12:35:19] Storing 10000 Traits (1362302/570000/580163)
[INFO] [2022-05-18 12:35:22] Storing 10000 MetaTraits (1362302/570000/580163)
[INFO] [2022-05-18 12:35:30] Storing 10000 Traits (1382302/580000/580163)
[INFO] [2022-05-18 12:35:33] Storing 10000 MetaTraits (1382302/580000/580163)
[INFO] [2022-05-18 12:35:36] Storing 162 Traits (1382626/580161/580163)
[INFO] [2022-05-18 12:35:36] Storing 162 MetaTraits (1382626/580161/580163)
[STOP] [2022-05-18 12:35:36] parse_diff_and_store
[START] [2022-05-18 12:35:36] resolve_keys
[2022-05-18 12:36:03] Resolving downloaded urls (this is not actually downloading them yet)
[INFO] [2022-05-18 12:36:36] Occurrences to nodes (through scientific_names)...
[INFO] [2022-05-18 12:36:51] traits to occurrences...
[INFO] [2022-05-18 12:37:12] traits to nodes (through occurrences)...
[INFO] [2022-05-18 12:37:25] Traits to sex term...
[INFO] [2022-05-18 12:37:34] Traits to lifestage term...
[INFO] [2022-05-18 12:37:43] MetaTraits to traits...
[INFO] [2022-05-18 12:38:04] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2022-05-18 12:38:05] Assocs to occurrences...
[INFO] [2022-05-18 12:38:05] Assocs to nodes...
[INFO] [2022-05-18 12:38:05] Assoc to sex term...
[INFO] [2022-05-18 12:38:05] Assoc to lifestage term...
[INFO] [2022-05-18 12:38:05] MetaAssoc to assocs...
[STOP] [2022-05-18 12:38:05] resolve_keys
[START] [2022-05-18 12:38:05] hold_for_later_1
[STOP] [2022-05-18 12:38:05] hold_for_later_1
[START] [2022-05-18 12:38:05] hold_for_later_2
[STOP] [2022-05-18 12:38:05] hold_for_later_2
[START] [2022-05-18 12:38:05] resolve_missing_parents
[STOP] [2022-05-18 12:38:08] resolve_missing_parents
[START] [2022-05-18 12:38:08] rebuild_nodes
[START] [2022-05-18 12:38:08] Flattener#flatten
[START] [2022-05-18 12:38:08] Flattener#study_resource
[START] [2022-05-18 12:38:08] Flattener#build_ancestry
[STOP] [2022-05-18 12:38:11] Flattener#build_ancestry
[INFO] [2022-05-18 12:38:11] 35537 ancestry keys
[START] [2022-05-18 12:38:11] build_node_ancestors
[INFO] [2022-05-18 12:38:11] old ancestors deleted.
[STOP] [2022-05-18 12:38:12] build_node_ancestors
[START] [2022-05-18 12:38:15] Flattener#propagate_ancestor_ids
[STOP] [2022-05-18 12:38:15] Flattener#propagate_ancestor_ids
[STOP] [2022-05-18 12:38:15] Flattener#flatten
[STOP] [2022-05-18 12:38:15] rebuild_nodes
[START] [2022-05-18 12:38:15] resolve_missing_media_owners
[STOP] [2022-05-18 12:38:15] resolve_missing_media_owners
[START] [2022-05-18 12:38:15] sanitize_media_verbatims
[STOP] [2022-05-18 12:38:15] sanitize_media_verbatims
[START] [2022-05-18 12:38:15] queue_downloads
[STOP] [2022-05-18 12:38:15] queue_downloads
[START] [2022-05-18 12:38:15] parse_names
[WARN] [2022-05-18 12:38:15] I see 35537 names which still need to be parsed.
[WARN] [2022-05-18 12:38:17] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[INFO] [2022-05-18 12:38:18] 0% of media downloaded
[INFO] [2022-05-18 12:38:18] 100% of media downloaded
[INFO] [2022-05-18 12:38:18] 100% of media downloaded
[WARN] [2022-05-18 12:38:24] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2022-05-18 12:38:32] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2022-05-18 12:38:39] Names to parse: 5537 formatted: 5537 learned: 5537 parsed: 5537
[STOP] [2022-05-18 12:38:43] parse_names
[START] [2022-05-18 12:38:43] denormalize_canonical_names_to_nodes
[STOP] [2022-05-18 12:38:43] denormalize_canonical_names_to_nodes
[START] [2022-05-18 12:38:43] match_nodes
[START] [2022-05-18 12:38:43] map_all_nodes_to_pages
[STOP] [2022-05-18 12:49:01] map_all_nodes_to_pages
[INFO] [2022-05-18 12:49:01] 3645 Unmatched nodes (of 35537)! That's too many to output. Full list in /app/public/data/usda_plants/unmatched_nodes.txt ; First 10: Canonical: Bastardia; Node#112946172; ResourceID: BASTA; Canonical: Gossypium hirsutum hirsutum; Node#112957479; ResourceID: GOHIH2; Canonical: Gossypium hirsutum marie-galante; Node#112957480; ResourceID: GOHIM; Canonical: Hibiscus elatus; Node#112958620; ResourceID: HIEL; Canonical: Hibiscus macrophyllus; Node#112958683; ResourceID: HIMA5; Canonical: Iliamna rivularis diversa; Node#112959407; ResourceID: ILRID; Canonical: Lavatera olbia; Node#112960577; ResourceID: LAOL2; Canonical: Lagunaria patersonia; Node#112960580; ResourceID: LAPA; Canonical: Lavatera thuringiaca; Node#112960660; ResourceID: LATH; Canonical: Lavatera; Node#112960684; ResourceID: LAVAT
[START] [2022-05-18 12:49:01] update_nodes
[STOP] [2022-05-18 12:49:19] update_nodes
[STOP] [2022-05-18 12:49:19] match_nodes
[START] [2022-05-18 12:49:19] reindex_search
[STOP] [2022-05-18 12:49:43] reindex_search
[START] [2022-05-18 12:49:43] normalize_units
[STOP] [2022-05-18 12:51:24] normalize_units
[START] [2022-05-18 12:51:24] calculate_statistics
[INFO] [2022-05-18 12:51:39] Duplicate page_id count: 0
[2022-05-18 12:51:44] (NEAR) DUPLICATE TRAITS FOUND! There are only 558206 (of 580161 total) unique traits.
[START] [2022-05-19 09:41:42] logged process: ec613da13a59e8ab5b959bc001fc093a2490fe5a Remove old refs to Link class (which was removed)
[INFO] [2022-05-19 09:41:42] Already completed stage create_harvest_instance, skipping...
[INFO] [2022-05-19 09:41:42] Already completed stage fetch_files, skipping...
[INFO] [2022-05-19 09:41:42] Already completed stage validate_each_file, skipping...
[INFO] [2022-05-19 09:41:42] Already completed stage convert_to_csv, skipping...
[INFO] [2022-05-19 09:41:42] Already completed stage calculate_delta, skipping...
[INFO] [2022-05-19 09:41:42] Already completed stage parse_diff_and_store, skipping...
[INFO] [2022-05-19 09:41:42] Already completed stage resolve_keys, skipping...
[INFO] [2022-05-19 09:41:42] Already completed stage hold_for_later_1, skipping...
[INFO] [2022-05-19 09:41:42] Already completed stage hold_for_later_2, skipping...
[INFO] [2022-05-19 09:41:42] Already completed stage resolve_missing_parents, skipping...
[INFO] [2022-05-19 09:41:42] Already completed stage rebuild_nodes, skipping...
[INFO] [2022-05-19 09:41:42] Already completed stage resolve_missing_media_owners, skipping...
[INFO] [2022-05-19 09:41:42] Already completed stage sanitize_media_verbatims, skipping...
[INFO] [2022-05-19 09:41:42] Already completed stage queue_downloads, skipping...
[INFO] [2022-05-19 09:41:42] Already completed stage parse_names, skipping...
[INFO] [2022-05-19 09:41:42] Already completed stage denormalize_canonical_names_to_nodes, skipping...
[INFO] [2022-05-19 09:41:42] Already completed stage match_nodes, skipping...
[INFO] [2022-05-19 09:41:42] Already completed stage reindex_search, skipping...
[INFO] [2022-05-19 09:41:42] Already completed stage normalize_units, skipping...
[START] [2022-05-19 09:41:42] calculate_statistics
[INFO] [2022-05-19 09:41:45] Duplicate page_id count: 0
[2022-05-19 09:41:51] (NEAR) DUPLICATE TRAITS FOUND! There are only 558206 (of 580161 total) unique traits.
[STOP] [2022-05-19 10:54:10] calculate_statistics
[STOP] [2022-05-19 10:54:10] logged process, took 4348.02
[START] [2022-05-19 11:03:19] logged process: ec613da13a59e8ab5b959bc001fc093a2490fe5a Remove old refs to Link class (which was removed)
[INFO] [2022-05-19 11:03:19] Already completed stage create_harvest_instance, skipping...
[INFO] [2022-05-19 11:03:19] Already completed stage fetch_files, skipping...
[INFO] [2022-05-19 11:03:19] Already completed stage validate_each_file, skipping...
[INFO] [2022-05-19 11:03:19] Already completed stage convert_to_csv, skipping...
[INFO] [2022-05-19 11:03:19] Already completed stage calculate_delta, skipping...
[INFO] [2022-05-19 11:03:19] Already completed stage parse_diff_and_store, skipping...
[INFO] [2022-05-19 11:03:19] Already completed stage resolve_keys, skipping...
[INFO] [2022-05-19 11:03:19] Already completed stage hold_for_later_1, skipping...
[INFO] [2022-05-19 11:03:19] Already completed stage hold_for_later_2, skipping...
[INFO] [2022-05-19 11:03:19] Already completed stage resolve_missing_parents, skipping...
[INFO] [2022-05-19 11:03:19] Already completed stage rebuild_nodes, skipping...
[INFO] [2022-05-19 11:03:19] Already completed stage resolve_missing_media_owners, skipping...
[INFO] [2022-05-19 11:03:19] Already completed stage sanitize_media_verbatims, skipping...
[INFO] [2022-05-19 11:03:19] Already completed stage queue_downloads, skipping...
[INFO] [2022-05-19 11:03:19] Already completed stage parse_names, skipping...
[INFO] [2022-05-19 11:03:19] Already completed stage denormalize_canonical_names_to_nodes, skipping...
[INFO] [2022-05-19 11:03:19] Already completed stage match_nodes, skipping...
[INFO] [2022-05-19 11:03:19] Already completed stage reindex_search, skipping...
[INFO] [2022-05-19 11:03:19] Already completed stage normalize_units, skipping...
[START] [2022-05-19 11:03:19] calculate_statistics
[INFO] [2022-05-19 11:03:22] Duplicate page_id count: 0
[2022-05-19 11:03:29] (NEAR) DUPLICATE TRAITS FOUND! There are only 558206 (of 580161 total) unique traits.
[STOP] [2022-05-19 13:33:09] calculate_statistics
[STOP] [2022-05-19 13:33:09] logged process, took 8990.12
[START] [2022-05-19 13:40:02] logged process: f73f353df9a1d1d5fe86620ddb421763ac872d1a Removed #check_for_duplicate_* on trait resources
[INFO] [2022-05-19 13:40:02] Already completed stage create_harvest_instance, skipping...
[INFO] [2022-05-19 13:40:02] Already completed stage fetch_files, skipping...
[INFO] [2022-05-19 13:40:02] Already completed stage validate_each_file, skipping...
[INFO] [2022-05-19 13:40:02] Already completed stage convert_to_csv, skipping...
[INFO] [2022-05-19 13:40:02] Already completed stage calculate_delta, skipping...
[INFO] [2022-05-19 13:40:02] Already completed stage parse_diff_and_store, skipping...
[INFO] [2022-05-19 13:40:02] Already completed stage resolve_keys, skipping...
[INFO] [2022-05-19 13:40:02] Already completed stage hold_for_later_1, skipping...
[INFO] [2022-05-19 13:40:02] Already completed stage hold_for_later_2, skipping...
[INFO] [2022-05-19 13:40:02] Already completed stage resolve_missing_parents, skipping...
[INFO] [2022-05-19 13:40:02] Already completed stage rebuild_nodes, skipping...
[INFO] [2022-05-19 13:40:02] Already completed stage resolve_missing_media_owners, skipping...
[INFO] [2022-05-19 13:40:02] Already completed stage sanitize_media_verbatims, skipping...
[INFO] [2022-05-19 13:40:02] Already completed stage queue_downloads, skipping...
[INFO] [2022-05-19 13:40:02] Already completed stage parse_names, skipping...
[INFO] [2022-05-19 13:40:02] Already completed stage denormalize_canonical_names_to_nodes, skipping...
[INFO] [2022-05-19 13:40:02] Already completed stage match_nodes, skipping...
[INFO] [2022-05-19 13:40:02] Already completed stage reindex_search, skipping...
[INFO] [2022-05-19 13:40:02] Already completed stage normalize_units, skipping...
[START] [2022-05-19 13:40:02] calculate_statistics
[INFO] [2022-05-19 13:40:05] Duplicate page_id count: 0
[STOP] [2022-05-19 13:40:06] calculate_statistics
[START] [2022-05-19 13:40:06] complete_harvest_instance
[START] [2022-05-19 13:40:06] overall_tsv_creation
[INFO] [2022-05-19 13:40:06] Processing group of 35537 in 4 batches of 10000
[INFO] [2022-05-19 13:42:54] 162828 Traits (unfiltered)...
[INFO] [2022-05-19 13:42:54] Building Traits map (this can take a while)...
[INFO] [2022-05-19 13:45:59] 10000 traits mapped (10000 meta)...
[INFO] [2022-05-19 13:48:08] 20000 traits mapped (20000 meta)...
[INFO] [2022-05-19 13:50:17] 30000 traits mapped (30000 meta)...
[INFO] [2022-05-19 13:52:26] 40000 traits mapped (40000 meta)...
[INFO] [2022-05-19 13:54:36] 50000 traits mapped (50000 meta)...
[INFO] [2022-05-19 13:56:48] 60000 traits mapped (60000 meta)...
[INFO] [2022-05-19 13:58:55] 70000 traits mapped (76553 meta)...
[INFO] [2022-05-19 14:01:06] 80000 traits mapped (97398 meta)...
[INFO] [2022-05-19 14:03:16] 90000 traits mapped (118003 meta)...
[INFO] [2022-05-19 14:05:26] 100000 traits mapped (138003 meta)...
[INFO] [2022-05-19 14:07:40] 110000 traits mapped (158003 meta)...
[INFO] [2022-05-19 14:09:47] 120000 traits mapped (178003 meta)...
[INFO] [2022-05-19 14:12:00] 130000 traits mapped (192787 meta)...
[INFO] [2022-05-19 14:14:10] 140000 traits mapped (202787 meta)...
[INFO] [2022-05-19 14:16:20] 150000 traits mapped (212787 meta)...
[INFO] [2022-05-19 14:18:30] 160000 traits mapped (222787 meta)...
[INFO] [2022-05-19 14:19:11] Done. 162828 traits mapped (225615 meta).
[INFO] [2022-05-19 14:19:11] Building Associations map (this can take a while)...
[INFO] [2022-05-19 14:19:23] Done. 0 assocs mapped (0 meta).
[INFO] [2022-05-19 14:19:23] Adding 162828 traits...
[INFO] [2022-05-19 14:19:38] 1461 metadata added.
[INFO] [2022-05-19 14:19:38] Adding 0 assocs...
[INFO] [2022-05-19 14:19:38] 0 metadata added.
[INFO] [2022-05-19 14:24:01] 150011 Traits (unfiltered)...
[INFO] [2022-05-19 14:24:01] Building Traits map (this can take a while)...
[INFO] [2022-05-19 14:27:02] 10000 traits mapped (10000 meta)...
[INFO] [2022-05-19 14:29:15] 20000 traits mapped (20000 meta)...
[INFO] [2022-05-19 14:31:23] 30000 traits mapped (30000 meta)...
[INFO] [2022-05-19 14:33:33] 40000 traits mapped (40000 meta)...
[INFO] [2022-05-19 14:35:41] 50000 traits mapped (50000 meta)...
[INFO] [2022-05-19 14:37:53] 60000 traits mapped (60000 meta)...
[INFO] [2022-05-19 14:40:02] 70000 traits mapped (78671 meta)...
[INFO] [2022-05-19 14:42:16] 80000 traits mapped (99301 meta)...
[INFO] [2022-05-19 14:44:24] 90000 traits mapped (119775 meta)...
[INFO] [2022-05-19 14:46:38] 100000 traits mapped (139775 meta)...
[INFO] [2022-05-19 14:48:46] 110000 traits mapped (159775 meta)...
[INFO] [2022-05-19 14:51:00] 120000 traits mapped (173271 meta)...
[INFO] [2022-05-19 14:53:10] 130000 traits mapped (183271 meta)...
[INFO] [2022-05-19 14:55:21] 140000 traits mapped (193271 meta)...
[INFO] [2022-05-19 14:57:34] 150000 traits mapped (203271 meta)...
[INFO] [2022-05-19 14:57:46] Done. 150011 traits mapped (203282 meta).
[INFO] [2022-05-19 14:57:46] Building Associations map (this can take a while)...
[INFO] [2022-05-19 14:57:55] Done. 0 assocs mapped (0 meta).
[INFO] [2022-05-19 14:57:55] Adding 150011 traits...
[INFO] [2022-05-19 14:58:08] 792 metadata added.
[INFO] [2022-05-19 14:58:08] Adding 0 assocs...
[INFO] [2022-05-19 14:58:08] 0 metadata added.
[INFO] [2022-05-19 15:01:03] 162955 Traits (unfiltered)...
[INFO] [2022-05-19 15:01:03] Building Traits map (this can take a while)...
[INFO] [2022-05-19 15:04:02] 10000 traits mapped (10000 meta)...
[INFO] [2022-05-19 15:06:17] 20000 traits mapped (20000 meta)...
[INFO] [2022-05-19 15:08:26] 30000 traits mapped (30000 meta)...
[INFO] [2022-05-19 15:10:36] 40000 traits mapped (40000 meta)...
[INFO] [2022-05-19 15:12:44] 50000 traits mapped (50000 meta)...
[INFO] [2022-05-19 15:14:55] 60000 traits mapped (60000 meta)...
[INFO] [2022-05-19 15:17:03] 70000 traits mapped (76695 meta)...
[INFO] [2022-05-19 15:19:15] 80000 traits mapped (97620 meta)...
[INFO] [2022-05-19 15:21:23] 90000 traits mapped (118229 meta)...
[INFO] [2022-05-19 15:23:35] 100000 traits mapped (138229 meta)...
[INFO] [2022-05-19 15:25:46] 110000 traits mapped (158229 meta)...
[INFO] [2022-05-19 15:27:59] 120000 traits mapped (178229 meta)...
[INFO] [2022-05-19 15:30:12] 130000 traits mapped (193476 meta)...
[INFO] [2022-05-19 15:32:20] 140000 traits mapped (203476 meta)...
[INFO] [2022-05-19 15:34:34] 150000 traits mapped (213476 meta)...
[INFO] [2022-05-19 15:36:43] 160000 traits mapped (223476 meta)...
[INFO] [2022-05-19 15:37:21] Done. 162955 traits mapped (226431 meta).
[INFO] [2022-05-19 15:37:21] Building Associations map (this can take a while)...
[INFO] [2022-05-19 15:37:28] Done. 0 assocs mapped (0 meta).
[INFO] [2022-05-19 15:37:29] Adding 162955 traits...
[INFO] [2022-05-19 15:37:50] 1037 metadata added.
[INFO] [2022-05-19 15:37:50] Adding 0 assocs...
[INFO] [2022-05-19 15:37:50] 0 metadata added.
[INFO] [2022-05-19 15:40:34] 93222 Traits (unfiltered)...
[INFO] [2022-05-19 15:40:34] Building Traits map (this can take a while)...
[INFO] [2022-05-19 15:43:34] 10000 traits mapped (10000 meta)...
[INFO] [2022-05-19 15:45:46] 20000 traits mapped (20000 meta)...
[INFO] [2022-05-19 15:47:53] 30000 traits mapped (30000 meta)...
[INFO] [2022-05-19 15:50:03] 40000 traits mapped (41152 meta)...
[INFO] [2022-05-19 15:52:11] 50000 traits mapped (61850 meta)...
[INFO] [2022-05-19 15:54:22] 60000 traits mapped (81850 meta)...
[INFO] [2022-05-19 15:56:30] 70000 traits mapped (101546 meta)...
[INFO] [2022-05-19 15:58:42] 80000 traits mapped (111546 meta)...
[INFO] [2022-05-19 16:00:50] 90000 traits mapped (121546 meta)...
[INFO] [2022-05-19 16:01:41] Done. 93222 traits mapped (124768 meta).
[INFO] [2022-05-19 16:01:41] Building Associations map (this can take a while)...
[INFO] [2022-05-19 16:01:51] Done. 0 assocs mapped (0 meta).
[INFO] [2022-05-19 16:01:51] Adding 93222 traits...
[INFO] [2022-05-19 16:02:01] 464 metadata added.
[INFO] [2022-05-19 16:02:01] Adding 0 assocs...
[INFO] [2022-05-19 16:02:01] 0 metadata added.
[INFO] [2022-05-19 16:03:12] Average Time: 2062.518
[INFO] [2022-05-19 16:03:12] Total Time: 2h23m7s
[STOP] [2022-05-19 16:03:12] overall_tsv_creation
[INFO] [2022-05-19 16:03:12] Done. Check your files:
[INFO] [2022-05-19 16:03:12] (35537 lines) /app/public/data/usda_plants/publish_nodes.tsv
[INFO] [2022-05-19 16:03:12] (35186 lines) /app/public/data/usda_plants/publish_identifiers.tsv
[INFO] [2022-05-19 16:03:13] (35186 lines) /app/public/data/usda_plants/publish_node_ancestors.tsv
[INFO] [2022-05-19 16:03:13] (35537 lines) /app/public/data/usda_plants/publish_scientific_names.tsv
[INFO] [2022-05-19 16:03:13] (47481 lines) /app/public/data/usda_plants/publish_vernaculars.tsv
[INFO] [2022-05-19 16:03:13] (569017 lines) /app/public/data/usda_plants/publish_traits.tsv
[INFO] [2022-05-19 16:03:13] (3755 lines) /app/public/data/usda_plants/publish_metadata.tsv
[STOP] [2022-05-19 16:03:14] complete_harvest_instance
[START] [2022-05-19 16:03:14] completed
[STOP] [2022-05-19 16:03:14] completed
[STOP] [2022-05-19 16:03:14] logged process, took 8591.87

Latest Process