Harvest for USDA Plants data Created 14 Jul 13:05

Stage: completed
Fetched: 14 Jul 13:05
Validated: 14 Jul 13:06
Deltas Created 14 Jul 13:06
Units Normalized: 14 Jul 15:39
Ancestry Built: 14 Jul 14:25
Nodes Matched: 14 Jul 15:36
Names Parsed: 14 Jul 14:26
New Models Stored: 14 Jul 14:20
Indexed: 14 Jul 15:37
Completed: 14 Jul 16:39
Time to Harvest: 4 minutes

Expected File Format Definitions

Harvesting Log (most recent first)

# Logfile created on 2020-03-26 12:34:55 -0400 by logger.rb/56815
[INFO] [2020-03-26 12:34:55] ## HARVEST: type = -harvest
[START] [2020-03-26 12:34:57] logged process
[START] [2020-03-26 12:34:57] create_harvest_instance
[STOP] [2020-03-26 12:34:59] create_harvest_instance
[START] [2020-03-26 12:34:59] fetch_files
[STOP] [2020-03-26 12:35:00] fetch_files
[START] [2020-03-26 12:35:00] validate_each_file
[STOP] [2020-03-26 12:36:24] validate_each_file
[START] [2020-03-26 12:36:24] convert_to_csv
[CMD] [2020-03-26 12:36:24] /usr/bin/sort /app/public/converted_csv/usda_plants_agents_20489.csv > /app/public/converted_csv/usda_plants_agents_20489.csv_sorted
[CMD] [2020-03-26 12:36:25] /usr/bin/sort /app/public/converted_csv/usda_plants_refs_20490.csv > /app/public/converted_csv/usda_plants_refs_20490.csv_sorted
[CMD] [2020-03-26 12:36:26] /usr/bin/sort /app/public/converted_csv/usda_plants_nodes_20491.csv > /app/public/converted_csv/usda_plants_nodes_20491.csv_sorted
[CMD] [2020-03-26 12:36:27] /usr/bin/sort /app/public/converted_csv/usda_plants_media_20492.csv > /app/public/converted_csv/usda_plants_media_20492.csv_sorted
[CMD] [2020-03-26 12:36:28] /usr/bin/sort /app/public/converted_csv/usda_plants_vernaculars_20493.csv > /app/public/converted_csv/usda_plants_vernaculars_20493.csv_sorted
[CMD] [2020-03-26 12:36:29] /usr/bin/sort /app/public/converted_csv/usda_plants_occurrences_20494.csv > /app/public/converted_csv/usda_plants_occurrences_20494.csv_sorted
[CMD] [2020-03-26 12:36:29] /usr/bin/sort /app/public/converted_csv/usda_plants_measurements_20495.csv > /app/public/converted_csv/usda_plants_measurements_20495.csv_sorted
[STOP] [2020-03-26 12:36:31] convert_to_csv
[START] [2020-03-26 12:36:31] calculate_delta
[CMD] [2020-03-26 12:36:31] echo "0a" > /app/public/diff/usda_plants_agents_20489.diff
[CMD] [2020-03-26 12:36:32] tail -n +1 /app/public/converted_csv/usda_plants_agents_20489.csv >> /app/public/diff/usda_plants_agents_20489.diff
[CMD] [2020-03-26 12:36:32] echo "." >> /app/public/diff/usda_plants_agents_20489.diff
[CMD] [2020-03-26 12:36:33] echo "0a" > /app/public/diff/usda_plants_refs_20490.diff
[CMD] [2020-03-26 12:36:34] tail -n +1 /app/public/converted_csv/usda_plants_refs_20490.csv >> /app/public/diff/usda_plants_refs_20490.diff
[CMD] [2020-03-26 12:36:35] echo "." >> /app/public/diff/usda_plants_refs_20490.diff
[CMD] [2020-03-26 12:36:36] echo "0a" > /app/public/diff/usda_plants_nodes_20491.diff
[CMD] [2020-03-26 12:36:36] tail -n +1 /app/public/converted_csv/usda_plants_nodes_20491.csv >> /app/public/diff/usda_plants_nodes_20491.diff
[CMD] [2020-03-26 12:36:37] echo "." >> /app/public/diff/usda_plants_nodes_20491.diff
[CMD] [2020-03-26 12:36:38] echo "0a" > /app/public/diff/usda_plants_media_20492.diff
[CMD] [2020-03-26 12:36:39] tail -n +1 /app/public/converted_csv/usda_plants_media_20492.csv >> /app/public/diff/usda_plants_media_20492.diff
[CMD] [2020-03-26 12:36:40] echo "." >> /app/public/diff/usda_plants_media_20492.diff
[CMD] [2020-03-26 12:36:40] echo "0a" > /app/public/diff/usda_plants_vernaculars_20493.diff
[CMD] [2020-03-26 12:36:41] tail -n +1 /app/public/converted_csv/usda_plants_vernaculars_20493.csv >> /app/public/diff/usda_plants_vernaculars_20493.diff
[CMD] [2020-03-26 12:36:42] echo "." >> /app/public/diff/usda_plants_vernaculars_20493.diff
[CMD] [2020-03-26 12:36:43] echo "0a" > /app/public/diff/usda_plants_occurrences_20494.diff
[CMD] [2020-03-26 12:36:44] tail -n +1 /app/public/converted_csv/usda_plants_occurrences_20494.csv >> /app/public/diff/usda_plants_occurrences_20494.diff
[CMD] [2020-03-26 12:36:45] echo "." >> /app/public/diff/usda_plants_occurrences_20494.diff
[CMD] [2020-03-26 12:36:45] echo "0a" > /app/public/diff/usda_plants_measurements_20495.diff
[CMD] [2020-03-26 12:36:46] tail -n +1 /app/public/converted_csv/usda_plants_measurements_20495.csv >> /app/public/diff/usda_plants_measurements_20495.diff
[CMD] [2020-03-26 12:36:47] echo "." >> /app/public/diff/usda_plants_measurements_20495.diff
[STOP] [2020-03-26 12:36:48] calculate_delta
[START] [2020-03-26 12:36:48] parse_diff_and_store
[INFO] [2020-03-26 12:36:49] Loading agents diff file into memory (true lines)...
[INFO] [2020-03-26 12:36:50] Loading refs diff file into memory (true lines)...
[INFO] [2020-03-26 12:36:50] Loading nodes diff file into memory (true lines)...
[INFO] [2020-03-26 12:37:10] Loading media diff file into memory (true lines)...
[INFO] [2020-03-26 12:37:11] Loading vernaculars diff file into memory (true lines)...
[INFO] [2020-03-26 12:38:06] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-03-26 12:40:08] Loading measurements diff file into memory (true lines)...
[INFO] [2020-03-26 13:44:30] Storing 1 Attributions
[INFO] [2020-03-26 13:44:30] Processing group of 1 in 1 groups of 1000
[INFO] [2020-03-26 13:44:30] Average Time: 0.0
[INFO] [2020-03-26 13:44:30] Total Time: 1s
[INFO] [2020-03-26 13:44:30] Storing 2 References
[INFO] [2020-03-26 13:44:30] Processing group of 2 in 1 groups of 1000
[INFO] [2020-03-26 13:44:30] Average Time: 0.0
[INFO] [2020-03-26 13:44:30] Total Time: 1s
[INFO] [2020-03-26 13:44:30] Storing 35956 ScientificNames
[INFO] [2020-03-26 13:44:30] Processing group of 35956 in 36 groups of 1000
[INFO] [2020-03-26 13:44:59] Average Time: 0.804
[INFO] [2020-03-26 13:44:59] Total Time: 30s
[INFO] [2020-03-26 13:44:59] last 3 / first 3: 11.04
[INFO] [2020-03-26 13:44:59] Std.Dev: 2.2509997778764883; Max: 13.92
[INFO] [2020-03-26 13:44:59] Storing 35956 Nodes
[INFO] [2020-03-26 13:44:59] Processing group of 35956 in 36 groups of 1000
[INFO] [2020-03-26 13:45:25] Average Time: 0.731
[INFO] [2020-03-26 13:45:25] Total Time: 27s
[INFO] [2020-03-26 13:45:25] last 3 / first 3: 0.06
[INFO] [2020-03-26 13:45:25] Std.Dev: 2.24610774452162; Max: 13.82
[INFO] [2020-03-26 13:45:25] Storing 35605 Identifiers
[INFO] [2020-03-26 13:45:25] Processing group of 35605 in 36 groups of 1000
[INFO] [2020-03-26 13:45:29] Average Time: 0.099
[INFO] [2020-03-26 13:45:29] Total Time: 4s
[INFO] [2020-03-26 13:45:29] last 3 / first 3: 0.67
[INFO] [2020-03-26 13:45:29] Std.Dev: 0.06324555320336758; Max: 0.44
[INFO] [2020-03-26 13:45:29] Storing 2 BibliographicCitations
[INFO] [2020-03-26 13:45:29] Processing group of 2 in 1 groups of 1000
[INFO] [2020-03-26 13:45:29] Average Time: 0.02
[INFO] [2020-03-26 13:45:29] Total Time: 1s
[INFO] [2020-03-26 13:45:29] Storing 2 ArticlesSections
[INFO] [2020-03-26 13:45:29] Processing group of 2 in 1 groups of 1000
[INFO] [2020-03-26 13:45:29] Average Time: 0.01
[INFO] [2020-03-26 13:45:29] Total Time: 1s
[INFO] [2020-03-26 13:45:29] Storing 2 Articles
[INFO] [2020-03-26 13:45:29] Processing group of 2 in 1 groups of 1000
[INFO] [2020-03-26 13:45:29] Average Time: 0.01
[INFO] [2020-03-26 13:45:29] Total Time: 1s
[INFO] [2020-03-26 13:45:29] Storing 1 ContentAttributions
[INFO] [2020-03-26 13:45:29] Processing group of 1 in 1 groups of 1000
[INFO] [2020-03-26 13:45:29] Average Time: 0.02
[INFO] [2020-03-26 13:45:29] Total Time: 1s
[INFO] [2020-03-26 13:45:29] Storing 3 Media
[INFO] [2020-03-26 13:45:29] Processing group of 3 in 1 groups of 1000
[INFO] [2020-03-26 13:45:29] Average Time: 0.01
[INFO] [2020-03-26 13:45:29] Total Time: 1s
[INFO] [2020-03-26 13:45:29] Storing 305965 Vernaculars
[INFO] [2020-03-26 13:45:29] Processing group of 305965 in 306 groups of 1000
[INFO] [2020-03-26 13:47:30] Average Time: 0.389
[INFO] [2020-03-26 13:47:30] Total Time: 2m1s
[INFO] [2020-03-26 13:47:30] last 3 / first 3: 1.0
[INFO] [2020-03-26 13:47:30] Std.Dev: 1.5959323293924463; Max: 14.37
[INFO] [2020-03-26 13:47:30] Storing 656907 Occurrences
[INFO] [2020-03-26 13:47:30] Processing group of 656907 in 657 groups of 1000
[INFO] [2020-03-26 13:51:24] Average Time: 0.328
[INFO] [2020-03-26 13:51:24] Total Time: 3m55s
[INFO] [2020-03-26 13:51:24] last 3 / first 3: 0.44
[INFO] [2020-03-26 13:51:24] Std.Dev: 1.7357995275952807; Max: 15.56
[INFO] [2020-03-26 13:51:24] Storing 602217 Traits
[INFO] [2020-03-26 13:51:24] Processing group of 602217 in 603 groups of 1000
[INFO] [2020-03-26 13:59:22] Average Time: 0.788
[INFO] [2020-03-26 13:59:22] Total Time: 7m58s
[INFO] [2020-03-26 13:59:22] last 3 / first 3: 0.66
[INFO] [2020-03-26 13:59:22] Std.Dev: 2.7624264696096437; Max: 17.94
[INFO] [2020-03-26 13:59:22] Storing 1489696 MetaTraits
[INFO] [2020-03-26 13:59:22] Processing group of 1489696 in 1490 groups of 1000
[INFO] [2020-03-26 14:09:32] Average Time: 0.405
[INFO] [2020-03-26 14:09:32] Total Time: 10m11s
[INFO] [2020-03-26 14:09:32] last 3 / first 3: 2.14
[INFO] [2020-03-26 14:09:32] Std.Dev: 2.2501111083677623; Max: 20.35
[STOP] [2020-03-26 14:09:32] parse_diff_and_store
[START] [2020-03-26 14:09:32] resolve_keys
[INFO] [2020-03-26 14:10:14] Occurrences to nodes (through scientific_names)...
[INFO] [2020-03-26 14:10:29] traits to occurrences...
[INFO] [2020-03-26 14:11:19] traits to nodes (through occurrences)...
[INFO] [2020-03-26 14:11:32] Traits to sex term...
[INFO] [2020-03-26 14:11:44] Traits to lifestage term...
[INFO] [2020-03-26 14:11:57] MetaTraits to traits...
[INFO] [2020-03-26 14:13:31] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-03-26 14:13:31] Assocs to occurrences...
[INFO] [2020-03-26 14:13:31] Assocs to nodes...
[INFO] [2020-03-26 14:13:31] Assoc to sex term...
[INFO] [2020-03-26 14:13:31] Assoc to lifestage term...
[STOP] [2020-03-26 14:13:31] resolve_keys
[START] [2020-03-26 14:13:31] hold_for_later_1
[STOP] [2020-03-26 14:13:31] hold_for_later_1
[START] [2020-03-26 14:13:31] hold_for_later_2
[STOP] [2020-03-26 14:13:31] hold_for_later_2
[START] [2020-03-26 14:13:31] resolve_missing_parents
[STOP] [2020-03-26 14:13:35] resolve_missing_parents
[START] [2020-03-26 14:13:35] rebuild_nodes
[START] [2020-03-26 14:13:35] Flattener#flatten
[START] [2020-03-26 14:13:35] Flattener#study_resource
[START] [2020-03-26 14:13:35] Flattener#build_ancestry
[STOP] [2020-03-26 14:14:08] Flattener#build_ancestry
[INFO] [2020-03-26 14:14:08] 35956 ancestry keys
[START] [2020-03-26 14:14:08] build_node_ancestors
[INFO] [2020-03-26 14:14:08] old ancestors deleted.
[STOP] [2020-03-26 14:14:10] build_node_ancestors
[START] [2020-03-26 14:14:13] Flattener#propagate_ancestor_ids
[STOP] [2020-03-26 14:14:14] Flattener#propagate_ancestor_ids
[STOP] [2020-03-26 14:14:14] Flattener#flatten
[STOP] [2020-03-26 14:14:14] rebuild_nodes
[START] [2020-03-26 14:14:14] resolve_missing_media_owners
[STOP] [2020-03-26 14:14:14] resolve_missing_media_owners
[START] [2020-03-26 14:14:14] sanitize_media_verbatims
[STOP] [2020-03-26 14:14:14] sanitize_media_verbatims
[START] [2020-03-26 14:14:14] queue_downloads
[STOP] [2020-03-26 14:14:14] queue_downloads
[START] [2020-03-26 14:14:14] parse_names
[WARN] [2020-03-26 14:14:14] I see 35956 names which still need to be parsed.
[STOP] [2020-03-26 14:14:44] parse_names
[START] [2020-03-26 14:14:44] denormalize_canonical_names_to_nodes
[STOP] [2020-03-26 14:14:45] denormalize_canonical_names_to_nodes
[START] [2020-03-26 14:14:45] match_nodes
[START] [2020-03-26 14:14:45] map_all_nodes_to_pages
[STOP] [2020-03-26 15:12:46] map_all_nodes_to_pages
[INFO] [2020-03-26 15:12:46] 2797 Unmatched nodes (of 35956)! That's too many to output. First 10: Abelmoschus (#67516645); Abutilon sandwicense (#67516700); Abutilon (#67516708); Alcea (#67517511); Allosidastrum (#67517649); Allowissadula (#67517653); Alcea pallida (#67517716); Althaea (#67517804); Anisodontea (#67518294); Anoda (#67518364)
[START] [2020-03-26 15:12:46] update_nodes
[STOP] [2020-03-26 15:12:59] update_nodes
[STOP] [2020-03-26 15:12:59] match_nodes
[START] [2020-03-26 15:12:59] reindex_search
[STOP] [2020-03-26 15:14:10] reindex_search
[START] [2020-03-26 15:14:10] normalize_units
[STOP] [2020-03-26 15:15:44] normalize_units
[START] [2020-03-26 15:15:44] calculate_statistics
[STOP] [2020-03-26 15:15:44] calculate_statistics
[START] [2020-03-26 15:15:44] complete_harvest_instance
[START] [2020-03-26 15:15:44] overall_tsv_creation
[INFO] [2020-03-26 15:15:44] Processing group of 35956 in 4 batches of 10000
[INFO] [2020-03-26 15:18:13] 168433 Traits (unfiltered)...
[INFO] [2020-03-26 15:18:27] 168433 Traits (filtered)...
[INFO] [2020-03-26 15:18:27] 0 Associations (filtered)...
[INFO] [2020-03-26 15:27:52] 417987 metadata added.
[INFO] [2020-03-26 15:27:52] 0 metadata added.
[INFO] [2020-03-26 15:30:31] 152651 Traits (unfiltered)...
[INFO] [2020-03-26 15:30:45] 152651 Traits (filtered)...
[INFO] [2020-03-26 15:30:45] 0 Associations (filtered)...
[INFO] [2020-03-26 15:40:09] 372736 metadata added.
[INFO] [2020-03-26 15:40:09] 0 metadata added.
[INFO] [2020-03-26 15:42:41] 164723 Traits (unfiltered)...
[INFO] [2020-03-26 15:42:55] 164723 Traits (filtered)...
[INFO] [2020-03-26 15:42:55] 0 Associations (filtered)...
[INFO] [2020-03-26 15:53:02] 409177 metadata added.
[INFO] [2020-03-26 15:53:02] 0 metadata added.
[INFO] [2020-03-26 15:55:01] 105831 Traits (unfiltered)...
[INFO] [2020-03-26 15:55:15] 105831 Traits (filtered)...
[INFO] [2020-03-26 15:55:15] 0 Associations (filtered)...
[INFO] [2020-03-26 16:01:42] 257300 metadata added.
[INFO] [2020-03-26 16:01:42] 0 metadata added.
[INFO] [2020-03-26 16:01:42] Average Time: 633.75
[INFO] [2020-03-26 16:01:42] Total Time: 45m58s
[STOP] [2020-03-26 16:01:42] overall_tsv_creation
[INFO] [2020-03-26 16:01:42] Done. Check your files:
[INFO] [2020-03-26 16:01:43] (35956 lines) /app/public/data/usda_plants/publish_nodes.tsv
[INFO] [2020-03-26 16:01:44] (35605 lines) /app/public/data/usda_plants/publish_identifiers.tsv
[INFO] [2020-03-26 16:01:44] (35605 lines) /app/public/data/usda_plants/publish_node_ancestors.tsv
[INFO] [2020-03-26 16:01:45] (35956 lines) /app/public/data/usda_plants/publish_scientific_names.tsv
[INFO] [2020-03-26 16:01:46] (305965 lines) /app/public/data/usda_plants/publish_vernaculars.tsv
[INFO] [2020-03-26 16:01:47] (591639 lines) /app/public/data/usda_plants/publish_traits.tsv
[INFO] [2020-03-26 16:01:48] (1457201 lines) /app/public/data/usda_plants/publish_metadata.tsv
[STOP] [2020-03-26 16:01:48] complete_harvest_instance
[START] [2020-03-26 16:01:48] completed
[STOP] [2020-03-26 16:01:48] completed
[STOP] [2020-03-26 16:01:48] logged process, took 12410.99
[INFO] [2020-07-14 12:57:19] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-07-14 12:57:24] ## remove_type: ScientificName
[INFO] [2020-07-14 12:57:24] ++ Calling delete_all on 35956 instances...
[INFO] [2020-07-14 12:57:28] [12:57:28.219] Removed 35956 Scientificnames
[INFO] [2020-07-14 12:57:28] ## remove_type: Vernacular
[INFO] [2020-07-14 12:57:28] ++ Batch removal of 305965 instances...
[INFO] [2020-07-14 12:57:28] [12:57:28.297] Batch 0...
[INFO] [2020-07-14 12:57:29] [12:57:29.147] Batch 1...
[INFO] [2020-07-14 12:57:30] [12:57:30.064] Batch 2...
[INFO] [2020-07-14 12:57:32] [12:57:32.305] Batch 3...
[INFO] [2020-07-14 12:57:34] [12:57:34.169] Batch 4...
[INFO] [2020-07-14 12:57:35] [12:57:35.766] Batch 5...
[INFO] [2020-07-14 12:57:37] [12:57:37.273] Batch 6...
[INFO] [2020-07-14 12:57:38] [12:57:38.094] Batch 7...
[INFO] [2020-07-14 12:57:39] [12:57:39.215] Batch 8...
[INFO] [2020-07-14 12:57:39] [12:57:39.823] Batch 9...
[INFO] [2020-07-14 12:57:40] [12:57:40.425] Batch 10...
[INFO] [2020-07-14 12:57:41] [12:57:41.512] Batch 11...
[INFO] [2020-07-14 12:57:42] [12:57:42.137] Batch 12...
[INFO] [2020-07-14 12:57:42] [12:57:42.739] Batch 13...
[INFO] [2020-07-14 12:57:43] [12:57:43.333] Batch 14...
[INFO] [2020-07-14 12:57:43] [12:57:43.924] Batch 15...
[INFO] [2020-07-14 12:57:44] [12:57:44.536] Batch 16...
[INFO] [2020-07-14 12:57:45] [12:57:45.122] Batch 17...
[INFO] [2020-07-14 12:57:45] [12:57:45.708] Batch 18...
[INFO] [2020-07-14 12:57:46] [12:57:46.297] Batch 19...
[INFO] [2020-07-14 12:57:46] [12:57:46.880] Batch 20...
[INFO] [2020-07-14 12:57:47] [12:57:47.463] Batch 21...
[INFO] [2020-07-14 12:57:48] [12:57:48.039] Batch 22...
[INFO] [2020-07-14 12:57:48] [12:57:48.618] Batch 23...
[INFO] [2020-07-14 12:57:49] [12:57:49.192] Batch 24...
[INFO] [2020-07-14 12:57:49] [12:57:49.768] Batch 25...
[INFO] [2020-07-14 12:57:50] [12:57:50.340] Batch 26...
[INFO] [2020-07-14 12:57:50] [12:57:50.911] Batch 27...
[INFO] [2020-07-14 12:57:51] [12:57:51.485] Batch 28...
[INFO] [2020-07-14 12:57:52] [12:57:52.047] Batch 29...
[INFO] [2020-07-14 12:57:52] [12:57:52.608] Batch 30...
[INFO] [2020-07-14 12:57:53] [12:57:53.144] Removed 305965 Vernaculars
[INFO] [2020-07-14 12:57:53] ## remove_type: Article
[INFO] [2020-07-14 12:57:53] ++ Calling delete_all on 2 instances...
[INFO] [2020-07-14 12:57:53] [12:57:53.191] Removed 2 Articles
[INFO] [2020-07-14 12:57:53] ## remove_type: Medium
[INFO] [2020-07-14 12:57:53] ++ Calling delete_all on 3 instances...
[INFO] [2020-07-14 12:57:53] [12:57:53.243] Removed 3 Media
[INFO] [2020-07-14 12:57:53] ## remove_type: Trait
[INFO] [2020-07-14 12:57:53] ++ Batch removal of 602217 instances...
[INFO] [2020-07-14 12:57:53] [12:57:53.344] Batch 0...
[INFO] [2020-07-14 12:57:55] [12:57:55.363] Batch 1...
[INFO] [2020-07-14 12:58:01] [12:58:01.558] Batch 2...
[INFO] [2020-07-14 12:58:06] [12:58:06.000] Batch 3...
[INFO] [2020-07-14 12:58:10] [12:58:10.727] Batch 4...
[INFO] [2020-07-14 12:58:15] [12:58:15.292] Batch 5...
[INFO] [2020-07-14 12:58:20] [12:58:20.474] Batch 6...
[INFO] [2020-07-14 12:58:25] [12:58:25.771] Batch 7...
[INFO] [2020-07-14 12:58:30] [12:58:30.476] Batch 8...
[INFO] [2020-07-14 12:58:34] [12:58:34.868] Batch 9...
[INFO] [2020-07-14 12:58:39] [12:58:39.833] Batch 10...
[INFO] [2020-07-14 12:58:44] [12:58:44.117] Batch 11...
[INFO] [2020-07-14 12:58:49] [12:58:49.096] Batch 12...
[INFO] [2020-07-14 12:58:53] [12:58:53.926] Batch 13...
[INFO] [2020-07-14 12:58:58] [12:58:58.534] Batch 14...
[INFO] [2020-07-14 12:59:03] [12:59:03.305] Batch 15...
[INFO] [2020-07-14 12:59:08] [12:59:08.058] Batch 16...
[INFO] [2020-07-14 12:59:12] [12:59:12.612] Batch 17...
[INFO] [2020-07-14 12:59:17] [12:59:17.309] Batch 18...
[INFO] [2020-07-14 12:59:22] [12:59:22.624] Batch 19...
[INFO] [2020-07-14 12:59:26] [12:59:26.951] Batch 20...
[INFO] [2020-07-14 12:59:31] [12:59:31.351] Batch 21...
[INFO] [2020-07-14 12:59:36] [12:59:36.050] Batch 22...
[INFO] [2020-07-14 12:59:41] [12:59:41.069] Batch 23...
[INFO] [2020-07-14 12:59:45] [12:59:45.955] Batch 24...
[INFO] [2020-07-14 12:59:50] [12:59:50.154] Batch 25...
[INFO] [2020-07-14 12:59:55] [12:59:55.142] Batch 26...
[INFO] [2020-07-14 12:59:59] [12:59:59.617] Batch 27...
[INFO] [2020-07-14 13:00:04] [13:00:04.321] Batch 28...
[INFO] [2020-07-14 13:00:08] [13:00:08.938] Batch 29...
[INFO] [2020-07-14 13:00:13] [13:00:13.712] Batch 30...
[INFO] [2020-07-14 13:00:18] [13:00:18.547] Batch 31...
[INFO] [2020-07-14 13:00:23] [13:00:23.474] Batch 32...
[INFO] [2020-07-14 13:00:28] [13:00:28.783] Batch 33...
[INFO] [2020-07-14 13:00:33] [13:00:33.799] Batch 34...
[INFO] [2020-07-14 13:00:38] [13:00:38.942] Batch 35...
[INFO] [2020-07-14 13:00:41] [13:00:41.711] Batch 36...
[INFO] [2020-07-14 13:00:49] [13:00:49.052] Batch 37...
[INFO] [2020-07-14 13:00:52] [13:00:52.265] Batch 38...
[INFO] [2020-07-14 13:00:54] [13:00:54.641] Batch 39...
[INFO] [2020-07-14 13:00:56] [13:00:56.790] Batch 40...
[INFO] [2020-07-14 13:00:59] [13:00:59.155] Batch 41...
[INFO] [2020-07-14 13:01:01] [13:01:01.301] Batch 42...
[INFO] [2020-07-14 13:01:03] [13:01:03.489] Batch 43...
[INFO] [2020-07-14 13:01:05] [13:01:05.456] Batch 44...
[INFO] [2020-07-14 13:01:07] [13:01:07.552] Batch 45...
[INFO] [2020-07-14 13:01:09] [13:01:09.551] Batch 46...
[INFO] [2020-07-14 13:01:11] [13:01:11.592] Batch 47...
[INFO] [2020-07-14 13:01:13] [13:01:13.607] Batch 48...
[INFO] [2020-07-14 13:01:15] [13:01:15.609] Batch 49...
[INFO] [2020-07-14 13:01:17] [13:01:17.535] Batch 50...
[INFO] [2020-07-14 13:01:19] [13:01:19.506] Batch 51...
[INFO] [2020-07-14 13:01:21] [13:01:21.471] Batch 52...
[INFO] [2020-07-14 13:01:23] [13:01:23.443] Batch 53...
[INFO] [2020-07-14 13:01:25] [13:01:25.526] Batch 54...
[INFO] [2020-07-14 13:01:27] [13:01:27.489] Batch 55...
[INFO] [2020-07-14 13:01:29] [13:01:29.463] Batch 56...
[INFO] [2020-07-14 13:01:31] [13:01:31.468] Batch 57...
[INFO] [2020-07-14 13:01:33] [13:01:33.422] Batch 58...
[INFO] [2020-07-14 13:01:35] [13:01:35.511] Batch 59...
[INFO] [2020-07-14 13:01:37] [13:01:37.470] Batch 60...
[INFO] [2020-07-14 13:01:38] [13:01:38.018] Removed 602217 Traits
[INFO] [2020-07-14 13:01:38] ## remove_type: MetaTrait
[INFO] [2020-07-14 13:01:38] ++ Batch removal of 1489696 instances...
[INFO] [2020-07-14 13:01:38] [13:01:38.761] Batch 0...
[INFO] [2020-07-14 13:01:39] [13:01:39.907] Batch 1...
[INFO] [2020-07-14 13:01:40] [13:01:40.999] Batch 2...
[INFO] [2020-07-14 13:01:42] [13:01:42.108] Batch 3...
[INFO] [2020-07-14 13:01:43] [13:01:43.177] Batch 4...
[INFO] [2020-07-14 13:01:44] [13:01:44.266] Batch 5...
[INFO] [2020-07-14 13:01:45] [13:01:45.345] Batch 6...
[INFO] [2020-07-14 13:01:46] [13:01:46.404] Batch 7...
[INFO] [2020-07-14 13:01:47] [13:01:47.482] Batch 8...
[INFO] [2020-07-14 13:01:48] [13:01:48.550] Batch 9...
[INFO] [2020-07-14 13:01:49] [13:01:49.637] Batch 10...
[INFO] [2020-07-14 13:01:50] [13:01:50.694] Batch 11...
[INFO] [2020-07-14 13:01:51] [13:01:51.758] Batch 12...
[INFO] [2020-07-14 13:01:52] [13:01:52.822] Batch 13...
[INFO] [2020-07-14 13:01:53] [13:01:53.860] Batch 14...
[INFO] [2020-07-14 13:01:54] [13:01:54.937] Batch 15...
[INFO] [2020-07-14 13:01:56] [13:01:56.012] Batch 16...
[INFO] [2020-07-14 13:01:57] [13:01:57.068] Batch 17...
[INFO] [2020-07-14 13:01:58] [13:01:58.127] Batch 18...
[INFO] [2020-07-14 13:01:59] [13:01:59.178] Batch 19...
[INFO] [2020-07-14 13:02:00] [13:02:00.228] Batch 20...
[INFO] [2020-07-14 13:02:01] [13:02:01.289] Batch 21...
[INFO] [2020-07-14 13:02:02] [13:02:02.346] Batch 22...
[INFO] [2020-07-14 13:02:03] [13:02:03.383] Batch 23...
[INFO] [2020-07-14 13:02:04] [13:02:04.431] Batch 24...
[INFO] [2020-07-14 13:02:05] [13:02:05.484] Batch 25...
[INFO] [2020-07-14 13:02:06] [13:02:06.511] Batch 26...
[INFO] [2020-07-14 13:02:07] [13:02:07.596] Batch 27...
[INFO] [2020-07-14 13:02:08] [13:02:08.650] Batch 28...
[INFO] [2020-07-14 13:02:09] [13:02:09.696] Batch 29...
[INFO] [2020-07-14 13:02:10] [13:02:10.743] Batch 30...
[INFO] [2020-07-14 13:02:11] [13:02:11.787] Batch 31...
[INFO] [2020-07-14 13:02:12] [13:02:12.834] Batch 32...
[INFO] [2020-07-14 13:02:13] [13:02:13.888] Batch 33...
[INFO] [2020-07-14 13:02:14] [13:02:14.907] Batch 34...
[INFO] [2020-07-14 13:02:15] [13:02:15.946] Batch 35...
[INFO] [2020-07-14 13:02:16] [13:02:16.997] Batch 36...
[INFO] [2020-07-14 13:02:17] [13:02:17.998] Batch 37...
[INFO] [2020-07-14 13:02:19] [13:02:19.025] Batch 38...
[INFO] [2020-07-14 13:02:20] [13:02:20.026] Batch 39...
[INFO] [2020-07-14 13:02:21] [13:02:21.024] Batch 40...
[INFO] [2020-07-14 13:02:22] [13:02:22.021] Batch 41...
[INFO] [2020-07-14 13:02:23] [13:02:23.104] Batch 42...
[INFO] [2020-07-14 13:02:24] [13:02:24.144] Batch 43...
[INFO] [2020-07-14 13:02:25] [13:02:25.150] Batch 44...
[INFO] [2020-07-14 13:02:26] [13:02:26.190] Batch 45...
[INFO] [2020-07-14 13:02:27] [13:02:27.187] Batch 46...
[INFO] [2020-07-14 13:02:28] [13:02:28.164] Batch 47...
[INFO] [2020-07-14 13:02:29] [13:02:29.215] Batch 48...
[INFO] [2020-07-14 13:02:30] [13:02:30.215] Batch 49...
[INFO] [2020-07-14 13:02:31] [13:02:31.220] Batch 50...
[INFO] [2020-07-14 13:02:32] [13:02:32.201] Batch 51...
[INFO] [2020-07-14 13:02:33] [13:02:33.201] Batch 52...
[INFO] [2020-07-14 13:02:34] [13:02:34.181] Batch 53...
[INFO] [2020-07-14 13:02:35] [13:02:35.189] Batch 54...
[INFO] [2020-07-14 13:02:36] [13:02:36.185] Batch 55...
[INFO] [2020-07-14 13:02:37] [13:02:37.190] Batch 56...
[INFO] [2020-07-14 13:02:38] [13:02:38.198] Batch 57...
[INFO] [2020-07-14 13:02:39] [13:02:39.175] Batch 58...
[INFO] [2020-07-14 13:02:40] [13:02:40.142] Batch 59...
[INFO] [2020-07-14 13:02:41] [13:02:41.127] Batch 60...
[INFO] [2020-07-14 13:02:42] [13:02:42.087] Batch 61...
[INFO] [2020-07-14 13:02:43] [13:02:43.008] Batch 62...
[INFO] [2020-07-14 13:02:43] [13:02:43.966] Batch 63...
[INFO] [2020-07-14 13:02:44] [13:02:44.924] Batch 64...
[INFO] [2020-07-14 13:02:45] [13:02:45.876] Batch 65...
[INFO] [2020-07-14 13:02:46] [13:02:46.837] Batch 66...
[INFO] [2020-07-14 13:02:47] [13:02:47.793] Batch 67...
[INFO] [2020-07-14 13:02:48] [13:02:48.778] Batch 68...
[INFO] [2020-07-14 13:02:49] [13:02:49.771] Batch 69...
[INFO] [2020-07-14 13:02:50] [13:02:50.744] Batch 70...
[INFO] [2020-07-14 13:02:51] [13:02:51.727] Batch 71...
[INFO] [2020-07-14 13:02:52] [13:02:52.693] Batch 72...
[INFO] [2020-07-14 13:02:53] [13:02:53.641] Batch 73...
[INFO] [2020-07-14 13:02:54] [13:02:54.589] Batch 74...
[INFO] [2020-07-14 13:02:55] [13:02:55.538] Batch 75...
[INFO] [2020-07-14 13:02:56] [13:02:56.479] Batch 76...
[INFO] [2020-07-14 13:02:57] [13:02:57.455] Batch 77...
[INFO] [2020-07-14 13:02:58] [13:02:58.430] Batch 78...
[INFO] [2020-07-14 13:02:59] [13:02:59.403] Batch 79...
[INFO] [2020-07-14 13:03:00] [13:03:00.410] Batch 80...
[INFO] [2020-07-14 13:03:01] [13:03:01.390] Batch 81...
[INFO] [2020-07-14 13:03:02] [13:03:02.348] Batch 82...
[INFO] [2020-07-14 13:03:03] [13:03:03.308] Batch 83...
[INFO] [2020-07-14 13:03:04] [13:03:04.251] Batch 84...
[INFO] [2020-07-14 13:03:05] [13:03:05.189] Batch 85...
[INFO] [2020-07-14 13:03:06] [13:03:06.138] Batch 86...
[INFO] [2020-07-14 13:03:07] [13:03:07.040] Batch 87...
[INFO] [2020-07-14 13:03:07] [13:03:07.971] Batch 88...
[INFO] [2020-07-14 13:03:08] [13:03:08.911] Batch 89...
[INFO] [2020-07-14 13:03:09] [13:03:09.857] Batch 90...
[INFO] [2020-07-14 13:03:10] [13:03:10.759] Batch 91...
[INFO] [2020-07-14 13:03:11] [13:03:11.724] Batch 92...
[INFO] [2020-07-14 13:03:12] [13:03:12.675] Batch 93...
[INFO] [2020-07-14 13:03:13] [13:03:13.639] Batch 94...
[INFO] [2020-07-14 13:03:14] [13:03:14.566] Batch 95...
[INFO] [2020-07-14 13:03:15] [13:03:15.495] Batch 96...
[INFO] [2020-07-14 13:03:16] [13:03:16.431] Batch 97...
[INFO] [2020-07-14 13:03:17] [13:03:17.391] Batch 98...
[INFO] [2020-07-14 13:03:18] [13:03:18.297] Batch 99...
[INFO] [2020-07-14 13:03:19] [13:03:19.229] Batch 100...
[INFO] [2020-07-14 13:03:20] [13:03:20.179] Batch 101...
[INFO] [2020-07-14 13:03:21] [13:03:21.066] Batch 102...
[INFO] [2020-07-14 13:03:21] [13:03:21.968] Batch 103...
[INFO] [2020-07-14 13:03:22] [13:03:22.862] Batch 104...
[INFO] [2020-07-14 13:03:23] [13:03:23.813] Batch 105...
[INFO] [2020-07-14 13:03:24] [13:03:24.722] Batch 106...
[INFO] [2020-07-14 13:03:25] [13:03:25.641] Batch 107...
[INFO] [2020-07-14 13:03:26] [13:03:26.559] Batch 108...
[INFO] [2020-07-14 13:03:27] [13:03:27.474] Batch 109...
[INFO] [2020-07-14 13:03:28] [13:03:28.383] Batch 110...
[INFO] [2020-07-14 13:03:29] [13:03:29.333] Batch 111...
[INFO] [2020-07-14 13:03:30] [13:03:30.199] Batch 112...
[INFO] [2020-07-14 13:03:31] [13:03:31.076] Batch 113...
[INFO] [2020-07-14 13:03:31] [13:03:31.952] Batch 114...
[INFO] [2020-07-14 13:03:32] [13:03:32.829] Batch 115...
[INFO] [2020-07-14 13:03:33] [13:03:33.732] Batch 116...
[INFO] [2020-07-14 13:03:34] [13:03:34.599] Batch 117...
[INFO] [2020-07-14 13:03:35] [13:03:35.487] Batch 118...
[INFO] [2020-07-14 13:03:36] [13:03:36.350] Batch 119...
[INFO] [2020-07-14 13:03:37] [13:03:37.258] Batch 120...
[INFO] [2020-07-14 13:03:38] [13:03:38.140] Batch 121...
[INFO] [2020-07-14 13:03:39] [13:03:39.007] Batch 122...
[INFO] [2020-07-14 13:03:39] [13:03:39.908] Batch 123...
[INFO] [2020-07-14 13:03:40] [13:03:40.795] Batch 124...
[INFO] [2020-07-14 13:03:41] [13:03:41.669] Batch 125...
[INFO] [2020-07-14 13:03:42] [13:03:42.534] Batch 126...
[INFO] [2020-07-14 13:03:43] [13:03:43.388] Batch 127...
[INFO] [2020-07-14 13:03:44] [13:03:44.252] Batch 128...
[INFO] [2020-07-14 13:03:45] [13:03:45.121] Batch 129...
[INFO] [2020-07-14 13:03:45] [13:03:45.969] Batch 130...
[INFO] [2020-07-14 13:03:46] [13:03:46.851] Batch 131...
[INFO] [2020-07-14 13:03:47] [13:03:47.699] Batch 132...
[INFO] [2020-07-14 13:03:48] [13:03:48.570] Batch 133...
[INFO] [2020-07-14 13:03:49] [13:03:49.440] Batch 134...
[INFO] [2020-07-14 13:03:50] [13:03:50.330] Batch 135...
[INFO] [2020-07-14 13:03:51] [13:03:51.182] Batch 136...
[INFO] [2020-07-14 13:03:52] [13:03:52.060] Batch 137...
[INFO] [2020-07-14 13:03:52] [13:03:52.910] Batch 138...
[INFO] [2020-07-14 13:03:53] [13:03:53.752] Batch 139...
[INFO] [2020-07-14 13:03:54] [13:03:54.600] Batch 140...
[INFO] [2020-07-14 13:03:55] [13:03:55.454] Batch 141...
[INFO] [2020-07-14 13:03:56] [13:03:56.303] Batch 142...
[INFO] [2020-07-14 13:03:57] [13:03:57.149] Batch 143...
[INFO] [2020-07-14 13:03:58] [13:03:58.017] Batch 144...
[INFO] [2020-07-14 13:03:58] [13:03:58.840] Batch 145...
[INFO] [2020-07-14 13:03:59] [13:03:59.663] Batch 146...
[INFO] [2020-07-14 13:04:00] [13:04:00.511] Batch 147...
[INFO] [2020-07-14 13:04:01] [13:04:01.378] Batch 148...
[INFO] [2020-07-14 13:04:02] [13:04:02.205] Removed 1489696 Metatraits
[INFO] [2020-07-14 13:04:02] ## remove_type: OccurrenceMetadatum
[INFO] [2020-07-14 13:04:02] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-14 13:04:02] [13:04:02.227] Removed 0 Occurrencemetadata
[INFO] [2020-07-14 13:04:02] ## remove_type: Assoc
[INFO] [2020-07-14 13:04:02] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-14 13:04:02] [13:04:02.230] Removed 0 Assocs
[INFO] [2020-07-14 13:04:02] ## remove_type: MetaAssoc
[INFO] [2020-07-14 13:04:02] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-14 13:04:02] [13:04:02.233] Removed 0 Metaassocs
[INFO] [2020-07-14 13:04:02] ## remove_type: Identifier
[INFO] [2020-07-14 13:04:02] ++ Calling delete_all on 35605 instances...
[INFO] [2020-07-14 13:04:05] [13:04:05.387] Removed 35605 Identifiers
[INFO] [2020-07-14 13:04:05] ## remove_type: Reference
[INFO] [2020-07-14 13:04:05] ++ Calling delete_all on 2 instances...
[INFO] [2020-07-14 13:04:05] [13:04:05.391] Removed 2 References
[INFO] [2020-07-14 13:04:07] Starting batch with ID 67549708...
[INFO] [2020-07-14 13:04:08] Starting batch with ID 67549708...
[INFO] [2020-07-14 13:04:09] Starting batch with ID 67540687...
[INFO] [2020-07-14 13:04:10] Starting batch with ID 67540687...
[INFO] [2020-07-14 13:04:11] Starting batch with ID 67543337...
[INFO] [2020-07-14 13:04:12] Starting batch with ID 67543337...
[INFO] [2020-07-14 13:04:13] Starting batch with ID 67546654...
[INFO] [2020-07-14 13:04:14] Starting batch with ID 67546654...
[INFO] [2020-07-14 13:04:15] Starting batch with ID 67543557...
[INFO] [2020-07-14 13:04:16] Starting batch with ID 67543557...
[INFO] [2020-07-14 13:04:17] Starting batch with ID 67539893...
[INFO] [2020-07-14 13:04:18] Starting batch with ID 67551800...
[INFO] [2020-07-14 13:04:20] Starting batch with ID 67530522...
[INFO] [2020-07-14 13:04:20] Starting batch with ID 67542924...
[INFO] [2020-07-14 13:04:20] ## remove_type: Node
[INFO] [2020-07-14 13:04:20] ++ Calling delete_all on 35956 instances...
[INFO] [2020-07-14 13:04:24] [13:04:24.797] Removed 35956 Nodes
[START] [2020-07-14 13:05:07] logged process
[START] [2020-07-14 13:05:07] Creating resource from OpenData
[START] [2020-07-14 13:05:12] logged process
[START] [2020-07-14 13:05:12] Parse meta.xml file and create formats with fields
[WARN] [2020-07-14 13:05:13] (common) IGNORED  (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2020-07-14 13:05:13] (common) IGNORED  (refs) field header: publicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2020-07-14 13:05:13] (common) IGNORED  (refs) field header: pageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2020-07-14 13:05:13] (common) IGNORED  (refs) field header: pageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2020-07-14 13:05:13] (common) IGNORED  (refs) field header: language term: http://purl.org/dc/terms/language
[STOP] [2020-07-14 13:05:13] Parse meta.xml file and create formats with fields
[STOP] [2020-07-14 13:05:13] Creating resource from OpenData
[START] [2020-07-14 13:05:13] logged process
[START] [2020-07-14 13:05:13] create_harvest_instance
[STOP] [2020-07-14 13:05:30] create_harvest_instance
[START] [2020-07-14 13:05:30] fetch_files
[STOP] [2020-07-14 13:05:30] fetch_files
[START] [2020-07-14 13:05:30] validate_each_file
[STOP] [2020-07-14 13:06:31] validate_each_file
[START] [2020-07-14 13:06:31] convert_to_csv
[CMD] [2020-07-14 13:06:31] /usr/bin/sort /app/public/converted_csv/usda_plants_agents_21809.csv > /app/public/converted_csv/usda_plants_agents_21809.csv_sorted
[CMD] [2020-07-14 13:06:31] /usr/bin/sort /app/public/converted_csv/usda_plants_refs_21810.csv > /app/public/converted_csv/usda_plants_refs_21810.csv_sorted
[CMD] [2020-07-14 13:06:31] /usr/bin/sort /app/public/converted_csv/usda_plants_nodes_21811.csv > /app/public/converted_csv/usda_plants_nodes_21811.csv_sorted
[CMD] [2020-07-14 13:06:31] /usr/bin/sort /app/public/converted_csv/usda_plants_media_21812.csv > /app/public/converted_csv/usda_plants_media_21812.csv_sorted
[CMD] [2020-07-14 13:06:31] /usr/bin/sort /app/public/converted_csv/usda_plants_vernaculars_21813.csv > /app/public/converted_csv/usda_plants_vernaculars_21813.csv_sorted
[CMD] [2020-07-14 13:06:31] /usr/bin/sort /app/public/converted_csv/usda_plants_occurrences_21814.csv > /app/public/converted_csv/usda_plants_occurrences_21814.csv_sorted
[CMD] [2020-07-14 13:06:31] /usr/bin/sort /app/public/converted_csv/usda_plants_measurements_21815.csv > /app/public/converted_csv/usda_plants_measurements_21815.csv_sorted
[STOP] [2020-07-14 13:06:32] convert_to_csv
[START] [2020-07-14 13:06:32] calculate_delta
[CMD] [2020-07-14 13:06:32] echo "0a" > /app/public/diff/usda_plants_agents_21809.diff
[CMD] [2020-07-14 13:06:32] tail -n +1 /app/public/converted_csv/usda_plants_agents_21809.csv >> /app/public/diff/usda_plants_agents_21809.diff
[CMD] [2020-07-14 13:06:32] echo "." >> /app/public/diff/usda_plants_agents_21809.diff
[CMD] [2020-07-14 13:06:32] echo "0a" > /app/public/diff/usda_plants_refs_21810.diff
[CMD] [2020-07-14 13:06:32] tail -n +1 /app/public/converted_csv/usda_plants_refs_21810.csv >> /app/public/diff/usda_plants_refs_21810.diff
[CMD] [2020-07-14 13:06:32] echo "." >> /app/public/diff/usda_plants_refs_21810.diff
[CMD] [2020-07-14 13:06:32] echo "0a" > /app/public/diff/usda_plants_nodes_21811.diff
[CMD] [2020-07-14 13:06:32] tail -n +1 /app/public/converted_csv/usda_plants_nodes_21811.csv >> /app/public/diff/usda_plants_nodes_21811.diff
[CMD] [2020-07-14 13:06:32] echo "." >> /app/public/diff/usda_plants_nodes_21811.diff
[CMD] [2020-07-14 13:06:32] echo "0a" > /app/public/diff/usda_plants_media_21812.diff
[CMD] [2020-07-14 13:06:32] tail -n +1 /app/public/converted_csv/usda_plants_media_21812.csv >> /app/public/diff/usda_plants_media_21812.diff
[CMD] [2020-07-14 13:06:32] echo "." >> /app/public/diff/usda_plants_media_21812.diff
[CMD] [2020-07-14 13:06:32] echo "0a" > /app/public/diff/usda_plants_vernaculars_21813.diff
[CMD] [2020-07-14 13:06:32] tail -n +1 /app/public/converted_csv/usda_plants_vernaculars_21813.csv >> /app/public/diff/usda_plants_vernaculars_21813.diff
[CMD] [2020-07-14 13:06:32] echo "." >> /app/public/diff/usda_plants_vernaculars_21813.diff
[CMD] [2020-07-14 13:06:32] echo "0a" > /app/public/diff/usda_plants_occurrences_21814.diff
[CMD] [2020-07-14 13:06:32] tail -n +1 /app/public/converted_csv/usda_plants_occurrences_21814.csv >> /app/public/diff/usda_plants_occurrences_21814.diff
[CMD] [2020-07-14 13:06:32] echo "." >> /app/public/diff/usda_plants_occurrences_21814.diff
[CMD] [2020-07-14 13:06:32] echo "0a" > /app/public/diff/usda_plants_measurements_21815.diff
[CMD] [2020-07-14 13:06:32] tail -n +1 /app/public/converted_csv/usda_plants_measurements_21815.csv >> /app/public/diff/usda_plants_measurements_21815.diff
[CMD] [2020-07-14 13:06:32] echo "." >> /app/public/diff/usda_plants_measurements_21815.diff
[STOP] [2020-07-14 13:06:32] calculate_delta
[START] [2020-07-14 13:06:32] parse_diff_and_store
[INFO] [2020-07-14 13:06:32] Loading agents diff file into memory (true lines)...
[INFO] [2020-07-14 13:06:32] Loading refs diff file into memory (true lines)...
[INFO] [2020-07-14 13:06:33] Loading nodes diff file into memory (true lines)...
[INFO] [2020-07-14 13:06:44] Loading media diff file into memory (true lines)...
[INFO] [2020-07-14 13:06:44] Loading vernaculars diff file into memory (true lines)...
[INFO] [2020-07-14 13:07:19] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-07-14 13:08:23] Loading measurements diff file into memory (true lines)...
[INFO] [2020-07-14 14:05:04] Storing 1 Attributions
[INFO] [2020-07-14 14:05:04] Processing group of 1 in 1 groups of 1000
[INFO] [2020-07-14 14:05:04] Average Time: 0.0
[INFO] [2020-07-14 14:05:04] Total Time: 1s
[INFO] [2020-07-14 14:05:04] Storing 2 References
[INFO] [2020-07-14 14:05:04] Processing group of 2 in 1 groups of 1000
[INFO] [2020-07-14 14:05:04] Average Time: 0.0
[INFO] [2020-07-14 14:05:04] Total Time: 1s
[INFO] [2020-07-14 14:05:04] Storing 35956 ScientificNames
[INFO] [2020-07-14 14:05:04] Processing group of 35956 in 36 groups of 1000
[INFO] [2020-07-14 14:05:31] Average Time: 0.753
[INFO] [2020-07-14 14:05:31] Total Time: 28s
[INFO] [2020-07-14 14:05:31] last 3 / first 3: 0.77
[INFO] [2020-07-14 14:05:31] Std.Dev: 2.281227739617419; Max: 14.05
[INFO] [2020-07-14 14:05:31] Storing 35956 Nodes
[INFO] [2020-07-14 14:05:31] Processing group of 35956 in 36 groups of 1000
[INFO] [2020-07-14 14:05:43] Average Time: 0.319
[INFO] [2020-07-14 14:05:43] Total Time: 12s
[INFO] [2020-07-14 14:05:43] last 3 / first 3: 1.08
[INFO] [2020-07-14 14:05:43] Std.Dev: 0.07745966692414834; Max: 0.6
[INFO] [2020-07-14 14:05:43] Storing 35605 Identifiers
[INFO] [2020-07-14 14:05:43] Processing group of 35605 in 36 groups of 1000
[INFO] [2020-07-14 14:05:46] Average Time: 0.082
[INFO] [2020-07-14 14:05:46] Total Time: 4s
[INFO] [2020-07-14 14:05:46] last 3 / first 3: 0.68
[INFO] [2020-07-14 14:05:46] Std.Dev: 0.0; Max: 0.15
[INFO] [2020-07-14 14:05:46] Storing 2 BibliographicCitations
[INFO] [2020-07-14 14:05:46] Processing group of 2 in 1 groups of 1000
[INFO] [2020-07-14 14:05:46] Average Time: 0.02
[INFO] [2020-07-14 14:05:46] Total Time: 1s
[INFO] [2020-07-14 14:05:46] Storing 2 ArticlesSections
[INFO] [2020-07-14 14:05:46] Processing group of 2 in 1 groups of 1000
[INFO] [2020-07-14 14:05:46] Average Time: 0.01
[INFO] [2020-07-14 14:05:46] Total Time: 1s
[INFO] [2020-07-14 14:05:46] Storing 2 Articles
[INFO] [2020-07-14 14:05:46] Processing group of 2 in 1 groups of 1000
[INFO] [2020-07-14 14:05:46] Average Time: 0.01
[INFO] [2020-07-14 14:05:46] Total Time: 1s
[INFO] [2020-07-14 14:05:46] Storing 1 ContentAttributions
[INFO] [2020-07-14 14:05:46] Processing group of 1 in 1 groups of 1000
[INFO] [2020-07-14 14:05:46] Average Time: 0.01
[INFO] [2020-07-14 14:05:46] Total Time: 1s
[INFO] [2020-07-14 14:05:46] Storing 3 Media
[INFO] [2020-07-14 14:05:46] Processing group of 3 in 1 groups of 1000
[INFO] [2020-07-14 14:05:46] Average Time: 0.02
[INFO] [2020-07-14 14:05:46] Total Time: 1s
[INFO] [2020-07-14 14:05:46] Storing 305965 Vernaculars
[INFO] [2020-07-14 14:05:46] Processing group of 305965 in 306 groups of 1000
[INFO] [2020-07-14 14:07:07] Average Time: 0.26
[INFO] [2020-07-14 14:07:07] Total Time: 1m21s
[INFO] [2020-07-14 14:07:07] last 3 / first 3: 0.94
[INFO] [2020-07-14 14:07:07] Std.Dev: 1.1171392035015153; Max: 14.0
[INFO] [2020-07-14 14:07:07] Storing 636471 Occurrences
[INFO] [2020-07-14 14:07:07] Processing group of 636471 in 637 groups of 1000
[INFO] [2020-07-14 14:10:20] Average Time: 0.275
[INFO] [2020-07-14 14:10:20] Total Time: 3m14s
[INFO] [2020-07-14 14:10:20] last 3 / first 3: 1.04
[INFO] [2020-07-14 14:10:20] Std.Dev: 1.5346009253222805; Max: 15.36
[INFO] [2020-07-14 14:10:20] Storing 581781 Traits
[INFO] [2020-07-14 14:10:20] Processing group of 581781 in 582 groups of 1000
[INFO] [2020-07-14 14:15:45] Average Time: 0.554
[INFO] [2020-07-14 14:15:45] Total Time: 5m25s
[INFO] [2020-07-14 14:15:45] last 3 / first 3: 0.89
[INFO] [2020-07-14 14:15:45] Std.Dev: 2.0799038439312523; Max: 16.76
[INFO] [2020-07-14 14:15:45] Storing 1428384 MetaTraits
[INFO] [2020-07-14 14:15:45] Processing group of 1428384 in 1429 groups of 1000
[INFO] [2020-07-14 14:20:52] Average Time: 0.21
[INFO] [2020-07-14 14:20:52] Total Time: 5m8s
[INFO] [2020-07-14 14:20:52] last 3 / first 3: 2.13
[INFO] [2020-07-14 14:20:52] Std.Dev: 1.2856904759700136; Max: 18.08
[STOP] [2020-07-14 14:20:52] parse_diff_and_store
[START] [2020-07-14 14:20:52] resolve_keys
[INFO] [2020-07-14 14:21:46] Occurrences to nodes (through scientific_names)...
[INFO] [2020-07-14 14:22:00] traits to occurrences...
[INFO] [2020-07-14 14:22:48] traits to nodes (through occurrences)...
[INFO] [2020-07-14 14:23:03] Traits to sex term...
[INFO] [2020-07-14 14:23:16] Traits to lifestage term...
[INFO] [2020-07-14 14:23:28] MetaTraits to traits...
[INFO] [2020-07-14 14:24:58] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-07-14 14:24:58] Assocs to occurrences...
[INFO] [2020-07-14 14:24:58] Assocs to nodes...
[INFO] [2020-07-14 14:24:58] Assoc to sex term...
[INFO] [2020-07-14 14:24:58] Assoc to lifestage term...
[STOP] [2020-07-14 14:24:58] resolve_keys
[START] [2020-07-14 14:24:58] hold_for_later_1
[STOP] [2020-07-14 14:24:58] hold_for_later_1
[START] [2020-07-14 14:24:58] hold_for_later_2
[STOP] [2020-07-14 14:24:58] hold_for_later_2
[START] [2020-07-14 14:24:58] resolve_missing_parents
[STOP] [2020-07-14 14:25:02] resolve_missing_parents
[START] [2020-07-14 14:25:02] rebuild_nodes
[START] [2020-07-14 14:25:02] Flattener#flatten
[START] [2020-07-14 14:25:02] Flattener#study_resource
[START] [2020-07-14 14:25:02] Flattener#build_ancestry
[STOP] [2020-07-14 14:25:27] Flattener#build_ancestry
[INFO] [2020-07-14 14:25:27] 35956 ancestry keys
[START] [2020-07-14 14:25:27] build_node_ancestors
[INFO] [2020-07-14 14:25:27] old ancestors deleted.
[STOP] [2020-07-14 14:25:29] build_node_ancestors
[START] [2020-07-14 14:25:31] Flattener#propagate_ancestor_ids
[STOP] [2020-07-14 14:25:32] Flattener#propagate_ancestor_ids
[STOP] [2020-07-14 14:25:32] Flattener#flatten
[STOP] [2020-07-14 14:25:32] rebuild_nodes
[START] [2020-07-14 14:25:32] resolve_missing_media_owners
[STOP] [2020-07-14 14:25:32] resolve_missing_media_owners
[START] [2020-07-14 14:25:32] sanitize_media_verbatims
[STOP] [2020-07-14 14:25:32] sanitize_media_verbatims
[START] [2020-07-14 14:25:32] queue_downloads
[STOP] [2020-07-14 14:25:32] queue_downloads
[START] [2020-07-14 14:25:32] parse_names
[WARN] [2020-07-14 14:25:32] I see 35956 names which still need to be parsed.
[STOP] [2020-07-14 14:26:00] parse_names
[START] [2020-07-14 14:26:00] denormalize_canonical_names_to_nodes
[STOP] [2020-07-14 14:26:01] denormalize_canonical_names_to_nodes
[START] [2020-07-14 14:26:01] match_nodes
[START] [2020-07-14 14:26:01] map_all_nodes_to_pages
[STOP] [2020-07-14 15:36:23] map_all_nodes_to_pages
[INFO] [2020-07-14 15:36:23] 2526 Unmatched nodes (of 35956)! That's too many to output. First 10: Abutilon malacum (#80164354); Abelmoschus moschatus (#80164361); Abutilon parishii (#80164370); Abutilon reventum (#80164377); Anoda abutiloides (#80165791); Anoda reticulata (#80166090); Callirhoe involucrata (#80170080); Hibiscus aculeatus (#80180997); Hibiscus biseptus (#80181012); Hibiscadelphus bombycinus (#80181017)
[START] [2020-07-14 15:36:23] update_nodes
[STOP] [2020-07-14 15:36:38] update_nodes
[STOP] [2020-07-14 15:36:38] match_nodes
[START] [2020-07-14 15:36:39] reindex_search
[STOP] [2020-07-14 15:37:51] reindex_search
[START] [2020-07-14 15:37:51] normalize_units
[STOP] [2020-07-14 15:39:30] normalize_units
[START] [2020-07-14 15:39:30] calculate_statistics
[STOP] [2020-07-14 15:39:31] calculate_statistics
[START] [2020-07-14 15:39:31] complete_harvest_instance
[START] [2020-07-14 15:39:31] overall_tsv_creation
[INFO] [2020-07-14 15:39:31] Processing group of 35956 in 4 batches of 10000
[INFO] [2020-07-14 15:41:14] 162326 Traits (unfiltered)...
[INFO] [2020-07-14 15:54:21] 162326 Traits (filtered)...
[INFO] [2020-07-14 15:54:24] 0 Associations (filtered)...
[INFO] [2020-07-14 15:54:43] 399663 metadata added.
[INFO] [2020-07-14 15:54:43] 0 metadata added.
[INFO] [2020-07-14 15:57:38] 147902 Traits (unfiltered)...
[INFO] [2020-07-14 16:10:08] 147902 Traits (filtered)...
[INFO] [2020-07-14 16:10:11] 0 Associations (filtered)...
[INFO] [2020-07-14 16:10:24] 358488 metadata added.
[INFO] [2020-07-14 16:10:24] 0 metadata added.
[INFO] [2020-07-14 16:13:33] 158813 Traits (unfiltered)...
[INFO] [2020-07-14 16:26:52] 158813 Traits (filtered)...
[INFO] [2020-07-14 16:26:55] 0 Associations (filtered)...
[INFO] [2020-07-14 16:27:21] 391447 metadata added.
[INFO] [2020-07-14 16:27:21] 0 metadata added.
[INFO] [2020-07-14 16:29:50] 102471 Traits (unfiltered)...
[INFO] [2020-07-14 16:38:28] 102471 Traits (filtered)...
[INFO] [2020-07-14 16:38:31] 0 Associations (filtered)...
[INFO] [2020-07-14 16:38:39] 247220 metadata added.
[INFO] [2020-07-14 16:38:39] 0 metadata added.
[INFO] [2020-07-14 16:38:58] Average Time: 850.553
[INFO] [2020-07-14 16:38:58] Total Time: 59m27s
[STOP] [2020-07-14 16:38:58] overall_tsv_creation
[INFO] [2020-07-14 16:38:58] Done. Check your files:
[INFO] [2020-07-14 16:38:58] (35956 lines) /app/public/data/usda_plants/publish_nodes.tsv
[INFO] [2020-07-14 16:38:59] (35605 lines) /app/public/data/usda_plants/publish_identifiers.tsv
[INFO] [2020-07-14 16:38:59] (35605 lines) /app/public/data/usda_plants/publish_node_ancestors.tsv
[INFO] [2020-07-14 16:38:59] (35956 lines) /app/public/data/usda_plants/publish_scientific_names.tsv
[INFO] [2020-07-14 16:39:00] (305965 lines) /app/public/data/usda_plants/publish_vernaculars.tsv
[INFO] [2020-07-14 16:39:00] (571513 lines) /app/public/data/usda_plants/publish_traits.tsv
[INFO] [2020-07-14 16:39:01] (41789 lines) /app/public/data/usda_plants/publish_metadata.tsv
[STOP] [2020-07-14 16:39:01] complete_harvest_instance
[START] [2020-07-14 16:39:01] completed
[STOP] [2020-07-14 16:39:01] completed
[STOP] [2020-07-14 16:39:01] logged process, took 12827.84

Latest Process