Harvest for North American Flora Created 23 Dec 11:16

Harvesting Log

[INFO] [2021-12-23 11:16:50] Created harvest instance #4086
[STOP] [2021-12-23 11:16:50] create_harvest_instance
[START] [2021-12-23 11:16:50] fetch_files
[STOP] [2021-12-23 11:16:50] fetch_files
[START] [2021-12-23 11:16:50] validate_each_file
[INFO] [2021-12-23 11:16:50] Looping over 5 formats...
[INFO] [2021-12-23 11:16:50] ...nodes (/app/public/data/n_am_flora/taxon.tab)
[INFO] [2021-12-23 11:16:50] Valid: /app/public/converted_csv/n_am_flora_nodes_4086.csv (21588 lines)
[INFO] [2021-12-23 11:16:50] ...media (/app/public/data/n_am_flora/media_resource.tab)
[INFO] [2021-12-23 11:16:51] Valid: /app/public/converted_csv/n_am_flora_media_4086.csv (16993 lines)
[INFO] [2021-12-23 11:16:51] ...occurrences (/app/public/data/n_am_flora/occurrence_specific.tab)
[INFO] [2021-12-23 11:16:53] Valid: /app/public/converted_csv/n_am_flora_occurrences_4086.csv (96181 lines)
[INFO] [2021-12-23 11:16:53] ...assocs (/app/public/data/n_am_flora/association.tab)
[INFO] [2021-12-23 11:16:54] Valid: /app/public/converted_csv/n_am_flora_assocs_4086.csv (5698 lines)
[INFO] [2021-12-23 11:16:54] ...measurements (/app/public/data/n_am_flora/measurement_or_fact_specific.tab)
[INFO] [2021-12-23 11:16:59] Valid: /app/public/converted_csv/n_am_flora_measurements_4086.csv (89904 lines)
[STOP] [2021-12-23 11:16:59] validate_each_file
[START] [2021-12-23 11:16:59] convert_to_csv
[INFO] [2021-12-23 11:16:59] Looping over 5 formats...
[INFO] [2021-12-23 11:16:59] ...nodes (/app/public/data/n_am_flora/taxon.tab)
[CMD] [2021-12-23 11:16:59] /usr/bin/sort /app/public/converted_csv/n_am_flora_nodes_4086.csv > /app/public/converted_csv/n_am_flora_nodes_4086.csv_sorted
[INFO] [2021-12-23 11:17:00] Converted: /app/public/converted_csv/n_am_flora_nodes_4086.csv (21588 lines)
[INFO] [2021-12-23 11:17:00] ...media (/app/public/data/n_am_flora/media_resource.tab)
[CMD] [2021-12-23 11:17:00] /usr/bin/sort /app/public/converted_csv/n_am_flora_media_4086.csv > /app/public/converted_csv/n_am_flora_media_4086.csv_sorted
[INFO] [2021-12-23 11:17:02] Converted: /app/public/converted_csv/n_am_flora_media_4086.csv (16993 lines)
[INFO] [2021-12-23 11:17:02] ...occurrences (/app/public/data/n_am_flora/occurrence_specific.tab)
[CMD] [2021-12-23 11:17:02] /usr/bin/sort /app/public/converted_csv/n_am_flora_occurrences_4086.csv > /app/public/converted_csv/n_am_flora_occurrences_4086.csv_sorted
[INFO] [2021-12-23 11:17:03] Converted: /app/public/converted_csv/n_am_flora_occurrences_4086.csv (96181 lines)
[INFO] [2021-12-23 11:17:03] ...assocs (/app/public/data/n_am_flora/association.tab)
[CMD] [2021-12-23 11:17:03] /usr/bin/sort /app/public/converted_csv/n_am_flora_assocs_4086.csv > /app/public/converted_csv/n_am_flora_assocs_4086.csv_sorted
[INFO] [2021-12-23 11:17:04] Converted: /app/public/converted_csv/n_am_flora_assocs_4086.csv (5698 lines)
[INFO] [2021-12-23 11:17:04] ...measurements (/app/public/data/n_am_flora/measurement_or_fact_specific.tab)
[CMD] [2021-12-23 11:17:04] /usr/bin/sort /app/public/converted_csv/n_am_flora_measurements_4086.csv > /app/public/converted_csv/n_am_flora_measurements_4086.csv_sorted
[INFO] [2021-12-23 11:17:06] Converted: /app/public/converted_csv/n_am_flora_measurements_4086.csv (89904 lines)
[STOP] [2021-12-23 11:17:06] convert_to_csv
[START] [2021-12-23 11:17:06] calculate_delta
[INFO] [2021-12-23 11:17:06] Looping over 5 formats...
[INFO] [2021-12-23 11:17:06] ...nodes (/app/public/data/n_am_flora/taxon.tab)
[CMD] [2021-12-23 11:17:06] echo "0a" > /app/public/diff/n_am_flora_nodes_4086.diff
[CMD] [2021-12-23 11:17:07] tail -n +1 /app/public/converted_csv/n_am_flora_nodes_4086.csv >> /app/public/diff/n_am_flora_nodes_4086.diff
[CMD] [2021-12-23 11:17:08] echo "." >> /app/public/diff/n_am_flora_nodes_4086.diff
[INFO] [2021-12-23 11:17:10] Created diff: /app/public/diff/n_am_flora_nodes_4086.diff (21590 lines)
[INFO] [2021-12-23 11:17:10] ...media (/app/public/data/n_am_flora/media_resource.tab)
[CMD] [2021-12-23 11:17:10] echo "0a" > /app/public/diff/n_am_flora_media_4086.diff
[CMD] [2021-12-23 11:17:11] tail -n +1 /app/public/converted_csv/n_am_flora_media_4086.csv >> /app/public/diff/n_am_flora_media_4086.diff
[CMD] [2021-12-23 11:17:12] echo "." >> /app/public/diff/n_am_flora_media_4086.diff
[INFO] [2021-12-23 11:17:14] Created diff: /app/public/diff/n_am_flora_media_4086.diff (16995 lines)
[INFO] [2021-12-23 11:17:14] ...occurrences (/app/public/data/n_am_flora/occurrence_specific.tab)
[CMD] [2021-12-23 11:17:14] echo "0a" > /app/public/diff/n_am_flora_occurrences_4086.diff
[CMD] [2021-12-23 11:17:15] tail -n +1 /app/public/converted_csv/n_am_flora_occurrences_4086.csv >> /app/public/diff/n_am_flora_occurrences_4086.diff
[CMD] [2021-12-23 11:17:16] echo "." >> /app/public/diff/n_am_flora_occurrences_4086.diff
[INFO] [2021-12-23 11:17:18] Created diff: /app/public/diff/n_am_flora_occurrences_4086.diff (96183 lines)
[INFO] [2021-12-23 11:17:18] ...assocs (/app/public/data/n_am_flora/association.tab)
[CMD] [2021-12-23 11:17:18] echo "0a" > /app/public/diff/n_am_flora_assocs_4086.diff
[CMD] [2021-12-23 11:17:19] tail -n +1 /app/public/converted_csv/n_am_flora_assocs_4086.csv >> /app/public/diff/n_am_flora_assocs_4086.diff
[CMD] [2021-12-23 11:17:20] echo "." >> /app/public/diff/n_am_flora_assocs_4086.diff
[INFO] [2021-12-23 11:17:22] Created diff: /app/public/diff/n_am_flora_assocs_4086.diff (5700 lines)
[INFO] [2021-12-23 11:17:22] ...measurements (/app/public/data/n_am_flora/measurement_or_fact_specific.tab)
[CMD] [2021-12-23 11:17:22] echo "0a" > /app/public/diff/n_am_flora_measurements_4086.diff
[CMD] [2021-12-23 11:17:23] tail -n +1 /app/public/converted_csv/n_am_flora_measurements_4086.csv >> /app/public/diff/n_am_flora_measurements_4086.diff
[CMD] [2021-12-23 11:17:24] echo "." >> /app/public/diff/n_am_flora_measurements_4086.diff
[INFO] [2021-12-23 11:17:26] Created diff: /app/public/diff/n_am_flora_measurements_4086.diff (89906 lines)
[STOP] [2021-12-23 11:17:26] calculate_delta
[START] [2021-12-23 11:17:26] parse_diff_and_store
[INFO] [2021-12-23 11:17:26] Handling diff: /app/public/diff/n_am_flora_nodes_4086.diff (21590 lines)
[INFO] [2021-12-23 11:17:27] Loading nodes diff file into memory (21590 /app/public/diff/n_am_flora_nodes_4086.diff lines)...
[INFO] [2021-12-23 11:17:34] Handling diff: /app/public/diff/n_am_flora_media_4086.diff (16995 lines)
[INFO] [2021-12-23 11:17:35] Loading media diff file into memory (16995 /app/public/diff/n_am_flora_media_4086.diff lines)...
[INFO] [2021-12-23 11:17:55] Handling diff: /app/public/diff/n_am_flora_occurrences_4086.diff (96183 lines)
[INFO] [2021-12-23 11:17:56] Loading occurrences diff file into memory (96183 /app/public/diff/n_am_flora_occurrences_4086.diff lines)...
[INFO] [2021-12-23 11:18:07] Handling diff: /app/public/diff/n_am_flora_assocs_4086.diff (5700 lines)
[INFO] [2021-12-23 11:18:08] Loading assocs diff file into memory (5700 /app/public/diff/n_am_flora_assocs_4086.diff lines)...
[INFO] [2021-12-23 11:18:12] Handling diff: /app/public/diff/n_am_flora_measurements_4086.diff (89906 lines)
[INFO] [2021-12-23 11:18:13] Loading measurements diff file into memory (89906 /app/public/diff/n_am_flora_measurements_4086.diff lines)...
[INFO] [2021-12-23 11:19:16] Storing 21588 ScientificNames
[INFO] [2021-12-23 11:19:16] Processing group of 21588 in 22 groups of 1000
[INFO] [2021-12-23 11:19:22] Average Time: 0.29
[INFO] [2021-12-23 11:19:22] Total Time: 7s
[INFO] [2021-12-23 11:19:22] last 3 / first 3: 0.77
[INFO] [2021-12-23 11:19:22] Std.Dev: 0.044721359549995794; Max: 0.38
[INFO] [2021-12-23 11:19:22] Storing 21588 Nodes
[INFO] [2021-12-23 11:19:22] Processing group of 21588 in 22 groups of 1000
[INFO] [2021-12-23 11:19:28] Average Time: 0.248
[INFO] [2021-12-23 11:19:28] Total Time: 6s
[INFO] [2021-12-23 11:19:28] last 3 / first 3: 0.85
[INFO] [2021-12-23 11:19:28] Std.Dev: 0.044721359549995794; Max: 0.36
[INFO] [2021-12-23 11:19:28] Storing 16993 BibliographicCitations
[INFO] [2021-12-23 11:19:28] Processing group of 16993 in 17 groups of 1000
[INFO] [2021-12-23 11:19:31] Average Time: 0.162
[INFO] [2021-12-23 11:19:31] Total Time: 3s
[INFO] [2021-12-23 11:19:31] last 3 / first 3: 1.12
[INFO] [2021-12-23 11:19:31] Std.Dev: 0.03162277660168379; Max: 0.25
[INFO] [2021-12-23 11:19:31] Storing 16993 ArticlesSections
[INFO] [2021-12-23 11:19:31] Processing group of 16993 in 17 groups of 1000
[INFO] [2021-12-23 11:19:32] Average Time: 0.051
[INFO] [2021-12-23 11:19:32] Total Time: 1s
[INFO] [2021-12-23 11:19:32] last 3 / first 3: 0.94
[INFO] [2021-12-23 11:19:32] Std.Dev: 0.0; Max: 0.06
[INFO] [2021-12-23 11:19:32] Storing 16993 Articles
[INFO] [2021-12-23 11:19:32] Processing group of 16993 in 17 groups of 1000
[INFO] [2021-12-23 11:19:37] Average Time: 0.342
[INFO] [2021-12-23 11:19:37] Total Time: 6s
[INFO] [2021-12-23 11:19:37] last 3 / first 3: 0.91
[INFO] [2021-12-23 11:19:37] Std.Dev: 0.044721359549995794; Max: 0.44
[INFO] [2021-12-23 11:19:37] Storing 96181 Occurrences
[INFO] [2021-12-23 11:19:37] Processing group of 96181 in 97 groups of 1000
[INFO] [2021-12-23 11:19:55] Average Time: 0.178
[INFO] [2021-12-23 11:19:55] Total Time: 18s
[INFO] [2021-12-23 11:19:55] last 3 / first 3: 2.5
[INFO] [2021-12-23 11:19:55] Std.Dev: 0.161245154965971; Max: 0.69
[INFO] [2021-12-23 11:19:55] Storing 86 OccurrenceMetadata
[INFO] [2021-12-23 11:19:55] Processing group of 86 in 1 groups of 1000
[INFO] [2021-12-23 11:19:55] Average Time: 0.02
[INFO] [2021-12-23 11:19:55] Total Time: 1s
[INFO] [2021-12-23 11:19:55] Storing 5698 Assocs
[INFO] [2021-12-23 11:19:55] Processing group of 5698 in 6 groups of 1000
[INFO] [2021-12-23 11:19:57] Average Time: 0.298
[INFO] [2021-12-23 11:19:57] Total Time: 2s
[INFO] [2021-12-23 11:19:57] Storing 11396 MetaAssocs
[INFO] [2021-12-23 11:19:57] Processing group of 11396 in 12 groups of 1000
[INFO] [2021-12-23 11:19:58] Average Time: 0.112
[INFO] [2021-12-23 11:19:58] Total Time: 2s
[INFO] [2021-12-23 11:19:58] last 3 / first 3: 1.06
[INFO] [2021-12-23 11:19:58] Std.Dev: 0.03162277660168379; Max: 0.19
[INFO] [2021-12-23 11:19:58] Storing 89904 Traits
[INFO] [2021-12-23 11:19:58] Processing group of 89904 in 90 groups of 1000
[INFO] [2021-12-23 11:20:27] Average Time: 0.314
[INFO] [2021-12-23 11:20:27] Total Time: 29s
[INFO] [2021-12-23 11:20:27] last 3 / first 3: 1.26
[INFO] [2021-12-23 11:20:27] Std.Dev: 0.05477225575051661; Max: 0.55
[INFO] [2021-12-23 11:20:27] Storing 179808 MetaTraits
[INFO] [2021-12-23 11:20:27] Processing group of 179808 in 180 groups of 1000
[INFO] [2021-12-23 11:20:59] Average Time: 0.169
[INFO] [2021-12-23 11:20:59] Total Time: 33s
[INFO] [2021-12-23 11:20:59] last 3 / first 3: 0.74
[INFO] [2021-12-23 11:20:59] Std.Dev: 0.14491376746189438; Max: 1.13
[STOP] [2021-12-23 11:20:59] parse_diff_and_store
[START] [2021-12-23 11:20:59] resolve_keys
[INFO] [2021-12-23 11:21:21] Occurrences to nodes (through scientific_names)...
[INFO] [2021-12-23 11:21:23] traits to occurrences...
[INFO] [2021-12-23 11:21:27] traits to nodes (through occurrences)...
[INFO] [2021-12-23 11:21:29] Traits to sex term...
[INFO] [2021-12-23 11:21:30] Traits to lifestage term...
[INFO] [2021-12-23 11:21:32] MetaTraits to traits...
[INFO] [2021-12-23 11:21:36] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-12-23 11:21:36] Assocs to occurrences...
[INFO] [2021-12-23 11:21:37] Assocs to nodes...
[INFO] [2021-12-23 11:21:37] Assoc to sex term...
[INFO] [2021-12-23 11:21:37] Assoc to lifestage term...
[INFO] [2021-12-23 11:21:37] MetaAssoc to assocs...
[STOP] [2021-12-23 11:21:37] resolve_keys
[START] [2021-12-23 11:21:37] hold_for_later_1
[STOP] [2021-12-23 11:21:37] hold_for_later_1
[START] [2021-12-23 11:21:37] hold_for_later_2
[STOP] [2021-12-23 11:21:37] hold_for_later_2
[START] [2021-12-23 11:21:37] resolve_missing_parents
[STOP] [2021-12-23 11:21:37] resolve_missing_parents
[START] [2021-12-23 11:21:37] rebuild_nodes
[START] [2021-12-23 11:21:37] Flattener#flatten
[START] [2021-12-23 11:21:37] Flattener#study_resource
[START] [2021-12-23 11:21:38] Flattener#build_ancestry
[STOP] [2021-12-23 11:21:39] Flattener#build_ancestry
[INFO] [2021-12-23 11:21:39] 21588 ancestry keys
[START] [2021-12-23 11:21:39] build_node_ancestors
[INFO] [2021-12-23 11:21:39] old ancestors deleted.
[STOP] [2021-12-23 11:21:39] build_node_ancestors
[WARN] [2021-12-23 11:21:39] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2021-12-23 11:21:39] Flattener#flatten
[STOP] [2021-12-23 11:21:39] rebuild_nodes
[START] [2021-12-23 11:21:39] resolve_missing_media_owners
[STOP] [2021-12-23 11:21:39] resolve_missing_media_owners
[START] [2021-12-23 11:21:39] sanitize_media_verbatims
[STOP] [2021-12-23 11:21:39] sanitize_media_verbatims
[START] [2021-12-23 11:21:39] queue_downloads
[STOP] [2021-12-23 11:21:39] queue_downloads
[START] [2021-12-23 11:21:39] parse_names
[WARN] [2021-12-23 11:21:39] I see 21588 names which still need to be parsed.
[WARN] [2021-12-23 11:21:40] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2021-12-23 11:21:50] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2021-12-23 11:21:56] Names to parse: 1588 formatted: 1588 learned: 1588 parsed: 1588
[STOP] [2021-12-23 11:21:58] parse_names
[START] [2021-12-23 11:21:58] denormalize_canonical_names_to_nodes
[STOP] [2021-12-23 11:21:59] denormalize_canonical_names_to_nodes
[START] [2021-12-23 11:21:59] match_nodes
[START] [2021-12-23 11:21:59] map_all_nodes_to_pages
[STOP] [2021-12-23 11:27:10] map_all_nodes_to_pages
[INFO] [2021-12-23 11:27:10] 4283 Unmatched nodes (of 21588)! That's too many to output. Full list in /app/public/data/n_am_flora/unmatched_nodes.txt ; First 10: Canonical: Phyllosticta astragali; Node#101634773; ResourceID: 000e8a82dbef8b81d660c02b2c46e584; Canonical: Acacia nllotica; Node#101634774; ResourceID: 000ec950b34f1dbd21294d01ec09c257; Canonical: Potentilla flabelilfolia; Node#101634775; ResourceID: 000fdea5d1f83803b4a4059f1ae99b3a; Canonical: Vernonia ervendbcrgii; Node#101634776; ResourceID: 00107b4bd9513a50646367c65f365e1d; Canonical: Meibomia bracteosa; Node#101634784; ResourceID: 001f626038359bedc896c7fbfb07db16; Canonical: Lophotocarpus spathulatus; Node#101634800; ResourceID: 004f54e1d686b576620dfe8045eadec2; Canonical: Cortinarius sericipes; Node#101634805; ResourceID: 005c832fcf690dfb3a3c23141eb81160; Canonical: Sedum diversifolium; Node#101634806; ResourceID: 006155d1a79cd956f96fbc0f4a0f5c0c; Canonical: Lyclum califomicum; Node#101634810; ResourceID: 00696988560d4a0a30e409e9feee3823; Canonical: Russula anomala; Node#101634814; ResourceID: 006db9cc46a38ca9575b66c3dcbacc3b
[START] [2021-12-23 11:27:10] update_nodes
[STOP] [2021-12-23 11:27:18] update_nodes
[STOP] [2021-12-23 11:27:18] match_nodes
[START] [2021-12-23 11:27:18] reindex_search
[STOP] [2021-12-23 11:27:30] reindex_search
[START] [2021-12-23 11:27:30] normalize_units
[STOP] [2021-12-23 11:29:17] normalize_units
[START] [2021-12-23 11:29:17] calculate_statistics
[2021-12-23 11:29:17] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[2021-12-23 11:29:18] (NEAR) DUPLICATE TRAITS FOUND! There are only 89093 (of 89904 total) unique traits.
[2021-12-23 11:41:28] (Near) duplicate trait pairs (up to 100):
[2021-12-23 11:41:28] (resource_pk: 0018b0e36e7d3cfc193e255775da8186_15441_ENV, id: 245570532), (resource_pk: fc5d09355c761238ad78ea9e59595c27_15441_ENV, id: 245659099)
[2021-12-23 11:41:28] (resource_pk: 00605f3ec0f7b85cd3820c0b2789b4a9_15441_ENV, id: 245570623), (resource_pk: 24ef8a43c70547884e8f4d18a15fe099_15441_ENV, id: 245583399)
[2021-12-23 11:41:28] (resource_pk: 0071007ca04cbc3026fb133fc4214a0f_15441, id: 245570645), (resource_pk: 93907eaaf9b0961397a10d7fbed59ef3_15441, id: 245622231)
[2021-12-23 11:41:28] (resource_pk: 00ea10a0583a2d14df99b11beef8d857_91116, id: 245570792), (resource_pk: 3d97ba08b7f614d8cc6de7875ebb3319_91116, id: 245591943)
[2021-12-23 11:41:28] (resource_pk: 00f79c3dbe4377958601d15b2f2053ed_15441_ENV, id: 245570815), (resource_pk: 06e5941880c80ac56796d9423c5ac85a_15441_ENV, id: 245572947)
[2021-12-23 11:41:28] (resource_pk: 01048ab8ff66cd76a09d88123202db89_15441, id: 245570832), (resource_pk: fb460e7c3aa58519a0bb9b394662d26a_15441, id: 245658749)
[2021-12-23 11:41:28] (resource_pk: 010b2bf6c4f31b7609e0102c469113e0_15441_ENV, id: 245570837), (resource_pk: d0522cb804b913bba8598b47e59e5069_15441_ENV, id: 245643761)
[2021-12-23 11:41:28] (resource_pk: 012d474e13385a994123623d937b747c_15441_ENV, id: 245570883), (resource_pk: 089e9cb3d1bbc0687c81d1527cc30a88_15441_ENV, id: 245573522)
[2021-12-23 11:41:28] (resource_pk: 0134de8db4623b07f67a5d127c9e89ce_15441, id: 245570891), (resource_pk: c1b34737049b9dcaf882fdfe5af88369_15441, id: 245638548)
[2021-12-23 11:41:28] (resource_pk: 0167c9d33545cbd8cb1642a94476a9d1_15441_ENV, id: 245570954), (resource_pk: b4048ad8410f59ea2a431f248e852231_15441_ENV, id: 245633649)
[2021-12-23 11:41:28] (resource_pk: 017ef17368747c9b5d2d026d1e24b620_15441_ENV, id: 245570989), (resource_pk: 68e2772a2cd428a4aadbfeedfb910b5b_15441_ENV, id: 245607153)
[2021-12-23 11:41:28] (resource_pk: 0185430e22e21d480a1d44b096c2004b_15441, id: 245571002), (resource_pk: 7c6a926087c1de7d43acecd1b4692476_15441, id: 245614126)
[2021-12-23 11:41:28] (resource_pk: 0191a30534f03deabe91bf7dd3887a51_15441_ENV, id: 245571026), (resource_pk: fa705cf591737bd1fd8fc2687c962f1b_15441_ENV, id: 245658464)
[2021-12-23 11:41:28] (resource_pk: 0194a8a6b9c8520418e6533a50025e00_15441_ENV, id: 245571032), (resource_pk: 6dceb74e7efd5d39ea8a87495a67ed3e_15441_ENV, id: 245608916)
[2021-12-23 11:41:28] (resource_pk: 019ce4687650b91b6e80425c98237b86_15441, id: 245571047), (resource_pk: 93777f4f48f0a8bf6f326e885ac8e36b_15441, id: 245622194)
[2021-12-23 11:41:28] (resource_pk: 019cf74b24f92d637989baeb3e33d410_15438, id: 245571048), (resource_pk: dde0a7ccd8dd5ffff37a1a47c7a15e57_15438, id: 245648422)
[2021-12-23 11:41:28] (resource_pk: 0233b3caf5668f9586a7be4507eb1ce4_15441, id: 245571254), (resource_pk: 8870ce7c411014ba4a6cae5e6357edc4_15441, id: 245618360)
[2021-12-23 11:41:28] (resource_pk: 023b51f48d1c61fe09d29dcf167334c7_15441, id: 245571260), (resource_pk: d97b935facdec16e923b4b3d61dcf057_15441, id: 245646870)
[2021-12-23 11:41:28] (resource_pk: 024ae67fc17d24be3fd82c87895e6ad2_15441, id: 245571290), (resource_pk: cf05b7072791b73516f170a37b4f0417_15441, id: 245643263)
[2021-12-23 11:41:28] (resource_pk: 028c08ddb024c25025474b91b48993b4_91529, id: 245571393), (resource_pk: 269ce7a20d24c494a3093f92642bcf65_91529, id: 245584021)
[2021-12-23 11:41:28] (resource_pk: 029a88f920e49640e4e2f49ef8da0dfc_15441, id: 245571423), (resource_pk: 30320a3c012748367c2c9113c627f9e4_15441, id: 245587312)
[2021-12-23 11:41:28] (resource_pk: 02c82c7c9362abfc3146b59e9b01775a_15441, id: 245571495), (resource_pk: 7eb253d943bd3c3d07dbac2f8f080803_15441, id: 245614868)
[2021-12-23 11:41:28] (resource_pk: 02eea9aa1853b479954de53317cd00f7_91534, id: 245571555), (resource_pk: 5e665ff9987ff853579412b8ceefbd67_91534, id: 245603450)
[2021-12-23 11:41:28] (resource_pk: 02fc426bd8f4b8b8e4649482ad4c8394_15441_ENV, id: 245571569), (resource_pk: 0bd689ca6ee0920218031c66cabb2050_15441_ENV, id: 245574652)
[2021-12-23 11:41:28] (resource_pk: 031dbc55c36c66b5fe7ea7c38a5993ed_15441_ENV, id: 245571619), (resource_pk: 88cb0ffb7bf32317817eb059880c74dd_15441_ENV, id: 245618482)
[2021-12-23 11:41:28] (resource_pk: 03875e76d74afefc57bd82941dc97c56_15441_ENV, id: 245571759), (resource_pk: 0bf80625d6fdcd4d11edc84369c70bc4_15441_ENV, id: 245574692)
[2021-12-23 11:41:28] (resource_pk: 03bfa8534eb51cb421c17733888bb822_15441, id: 245571831), (resource_pk: 1ec0ab9ccc3ccc4bb2f4a87b441fcf4c_15441, id: 245581226)
[2021-12-23 11:41:28] (resource_pk: 03cddcfc028102fb8bbeddbe917630b1_15441_ENV, id: 245571846), (resource_pk: c21ccf90376f63fc2f5fda014952ce68_15441_ENV, id: 245638696)
[2021-12-23 11:41:28] (resource_pk: 03e5270d69552e0b5ea7fceb60ae9f41_15441, id: 245571882), (resource_pk: cd62fcc348428f0d29fda0b2eef1bda0_15441, id: 245642624)
[2021-12-23 11:41:28] (resource_pk: 03edda154af36737f41bab6c1a75c5a1_15441_ENV, id: 245571897), (resource_pk: bdffda0a1ee59dd8e386d5bc26b06ac3_15441_ENV, id: 245637243)
[2021-12-23 11:41:28] (resource_pk: 0406dde3a838cae2cb7010ef48053f35_15441_ENV, id: 245571943), (resource_pk: 0e60b36bb66af276957a2e0423c8db27_15441_ENV, id: 245575529)
[2021-12-23 11:41:28] (resource_pk: 044ab20a3c3331821be8b2e9cd6443aa_15441, id: 245572040), (resource_pk: 7fc32b8bddf687bd26c283ec7e886d18_15441, id: 245615249)
[2021-12-23 11:41:28] (resource_pk: 04d924d625da417d7f2fb8d920290735_15441_ENV, id: 245572229), (resource_pk: 518ee9cafaeb8f7db86a37d43ede750e_15441_ENV, id: 245598917)
[2021-12-23 11:41:28] (resource_pk: 04f3ccb9030980d90f8bc6f2544c639b_15441_ENV, id: 245572264), (resource_pk: c3aac9fc1f8b74d1d74e7d48bc1861e6_15441_ENV, id: 245639245)
[2021-12-23 11:41:28] (resource_pk: 053d4d29f5c169a0251373994ee6bdcb_15441_ENV, id: 245572361), (resource_pk: 7bb885a15e315a7b9a36bd6f8dc27d8b_15441_ENV, id: 245613883)
[2021-12-23 11:41:28] (resource_pk: 0550c1dc341fb3f17f9aa6abbe924a7d_15441_ENV, id: 245572393), (resource_pk: 4792d484eb1098cd5c07e1047448f567_15441_ENV, id: 245595488)
[2021-12-23 11:41:28] (resource_pk: 055e5d16c7b245b9507050297a772864_15441, id: 245572411), (resource_pk: 28ddc7d2945bab08de3f148c7db45ea4_15441, id: 245584761)
[2021-12-23 11:41:28] (resource_pk: 0594181a2af3220afc37c03c9100994a_91538, id: 245572500), (resource_pk: f3ab726338c99bc9ce0e988498765bd2_91538, id: 245656044)
[2021-12-23 11:41:28] (resource_pk: 05a0c2879002b8d42710f6f5ec303d29_15441_ENV, id: 245572521), (resource_pk: e3b3f55c219c5224ba2e3a97a32a9b07_15441_ENV, id: 245650431)
[2021-12-23 11:41:28] (resource_pk: 06334c569076226c3a763912ca62a90a_15441_ENV, id: 245572708), (resource_pk: be6e390a3f38298ec98bf9e842e14032_15441_ENV, id: 245637390)
[2021-12-23 11:41:28] (resource_pk: 064961283b8d9d14831368a99046bb00_15441, id: 245572741), (resource_pk: d7ef645ccf9666f4d7a2e437874cb152_15441, id: 245646369)
[2021-12-23 11:41:28] (resource_pk: 066303bf1289be38e0e3c20ecae15d57_15441_ENV, id: 245572779), (resource_pk: cb7be998283e89820d6d7663a91acb5e_15441_ENV, id: 245641970)
[2021-12-23 11:41:28] (resource_pk: 067ba9b0c35d1f362a3ea1eb1ac14978_15441_ENV, id: 245572808), (resource_pk: 6180872a393e3c26c8945879af38ef11_15441_ENV, id: 245604528)
[2021-12-23 11:41:28] (resource_pk: 0680af65e63c35b798f38e729da8d3d3_15441_ENV, id: 245572821), (resource_pk: 216f09bc90e87a89519f538107df5755_15441_ENV, id: 245582129)
[2021-12-23 11:41:28] (resource_pk: 06823e084d0d16fb39ec71e7d6c04a7e_15441, id: 245572822), (resource_pk: 355478f2c5808783573480d8a783f846_15441, id: 245589019)
[2021-12-23 11:41:28] (resource_pk: 0708f033f6e23b5f8ee62e9b912d7d12_15441, id: 245573001), (resource_pk: a9e7957ed5940c1277ea008b26a21066_15441, id: 245630090)
[2021-12-23 11:41:28] (resource_pk: 07192ccf8ebae55137d162f2629614b2_15441, id: 245573021), (resource_pk: b76f0c7066dd18110e2420e767c20f41_15441, id: 245634849)
[2021-12-23 11:41:28] (resource_pk: 074acea250a4a5337a12809aca568043_15441_ENV, id: 245573093), (resource_pk: d5dd45ce4a77501f37a15a9a352ed496_15441_ENV, id: 245645673)
[2021-12-23 11:41:28] (resource_pk: 07508e156dc7c4f66b7cbf74e1d458f1_15441_ENV, id: 245573101), (resource_pk: 1d0031f18b1ce465b7db45fdfd7982c5_15441_ENV, id: 245580596)
[2021-12-23 11:41:28] (resource_pk: 076d2ca68ca22dc542b1b92e4114a395_15441_ENV, id: 245573133), (resource_pk: d65da1b1686aa4fb52585adc449d3f7d_15441_ENV, id: 245645826)
[2021-12-23 11:41:28] (resource_pk: 07772ee3b6c76007443a55a11dc9b6d0_15441_ENV, id: 245573154), (resource_pk: f24e4fc318852314195be38514248c53_15441_ENV, id: 245655588)
[2021-12-23 11:41:28] (resource_pk: 080e96194cf6244acf4971fbd5cbe6a6_15438_ENV, id: 245573342), (resource_pk: 806cd51037f2e1c79642fb0024a228e1_15438_ENV, id: 245615489)
[2021-12-23 11:41:28] (resource_pk: 081494cfdbd684cbad49d6f7b0ca7ec9_15441_ENV, id: 245573351), (resource_pk: ea8f40921ede2a1e689882b22ea903d0_15441_ENV, id: 245652862)
[2021-12-23 11:41:28] (resource_pk: 087e38d144d8517fd2053bb6dacd3bb0_15441, id: 245573482), (resource_pk: 0d8c15c41d523463649db843b01f4d41_15441, id: 245575267)
[2021-12-23 11:41:28] (resource_pk: 088afae1760b52d4c82b3ab943b226b8_15441, id: 245573494), (resource_pk: f18698b1b513a7ab8973cf73847e7ec6_15441, id: 245655305)
[2021-12-23 11:41:28] (resource_pk: 088b546f1ae096bb443058b4b8a0bbed_15441, id: 245573496), (resource_pk: d9fb662bd6618c29420851215582d196_15441, id: 245647084)
[2021-12-23 11:41:28] (resource_pk: 08c27a7f3bbe1cdd5715bcbe0cbe1481_15441_ENV, id: 245573579), (resource_pk: 0c6d58ac4247ef6e419f0e67a0a225a1_15441_ENV, id: 245574852)
[2021-12-23 11:41:28] (resource_pk: 08e7c3d7154e10a3aff30b41e63f16c1_91527, id: 245573634), (resource_pk: b6e72ed916e222e0fae497e8fa581e7d_91527, id: 245634667)
[2021-12-23 11:41:28] (resource_pk: 08e8cd2e5d1d71dc8e5795f21066a1ad_15441, id: 245573635), (resource_pk: 33a62c3dbf3d2fc084bf5115160a5c81_15441, id: 245588485)
[2021-12-23 11:41:28] (resource_pk: 08fa630b04bc3b36036bdf7951859339_15441, id: 245573657), (resource_pk: 6e3d4ea01e7c2d874d3671966c5d97cb_15441, id: 245609080)
[2021-12-23 11:41:28] (resource_pk: 090adcb382d8e50c30ac4821383d3b47_91116, id: 245573685), (resource_pk: 283996f20ac22fba99ba1bbf5266a7f1_91116, id: 245584557)
[2021-12-23 11:41:28] (resource_pk: 0976010f257468b96daea69dd18ae08a_15441_ENV, id: 245573831), (resource_pk: 95bcb6add61aeefc86665e9d03913c1a_15441_ENV, id: 245622978)
[2021-12-23 11:41:28] (resource_pk: 099f9e770a655679731df36516c8d735_91527_ENV, id: 245573890), (resource_pk: ffcd5cf3f54ecb000e93a574df34a02a_91527_ENV, id: 245660334)
[2021-12-23 11:41:28] (resource_pk: 09e2c4dc626cf595e2ba2c2e9d86726c_15441, id: 245573972), (resource_pk: f39ff0d1a3d362d3631fbf122ba2d5e2_15441, id: 245656031)
[2021-12-23 11:41:28] (resource_pk: 09e3d30b52274a8bca063af4d205910c_15441_ENV, id: 245573974), (resource_pk: 1bc9ddfae958f3bea3e22a72bfca82c3_15441_ENV, id: 245580169)
[2021-12-23 11:41:28] (resource_pk: 0a3dfe292c9c1c005cd2f7dc8c7759a5_15441_ENV, id: 245574092), (resource_pk: ed907451279f5c1ec4eaad0558ec7120_15441_ENV, id: 245653915)
[2021-12-23 11:41:28] (resource_pk: 0a46c994ab504a71845dcddfd2c3205f_15441_ENV, id: 245574117), (resource_pk: 1e641da0dc180576be5a8182061671e6_15441_ENV, id: 245581099)
[2021-12-23 11:41:28] (resource_pk: 0aacdbe36980b208bb9cde11003d04dc_15441, id: 245574244), (resource_pk: 6c06fa5f4a75a84976e7f48799afd4e3_15441, id: 245608321)
[2021-12-23 11:41:28] (resource_pk: 0acd1da0a1d343b547481ccc6d54f59b_15441, id: 245574277), (resource_pk: 1e5a555a93ca9bbe0b94a7278aaeff53_15441, id: 245581083)
[2021-12-23 11:41:28] (resource_pk: 0ad8ab9f50fa848c0a9fe247ee3c5c14_15441, id: 245574295), (resource_pk: 2fa89ddad2f258f4fe5f79059ad29d55_15441, id: 245587135)
[2021-12-23 11:41:28] (resource_pk: 0b016db7582cf25eebaf28778b99c78a_15441, id: 245574343), (resource_pk: 16fce11fe204206926238ec60d1ac7c4_15441, id: 245578518)
[2021-12-23 11:41:28] (resource_pk: 0b61ae595e1da4cbb69ba3981c70e180_15441, id: 245574478), (resource_pk: 87c00db9c7ed95c3916174ac7527885c_15441, id: 245618128)
[2021-12-23 11:41:28] (resource_pk: 0b93c3f13be2d5232e157240b258629d_15441, id: 245574547), (resource_pk: 7054de49ea9d04624f5bb2c6e3ebb99d_15441, id: 245609788)
[2021-12-23 11:41:28] (resource_pk: 0b96602f596f8c2b6672614c1fbb2482_15441_ENV, id: 245574552), (resource_pk: fa3caa50c7b74d826882ffc993caf0a4_15441_ENV, id: 245658388)
[2021-12-23 11:41:28] (resource_pk: 0bdce586c1e9423a0bf0d9e33f152e16_91155_ENV, id: 245574656), (resource_pk: 4bdcfef1ee55a9b0477bad06748fcd35_91155_ENV, id: 245596969)
[2021-12-23 11:41:28] (resource_pk: 0bff05d8521bd533241daceaf82203fe_15441, id: 245574700), (resource_pk: 836e52f8804c2ce6fcb2969c3e09157e_15441, id: 245616590)
[2021-12-23 11:41:28] (resource_pk: 0c04d51014dc99334c8aa954ebc9c4da_15438_ENV, id: 245574705), (resource_pk: e665b482c77db70c6112fa8c10e12915_15438_ENV, id: 245651416)
[2021-12-23 11:41:28] (resource_pk: 0c3ffca3e84f4425761d2b990c25e83c_15441_ENV, id: 245574790), (resource_pk: abd7a4d4c8ec2078ff610aad6f95d1e2_15441_ENV, id: 245630735)
[2021-12-23 11:41:28] (resource_pk: 0c416d0464e4b7433592c264f3d35034_15441_ENV, id: 245574795), (resource_pk: 94ceff2864cfaaae01c5d8a611fa8180_15441_ENV, id: 245622655)
[2021-12-23 11:41:28] (resource_pk: 0c6de952c9ee9df01baad858f4d1517e_15441, id: 245574853), (resource_pk: d1042ee5404067801873d0e319082df6_15441, id: 245644031)
[2021-12-23 11:41:28] (resource_pk: 0c8a5eda55631ecdf000b314a37f809e_15441, id: 245574890), (resource_pk: 1fe342dcc4c5390113415be998c1f128_15441, id: 245581615)
[2021-12-23 11:41:28] (resource_pk: 0cbb0067b10dddfc92c421fb3ea42f64_15441, id: 245574976), (resource_pk: a1e9c71ac4de76ae23d37d38886ffd84_15441, id: 245627310)
[2021-12-23 11:41:28] (resource_pk: 0cbecacd92b4d37271af695aae56f192_15441_ENV, id: 245574982), (resource_pk: 52bc344f597d37919ec79bae26ba742c_15441_ENV, id: 245599300)
[2021-12-23 11:41:28] (resource_pk: 0cca9a9c1d5aba264c0b371bfcea7cb8_15441, id: 245575004), (resource_pk: 58365e6a3d8b44942891e164ff3a55af_15441, id: 245601240)
[2021-12-23 11:41:28] (resource_pk: 0cd24d55d33ed2b067546d84327e033b_15441_ENV, id: 245575014), (resource_pk: 89bb92f13216421d3e707b5723754ebb_15441_ENV, id: 245618833)
[2021-12-23 11:41:28] (resource_pk: 0d09bea0871935e8fe76590de53bc21b_15441, id: 245575076), (resource_pk: 58a16684ee8b8d189e614809838a0a3a_15441, id: 245601395)
[2021-12-23 11:41:28] (resource_pk: 0d2496d29561fac1ba322c01b5c3bd65_15441, id: 245575118), (resource_pk: c35f26e0854570eddbfe7eb8be838793_15441, id: 245639133)
[2021-12-23 11:41:28] (resource_pk: 0d2cea0780a9e9f17d4a20371215f5ff_15441, id: 245575135), (resource_pk: 8f920392a7ba1005cceb92f7508db292_15441, id: 245620845)
[2021-12-23 11:41:28] (resource_pk: 0d331d6ef069a9194d06887dbbb2009f_15441_ENV, id: 245575138), (resource_pk: 78253b0acc92d1f88516089e3563dc1f_15441_ENV, id: 245612578)
[2021-12-23 11:41:28] (resource_pk: 0d377ed85154944a4a95fe6b14183215_15441_ENV, id: 245575146), (resource_pk: 77cffe491fb2022a6760514fe4a2e1d2_15441_ENV, id: 245612464)
[2021-12-23 11:41:28] (resource_pk: 0d3c3b43addb0b6138c76996bbcaa265_91535, id: 245575158), (resource_pk: be96eabf48f6aaf6978868aef8012716_91535, id: 245637444)
[2021-12-23 11:41:28] (resource_pk: 0d4441db76e30c594ca0d34d19e54cff_15441_ENV, id: 245575170), (resource_pk: d3bd82228bbe650c6d5fab4cc3c0dfd4_15441_ENV, id: 245644928)
[2021-12-23 11:41:28] (resource_pk: 0db9f4d6072795d1bfb273e5586d51d6_15441, id: 245575315), (resource_pk: 3932c0f9837f2f4c1a6ac76eabfc1fb2_15441, id: 245590377)
[2021-12-23 11:41:28] (resource_pk: 0dc0fe220fb4f059709197d4bbfa7005_15441, id: 245575325), (resource_pk: 51cc215996d149e637bb28fafb423c2a_15441, id: 245599006)
[2021-12-23 11:41:28] (resource_pk: 0dd1ffa338922258f96f41839f670493_15441_ENV, id: 245575352), (resource_pk: 3210be16cf2b6a1cdb96623f82b5ed29_15441_ENV, id: 245587956)
[2021-12-23 11:41:28] (resource_pk: 0def96cbf9c5ecd96f897f55bab6f26b_15441_ENV, id: 245575387), (resource_pk: ba29edd6095b0635a1c712d292d25c50_15441_ENV, id: 245635837)
[2021-12-23 11:41:28] (resource_pk: 0df226a34ac4fb467a9b730a6f02ee85_15441_ENV, id: 245575393), (resource_pk: e02c77853fd78b3d0668ebd363baf629_15441_ENV, id: 245649209)
[2021-12-23 11:41:28] (resource_pk: 0e0ee18695d387a8ed6c68ae705b16ab_15441_ENV, id: 245575433), (resource_pk: 3939fb0f2912e6e1db1b0a528b81ea36_15441_ENV, id: 245590384)
[2021-12-23 11:41:28] (resource_pk: 0e44b504647f71b1a3ba5c7a98df8ffb_15441_ENV, id: 245575493), (resource_pk: acc7bb1d850c2189ef94113c1c7c48c1_15441_ENV, id: 245631077)
[2021-12-23 11:41:28] (resource_pk: 0ede32e475ab429a9b02845e565bfc78_15441, id: 245575722), (resource_pk: 8a52168342a1fec15e49355a5c39c9d1_15441, id: 245619021)
[STOP] [2021-12-23 11:41:28] calculate_statistics
[START] [2021-12-23 11:41:28] complete_harvest_instance
[START] [2021-12-23 11:41:28] overall_tsv_creation
[INFO] [2021-12-23 11:41:29] Processing group of 21588 in 3 batches of 10000
[INFO] [2021-12-23 11:44:14] 41842 Traits (unfiltered)...
[INFO] [2021-12-23 11:51:31] 41842 Traits (filtered)...
[INFO] [2021-12-23 11:52:02] 2672 Associations (filtered)...
[INFO] [2021-12-23 11:52:06] 38 metadata added.
[INFO] [2021-12-23 11:52:06] 0 metadata added.
[INFO] [2021-12-23 11:54:35] 41358 Traits (unfiltered)...
[INFO] [2021-12-23 12:01:48] 41358 Traits (filtered)...
[INFO] [2021-12-23 12:02:19] 2614 Associations (filtered)...
[INFO] [2021-12-23 12:02:23] 35 metadata added.
[INFO] [2021-12-23 12:02:23] 0 metadata added.
[INFO] [2021-12-23 12:04:04] 6704 Traits (unfiltered)...
[INFO] [2021-12-23 12:05:49] 6704 Traits (filtered)...
[INFO] [2021-12-23 12:05:58] 412 Associations (filtered)...
[INFO] [2021-12-23 12:05:59] 4 metadata added.
[INFO] [2021-12-23 12:05:59] 0 metadata added.
[INFO] [2021-12-23 12:06:38] Average Time: 434.463
[INFO] [2021-12-23 12:06:38] Total Time: 25m10s
[STOP] [2021-12-23 12:06:38] overall_tsv_creation
[INFO] [2021-12-23 12:06:38] Done. Check your files:
[INFO] [2021-12-23 12:06:39] (16993 lines) /app/public/data/n_am_flora/publish_bibliographic_citations.tsv
[INFO] [2021-12-23 12:06:41] (21588 lines) /app/public/data/n_am_flora/publish_nodes.tsv
[INFO] [2021-12-23 12:06:42] (21588 lines) /app/public/data/n_am_flora/publish_scientific_names.tsv
[INFO] [2021-12-23 12:06:43] (16993 lines) /app/public/data/n_am_flora/publish_articles.tsv
[INFO] [2021-12-23 12:06:44] (16993 lines) /app/public/data/n_am_flora/publish_content_sections.tsv
[INFO] [2021-12-23 12:06:46] (95603 lines) /app/public/data/n_am_flora/publish_traits.tsv
[INFO] [2021-12-23 12:06:47] (78 lines) /app/public/data/n_am_flora/publish_metadata.tsv
[STOP] [2021-12-23 12:06:47] complete_harvest_instance
[START] [2021-12-23 12:06:47] completed
[STOP] [2021-12-23 12:06:47] completed
[STOP] [2021-12-23 12:06:47] logged process, took 2998.81

Latest Process

23 Dec 11:16 EXITED currently processing 0