Harvest for TRY summarized records Created 27 May 15:43

Stage: completed
Fetched: 27 May 15:43
Validated: 27 May 15:44
Deltas Created 27 May 15:45
Units Normalized: 27 May 16:36
Ancestry Built: 27 May 16:13
Nodes Matched: 27 May 16:29
Names Parsed: 27 May 16:14
New Models Stored: 27 May 16:09
Indexed: 27 May 16:31
Completed: 27 May 18:26
Time to Harvest: 3 minutes

Harvesting Log

(339 lines)
[INFO] [2021-05-27 15:43:39] Created harvest instance #3894
[STOP] [2021-05-27 15:43:39] create_harvest_instance
[START] [2021-05-27 15:43:39] fetch_files
[STOP] [2021-05-27 15:43:39] fetch_files
[START] [2021-05-27 15:43:39] validate_each_file
[INFO] [2021-05-27 15:43:39] Looping over 4 formats...
[INFO] [2021-05-27 15:43:39] ...refs (/app/public/data/try_summarized_r/reference.tab)
[INFO] [2021-05-27 15:43:40] Valid: /app/public/converted_csv/try_summarized_r_refs_3894.csv (182 lines)
[INFO] [2021-05-27 15:43:40] ...nodes (/app/public/data/try_summarized_r/taxon.tab)
[INFO] [2021-05-27 15:43:42] Valid: /app/public/converted_csv/try_summarized_r_nodes_3894.csv (103373 lines)
[INFO] [2021-05-27 15:43:42] ...occurrences (/app/public/data/try_summarized_r/occurrence.tab)
[INFO] [2021-05-27 15:43:45] Valid: /app/public/converted_csv/try_summarized_r_occurrences_3894.csv (164440 lines)
[INFO] [2021-05-27 15:43:45] ...measurements (/app/public/data/try_summarized_r/measurement_or_fact_specific.tab)
[INFO] [2021-05-27 15:44:33] Valid: /app/public/converted_csv/try_summarized_r_measurements_3894.csv (548631 lines)
[STOP] [2021-05-27 15:44:33] validate_each_file
[START] [2021-05-27 15:44:33] convert_to_csv
[INFO] [2021-05-27 15:44:33] Looping over 4 formats...
[INFO] [2021-05-27 15:44:33] ...refs (/app/public/data/try_summarized_r/reference.tab)
[CMD] [2021-05-27 15:44:33] /usr/bin/sort /app/public/converted_csv/try_summarized_r_refs_3894.csv > /app/public/converted_csv/try_summarized_r_refs_3894.csv_sorted
[INFO] [2021-05-27 15:44:35] Converted: /app/public/converted_csv/try_summarized_r_refs_3894.csv (182 lines)
[INFO] [2021-05-27 15:44:35] ...nodes (/app/public/data/try_summarized_r/taxon.tab)
[CMD] [2021-05-27 15:44:35] /usr/bin/sort /app/public/converted_csv/try_summarized_r_nodes_3894.csv > /app/public/converted_csv/try_summarized_r_nodes_3894.csv_sorted
[INFO] [2021-05-27 15:44:36] Converted: /app/public/converted_csv/try_summarized_r_nodes_3894.csv (103373 lines)
[INFO] [2021-05-27 15:44:36] ...occurrences (/app/public/data/try_summarized_r/occurrence.tab)
[CMD] [2021-05-27 15:44:36] /usr/bin/sort /app/public/converted_csv/try_summarized_r_occurrences_3894.csv > /app/public/converted_csv/try_summarized_r_occurrences_3894.csv_sorted
[INFO] [2021-05-27 15:44:37] Converted: /app/public/converted_csv/try_summarized_r_occurrences_3894.csv (164440 lines)
[INFO] [2021-05-27 15:44:37] ...measurements (/app/public/data/try_summarized_r/measurement_or_fact_specific.tab)
[CMD] [2021-05-27 15:44:37] /usr/bin/sort /app/public/converted_csv/try_summarized_r_measurements_3894.csv > /app/public/converted_csv/try_summarized_r_measurements_3894.csv_sorted
[INFO] [2021-05-27 15:44:43] Converted: /app/public/converted_csv/try_summarized_r_measurements_3894.csv (548631 lines)
[STOP] [2021-05-27 15:44:43] convert_to_csv
[START] [2021-05-27 15:44:43] calculate_delta
[INFO] [2021-05-27 15:44:43] Looping over 4 formats...
[INFO] [2021-05-27 15:44:43] ...refs (/app/public/data/try_summarized_r/reference.tab)
[CMD] [2021-05-27 15:44:43] echo "0a" > /app/public/diff/try_summarized_r_refs_3894.diff
[CMD] [2021-05-27 15:44:44] tail -n +1 /app/public/converted_csv/try_summarized_r_refs_3894.csv >> /app/public/diff/try_summarized_r_refs_3894.diff
[CMD] [2021-05-27 15:44:46] echo "." >> /app/public/diff/try_summarized_r_refs_3894.diff
[INFO] [2021-05-27 15:44:47] Created diff: /app/public/diff/try_summarized_r_refs_3894.diff (184 lines)
[INFO] [2021-05-27 15:44:47] ...nodes (/app/public/data/try_summarized_r/taxon.tab)
[CMD] [2021-05-27 15:44:47] echo "0a" > /app/public/diff/try_summarized_r_nodes_3894.diff
[CMD] [2021-05-27 15:44:48] tail -n +1 /app/public/converted_csv/try_summarized_r_nodes_3894.csv >> /app/public/diff/try_summarized_r_nodes_3894.diff
[CMD] [2021-05-27 15:44:50] echo "." >> /app/public/diff/try_summarized_r_nodes_3894.diff
[INFO] [2021-05-27 15:44:51] Created diff: /app/public/diff/try_summarized_r_nodes_3894.diff (103375 lines)
[INFO] [2021-05-27 15:44:51] ...occurrences (/app/public/data/try_summarized_r/occurrence.tab)
[CMD] [2021-05-27 15:44:51] echo "0a" > /app/public/diff/try_summarized_r_occurrences_3894.diff
[CMD] [2021-05-27 15:44:52] tail -n +1 /app/public/converted_csv/try_summarized_r_occurrences_3894.csv >> /app/public/diff/try_summarized_r_occurrences_3894.diff
[CMD] [2021-05-27 15:44:54] echo "." >> /app/public/diff/try_summarized_r_occurrences_3894.diff
[INFO] [2021-05-27 15:44:55] Created diff: /app/public/diff/try_summarized_r_occurrences_3894.diff (164442 lines)
[INFO] [2021-05-27 15:44:55] ...measurements (/app/public/data/try_summarized_r/measurement_or_fact_specific.tab)
[CMD] [2021-05-27 15:44:55] echo "0a" > /app/public/diff/try_summarized_r_measurements_3894.diff
[CMD] [2021-05-27 15:44:56] tail -n +1 /app/public/converted_csv/try_summarized_r_measurements_3894.csv >> /app/public/diff/try_summarized_r_measurements_3894.diff
[CMD] [2021-05-27 15:44:59] echo "." >> /app/public/diff/try_summarized_r_measurements_3894.diff
[INFO] [2021-05-27 15:45:03] Created diff: /app/public/diff/try_summarized_r_measurements_3894.diff (548633 lines)
[STOP] [2021-05-27 15:45:03] calculate_delta
[START] [2021-05-27 15:45:03] parse_diff_and_store
[INFO] [2021-05-27 15:45:03] Handling diff: /app/public/diff/try_summarized_r_refs_3894.diff (184 lines)
[INFO] [2021-05-27 15:45:04] Loading refs diff file into memory (184 /app/public/diff/try_summarized_r_refs_3894.diff lines)...
[INFO] [2021-05-27 15:45:06] Handling diff: /app/public/diff/try_summarized_r_nodes_3894.diff (103375 lines)
[INFO] [2021-05-27 15:45:07] Loading nodes diff file into memory (103375 /app/public/diff/try_summarized_r_nodes_3894.diff lines)...
[WARN] [2021-05-27 15:45:14] Filtered Scientific Name `Carex  comans` to `Carex comans`
[WARN] [2021-05-27 15:45:14] Filtered Scientific Name `Cephalotaxus harringtonii var.  wilsoniana` to `Cephalotaxus harringtonii var. wilsoniana`
[WARN] [2021-05-27 15:45:34] Filtered Scientific Name `Sida  limestone` to `Sida limestone`
[INFO] [2021-05-27 15:45:38] Handling diff: /app/public/diff/try_summarized_r_occurrences_3894.diff (164442 lines)
[INFO] [2021-05-27 15:45:40] Loading occurrences diff file into memory (164442 /app/public/diff/try_summarized_r_occurrences_3894.diff lines)...
[INFO] [2021-05-27 15:46:13] Handling diff: /app/public/diff/try_summarized_r_measurements_3894.diff (548633 lines)
[INFO] [2021-05-27 15:46:14] Loading measurements diff file into memory (548633 /app/public/diff/try_summarized_r_measurements_3894.diff lines)...
[INFO] [2021-05-27 15:58:04] Storing 182 References
[INFO] [2021-05-27 15:58:04] Processing group of 182 in 1 groups of 1000
[INFO] [2021-05-27 15:58:04] Average Time: 0.04
[INFO] [2021-05-27 15:58:04] Total Time: 1s
[INFO] [2021-05-27 15:58:04] Storing 103791 ScientificNames
[INFO] [2021-05-27 15:58:04] Processing group of 103791 in 104 groups of 1000
[INFO] [2021-05-27 15:58:36] Average Time: 0.309
[INFO] [2021-05-27 15:58:36] Total Time: 33s
[INFO] [2021-05-27 15:58:36] last 3 / first 3: 0.88
[INFO] [2021-05-27 15:58:36] Std.Dev: 0.06324555320336758; Max: 0.58
[INFO] [2021-05-27 15:58:36] Storing 103791 Nodes
[INFO] [2021-05-27 15:58:36] Processing group of 103791 in 104 groups of 1000
[INFO] [2021-05-27 15:59:31] Average Time: 0.522
[INFO] [2021-05-27 15:59:31] Total Time: 55s
[INFO] [2021-05-27 15:59:31] last 3 / first 3: 0.28
[INFO] [2021-05-27 15:59:31] Std.Dev: 1.418097316829843; Max: 13.33
[INFO] [2021-05-27 15:59:31] Storing 164440 Occurrences
[INFO] [2021-05-27 15:59:31] Processing group of 164440 in 165 groups of 1000
[INFO] [2021-05-27 15:59:54] Average Time: 0.135
[INFO] [2021-05-27 15:59:54] Total Time: 23s
[INFO] [2021-05-27 15:59:54] last 3 / first 3: 7.5
[INFO] [2021-05-27 15:59:54] Std.Dev: 0.1414213562373095; Max: 1.33
[INFO] [2021-05-27 15:59:54] Storing 548631 Traits
[INFO] [2021-05-27 15:59:54] Processing group of 548631 in 549 groups of 1000
[INFO] [2021-05-27 16:03:59] Average Time: 0.441
[INFO] [2021-05-27 16:03:59] Total Time: 4m5s
[INFO] [2021-05-27 16:03:59] last 3 / first 3: 0.17
[INFO] [2021-05-27 16:03:59] Std.Dev: 0.8148619514985346; Max: 13.82
[INFO] [2021-05-27 16:03:59] Storing 1387572 MetaTraits
[INFO] [2021-05-27 16:03:59] Processing group of 1387572 in 1388 groups of 1000
[INFO] [2021-05-27 16:09:19] Average Time: 0.225
[INFO] [2021-05-27 16:09:19] Total Time: 5m20s
[INFO] [2021-05-27 16:09:19] last 3 / first 3: 0.85
[INFO] [2021-05-27 16:09:19] Std.Dev: 0.4312771730569565; Max: 7.46
[INFO] [2021-05-27 16:09:19] Storing 284410 TraitsReferences
[INFO] [2021-05-27 16:09:19] Processing group of 284410 in 285 groups of 1000
[INFO] [2021-05-27 16:09:40] Average Time: 0.072
[INFO] [2021-05-27 16:09:40] Total Time: 22s
[INFO] [2021-05-27 16:09:40] last 3 / first 3: 0.71
[INFO] [2021-05-27 16:09:40] Std.Dev: 0.044721359549995794; Max: 0.38
[STOP] [2021-05-27 16:09:40] parse_diff_and_store
[START] [2021-05-27 16:09:40] resolve_keys
[INFO] [2021-05-27 16:10:19] Occurrences to nodes (through scientific_names)...
[INFO] [2021-05-27 16:10:23] traits to occurrences...
[INFO] [2021-05-27 16:10:40] traits to nodes (through occurrences)...
[INFO] [2021-05-27 16:10:52] Traits to sex term...
[INFO] [2021-05-27 16:10:58] Traits to lifestage term...
[INFO] [2021-05-27 16:11:04] MetaTraits to traits...
[INFO] [2021-05-27 16:11:38] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-05-27 16:11:54] Assocs to occurrences...
[INFO] [2021-05-27 16:11:54] Assocs to nodes...
[INFO] [2021-05-27 16:11:54] Assoc to sex term...
[INFO] [2021-05-27 16:11:54] Assoc to lifestage term...
[INFO] [2021-05-27 16:11:54] MetaAssoc to assocs...
[STOP] [2021-05-27 16:11:54] resolve_keys
[START] [2021-05-27 16:11:54] hold_for_later_1
[STOP] [2021-05-27 16:11:54] hold_for_later_1
[START] [2021-05-27 16:11:54] hold_for_later_2
[STOP] [2021-05-27 16:11:54] hold_for_later_2
[START] [2021-05-27 16:11:54] resolve_missing_parents
[STOP] [2021-05-27 16:12:01] resolve_missing_parents
[START] [2021-05-27 16:12:01] rebuild_nodes
[START] [2021-05-27 16:12:01] Flattener#flatten
[START] [2021-05-27 16:12:01] Flattener#study_resource
[START] [2021-05-27 16:12:02] Flattener#build_ancestry
[STOP] [2021-05-27 16:12:45] Flattener#build_ancestry
[INFO] [2021-05-27 16:12:45] 103791 ancestry keys
[START] [2021-05-27 16:12:45] build_node_ancestors
[INFO] [2021-05-27 16:12:45] old ancestors deleted.
[STOP] [2021-05-27 16:12:57] build_node_ancestors
[START] [2021-05-27 16:13:02] Flattener#propagate_ancestor_ids
[STOP] [2021-05-27 16:13:06] Flattener#propagate_ancestor_ids
[STOP] [2021-05-27 16:13:06] Flattener#flatten
[STOP] [2021-05-27 16:13:06] rebuild_nodes
[START] [2021-05-27 16:13:06] resolve_missing_media_owners
[STOP] [2021-05-27 16:13:06] resolve_missing_media_owners
[START] [2021-05-27 16:13:06] sanitize_media_verbatims
[STOP] [2021-05-27 16:13:06] sanitize_media_verbatims
[START] [2021-05-27 16:13:06] queue_downloads
[STOP] [2021-05-27 16:13:06] queue_downloads
[START] [2021-05-27 16:13:06] parse_names
[WARN] [2021-05-27 16:13:06] I see 103791 names which still need to be parsed.
[WARN] [2021-05-27 16:14:17] I see 17 names which still need to be parsed.
[STOP] [2021-05-27 16:14:19] parse_names
[START] [2021-05-27 16:14:19] denormalize_canonical_names_to_nodes
[STOP] [2021-05-27 16:14:21] denormalize_canonical_names_to_nodes
[START] [2021-05-27 16:14:21] match_nodes
[START] [2021-05-27 16:14:21] map_all_nodes_to_pages
[STOP] [2021-05-27 16:29:49] map_all_nodes_to_pages
[INFO] [2021-05-27 16:29:49] 5228 Unmatched nodes (of 103791)! That's too many to output. Full list in /app/public/data/try_summarized_r/unmatched_nodes.txt ; First 10: Canonical: Abacaba palm; Node#94709346; ResourceID: ABACABA PALM; Canonical: Abdilobarana; Node#94709347; ResourceID: ABDILOBARANA sp; Canonical: Abiu casca; Node#94709349; ResourceID: ABIU CASCA; Canonical: Abiu cutiti; Node#94709350; ResourceID: ABIU CUTITI; Canonical: Abiu pitomba; Node#94709351; ResourceID: ABIU PITOMBA; Canonical: Abiurana; Node#94709352; ResourceID: ABIURANA sp; Canonical: Acariquara; Node#94709353; ResourceID: ACARIQUARA sp; Canonical: Acariquarana; Node#94709354; ResourceID: ACARIQUARANA sp; Canonical: Achicha; Node#94709355; ResourceID: ACHICHA sp; Canonical: Achua; Node#94709356; ResourceID: ACHUA sp
[START] [2021-05-27 16:29:49] update_nodes
[STOP] [2021-05-27 16:29:50] update_nodes
[STOP] [2021-05-27 16:29:50] match_nodes
[START] [2021-05-27 16:29:50] reindex_search
[STOP] [2021-05-27 16:31:06] reindex_search
[START] [2021-05-27 16:31:06] normalize_units
[STOP] [2021-05-27 16:36:30] normalize_units
[START] [2021-05-27 16:36:30] calculate_statistics
[2021-05-27 16:36:36] (NEAR) DUPLICATE TRAITS FOUND! There are only 531601 (of 548631 total) unique traits.
[2021-05-27 17:01:06] (Near) duplicate trait pairs (up to 100):
[2021-05-27 17:01:06] (resource_pk: 00001c24d82c8c911ad07f7c1897c201_try, id: 211881460), (resource_pk: 58428a3e1e668f89ecc07ec3906dd562_try, id: 212070710)
[2021-05-27 17:01:06] (resource_pk: 0000beebd73fc1770971c40b69215b7e_try, id: 211881467), (resource_pk: a96ea1db0e5aaea4174726b03c1b0437_try, id: 212244647)
[2021-05-27 17:01:06] (resource_pk: 0003c654ad70654407e18c7b22326f71_try, id: 211881489), (resource_pk: 0484fd0f8734e976f6aa3f3c431fddc4_try, id: 211891284)
[2021-05-27 17:01:06] (resource_pk: 000549478620f3acc89fca51b55775b2_try, id: 211881502), (resource_pk: f1dc5a4a3832c9bcc6fe89e1443e95af_try, id: 212399725)
[2021-05-27 17:01:06] (resource_pk: 000d1feb23fe0b9613a4933c4eedbac9_try, id: 211881566), (resource_pk: 17ec7c4e71ff8330c3afce4983a068bd_try, id: 211932649)
[2021-05-27 17:01:06] (resource_pk: 000f645a34fbaa91bdd3229ec96b6941_try, id: 211881581), (resource_pk: 701688e5cada6f2c8ef5590848a1e556_try, id: 212121643)
[2021-05-27 17:01:06] (resource_pk: 00122b5f26588e3544be88d0ede0af52_try, id: 211881595), (resource_pk: d4df530358d7635df4ece4f0970d74b3_try, id: 212337625)
[2021-05-27 17:01:06] (resource_pk: 00130d322f39ae56f1542fa483ce17f2_try, id: 211881603), (resource_pk: af0a709c4b583e4801aa1f94b207a3fc_try, id: 212256591)
[2021-05-27 17:01:06] (resource_pk: 001477c4a9e4c727c29f060880d521b1_try, id: 211881619), (resource_pk: 86337cc2829517dad7ea2d5a848a6ca3_try, id: 212169149)
[2021-05-27 17:01:06] (resource_pk: 001574362441f3d95776204842e4f232_try, id: 211881628), (resource_pk: ef2d864d776d88981f1284ca2b61cfb5_try, id: 212393999)
[2021-05-27 17:01:06] (resource_pk: 001852b198960c6b3a7ba221479b2439_try, id: 211881655), (resource_pk: d78b6282cd9cc5d5089a8b7b86036187_try, id: 212343400)
[2021-05-27 17:01:06] (resource_pk: 0018ffd1a9639639beb4900c07cc99c6_try, id: 211881660), (resource_pk: 85bc669828fea6919078be5a803f1852_try, id: 212168097)
[2021-05-27 17:01:06] (resource_pk: 001b34ea136874e2dbe1a6bb4f6346f7_try, id: 211881677), (resource_pk: ee1b6896cccd410e2cf9d2674d23d6db_try, id: 212391689)
[2021-05-27 17:01:06] (resource_pk: 001b58d26d84a5c474b924e0dbe47f1d_try, id: 211881679), (resource_pk: ae60e48e8ab121e8e9dd02b3296396ba_try, id: 212255213)
[2021-05-27 17:01:06] (resource_pk: 001b91e5989fc828ed426010755ba26b_try, id: 211881684), (resource_pk: 713d0fd965ddc73c9639d0781a93bdd7_try, id: 212124149)
[2021-05-27 17:01:06] (resource_pk: 00223f2e409b3a29756a3db6876b8e1a_try, id: 211881732), (resource_pk: 4026b5db8117ed62e14de7cd6fdae808_try, id: 212018907)
[2021-05-27 17:01:06] (resource_pk: 00224e2661589007402907571132938d_try, id: 211881735), (resource_pk: 2d584db45a9e1b01e3f3ab9d753c78f5_try, id: 211978624)
[2021-05-27 17:01:06] (resource_pk: 0023efffbb5a9a95413a7df230816b47_try, id: 211881753), (resource_pk: 2a451f7a150c78273c1bcde7414fc1ea_try, id: 211972154)
[2021-05-27 17:01:06] (resource_pk: 002580c982caab79103e928b900d92f1_try, id: 211881769), (resource_pk: cf223c49bbcf33e54aac33b02cb4a816_try, id: 212325369)
[2021-05-27 17:01:06] (resource_pk: 0027c49783372a1c81e87b41bef10396_try, id: 211881791), (resource_pk: fb4c0519beef77ccac269e447d983456_try, id: 212419965)
[2021-05-27 17:01:06] (resource_pk: 002a4d455dcf8996025771ac8cf189d5_try, id: 211881807), (resource_pk: f8d52479a3ae56248b89250b72ea4f87_try, id: 212414701)
[2021-05-27 17:01:06] (resource_pk: 002cc6c413cc949829863b485e8585d5_try, id: 211881821), (resource_pk: 34117a61b2bd2ed22d86b6a9e070fb8e_try, id: 211993184)
[2021-05-27 17:01:06] (resource_pk: 002d07d4be91bf4f688bd90bcb1f1d09_try, id: 211881824), (resource_pk: f0cff0bc5f1e4153d59aa46fd28ba74e_try, id: 212397463)
[2021-05-27 17:01:06] (resource_pk: 002d60d345aca5f8687faa3fc0e7b7b4_try, id: 211881826), (resource_pk: a8cd8cce56496553976a292805548c40_try, id: 212243262)
[2021-05-27 17:01:06] (resource_pk: 002d7f7241c4c8f3cd3d337abeda5243_try, id: 211881829), (resource_pk: ae5aa439250e0a238d64ebfe8086989b_try, id: 212255153)
[2021-05-27 17:01:06] (resource_pk: 002ebdd843146c0fb96079c9ddd8fee4_try, id: 211881835), (resource_pk: b9086382f27df4ee45df084b51c7b5c6_try, id: 212278263)
[2021-05-27 17:01:06] (resource_pk: 0035f226cdda87df18c32bba15ade7d8_try, id: 211881897), (resource_pk: e8c09ada20d42e22dcab7385d80fc068_try, id: 212380251)
[2021-05-27 17:01:06] (resource_pk: 003604ee14cf555f8f2c600ff90b5bc5_try, id: 211881898), (resource_pk: af293699e15c50ecc23b97a4b7286a1c_try, id: 212256882)
[2021-05-27 17:01:06] (resource_pk: 00361921823bb9deae48dbb985874967_try, id: 211881899), (resource_pk: ce82ec3dfa96904e70f11c535df6c714_try, id: 212324023)
[2021-05-27 17:01:06] (resource_pk: 003d230e5b5795ab32078c5981e5db63_try, id: 211881951), (resource_pk: f915fcc74b8155872da57d03caef18a6_try, id: 212415227)
[2021-05-27 17:01:06] (resource_pk: 003deb3225ee7e469766c456d9c5b57a_try, id: 211881961), (resource_pk: 022c3768f5f4e5f0da74ec00cce569eb_try, id: 211886193)
[2021-05-27 17:01:06] (resource_pk: 00431a117afb60f9a05129f698794591_try, id: 211882001), (resource_pk: 7ee45943e7089168b679f6ca8b4ab955_try, id: 212153440)
[2021-05-27 17:01:06] (resource_pk: 0044f122b06eefe6e209f7a75910e0ac_try, id: 211882021), (resource_pk: 8dfe3f4f54961d4d4862e625b182f2eb_try, id: 212185775)
[2021-05-27 17:01:06] (resource_pk: 004559f240cb2dae308ae86a19ddaa90_try, id: 211882027), (resource_pk: 14088eb21feb7d6783752e38c5d6a834_try, id: 211924286)
[2021-05-27 17:01:06] (resource_pk: 00474b7b96b12740c87258916fd9e169_try, id: 211882043), (resource_pk: 06cb4b6e719abd05ea62aa6582154fd7_try, id: 211896165)
[2021-05-27 17:01:06] (resource_pk: 00475eff071ba2acb39a3506b5908db4_try, id: 211882044), (resource_pk: 0dcdaa580f61550d86e7ec4f31cf33c1_try, id: 211911207)
[2021-05-27 17:01:06] (resource_pk: 004a8392466b7c70810a41b19dea4d14_try, id: 211882079), (resource_pk: d9688d1c207c3886ed75a4c880727594_try, id: 212347488)
[2021-05-27 17:01:06] (resource_pk: 004bde353b48483c17df3ea2acaa88bf_try, id: 211882086), (resource_pk: ccfcc719b6e42402a77b3bbd72bf71f9_try, id: 212320790)
[2021-05-27 17:01:06] (resource_pk: 004e1777da553a0f9a4f2aa6d8f5749c_try, id: 211882100), (resource_pk: e4c46845dc9963d905b5f3802b936c79_try, id: 212371768)
[2021-05-27 17:01:06] (resource_pk: 004ff538b214fe4175ea7ef5b898a774_try, id: 211882114), (resource_pk: 8f71df9e0eb0ea8397852f0d0fd3ca4e_try, id: 212188886)
[2021-05-27 17:01:06] (resource_pk: 0052452aa8a571c585121acd31f310a7_try, id: 211882137), (resource_pk: 043da8a09b2ad0305d40a5839d6ea72e_try, id: 211890665)
[2021-05-27 17:01:06] (resource_pk: 005eaf7601ecdc24711e641a6054ce77_try, id: 211882246), (resource_pk: d0a7e7d97b140c55d09262921970a11a_try, id: 212328542)
[2021-05-27 17:01:06] (resource_pk: 0060b22088af7c1dff59e1c5e992ac24_try, id: 211882263), (resource_pk: 725c9a050f791f1f9a2d76fbd9bceee2_try, id: 212126493)
[2021-05-27 17:01:06] (resource_pk: 0065598c3c588d38c6894ab264f21723_try, id: 211882295), (resource_pk: 5d504a19a1ba0d666b3b3863641e4587_try, id: 212081437)
[2021-05-27 17:01:06] (resource_pk: 00686783476d46841a6a5449040b814a_try, id: 211882315), (resource_pk: 24cdfae16c751bb0db45b24dceafccb6_try, id: 211960405)
[2021-05-27 17:01:06] (resource_pk: 0068e9fc56476912ffce289567cf7354_try, id: 211882318), (resource_pk: 01a049c17356efd33a303d4a11234aff_try, id: 211884989)
[2021-05-27 17:01:06] (resource_pk: 00696e2f24f1d93fedd682c30ba59e5e_try, id: 211882321), (resource_pk: c47698e6d0b729df068272d9160a0afe_try, id: 212302672)
[2021-05-27 17:01:06] (resource_pk: 006b9716daaa9d689f80689d17fb5fbe_try, id: 211882333), (resource_pk: 4b5a38bce4b246d695db30613d4b91fa_try, id: 212043120)
[2021-05-27 17:01:06] (resource_pk: 006d6582e1f749fea4d24e456151fdf2_try, id: 211882348), (resource_pk: e696c674f0601523dd1bd26349d32c4a_try, id: 212375658)
[2021-05-27 17:01:06] (resource_pk: 007149681bfb1d3f6423967ecf9261fd_try, id: 211882390), (resource_pk: ca79fd0743399dcf80629308d762a923_try, id: 212315495)
[2021-05-27 17:01:06] (resource_pk: 00714b18dd5f944302907f4994b84b15_try, id: 211882391), (resource_pk: 49d0829c1d41fddddadc22a3c1cb7229_try, id: 212039853)
[2021-05-27 17:01:06] (resource_pk: 00728eed35b7291190303fe6dbd3ef1c_try, id: 211882399), (resource_pk: eb25a845b5526ab4c5d5131f26f586b5_try, id: 212385422)
[2021-05-27 17:01:06] (resource_pk: 007319334e92b82361ad363b429ba156_try, id: 211882404), (resource_pk: dd5bd9dff9b18f61c9487aa02c952451_try, id: 212355864)
[2021-05-27 17:01:06] (resource_pk: 007559ab3c1529a6bcddc12365f7541b_try, id: 211882420), (resource_pk: 0e0d410e3ba37762231efb4ca31eba3d_try, id: 211911707)
[2021-05-27 17:01:06] (resource_pk: 00795ea4ea42e65a4fa8f0757e3df5bd_try, id: 211882445), (resource_pk: 87fee0d7b051f2e37bee62920032487f_try, id: 212172910)
[2021-05-27 17:01:06] (resource_pk: 007971ba14c2fa7ab53faeb688c6b379_try, id: 211882447), (resource_pk: bd6760263b744bd8430c9b61017d5cfa_try, id: 212287653)
[2021-05-27 17:01:06] (resource_pk: 007ce29fe32f10eb62bca9eeba119c82_try, id: 211882473), (resource_pk: 2efaed7da7f598c4dfe64ab940b88db4_try, id: 211982122)
[2021-05-27 17:01:06] (resource_pk: 007f52cfde564119ca829694c7bf9010_try, id: 211882488), (resource_pk: 71e54b94f63290aff11bebdc2e329205_try, id: 212125530)
[2021-05-27 17:01:06] (resource_pk: 00806ba0d0451c038ee9443ca85725c3_try, id: 211882500), (resource_pk: c8fe8b9dd4df680cd981e70003507473_try, id: 212312332)
[2021-05-27 17:01:06] (resource_pk: 00823d20cbed5690f98f88fb5d4094a2_try, id: 211882514), (resource_pk: 903d387c9ef7c6059a55165009057ec8_try, id: 212190498)
[2021-05-27 17:01:06] (resource_pk: 008450f580d917e388333be1d175d850_try, id: 211882532), (resource_pk: bddb4cb32b7da532d313fdf4f5002cb1_try, id: 212288608)
[2021-05-27 17:01:06] (resource_pk: 00878402253255ad96c72219b3b939ff_try, id: 211882550), (resource_pk: 0d5115781e0b2962b7bb8cac3b4d4d0b_try, id: 211910166)
[2021-05-27 17:01:06] (resource_pk: 00899decd48674718161a33c1d0e889e_try, id: 211882566), (resource_pk: 95e4e6fc469f7f6924fe0b3c7bd690fa_try, id: 212202675)
[2021-05-27 17:01:06] (resource_pk: 008b67ef0e5d2d3b83ef4a6d266ce061_try, id: 211882580), (resource_pk: 51d8e4c20f1b7daf10dcf75cf24ee43e_try, id: 212057158)
[2021-05-27 17:01:06] (resource_pk: 009004e31ab5dd9b79967ada9bd1e38a_try, id: 211882628), (resource_pk: 6ac3bc6ddb4bcd94030dfc99991f2d53_try, id: 212110315)
[2021-05-27 17:01:06] (resource_pk: 00908d3a663955c78e5f3a9bdf0ec37f_try, id: 211882631), (resource_pk: c435202fecb8726413d6b0739792b4eb_try, id: 212302117)
[2021-05-27 17:01:06] (resource_pk: 00911cceb8fc0001794f32b8eea20714_try, id: 211882638), (resource_pk: 2bb26d6d4e5084187456ec6e325b5414_try, id: 211975136)
[2021-05-27 17:01:06] (resource_pk: 00913cbe51873fd45a8c2b81a2649f04_try, id: 211882641), (resource_pk: 7acf58d1f3bc580f6d08bcfc1fd70ca8_try, id: 212144758)
[2021-05-27 17:01:06] (resource_pk: 00914c8a9b1094c88b49fe887d071c64_try, id: 211882642), (resource_pk: 3f07df4e435087b78bd1fb932af88445_try, id: 212016579)
[2021-05-27 17:01:06] (resource_pk: 009169711b7e2caaaffc9f44cd08f257_try, id: 211882643), (resource_pk: ad5d2a41561aee4b9237be187b27f00c_try, id: 212253091)
[2021-05-27 17:01:06] (resource_pk: 0092e27de919ea591228938906625746_try, id: 211882659), (resource_pk: db42044e9116460cae00ca30399c4281_try, id: 212351379)
[2021-05-27 17:01:06] (resource_pk: 009868fea3f69859944cf810f01f1666_try, id: 211882700), (resource_pk: 1fa95e8738c3f9aac4f636a9e82cdaf4_try, id: 211949315)
[2021-05-27 17:01:06] (resource_pk: 009a74a5550077ca7308234def2ee80b_try, id: 211882709), (resource_pk: 68306697e815e09c291519dd84b89b4f_try, id: 212104723)
[2021-05-27 17:01:06] (resource_pk: 009ad2e171357fb568a34a9b24130ed9_try, id: 211882715), (resource_pk: c872a90eb0d6018603178a333ed4519d_try, id: 212311105)
[2021-05-27 17:01:06] (resource_pk: 009b63cc43256cbd135b149a37791c71_try, id: 211882721), (resource_pk: a4c9529e8065c53051431d15f8a0a699_try, id: 212234439)
[2021-05-27 17:01:06] (resource_pk: 009d93523782e3d702a7aedfbd4f31d3_try, id: 211882737), (resource_pk: 42a7c799854dedfa09e8e547af128713_try, id: 212024257)
[2021-05-27 17:01:06] (resource_pk: 009ec81500206573b9593b3bfd7f72fb_try, id: 211882748), (resource_pk: 32624ad2e03031b425f4112ad89d252a_try, id: 211989533)
[2021-05-27 17:01:06] (resource_pk: 00a1a6514ff6a9aab22990ae96473443_try, id: 211882773), (resource_pk: c47b90359cba2517b7f27c396e16e41b_try, id: 212302703)
[2021-05-27 17:01:06] (resource_pk: 00a25d49eb92062ed9adcf294131c138_try, id: 211882779), (resource_pk: df22d3a000bab288516a6f638f5b0a53_try, id: 212359648)
[2021-05-27 17:01:06] (resource_pk: 00a2c0ee3435873200ea5c868d379515_try, id: 211882782), (resource_pk: 07be42ffb1e7bf0621ee404d47dcf583_try, id: 211898203)
[2021-05-27 17:01:06] (resource_pk: 00a6b2c6b62e26670df5ec98dc4206bd_try, id: 211882817), (resource_pk: bc1c9c7d5389273e1feefe0955e96944_try, id: 212284887)
[2021-05-27 17:01:06] (resource_pk: 00a70af5138843a0a75cfeebaa035f19_try, id: 211882823), (resource_pk: 8fe76c24db47db62f1d958afa334a60b_try, id: 212189824)
[2021-05-27 17:01:06] (resource_pk: 00a8e875ccb2d2f5b85bfbf1d13ad0ab_try, id: 211882837), (resource_pk: 42e28372007482d9c21f54fab89fb429_try, id: 212024734)
[2021-05-27 17:01:06] (resource_pk: 00aa8a157715f29bab6f1e3ca41eff5b_try, id: 211882848), (resource_pk: 1034f2685676fc9f4d28c03cc9e552b2_try, id: 211916192)
[2021-05-27 17:01:06] (resource_pk: 00ad2415ceb84347df488ffeba85f1f3_try, id: 211882882), (resource_pk: 44c8b86e5dcc1fcc7280a8b6fa163d32_try, id: 212028884)
[2021-05-27 17:01:06] (resource_pk: 00adcfc88ebb3ffeb452dbed5bbf9499_try, id: 211882888), (resource_pk: d2fa64ab3235e42c8c9ffefb3b6218b1_try, id: 212333533)
[2021-05-27 17:01:06] (resource_pk: 00b272e61c6e2bfe0f2c217acadf8c3d_try, id: 211882927), (resource_pk: 0f8dfee62c428cbead3cbba023920b31_try, id: 211914889)
[2021-05-27 17:01:06] (resource_pk: 00b5eb79cac31f19e3db4759e6269b10_try, id: 211882953), (resource_pk: e86117c104d8c2a3de7568a98ddf8ca5_try, id: 212379490)
[2021-05-27 17:01:06] (resource_pk: 00b76a3c56693c25f208f15523249cd8_try, id: 211882973), (resource_pk: 33b62722d409454a5777f09c71d18ada_try, id: 211992443)
[2021-05-27 17:01:06] (resource_pk: 00baf9978f5e410ca1f7f126f0af99d5_try, id: 211883005), (resource_pk: 771d804039322330ea0fa4bf228a40b8_try, id: 212136794)
[2021-05-27 17:01:06] (resource_pk: 00bc88a26b332eb1596e3dc58a071ef1_try, id: 211883017), (resource_pk: 4d6e12bbdff39308200e4018afa03399_try, id: 212047783)
[2021-05-27 17:01:06] (resource_pk: 00c1f0978e9451c3bbcde6cfa4205706_try, id: 211883069), (resource_pk: 9786cc6dc36a27b5b3bf8987b07aa9a9_try, id: 212206139)
[2021-05-27 17:01:06] (resource_pk: 00c564760d8e29f866473a90ea2e7912_try, id: 211883099), (resource_pk: 28d3a191ba4141983dc661a71887915b_try, id: 211969042)
[2021-05-27 17:01:06] (resource_pk: 00ccdcba0b6133cb177ac4f5aeb0b1f9_try, id: 211883159), (resource_pk: 6ee604ffa523b99fe6f0ddcf5f6819ec_try, id: 212119051)
[2021-05-27 17:01:06] (resource_pk: 00ccf9f4a0f6d1f94276724d7cbd02db_try, id: 211883161), (resource_pk: afdf2c869ef72b35fd1af79f100d8a0b_try, id: 212258405)
[2021-05-27 17:01:06] (resource_pk: 00ce12e5ab3be666068ab4c484968fcd_try, id: 211883169), (resource_pk: d7234f56b70ba87d0523fa441bd15045_try, id: 212342532)
[2021-05-27 17:01:06] (resource_pk: 00ce9a41d186be48047fae7f62c6acac_try, id: 211883176), (resource_pk: 4dfdd017a028414aad3804c5795d9ae5_try, id: 212048940)
[2021-05-27 17:01:06] (resource_pk: 00cfff39f4ec3b0b979e3d21ab1ee863_try, id: 211883190), (resource_pk: 65c55926f1bbbf039c1bb77afd739fb8_try, id: 212099666)
[2021-05-27 17:01:06] (resource_pk: 00d2eda390e7c2389c81cec6b6823475_try, id: 211883215), (resource_pk: 05074ca221012d068d72433c1683b316_try, id: 211892389)
[2021-05-27 17:01:06] (resource_pk: 00d43ead3497f57f533d5b090622075b_try, id: 211883222), (resource_pk: 2f52bac9a735892c1a74b73cfe1c3633_try, id: 211982901)
[STOP] [2021-05-27 17:01:07] calculate_statistics
[START] [2021-05-27 17:01:07] complete_harvest_instance
[START] [2021-05-27 17:01:07] overall_tsv_creation
[INFO] [2021-05-27 17:01:07] Processing group of 103791 in 11 batches of 10000
[INFO] [2021-05-27 17:02:17] 55175 Traits (unfiltered)...
[INFO] [2021-05-27 17:08:26] 55175 Traits (filtered)...
[INFO] [2021-05-27 17:08:28] 0 Associations (filtered)...
[INFO] [2021-05-27 17:08:41] 49356 metadata added.
[INFO] [2021-05-27 17:08:41] 0 metadata added.
[INFO] [2021-05-27 17:10:31] 53226 Traits (unfiltered)...
[INFO] [2021-05-27 17:16:31] 53226 Traits (filtered)...
[INFO] [2021-05-27 17:16:34] 0 Associations (filtered)...
[INFO] [2021-05-27 17:16:47] 51447 metadata added.
[INFO] [2021-05-27 17:16:47] 0 metadata added.
[INFO] [2021-05-27 17:18:41] 52015 Traits (unfiltered)...
[INFO] [2021-05-27 17:24:36] 52015 Traits (filtered)...
[INFO] [2021-05-27 17:24:39] 0 Associations (filtered)...
[INFO] [2021-05-27 17:24:50] 46787 metadata added.
[INFO] [2021-05-27 17:24:50] 0 metadata added.
[INFO] [2021-05-27 17:26:48] 53365 Traits (unfiltered)...
[INFO] [2021-05-27 17:32:50] 53365 Traits (filtered)...
[INFO] [2021-05-27 17:32:53] 0 Associations (filtered)...
[INFO] [2021-05-27 17:33:05] 46926 metadata added.
[INFO] [2021-05-27 17:33:05] 0 metadata added.
[INFO] [2021-05-27 17:34:55] 53955 Traits (unfiltered)...
[INFO] [2021-05-27 17:41:00] 53955 Traits (filtered)...
[INFO] [2021-05-27 17:41:03] 0 Associations (filtered)...
[INFO] [2021-05-27 17:41:12] 48940 metadata added.
[INFO] [2021-05-27 17:41:12] 0 metadata added.
[INFO] [2021-05-27 17:43:09] 57390 Traits (unfiltered)...
[INFO] [2021-05-27 17:49:38] 57390 Traits (filtered)...
[INFO] [2021-05-27 17:49:41] 0 Associations (filtered)...
[INFO] [2021-05-27 17:49:51] 51672 metadata added.
[INFO] [2021-05-27 17:49:51] 0 metadata added.
[INFO] [2021-05-27 17:51:48] 48317 Traits (unfiltered)...
[INFO] [2021-05-27 17:57:20] 48317 Traits (filtered)...
[INFO] [2021-05-27 17:57:23] 0 Associations (filtered)...
[INFO] [2021-05-27 17:57:35] 44862 metadata added.
[INFO] [2021-05-27 17:57:35] 0 metadata added.
[INFO] [2021-05-27 17:59:24] 48565 Traits (unfiltered)...
[INFO] [2021-05-27 18:04:58] 48565 Traits (filtered)...
[INFO] [2021-05-27 18:05:01] 0 Associations (filtered)...
[INFO] [2021-05-27 18:05:12] 49315 metadata added.
[INFO] [2021-05-27 18:05:12] 0 metadata added.
[INFO] [2021-05-27 18:07:03] 53149 Traits (unfiltered)...
[INFO] [2021-05-27 18:13:07] 53149 Traits (filtered)...
[INFO] [2021-05-27 18:13:10] 0 Associations (filtered)...
[INFO] [2021-05-27 18:13:21] 50102 metadata added.
[INFO] [2021-05-27 18:13:21] 0 metadata added.
[INFO] [2021-05-27 18:15:17] 52263 Traits (unfiltered)...
[INFO] [2021-05-27 18:21:16] 52263 Traits (filtered)...
[INFO] [2021-05-27 18:21:19] 0 Associations (filtered)...
[INFO] [2021-05-27 18:21:28] 49849 metadata added.
[INFO] [2021-05-27 18:21:28] 0 metadata added.
[INFO] [2021-05-27 18:23:04] 21206 Traits (unfiltered)...
[INFO] [2021-05-27 18:25:47] 21206 Traits (filtered)...
[INFO] [2021-05-27 18:25:50] 0 Associations (filtered)...
[INFO] [2021-05-27 18:25:54] 19825 metadata added.
[INFO] [2021-05-27 18:25:54] 0 metadata added.
[INFO] [2021-05-27 18:26:27] Average Time: 435.781
[INFO] [2021-05-27 18:26:27] Total Time: 1h25m20s
[INFO] [2021-05-27 18:26:27] last 3 / first 3: 0.84
[INFO] [2021-05-27 18:26:27] Std.Dev: 71.20133425716122; Max: 489.64
[STOP] [2021-05-27 18:26:27] overall_tsv_creation
[INFO] [2021-05-27 18:26:27] Done. Check your files:
[INFO] [2021-05-27 18:26:29] (103791 lines) /app/public/data/try_summarized_r/publish_nodes.tsv
[INFO] [2021-05-27 18:26:30] (171203 lines) /app/public/data/try_summarized_r/publish_node_ancestors.tsv
[INFO] [2021-05-27 18:26:31] (103791 lines) /app/public/data/try_summarized_r/publish_scientific_names.tsv
[INFO] [2021-05-27 18:26:33] (548627 lines) /app/public/data/try_summarized_r/publish_traits.tsv
[INFO] [2021-05-27 18:26:34] (509082 lines) /app/public/data/try_summarized_r/publish_metadata.tsv
[STOP] [2021-05-27 18:26:35] complete_harvest_instance
[START] [2021-05-27 18:26:35] completed
[STOP] [2021-05-27 18:26:35] completed
[STOP] [2021-05-27 18:26:35] logged process, took 9776.66

Latest Process