Harvest for coraltraits Created 24 Nov 13:02

Stage: completed
Fetched: 24 Nov 13:02
Validated: 24 Nov 13:02
Deltas Created 24 Nov 13:03
Units Normalized: 24 Nov 13:08
Ancestry Built: 24 Nov 13:05
Nodes Matched: 24 Nov 13:06
Names Parsed: 24 Nov 13:05
New Models Stored: 24 Nov 13:04
Indexed: 24 Nov 13:06
Completed: 24 Nov 13:16
Time to Harvest: less than a minute

Harvesting Log (most recent first)

[INFO] [2021-11-24 13:02:38] Created harvest instance #4078
[STOP] [2021-11-24 13:02:38] create_harvest_instance
[START] [2021-11-24 13:02:38] fetch_files
[STOP] [2021-11-24 13:02:38] fetch_files
[START] [2021-11-24 13:02:38] validate_each_file
[INFO] [2021-11-24 13:02:38] Looping over 4 formats...
[INFO] [2021-11-24 13:02:38] ...refs (/app/public/data/coraltraits_tar_/reference.tab)
[INFO] [2021-11-24 13:02:38] Valid: /app/public/converted_csv/coraltraits_tar__refs_4078.csv (555 lines)
[INFO] [2021-11-24 13:02:38] ...nodes (/app/public/data/coraltraits_tar_/taxon.tab)
[INFO] [2021-11-24 13:02:38] Valid: /app/public/converted_csv/coraltraits_tar__nodes_4078.csv (1547 lines)
[INFO] [2021-11-24 13:02:38] ...occurrences (/app/public/data/coraltraits_tar_/occurrence.tab)
[INFO] [2021-11-24 13:02:39] Valid: /app/public/converted_csv/coraltraits_tar__occurrences_4078.csv (31858 lines)
[INFO] [2021-11-24 13:02:39] ...measurements (/app/public/data/coraltraits_tar_/measurement_or_fact_specific.tab)
[INFO] [2021-11-24 13:02:42] Valid: /app/public/converted_csv/coraltraits_tar__measurements_4078.csv (53008 lines)
[STOP] [2021-11-24 13:02:42] validate_each_file
[START] [2021-11-24 13:02:42] convert_to_csv
[INFO] [2021-11-24 13:02:42] Looping over 4 formats...
[INFO] [2021-11-24 13:02:42] ...refs (/app/public/data/coraltraits_tar_/reference.tab)
[CMD] [2021-11-24 13:02:42] /usr/bin/sort /app/public/converted_csv/coraltraits_tar__refs_4078.csv > /app/public/converted_csv/coraltraits_tar__refs_4078.csv_sorted
[INFO] [2021-11-24 13:02:44] Converted: /app/public/converted_csv/coraltraits_tar__refs_4078.csv (555 lines)
[INFO] [2021-11-24 13:02:44] ...nodes (/app/public/data/coraltraits_tar_/taxon.tab)
[CMD] [2021-11-24 13:02:44] /usr/bin/sort /app/public/converted_csv/coraltraits_tar__nodes_4078.csv > /app/public/converted_csv/coraltraits_tar__nodes_4078.csv_sorted
[INFO] [2021-11-24 13:02:45] Converted: /app/public/converted_csv/coraltraits_tar__nodes_4078.csv (1547 lines)
[INFO] [2021-11-24 13:02:45] ...occurrences (/app/public/data/coraltraits_tar_/occurrence.tab)
[CMD] [2021-11-24 13:02:45] /usr/bin/sort /app/public/converted_csv/coraltraits_tar__occurrences_4078.csv > /app/public/converted_csv/coraltraits_tar__occurrences_4078.csv_sorted
[INFO] [2021-11-24 13:02:47] Converted: /app/public/converted_csv/coraltraits_tar__occurrences_4078.csv (31858 lines)
[INFO] [2021-11-24 13:02:47] ...measurements (/app/public/data/coraltraits_tar_/measurement_or_fact_specific.tab)
[CMD] [2021-11-24 13:02:47] /usr/bin/sort /app/public/converted_csv/coraltraits_tar__measurements_4078.csv > /app/public/converted_csv/coraltraits_tar__measurements_4078.csv_sorted
[INFO] [2021-11-24 13:02:48] Converted: /app/public/converted_csv/coraltraits_tar__measurements_4078.csv (53008 lines)
[STOP] [2021-11-24 13:02:48] convert_to_csv
[START] [2021-11-24 13:02:49] calculate_delta
[INFO] [2021-11-24 13:02:49] Looping over 4 formats...
[INFO] [2021-11-24 13:02:49] ...refs (/app/public/data/coraltraits_tar_/reference.tab)
[CMD] [2021-11-24 13:02:49] echo "0a" > /app/public/diff/coraltraits_tar__refs_4078.diff
[CMD] [2021-11-24 13:02:50] tail -n +1 /app/public/converted_csv/coraltraits_tar__refs_4078.csv >> /app/public/diff/coraltraits_tar__refs_4078.diff
[CMD] [2021-11-24 13:02:52] echo "." >> /app/public/diff/coraltraits_tar__refs_4078.diff
[INFO] [2021-11-24 13:02:53] Created diff: /app/public/diff/coraltraits_tar__refs_4078.diff (557 lines)
[INFO] [2021-11-24 13:02:53] ...nodes (/app/public/data/coraltraits_tar_/taxon.tab)
[CMD] [2021-11-24 13:02:53] echo "0a" > /app/public/diff/coraltraits_tar__nodes_4078.diff
[CMD] [2021-11-24 13:02:55] tail -n +1 /app/public/converted_csv/coraltraits_tar__nodes_4078.csv >> /app/public/diff/coraltraits_tar__nodes_4078.diff
[CMD] [2021-11-24 13:02:56] echo "." >> /app/public/diff/coraltraits_tar__nodes_4078.diff
[INFO] [2021-11-24 13:02:58] Created diff: /app/public/diff/coraltraits_tar__nodes_4078.diff (1549 lines)
[INFO] [2021-11-24 13:02:58] ...occurrences (/app/public/data/coraltraits_tar_/occurrence.tab)
[CMD] [2021-11-24 13:02:58] echo "0a" > /app/public/diff/coraltraits_tar__occurrences_4078.diff
[CMD] [2021-11-24 13:03:00] tail -n +1 /app/public/converted_csv/coraltraits_tar__occurrences_4078.csv >> /app/public/diff/coraltraits_tar__occurrences_4078.diff
[CMD] [2021-11-24 13:03:01] echo "." >> /app/public/diff/coraltraits_tar__occurrences_4078.diff
[INFO] [2021-11-24 13:03:03] Created diff: /app/public/diff/coraltraits_tar__occurrences_4078.diff (31860 lines)
[INFO] [2021-11-24 13:03:03] ...measurements (/app/public/data/coraltraits_tar_/measurement_or_fact_specific.tab)
[CMD] [2021-11-24 13:03:03] echo "0a" > /app/public/diff/coraltraits_tar__measurements_4078.diff
[CMD] [2021-11-24 13:03:04] tail -n +1 /app/public/converted_csv/coraltraits_tar__measurements_4078.csv >> /app/public/diff/coraltraits_tar__measurements_4078.diff
[CMD] [2021-11-24 13:03:06] echo "." >> /app/public/diff/coraltraits_tar__measurements_4078.diff
[INFO] [2021-11-24 13:03:08] Created diff: /app/public/diff/coraltraits_tar__measurements_4078.diff (53010 lines)
[STOP] [2021-11-24 13:03:08] calculate_delta
[START] [2021-11-24 13:03:08] parse_diff_and_store
[INFO] [2021-11-24 13:03:08] Handling diff: /app/public/diff/coraltraits_tar__refs_4078.diff (557 lines)
[INFO] [2021-11-24 13:03:09] Loading refs diff file into memory (557 /app/public/diff/coraltraits_tar__refs_4078.diff lines)...
[INFO] [2021-11-24 13:03:11] Handling diff: /app/public/diff/coraltraits_tar__nodes_4078.diff (1549 lines)
[INFO] [2021-11-24 13:03:13] Loading nodes diff file into memory (1549 /app/public/diff/coraltraits_tar__nodes_4078.diff lines)...
[INFO] [2021-11-24 13:03:15] Handling diff: /app/public/diff/coraltraits_tar__occurrences_4078.diff (31860 lines)
[INFO] [2021-11-24 13:03:16] Loading occurrences diff file into memory (31860 /app/public/diff/coraltraits_tar__occurrences_4078.diff lines)...
[INFO] [2021-11-24 13:03:26] Handling diff: /app/public/diff/coraltraits_tar__measurements_4078.diff (53010 lines)
[INFO] [2021-11-24 13:03:28] Loading measurements diff file into memory (53010 /app/public/diff/coraltraits_tar__measurements_4078.diff lines)...
[INFO] [2021-11-24 13:04:01] Storing 555 References
[INFO] [2021-11-24 13:04:01] Processing group of 555 in 1 groups of 1000
[INFO] [2021-11-24 13:04:01] Average Time: 0.1
[INFO] [2021-11-24 13:04:01] Total Time: 1s
[INFO] [2021-11-24 13:04:01] Storing 1551 ScientificNames
[INFO] [2021-11-24 13:04:01] Processing group of 1551 in 2 groups of 1000
[INFO] [2021-11-24 13:04:02] Average Time: 0.275
[INFO] [2021-11-24 13:04:02] Total Time: 1s
[INFO] [2021-11-24 13:04:02] Storing 1551 Nodes
[INFO] [2021-11-24 13:04:02] Processing group of 1551 in 2 groups of 1000
[INFO] [2021-11-24 13:04:02] Average Time: 0.19
[INFO] [2021-11-24 13:04:02] Total Time: 1s
[INFO] [2021-11-24 13:04:02] Storing 31858 Occurrences
[INFO] [2021-11-24 13:04:02] Processing group of 31858 in 32 groups of 1000
[INFO] [2021-11-24 13:04:06] Average Time: 0.11
[INFO] [2021-11-24 13:04:06] Total Time: 4s
[INFO] [2021-11-24 13:04:06] last 3 / first 3: 1.03
[INFO] [2021-11-24 13:04:06] Std.Dev: 0.03162277660168379; Max: 0.2
[INFO] [2021-11-24 13:04:06] Storing 51213 OccurrenceMetadata
[INFO] [2021-11-24 13:04:06] Processing group of 51213 in 52 groups of 1000
[INFO] [2021-11-24 13:04:12] Average Time: 0.114
[INFO] [2021-11-24 13:04:12] Total Time: 7s
[INFO] [2021-11-24 13:04:12] last 3 / first 3: 1.03
[INFO] [2021-11-24 13:04:12] Std.Dev: 0.0; Max: 0.2
[INFO] [2021-11-24 13:04:12] Storing 41348 TraitsReferences
[INFO] [2021-11-24 13:04:12] Processing group of 41348 in 42 groups of 1000
[INFO] [2021-11-24 13:04:15] Average Time: 0.066
[INFO] [2021-11-24 13:04:15] Total Time: 4s
[INFO] [2021-11-24 13:04:15] last 3 / first 3: 0.71
[INFO] [2021-11-24 13:04:15] Std.Dev: 0.0; Max: 0.17
[INFO] [2021-11-24 13:04:15] Storing 53008 Traits
[INFO] [2021-11-24 13:04:15] Processing group of 53008 in 54 groups of 1000
[INFO] [2021-11-24 13:04:33] Average Time: 0.333
[INFO] [2021-11-24 13:04:33] Total Time: 19s
[INFO] [2021-11-24 13:04:33] last 3 / first 3: 0.68
[INFO] [2021-11-24 13:04:33] Std.Dev: 0.130384048104053; Max: 0.89
[INFO] [2021-11-24 13:04:33] Storing 97276 MetaTraits
[INFO] [2021-11-24 13:04:33] Processing group of 97276 in 98 groups of 1000
[INFO] [2021-11-24 13:04:47] Average Time: 0.129
[INFO] [2021-11-24 13:04:47] Total Time: 14s
[INFO] [2021-11-24 13:04:47] last 3 / first 3: 0.74
[INFO] [2021-11-24 13:04:47] Std.Dev: 0.05477225575051661; Max: 0.41
[STOP] [2021-11-24 13:04:47] parse_diff_and_store
[START] [2021-11-24 13:04:47] resolve_keys
[INFO] [2021-11-24 13:05:05] Occurrences to nodes (through scientific_names)...
[INFO] [2021-11-24 13:05:07] traits to occurrences...
[INFO] [2021-11-24 13:05:09] traits to nodes (through occurrences)...
[INFO] [2021-11-24 13:05:09] Traits to sex term...
[INFO] [2021-11-24 13:05:10] Traits to lifestage term...
[INFO] [2021-11-24 13:05:10] MetaTraits to traits...
[INFO] [2021-11-24 13:05:13] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-11-24 13:05:16] Assocs to occurrences...
[INFO] [2021-11-24 13:05:16] Assocs to nodes...
[INFO] [2021-11-24 13:05:16] Assoc to sex term...
[INFO] [2021-11-24 13:05:16] Assoc to lifestage term...
[INFO] [2021-11-24 13:05:16] MetaAssoc to assocs...
[STOP] [2021-11-24 13:05:16] resolve_keys
[START] [2021-11-24 13:05:16] hold_for_later_1
[STOP] [2021-11-24 13:05:16] hold_for_later_1
[START] [2021-11-24 13:05:16] hold_for_later_2
[STOP] [2021-11-24 13:05:16] hold_for_later_2
[START] [2021-11-24 13:05:16] resolve_missing_parents
[STOP] [2021-11-24 13:05:16] resolve_missing_parents
[START] [2021-11-24 13:05:16] rebuild_nodes
[START] [2021-11-24 13:05:16] Flattener#flatten
[START] [2021-11-24 13:05:16] Flattener#study_resource
[START] [2021-11-24 13:05:16] Flattener#build_ancestry
[STOP] [2021-11-24 13:05:16] Flattener#build_ancestry
[INFO] [2021-11-24 13:05:16] 1551 ancestry keys
[START] [2021-11-24 13:05:16] build_node_ancestors
[INFO] [2021-11-24 13:05:16] old ancestors deleted.
[STOP] [2021-11-24 13:05:16] build_node_ancestors
[START] [2021-11-24 13:05:16] Flattener#propagate_ancestor_ids
[STOP] [2021-11-24 13:05:16] Flattener#propagate_ancestor_ids
[STOP] [2021-11-24 13:05:16] Flattener#flatten
[STOP] [2021-11-24 13:05:17] rebuild_nodes
[START] [2021-11-24 13:05:17] resolve_missing_media_owners
[STOP] [2021-11-24 13:05:17] resolve_missing_media_owners
[START] [2021-11-24 13:05:17] sanitize_media_verbatims
[STOP] [2021-11-24 13:05:17] sanitize_media_verbatims
[START] [2021-11-24 13:05:17] queue_downloads
[STOP] [2021-11-24 13:05:17] queue_downloads
[START] [2021-11-24 13:05:17] parse_names
[WARN] [2021-11-24 13:05:17] I see 1551 names which still need to be parsed.
[WARN] [2021-11-24 13:05:17] Names to parse: 1551 formatted: 1551 learned: 1551 parsed: 1551
[STOP] [2021-11-24 13:05:19] parse_names
[START] [2021-11-24 13:05:19] denormalize_canonical_names_to_nodes
[STOP] [2021-11-24 13:05:19] denormalize_canonical_names_to_nodes
[START] [2021-11-24 13:05:19] match_nodes
[START] [2021-11-24 13:05:19] map_all_nodes_to_pages
[STOP] [2021-11-24 13:06:42] map_all_nodes_to_pages
[INFO] [2021-11-24 13:06:42] 44 Unmatched nodes (of 1551)! That's too many to output. Full list in /app/public/data/coraltraits_tar_/unmatched_nodes.txt ; First 10: Canonical: Acropora lutkeni; Node#101270671; ResourceID: 101; Canonical: Montipora granulosa; Node#101270680; ResourceID: 1018; Canonical: Montipora hirsuta; Node#101270684; ResourceID: 1021; Canonical: Oulangia stokesianamiltoni; Node#101270768; ResourceID: 1099; Canonical: Oulangia stokesianastokesiana; Node#101270770; ResourceID: 1100; Canonical: Paraconotrochus antarctica; Node#101270793; ResourceID: 1121; Canonical: Paracyathus indicusgracilis; Node#101270805; ResourceID: 1132; Canonical: Paracyathus indicusindicus; Node#101270806; ResourceID: 1133; Canonical: Phyllangia americanaamericana; Node#101270858; ResourceID: 1181; Canonical: Phyllangia americanamouchezii; Node#101270859; ResourceID: 1182
[START] [2021-11-24 13:06:42] update_nodes
[STOP] [2021-11-24 13:06:43] update_nodes
[STOP] [2021-11-24 13:06:43] match_nodes
[START] [2021-11-24 13:06:43] reindex_search
[STOP] [2021-11-24 13:06:46] reindex_search
[START] [2021-11-24 13:06:46] normalize_units
[STOP] [2021-11-24 13:08:08] normalize_units
[START] [2021-11-24 13:08:08] calculate_statistics
[2021-11-24 13:08:09] (NEAR) DUPLICATE TRAITS FOUND! There are only 28702 (of 32568 total) unique traits.
[2021-11-24 13:08:32] (Near) duplicate trait pairs (up to 100):
[2021-11-24 13:08:32] (resource_pk: 00083fd32a5b2751e2e3c0f8f296cc6f_cotr, id: 240993932), (resource_pk: 4d2f3557b223a63ccb2241915fb1ba96_cotr, id: 241009953)
[2021-11-24 13:08:32] (resource_pk: 000e2503abfc3e9e8be33e4b650478a4_cotr, id: 240993937), (resource_pk: fbb201415f4609bbaa6463bcadaaf21d_cotr, id: 241046052)
[2021-11-24 13:08:32] (resource_pk: 001cc1450614d920bdb8dab21a873c78_cotr, id: 240993950), (resource_pk: 25aa165e62ae4f73b3d76043eafe8592_cotr, id: 241001656)
[2021-11-24 13:08:32] (resource_pk: 0030e2597619e594481e650a08b20ebb_cotr, id: 240993965), (resource_pk: 6936186449affdc47a4363cbf67c6444_cotr, id: 241015664)
[2021-11-24 13:08:32] (resource_pk: 003212709adaea2571c3be01e3fd3067_cotr, id: 240993967), (resource_pk: bca5ba512b360ea4459a19e8a3cb660f_cotr, id: 241032861)
[2021-11-24 13:08:32] (resource_pk: 00324ade4ee609608c502fce001dc873_cotr, id: 240993968), (resource_pk: defd8926d2e837a3bf5c05092f53a3b9_cotr, id: 241040014)
[2021-11-24 13:08:32] (resource_pk: 0034c563e04f0ebd9cecf34588bbc84f_cotr, id: 240993970), (resource_pk: 7a4229134d4988c1990d43b547b392c0_cotr, id: 241019097)
[2021-11-24 13:08:32] (resource_pk: 0058829a685c56d9417bff517d34f2b6_cotr, id: 240993996), (resource_pk: 736fa531ae6bbe03e074b17415c70ed7_cotr, id: 241017693)
[2021-11-24 13:08:32] (resource_pk: 0058829a685c56d9417bff517d34f2b6_cotr, id: 240993996), (resource_pk: 7c3c03b407b3b2b73299b013f17645c2_cotr, id: 241019496)
[2021-11-24 13:08:32] (resource_pk: 0058829a685c56d9417bff517d34f2b6_cotr, id: 240993996), (resource_pk: b5cea0e753161f448b2312cade33cfab_cotr, id: 241031396)
[2021-11-24 13:08:32] (resource_pk: 0058829a685c56d9417bff517d34f2b6_cotr, id: 240993996), (resource_pk: fc8b693509d8d35335732a378b74def4_cotr, id: 241046246)
[2021-11-24 13:08:32] (resource_pk: 005faa42d25bff92b23bd0e1ba54093a_cotr, id: 240994001), (resource_pk: 9146091222a50bd941ad187043d593b7_cotr, id: 241023884)
[2021-11-24 13:08:32] (resource_pk: 005faa42d25bff92b23bd0e1ba54093a_cotr, id: 240994001), (resource_pk: d1877b56fa88be1176f52e67a9eb618d_cotr, id: 241037183)
[2021-11-24 13:08:32] (resource_pk: 005faa42d25bff92b23bd0e1ba54093a_cotr, id: 240994001), (resource_pk: f8dda0602d3033058093bc920dac8a05_cotr, id: 241045375)
[2021-11-24 13:08:32] (resource_pk: 0060b41ca27c575b709058052bab1c9b_cotr, id: 240994003), (resource_pk: 6e7d10e627337613b08aadd786a05b8a_cotr, id: 241016732)
[2021-11-24 13:08:32] (resource_pk: 006487ec75655ebc4144f35b88524084_cotr, id: 240994007), (resource_pk: 595eb8e224205ef670af77d352c789fc_cotr, id: 241012421)
[2021-11-24 13:08:32] (resource_pk: 006855ca2e2720fd55cb38a7065250aa_cotr, id: 240994013), (resource_pk: fd212eee88d01ad2f7928303cd03cf35_cotr, id: 241046360)
[2021-11-24 13:08:32] (resource_pk: 0070fd7a744122e8d403f3e52a6a5314_cotr, id: 240994020), (resource_pk: 048fd4109eda6853777537c6151bd4ad_cotr, id: 240994854)
[2021-11-24 13:08:32] (resource_pk: 0070fd7a744122e8d403f3e52a6a5314_cotr, id: 240994020), (resource_pk: c483fb5d8bdd8cf9353d05eaa3e9ee94_cotr, id: 241034510)
[2021-11-24 13:08:32] (resource_pk: 00716a8b6f4b06910c7a292c5e3dc1be_cotr, id: 240994021), (resource_pk: 3658cf0b44aae0b41e57bc247d802bb7_cotr, id: 241005273)
[2021-11-24 13:08:32] (resource_pk: 007feb8a3b1a07aaa56f23aae5d7ee54_cotr, id: 240994031), (resource_pk: 2a77b8013ec4e750e2f319bf1271256c_cotr, id: 241002713)
[2021-11-24 13:08:32] (resource_pk: 008a069633dc702dd0bd360d8015d89a_cotr, id: 240994037), (resource_pk: b581b277b8dbdba7fcb07b225aa6a974_cotr, id: 241031352)
[2021-11-24 13:08:32] (resource_pk: 008b66ab914144f8d25d400ba92d8664_cotr, id: 240994039), (resource_pk: 58770178ed768cb01866a5a6fd2f3f45_cotr, id: 241012236)
[2021-11-24 13:08:32] (resource_pk: 00936c662046f81c49b0b9c6ae8e8898_cotr, id: 240994043), (resource_pk: 455a73d806a9a096c0145bbab9915aa5_cotr, id: 241008345)
[2021-11-24 13:08:32] (resource_pk: 0095de90a022ed6dc01a2d1e6becf062_cotr, id: 240994047), (resource_pk: e79ec631a40d137ae656a11fc3d80f60_cotr, id: 241041814)
[2021-11-24 13:08:32] (resource_pk: 009d2bff09e27ecb5aed706033ba7ea2_cotr, id: 240994053), (resource_pk: 3aafdad17f949632e6eddb892ac4387c_cotr, id: 241006183)
[2021-11-24 13:08:32] (resource_pk: 009d2bff09e27ecb5aed706033ba7ea2_cotr, id: 240994053), (resource_pk: 6a1a35b99ab55bf739f3e9daeb85d1fd_cotr, id: 241015847)
[2021-11-24 13:08:32] (resource_pk: 009d2bff09e27ecb5aed706033ba7ea2_cotr, id: 240994053), (resource_pk: be1c8eee4b5c0934b4e457fc653cea89_cotr, id: 241033151)
[2021-11-24 13:08:32] (resource_pk: 009eff02e956d8e0d1deafee17d8cd40_cotr, id: 240994056), (resource_pk: d1b065d1d28425d1dbc0218a83674700_cotr, id: 241037219)
[2021-11-24 13:08:32] (resource_pk: 00aa974967d5de782b69240678ac93dc_cotr, id: 240994062), (resource_pk: 5b99dccd0f594688d4f6e1b9fe88da95_cotr, id: 241012903)
[2021-11-24 13:08:32] (resource_pk: 00adf2d9c1c5ed00e65fc823bfa94f87_cotr, id: 240994064), (resource_pk: 987a3f0e62b20d9d28846ed713cea0fb_cotr, id: 241025424)
[2021-11-24 13:08:32] (resource_pk: 00afa720a12579e71afffac03c04f455_cotr, id: 240994066), (resource_pk: ee61e45411923de0b58eb10b5db2c521_cotr, id: 241043184)
[2021-11-24 13:08:32] (resource_pk: 00b5159aeb3955c9bf925c352b44eb72_cotr, id: 240994076), (resource_pk: 9eab2d977c78ebc92b4529cd3b0203f7_cotr, id: 241026737)
[2021-11-24 13:08:32] (resource_pk: 00bb0582bd08f41026a00492fe523fcb_cotr, id: 240994081), (resource_pk: 64f4fcee42d41a25ba435c9baad61e7d_cotr, id: 241014761)
[2021-11-24 13:08:32] (resource_pk: 00bd1b7b00812326865dd4dec1b2b1e5_cotr, id: 240994082), (resource_pk: 1b0f2435f9d6f7c5c1312e77a8b322a2_cotr, id: 240999490)
[2021-11-24 13:08:32] (resource_pk: 00be2178070ac950696c589b341429a0_cotr, id: 240994085), (resource_pk: 7259c4c98ab293a9ee7aef2761edb4bd_cotr, id: 241017469)
[2021-11-24 13:08:32] (resource_pk: 00be2178070ac950696c589b341429a0_cotr, id: 240994085), (resource_pk: 872dcd817d9c75fe4980735691f2f094_cotr, id: 241021798)
[2021-11-24 13:08:32] (resource_pk: 00c5465d2a7b10e2fefe748d5ad4883e_cotr, id: 240994091), (resource_pk: 37e20783427406e86506ddc0187b4d5c_cotr, id: 241005604)
[2021-11-24 13:08:32] (resource_pk: 00d46a417d419f75b98727f9ff716c8a_cotr, id: 240994102), (resource_pk: 25a7e24846b700dd1aba762e78a473fb_cotr, id: 241001654)
[2021-11-24 13:08:32] (resource_pk: 00dcc04f08613f351cede735d9742eb0_cotr, id: 240994111), (resource_pk: 94b5625c0c71b4334a15a0df6a222ad1_cotr, id: 241024627)
[2021-11-24 13:08:32] (resource_pk: 00e345df64bbd3b9b1ae647c8fbdc045_cotr, id: 240994116), (resource_pk: ae340def286b6a8a333214255aa5afb9_cotr, id: 241029841)
[2021-11-24 13:08:32] (resource_pk: 00f96911c9538f54a93741acd85991aa_cotr, id: 240994132), (resource_pk: cea23dc3e77e86ae82993157916e46b8_cotr, id: 241036621)
[2021-11-24 13:08:32] (resource_pk: 010a53fb19ca72f22847a3fe21bf36a9_cotr, id: 240994145), (resource_pk: 7f61de88e7b111d56144bcd51a624de8_cotr, id: 241020162)
[2021-11-24 13:08:32] (resource_pk: 010a53fb19ca72f22847a3fe21bf36a9_cotr, id: 240994145), (resource_pk: dd7fcf81b587709593ef1d75f53e98d1_cotr, id: 241039700)
[2021-11-24 13:08:32] (resource_pk: 011ea279cc2b884cd8f3c2ed9255a78f_cotr, id: 240994159), (resource_pk: 1c940c015e6e2173fd324cd908f246ea_cotr, id: 240999790)
[2021-11-24 13:08:32] (resource_pk: 011fe4d24d09bfc5775674eb1eedac4c_cotr, id: 240994160), (resource_pk: f6329aaf125e7f4ace8b36852867b394_cotr, id: 241044820)
[2021-11-24 13:08:32] (resource_pk: 012674702317800d738ffb9b7cdb5659_cotr, id: 240994164), (resource_pk: 0b90594673c0cb054dd2fa944d6c47f0_cotr, id: 240996314)
[2021-11-24 13:08:32] (resource_pk: 01322aa27a84e743a4dc5e4e34d34af7_cotr, id: 240994170), (resource_pk: 1bd61c3f5839b3d5ce7ddbe4cddd387b_cotr, id: 240999641)
[2021-11-24 13:08:32] (resource_pk: 01324de5d1d911ed30291762d5e7139f_cotr, id: 240994171), (resource_pk: 4a82ad498068fbb292168450681dbad0_cotr, id: 241009441)
[2021-11-24 13:08:32] (resource_pk: 01457aeae9ae755422121a5e3b2f135d_cotr, id: 240994185), (resource_pk: 7e5ab7f9353969faa3b5bb7fb7cdaf6e_cotr, id: 241019938)
[2021-11-24 13:08:32] (resource_pk: 01457aeae9ae755422121a5e3b2f135d_cotr, id: 240994185), (resource_pk: 893cad71699529222be8a118d6d20ad6_cotr, id: 241022215)
[2021-11-24 13:08:32] (resource_pk: 0149edccd5181fbab9aa4a5da8369655_cotr, id: 240994189), (resource_pk: 9d3bb3b224c8486657c37a9d66c3b979_cotr, id: 241026433)
[2021-11-24 13:08:32] (resource_pk: 0153bb0e8932343be3a64394566c92bc_cotr, id: 240994197), (resource_pk: 72b3d5bf78590e4889dfb3a921e86013_cotr, id: 241017540)
[2021-11-24 13:08:32] (resource_pk: 016117c0b22a2878cd7561bf62fc8a50_cotr, id: 240994208), (resource_pk: cb80f0901c6b632521c671fe34e5565d_cotr, id: 241035984)
[2021-11-24 13:08:32] (resource_pk: 01642b13b3a71f695849797b74bb9f0e_cotr, id: 240994215), (resource_pk: 5cc6d1de361479695e4da6706b1f5e1f_cotr, id: 241013147)
[2021-11-24 13:08:32] (resource_pk: 016d0976c39e149afa80defbe330c8eb_cotr, id: 240994218), (resource_pk: d01fe7e1d69b0db70a44b45112f700fd_cotr, id: 241036914)
[2021-11-24 13:08:32] (resource_pk: 0177279be92b40c835b274617996c36b_cotr, id: 240994229), (resource_pk: 649ce60dd0916dcb3162b37fc94ee250_cotr, id: 241014680)
[2021-11-24 13:08:32] (resource_pk: 01821272dcb750d9d1075f64e9e87fce_cotr, id: 240994238), (resource_pk: a27f8e202879227cdaffcd6b755bfc4d_cotr, id: 241027509)
[2021-11-24 13:08:32] (resource_pk: 01af927b7076099daa2a616f44857fe0_cotr, id: 240994261), (resource_pk: cf2c6a5d177c13006917beb06603191b_cotr, id: 241036724)
[2021-11-24 13:08:32] (resource_pk: 01b1fcec86548b085af38ee573616051_cotr, id: 240994262), (resource_pk: 78ebebd11420be54101aabef2537523c_cotr, id: 241018845)
[2021-11-24 13:08:32] (resource_pk: 01b240b36c1575c6a60ff3d183315502_cotr, id: 240994263), (resource_pk: f74c3f80c66b218f123cb8346a299bc0_cotr, id: 241045042)
[2021-11-24 13:08:32] (resource_pk: 01cb29a960f3ae564ccb082374c26fe2_cotr, id: 240994278), (resource_pk: 85736f9303bea11bc403de95254adc0f_cotr, id: 241021431)
[2021-11-24 13:08:32] (resource_pk: 01cba6535614c2adfd7912d9c3ae41fc_cotr, id: 240994279), (resource_pk: eabfaf1071a1b31b8468532e9f9dc4a3_cotr, id: 241042462)
[2021-11-24 13:08:32] (resource_pk: 01cdd679570cc9d82f0516508a3357a8_cotr, id: 240994283), (resource_pk: fc1116bdbeea56d52e3271421a4086aa_cotr, id: 241046131)
[2021-11-24 13:08:32] (resource_pk: 01ce65798d7b42d3692f9d9d42b927ed_cotr, id: 240994284), (resource_pk: 91f7571cf0048a2a0965ee622e2f1084_cotr, id: 241024034)
[2021-11-24 13:08:32] (resource_pk: 01d07703b861341137a84192e6e1e377_cotr, id: 240994287), (resource_pk: 2a404ab406d9f95b79471953048fdc08_cotr, id: 241002670)
[2021-11-24 13:08:32] (resource_pk: 01d667d217474c0a054a0e65b51baf79_cotr, id: 240994293), (resource_pk: e55ca5b2658cfb9f6b1a7671e3407250_cotr, id: 241041335)
[2021-11-24 13:08:32] (resource_pk: 01de2082cb08a9e6b08fd7f4397c80a3_cotr, id: 240994298), (resource_pk: 03f06d1e73fc9bcef28b07dfe4d88254_cotr, id: 240994727)
[2021-11-24 13:08:32] (resource_pk: 01deddeeb4594d1218e4b41bf146beb0_cotr, id: 240994299), (resource_pk: d079f456d68dff7c447cc2c09a294ed6_cotr, id: 241036985)
[2021-11-24 13:08:32] (resource_pk: 01e7236fabc2594996e081d0d884c327_cotr, id: 240994306), (resource_pk: e274552dec290d83cadff18b73be94ae_cotr, id: 241040745)
[2021-11-24 13:08:32] (resource_pk: 01eb76c7154cc7d2eb500093603ba823_cotr, id: 240994311), (resource_pk: 8c1263be2016d339613d2938b4f49738_cotr, id: 241022811)
[2021-11-24 13:08:32] (resource_pk: 01f131ed1414ae3fb25a55d350ace51c_cotr, id: 240994315), (resource_pk: acc7b41d8e9c9ec768d01966e57afe83_cotr, id: 241029542)
[2021-11-24 13:08:32] (resource_pk: 020a18e93a0830cd894fe705eb1731a6_cotr, id: 240994333), (resource_pk: 7ad5cd864c984849f873abc8528e4b15_cotr, id: 241019215)
[2021-11-24 13:08:32] (resource_pk: 021582f481c962a15050cab7f820ec12_cotr, id: 240994341), (resource_pk: a70d5752767c068340b32c8d8d75bc88_cotr, id: 241028401)
[2021-11-24 13:08:32] (resource_pk: 021ee39f023a395da42234ec829c3649_cotr, id: 240994350), (resource_pk: 841b0741b98ffae5b155787d37a2291f_cotr, id: 241021134)
[2021-11-24 13:08:32] (resource_pk: 022ca9a6b4ca983ceac2639b6385faa4_cotr, id: 240994358), (resource_pk: 079178e365567776e6ddba78a94e1d3f_cotr, id: 240995456)
[2021-11-24 13:08:32] (resource_pk: 0231eb7a547f30faccdd2e684b068249_cotr, id: 240994365), (resource_pk: 1336a09a5ea8263b231081a01a37e0ef_cotr, id: 240997922)
[2021-11-24 13:08:32] (resource_pk: 0231eb7a547f30faccdd2e684b068249_cotr, id: 240994365), (resource_pk: d6b45920a2e7988ad03d273acc8721cc_cotr, id: 241038273)
[2021-11-24 13:08:32] (resource_pk: 0232bb29ad9986de7cc5230d7a7011f6_cotr, id: 240994367), (resource_pk: 8043fe470e98bde8408306af1254c4d1_cotr, id: 241020337)
[2021-11-24 13:08:32] (resource_pk: 023ab127fd8b1e18035360b10d377e86_cotr, id: 240994370), (resource_pk: 4737494e7fa8c7413789bc1760cda7c8_cotr, id: 241008735)
[2021-11-24 13:08:32] (resource_pk: 024b06cb16de2d7a8916fbbdd3cf0c1c_cotr, id: 240994390), (resource_pk: 093b245c43b8383862eaed43ddd009cb_cotr, id: 240995851)
[2021-11-24 13:08:32] (resource_pk: 024e68eb73f9443491abdbb84e6a4275_cotr, id: 240994391), (resource_pk: 0b0c14647facf1f33280bf4a02cccffe_cotr, id: 240996208)
[2021-11-24 13:08:32] (resource_pk: 024e68eb73f9443491abdbb84e6a4275_cotr, id: 240994391), (resource_pk: 1418f93c2836ce3bac8c33d3a38bec0c_cotr, id: 240998108)
[2021-11-24 13:08:32] (resource_pk: 024e68eb73f9443491abdbb84e6a4275_cotr, id: 240994391), (resource_pk: 1a3353ef0d6d0554384763a35f982769_cotr, id: 240999337)
[2021-11-24 13:08:32] (resource_pk: 024e68eb73f9443491abdbb84e6a4275_cotr, id: 240994391), (resource_pk: 1f576bde583adcd1a0a83b2d445b49b5_cotr, id: 241000376)
[2021-11-24 13:08:32] (resource_pk: 024e68eb73f9443491abdbb84e6a4275_cotr, id: 240994391), (resource_pk: 2beaf33aa15ee1654d02fc232115dbbe_cotr, id: 241003045)
[2021-11-24 13:08:32] (resource_pk: 024e68eb73f9443491abdbb84e6a4275_cotr, id: 240994391), (resource_pk: 4314e156334cf2be8a329cb7aa208e86_cotr, id: 241007884)
[2021-11-24 13:08:32] (resource_pk: 024e68eb73f9443491abdbb84e6a4275_cotr, id: 240994391), (resource_pk: 4def276788debe1fc00c5c58a3892a8f_cotr, id: 241010103)
[2021-11-24 13:08:32] (resource_pk: 024e68eb73f9443491abdbb84e6a4275_cotr, id: 240994391), (resource_pk: 53048382622f73d433bd69b1eb5d810a_cotr, id: 241011149)
[2021-11-24 13:08:32] (resource_pk: 024e68eb73f9443491abdbb84e6a4275_cotr, id: 240994391), (resource_pk: 5b00afade2f75eb2e50e862305f52233_cotr, id: 241012778)
[2021-11-24 13:08:32] (resource_pk: 024e68eb73f9443491abdbb84e6a4275_cotr, id: 240994391), (resource_pk: 5b32b1ccc358316e342a57020f5cb306_cotr, id: 241012817)
[2021-11-24 13:08:32] (resource_pk: 024e68eb73f9443491abdbb84e6a4275_cotr, id: 240994391), (resource_pk: 651958b2a6d5ab80ff9be4fb011340ff_cotr, id: 241014789)
[2021-11-24 13:08:32] (resource_pk: 024e68eb73f9443491abdbb84e6a4275_cotr, id: 240994391), (resource_pk: 7c45fa689dc60c56234117a79630ea09_cotr, id: 241019501)
[2021-11-24 13:08:32] (resource_pk: 024e68eb73f9443491abdbb84e6a4275_cotr, id: 240994391), (resource_pk: 853d42645f804d5bd962a8ecaa172584_cotr, id: 241021380)
[2021-11-24 13:08:32] (resource_pk: 024e68eb73f9443491abdbb84e6a4275_cotr, id: 240994391), (resource_pk: 87dbf21e22989fdd2e90d2abcf365d78_cotr, id: 241021946)
[2021-11-24 13:08:32] (resource_pk: 024e68eb73f9443491abdbb84e6a4275_cotr, id: 240994391), (resource_pk: 8f69bdc208c48c1d8aada9603b2769de_cotr, id: 241023513)
[2021-11-24 13:08:32] (resource_pk: 024e68eb73f9443491abdbb84e6a4275_cotr, id: 240994391), (resource_pk: b51cfc6045bb68a2e47a5bb42401fb65_cotr, id: 241031269)
[2021-11-24 13:08:32] (resource_pk: 024e68eb73f9443491abdbb84e6a4275_cotr, id: 240994391), (resource_pk: bbb758806d048a00fbc7ebdadae22d5f_cotr, id: 241032665)
[2021-11-24 13:08:32] (resource_pk: 024e68eb73f9443491abdbb84e6a4275_cotr, id: 240994391), (resource_pk: c4670b5239a3405a115f318d1e2022fe_cotr, id: 241034491)
[2021-11-24 13:08:32] (resource_pk: 024e68eb73f9443491abdbb84e6a4275_cotr, id: 240994391), (resource_pk: c66afb08bbcfaf590978fc3fc3954d40_cotr, id: 241034931)
[STOP] [2021-11-24 13:08:32] calculate_statistics
[START] [2021-11-24 13:08:32] complete_harvest_instance
[START] [2021-11-24 13:08:32] overall_tsv_creation
[INFO] [2021-11-24 13:08:32] Processing group of 1551 in 1 batches of 10000
[INFO] [2021-11-24 13:09:28] 32568 Traits (unfiltered)...
[INFO] [2021-11-24 13:15:16] 32568 Traits (filtered)...
[INFO] [2021-11-24 13:15:16] 0 Associations (filtered)...
[INFO] [2021-11-24 13:15:30] 113066 metadata added.
[INFO] [2021-11-24 13:15:30] 0 metadata added.
[INFO] [2021-11-24 13:16:15] Average Time: 426.71
[INFO] [2021-11-24 13:16:15] Total Time: 7m44s
[STOP] [2021-11-24 13:16:15] overall_tsv_creation
[INFO] [2021-11-24 13:16:15] Done. Check your files:
[INFO] [2021-11-24 13:16:16] (1551 lines) /app/public/data/coraltraits_tar_/publish_nodes.tsv
[INFO] [2021-11-24 13:16:18] (6194 lines) /app/public/data/coraltraits_tar_/publish_node_ancestors.tsv
[INFO] [2021-11-24 13:16:20] (1551 lines) /app/public/data/coraltraits_tar_/publish_scientific_names.tsv
[INFO] [2021-11-24 13:16:21] (32569 lines) /app/public/data/coraltraits_tar_/publish_traits.tsv
[INFO] [2021-11-24 13:16:23] (113067 lines) /app/public/data/coraltraits_tar_/publish_metadata.tsv
[STOP] [2021-11-24 13:16:23] complete_harvest_instance
[START] [2021-11-24 13:16:23] completed
[STOP] [2021-11-24 13:16:23] completed
[STOP] [2021-11-24 13:16:23] logged process, took 827.42

Latest Process