Harvest for Smithsonian Contributions to Zoology Created 28 Oct 16:30

Stage: completed
Fetched: 28 Oct 16:30
Validated: 28 Oct 16:30
Deltas Created 28 Oct 16:30
Units Normalized: 28 Oct 16:35
Ancestry Built: 28 Oct 16:33
Nodes Matched: 28 Oct 16:35
Names Parsed: 28 Oct 16:33
New Models Stored: 28 Oct 16:32
Indexed: 28 Oct 16:35
Completed: 28 Oct 16:54
Time to Harvest: less than a minute
Harvesting Log

(320 lines)
[INFO] [2021-10-28 16:30:30] Created harvest instance #4070
[STOP] [2021-10-28 16:30:30] create_harvest_instance
[START] [2021-10-28 16:30:30] fetch_files
[STOP] [2021-10-28 16:30:30] fetch_files
[START] [2021-10-28 16:30:30] validate_each_file
[INFO] [2021-10-28 16:30:30] Looping over 5 formats...
[INFO] [2021-10-28 16:30:30] ...nodes (/app/public/data/si_contributions/taxon.tab)
[INFO] [2021-10-28 16:30:30] Valid: /app/public/converted_csv/si_contributions_nodes_4070.csv (12217 lines)
[INFO] [2021-10-28 16:30:30] ...media (/app/public/data/si_contributions/media_resource.tab)
[INFO] [2021-10-28 16:30:32] Valid: /app/public/converted_csv/si_contributions_media_4070.csv (11462 lines)
[INFO] [2021-10-28 16:30:32] ...occurrences (/app/public/data/si_contributions/occurrence_specific.tab)
[INFO] [2021-10-28 16:30:32] Valid: /app/public/converted_csv/si_contributions_occurrences_4070.csv (43179 lines)
[INFO] [2021-10-28 16:30:32] ...assocs (/app/public/data/si_contributions/association.tab)
[INFO] [2021-10-28 16:30:32] Valid: /app/public/converted_csv/si_contributions_assocs_4070.csv (376 lines)
[INFO] [2021-10-28 16:30:32] ...measurements (/app/public/data/si_contributions/measurement_or_fact_specific.tab)
[INFO] [2021-10-28 16:30:34] Valid: /app/public/converted_csv/si_contributions_measurements_4070.csv (42743 lines)
[STOP] [2021-10-28 16:30:34] validate_each_file
[START] [2021-10-28 16:30:34] convert_to_csv
[INFO] [2021-10-28 16:30:34] Looping over 5 formats...
[INFO] [2021-10-28 16:30:34] ...nodes (/app/public/data/si_contributions/taxon.tab)
[CMD] [2021-10-28 16:30:34] /usr/bin/sort /app/public/converted_csv/si_contributions_nodes_4070.csv > /app/public/converted_csv/si_contributions_nodes_4070.csv_sorted
[INFO] [2021-10-28 16:30:34] Converted: /app/public/converted_csv/si_contributions_nodes_4070.csv (12217 lines)
[INFO] [2021-10-28 16:30:34] ...media (/app/public/data/si_contributions/media_resource.tab)
[CMD] [2021-10-28 16:30:34] /usr/bin/sort /app/public/converted_csv/si_contributions_media_4070.csv > /app/public/converted_csv/si_contributions_media_4070.csv_sorted
[INFO] [2021-10-28 16:30:35] Converted: /app/public/converted_csv/si_contributions_media_4070.csv (11462 lines)
[INFO] [2021-10-28 16:30:35] ...occurrences (/app/public/data/si_contributions/occurrence_specific.tab)
[CMD] [2021-10-28 16:30:35] /usr/bin/sort /app/public/converted_csv/si_contributions_occurrences_4070.csv > /app/public/converted_csv/si_contributions_occurrences_4070.csv_sorted
[INFO] [2021-10-28 16:30:35] Converted: /app/public/converted_csv/si_contributions_occurrences_4070.csv (43179 lines)
[INFO] [2021-10-28 16:30:35] ...assocs (/app/public/data/si_contributions/association.tab)
[CMD] [2021-10-28 16:30:35] /usr/bin/sort /app/public/converted_csv/si_contributions_assocs_4070.csv > /app/public/converted_csv/si_contributions_assocs_4070.csv_sorted
[INFO] [2021-10-28 16:30:35] Converted: /app/public/converted_csv/si_contributions_assocs_4070.csv (376 lines)
[INFO] [2021-10-28 16:30:36] ...measurements (/app/public/data/si_contributions/measurement_or_fact_specific.tab)
[CMD] [2021-10-28 16:30:36] /usr/bin/sort /app/public/converted_csv/si_contributions_measurements_4070.csv > /app/public/converted_csv/si_contributions_measurements_4070.csv_sorted
[INFO] [2021-10-28 16:30:36] Converted: /app/public/converted_csv/si_contributions_measurements_4070.csv (42743 lines)
[STOP] [2021-10-28 16:30:36] convert_to_csv
[START] [2021-10-28 16:30:36] calculate_delta
[INFO] [2021-10-28 16:30:36] Looping over 5 formats...
[INFO] [2021-10-28 16:30:36] ...nodes (/app/public/data/si_contributions/taxon.tab)
[CMD] [2021-10-28 16:30:36] echo "0a" > /app/public/diff/si_contributions_nodes_4070.diff
[CMD] [2021-10-28 16:30:36] tail -n +1 /app/public/converted_csv/si_contributions_nodes_4070.csv >> /app/public/diff/si_contributions_nodes_4070.diff
[CMD] [2021-10-28 16:30:37] echo "." >> /app/public/diff/si_contributions_nodes_4070.diff
[INFO] [2021-10-28 16:30:37] Created diff: /app/public/diff/si_contributions_nodes_4070.diff (12219 lines)
[INFO] [2021-10-28 16:30:37] ...media (/app/public/data/si_contributions/media_resource.tab)
[CMD] [2021-10-28 16:30:37] echo "0a" > /app/public/diff/si_contributions_media_4070.diff
[CMD] [2021-10-28 16:30:37] tail -n +1 /app/public/converted_csv/si_contributions_media_4070.csv >> /app/public/diff/si_contributions_media_4070.diff
[CMD] [2021-10-28 16:30:37] echo "." >> /app/public/diff/si_contributions_media_4070.diff
[INFO] [2021-10-28 16:30:38] Created diff: /app/public/diff/si_contributions_media_4070.diff (11464 lines)
[INFO] [2021-10-28 16:30:38] ...occurrences (/app/public/data/si_contributions/occurrence_specific.tab)
[CMD] [2021-10-28 16:30:38] echo "0a" > /app/public/diff/si_contributions_occurrences_4070.diff
[CMD] [2021-10-28 16:30:38] tail -n +1 /app/public/converted_csv/si_contributions_occurrences_4070.csv >> /app/public/diff/si_contributions_occurrences_4070.diff
[CMD] [2021-10-28 16:30:38] echo "." >> /app/public/diff/si_contributions_occurrences_4070.diff
[INFO] [2021-10-28 16:30:39] Created diff: /app/public/diff/si_contributions_occurrences_4070.diff (43181 lines)
[INFO] [2021-10-28 16:30:39] ...assocs (/app/public/data/si_contributions/association.tab)
[CMD] [2021-10-28 16:30:39] echo "0a" > /app/public/diff/si_contributions_assocs_4070.diff
[CMD] [2021-10-28 16:30:39] tail -n +1 /app/public/converted_csv/si_contributions_assocs_4070.csv >> /app/public/diff/si_contributions_assocs_4070.diff
[CMD] [2021-10-28 16:30:39] echo "." >> /app/public/diff/si_contributions_assocs_4070.diff
[INFO] [2021-10-28 16:30:39] Created diff: /app/public/diff/si_contributions_assocs_4070.diff (378 lines)
[INFO] [2021-10-28 16:30:39] ...measurements (/app/public/data/si_contributions/measurement_or_fact_specific.tab)
[CMD] [2021-10-28 16:30:39] echo "0a" > /app/public/diff/si_contributions_measurements_4070.diff
[CMD] [2021-10-28 16:30:40] tail -n +1 /app/public/converted_csv/si_contributions_measurements_4070.csv >> /app/public/diff/si_contributions_measurements_4070.diff
[CMD] [2021-10-28 16:30:40] echo "." >> /app/public/diff/si_contributions_measurements_4070.diff
[INFO] [2021-10-28 16:30:40] Created diff: /app/public/diff/si_contributions_measurements_4070.diff (42745 lines)
[STOP] [2021-10-28 16:30:40] calculate_delta
[START] [2021-10-28 16:30:40] parse_diff_and_store
[INFO] [2021-10-28 16:30:40] Handling diff: /app/public/diff/si_contributions_nodes_4070.diff (12219 lines)
[INFO] [2021-10-28 16:30:40] Loading nodes diff file into memory (12219 /app/public/diff/si_contributions_nodes_4070.diff lines)...
[INFO] [2021-10-28 16:30:44] Handling diff: /app/public/diff/si_contributions_media_4070.diff (11464 lines)
[INFO] [2021-10-28 16:30:44] Loading media diff file into memory (11464 /app/public/diff/si_contributions_media_4070.diff lines)...
[INFO] [2021-10-28 16:31:07] Handling diff: /app/public/diff/si_contributions_occurrences_4070.diff (43181 lines)
[INFO] [2021-10-28 16:31:08] Loading occurrences diff file into memory (43181 /app/public/diff/si_contributions_occurrences_4070.diff lines)...
[INFO] [2021-10-28 16:31:12] Handling diff: /app/public/diff/si_contributions_assocs_4070.diff (378 lines)
[INFO] [2021-10-28 16:31:12] Loading assocs diff file into memory (378 /app/public/diff/si_contributions_assocs_4070.diff lines)...
[INFO] [2021-10-28 16:31:12] Handling diff: /app/public/diff/si_contributions_measurements_4070.diff (42745 lines)
[INFO] [2021-10-28 16:31:13] Loading measurements diff file into memory (42745 /app/public/diff/si_contributions_measurements_4070.diff lines)...
[INFO] [2021-10-28 16:31:37] Storing 12217 ScientificNames
[INFO] [2021-10-28 16:31:37] Processing group of 12217 in 13 groups of 1000
[INFO] [2021-10-28 16:31:41] Average Time: 0.277
[INFO] [2021-10-28 16:31:41] Total Time: 4s
[INFO] [2021-10-28 16:31:41] last 3 / first 3: 0.78
[INFO] [2021-10-28 16:31:41] Std.Dev: 0.07071067811865475; Max: 0.39
[INFO] [2021-10-28 16:31:41] Storing 12217 Nodes
[INFO] [2021-10-28 16:31:41] Processing group of 12217 in 13 groups of 1000
[INFO] [2021-10-28 16:31:44] Average Time: 0.228
[INFO] [2021-10-28 16:31:44] Total Time: 4s
[INFO] [2021-10-28 16:31:44] last 3 / first 3: 0.83
[INFO] [2021-10-28 16:31:44] Std.Dev: 0.05477225575051661; Max: 0.3
[INFO] [2021-10-28 16:31:44] Storing 11311 BibliographicCitations
[INFO] [2021-10-28 16:31:44] Processing group of 11311 in 12 groups of 1000
[INFO] [2021-10-28 16:31:46] Average Time: 0.156
[INFO] [2021-10-28 16:31:46] Total Time: 2s
[INFO] [2021-10-28 16:31:46] last 3 / first 3: 0.82
[INFO] [2021-10-28 16:31:46] Std.Dev: 0.044721359549995794; Max: 0.22
[INFO] [2021-10-28 16:31:46] Storing 11462 ArticlesSections
[INFO] [2021-10-28 16:31:46] Processing group of 11462 in 12 groups of 1000
[INFO] [2021-10-28 16:31:47] Average Time: 0.054
[INFO] [2021-10-28 16:31:47] Total Time: 1s
[INFO] [2021-10-28 16:31:47] last 3 / first 3: 0.8
[INFO] [2021-10-28 16:31:47] Std.Dev: 0.03162277660168379; Max: 0.13
[INFO] [2021-10-28 16:31:47] Storing 11462 Articles
[INFO] [2021-10-28 16:31:47] Processing group of 11462 in 12 groups of 1000
[INFO] [2021-10-28 16:31:54] Average Time: 0.573
[INFO] [2021-10-28 16:31:54] Total Time: 8s
[INFO] [2021-10-28 16:31:54] last 3 / first 3: 0.79
[INFO] [2021-10-28 16:31:54] Std.Dev: 0.1816590212458495; Max: 0.95
[INFO] [2021-10-28 16:31:54] Storing 43179 Occurrences
[INFO] [2021-10-28 16:31:54] Processing group of 43179 in 44 groups of 1000
[INFO] [2021-10-28 16:32:00] Average Time: 0.127
[INFO] [2021-10-28 16:32:00] Total Time: 6s
[INFO] [2021-10-28 16:32:00] last 3 / first 3: 0.69
[INFO] [2021-10-28 16:32:00] Std.Dev: 0.07745966692414834; Max: 0.58
[INFO] [2021-10-28 16:32:00] Storing 376 Assocs
[INFO] [2021-10-28 16:32:00] Processing group of 376 in 1 groups of 1000
[INFO] [2021-10-28 16:32:00] Average Time: 0.1
[INFO] [2021-10-28 16:32:00] Total Time: 1s
[INFO] [2021-10-28 16:32:00] Storing 752 MetaAssocs
[INFO] [2021-10-28 16:32:00] Processing group of 752 in 1 groups of 1000
[INFO] [2021-10-28 16:32:00] Average Time: 0.11
[INFO] [2021-10-28 16:32:00] Total Time: 1s
[INFO] [2021-10-28 16:32:00] Storing 42743 Traits
[INFO] [2021-10-28 16:32:00] Processing group of 42743 in 43 groups of 1000
[INFO] [2021-10-28 16:32:15] Average Time: 0.352
[INFO] [2021-10-28 16:32:15] Total Time: 16s
[INFO] [2021-10-28 16:32:15] last 3 / first 3: 0.87
[INFO] [2021-10-28 16:32:15] Std.Dev: 0.130384048104053; Max: 0.85
[INFO] [2021-10-28 16:32:15] Storing 85169 MetaTraits
[INFO] [2021-10-28 16:32:15] Processing group of 85169 in 86 groups of 1000
[INFO] [2021-10-28 16:32:27] Average Time: 0.131
[INFO] [2021-10-28 16:32:27] Total Time: 12s
[INFO] [2021-10-28 16:32:27] last 3 / first 3: 0.56
[INFO] [2021-10-28 16:32:27] Std.Dev: 0.03162277660168379; Max: 0.31
[STOP] [2021-10-28 16:32:27] parse_diff_and_store
[START] [2021-10-28 16:32:27] resolve_keys
[INFO] [2021-10-28 16:32:52] Occurrences to nodes (through scientific_names)...
[INFO] [2021-10-28 16:32:52] traits to occurrences...
[INFO] [2021-10-28 16:32:54] traits to nodes (through occurrences)...
[INFO] [2021-10-28 16:32:55] Traits to sex term...
[INFO] [2021-10-28 16:32:56] Traits to lifestage term...
[INFO] [2021-10-28 16:32:57] MetaTraits to traits...
[INFO] [2021-10-28 16:32:59] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-10-28 16:32:59] Assocs to occurrences...
[INFO] [2021-10-28 16:32:59] Assocs to nodes...
[INFO] [2021-10-28 16:33:00] Assoc to sex term...
[INFO] [2021-10-28 16:33:00] Assoc to lifestage term...
[INFO] [2021-10-28 16:33:00] MetaAssoc to assocs...
[STOP] [2021-10-28 16:33:00] resolve_keys
[START] [2021-10-28 16:33:00] hold_for_later_1
[STOP] [2021-10-28 16:33:00] hold_for_later_1
[START] [2021-10-28 16:33:00] hold_for_later_2
[STOP] [2021-10-28 16:33:00] hold_for_later_2
[START] [2021-10-28 16:33:00] resolve_missing_parents
[STOP] [2021-10-28 16:33:00] resolve_missing_parents
[START] [2021-10-28 16:33:00] rebuild_nodes
[START] [2021-10-28 16:33:00] Flattener#flatten
[START] [2021-10-28 16:33:00] Flattener#study_resource
[START] [2021-10-28 16:33:00] Flattener#build_ancestry
[STOP] [2021-10-28 16:33:00] Flattener#build_ancestry
[INFO] [2021-10-28 16:33:00] 12217 ancestry keys
[START] [2021-10-28 16:33:00] build_node_ancestors
[INFO] [2021-10-28 16:33:00] old ancestors deleted.
[STOP] [2021-10-28 16:33:00] build_node_ancestors
[WARN] [2021-10-28 16:33:00] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2021-10-28 16:33:00] Flattener#flatten
[STOP] [2021-10-28 16:33:00] rebuild_nodes
[START] [2021-10-28 16:33:00] resolve_missing_media_owners
[STOP] [2021-10-28 16:33:00] resolve_missing_media_owners
[START] [2021-10-28 16:33:00] sanitize_media_verbatims
[STOP] [2021-10-28 16:33:00] sanitize_media_verbatims
[START] [2021-10-28 16:33:00] queue_downloads
[STOP] [2021-10-28 16:33:00] queue_downloads
[START] [2021-10-28 16:33:00] parse_names
[WARN] [2021-10-28 16:33:00] I see 12217 names which still need to be parsed.
[WARN] [2021-10-28 16:33:01] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2021-10-28 16:33:09] Names to parse: 2217 formatted: 2217 learned: 2217 parsed: 2217
[STOP] [2021-10-28 16:33:11] parse_names
[START] [2021-10-28 16:33:11] denormalize_canonical_names_to_nodes
[STOP] [2021-10-28 16:33:11] denormalize_canonical_names_to_nodes
[START] [2021-10-28 16:33:11] match_nodes
[START] [2021-10-28 16:33:11] map_all_nodes_to_pages
[STOP] [2021-10-28 16:35:38] map_all_nodes_to_pages
[INFO] [2021-10-28 16:35:38] 1827 Unmatched nodes (of 12217)! That's too many to output. Full list in /app/public/data/si_contributions/unmatched_nodes.txt ; First 10: Canonical: Ataenius nocturnus; Node#100941104; ResourceID: 0011ed12d7f09df2e268acc4bdf3e7c9; Canonical: Dysoptus prolatus; Node#100941132; ResourceID: 00810839c6f7e80bfdcf83a35c7f3a83; Canonical: Limonia recondita civica; Node#100941140; ResourceID: 00a220bfdebaa126c5c88c21b830135e; Canonical: Lempos waipio; Node#100941144; ResourceID: 00d92560a02e8a576f30adedd4490543; Canonical: Antarctia honora; Node#100941145; ResourceID: 00db7fa4c2dc84aa8c92c1b40662c8a4; Canonical: Petrolisthes rosariensis; Node#100941159; ResourceID: 011573a897f62f1f74365322bea398ba; Canonical: Gammaropsis pali; Node#100941176; ResourceID: 016fecac26b8751c1eb3067c90faef71; Canonical: Liris evansi; Node#100941180; ResourceID: 019611738d19bddf97801e2677294cb6; Canonical: Octopus oshimai; Node#100941181; ResourceID: 019e09f3a5618ea9a80497c6195e027c; Canonical: Creagrudite melanzonus; Node#100941182; ResourceID: 01a0d0b8c5688e10e17bb320d3f76af7
[START] [2021-10-28 16:35:38] update_nodes
[STOP] [2021-10-28 16:35:42] update_nodes
[STOP] [2021-10-28 16:35:42] match_nodes
[START] [2021-10-28 16:35:42] reindex_search
[STOP] [2021-10-28 16:35:49] reindex_search
[START] [2021-10-28 16:35:49] normalize_units
[STOP] [2021-10-28 16:35:49] normalize_units
[START] [2021-10-28 16:35:49] calculate_statistics
[2021-10-28 16:35:49] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[2021-10-28 16:35:50] (NEAR) DUPLICATE TRAITS FOUND! There are only 42459 (of 42743 total) unique traits.
[2021-10-28 16:43:34] (Near) duplicate trait pairs (up to 100):
[2021-10-28 16:43:34] (resource_pk: 001b2ed8ef00c364f7b9353827090230_10088_5097_ENV, id: 235846080), (resource_pk: 450ee630da7a7c9a91bfdf0ebeab8eac_10088_5097_ENV, id: 235857681)
[2021-10-28 16:43:34] (resource_pk: 0033a65d59ad159d9dc9f5a2263c813a_10088_5097_ENV, id: 235846092), (resource_pk: 2c30048941bbda61fa1a3eb27b3cc5eb_10088_5097_ENV, id: 235853467)
[2021-10-28 16:43:34] (resource_pk: 00705e3cbaf068584ad4b47f1e47d79e_10088_5097_ENV, id: 235846131), (resource_pk: f80453d0296914e4c0e54ac8395a140b_10088_5097_ENV, id: 235887470)
[2021-10-28 16:43:34] (resource_pk: 00d3855fbfe8094aa50257f67b6a30cd_10088_5097_ENV, id: 235846201), (resource_pk: ef68ff2c4d6df497dc5ca3ed15e5f88a_10088_5097_ENV, id: 235886045)
[2021-10-28 16:43:34] (resource_pk: 00fdb24a0eb9e84becf610a130cb9924_10088_5097_ENV, id: 235846228), (resource_pk: 440030840832278a6cab9f26134ea171_10088_5097_ENV, id: 235857491)
[2021-10-28 16:43:34] (resource_pk: 018880f15edba09712d3a02051407b06_10088_5097_ENV, id: 235846326), (resource_pk: c6eb7d5ffc073f7972aec32838cbcaa1_10088_5097_ENV, id: 235879410)
[2021-10-28 16:43:34] (resource_pk: 0261462e5fa6bf0236ed120662bd9386_10088_5097_ENV, id: 235846454), (resource_pk: d0ae4e729e77bb51c9d3f576fb4b1eee_10088_5097_ENV, id: 235881008)
[2021-10-28 16:43:34] (resource_pk: 0308685ed36f56b84deb84b112c8a4a2_10088_5097_ENV, id: 235846577), (resource_pk: 7fbcd8088e4a2cfe5fdb9d744f08c355_10088_5097_ENV, id: 235867480)
[2021-10-28 16:43:34] (resource_pk: 0308685ed36f56b84deb84b112c8a4a2_10088_5097_ENV, id: 235846577), (resource_pk: b5e6c591bd5061da57bc2e873c55eb9b_10088_5097_ENV, id: 235876522)
[2021-10-28 16:43:34] (resource_pk: 030e73e7efbdf465da8dede0a93b35f1_10088_5097_ENV, id: 235846582), (resource_pk: 9fe9b08bbb0cc6948f5cc28a4b432566_10088_5097_ENV, id: 235872877)
[2021-10-28 16:43:34] (resource_pk: 032351440e9f8e144e539045d5a91ac8_10088_5097_ENV, id: 235846594), (resource_pk: fe73fbb8c9044e0cb98f403ebc54029f_10088_5097_ENV, id: 235888536)
[2021-10-28 16:43:34] (resource_pk: 041540e83daa22dffdc52ef10c75c3e2_10088_5097_ENV, id: 235846753), (resource_pk: 24a7e134f19eea7ab518a58f6f66a511_10088_5097_ENV, id: 235852174)
[2021-10-28 16:43:34] (resource_pk: 0436b8885b525a1d79d94262ed7a144a_10088_5097_ENV, id: 235846772), (resource_pk: 668b8f75ae1ee02faa40a7387e3d1646_10088_5097_ENV, id: 235863165)
[2021-10-28 16:43:34] (resource_pk: 045d83f7f500c06668c71a7e38993dca_10088_5097_ENV, id: 235846798), (resource_pk: e8184b036a3fa4f67f4d6396e84f18bc_10088_5097_ENV, id: 235884885)
[2021-10-28 16:43:34] (resource_pk: 0484854f7d6ee635be0e9853041d8e13_10088_5097_ENV, id: 235846830), (resource_pk: 7845fda0f1a0fadc243ec8fd95b2960f_10088_5097_ENV, id: 235866223)
[2021-10-28 16:43:34] (resource_pk: 04d985ce62a3d1e88242e943d05a7a0d_10088_5097_ENV, id: 235846899), (resource_pk: b31a7ef4be541c02ff82e688d3b8e8f7_10088_5097_ENV, id: 235876057)
[2021-10-28 16:43:34] (resource_pk: 066c373bda2996bf2fc27755d12fa545_10088_5097_ENV, id: 235847147), (resource_pk: 3b2b51e75c3c1f82f2783c90e92aa2b4_10088_5097_ENV, id: 235856014)
[2021-10-28 16:43:34] (resource_pk: 068d41060b0a6ba467efb501b577c3ec_10088_5097_ENV, id: 235847168), (resource_pk: 8b2c63dfc81db3896d29c208c32f9863_10088_5097_ENV, id: 235869407)
[2021-10-28 16:43:34] (resource_pk: 06a961f949579c9912beab7f283c1b2c_10088_5097_ENV, id: 235847197), (resource_pk: 58b02f17d4a247a3689e885306157d17_10088_5097_ENV, id: 235860910)
[2021-10-28 16:43:34] (resource_pk: 07519825ab306e1c8c594ba9184de14a_10088_5097_ENV, id: 235847300), (resource_pk: e789ece3bbc9e3b447bd7d9b9a497c64_10088_5097_ENV, id: 235884808)
[2021-10-28 16:43:34] (resource_pk: 07756964030d1495806fec8091b6debf_10088_5097_ENV, id: 235847325), (resource_pk: f825ac5674996a7a8e25529467bff0da_10088_5097_ENV, id: 235887485)
[2021-10-28 16:43:34] (resource_pk: 0811d833ce6a5c63d933de9d7346b03b_10088_5097_ENV, id: 235847436), (resource_pk: f38e4f261540e7151ac7ad13b92b2a87_10088_5097_ENV, id: 235886745)
[2021-10-28 16:43:34] (resource_pk: 085713a99be86af1945df7d03448ad4f_10088_5097_ENV, id: 235847486), (resource_pk: 5714229ad2f37b716d8a3ef6cbf11491_10088_5097_ENV, id: 235860660)
[2021-10-28 16:43:34] (resource_pk: 085713a99be86af1945df7d03448ad4f_10088_5097_ENV, id: 235847486), (resource_pk: bb47b383c14c7a009c3f91584e645021_10088_5097_ENV, id: 235877443)
[2021-10-28 16:43:34] (resource_pk: 086b649f5b557392140c1b0808f4e917_10088_5097_ENV, id: 235847501), (resource_pk: 31c92691a92c3e954d3ee32ef8ddd1a4_10088_5097_ENV, id: 235854409)
[2021-10-28 16:43:34] (resource_pk: 09087764031bd62cb85ac87803f04e77_10088_5097_ENV, id: 235847579), (resource_pk: a6851f2be22015370be206d30e8f6447_10088_5097_ENV, id: 235873971)
[2021-10-28 16:43:34] (resource_pk: 09521f3e4527922bd56114f0638bb857_10088_5097_ENV, id: 235847637), (resource_pk: 1c7740e057eeeddf373ae696495a9eb1_10088_5097_ENV, id: 235850797)
[2021-10-28 16:43:34] (resource_pk: 0a57abbcd9ea277f0b205222284cf9dd_10088_5097_ENV, id: 235847808), (resource_pk: 31d0eda92acb35ecbb5c7b0a41560598_10088_5097_ENV, id: 235854414)
[2021-10-28 16:43:34] (resource_pk: 0c2bd2076ce776872f797f3507e5d876_10088_5097_ENV, id: 235848126), (resource_pk: b47e8479393e5118bb7334cde0789c37_10088_5097_ENV, id: 235876279)
[2021-10-28 16:43:34] (resource_pk: 0c3599ef9544888747ffac9949dd2788_10088_5097_ENV, id: 235848132), (resource_pk: bb1b6939a81dc658c231ef4f46b24b81_10088_5097_ENV, id: 235877405)
[2021-10-28 16:43:34] (resource_pk: 0c61e7f21c0dbd02e3cd0f2135811c03_10088_5097_ENV, id: 235848172), (resource_pk: 73dfbf3432dfd3b43b9664eb97d41ba3_10088_5097_ENV, id: 235865477)
[2021-10-28 16:43:34] (resource_pk: 0d187cb945bff27a05d6ff82f96b260e_10088_5097_ENV, id: 235848290), (resource_pk: b6ce8f5332bb68d9d62b3f0481fdf5bb_10088_5097_ENV, id: 235876683)
[2021-10-28 16:43:34] (resource_pk: 0d628788ba751a97f6fd2ab85e91e01c_10088_5097_ENV, id: 235848347), (resource_pk: 132c00efc3e9d074b37fdc2811affc13_10088_5097_ENV, id: 235849285)
[2021-10-28 16:43:34] (resource_pk: 0dd8bd825831b74ecd9e2183df13187f_10088_5097_ENV, id: 235848436), (resource_pk: 837dc3631407d42a51fc10a245ed3c72_10088_5097_ENV, id: 235868136)
[2021-10-28 16:43:34] (resource_pk: 0dd8bd825831b74ecd9e2183df13187f_10088_5097_ENV, id: 235848436), (resource_pk: e4d81b64e8c833f691319b34e83e60fa_10088_5097_ENV, id: 235884406)
[2021-10-28 16:43:34] (resource_pk: 0e24553c3773e20a8eb20fd2becdc708_10088_5097_ENV, id: 235848486), (resource_pk: d709ee4a62810c47a5b352c668ef3d06_10088_5097_ENV, id: 235882095)
[2021-10-28 16:43:34] (resource_pk: 0e65099386ae974d0405e16f4ac86043_10088_5097_ENV, id: 235848532), (resource_pk: 4b1b6637342790f81cd99544ade7469b_10088_5097_ENV, id: 235858659)
[2021-10-28 16:43:34] (resource_pk: 0e82849bf82a33d130d0fff986718dd8_10088_5097_ENV, id: 235848550), (resource_pk: 71da64c69edc62c49e49dbc861144189_10088_5097_ENV, id: 235865132)
[2021-10-28 16:43:34] (resource_pk: 0ec0fa0da01359370e57b627420f4b00_10088_5097_ENV, id: 235848603), (resource_pk: b4b15af78e73c9147462a4e3fde0cb42_10088_5097_ENV, id: 235876302)
[2021-10-28 16:43:34] (resource_pk: 0ecf26443f5f6982eb0ebeae042597e1_10088_5097_ENV, id: 235848613), (resource_pk: e3c20450edc33510318597815cf3cd9a_10088_5097_ENV, id: 235884222)
[2021-10-28 16:43:34] (resource_pk: 0f0e59fc25cb40a9e7dde506351ff915_10088_5097_ENV, id: 235848654), (resource_pk: ad858c60f64e36febea3c2d005d3032a_10088_5097_ENV, id: 235875178)
[2021-10-28 16:43:34] (resource_pk: 0f8f247c466260cc4f0190f6aad0c4b8_10088_5097_ENV, id: 235848736), (resource_pk: 3d1ee4f3ee1b3189d6d00cb049db04e3_10088_5097_ENV, id: 235856357)
[2021-10-28 16:43:34] (resource_pk: 0ff03feec33ffa9cd9ef4df068fc79ca_10088_5097_ENV, id: 235848801), (resource_pk: c8ab64e35dffb879139db56916853f59_10088_5097_ENV, id: 235879705)
[2021-10-28 16:43:34] (resource_pk: 100d5a42bf55c5d268b5e06762743d6c_10088_5097_ENV, id: 235848821), (resource_pk: 9791df0bbac1261fab194cd34f587cac_10088_5097_ENV, id: 235871551)
[2021-10-28 16:43:34] (resource_pk: 107081dcee94329265bd200a761f2278_10088_5097_ENV, id: 235848891), (resource_pk: b7897a660ce14392ee4fd8752450dbd7_10088_5097_ENV, id: 235876805)
[2021-10-28 16:43:34] (resource_pk: 10e59d95e07e95cae8299024aa1bc3a3_10088_5097_ENV, id: 235848953), (resource_pk: 5a10b6a115585f1682bf3a58acd26a6a_10088_5097_ENV, id: 235861129)
[2021-10-28 16:43:34] (resource_pk: 111469bd40fcdc7fc6016b986c8cbadb_10088_5097_ENV, id: 235848980), (resource_pk: 9313c913132d824d6148821f622a3872_10088_5097_ENV, id: 235870755)
[2021-10-28 16:43:34] (resource_pk: 1141a0933bfcac262b2f43016f2f2ea7_10088_5097_ENV, id: 235849007), (resource_pk: a542970119d27aa45012bde22d6b59f6_10088_5097_ENV, id: 235873764)
[2021-10-28 16:43:34] (resource_pk: 11f0ea70eff665c4a0179c99983fac6f_10088_5097_ENV, id: 235849095), (resource_pk: a47816c3c3de1fb58351a7338d303f9b_10088_5097_ENV, id: 235873622)
[2021-10-28 16:43:34] (resource_pk: 14259be48df1e0479e048d15165b2be4_10088_5097_ENV, id: 235849439), (resource_pk: b309e18d3fcee2b91579fc858f0eaae1_10088_5097_ENV, id: 235876044)
[2021-10-28 16:43:34] (resource_pk: 143b8244fa7333f9bed6874b140d37ee_10088_5097_ENV, id: 235849450), (resource_pk: 2dd28a71a594b6e9af8f4ba616689126_10088_5097_ENV, id: 235853721)
[2021-10-28 16:43:34] (resource_pk: 1473618ba5b9b18634a796326c157f90_10088_5097_ENV, id: 235849481), (resource_pk: a52ff5b6f43fae554a4568504bc1cda1_10088_5097_ENV, id: 235873755)
[2021-10-28 16:43:34] (resource_pk: 14dc2c28d8d2918955d6f0d910539c21_10088_5097_ENV, id: 235849539), (resource_pk: d073cdbbbf4e04b17c42e8363b82393d_10088_5097_ENV, id: 235880957)
[2021-10-28 16:43:34] (resource_pk: 14e0120e808162c25a0acf2deac80943_10088_5097_ENV, id: 235849546), (resource_pk: e0781ff16ce7defd352f78816fa181ca_10088_5097_ENV, id: 235883662)
[2021-10-28 16:43:34] (resource_pk: 15651674d293bdc7e45cc25ba895e02f_10088_5097_ENV, id: 235849633), (resource_pk: ff0c5798e61e52d9c6f6aff1376599c9_10088_5097_ENV, id: 235888635)
[2021-10-28 16:43:34] (resource_pk: 167e8af7b13272e4cd8cd27b90f5143d_10088_5097_ENV, id: 235849799), (resource_pk: 8a5da5be7e94e3ef1559eb74fe051c10_10088_5097_ENV, id: 235869255)
[2021-10-28 16:43:34] (resource_pk: 17afe6c652833de16b2b535a63adbc8e_10088_5097_ENV, id: 235849990), (resource_pk: c285f25211e5ae1b6be5e1b0ba6f3a32_10088_5097_ENV, id: 235878618)
[2021-10-28 16:43:34] (resource_pk: 18cf84ba5ed9bde17dad05207be4875c_10088_5097_ENV, id: 235850164), (resource_pk: 2a58cce0c8923795aa0f701c27a9cf37_10088_5097_ENV, id: 235853154)
[2021-10-28 16:43:34] (resource_pk: 18d5ff2fc4971d1fb4b7c11a0e58137f_10088_5097_ENV, id: 235850165), (resource_pk: 678a3bb41f0d88a1f7a9645e91e972d8_10088_5097_ENV, id: 235863327)
[2021-10-28 16:43:34] (resource_pk: 19150d97b4992f27e4816c7c0f5517a9_10088_5097_ENV, id: 235850218), (resource_pk: 989c3339a5a5c896b38f544f7cafd3cd_10088_5097_ENV, id: 235871711)
[2021-10-28 16:43:34] (resource_pk: 198be0f63ab4d30562e45cabdbb77b70_10088_5097_ENV, id: 235850278), (resource_pk: 456082525c7d8902d0aa8655aac3a0b3_10088_5097_ENV, id: 235857723)
[2021-10-28 16:43:34] (resource_pk: 1a9fc6c5ae3dfe6d2ce45ca9fd4acc47_10088_5097_ENV, id: 235850477), (resource_pk: 894e4a05ca1352e90f0d37d89abb05b6_10088_5097_ENV, id: 235869052)
[2021-10-28 16:43:34] (resource_pk: 1aa0dbad1ce701087d8e9ad8f73a34a9_10088_5097_ENV, id: 235850478), (resource_pk: 571cf42eb6b5ac998b653ff631040eb0_10088_5097_ENV, id: 235860672)
[2021-10-28 16:43:34] (resource_pk: 1b24bdc6bedcf91df2e86cf783ee2df3_10088_5097_ENV, id: 235850561), (resource_pk: ae999df7e1764334de82dbde77256e6a_10088_5097_ENV, id: 235875336)
[2021-10-28 16:43:34] (resource_pk: 1b8601aaaa8c9976c17497616a431591_10088_5097_ENV, id: 235850634), (resource_pk: b1913a1e1445fef31e1d014313214d89_10088_5097_ENV, id: 235875823)
[2021-10-28 16:43:34] (resource_pk: 1bddf19335917401b3abe227ccd8cde7_10088_5097_ENV, id: 235850693), (resource_pk: 38fbe7647dbd5ca70a47cecefd74c374_10088_5097_ENV, id: 235855665)
[2021-10-28 16:43:34] (resource_pk: 1c265aa3de80aed22788e4e6a864e7a9_10088_5097_ENV, id: 235850743), (resource_pk: 4091f2dfe65a35475b17867a7381612c_10088_5097_ENV, id: 235856919)
[2021-10-28 16:43:34] (resource_pk: 1c60f207b2e6a12e64d445c3de44cb47_10088_5097_ENV, id: 235850780), (resource_pk: b0f7814951381fb14c4f49c2542b284a_10088_5097_ENV, id: 235875732)
[2021-10-28 16:43:34] (resource_pk: 1e127aff3932098ab84dbc6825449dd1_10088_5097_ENV, id: 235851051), (resource_pk: 6327ca94e8f99d834205c0f4b0c9ce90_10088_5097_ENV, id: 235862611)
[2021-10-28 16:43:34] (resource_pk: 1e1756bbe2442834a0dae71f64d722d5_10088_5097_ENV, id: 235851057), (resource_pk: 35f1900294fafd09a46ff8b69a43d194_10088_5097_ENV, id: 235855122)
[2021-10-28 16:43:34] (resource_pk: 20b65c30f7e11940334f4294c31429a2_10088_5097_ENV, id: 235851502), (resource_pk: c5cbd68a8218df8ac365e566bd5a475b_10088_5097_ENV, id: 235879200)
[2021-10-28 16:43:34] (resource_pk: 20c901171b3070d56b47af4aed4cd04e_10088_5097_ENV, id: 235851517), (resource_pk: ee60ee3822334ca3d5303bc1030ce2b5_10088_5097_ENV, id: 235885903)
[2021-10-28 16:43:34] (resource_pk: 214426d1eded8c690b6f53cc3444c4f7_10088_5097_ENV, id: 235851609), (resource_pk: 509de090af279277da674cede4b01744_10088_5097_ENV, id: 235859574)
[2021-10-28 16:43:34] (resource_pk: 214426d1eded8c690b6f53cc3444c4f7_10088_5097_ENV, id: 235851609), (resource_pk: a1d180eb3b44c4ec760fe872fbedd44e_10088_5097_ENV, id: 235873185)
[2021-10-28 16:43:34] (resource_pk: 214426d1eded8c690b6f53cc3444c4f7_10088_5097_ENV, id: 235851609), (resource_pk: b0600c44d506975b658838f9b65fe548_10088_5097_ENV, id: 235875644)
[2021-10-28 16:43:34] (resource_pk: 214426d1eded8c690b6f53cc3444c4f7_10088_5097_ENV, id: 235851609), (resource_pk: d30db32d60a83c4722005f890cdc5bea_10088_5097_ENV, id: 235881390)
[2021-10-28 16:43:34] (resource_pk: 2175ca13cd4ec055996402c794a2c99b_10088_5097_ENV, id: 235851643), (resource_pk: 755c2bd932b493faf2662510c05e9c60_10088_5097_ENV, id: 235865752)
[2021-10-28 16:43:34] (resource_pk: 220581d00a62a38b1d1a25b1ce5738a3_10088_5097_ENV, id: 235851734), (resource_pk: 42bdb8a4f9adf1e89f1834b0dca882f7_10088_5097_ENV, id: 235857273)
[2021-10-28 16:43:34] (resource_pk: 23f65bc5044de29d06a65b602ba77b17_10088_5097_ENV, id: 235852042), (resource_pk: f62eea4efb82e11a90e441d4666d0341_10088_5097_ENV, id: 235887164)
[2021-10-28 16:43:34] (resource_pk: 24a8ca88dc7d16094ea3a2ce5229603d_10088_5097_ENV, id: 235852177), (resource_pk: 4bcc787dcbda10a3e1b6187dbb5fa176_10088_5097_ENV, id: 235858777)
[2021-10-28 16:43:34] (resource_pk: 25d5d53ad1a44565a99ebcafcf145315_10088_5097_ENV, id: 235852397), (resource_pk: 60f4c3c2fbda4b8efa9d7e2ce3e5edaa_10088_5097_ENV, id: 235862255)
[2021-10-28 16:43:34] (resource_pk: 2643043141df33a7c76448ca88ac97a4_10088_5097_ENV, id: 235852462), (resource_pk: 69a2ea05d5b8d0de2524e83a42809a74_10088_5097_ENV, id: 235863708)
[2021-10-28 16:43:34] (resource_pk: 26fa5d8bb10edcf5f98c610f38d2f8ae_10088_5097_ENV, id: 235852595), (resource_pk: 647f5f280e94f7e0aab2b4c1f2a7e2c4_10088_5097_ENV, id: 235862830)
[2021-10-28 16:43:34] (resource_pk: 277c093db613038245884348e5edb1a9_10088_5097_ENV, id: 235852683), (resource_pk: 5224a13fafbddb951f46bf9d37cc3708_10088_5097_ENV, id: 235859831)
[2021-10-28 16:43:34] (resource_pk: 28a2dbe3b6ef65783032b16c62e2e3d1_10088_5097_ENV, id: 235852876), (resource_pk: 84c0ff4f30ed0a6f10fccbca95dd9ba2_10088_5097_ENV, id: 235868297)
[2021-10-28 16:43:34] (resource_pk: 29427ee1909db218839d3e44229ffc79_10088_5097_ENV, id: 235852964), (resource_pk: 2f0e2f9e1fb3d7dcca4e4cc679fb2875_10088_5097_ENV, id: 235853935)
[2021-10-28 16:43:34] (resource_pk: 294359aeb56b5f5db0031d421b1bcdc2_10088_5097_ENV, id: 235852965), (resource_pk: 35572d3c54d504e2f0e158c376332399_10088_5097_ENV, id: 235855014)
[2021-10-28 16:43:34] (resource_pk: 297f5a05ab91e68eb43a3a709c6a52f7_10088_5097_ENV, id: 235853009), (resource_pk: 396e818e12141a293cb62669ce3497fc_10088_5097_ENV, id: 235855738)
[2021-10-28 16:43:34] (resource_pk: 298de981b4a5c4a78d19b9348be5a649_10088_5097_ENV, id: 235853023), (resource_pk: e1c72d951cd3c2fcea04b085a1576d82_10088_5097_ENV, id: 235883896)
[2021-10-28 16:43:34] (resource_pk: 2a66f6fdc54973550c644967cd2ff668_10088_5097_ENV, id: 235853163), (resource_pk: c033308e4e91dd32e2e37b65caa76631_10088_5097_ENV, id: 235878220)
[2021-10-28 16:43:34] (resource_pk: 2aef0b5fb55923d7a1401d0c729bb03c_10088_5097_ENV, id: 235853254), (resource_pk: d976438e0af55435099886b91732001b_10088_5097_ENV, id: 235882484)
[2021-10-28 16:43:34] (resource_pk: 2b04be89173a42c68fa02227cfc8deba_10088_5097_ENV, id: 235853270), (resource_pk: d1dfa4ad2f5fd4d706037d1563c579d7_10088_5097_ENV, id: 235881207)
[2021-10-28 16:43:34] (resource_pk: 2c6b0963e8e949e1c2a266ec994ffa0e_10088_5097_ENV, id: 235853493), (resource_pk: 8ac312e4700f28303b13c31b4963ba2c_10088_5097_ENV, id: 235869325)
[2021-10-28 16:43:34] (resource_pk: 2cf9c1cf65bb683865b0b8f0e052124e_10088_5097_ENV, id: 235853583), (resource_pk: a1f4691b11f7e370b4250020af167739_10088_5097_ENV, id: 235873212)
[2021-10-28 16:43:34] (resource_pk: 2d3f3d0cc766c35ca2de0cbabbdc8c57_10088_5097_ENV, id: 235853623), (resource_pk: 421fe911dacbb3b155ff82eb7df26881_10088_5097_ENV, id: 235857172)
[2021-10-28 16:43:34] (resource_pk: 2e1bce6a1aaf7424fdd2d877b48658fb_10088_5097_ENV, id: 235853768), (resource_pk: b6faf4c10116a88b436d943b252feb80_10088_5097_ENV, id: 235876718)
[2021-10-28 16:43:34] (resource_pk: 2ec728c2e1fd550293db08c14e7da30d_10088_5097_ENV, id: 235853886), (resource_pk: f5b14610b1e803cf4980f0f14f268016_10088_5097_ENV, id: 235887086)
[2021-10-28 16:43:34] (resource_pk: 2fcd73b0dfc38916e73d1bb0aa719dce_10088_5097_ENV, id: 235854078), (resource_pk: 670ca76140e0787d7cfdbebb615e2b87_10088_5097_ENV, id: 235863238)
[2021-10-28 16:43:34] (resource_pk: 301e32c6c7658ec12294038fe3056c2f_10088_5097_ENV, id: 235854118), (resource_pk: d469289e6cd79ef6ef777fa8d684fe4d_10088_5097_ENV, id: 235881639)
[2021-10-28 16:43:34] (resource_pk: 302a65431b60ef7089ef849c1d4ffe91_10088_5097_ENV, id: 235854121), (resource_pk: 6a691e9d06a0a40638970453de09a2c7_10088_5097_ENV, id: 235863842)
[STOP] [2021-10-28 16:43:34] calculate_statistics
[START] [2021-10-28 16:43:34] complete_harvest_instance
[START] [2021-10-28 16:43:34] overall_tsv_creation
[INFO] [2021-10-28 16:43:34] Processing group of 12217 in 2 batches of 10000
[INFO] [2021-10-28 16:44:55] 34852 Traits (unfiltered)...
[INFO] [2021-10-28 16:49:56] 34852 Traits (filtered)...
[INFO] [2021-10-28 16:50:07] 314 Associations (filtered)...
[INFO] [2021-10-28 16:50:10] 0 metadata added.
[INFO] [2021-10-28 16:50:11] 0 metadata added.
[INFO] [2021-10-28 16:51:42] 7891 Traits (unfiltered)...
[INFO] [2021-10-28 16:53:17] 7891 Traits (filtered)...
[INFO] [2021-10-28 16:53:24] 62 Associations (filtered)...
[INFO] [2021-10-28 16:53:26] 0 metadata added.
[INFO] [2021-10-28 16:53:26] 0 metadata added.
[INFO] [2021-10-28 16:53:59] Average Time: 277.48
[INFO] [2021-10-28 16:53:59] Total Time: 10m26s
[STOP] [2021-10-28 16:53:59] overall_tsv_creation
[INFO] [2021-10-28 16:53:59] Done. Check your files:
[INFO] [2021-10-28 16:53:59] (11311 lines) /app/public/data/si_contributions/publish_bibliographic_citations.tsv
[INFO] [2021-10-28 16:54:00] (12217 lines) /app/public/data/si_contributions/publish_nodes.tsv
[INFO] [2021-10-28 16:54:00] (12217 lines) /app/public/data/si_contributions/publish_scientific_names.tsv
[INFO] [2021-10-28 16:54:00] (11462 lines) /app/public/data/si_contributions/publish_articles.tsv
[INFO] [2021-10-28 16:54:00] (11462 lines) /app/public/data/si_contributions/publish_content_sections.tsv
[INFO] [2021-10-28 16:54:01] (43120 lines) /app/public/data/si_contributions/publish_traits.tsv
[INFO] [2021-10-28 16:54:01] (1 lines) /app/public/data/si_contributions/publish_metadata.tsv
[STOP] [2021-10-28 16:54:01] complete_harvest_instance
[START] [2021-10-28 16:54:01] completed
[STOP] [2021-10-28 16:54:01] completed
[STOP] [2021-10-28 16:54:01] logged process, took 1411.26
Latest Process

28 Oct 16:30 EXITED currently processing 0