Harvest for Harvard Museum of Comparative Zoology Created 08 Jun 14:02

Harvesting Log

[INFO] [2021-06-08 14:02:42] Created harvest instance #4000
[STOP] [2021-06-08 14:02:42] create_harvest_instance
[START] [2021-06-08 14:02:42] fetch_files
[STOP] [2021-06-08 14:02:42] fetch_files
[START] [2021-06-08 14:02:42] validate_each_file
[INFO] [2021-06-08 14:02:42] Looping over 5 formats...
[INFO] [2021-06-08 14:02:42] ...agents (/app/public/data/hvd_museum_compa/agent.tab)
[INFO] [2021-06-08 14:02:42] Valid: /app/public/converted_csv/hvd_museum_compa_agents_4000.csv (29 lines)
[INFO] [2021-06-08 14:02:42] ...nodes (/app/public/data/hvd_museum_compa/taxon.tab)
[INFO] [2021-06-08 14:02:43] Valid: /app/public/converted_csv/hvd_museum_compa_nodes_4000.csv (28808 lines)
[INFO] [2021-06-08 14:02:43] ...media (/app/public/data/hvd_museum_compa/media_resource.tab)
[INFO] [2021-06-08 14:02:56] Valid: /app/public/converted_csv/hvd_museum_compa_media_4000.csv (204028 lines)
[INFO] [2021-06-08 14:02:56] ...occurrences (/app/public/data/hvd_museum_compa/occurrence_specific.tab)
[INFO] [2021-06-08 14:02:57] Valid: /app/public/converted_csv/hvd_museum_compa_occurrences_4000.csv (47607 lines)
[INFO] [2021-06-08 14:02:57] ...measurements (/app/public/data/hvd_museum_compa/measurement_or_fact.tab)
[INFO] [2021-06-08 14:03:09] Valid: /app/public/converted_csv/hvd_museum_compa_measurements_4000.csv (290917 lines)
[STOP] [2021-06-08 14:03:09] validate_each_file
[START] [2021-06-08 14:03:09] convert_to_csv
[INFO] [2021-06-08 14:03:09] Looping over 5 formats...
[INFO] [2021-06-08 14:03:09] ...agents (/app/public/data/hvd_museum_compa/agent.tab)
[CMD] [2021-06-08 14:03:09] /usr/bin/sort /app/public/converted_csv/hvd_museum_compa_agents_4000.csv > /app/public/converted_csv/hvd_museum_compa_agents_4000.csv_sorted
[INFO] [2021-06-08 14:03:09] Converted: /app/public/converted_csv/hvd_museum_compa_agents_4000.csv (29 lines)
[INFO] [2021-06-08 14:03:09] ...nodes (/app/public/data/hvd_museum_compa/taxon.tab)
[CMD] [2021-06-08 14:03:09] /usr/bin/sort /app/public/converted_csv/hvd_museum_compa_nodes_4000.csv > /app/public/converted_csv/hvd_museum_compa_nodes_4000.csv_sorted
[INFO] [2021-06-08 14:03:09] Converted: /app/public/converted_csv/hvd_museum_compa_nodes_4000.csv (28808 lines)
[INFO] [2021-06-08 14:03:09] ...media (/app/public/data/hvd_museum_compa/media_resource.tab)
[CMD] [2021-06-08 14:03:09] /usr/bin/sort /app/public/converted_csv/hvd_museum_compa_media_4000.csv > /app/public/converted_csv/hvd_museum_compa_media_4000.csv_sorted
[INFO] [2021-06-08 14:03:10] Converted: /app/public/converted_csv/hvd_museum_compa_media_4000.csv (204028 lines)
[INFO] [2021-06-08 14:03:10] ...occurrences (/app/public/data/hvd_museum_compa/occurrence_specific.tab)
[CMD] [2021-06-08 14:03:10] /usr/bin/sort /app/public/converted_csv/hvd_museum_compa_occurrences_4000.csv > /app/public/converted_csv/hvd_museum_compa_occurrences_4000.csv_sorted
[INFO] [2021-06-08 14:03:10] Converted: /app/public/converted_csv/hvd_museum_compa_occurrences_4000.csv (47607 lines)
[INFO] [2021-06-08 14:03:10] ...measurements (/app/public/data/hvd_museum_compa/measurement_or_fact.tab)
[CMD] [2021-06-08 14:03:10] /usr/bin/sort /app/public/converted_csv/hvd_museum_compa_measurements_4000.csv > /app/public/converted_csv/hvd_museum_compa_measurements_4000.csv_sorted
[INFO] [2021-06-08 14:03:11] Converted: /app/public/converted_csv/hvd_museum_compa_measurements_4000.csv (290917 lines)
[STOP] [2021-06-08 14:03:11] convert_to_csv
[START] [2021-06-08 14:03:11] calculate_delta
[INFO] [2021-06-08 14:03:11] Looping over 5 formats...
[INFO] [2021-06-08 14:03:11] ...agents (/app/public/data/hvd_museum_compa/agent.tab)
[CMD] [2021-06-08 14:03:11] echo "0a" > /app/public/diff/hvd_museum_compa_agents_4000.diff
[CMD] [2021-06-08 14:03:11] tail -n +1 /app/public/converted_csv/hvd_museum_compa_agents_4000.csv >> /app/public/diff/hvd_museum_compa_agents_4000.diff
[CMD] [2021-06-08 14:03:11] echo "." >> /app/public/diff/hvd_museum_compa_agents_4000.diff
[INFO] [2021-06-08 14:03:12] Created diff: /app/public/diff/hvd_museum_compa_agents_4000.diff (31 lines)
[INFO] [2021-06-08 14:03:12] ...nodes (/app/public/data/hvd_museum_compa/taxon.tab)
[CMD] [2021-06-08 14:03:12] echo "0a" > /app/public/diff/hvd_museum_compa_nodes_4000.diff
[CMD] [2021-06-08 14:03:12] tail -n +1 /app/public/converted_csv/hvd_museum_compa_nodes_4000.csv >> /app/public/diff/hvd_museum_compa_nodes_4000.diff
[CMD] [2021-06-08 14:03:12] echo "." >> /app/public/diff/hvd_museum_compa_nodes_4000.diff
[INFO] [2021-06-08 14:03:12] Created diff: /app/public/diff/hvd_museum_compa_nodes_4000.diff (28810 lines)
[INFO] [2021-06-08 14:03:12] ...media (/app/public/data/hvd_museum_compa/media_resource.tab)
[CMD] [2021-06-08 14:03:12] echo "0a" > /app/public/diff/hvd_museum_compa_media_4000.diff
[CMD] [2021-06-08 14:03:13] tail -n +1 /app/public/converted_csv/hvd_museum_compa_media_4000.csv >> /app/public/diff/hvd_museum_compa_media_4000.diff
[CMD] [2021-06-08 14:03:13] echo "." >> /app/public/diff/hvd_museum_compa_media_4000.diff
[INFO] [2021-06-08 14:03:13] Created diff: /app/public/diff/hvd_museum_compa_media_4000.diff (204030 lines)
[INFO] [2021-06-08 14:03:13] ...occurrences (/app/public/data/hvd_museum_compa/occurrence_specific.tab)
[CMD] [2021-06-08 14:03:13] echo "0a" > /app/public/diff/hvd_museum_compa_occurrences_4000.diff
[CMD] [2021-06-08 14:03:14] tail -n +1 /app/public/converted_csv/hvd_museum_compa_occurrences_4000.csv >> /app/public/diff/hvd_museum_compa_occurrences_4000.diff
[CMD] [2021-06-08 14:03:14] echo "." >> /app/public/diff/hvd_museum_compa_occurrences_4000.diff
[INFO] [2021-06-08 14:03:14] Created diff: /app/public/diff/hvd_museum_compa_occurrences_4000.diff (47609 lines)
[INFO] [2021-06-08 14:03:14] ...measurements (/app/public/data/hvd_museum_compa/measurement_or_fact.tab)
[CMD] [2021-06-08 14:03:14] echo "0a" > /app/public/diff/hvd_museum_compa_measurements_4000.diff
[CMD] [2021-06-08 14:03:14] tail -n +1 /app/public/converted_csv/hvd_museum_compa_measurements_4000.csv >> /app/public/diff/hvd_museum_compa_measurements_4000.diff
[CMD] [2021-06-08 14:03:15] echo "." >> /app/public/diff/hvd_museum_compa_measurements_4000.diff
[INFO] [2021-06-08 14:03:15] Created diff: /app/public/diff/hvd_museum_compa_measurements_4000.diff (290919 lines)
[STOP] [2021-06-08 14:03:15] calculate_delta
[START] [2021-06-08 14:03:15] parse_diff_and_store
[INFO] [2021-06-08 14:03:15] Handling diff: /app/public/diff/hvd_museum_compa_agents_4000.diff (31 lines)
[INFO] [2021-06-08 14:03:15] Loading agents diff file into memory (31 /app/public/diff/hvd_museum_compa_agents_4000.diff lines)...
[INFO] [2021-06-08 14:03:16] Handling diff: /app/public/diff/hvd_museum_compa_nodes_4000.diff (28810 lines)
[INFO] [2021-06-08 14:03:16] Loading nodes diff file into memory (28810 /app/public/diff/hvd_museum_compa_nodes_4000.diff lines)...
[INFO] [2021-06-08 14:03:27] Handling diff: /app/public/diff/hvd_museum_compa_media_4000.diff (204030 lines)
[INFO] [2021-06-08 14:03:28] Loading media diff file into memory (204030 /app/public/diff/hvd_museum_compa_media_4000.diff lines)...
[INFO] [2021-06-08 14:05:08] Handling diff: /app/public/diff/hvd_museum_compa_occurrences_4000.diff (47609 lines)
[INFO] [2021-06-08 14:05:08] Loading occurrences diff file into memory (47609 /app/public/diff/hvd_museum_compa_occurrences_4000.diff lines)...
[INFO] [2021-06-08 14:05:20] Handling diff: /app/public/diff/hvd_museum_compa_measurements_4000.diff (290919 lines)
[INFO] [2021-06-08 14:05:20] Loading measurements diff file into memory (290919 /app/public/diff/hvd_museum_compa_measurements_4000.diff lines)...
[INFO] [2021-06-08 14:07:27] Storing 29 Attributions
[INFO] [2021-06-08 14:07:27] Processing group of 29 in 1 groups of 1000
[INFO] [2021-06-08 14:07:27] Average Time: 0.01
[INFO] [2021-06-08 14:07:27] Total Time: 1s
[INFO] [2021-06-08 14:07:27] Storing 40077 ScientificNames
[INFO] [2021-06-08 14:07:27] Processing group of 40077 in 41 groups of 1000
[INFO] [2021-06-08 14:07:45] Average Time: 0.447
[INFO] [2021-06-08 14:07:45] Total Time: 19s
[INFO] [2021-06-08 14:07:45] last 3 / first 3: 0.68
[INFO] [2021-06-08 14:07:45] Std.Dev: 0.3807886552931954; Max: 1.63
[INFO] [2021-06-08 14:07:45] Storing 40077 Nodes
[INFO] [2021-06-08 14:07:45] Processing group of 40077 in 41 groups of 1000
[INFO] [2021-06-08 14:07:59] Average Time: 0.336
[INFO] [2021-06-08 14:07:59] Total Time: 14s
[INFO] [2021-06-08 14:07:59] last 3 / first 3: 0.67
[INFO] [2021-06-08 14:07:59] Std.Dev: 0.07745966692414834; Max: 0.5
[INFO] [2021-06-08 14:07:59] Storing 204028 ContentAttributions
[INFO] [2021-06-08 14:07:59] Processing group of 204028 in 205 groups of 1000
[INFO] [2021-06-08 14:08:25] Average Time: 0.123
[INFO] [2021-06-08 14:08:25] Total Time: 26s
[INFO] [2021-06-08 14:08:25] last 3 / first 3: 0.42
[INFO] [2021-06-08 14:08:25] Std.Dev: 0.044721359549995794; Max: 0.41
[INFO] [2021-06-08 14:08:25] Storing 204028 Media
[INFO] [2021-06-08 14:08:25] Processing group of 204028 in 205 groups of 1000
[INFO] [2021-06-08 14:09:49] Average Time: 0.405
[INFO] [2021-06-08 14:09:49] Total Time: 1m24s
[INFO] [2021-06-08 14:09:49] last 3 / first 3: 0.7
[INFO] [2021-06-08 14:09:49] Std.Dev: 0.3331666249791536; Max: 2.63
[INFO] [2021-06-08 14:09:49] Storing 47607 Occurrences
[INFO] [2021-06-08 14:09:49] Processing group of 47607 in 48 groups of 1000
[INFO] [2021-06-08 14:09:57] Average Time: 0.149
[INFO] [2021-06-08 14:09:57] Total Time: 8s
[INFO] [2021-06-08 14:09:57] last 3 / first 3: 0.98
[INFO] [2021-06-08 14:09:57] Std.Dev: 0.044721359549995794; Max: 0.3
[INFO] [2021-06-08 14:09:57] Storing 384750 OccurrenceMetadata
[INFO] [2021-06-08 14:09:57] Processing group of 384750 in 385 groups of 1000
[INFO] [2021-06-08 14:11:02] Average Time: 0.167
[INFO] [2021-06-08 14:11:02] Total Time: 1m6s
[INFO] [2021-06-08 14:11:02] last 3 / first 3: 0.72
[INFO] [2021-06-08 14:11:02] Std.Dev: 0.18708286933869708; Max: 1.48
[INFO] [2021-06-08 14:11:02] Storing 50935 Traits
[INFO] [2021-06-08 14:11:02] Processing group of 50935 in 51 groups of 1000
[INFO] [2021-06-08 14:11:20] Average Time: 0.335
[INFO] [2021-06-08 14:11:20] Total Time: 18s
[INFO] [2021-06-08 14:11:20] last 3 / first 3: 0.82
[INFO] [2021-06-08 14:11:20] Std.Dev: 0.06324555320336758; Max: 0.56
[INFO] [2021-06-08 14:11:20] Storing 101870 MetaTraits
[INFO] [2021-06-08 14:11:20] Processing group of 101870 in 102 groups of 1000
[INFO] [2021-06-08 14:11:33] Average Time: 0.129
[INFO] [2021-06-08 14:11:33] Total Time: 14s
[INFO] [2021-06-08 14:11:33] last 3 / first 3: 1.3
[INFO] [2021-06-08 14:11:33] Std.Dev: 0.03162277660168379; Max: 0.28
[STOP] [2021-06-08 14:11:33] parse_diff_and_store
[START] [2021-06-08 14:11:33] resolve_keys
[INFO] [2021-06-08 14:12:38] Occurrences to nodes (through scientific_names)...
[INFO] [2021-06-08 14:12:55] traits to occurrences...
[INFO] [2021-06-08 14:12:57] traits to nodes (through occurrences)...
[INFO] [2021-06-08 14:12:58] Traits to sex term...
[INFO] [2021-06-08 14:12:58] Traits to lifestage term...
[INFO] [2021-06-08 14:12:59] MetaTraits to traits...
[INFO] [2021-06-08 14:13:01] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-06-08 14:13:02] Assocs to occurrences...
[INFO] [2021-06-08 14:13:02] Assocs to nodes...
[INFO] [2021-06-08 14:13:02] Assoc to sex term...
[INFO] [2021-06-08 14:13:02] Assoc to lifestage term...
[INFO] [2021-06-08 14:13:02] MetaAssoc to assocs...
[STOP] [2021-06-08 14:13:09] resolve_keys
[START] [2021-06-08 14:13:09] hold_for_later_1
[STOP] [2021-06-08 14:13:09] hold_for_later_1
[START] [2021-06-08 14:13:09] hold_for_later_2
[STOP] [2021-06-08 14:13:09] hold_for_later_2
[START] [2021-06-08 14:13:09] resolve_missing_parents
[STOP] [2021-06-08 14:13:11] resolve_missing_parents
[START] [2021-06-08 14:13:11] rebuild_nodes
[START] [2021-06-08 14:13:11] Flattener#flatten
[START] [2021-06-08 14:13:11] Flattener#study_resource
[START] [2021-06-08 14:13:11] Flattener#build_ancestry
[STOP] [2021-06-08 14:13:22] Flattener#build_ancestry
[INFO] [2021-06-08 14:13:22] 40077 ancestry keys
[START] [2021-06-08 14:13:22] build_node_ancestors
[INFO] [2021-06-08 14:13:22] old ancestors deleted.
[STOP] [2021-06-08 14:13:52] build_node_ancestors
[START] [2021-06-08 14:13:54] Flattener#propagate_ancestor_ids
[STOP] [2021-06-08 14:13:59] Flattener#propagate_ancestor_ids
[STOP] [2021-06-08 14:13:59] Flattener#flatten
[STOP] [2021-06-08 14:13:59] rebuild_nodes
[START] [2021-06-08 14:13:59] resolve_missing_media_owners
[STOP] [2021-06-08 14:13:59] resolve_missing_media_owners
[START] [2021-06-08 14:13:59] sanitize_media_verbatims
[STOP] [2021-06-08 14:13:59] sanitize_media_verbatims
[START] [2021-06-08 14:13:59] queue_downloads
[STOP] [2021-06-08 14:14:00] queue_downloads
[START] [2021-06-08 14:14:00] parse_names
[WARN] [2021-06-08 14:14:00] I see 40077 names which still need to be parsed.
[ERR] [2021-06-08 14:14:04][hdls] download_and_prep FAILED for Medium.find(12910538): 404 Not found
[WARN] [2021-06-08 14:14:31] I see 1097 names which still need to be parsed.
[WARN] [2021-06-08 14:14:33] I see 52 names which still need to be parsed.
[WARN] [2021-06-08 14:14:34] I see 3 names which still need to be parsed.
[STOP] [2021-06-08 14:14:36] parse_names
[START] [2021-06-08 14:14:36] denormalize_canonical_names_to_nodes
[STOP] [2021-06-08 14:14:36] denormalize_canonical_names_to_nodes
[START] [2021-06-08 14:14:36] match_nodes
[START] [2021-06-08 14:14:36] map_all_nodes_to_pages
[ERR] [2021-06-08 14:17:22][hdls] download_and_prep FAILED for Medium.find(12911938): 404 Not found
[ERR] [2021-06-08 14:17:22][hdls] download_and_prep FAILED for Medium.find(12911939): 404 Not found
[ERR] [2021-06-08 14:17:22][hdls] download_and_prep FAILED for Medium.find(12911940): 404 Not found
[ERR] [2021-06-08 14:27:31][hdls] download_and_prep FAILED for Medium.find(12916366): 404 Not found
[ERR] [2021-06-08 14:27:32][hdls] download_and_prep FAILED for Medium.find(12916368): 404 Not found
[STOP] [2021-06-08 14:57:43] map_all_nodes_to_pages
[INFO] [2021-06-08 14:57:43] 8440 Unmatched nodes (of 40077)! That's too many to output. Full list in /app/public/data/hvd_museum_compa/unmatched_nodes.txt ; First 10: Canonical: Pomacea maculata haustrum; Node#95828493; ResourceID: 52935; Canonical: Pomacea maculata australis; Node#95830180; ResourceID: 69899; Canonical: Pomacea urceus nobilis; Node#95830787; ResourceID: 86156; Canonical: Pomacea canaliculata immersa; Node#95830825; ResourceID: 87364; Canonical: Triforis; Node#95806415; ResourceID: Triforis; Canonical: Triforis torticulus; Node#95817492; ResourceID: 41122; Canonical: Aclididae; Node#95825171; ResourceID: Aclididae; Canonical: Cioniscus; Node#95825172; ResourceID: Cioniscus; Canonical: Cyclophorus chinense bifrons; Node#95830668; ResourceID: 83338; Canonical: Cyclotus approximans; Node#95830652; ResourceID: 82989
[START] [2021-06-08 14:57:43] update_nodes
[STOP] [2021-06-08 14:58:04] update_nodes
[STOP] [2021-06-08 14:58:04] match_nodes
[START] [2021-06-08 14:58:04] reindex_search
[STOP] [2021-06-08 14:58:49] reindex_search
[START] [2021-06-08 14:58:49] normalize_units
[STOP] [2021-06-08 14:58:49] normalize_units
[START] [2021-06-08 14:58:49] calculate_statistics
[2021-06-08 14:58:51] (NEAR) DUPLICATE TRAITS FOUND! There are only 47607 (of 50935 total) unique traits.
[2021-06-08 14:59:28] (Near) duplicate trait pairs (up to 100):
[2021-06-08 14:59:28] (resource_pk: 000226d1e9872ad1413322e650e8cec5_201, id: 219173055), (resource_pk: 93a5dff6fe88b37bf392f1721bf8549c_201, id: 219202321)
[2021-06-08 14:59:28] (resource_pk: 000226d1e9872ad1413322e650e8cec5_201, id: 219173055), (resource_pk: ad5811466c001a528628cba5d20454f6_201, id: 219207483)
[2021-06-08 14:59:28] (resource_pk: 0004ff6f7fc6e4b9b5a08212b4d24814_201, id: 219173058), (resource_pk: 96bcb119444f74fd9a16a7e5aac37cc8_201, id: 219202977)
[2021-06-08 14:59:28] (resource_pk: 000bd860fe283ae73a6ad353d6514eb2_201, id: 219173063), (resource_pk: e7367487598689dff4d8a676affd8e81_201, id: 219219067)
[2021-06-08 14:59:28] (resource_pk: 0014f5aff5463ae9471ecfac5549b049_201, id: 219173070), (resource_pk: d785d515de6c03f7710f85686a836cc4_201, id: 219215889)
[2021-06-08 14:59:28] (resource_pk: 002a9b58297cf749586f3a8bdf210777_201, id: 219173079), (resource_pk: 15f1e301acdbd790087245f18a703457_201, id: 219177323)
[2021-06-08 14:59:28] (resource_pk: 00334599812b952fc713e26d15a34f4d_201, id: 219173085), (resource_pk: e7df6dfe258e08ee25d4123215a3afd4_201, id: 219219202)
[2021-06-08 14:59:28] (resource_pk: 0034149da637913bb3c03e9791b180eb_201, id: 219173086), (resource_pk: 5ac74968fcae36de9234063490d0197a_201, id: 219190987)
[2021-06-08 14:59:28] (resource_pk: 0039de1eb6da1104459760b5cecd7139_201, id: 219173093), (resource_pk: 280f133edf9a3a8642db6023e0e0fbc4_201, id: 219180917)
[2021-06-08 14:59:28] (resource_pk: 00433ebaf989edd0a4ce3d2af636f80b_201, id: 219173098), (resource_pk: 510adf00072222ccadb8b32585afc12b_201, id: 219189072)
[2021-06-08 14:59:28] (resource_pk: 004a6c62559c5fd374757dbb1dbf55eb_201, id: 219173104), (resource_pk: 0fd0ed08c291ac53c7fbe5b78964d246_201, id: 219176147)
[2021-06-08 14:59:28] (resource_pk: 004a6c62559c5fd374757dbb1dbf55eb_201, id: 219173104), (resource_pk: 7d99890dd6fba333249a6eca17d32625_201, id: 219198055)
[2021-06-08 14:59:28] (resource_pk: 004a6c62559c5fd374757dbb1dbf55eb_201, id: 219173104), (resource_pk: e7afd44fe91c0b873e7766090df7ffb9_201, id: 219219162)
[2021-06-08 14:59:28] (resource_pk: 005c7efa7c5b45af3903d3dff216e09e_201, id: 219173124), (resource_pk: 2bf9adab25f5628344f2446a485d7913_201, id: 219181672)
[2021-06-08 14:59:28] (resource_pk: 005d1601741be4f61bc24cd64ee5f9d4_201, id: 219173125), (resource_pk: 19faf047e2ff5112b6da02fd40d87aeb_201, id: 219178123)
[2021-06-08 14:59:28] (resource_pk: 005d67f914bf6432f1bb8d5a9e6423f2_201, id: 219173126), (resource_pk: 5d96a6de4f6468fc29ae64faeb4c8545_201, id: 219191574)
[2021-06-08 14:59:28] (resource_pk: 00628d7b29b38a037eaa7b8aa3777a43_201, id: 219173129), (resource_pk: 77080e95097883acd40c97a65addf8d7_201, id: 219196741)
[2021-06-08 14:59:28] (resource_pk: 006325eb6d3d60934e32e4040d6f26bc_201, id: 219173131), (resource_pk: 9592bffdea62c2df72cdd4fa25ae29d7_201, id: 219202732)
[2021-06-08 14:59:28] (resource_pk: 0065978fca717ca0d44905600b0f493f_201, id: 219173134), (resource_pk: 39c325f999dda44cbf862d814f66f018_201, id: 219184429)
[2021-06-08 14:59:28] (resource_pk: 006a6b0b62180af82ea3586b39150523_201, id: 219173138), (resource_pk: 4fba1cd594cbacafe36f715d13dc5895_201, id: 219188843)
[2021-06-08 14:59:28] (resource_pk: 00747e127da64744e8a8b095b71fecc1_201, id: 219173149), (resource_pk: 257d13057c20da0c8c4f847eb81f64ba_201, id: 219180391)
[2021-06-08 14:59:28] (resource_pk: 00747e127da64744e8a8b095b71fecc1_201, id: 219173149), (resource_pk: 331474ac08988e398c33ee00519730e1_201, id: 219183080)
[2021-06-08 14:59:28] (resource_pk: 0074b1e87788b69cb29a10ea706741d0_201, id: 219173150), (resource_pk: d63e4d74ee3b241eeb56c79c77f92e23_201, id: 219215665)
[2021-06-08 14:59:28] (resource_pk: 007ae6cbfb77ebf76b6006bcc6954a56_201, id: 219173158), (resource_pk: c144f8461003fecbb75eb03efe0f2bc0_201, id: 219211494)
[2021-06-08 14:59:28] (resource_pk: 00817325a326dd637fd50bc05f3f043b_201, id: 219173161), (resource_pk: ee38dfc7372ed6dc4fdafe069f873187_201, id: 219220443)
[2021-06-08 14:59:28] (resource_pk: 008470202312bdaf0c7b418b99159cef_201, id: 219173164), (resource_pk: 9409fbd72c9bbd83fdc23740145d5525_201, id: 219202399)
[2021-06-08 14:59:28] (resource_pk: 008a52d126f4e4fbd77b5478362c52f4_201, id: 219173169), (resource_pk: a477c909748588469c5fb9e85e106238_201, id: 219205682)
[2021-06-08 14:59:28] (resource_pk: 008a75efe8d4a3d92c049bcd9af3ba88_201, id: 219173170), (resource_pk: f44b4be52ebb66fea9019dc44a7d1a5a_201, id: 219221622)
[2021-06-08 14:59:28] (resource_pk: 008b369bd4eff64dabbdfaa05b64e2cb_201, id: 219173171), (resource_pk: 11923891d509eafebdc4280cda49f1b6_201, id: 219176476)
[2021-06-08 14:59:28] (resource_pk: 008b369bd4eff64dabbdfaa05b64e2cb_201, id: 219173171), (resource_pk: 11fc92c2fd81e868ac86c46a7ca3cb96_201, id: 219176564)
[2021-06-08 14:59:28] (resource_pk: 008b369bd4eff64dabbdfaa05b64e2cb_201, id: 219173171), (resource_pk: e3e1ec851e54323c621cba3977e0e26e_201, id: 219218387)
[2021-06-08 14:59:28] (resource_pk: 008c15c652074839d74c734b4e3c5e25_201, id: 219173172), (resource_pk: 595c70f1903580efc10269f20869c1a2_201, id: 219190718)
[2021-06-08 14:59:28] (resource_pk: 008e7882f4a605c534b492a0383c3f4a_201, id: 219173175), (resource_pk: 17c38b130428eef977404be97b280b9a_201, id: 219177693)
[2021-06-08 14:59:28] (resource_pk: 008e7882f4a605c534b492a0383c3f4a_201, id: 219173175), (resource_pk: 4737e48c2100bda87e1a00539ecfb4f2_201, id: 219187172)
[2021-06-08 14:59:28] (resource_pk: 008e7882f4a605c534b492a0383c3f4a_201, id: 219173175), (resource_pk: 54aa7cb6d53a13806b808a9ee609db49_201, id: 219189793)
[2021-06-08 14:59:28] (resource_pk: 009142a54fcbf56895828ec3c43013a3_201, id: 219173178), (resource_pk: b9b46f2c993765f1be1750755f15117c_201, id: 219209980)
[2021-06-08 14:59:28] (resource_pk: 009263877c7563dd9b6be63a26706ac1_201, id: 219173181), (resource_pk: 108099c89932926cfc5d06f93d8d4de2_201, id: 219176270)
[2021-06-08 14:59:28] (resource_pk: 009965e58b2014067e048861cb204abd_201, id: 219173189), (resource_pk: b9423903d913f8be86b42fdfaf4b64c3_201, id: 219209898)
[2021-06-08 14:59:28] (resource_pk: 00a13687ef566cc74e9c4a6010fb522d_201, id: 219173193), (resource_pk: 6772022f9b51171dfb8f11487732d134_201, id: 219193573)
[2021-06-08 14:59:28] (resource_pk: 00a13687ef566cc74e9c4a6010fb522d_201, id: 219173193), (resource_pk: 94dd4950faea757863a325de7f3306f8_201, id: 219202581)
[2021-06-08 14:59:28] (resource_pk: 00a13687ef566cc74e9c4a6010fb522d_201, id: 219173193), (resource_pk: d6ca48cb471f094cfe18b254b077dc46_201, id: 219215764)
[2021-06-08 14:59:28] (resource_pk: 00bae3b549aa8490923cb33faf22ddd3_201, id: 219173208), (resource_pk: 83fff2ff6983c2c279861c37c451da36_201, id: 219199288)
[2021-06-08 14:59:28] (resource_pk: 00bae3b549aa8490923cb33faf22ddd3_201, id: 219173208), (resource_pk: fbbe5133229b29335061c417a6d9fd34_201, id: 219223134)
[2021-06-08 14:59:28] (resource_pk: 00d581194d4ca08a83f841d3623feabe_201, id: 219173227), (resource_pk: 50bfcb34a4b475591f44f86a56d2a8d4_201, id: 219189010)
[2021-06-08 14:59:28] (resource_pk: 00ec4ad4cb8827740bfcf394e9ce81ed_201, id: 219173247), (resource_pk: 6f4313ad2506146c86cf7fc9bd2abee3_201, id: 219195214)
[2021-06-08 14:59:28] (resource_pk: 00eeee151872654071440590cace177c_201, id: 219173249), (resource_pk: 062ed1e1a3c3968579f5b378576620e4_201, id: 219174254)
[2021-06-08 14:59:28] (resource_pk: 00f720c8765e617811bd04ebcf9f7f75_201, id: 219173256), (resource_pk: c545ae2579ebf1139c39f40e103a5c64_201, id: 219212305)
[2021-06-08 14:59:28] (resource_pk: 00f720c8765e617811bd04ebcf9f7f75_201, id: 219173256), (resource_pk: f4b2109c72e782522f9deed44d3bf5d5_201, id: 219221701)
[2021-06-08 14:59:28] (resource_pk: 00faa156046b0ec38b7a53bc7adafd14_201, id: 219173258), (resource_pk: f72707f2225d27854b817b602000cc91_201, id: 219222178)
[2021-06-08 14:59:28] (resource_pk: 010105a34d0f613f795e21b0da69878b_201, id: 219173261), (resource_pk: 1eeb482bfd6392dc1f88976a837f00b4_201, id: 219179132)
[2021-06-08 14:59:28] (resource_pk: 010212e0567f009e2ba4431825f1f8da_201, id: 219173264), (resource_pk: a38a2b3f6ebf624442df72f4240fc356_201, id: 219205503)
[2021-06-08 14:59:28] (resource_pk: 01023a43d3383e5f6f424beb73165532_201, id: 219173265), (resource_pk: 542e81aa28f534d1d5e98674218e9291_201, id: 219189687)
[2021-06-08 14:59:28] (resource_pk: 010d644973e66303171fb339329261ca_201, id: 219173273), (resource_pk: 730388d803d1b663b690e5cf94df1dec_201, id: 219195964)
[2021-06-08 14:59:28] (resource_pk: 011255f27d3ada9444d7b040f2f8715d_201, id: 219173278), (resource_pk: 6f49f5935291e0173564eba691932460_201, id: 219195216)
[2021-06-08 14:59:28] (resource_pk: 011255f27d3ada9444d7b040f2f8715d_201, id: 219173278), (resource_pk: ba2569ba90376a343ecd7d18b1531386_201, id: 219210068)
[2021-06-08 14:59:28] (resource_pk: 01225b3ff38ae3c4613209d5ae47d9c3_201, id: 219173287), (resource_pk: c0b2338338ceee35b787f5aa87a8f9e0_201, id: 219211356)
[2021-06-08 14:59:28] (resource_pk: 01241d4863117f01b8f34b1c411943f4_201, id: 219173294), (resource_pk: cb4c24370bb0ae861ed071b70aa619ef_201, id: 219213503)
[2021-06-08 14:59:28] (resource_pk: 0127552e2ebed21dcd3b3a10f6c72f3e_201, id: 219173297), (resource_pk: 4a5d1fb2285c8c49db74793059bd959f_201, id: 219187781)
[2021-06-08 14:59:28] (resource_pk: 0127deacc7b7401f4285b09ac271c660_201, id: 219173299), (resource_pk: 180790714bf66275610f2e8a73a889de_201, id: 219177758)
[2021-06-08 14:59:28] (resource_pk: 01290bdf8346ae1ff5e174e26846ed00_201, id: 219173300), (resource_pk: ed00fe56453f35f6849df0092d0f9d5b_201, id: 219220192)
[2021-06-08 14:59:28] (resource_pk: 013cccf4b376ffbb29f4adcb3091b16c_201, id: 219173311), (resource_pk: 21e16fa3171787c867ffebba7b78bb34_201, id: 219179699)
[2021-06-08 14:59:28] (resource_pk: 013cccf4b376ffbb29f4adcb3091b16c_201, id: 219173311), (resource_pk: 240e6f0c90b2e5f0f17cc55bd51b4ffb_201, id: 219180132)
[2021-06-08 14:59:28] (resource_pk: 013cccf4b376ffbb29f4adcb3091b16c_201, id: 219173311), (resource_pk: 44cfc95c25cb40a5ef1ddbe247b720a2_201, id: 219186712)
[2021-06-08 14:59:28] (resource_pk: 013cccf4b376ffbb29f4adcb3091b16c_201, id: 219173311), (resource_pk: 6bb087a55e9c875b0e12fc76c6657fa0_201, id: 219194417)
[2021-06-08 14:59:28] (resource_pk: 013cccf4b376ffbb29f4adcb3091b16c_201, id: 219173311), (resource_pk: e2ca87f8e83211b04013ffb0bbefd330_201, id: 219218174)
[2021-06-08 14:59:28] (resource_pk: 016a7510c56c9ec28c08201c6f5d22bc_201, id: 219173352), (resource_pk: 43f798504454ad7e1da1c09cffd6ce20_201, id: 219186525)
[2021-06-08 14:59:28] (resource_pk: 016cde56fb0975c167ed43a5f00d679a_201, id: 219173354), (resource_pk: 7e64776b68f8c3242d5b7007dd4cdb33_201, id: 219198214)
[2021-06-08 14:59:28] (resource_pk: 017163a32acaa04013e802b7bfeda201_201, id: 219173357), (resource_pk: 10e14deb96367303e3417e4ca4f172fd_201, id: 219176363)
[2021-06-08 14:59:28] (resource_pk: 01800a6b87cf6d3ab385b63ca1dd3aa9_201, id: 219173369), (resource_pk: b47a225f2b488086a53c63bac74a795f_201, id: 219208899)
[2021-06-08 14:59:28] (resource_pk: 0185a0814dab583b83e12c188fb4d589_201, id: 219173375), (resource_pk: 7b51d9ca1f1dbdbd2cef55fea9c925fb_201, id: 219197590)
[2021-06-08 14:59:28] (resource_pk: 018f49df2e2ec1d4d5c2b3338efae301_201, id: 219173379), (resource_pk: e890ba2c31deb17c0b20854de1eda480_201, id: 219219330)
[2021-06-08 14:59:28] (resource_pk: 01914abab2cf4183919b50183b87ed39_201, id: 219173380), (resource_pk: 999baabed608cb774a8843778e19466d_201, id: 219203544)
[2021-06-08 14:59:28] (resource_pk: 0193eab3c8478dfb4141c4ff7ae6de7f_201, id: 219173381), (resource_pk: 3bc78158dca09000fda59ce9d6b31b3c_201, id: 219184830)
[2021-06-08 14:59:28] (resource_pk: 0193eab3c8478dfb4141c4ff7ae6de7f_201, id: 219173381), (resource_pk: fbb705204eb721ec5c60632ca14f09c3_201, id: 219223131)
[2021-06-08 14:59:28] (resource_pk: 0199abcbed7ce688b387d6746f70dcdc_201, id: 219173383), (resource_pk: e2972147803d5eeba11d830c7f790ee5_201, id: 219218135)
[2021-06-08 14:59:28] (resource_pk: 019fd9a8fab0579890a6c002c915a9ba_201, id: 219173393), (resource_pk: f9f35f83f02fd14600b51c412da74459_201, id: 219222758)
[2021-06-08 14:59:28] (resource_pk: 01b23e9f3b802a71beb7bf190439f400_201, id: 219173404), (resource_pk: 43158d4fe59ea5824b2f07a4e9667e5f_201, id: 219186327)
[2021-06-08 14:59:28] (resource_pk: 01b23e9f3b802a71beb7bf190439f400_201, id: 219173404), (resource_pk: 92ca83bf54d86223534e4a497b357967_201, id: 219202138)
[2021-06-08 14:59:28] (resource_pk: 01b2fa235ce9cdc86e83a156cdf817c0_201, id: 219173405), (resource_pk: b857d0c3a66beebe1b1a98f2922695c3_201, id: 219209707)
[2021-06-08 14:59:28] (resource_pk: 01c44479f30452d4981cc4e55037744b_201, id: 219173419), (resource_pk: e4892f5c53a2153f4b4eefc271963972_201, id: 219218516)
[2021-06-08 14:59:28] (resource_pk: 01d7eb7542daf9d906a6bf8aef45a13b_201, id: 219173433), (resource_pk: 48e569756e8819c0b2c939a6181e2ef5_201, id: 219187508)
[2021-06-08 14:59:28] (resource_pk: 01dcef40efdd7b7e3f8ac0a6e012e2d0_201, id: 219173439), (resource_pk: 1b2e908b2ba77c38d9f63ae2b0fcabf5_201, id: 219178369)
[2021-06-08 14:59:28] (resource_pk: 01eb7a171be6f4aea477ecf90f791836_201, id: 219173445), (resource_pk: 257c3db96283721daaea117601e64412_201, id: 219180390)
[2021-06-08 14:59:28] (resource_pk: 01ecee460152457d2d05416697cd0571_201, id: 219173447), (resource_pk: 073c3dc5c0fdd296d37ab929c49806d4_201, id: 219174450)
[2021-06-08 14:59:28] (resource_pk: 0206ba4bb07907730c1d2235a28827ab_201, id: 219173459), (resource_pk: 28c5ee7233c3e1ca919b19ff3d57e710_201, id: 219181052)
[2021-06-08 14:59:28] (resource_pk: 0206ba4bb07907730c1d2235a28827ab_201, id: 219173459), (resource_pk: 5c80cbf96cf2825ce4c71d12d494ab0e_201, id: 219191353)
[2021-06-08 14:59:28] (resource_pk: 0206ba4bb07907730c1d2235a28827ab_201, id: 219173459), (resource_pk: d4d2f4f686114d6ba818b3cafb7decaf_201, id: 219215389)
[2021-06-08 14:59:28] (resource_pk: 0206bc39a62de938dc22c8f38c713528_201, id: 219173460), (resource_pk: 7015277862516bee973b907b5bbf64e1_201, id: 219195390)
[2021-06-08 14:59:28] (resource_pk: 020effe5779540b22ae41d88471332dc_201, id: 219173467), (resource_pk: 187482b849319186cb7e85762a742954_201, id: 219177850)
[2021-06-08 14:59:28] (resource_pk: 020effe5779540b22ae41d88471332dc_201, id: 219173467), (resource_pk: 3fdb1eaa8d0ec490e7d5a936a43380eb_201, id: 219185638)
[2021-06-08 14:59:28] (resource_pk: 022134c1a22f4f7212fb0a858273c3e9_201, id: 219173476), (resource_pk: 9d7bee259068094888425ae13d138236_201, id: 219204318)
[2021-06-08 14:59:28] (resource_pk: 022b1022c4a276e924571859eade9888_201, id: 219173483), (resource_pk: 78e1d50cecd3b548f30f5619f2518fd3_201, id: 219197142)
[2021-06-08 14:59:28] (resource_pk: 0233483649d707213509b167ff1cb3b6_201, id: 219173490), (resource_pk: 53437620e1748fbcee2c4ff15eac6aa4_201, id: 219189508)
[2021-06-08 14:59:28] (resource_pk: 0235ac0dcdcde1025293f8674ad85fe1_201, id: 219173494), (resource_pk: fbeec27e69a4d15d4fee66c568f47054_201, id: 219223171)
[2021-06-08 14:59:28] (resource_pk: 02365aefcf0c4ce5b9e7a44331db7ba5_201, id: 219173495), (resource_pk: b3d41c89939a90dea1b4b710ff57e1d9_201, id: 219208767)
[2021-06-08 14:59:28] (resource_pk: 024012ea21ca07b1f54c0a4fdd8ed235_201, id: 219173503), (resource_pk: 6345c9a022271d12f3855f38fdc66fac_201, id: 219192704)
[2021-06-08 14:59:28] (resource_pk: 02419e51b7448b6537a988c432471aa4_201, id: 219173505), (resource_pk: dc346b9cbd490009a81938b098170054_201, id: 219216827)
[2021-06-08 14:59:28] (resource_pk: 0245c3ba265142f4f9a321e111e1c6fe_201, id: 219173512), (resource_pk: ec6890e99213d664ad5f811a5f29ee27_201, id: 219220076)
[2021-06-08 14:59:28] (resource_pk: 02489f0ed5fcb70a3251dbcd39a4d3db_201, id: 219173514), (resource_pk: 14c2cf3aa6c7b842f67b33f65cb55c0e_201, id: 219177087)
[2021-06-08 14:59:28] (resource_pk: 024df0d0c150f36dbf21d2ab752468bf_201, id: 219173519), (resource_pk: 41acf0c7628f9388e085d328c74962bb_201, id: 219186054)
[STOP] [2021-06-08 14:59:29] calculate_statistics
[START] [2021-06-08 14:59:29] complete_harvest_instance
[START] [2021-06-08 14:59:29] overall_tsv_creation
[INFO] [2021-06-08 14:59:29] Processing group of 40077 in 5 batches of 10000
[INFO] [2021-06-08 15:02:57] 12302 Traits (unfiltered)...
[INFO] [2021-06-08 15:05:00] 12302 Traits (filtered)...
[INFO] [2021-06-08 15:05:02] 0 Associations (filtered)...
[INFO] [2021-06-08 15:05:17] 112114 metadata added.
[INFO] [2021-06-08 15:05:17] 0 metadata added.
[ERR] [2021-06-08 15:05:42][hdls] download_and_prep FAILED for Medium.find(12930970): 404 Not found
[INFO] [2021-06-08 15:08:36] 13686 Traits (unfiltered)...
[INFO] [2021-06-08 15:10:51] 13686 Traits (filtered)...
[INFO] [2021-06-08 15:10:54] 0 Associations (filtered)...
[INFO] [2021-06-08 15:11:11] 126100 metadata added.
[INFO] [2021-06-08 15:11:11] 0 metadata added.
[ERR] [2021-06-08 15:12:27][hdls] download_and_prep FAILED for Medium.find(12933882): 404 Not found
[ERR] [2021-06-08 15:12:30][hdls] download_and_prep FAILED for Medium.find(12933922): 404 Not found
[ERR] [2021-06-08 15:12:30][hdls] download_and_prep FAILED for Medium.find(12933923): 404 Not found
[ERR] [2021-06-08 15:12:31][hdls] download_and_prep FAILED for Medium.find(12933924): 404 Not found
[ERR] [2021-06-08 15:12:31][hdls] download_and_prep FAILED for Medium.find(12933925): 404 Not found
[ERR] [2021-06-08 15:12:31][hdls] download_and_prep FAILED for Medium.find(12933926): 404 Not found
[ERR] [2021-06-08 15:12:31][hdls] download_and_prep FAILED for Medium.find(12933927): 404 Not found
[ERR] [2021-06-08 15:12:31][hdls] download_and_prep FAILED for Medium.find(12933928): 404 Not found
[ERR] [2021-06-08 15:12:31][hdls] download_and_prep FAILED for Medium.find(12933929): 404 Not found
[ERR] [2021-06-08 15:12:31][hdls] download_and_prep FAILED for Medium.find(12933930): 404 Not found
[ERR] [2021-06-08 15:12:31][hdls] download_and_prep FAILED for Medium.find(12933931): 404 Not found
[ERR] [2021-06-08 15:12:31][hdls] download_and_prep FAILED for Medium.find(12933932): 404 Not found
[ERR] [2021-06-08 15:12:32][hdls] download_and_prep FAILED for Medium.find(12933933): 404 Not found
[ERR] [2021-06-08 15:12:32][hdls] download_and_prep FAILED for Medium.find(12933934): 404 Not found
[ERR] [2021-06-08 15:12:32][hdls] download_and_prep FAILED for Medium.find(12933935): 404 Not found
[ERR] [2021-06-08 15:12:32][hdls] download_and_prep FAILED for Medium.find(12933936): 404 Not found
[ERR] [2021-06-08 15:12:32][hdls] download_and_prep FAILED for Medium.find(12933937): 404 Not found
[ERR] [2021-06-08 15:12:32][hdls] download_and_prep FAILED for Medium.find(12933938): 404 Not found
[ERR] [2021-06-08 15:12:32][hdls] download_and_prep FAILED for Medium.find(12933939): 404 Not found
[ERR] [2021-06-08 15:12:32][hdls] download_and_prep FAILED for Medium.find(12933940): 404 Not found
[ERR] [2021-06-08 15:12:32][hdls] download_and_prep FAILED for Medium.find(12933941): 404 Not found
[ERR] [2021-06-08 15:12:32][hdls] download_and_prep FAILED for Medium.find(12933942): 404 Not found
[ERR] [2021-06-08 15:12:32][hdls] download_and_prep FAILED for Medium.find(12933943): 404 Not found
[ERR] [2021-06-08 15:12:32][hdls] download_and_prep FAILED for Medium.find(12933944): 404 Not found
[ERR] [2021-06-08 15:12:33][hdls] download_and_prep FAILED for Medium.find(12933945): 404 Not found
[ERR] [2021-06-08 15:12:33][hdls] download_and_prep FAILED for Medium.find(12933946): 404 Not found
[ERR] [2021-06-08 15:12:33][hdls] download_and_prep FAILED for Medium.find(12933947): 404 Not found
[ERR] [2021-06-08 15:14:13][hdls] download_and_prep FAILED for Medium.find(12934651): 404 Not found
[ERR] [2021-06-08 15:14:13][hdls] download_and_prep FAILED for Medium.find(12934650): 404 Not found
[ERR] [2021-06-08 15:14:13][hdls] download_and_prep FAILED for Medium.find(12934652): 404 Not found
[INFO] [2021-06-08 15:14:41] 16438 Traits (unfiltered)...
[ERR] [2021-06-08 15:16:25][hdls] download_and_prep FAILED for Medium.find(12935606): 404 Not found
[ERR] [2021-06-08 15:16:26][hdls] download_and_prep FAILED for Medium.find(12935634): 404 Not found
[ERR] [2021-06-08 15:16:26][hdls] download_and_prep FAILED for Medium.find(12935635): 404 Not found
[ERR] [2021-06-08 15:16:27][hdls] download_and_prep FAILED for Medium.find(12935636): 404 Not found
[ERR] [2021-06-08 15:16:27][hdls] download_and_prep FAILED for Medium.find(12935637): 404 Not found
[ERR] [2021-06-08 15:16:27][hdls] download_and_prep FAILED for Medium.find(12935638): 404 Not found
[ERR] [2021-06-08 15:16:27][hdls] download_and_prep FAILED for Medium.find(12935639): 404 Not found
[ERR] [2021-06-08 15:16:27][hdls] download_and_prep FAILED for Medium.find(12935616): 404 Not found
[ERR] [2021-06-08 15:16:28][hdls] download_and_prep FAILED for Medium.find(12935625): 404 Not found
[ERR] [2021-06-08 15:16:28][hdls] download_and_prep FAILED for Medium.find(12935626): 404 Not found
[ERR] [2021-06-08 15:16:28][hdls] download_and_prep FAILED for Medium.find(12935627): 404 Not found
[ERR] [2021-06-08 15:16:28][hdls] download_and_prep FAILED for Medium.find(12935628): 404 Not found
[ERR] [2021-06-08 15:16:28][hdls] download_and_prep FAILED for Medium.find(12935629): 404 Not found
[ERR] [2021-06-08 15:16:28][hdls] download_and_prep FAILED for Medium.find(12935630): 404 Not found
[ERR] [2021-06-08 15:16:29][hdls] download_and_prep FAILED for Medium.find(12935632): 404 Not found
[ERR] [2021-06-08 15:16:29][hdls] download_and_prep FAILED for Medium.find(12935631): 404 Not found
[ERR] [2021-06-08 15:16:29][hdls] download_and_prep FAILED for Medium.find(12935633): 404 Not found
[ERR] [2021-06-08 15:16:29][hdls] download_and_prep FAILED for Medium.find(12935640): 404 Not found
[ERR] [2021-06-08 15:16:29][hdls] download_and_prep FAILED for Medium.find(12935641): 404 Not found
[ERR] [2021-06-08 15:16:29][hdls] download_and_prep FAILED for Medium.find(12935642): 404 Not found
[ERR] [2021-06-08 15:16:29][hdls] download_and_prep FAILED for Medium.find(12935643): 404 Not found
[ERR] [2021-06-08 15:16:29][hdls] download_and_prep FAILED for Medium.find(12935644): 404 Not found
[ERR] [2021-06-08 15:16:29][hdls] download_and_prep FAILED for Medium.find(12935645): 404 Not found
[ERR] [2021-06-08 15:16:29][hdls] download_and_prep FAILED for Medium.find(12935646): 404 Not found
[ERR] [2021-06-08 15:16:29][hdls] download_and_prep FAILED for Medium.find(12935647): 404 Not found
[ERR] [2021-06-08 15:16:30][hdls] download_and_prep FAILED for Medium.find(12935648): 404 Not found
[ERR] [2021-06-08 15:16:30][hdls] download_and_prep FAILED for Medium.find(12935649): 404 Not found
[ERR] [2021-06-08 15:16:30][hdls] download_and_prep FAILED for Medium.find(12935650): 404 Not found
[ERR] [2021-06-08 15:16:30][hdls] download_and_prep FAILED for Medium.find(12935651): 404 Not found
[ERR] [2021-06-08 15:16:30][hdls] download_and_prep FAILED for Medium.find(12935652): 404 Not found
[ERR] [2021-06-08 15:16:30][hdls] download_and_prep FAILED for Medium.find(12935653): 404 Not found
[ERR] [2021-06-08 15:16:31][hdls] download_and_prep FAILED for Medium.find(12935654): 404 Not found
[ERR] [2021-06-08 15:16:31][hdls] download_and_prep FAILED for Medium.find(12935655): 404 Not found
[ERR] [2021-06-08 15:16:31][hdls] download_and_prep FAILED for Medium.find(12935657): 404 Not found
[ERR] [2021-06-08 15:16:31][hdls] download_and_prep FAILED for Medium.find(12935656): 404 Not found
[ERR] [2021-06-08 15:16:31][hdls] download_and_prep FAILED for Medium.find(12935658): 404 Not found
[ERR] [2021-06-08 15:16:31][hdls] download_and_prep FAILED for Medium.find(12935661): 404 Not found
[ERR] [2021-06-08 15:16:31][hdls] download_and_prep FAILED for Medium.find(12935659): 404 Not found
[ERR] [2021-06-08 15:16:31][hdls] download_and_prep FAILED for Medium.find(12935660): 404 Not found
[ERR] [2021-06-08 15:16:31][hdls] download_and_prep FAILED for Medium.find(12935662): 404 Not found
[ERR] [2021-06-08 15:16:32][hdls] download_and_prep FAILED for Medium.find(12935664): 404 Not found
[ERR] [2021-06-08 15:16:32][hdls] download_and_prep FAILED for Medium.find(12935663): 404 Not found
[ERR] [2021-06-08 15:16:32][hdls] download_and_prep FAILED for Medium.find(12935665): 404 Not found
[ERR] [2021-06-08 15:16:32][hdls] download_and_prep FAILED for Medium.find(12935666): 404 Not found
[ERR] [2021-06-08 15:16:32][hdls] download_and_prep FAILED for Medium.find(12935667): 404 Not found
[ERR] [2021-06-08 15:16:32][hdls] download_and_prep FAILED for Medium.find(12935668): 404 Not found
[ERR] [2021-06-08 15:16:32][hdls] download_and_prep FAILED for Medium.find(12935669): 404 Not found
[ERR] [2021-06-08 15:16:32][hdls] download_and_prep FAILED for Medium.find(12935670): 404 Not found
[ERR] [2021-06-08 15:16:32][hdls] download_and_prep FAILED for Medium.find(12935671): 404 Not found
[ERR] [2021-06-08 15:16:32][hdls] download_and_prep FAILED for Medium.find(12935672): 404 Not found
[ERR] [2021-06-08 15:16:33][hdls] download_and_prep FAILED for Medium.find(12935673): 404 Not found
[ERR] [2021-06-08 15:16:33][hdls] download_and_prep FAILED for Medium.find(12935674): 404 Not found
[ERR] [2021-06-08 15:16:33][hdls] download_and_prep FAILED for Medium.find(12935675): 404 Not found
[ERR] [2021-06-08 15:16:33][hdls] download_and_prep FAILED for Medium.find(12935676): 404 Not found
[ERR] [2021-06-08 15:16:33][hdls] download_and_prep FAILED for Medium.find(12935677): 404 Not found
[ERR] [2021-06-08 15:16:33][hdls] download_and_prep FAILED for Medium.find(12935678): 404 Not found
[ERR] [2021-06-08 15:16:34][hdls] download_and_prep FAILED for Medium.find(12935679): 404 Not found
[ERR] [2021-06-08 15:16:34][hdls] download_and_prep FAILED for Medium.find(12935680): 404 Not found
[ERR] [2021-06-08 15:16:34][hdls] download_and_prep FAILED for Medium.find(12935681): 404 Not found
[ERR] [2021-06-08 15:17:01][hdls] download_and_prep FAILED for Medium.find(12935889): 404 Not found
[ERR] [2021-06-08 15:17:01][hdls] download_and_prep FAILED for Medium.find(12935890): 404 Not found
[ERR] [2021-06-08 15:17:01][hdls] download_and_prep FAILED for Medium.find(12935893): 404 Not found
[ERR] [2021-06-08 15:17:01][hdls] download_and_prep FAILED for Medium.find(12935891): 404 Not found
[ERR] [2021-06-08 15:17:01][hdls] download_and_prep FAILED for Medium.find(12935892): 404 Not found
[ERR] [2021-06-08 15:17:01][hdls] download_and_prep FAILED for Medium.find(12935894): 404 Not found
[ERR] [2021-06-08 15:17:01][hdls] download_and_prep FAILED for Medium.find(12935896): 404 Not found
[ERR] [2021-06-08 15:17:01][hdls] download_and_prep FAILED for Medium.find(12935895): 404 Not found
[ERR] [2021-06-08 15:17:02][hdls] download_and_prep FAILED for Medium.find(12935897): 404 Not found
[ERR] [2021-06-08 15:17:02][hdls] download_and_prep FAILED for Medium.find(12935898): 404 Not found
[ERR] [2021-06-08 15:17:02][hdls] download_and_prep FAILED for Medium.find(12935899): 404 Not found
[ERR] [2021-06-08 15:17:02][hdls] download_and_prep FAILED for Medium.find(12935902): 404 Not found
[ERR] [2021-06-08 15:17:02][hdls] download_and_prep FAILED for Medium.find(12935901): 404 Not found
[ERR] [2021-06-08 15:17:02][hdls] download_and_prep FAILED for Medium.find(12935900): 404 Not found
[ERR] [2021-06-08 15:17:02][hdls] download_and_prep FAILED for Medium.find(12935903): 404 Not found
[ERR] [2021-06-08 15:17:02][hdls] download_and_prep FAILED for Medium.find(12935904): 404 Not found
[ERR] [2021-06-08 15:17:02][hdls] download_and_prep FAILED for Medium.find(12935907): 404 Not found
[ERR] [2021-06-08 15:17:02][hdls] download_and_prep FAILED for Medium.find(12935905): 404 Not found
[ERR] [2021-06-08 15:17:02][hdls] download_and_prep FAILED for Medium.find(12935906): 404 Not found
[ERR] [2021-06-08 15:17:02][hdls] download_and_prep FAILED for Medium.find(12935908): 404 Not found
[ERR] [2021-06-08 15:17:02][hdls] download_and_prep FAILED for Medium.find(12935909): 404 Not found
[ERR] [2021-06-08 15:17:13][hdls] download_and_prep FAILED for Medium.find(12935967): 404 Not found
[ERR] [2021-06-08 15:17:13][hdls] download_and_prep FAILED for Medium.find(12935968): 404 Not found
[ERR] [2021-06-08 15:17:13][hdls] download_and_prep FAILED for Medium.find(12935969): 404 Not found
[ERR] [2021-06-08 15:17:13][hdls] download_and_prep FAILED for Medium.find(12935970): 404 Not found
[ERR] [2021-06-08 15:17:13][hdls] download_and_prep FAILED for Medium.find(12935972): 404 Not found
[ERR] [2021-06-08 15:17:13][hdls] download_and_prep FAILED for Medium.find(12935971): 404 Not found
[ERR] [2021-06-08 15:17:13][hdls] download_and_prep FAILED for Medium.find(12935973): 404 Not found
[ERR] [2021-06-08 15:17:13][hdls] download_and_prep FAILED for Medium.find(12935976): 404 Not found
[ERR] [2021-06-08 15:17:13][hdls] download_and_prep FAILED for Medium.find(12935974): 404 Not found
[ERR] [2021-06-08 15:17:13][hdls] download_and_prep FAILED for Medium.find(12935975): 404 Not found
[ERR] [2021-06-08 15:17:13][hdls] download_and_prep FAILED for Medium.find(12935977): 404 Not found
[ERR] [2021-06-08 15:17:14][hdls] download_and_prep FAILED for Medium.find(12935978): 404 Not found
[ERR] [2021-06-08 15:17:14][hdls] download_and_prep FAILED for Medium.find(12935981): 404 Not found
[ERR] [2021-06-08 15:17:14][hdls] download_and_prep FAILED for Medium.find(12935980): 404 Not found
[ERR] [2021-06-08 15:17:14][hdls] download_and_prep FAILED for Medium.find(12935979): 404 Not found
[ERR] [2021-06-08 15:17:14][hdls] download_and_prep FAILED for Medium.find(12935982): 404 Not found
[ERR] [2021-06-08 15:17:14][hdls] download_and_prep FAILED for Medium.find(12935983): 404 Not found
[ERR] [2021-06-08 15:17:14][hdls] download_and_prep FAILED for Medium.find(12935984): 404 Not found
[ERR] [2021-06-08 15:17:15][hdls] download_and_prep FAILED for Medium.find(12935985): 404 Not found
[INFO] [2021-06-08 15:17:17] 16438 Traits (filtered)...
[INFO] [2021-06-08 15:17:20] 0 Associations (filtered)...
[INFO] [2021-06-08 15:17:42] 151620 metadata added.
[INFO] [2021-06-08 15:17:42] 0 metadata added.
[INFO] [2021-06-08 15:21:49] 8123 Traits (unfiltered)...
[ERR] [2021-06-08 15:22:09][hdls] download_and_prep FAILED for Medium.find(12938063): 404 Not found
[INFO] [2021-06-08 15:23:24] 8123 Traits (filtered)...
[INFO] [2021-06-08 15:23:26] 0 Associations (filtered)...
[INFO] [2021-06-08 15:23:39] 73259 metadata added.
[INFO] [2021-06-08 15:23:39] 0 metadata added.
[INFO] [2021-06-08 15:24:48] 386 Traits (unfiltered)...
[INFO] [2021-06-08 15:25:24] 386 Traits (filtered)...
[INFO] [2021-06-08 15:25:24] 0 Associations (filtered)...
[INFO] [2021-06-08 15:25:25] 3474 metadata added.
[INFO] [2021-06-08 15:25:25] 0 metadata added.
[INFO] [2021-06-08 15:25:54] Average Time: 249.982
[INFO] [2021-06-08 15:25:54] Total Time: 26m26s
[STOP] [2021-06-08 15:25:54] overall_tsv_creation
[INFO] [2021-06-08 15:25:54] Done. Check your files:
[INFO] [2021-06-08 15:25:54] (39938 lines) /app/public/data/hvd_museum_compa/publish_nodes.tsv
[INFO] [2021-06-08 15:25:54] (222256 lines) /app/public/data/hvd_museum_compa/publish_node_ancestors.tsv
[INFO] [2021-06-08 15:25:54] (40077 lines) /app/public/data/hvd_museum_compa/publish_scientific_names.tsv
[INFO] [2021-06-08 15:25:55] (204028 lines) /app/public/data/hvd_museum_compa/publish_media.tsv
[INFO] [2021-06-08 15:25:55] (22374 lines) /app/public/data/hvd_museum_compa/publish_image_info.tsv
[INFO] [2021-06-08 15:25:55] (204028 lines) /app/public/data/hvd_museum_compa/publish_attributions.tsv
[INFO] [2021-06-08 15:25:56] (50936 lines) /app/public/data/hvd_museum_compa/publish_traits.tsv
[INFO] [2021-06-08 15:25:56] (466568 lines) /app/public/data/hvd_museum_compa/publish_metadata.tsv
[STOP] [2021-06-08 15:25:56] complete_harvest_instance
[START] [2021-06-08 15:25:56] completed
[STOP] [2021-06-08 15:25:56] completed
[STOP] [2021-06-08 15:25:56] logged process, took 4994.0
[ERR] [2021-06-08 15:32:14][hdls] download_and_prep FAILED for Medium.find(12942303): 404 Not found
[ERR] [2021-06-08 15:32:15][hdls] download_and_prep FAILED for Medium.find(12942310): 404 Not found
[ERR] [2021-06-08 15:32:18][hdls] download_and_prep FAILED for Medium.find(12942342): 404 Not found
[ERR] [2021-06-08 15:32:18][hdls] download_and_prep FAILED for Medium.find(12942343): 404 Not found
[ERR] [2021-06-08 15:34:19][hdls] download_and_prep FAILED for Medium.find(12943185): 404 Not found
[ERR] [2021-06-08 15:34:19][hdls] download_and_prep FAILED for Medium.find(12943186): 404 Not found
[ERR] [2021-06-08 15:41:12][hdls] download_and_prep FAILED for Medium.find(12946163): 404 Not found
[ERR] [2021-06-08 15:43:28][hdls] download_and_prep FAILED for Medium.find(12947108): 404 Not found
[ERR] [2021-06-08 15:43:28][hdls] download_and_prep FAILED for Medium.find(12947109): 404 Not found
[ERR] [2021-06-08 15:43:29][hdls] download_and_prep FAILED for Medium.find(12947110): 404 Not found
[ERR] [2021-06-08 15:44:13][hdls] download_and_prep FAILED for Medium.find(12947452): 404 Not found
[ERR] [2021-06-08 15:45:33][hdls] download_and_prep FAILED for Medium.find(12948018): 404 Not found
[ERR] [2021-06-08 15:52:59][hdls] download_and_prep FAILED for Medium.find(12951093): 404 Not found
[ERR] [2021-06-08 15:54:01][hdls] download_and_prep FAILED for Medium.find(12951568): 404 Not found
[ERR] [2021-06-08 15:57:32][hdls] download_and_prep FAILED for Medium.find(12953065): 404 Not found
[ERR] [2021-06-08 15:57:33][hdls] download_and_prep FAILED for Medium.find(12953071): 404 Not found
[ERR] [2021-06-08 15:57:34][hdls] download_and_prep FAILED for Medium.find(12953084): 404 Not found
[ERR] [2021-06-08 16:01:49][hdls] download_and_prep FAILED for Medium.find(12954915): 404 Not found
[ERR] [2021-06-08 16:06:46][hdls] Couldn't parse image for Medium #12957107 (zero-length blob not permitted `' @ error/blob.c/BlobToImage/349)
[ERR] [2021-06-08 16:06:46][hdls] download_and_prep FAILED for Medium.find(12957107): unparsable
[ERR] [2021-06-08 16:08:27][hdls] Couldn't parse image for Medium #12957854 (zero-length blob not permitted `' @ error/blob.c/BlobToImage/349)
[ERR] [2021-06-08 16:08:27][hdls] download_and_prep FAILED for Medium.find(12957854): unparsable
[ERR] [2021-06-08 16:10:53][hdls] Couldn't parse image for Medium #12958884 (zero-length blob not permitted `' @ error/blob.c/BlobToImage/349)
[ERR] [2021-06-08 16:10:53][hdls] download_and_prep FAILED for Medium.find(12958884): unparsable
[ERR] [2021-06-08 16:20:08][hdls] download_and_prep FAILED for Medium.find(12962848): 404 Not found
[ERR] [2021-06-08 16:29:34][hdls] download_and_prep FAILED for Medium.find(12967026): 404 Not found
[ERR] [2021-06-08 16:34:21][hdls] download_and_prep FAILED for Medium.find(12969081): 404 Not found
[ERR] [2021-06-08 16:37:08][hdls] Couldn't parse image for Medium #12970205 (zero-length blob not permitted `' @ error/blob.c/BlobToImage/349)
[ERR] [2021-06-08 16:37:08][hdls] download_and_prep FAILED for Medium.find(12970205): unparsable
[ERR] [2021-06-08 16:43:34][hdls] download_and_prep FAILED for Medium.find(12972746): 404 Not found
[ERR] [2021-06-08 16:43:44][hdls] download_and_prep FAILED for Medium.find(12972816): 404 Not found
[ERR] [2021-06-08 17:49:46][hdls] download_and_prep FAILED for Medium.find(13000411): 404 Not found
[ERR] [2021-06-08 17:49:59][hdls] download_and_prep FAILED for Medium.find(13000472): 404 Not found
[ERR] [2021-06-08 17:51:04][hdls] download_and_prep FAILED for Medium.find(13000864): 404 Not found
[ERR] [2021-06-08 19:32:17][hdls] download_and_prep FAILED for Medium.find(13032186): 404 Not found
[ERR] [2021-06-08 19:59:27][hdls] download_and_prep FAILED for Medium.find(13041423): execution expired
[ERR] [2021-06-08 20:45:00][hdls] download_and_prep FAILED for Medium.find(13053987): 404 Not found
[ERR] [2021-06-08 20:45:00][hdls] download_and_prep FAILED for Medium.find(13053990): 404 Not found
[ERR] [2021-06-08 21:15:59][hdls] download_and_prep FAILED for Medium.find(13060400): 404 Not found
[ERR] [2021-06-08 21:15:59][hdls] download_and_prep FAILED for Medium.find(13060399): 404 Not found
[ERR] [2021-06-08 21:16:00][hdls] download_and_prep FAILED for Medium.find(13060401): 404 Not found
[ERR] [2021-06-08 21:52:08][hdls] download_and_prep FAILED for Medium.find(13067332): 404 Not found
[ERR] [2021-06-08 21:58:15][hdls] download_and_prep FAILED for Medium.find(13071540): 404 Not found
[ERR] [2021-06-08 21:58:16][hdls] download_and_prep FAILED for Medium.find(13071540): 404 Not found
[ERR] [2021-06-08 22:16:58][hdls] download_and_prep FAILED for Medium.find(13078403): 404 Not found
[ERR] [2021-06-08 22:18:25][hdls] download_and_prep FAILED for Medium.find(13079513): execution expired
[ERR] [2021-06-08 22:23:11][hdls] download_and_prep FAILED for Medium.find(13081712): 404 Not found
[ERR] [2021-06-08 22:27:27][hdls] download_and_prep FAILED for Medium.find(13084246): 404 Not found
[ERR] [2021-06-08 22:29:10][hdls] download_and_prep FAILED for Medium.find(13085214): 404 Not found
[ERR] [2021-06-08 22:31:49][hdls] download_and_prep FAILED for Medium.find(13087217): 404 Not found
[ERR] [2021-06-08 22:31:49][hdls] download_and_prep FAILED for Medium.find(13087217): 404 Not found
[ERR] [2021-06-08 22:39:10][hdls] download_and_prep FAILED for Medium.find(13091965): 404 Not found
[ERR] [2021-06-08 22:39:10][hdls] download_and_prep FAILED for Medium.find(13091964): 404 Not found
[ERR] [2021-06-08 22:39:10][hdls] download_and_prep FAILED for Medium.find(13091967): 404 Not found
[ERR] [2021-06-08 22:39:11][hdls] download_and_prep FAILED for Medium.find(13091965): 404 Not found
[ERR] [2021-06-08 23:21:53][hdls] download_and_prep FAILED for Medium.find(13109626): 404 Not found
[ERR] [2021-06-08 23:28:12][hdls] download_and_prep FAILED for Medium.find(13113807): 404 Not found
[ERR] [2021-06-08 23:28:27][hdls] download_and_prep FAILED for Medium.find(13113967): 404 Not found
[ERR] [2021-06-08 23:28:27][hdls] download_and_prep FAILED for Medium.find(13113968): 404 Not found
[ERR] [2021-06-08 23:28:29][hdls] download_and_prep FAILED for Medium.find(13113978): 404 Not found
[ERR] [2021-06-08 23:29:15][hdls] download_and_prep FAILED for Medium.find(13114454): 404 Not found

Latest Process

08 Jun 14:02 EXITED currently processing 0