harvested resources
sign in

Harvest for Arctic Arthropods Created 17 May 14:08

Stage: completed
Fetched: 17 May 14:08
Validated: 17 May 14:08
Deltas Created 17 May 14:08
Units Normalized: 17 May 14:11
Ancestry Built: 17 May 14:08
Nodes Matched: 17 May 14:11
Names Parsed: 17 May 14:08
New Models Stored: 17 May 14:08
Indexed: 17 May 14:11
Completed: 17 May 14:15
Time to Harvest: less than a minute

Harvesting Log

(316 lines) 
[INFO] [2021-05-17 14:08:01] Created harvest instance #3864
[STOP] [2021-05-17 14:08:01] create_harvest_instance
[START] [2021-05-17 14:08:01] fetch_files
[STOP] [2021-05-17 14:08:01] fetch_files
[START] [2021-05-17 14:08:01] validate_each_file
[INFO] [2021-05-17 14:08:01] Looping over 8 formats...
[INFO] [2021-05-17 14:08:01] ...agents (/app/public/data/Arctic_Arthropod/agents.txt)
[INFO] [2021-05-17 14:08:01] Valid: /app/public/converted_csv/Arctic_Arthropod_agents_3864.csv (0 lines)
[INFO] [2021-05-17 14:08:01] ...refs (/app/public/data/Arctic_Arthropod/references.txt)
[INFO] [2021-05-17 14:08:01] Valid: /app/public/converted_csv/Arctic_Arthropod_refs_3864.csv (0 lines)
[INFO] [2021-05-17 14:08:01] ...nodes (/app/public/data/Arctic_Arthropod/taxa.txt)
[INFO] [2021-05-17 14:08:01] Valid: /app/public/converted_csv/Arctic_Arthropod_nodes_3864.csv (1849 lines)
[INFO] [2021-05-17 14:08:01] ...media (/app/public/data/Arctic_Arthropod/media.txt)
[INFO] [2021-05-17 14:08:02] Valid: /app/public/converted_csv/Arctic_Arthropod_media_3864.csv (0 lines)
[INFO] [2021-05-17 14:08:02] ...vernaculars (/app/public/data/Arctic_Arthropod/common names.txt)
[INFO] [2021-05-17 14:08:02] Valid: /app/public/converted_csv/Arctic_Arthropod_vernaculars_3864.csv (0 lines)
[INFO] [2021-05-17 14:08:02] ...occurrences (/app/public/data/Arctic_Arthropod/occurrences.txt)
[INFO] [2021-05-17 14:08:02] Valid: /app/public/converted_csv/Arctic_Arthropod_occurrences_3864.csv (1849 lines)
[INFO] [2021-05-17 14:08:02] ...assocs (/app/public/data/Arctic_Arthropod/associations.txt)
[INFO] [2021-05-17 14:08:02] Valid: /app/public/converted_csv/Arctic_Arthropod_assocs_3864.csv (0 lines)
[INFO] [2021-05-17 14:08:02] ...measurements (/app/public/data/Arctic_Arthropod/measurements or facts.txt)
[INFO] [2021-05-17 14:08:03] Valid: /app/public/converted_csv/Arctic_Arthropod_measurements_3864.csv (14961 lines)
[STOP] [2021-05-17 14:08:03] validate_each_file
[START] [2021-05-17 14:08:03] convert_to_csv
[INFO] [2021-05-17 14:08:03] Looping over 8 formats...
[INFO] [2021-05-17 14:08:03] ...agents (/app/public/data/Arctic_Arthropod/agents.txt)
[CMD] [2021-05-17 14:08:03] /usr/bin/sort /app/public/converted_csv/Arctic_Arthropod_agents_3864.csv > /app/public/converted_csv/Arctic_Arthropod_agents_3864.csv_sorted
[INFO] [2021-05-17 14:08:03] Converted: /app/public/converted_csv/Arctic_Arthropod_agents_3864.csv (0 lines)
[INFO] [2021-05-17 14:08:03] ...refs (/app/public/data/Arctic_Arthropod/references.txt)
[CMD] [2021-05-17 14:08:03] /usr/bin/sort /app/public/converted_csv/Arctic_Arthropod_refs_3864.csv > /app/public/converted_csv/Arctic_Arthropod_refs_3864.csv_sorted
[INFO] [2021-05-17 14:08:03] Converted: /app/public/converted_csv/Arctic_Arthropod_refs_3864.csv (0 lines)
[INFO] [2021-05-17 14:08:03] ...nodes (/app/public/data/Arctic_Arthropod/taxa.txt)
[CMD] [2021-05-17 14:08:03] /usr/bin/sort /app/public/converted_csv/Arctic_Arthropod_nodes_3864.csv > /app/public/converted_csv/Arctic_Arthropod_nodes_3864.csv_sorted
[INFO] [2021-05-17 14:08:03] Converted: /app/public/converted_csv/Arctic_Arthropod_nodes_3864.csv (1849 lines)
[INFO] [2021-05-17 14:08:03] ...media (/app/public/data/Arctic_Arthropod/media.txt)
[CMD] [2021-05-17 14:08:03] /usr/bin/sort /app/public/converted_csv/Arctic_Arthropod_media_3864.csv > /app/public/converted_csv/Arctic_Arthropod_media_3864.csv_sorted
[INFO] [2021-05-17 14:08:03] Converted: /app/public/converted_csv/Arctic_Arthropod_media_3864.csv (0 lines)
[INFO] [2021-05-17 14:08:03] ...vernaculars (/app/public/data/Arctic_Arthropod/common names.txt)
[CMD] [2021-05-17 14:08:03] /usr/bin/sort /app/public/converted_csv/Arctic_Arthropod_vernaculars_3864.csv > /app/public/converted_csv/Arctic_Arthropod_vernaculars_3864.csv_sorted
[INFO] [2021-05-17 14:08:03] Converted: /app/public/converted_csv/Arctic_Arthropod_vernaculars_3864.csv (0 lines)
[INFO] [2021-05-17 14:08:03] ...occurrences (/app/public/data/Arctic_Arthropod/occurrences.txt)
[CMD] [2021-05-17 14:08:03] /usr/bin/sort /app/public/converted_csv/Arctic_Arthropod_occurrences_3864.csv > /app/public/converted_csv/Arctic_Arthropod_occurrences_3864.csv_sorted
[INFO] [2021-05-17 14:08:03] Converted: /app/public/converted_csv/Arctic_Arthropod_occurrences_3864.csv (1849 lines)
[INFO] [2021-05-17 14:08:03] ...assocs (/app/public/data/Arctic_Arthropod/associations.txt)
[CMD] [2021-05-17 14:08:03] /usr/bin/sort /app/public/converted_csv/Arctic_Arthropod_assocs_3864.csv > /app/public/converted_csv/Arctic_Arthropod_assocs_3864.csv_sorted
[INFO] [2021-05-17 14:08:03] Converted: /app/public/converted_csv/Arctic_Arthropod_assocs_3864.csv (0 lines)
[INFO] [2021-05-17 14:08:03] ...measurements (/app/public/data/Arctic_Arthropod/measurements or facts.txt)
[CMD] [2021-05-17 14:08:03] /usr/bin/sort /app/public/converted_csv/Arctic_Arthropod_measurements_3864.csv > /app/public/converted_csv/Arctic_Arthropod_measurements_3864.csv_sorted
[INFO] [2021-05-17 14:08:03] Converted: /app/public/converted_csv/Arctic_Arthropod_measurements_3864.csv (14961 lines)
[STOP] [2021-05-17 14:08:03] convert_to_csv
[START] [2021-05-17 14:08:03] calculate_delta
[INFO] [2021-05-17 14:08:03] Looping over 8 formats...
[INFO] [2021-05-17 14:08:03] ...agents (/app/public/data/Arctic_Arthropod/agents.txt)
[CMD] [2021-05-17 14:08:03] echo "0a" > /app/public/diff/Arctic_Arthropod_agents_3864.diff
[CMD] [2021-05-17 14:08:03] tail -n +1 /app/public/converted_csv/Arctic_Arthropod_agents_3864.csv >> /app/public/diff/Arctic_Arthropod_agents_3864.diff
[CMD] [2021-05-17 14:08:03] echo "." >> /app/public/diff/Arctic_Arthropod_agents_3864.diff
[INFO] [2021-05-17 14:08:03] Created diff: /app/public/diff/Arctic_Arthropod_agents_3864.diff (2 lines)
[INFO] [2021-05-17 14:08:03] ...refs (/app/public/data/Arctic_Arthropod/references.txt)
[CMD] [2021-05-17 14:08:03] echo "0a" > /app/public/diff/Arctic_Arthropod_refs_3864.diff
[CMD] [2021-05-17 14:08:03] tail -n +1 /app/public/converted_csv/Arctic_Arthropod_refs_3864.csv >> /app/public/diff/Arctic_Arthropod_refs_3864.diff
[CMD] [2021-05-17 14:08:03] echo "." >> /app/public/diff/Arctic_Arthropod_refs_3864.diff
[INFO] [2021-05-17 14:08:03] Created diff: /app/public/diff/Arctic_Arthropod_refs_3864.diff (2 lines)
[INFO] [2021-05-17 14:08:03] ...nodes (/app/public/data/Arctic_Arthropod/taxa.txt)
[CMD] [2021-05-17 14:08:03] echo "0a" > /app/public/diff/Arctic_Arthropod_nodes_3864.diff
[CMD] [2021-05-17 14:08:03] tail -n +1 /app/public/converted_csv/Arctic_Arthropod_nodes_3864.csv >> /app/public/diff/Arctic_Arthropod_nodes_3864.diff
[CMD] [2021-05-17 14:08:03] echo "." >> /app/public/diff/Arctic_Arthropod_nodes_3864.diff
[INFO] [2021-05-17 14:08:03] Created diff: /app/public/diff/Arctic_Arthropod_nodes_3864.diff (1851 lines)
[INFO] [2021-05-17 14:08:03] ...media (/app/public/data/Arctic_Arthropod/media.txt)
[CMD] [2021-05-17 14:08:03] echo "0a" > /app/public/diff/Arctic_Arthropod_media_3864.diff
[CMD] [2021-05-17 14:08:03] tail -n +1 /app/public/converted_csv/Arctic_Arthropod_media_3864.csv >> /app/public/diff/Arctic_Arthropod_media_3864.diff
[CMD] [2021-05-17 14:08:04] echo "." >> /app/public/diff/Arctic_Arthropod_media_3864.diff
[INFO] [2021-05-17 14:08:04] Created diff: /app/public/diff/Arctic_Arthropod_media_3864.diff (2 lines)
[INFO] [2021-05-17 14:08:04] ...vernaculars (/app/public/data/Arctic_Arthropod/common names.txt)
[CMD] [2021-05-17 14:08:04] echo "0a" > /app/public/diff/Arctic_Arthropod_vernaculars_3864.diff
[CMD] [2021-05-17 14:08:04] tail -n +1 /app/public/converted_csv/Arctic_Arthropod_vernaculars_3864.csv >> /app/public/diff/Arctic_Arthropod_vernaculars_3864.diff
[CMD] [2021-05-17 14:08:04] echo "." >> /app/public/diff/Arctic_Arthropod_vernaculars_3864.diff
[INFO] [2021-05-17 14:08:04] Created diff: /app/public/diff/Arctic_Arthropod_vernaculars_3864.diff (2 lines)
[INFO] [2021-05-17 14:08:04] ...occurrences (/app/public/data/Arctic_Arthropod/occurrences.txt)
[CMD] [2021-05-17 14:08:04] echo "0a" > /app/public/diff/Arctic_Arthropod_occurrences_3864.diff
[CMD] [2021-05-17 14:08:04] tail -n +1 /app/public/converted_csv/Arctic_Arthropod_occurrences_3864.csv >> /app/public/diff/Arctic_Arthropod_occurrences_3864.diff
[CMD] [2021-05-17 14:08:04] echo "." >> /app/public/diff/Arctic_Arthropod_occurrences_3864.diff
[INFO] [2021-05-17 14:08:04] Created diff: /app/public/diff/Arctic_Arthropod_occurrences_3864.diff (1851 lines)
[INFO] [2021-05-17 14:08:04] ...assocs (/app/public/data/Arctic_Arthropod/associations.txt)
[CMD] [2021-05-17 14:08:04] echo "0a" > /app/public/diff/Arctic_Arthropod_assocs_3864.diff
[CMD] [2021-05-17 14:08:04] tail -n +1 /app/public/converted_csv/Arctic_Arthropod_assocs_3864.csv >> /app/public/diff/Arctic_Arthropod_assocs_3864.diff
[CMD] [2021-05-17 14:08:04] echo "." >> /app/public/diff/Arctic_Arthropod_assocs_3864.diff
[INFO] [2021-05-17 14:08:04] Created diff: /app/public/diff/Arctic_Arthropod_assocs_3864.diff (2 lines)
[INFO] [2021-05-17 14:08:04] ...measurements (/app/public/data/Arctic_Arthropod/measurements or facts.txt)
[CMD] [2021-05-17 14:08:04] echo "0a" > /app/public/diff/Arctic_Arthropod_measurements_3864.diff
[CMD] [2021-05-17 14:08:04] tail -n +1 /app/public/converted_csv/Arctic_Arthropod_measurements_3864.csv >> /app/public/diff/Arctic_Arthropod_measurements_3864.diff
[CMD] [2021-05-17 14:08:04] echo "." >> /app/public/diff/Arctic_Arthropod_measurements_3864.diff
[INFO] [2021-05-17 14:08:04] Created diff: /app/public/diff/Arctic_Arthropod_measurements_3864.diff (14963 lines)
[STOP] [2021-05-17 14:08:04] calculate_delta
[START] [2021-05-17 14:08:04] parse_diff_and_store
[INFO] [2021-05-17 14:08:04] Handling diff: /app/public/diff/Arctic_Arthropod_agents_3864.diff (2 lines)
[INFO] [2021-05-17 14:08:04] Loading agents diff file into memory (2 /app/public/diff/Arctic_Arthropod_agents_3864.diff lines)...
[INFO] [2021-05-17 14:08:04] Handling diff: /app/public/diff/Arctic_Arthropod_refs_3864.diff (2 lines)
[INFO] [2021-05-17 14:08:04] Loading refs diff file into memory (2 /app/public/diff/Arctic_Arthropod_refs_3864.diff lines)...
[INFO] [2021-05-17 14:08:04] Handling diff: /app/public/diff/Arctic_Arthropod_nodes_3864.diff (1851 lines)
[INFO] [2021-05-17 14:08:04] Loading nodes diff file into memory (1851 /app/public/diff/Arctic_Arthropod_nodes_3864.diff lines)...
[INFO] [2021-05-17 14:08:05] Handling diff: /app/public/diff/Arctic_Arthropod_media_3864.diff (2 lines)
[INFO] [2021-05-17 14:08:05] Loading media diff file into memory (2 /app/public/diff/Arctic_Arthropod_media_3864.diff lines)...
[INFO] [2021-05-17 14:08:05] Handling diff: /app/public/diff/Arctic_Arthropod_vernaculars_3864.diff (2 lines)
[INFO] [2021-05-17 14:08:05] Loading vernaculars diff file into memory (2 /app/public/diff/Arctic_Arthropod_vernaculars_3864.diff lines)...
[INFO] [2021-05-17 14:08:05] Handling diff: /app/public/diff/Arctic_Arthropod_occurrences_3864.diff (1851 lines)
[INFO] [2021-05-17 14:08:05] Loading occurrences diff file into memory (1851 /app/public/diff/Arctic_Arthropod_occurrences_3864.diff lines)...
[INFO] [2021-05-17 14:08:05] Handling diff: /app/public/diff/Arctic_Arthropod_assocs_3864.diff (2 lines)
[INFO] [2021-05-17 14:08:05] Loading assocs diff file into memory (2 /app/public/diff/Arctic_Arthropod_assocs_3864.diff lines)...
[INFO] [2021-05-17 14:08:05] Handling diff: /app/public/diff/Arctic_Arthropod_measurements_3864.diff (14963 lines)
[INFO] [2021-05-17 14:08:05] Loading measurements diff file into memory (14963 /app/public/diff/Arctic_Arthropod_measurements_3864.diff lines)...
[INFO] [2021-05-17 14:08:14] Storing 2863 ScientificNames
[INFO] [2021-05-17 14:08:14] Processing group of 2863 in 3 groups of 1000
[INFO] [2021-05-17 14:08:14] Average Time: 0.287
[INFO] [2021-05-17 14:08:14] Total Time: 1s
[INFO] [2021-05-17 14:08:14] Storing 2863 Nodes
[INFO] [2021-05-17 14:08:14] Processing group of 2863 in 3 groups of 1000
[INFO] [2021-05-17 14:08:15] Average Time: 0.263
[INFO] [2021-05-17 14:08:15] Total Time: 1s
[INFO] [2021-05-17 14:08:15] Storing 1849 Occurrences
[INFO] [2021-05-17 14:08:15] Processing group of 1849 in 2 groups of 1000
[INFO] [2021-05-17 14:08:15] Average Time: 0.095
[INFO] [2021-05-17 14:08:15] Total Time: 1s
[INFO] [2021-05-17 14:08:15] Storing 14961 Traits
[INFO] [2021-05-17 14:08:15] Processing group of 14961 in 15 groups of 1000
[INFO] [2021-05-17 14:08:20] Average Time: 0.283
[INFO] [2021-05-17 14:08:20] Total Time: 5s
[INFO] [2021-05-17 14:08:20] last 3 / first 3: 1.0
[INFO] [2021-05-17 14:08:20] Std.Dev: 0.03162277660168379; Max: 0.35
[INFO] [2021-05-17 14:08:20] Storing 30026 MetaTraits
[INFO] [2021-05-17 14:08:20] Processing group of 30026 in 31 groups of 1000
[INFO] [2021-05-17 14:08:24] Average Time: 0.119
[INFO] [2021-05-17 14:08:24] Total Time: 4s
[INFO] [2021-05-17 14:08:24] last 3 / first 3: 0.76
[INFO] [2021-05-17 14:08:24] Std.Dev: 0.03162277660168379; Max: 0.19
[STOP] [2021-05-17 14:08:24] parse_diff_and_store
[START] [2021-05-17 14:08:24] resolve_keys
[INFO] [2021-05-17 14:08:33] Occurrences to nodes (through scientific_names)...
[INFO] [2021-05-17 14:08:33] traits to occurrences...
[INFO] [2021-05-17 14:08:34] traits to nodes (through occurrences)...
[INFO] [2021-05-17 14:08:34] Traits to sex term...
[INFO] [2021-05-17 14:08:34] Traits to lifestage term...
[INFO] [2021-05-17 14:08:34] MetaTraits to traits...
[INFO] [2021-05-17 14:08:35] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-05-17 14:08:35] Assocs to occurrences...
[INFO] [2021-05-17 14:08:35] Assocs to nodes...
[INFO] [2021-05-17 14:08:35] Assoc to sex term...
[INFO] [2021-05-17 14:08:35] Assoc to lifestage term...
[INFO] [2021-05-17 14:08:35] MetaAssoc to assocs...
[STOP] [2021-05-17 14:08:35] resolve_keys
[START] [2021-05-17 14:08:35] hold_for_later_1
[STOP] [2021-05-17 14:08:35] hold_for_later_1
[START] [2021-05-17 14:08:35] hold_for_later_2
[STOP] [2021-05-17 14:08:35] hold_for_later_2
[START] [2021-05-17 14:08:35] resolve_missing_parents
[STOP] [2021-05-17 14:08:35] resolve_missing_parents
[START] [2021-05-17 14:08:35] rebuild_nodes
[START] [2021-05-17 14:08:35] Flattener#flatten
[START] [2021-05-17 14:08:35] Flattener#study_resource
[START] [2021-05-17 14:08:36] Flattener#build_ancestry
[STOP] [2021-05-17 14:08:36] Flattener#build_ancestry
[INFO] [2021-05-17 14:08:36] 2863 ancestry keys
[START] [2021-05-17 14:08:36] build_node_ancestors
[INFO] [2021-05-17 14:08:36] old ancestors deleted.
[STOP] [2021-05-17 14:08:36] build_node_ancestors
[START] [2021-05-17 14:08:37] Flattener#propagate_ancestor_ids
[STOP] [2021-05-17 14:08:37] Flattener#propagate_ancestor_ids
[STOP] [2021-05-17 14:08:37] Flattener#flatten
[STOP] [2021-05-17 14:08:37] rebuild_nodes
[START] [2021-05-17 14:08:37] resolve_missing_media_owners
[STOP] [2021-05-17 14:08:37] resolve_missing_media_owners
[START] [2021-05-17 14:08:37] sanitize_media_verbatims
[STOP] [2021-05-17 14:08:37] sanitize_media_verbatims
[START] [2021-05-17 14:08:37] queue_downloads
[STOP] [2021-05-17 14:08:37] queue_downloads
[START] [2021-05-17 14:08:37] parse_names
[WARN] [2021-05-17 14:08:37] I see 2863 names which still need to be parsed.
[WARN] [2021-05-17 14:08:40] I see 2 names which still need to be parsed.
[STOP] [2021-05-17 14:08:41] parse_names
[START] [2021-05-17 14:08:41] denormalize_canonical_names_to_nodes
[STOP] [2021-05-17 14:08:41] denormalize_canonical_names_to_nodes
[START] [2021-05-17 14:08:41] match_nodes
[START] [2021-05-17 14:08:41] map_all_nodes_to_pages
[STOP] [2021-05-17 14:11:22] map_all_nodes_to_pages
[INFO] [2021-05-17 14:11:22] 586 Unmatched nodes (of 2863)! That's too many to output. Full list in /app/public/data/Arctic_Arthropod/unmatched_nodes.txt ; First 10: Canonical: Chironomus grandivalba; Node#94022292; ResourceID: Chironomus_grandivalba; Canonical: Chironomus viridulus; Node#94022297; ResourceID: Chironomus_viridulus; Canonical: Cladotanytarsus; Node#94022334; ResourceID: Cladotanytarsus_sp; Canonical: Corynocera arctica; Node#94022437; ResourceID: Corynocera_arctica; Canonical: Cricotopus basalis; Node#94022449; ResourceID: Cricotopus_basalis; Canonical: Cricotopus humeralis; Node#94022453; ResourceID: Cricotopus_humeralis; Canonical: Cricotopus pavidus; Node#94022455; ResourceID: Cricotopus_pavidus; Canonical: Cricotopus vitripennis; Node#94022458; ResourceID: Cricotopus_vitripennis; Canonical: Diamesa appendiculata; Node#94022543; ResourceID: Diamesa_appendiculata; Canonical: Diamesa parva; Node#94022550; ResourceID: Diamesa_parva
[START] [2021-05-17 14:11:22] update_nodes
[STOP] [2021-05-17 14:11:23] update_nodes
[STOP] [2021-05-17 14:11:23] match_nodes
[START] [2021-05-17 14:11:23] reindex_search
[STOP] [2021-05-17 14:11:25] reindex_search
[START] [2021-05-17 14:11:25] normalize_units
[STOP] [2021-05-17 14:11:26] normalize_units
[START] [2021-05-17 14:11:26] calculate_statistics
[2021-05-17 14:11:26] (NEAR) DUPLICATE TRAITS FOUND! There are only 13872 (of 14961 total) unique traits.
[2021-05-17 14:12:42] (Near) duplicate trait pairs (up to 100):
[2021-05-17 14:12:42] (resource_pk: M_00004449, id: 204522502), (resource_pk: M_00005976, id: 204524029)
[2021-05-17 14:12:42] (resource_pk: M_00004453, id: 204522506), (resource_pk: M_00005977, id: 204524030)
[2021-05-17 14:12:42] (resource_pk: M_00004454, id: 204522507), (resource_pk: M_00005973, id: 204524026)
[2021-05-17 14:12:42] (resource_pk: M_00004455, id: 204522508), (resource_pk: M_00005978, id: 204524031)
[2021-05-17 14:12:42] (resource_pk: M_00004456, id: 204522509), (resource_pk: M_00005979, id: 204524032)
[2021-05-17 14:12:42] (resource_pk: M_00004457, id: 204522510), (resource_pk: M_00005980, id: 204524033)
[2021-05-17 14:12:42] (resource_pk: M_00004458, id: 204522511), (resource_pk: M_00005981, id: 204524034)
[2021-05-17 14:12:42] (resource_pk: M_00004459, id: 204522512), (resource_pk: M_00005982, id: 204524035)
[2021-05-17 14:12:42] (resource_pk: M_00004460, id: 204522513), (resource_pk: M_00005983, id: 204524036)
[2021-05-17 14:12:42] (resource_pk: M_00004461, id: 204522514), (resource_pk: M_00005984, id: 204524037)
[2021-05-17 14:12:42] (resource_pk: M_00004462, id: 204522515), (resource_pk: M_00005985, id: 204524038)
[2021-05-17 14:12:42] (resource_pk: M_00004463, id: 204522516), (resource_pk: M_00005986, id: 204524039)
[2021-05-17 14:12:42] (resource_pk: M_00004464, id: 204522517), (resource_pk: M_00005987, id: 204524040)
[2021-05-17 14:12:42] (resource_pk: M_00004465, id: 204522518), (resource_pk: M_00005988, id: 204524041)
[2021-05-17 14:12:42] (resource_pk: M_00004466, id: 204522519), (resource_pk: M_00005989, id: 204524042)
[2021-05-17 14:12:42] (resource_pk: M_00004467, id: 204522520), (resource_pk: M_00005990, id: 204524043)
[2021-05-17 14:12:42] (resource_pk: M_00004469, id: 204522522), (resource_pk: M_00005991, id: 204524044)
[2021-05-17 14:12:42] (resource_pk: M_00004470, id: 204522523), (resource_pk: M_00005992, id: 204524045)
[2021-05-17 14:12:42] (resource_pk: M_00004471, id: 204522524), (resource_pk: M_00005993, id: 204524046)
[2021-05-17 14:12:42] (resource_pk: M_00004472, id: 204522525), (resource_pk: M_00005994, id: 204524047)
[2021-05-17 14:12:42] (resource_pk: M_00004473, id: 204522526), (resource_pk: M_00005995, id: 204524048)
[2021-05-17 14:12:42] (resource_pk: M_00004474, id: 204522527), (resource_pk: M_00005996, id: 204524049)
[2021-05-17 14:12:42] (resource_pk: M_00004475, id: 204522528), (resource_pk: M_00005997, id: 204524050)
[2021-05-17 14:12:42] (resource_pk: M_00004479, id: 204522532), (resource_pk: M_00005998, id: 204524051)
[2021-05-17 14:12:42] (resource_pk: M_00004480, id: 204522533), (resource_pk: M_00005999, id: 204524052)
[2021-05-17 14:12:42] (resource_pk: M_00004481, id: 204522534), (resource_pk: M_00006000, id: 204524053)
[2021-05-17 14:12:42] (resource_pk: M_00004482, id: 204522535), (resource_pk: M_00006001, id: 204524054)
[2021-05-17 14:12:42] (resource_pk: M_00004483, id: 204522536), (resource_pk: M_00006002, id: 204524055)
[2021-05-17 14:12:42] (resource_pk: M_00004484, id: 204522537), (resource_pk: M_00006003, id: 204524056)
[2021-05-17 14:12:42] (resource_pk: M_00004485, id: 204522538), (resource_pk: M_00006004, id: 204524057)
[2021-05-17 14:12:42] (resource_pk: M_00004487, id: 204522540), (resource_pk: M_00006005, id: 204524058)
[2021-05-17 14:12:42] (resource_pk: M_00004488, id: 204522541), (resource_pk: M_00006006, id: 204524059)
[2021-05-17 14:12:42] (resource_pk: M_00004489, id: 204522542), (resource_pk: M_00006007, id: 204524060)
[2021-05-17 14:12:42] (resource_pk: M_00004490, id: 204522543), (resource_pk: M_00006008, id: 204524061)
[2021-05-17 14:12:42] (resource_pk: M_00004491, id: 204522544), (resource_pk: M_00006009, id: 204524062)
[2021-05-17 14:12:42] (resource_pk: M_00004492, id: 204522545), (resource_pk: M_00006010, id: 204524063)
[2021-05-17 14:12:42] (resource_pk: M_00004494, id: 204522547), (resource_pk: M_00006011, id: 204524064)
[2021-05-17 14:12:42] (resource_pk: M_00004495, id: 204522548), (resource_pk: M_00006012, id: 204524065)
[2021-05-17 14:12:42] (resource_pk: M_00004496, id: 204522549), (resource_pk: M_00006013, id: 204524066)
[2021-05-17 14:12:42] (resource_pk: M_00004497, id: 204522550), (resource_pk: M_00006014, id: 204524067)
[2021-05-17 14:12:42] (resource_pk: M_00004498, id: 204522551), (resource_pk: M_00006015, id: 204524068)
[2021-05-17 14:12:42] (resource_pk: M_00004499, id: 204522552), (resource_pk: M_00006016, id: 204524069)
[2021-05-17 14:12:42] (resource_pk: M_00004501, id: 204522554), (resource_pk: M_00006017, id: 204524070)
[2021-05-17 14:12:42] (resource_pk: M_00004502, id: 204522555), (resource_pk: M_00006018, id: 204524071)
[2021-05-17 14:12:42] (resource_pk: M_00004503, id: 204522556), (resource_pk: M_00006019, id: 204524072)
[2021-05-17 14:12:42] (resource_pk: M_00004504, id: 204522557), (resource_pk: M_00006020, id: 204524073)
[2021-05-17 14:12:42] (resource_pk: M_00004505, id: 204522558), (resource_pk: M_00006021, id: 204524074)
[2021-05-17 14:12:42] (resource_pk: M_00004506, id: 204522559), (resource_pk: M_00006022, id: 204524075)
[2021-05-17 14:12:42] (resource_pk: M_00004507, id: 204522560), (resource_pk: M_00006023, id: 204524076)
[2021-05-17 14:12:42] (resource_pk: M_00004509, id: 204522562), (resource_pk: M_00006024, id: 204524077)
[2021-05-17 14:12:42] (resource_pk: M_00004510, id: 204522563), (resource_pk: M_00006025, id: 204524078)
[2021-05-17 14:12:42] (resource_pk: M_00004511, id: 204522564), (resource_pk: M_00006026, id: 204524079)
[2021-05-17 14:12:42] (resource_pk: M_00004512, id: 204522565), (resource_pk: M_00006027, id: 204524080)
[2021-05-17 14:12:42] (resource_pk: M_00004513, id: 204522566), (resource_pk: M_00006028, id: 204524081)
[2021-05-17 14:12:42] (resource_pk: M_00004514, id: 204522567), (resource_pk: M_00006029, id: 204524082)
[2021-05-17 14:12:42] (resource_pk: M_00004515, id: 204522568), (resource_pk: M_00006030, id: 204524083)
[2021-05-17 14:12:42] (resource_pk: M_00004516, id: 204522569), (resource_pk: M_00006031, id: 204524084)
[2021-05-17 14:12:42] (resource_pk: M_00004517, id: 204522570), (resource_pk: M_00006032, id: 204524085)
[2021-05-17 14:12:42] (resource_pk: M_00004518, id: 204522571), (resource_pk: M_00006033, id: 204524086)
[2021-05-17 14:12:42] (resource_pk: M_00004519, id: 204522572), (resource_pk: M_00006034, id: 204524087)
[2021-05-17 14:12:42] (resource_pk: M_00004520, id: 204522573), (resource_pk: M_00006035, id: 204524088)
[2021-05-17 14:12:42] (resource_pk: M_00004521, id: 204522574), (resource_pk: M_00006036, id: 204524089)
[2021-05-17 14:12:42] (resource_pk: M_00004522, id: 204522575), (resource_pk: M_00006037, id: 204524090)
[2021-05-17 14:12:42] (resource_pk: M_00004523, id: 204522576), (resource_pk: M_00006038, id: 204524091)
[2021-05-17 14:12:42] (resource_pk: M_00004524, id: 204522577), (resource_pk: M_00006039, id: 204524092)
[2021-05-17 14:12:42] (resource_pk: M_00004525, id: 204522578), (resource_pk: M_00006040, id: 204524093)
[2021-05-17 14:12:42] (resource_pk: M_00004526, id: 204522579), (resource_pk: M_00006041, id: 204524094)
[2021-05-17 14:12:42] (resource_pk: M_00004527, id: 204522580), (resource_pk: M_00006042, id: 204524095)
[2021-05-17 14:12:42] (resource_pk: M_00004530, id: 204522583), (resource_pk: M_00006043, id: 204524096)
[2021-05-17 14:12:42] (resource_pk: M_00004531, id: 204522584), (resource_pk: M_00006044, id: 204524097)
[2021-05-17 14:12:42] (resource_pk: M_00004532, id: 204522585), (resource_pk: M_00006045, id: 204524098)
[2021-05-17 14:12:42] (resource_pk: M_00004533, id: 204522586), (resource_pk: M_00006046, id: 204524099)
[2021-05-17 14:12:42] (resource_pk: M_00004534, id: 204522587), (resource_pk: M_00006047, id: 204524100)
[2021-05-17 14:12:42] (resource_pk: M_00004535, id: 204522588), (resource_pk: M_00006048, id: 204524101)
[2021-05-17 14:12:42] (resource_pk: M_00004536, id: 204522589), (resource_pk: M_00006049, id: 204524102)
[2021-05-17 14:12:42] (resource_pk: M_00004537, id: 204522590), (resource_pk: M_00006050, id: 204524103)
[2021-05-17 14:12:42] (resource_pk: M_00004538, id: 204522591), (resource_pk: M_00006051, id: 204524104)
[2021-05-17 14:12:42] (resource_pk: M_00004541, id: 204522594), (resource_pk: M_00006052, id: 204524105)
[2021-05-17 14:12:42] (resource_pk: M_00004542, id: 204522595), (resource_pk: M_00006053, id: 204524106)
[2021-05-17 14:12:42] (resource_pk: M_00004544, id: 204522597), (resource_pk: M_00006054, id: 204524107)
[2021-05-17 14:12:42] (resource_pk: M_00004545, id: 204522598), (resource_pk: M_00006055, id: 204524108)
[2021-05-17 14:12:42] (resource_pk: M_00004546, id: 204522599), (resource_pk: M_00006056, id: 204524109)
[2021-05-17 14:12:42] (resource_pk: M_00004547, id: 204522600), (resource_pk: M_00006057, id: 204524110)
[2021-05-17 14:12:42] (resource_pk: M_00004548, id: 204522601), (resource_pk: M_00006058, id: 204524111)
[2021-05-17 14:12:42] (resource_pk: M_00004551, id: 204522604), (resource_pk: M_00006059, id: 204524112)
[2021-05-17 14:12:42] (resource_pk: M_00004552, id: 204522605), (resource_pk: M_00006060, id: 204524113)
[2021-05-17 14:12:42] (resource_pk: M_00004553, id: 204522606), (resource_pk: M_00006061, id: 204524114)
[2021-05-17 14:12:42] (resource_pk: M_00004554, id: 204522607), (resource_pk: M_00006062, id: 204524115)
[2021-05-17 14:12:42] (resource_pk: M_00004555, id: 204522608), (resource_pk: M_00006063, id: 204524116)
[2021-05-17 14:12:42] (resource_pk: M_00004556, id: 204522609), (resource_pk: M_00006064, id: 204524117)
[2021-05-17 14:12:42] (resource_pk: M_00004557, id: 204522610), (resource_pk: M_00006065, id: 204524118)
[2021-05-17 14:12:42] (resource_pk: M_00004558, id: 204522611), (resource_pk: M_00006066, id: 204524119)
[2021-05-17 14:12:42] (resource_pk: M_00004560, id: 204522613), (resource_pk: M_00006067, id: 204524120)
[2021-05-17 14:12:42] (resource_pk: M_00004561, id: 204522614), (resource_pk: M_00006068, id: 204524121)
[2021-05-17 14:12:42] (resource_pk: M_00004562, id: 204522615), (resource_pk: M_00006069, id: 204524122)
[2021-05-17 14:12:42] (resource_pk: M_00004563, id: 204522616), (resource_pk: M_00006070, id: 204524123)
[2021-05-17 14:12:42] (resource_pk: M_00004564, id: 204522617), (resource_pk: M_00006071, id: 204524124)
[2021-05-17 14:12:42] (resource_pk: M_00004565, id: 204522618), (resource_pk: M_00006072, id: 204524125)
[2021-05-17 14:12:42] (resource_pk: M_00004568, id: 204522621), (resource_pk: M_00006073, id: 204524126)
[2021-05-17 14:12:42] (resource_pk: M_00004569, id: 204522622), (resource_pk: M_00006074, id: 204524127)
[STOP] [2021-05-17 14:12:42] calculate_statistics
[START] [2021-05-17 14:12:42] complete_harvest_instance
[START] [2021-05-17 14:12:42] overall_tsv_creation
[INFO] [2021-05-17 14:12:42] Processing group of 2863 in 1 batches of 10000
[INFO] [2021-05-17 14:13:28] 14961 Traits (unfiltered)...
[INFO] [2021-05-17 14:15:26] 14961 Traits (filtered)...
[INFO] [2021-05-17 14:15:26] 0 Associations (filtered)...
[INFO] [2021-05-17 14:15:27] 104 metadata added.
[INFO] [2021-05-17 14:15:27] 0 metadata added.
[INFO] [2021-05-17 14:15:57] Average Time: 167.61
[INFO] [2021-05-17 14:15:57] Total Time: 3m16s
[STOP] [2021-05-17 14:15:57] overall_tsv_creation
[INFO] [2021-05-17 14:15:57] Done. Check your files:
[INFO] [2021-05-17 14:15:57] (2861 lines) /app/public/data/Arctic_Arthropod/publish_nodes.tsv
[INFO] [2021-05-17 14:15:57] (15694 lines) /app/public/data/Arctic_Arthropod/publish_node_ancestors.tsv
[INFO] [2021-05-17 14:15:57] (2863 lines) /app/public/data/Arctic_Arthropod/publish_scientific_names.tsv
[INFO] [2021-05-17 14:15:57] (14962 lines) /app/public/data/Arctic_Arthropod/publish_traits.tsv
[INFO] [2021-05-17 14:15:57] (105 lines) /app/public/data/Arctic_Arthropod/publish_metadata.tsv
[STOP] [2021-05-17 14:15:57] complete_harvest_instance
[START] [2021-05-17 14:15:57] completed
[STOP] [2021-05-17 14:15:57] completed
[STOP] [2021-05-17 14:15:57] logged process, took 476.05

Latest Process