Harvest for Alaskan Arthropods Created 10 Jun 11:57

Harvesting Log

[INFO] [2021-06-10 11:57:50] Created harvest instance #4019
[STOP] [2021-06-10 11:57:50] create_harvest_instance
[START] [2021-06-10 11:57:50] fetch_files
[STOP] [2021-06-10 11:57:50] fetch_files
[START] [2021-06-10 11:57:50] validate_each_file
[INFO] [2021-06-10 11:57:50] Looping over 8 formats...
[INFO] [2021-06-10 11:57:50] ...agents (/app/public/data/Alaskan_Arthropo/agents.txt)
[INFO] [2021-06-10 11:57:50] Valid: /app/public/converted_csv/Alaskan_Arthropo_agents_4019.csv (0 lines)
[INFO] [2021-06-10 11:57:50] ...refs (/app/public/data/Alaskan_Arthropo/references.txt)
[INFO] [2021-06-10 11:57:50] Valid: /app/public/converted_csv/Alaskan_Arthropo_refs_4019.csv (443 lines)
[INFO] [2021-06-10 11:57:50] ...nodes (/app/public/data/Alaskan_Arthropo/taxa.txt)
[INFO] [2021-06-10 11:57:51] Valid: /app/public/converted_csv/Alaskan_Arthropo_nodes_4019.csv (7682 lines)
[INFO] [2021-06-10 11:57:51] ...media (/app/public/data/Alaskan_Arthropo/media.txt)
[INFO] [2021-06-10 11:57:51] Valid: /app/public/converted_csv/Alaskan_Arthropo_media_4019.csv (0 lines)
[INFO] [2021-06-10 11:57:51] ...vernaculars (/app/public/data/Alaskan_Arthropo/common names.txt)
[INFO] [2021-06-10 11:57:51] Valid: /app/public/converted_csv/Alaskan_Arthropo_vernaculars_4019.csv (0 lines)
[INFO] [2021-06-10 11:57:51] ...occurrences (/app/public/data/Alaskan_Arthropo/occurrences.txt)
[INFO] [2021-06-10 11:57:52] Valid: /app/public/converted_csv/Alaskan_Arthropo_occurrences_4019.csv (7682 lines)
[INFO] [2021-06-10 11:57:52] ...assocs (/app/public/data/Alaskan_Arthropo/associations.txt)
[INFO] [2021-06-10 11:57:52] Valid: /app/public/converted_csv/Alaskan_Arthropo_assocs_4019.csv (0 lines)
[INFO] [2021-06-10 11:57:52] ...measurements (/app/public/data/Alaskan_Arthropo/measurements or facts.txt)
[INFO] [2021-06-10 11:57:53] Valid: /app/public/converted_csv/Alaskan_Arthropo_measurements_4019.csv (7923 lines)
[STOP] [2021-06-10 11:57:53] validate_each_file
[START] [2021-06-10 11:57:53] convert_to_csv
[INFO] [2021-06-10 11:57:53] Looping over 8 formats...
[INFO] [2021-06-10 11:57:53] ...agents (/app/public/data/Alaskan_Arthropo/agents.txt)
[CMD] [2021-06-10 11:57:53] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_agents_4019.csv > /app/public/converted_csv/Alaskan_Arthropo_agents_4019.csv_sorted
[INFO] [2021-06-10 11:57:53] Converted: /app/public/converted_csv/Alaskan_Arthropo_agents_4019.csv (0 lines)
[INFO] [2021-06-10 11:57:53] ...refs (/app/public/data/Alaskan_Arthropo/references.txt)
[CMD] [2021-06-10 11:57:53] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_refs_4019.csv > /app/public/converted_csv/Alaskan_Arthropo_refs_4019.csv_sorted
[INFO] [2021-06-10 11:57:53] Converted: /app/public/converted_csv/Alaskan_Arthropo_refs_4019.csv (443 lines)
[INFO] [2021-06-10 11:57:53] ...nodes (/app/public/data/Alaskan_Arthropo/taxa.txt)
[CMD] [2021-06-10 11:57:53] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_nodes_4019.csv > /app/public/converted_csv/Alaskan_Arthropo_nodes_4019.csv_sorted
[INFO] [2021-06-10 11:57:54] Converted: /app/public/converted_csv/Alaskan_Arthropo_nodes_4019.csv (7682 lines)
[INFO] [2021-06-10 11:57:54] ...media (/app/public/data/Alaskan_Arthropo/media.txt)
[CMD] [2021-06-10 11:57:54] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_media_4019.csv > /app/public/converted_csv/Alaskan_Arthropo_media_4019.csv_sorted
[INFO] [2021-06-10 11:57:54] Converted: /app/public/converted_csv/Alaskan_Arthropo_media_4019.csv (0 lines)
[INFO] [2021-06-10 11:57:54] ...vernaculars (/app/public/data/Alaskan_Arthropo/common names.txt)
[CMD] [2021-06-10 11:57:54] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_vernaculars_4019.csv > /app/public/converted_csv/Alaskan_Arthropo_vernaculars_4019.csv_sorted
[INFO] [2021-06-10 11:57:55] Converted: /app/public/converted_csv/Alaskan_Arthropo_vernaculars_4019.csv (0 lines)
[INFO] [2021-06-10 11:57:55] ...occurrences (/app/public/data/Alaskan_Arthropo/occurrences.txt)
[CMD] [2021-06-10 11:57:55] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_occurrences_4019.csv > /app/public/converted_csv/Alaskan_Arthropo_occurrences_4019.csv_sorted
[INFO] [2021-06-10 11:57:55] Converted: /app/public/converted_csv/Alaskan_Arthropo_occurrences_4019.csv (7682 lines)
[INFO] [2021-06-10 11:57:55] ...assocs (/app/public/data/Alaskan_Arthropo/associations.txt)
[CMD] [2021-06-10 11:57:55] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_assocs_4019.csv > /app/public/converted_csv/Alaskan_Arthropo_assocs_4019.csv_sorted
[INFO] [2021-06-10 11:57:56] Converted: /app/public/converted_csv/Alaskan_Arthropo_assocs_4019.csv (0 lines)
[INFO] [2021-06-10 11:57:56] ...measurements (/app/public/data/Alaskan_Arthropo/measurements or facts.txt)
[CMD] [2021-06-10 11:57:56] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_measurements_4019.csv > /app/public/converted_csv/Alaskan_Arthropo_measurements_4019.csv_sorted
[INFO] [2021-06-10 11:57:56] Converted: /app/public/converted_csv/Alaskan_Arthropo_measurements_4019.csv (7923 lines)
[STOP] [2021-06-10 11:57:56] convert_to_csv
[START] [2021-06-10 11:57:56] calculate_delta
[INFO] [2021-06-10 11:57:56] Looping over 8 formats...
[INFO] [2021-06-10 11:57:56] ...agents (/app/public/data/Alaskan_Arthropo/agents.txt)
[CMD] [2021-06-10 11:57:56] echo "0a" > /app/public/diff/Alaskan_Arthropo_agents_4019.diff
[CMD] [2021-06-10 11:57:57] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_agents_4019.csv >> /app/public/diff/Alaskan_Arthropo_agents_4019.diff
[CMD] [2021-06-10 11:57:57] echo "." >> /app/public/diff/Alaskan_Arthropo_agents_4019.diff
[INFO] [2021-06-10 11:57:58] Created diff: /app/public/diff/Alaskan_Arthropo_agents_4019.diff (2 lines)
[INFO] [2021-06-10 11:57:58] ...refs (/app/public/data/Alaskan_Arthropo/references.txt)
[CMD] [2021-06-10 11:57:58] echo "0a" > /app/public/diff/Alaskan_Arthropo_refs_4019.diff
[CMD] [2021-06-10 11:57:58] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_refs_4019.csv >> /app/public/diff/Alaskan_Arthropo_refs_4019.diff
[CMD] [2021-06-10 11:57:58] echo "." >> /app/public/diff/Alaskan_Arthropo_refs_4019.diff
[INFO] [2021-06-10 11:57:59] Created diff: /app/public/diff/Alaskan_Arthropo_refs_4019.diff (445 lines)
[INFO] [2021-06-10 11:57:59] ...nodes (/app/public/data/Alaskan_Arthropo/taxa.txt)
[CMD] [2021-06-10 11:57:59] echo "0a" > /app/public/diff/Alaskan_Arthropo_nodes_4019.diff
[CMD] [2021-06-10 11:57:59] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_nodes_4019.csv >> /app/public/diff/Alaskan_Arthropo_nodes_4019.diff
[CMD] [2021-06-10 11:58:00] echo "." >> /app/public/diff/Alaskan_Arthropo_nodes_4019.diff
[INFO] [2021-06-10 11:58:00] Created diff: /app/public/diff/Alaskan_Arthropo_nodes_4019.diff (7684 lines)
[INFO] [2021-06-10 11:58:00] ...media (/app/public/data/Alaskan_Arthropo/media.txt)
[CMD] [2021-06-10 11:58:00] echo "0a" > /app/public/diff/Alaskan_Arthropo_media_4019.diff
[CMD] [2021-06-10 11:58:01] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_media_4019.csv >> /app/public/diff/Alaskan_Arthropo_media_4019.diff
[CMD] [2021-06-10 11:58:01] echo "." >> /app/public/diff/Alaskan_Arthropo_media_4019.diff
[INFO] [2021-06-10 11:58:02] Created diff: /app/public/diff/Alaskan_Arthropo_media_4019.diff (2 lines)
[INFO] [2021-06-10 11:58:02] ...vernaculars (/app/public/data/Alaskan_Arthropo/common names.txt)
[CMD] [2021-06-10 11:58:02] echo "0a" > /app/public/diff/Alaskan_Arthropo_vernaculars_4019.diff
[CMD] [2021-06-10 11:58:02] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_vernaculars_4019.csv >> /app/public/diff/Alaskan_Arthropo_vernaculars_4019.diff
[CMD] [2021-06-10 11:58:03] echo "." >> /app/public/diff/Alaskan_Arthropo_vernaculars_4019.diff
[INFO] [2021-06-10 11:58:03] Created diff: /app/public/diff/Alaskan_Arthropo_vernaculars_4019.diff (2 lines)
[INFO] [2021-06-10 11:58:03] ...occurrences (/app/public/data/Alaskan_Arthropo/occurrences.txt)
[CMD] [2021-06-10 11:58:03] echo "0a" > /app/public/diff/Alaskan_Arthropo_occurrences_4019.diff
[CMD] [2021-06-10 11:58:03] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_occurrences_4019.csv >> /app/public/diff/Alaskan_Arthropo_occurrences_4019.diff
[CMD] [2021-06-10 11:58:04] echo "." >> /app/public/diff/Alaskan_Arthropo_occurrences_4019.diff
[INFO] [2021-06-10 11:58:04] Created diff: /app/public/diff/Alaskan_Arthropo_occurrences_4019.diff (7684 lines)
[INFO] [2021-06-10 11:58:04] ...assocs (/app/public/data/Alaskan_Arthropo/associations.txt)
[CMD] [2021-06-10 11:58:04] echo "0a" > /app/public/diff/Alaskan_Arthropo_assocs_4019.diff
[CMD] [2021-06-10 11:58:05] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_assocs_4019.csv >> /app/public/diff/Alaskan_Arthropo_assocs_4019.diff
[CMD] [2021-06-10 11:58:05] echo "." >> /app/public/diff/Alaskan_Arthropo_assocs_4019.diff
[INFO] [2021-06-10 11:58:06] Created diff: /app/public/diff/Alaskan_Arthropo_assocs_4019.diff (2 lines)
[INFO] [2021-06-10 11:58:06] ...measurements (/app/public/data/Alaskan_Arthropo/measurements or facts.txt)
[CMD] [2021-06-10 11:58:06] echo "0a" > /app/public/diff/Alaskan_Arthropo_measurements_4019.diff
[CMD] [2021-06-10 11:58:06] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_measurements_4019.csv >> /app/public/diff/Alaskan_Arthropo_measurements_4019.diff
[CMD] [2021-06-10 11:58:07] echo "." >> /app/public/diff/Alaskan_Arthropo_measurements_4019.diff
[INFO] [2021-06-10 11:58:07] Created diff: /app/public/diff/Alaskan_Arthropo_measurements_4019.diff (7925 lines)
[STOP] [2021-06-10 11:58:07] calculate_delta
[START] [2021-06-10 11:58:07] parse_diff_and_store
[INFO] [2021-06-10 11:58:07] Handling diff: /app/public/diff/Alaskan_Arthropo_agents_4019.diff (2 lines)
[INFO] [2021-06-10 11:58:08] Loading agents diff file into memory (2 /app/public/diff/Alaskan_Arthropo_agents_4019.diff lines)...
[INFO] [2021-06-10 11:58:08] Handling diff: /app/public/diff/Alaskan_Arthropo_refs_4019.diff (445 lines)
[INFO] [2021-06-10 11:58:09] Loading refs diff file into memory (445 /app/public/diff/Alaskan_Arthropo_refs_4019.diff lines)...
[INFO] [2021-06-10 11:58:09] Handling diff: /app/public/diff/Alaskan_Arthropo_nodes_4019.diff (7684 lines)
[INFO] [2021-06-10 11:58:10] Loading nodes diff file into memory (7684 /app/public/diff/Alaskan_Arthropo_nodes_4019.diff lines)...
[INFO] [2021-06-10 11:58:13] Handling diff: /app/public/diff/Alaskan_Arthropo_media_4019.diff (2 lines)
[INFO] [2021-06-10 11:58:14] Loading media diff file into memory (2 /app/public/diff/Alaskan_Arthropo_media_4019.diff lines)...
[INFO] [2021-06-10 11:58:14] Handling diff: /app/public/diff/Alaskan_Arthropo_vernaculars_4019.diff (2 lines)
[INFO] [2021-06-10 11:58:15] Loading vernaculars diff file into memory (2 /app/public/diff/Alaskan_Arthropo_vernaculars_4019.diff lines)...
[INFO] [2021-06-10 11:58:15] Handling diff: /app/public/diff/Alaskan_Arthropo_occurrences_4019.diff (7684 lines)
[INFO] [2021-06-10 11:58:16] Loading occurrences diff file into memory (7684 /app/public/diff/Alaskan_Arthropo_occurrences_4019.diff lines)...
[INFO] [2021-06-10 11:58:17] Handling diff: /app/public/diff/Alaskan_Arthropo_assocs_4019.diff (2 lines)
[INFO] [2021-06-10 11:58:18] Loading assocs diff file into memory (2 /app/public/diff/Alaskan_Arthropo_assocs_4019.diff lines)...
[INFO] [2021-06-10 11:58:18] Handling diff: /app/public/diff/Alaskan_Arthropo_measurements_4019.diff (7925 lines)
[INFO] [2021-06-10 11:58:19] Loading measurements diff file into memory (7925 /app/public/diff/Alaskan_Arthropo_measurements_4019.diff lines)...
[INFO] [2021-06-10 11:58:22] Storing 443 References
[INFO] [2021-06-10 11:58:22] Processing group of 443 in 1 groups of 1000
[INFO] [2021-06-10 11:58:22] Average Time: 0.08
[INFO] [2021-06-10 11:58:22] Total Time: 1s
[INFO] [2021-06-10 11:58:22] Storing 11116 ScientificNames
[INFO] [2021-06-10 11:58:22] Processing group of 11116 in 12 groups of 1000
[INFO] [2021-06-10 11:58:26] Average Time: 0.262
[INFO] [2021-06-10 11:58:26] Total Time: 4s
[INFO] [2021-06-10 11:58:26] last 3 / first 3: 0.66
[INFO] [2021-06-10 11:58:26] Std.Dev: 0.07745966692414834; Max: 0.33
[INFO] [2021-06-10 11:58:26] Storing 11116 Nodes
[INFO] [2021-06-10 11:58:26] Processing group of 11116 in 12 groups of 1000
[INFO] [2021-06-10 11:58:31] Average Time: 0.403
[INFO] [2021-06-10 11:58:31] Total Time: 5s
[INFO] [2021-06-10 11:58:31] last 3 / first 3: 0.35
[INFO] [2021-06-10 11:58:31] Std.Dev: 0.3563705936241092; Max: 1.31
[INFO] [2021-06-10 11:58:31] Storing 7682 Occurrences
[INFO] [2021-06-10 11:58:31] Processing group of 7682 in 8 groups of 1000
[INFO] [2021-06-10 11:58:32] Average Time: 0.125
[INFO] [2021-06-10 11:58:32] Total Time: 2s
[INFO] [2021-06-10 11:58:32] last 3 / first 3: 1.5
[INFO] [2021-06-10 11:58:32] Std.Dev: 0.07071067811865475; Max: 0.29
[INFO] [2021-06-10 11:58:32] Storing 8258 TraitsReferences
[INFO] [2021-06-10 11:58:32] Processing group of 8258 in 9 groups of 1000
[INFO] [2021-06-10 11:58:32] Average Time: 0.061
[INFO] [2021-06-10 11:58:32] Total Time: 1s
[INFO] [2021-06-10 11:58:32] last 3 / first 3: 0.61
[INFO] [2021-06-10 11:58:32] Std.Dev: 0.03162277660168379; Max: 0.11
[INFO] [2021-06-10 11:58:32] Storing 7923 Traits
[INFO] [2021-06-10 11:58:32] Processing group of 7923 in 8 groups of 1000
[INFO] [2021-06-10 11:58:35] Average Time: 0.325
[INFO] [2021-06-10 11:58:35] Total Time: 3s
[INFO] [2021-06-10 11:58:35] last 3 / first 3: 1.0
[INFO] [2021-06-10 11:58:35] Std.Dev: 0.07071067811865475; Max: 0.46
[INFO] [2021-06-10 11:58:35] Storing 1195 MetaTraits
[INFO] [2021-06-10 11:58:35] Processing group of 1195 in 2 groups of 1000
[INFO] [2021-06-10 11:58:35] Average Time: 0.075
[INFO] [2021-06-10 11:58:35] Total Time: 1s
[STOP] [2021-06-10 11:58:35] parse_diff_and_store
[START] [2021-06-10 11:58:35] resolve_keys
[INFO] [2021-06-10 11:58:45] Occurrences to nodes (through scientific_names)...
[INFO] [2021-06-10 11:58:46] traits to occurrences...
[INFO] [2021-06-10 11:58:46] traits to nodes (through occurrences)...
[INFO] [2021-06-10 11:58:46] Traits to sex term...
[INFO] [2021-06-10 11:58:46] Traits to lifestage term...
[INFO] [2021-06-10 11:58:46] MetaTraits to traits...
[INFO] [2021-06-10 11:58:46] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-06-10 11:58:47] Assocs to occurrences...
[INFO] [2021-06-10 11:58:47] Assocs to nodes...
[INFO] [2021-06-10 11:58:47] Assoc to sex term...
[INFO] [2021-06-10 11:58:47] Assoc to lifestage term...
[INFO] [2021-06-10 11:58:47] MetaAssoc to assocs...
[STOP] [2021-06-10 11:58:47] resolve_keys
[START] [2021-06-10 11:58:47] hold_for_later_1
[STOP] [2021-06-10 11:58:47] hold_for_later_1
[START] [2021-06-10 11:58:47] hold_for_later_2
[STOP] [2021-06-10 11:58:47] hold_for_later_2
[START] [2021-06-10 11:58:47] resolve_missing_parents
[STOP] [2021-06-10 11:58:47] resolve_missing_parents
[START] [2021-06-10 11:58:47] rebuild_nodes
[START] [2021-06-10 11:58:47] Flattener#flatten
[START] [2021-06-10 11:58:47] Flattener#study_resource
[START] [2021-06-10 11:58:47] Flattener#build_ancestry
[STOP] [2021-06-10 11:58:48] Flattener#build_ancestry
[INFO] [2021-06-10 11:58:48] 11116 ancestry keys
[START] [2021-06-10 11:58:48] build_node_ancestors
[INFO] [2021-06-10 11:58:48] old ancestors deleted.
[STOP] [2021-06-10 11:58:49] build_node_ancestors
[START] [2021-06-10 11:58:53] Flattener#propagate_ancestor_ids
[STOP] [2021-06-10 11:58:54] Flattener#propagate_ancestor_ids
[STOP] [2021-06-10 11:58:54] Flattener#flatten
[STOP] [2021-06-10 11:58:54] rebuild_nodes
[START] [2021-06-10 11:58:54] resolve_missing_media_owners
[STOP] [2021-06-10 11:58:54] resolve_missing_media_owners
[START] [2021-06-10 11:58:54] sanitize_media_verbatims
[STOP] [2021-06-10 11:58:54] sanitize_media_verbatims
[START] [2021-06-10 11:58:54] queue_downloads
[STOP] [2021-06-10 11:58:54] queue_downloads
[START] [2021-06-10 11:58:54] parse_names
[WARN] [2021-06-10 11:58:54] I see 11116 names which still need to be parsed.
[WARN] [2021-06-10 11:59:03] I see 54 names which still need to be parsed.
[STOP] [2021-06-10 11:59:04] parse_names
[START] [2021-06-10 11:59:04] denormalize_canonical_names_to_nodes
[STOP] [2021-06-10 11:59:04] denormalize_canonical_names_to_nodes
[START] [2021-06-10 11:59:04] match_nodes
[START] [2021-06-10 11:59:04] map_all_nodes_to_pages
[STOP] [2021-06-10 12:00:46] map_all_nodes_to_pages
[INFO] [2021-06-10 12:00:46] 1080 Unmatched nodes (of 11116)! That's too many to output. Full list in /app/public/data/Alaskan_Arthropo/unmatched_nodes.txt ; First 10: Canonical: Acronicta lupini; Node#95932360; ResourceID: Acronicta lupini; Canonical: Acronicta vulpina; Node#95932362; ResourceID: Acronicta vulpina; Canonical: Actebia balanitis; Node#95932388; ResourceID: Actebia balanitis; Canonical: Alypia langtoni; Node#95932778; ResourceID: Alypia langtoni; Canonical: Anarta nigrolunata; Node#95932927; ResourceID: Anarta nigrolunata; Canonical: Anarta trifolii; Node#95932928; ResourceID: Anarta trifolii; Canonical: Androloma; Node#95932979; ResourceID: Androloma; Canonical: Androloma maccullochii; Node#95932980; ResourceID: Androloma maccullochii; Canonical: Apamea amputatrix; Node#95933124; ResourceID: Apamea amputatrix; Canonical: Apamea cogitata; Node#95933126; ResourceID: Apamea cogitata
[START] [2021-06-10 12:00:46] update_nodes
[STOP] [2021-06-10 12:00:51] update_nodes
[STOP] [2021-06-10 12:00:51] match_nodes
[START] [2021-06-10 12:00:51] reindex_search
[STOP] [2021-06-10 12:01:01] reindex_search
[START] [2021-06-10 12:01:01] normalize_units
[STOP] [2021-06-10 12:01:07] normalize_units
[START] [2021-06-10 12:01:07] calculate_statistics
[2021-06-10 12:01:07] (NEAR) DUPLICATE TRAITS FOUND! There are only 5875 (of 7880 total) unique traits.
[2021-06-10 12:01:22] (Near) duplicate trait pairs (up to 100):
[2021-06-10 12:01:22] (resource_pk: 1052, id: 219442395), (resource_pk: 1053, id: 219442396)
[2021-06-10 12:01:22] (resource_pk: 1060, id: 219442404), (resource_pk: 1061, id: 219442405)
[2021-06-10 12:01:22] (resource_pk: 1065, id: 219442409), (resource_pk: 1066, id: 219442410)
[2021-06-10 12:01:22] (resource_pk: 1089, id: 219442435), (resource_pk: 1090, id: 219442437)
[2021-06-10 12:01:22] (resource_pk: 1111, id: 219442461), (resource_pk: 1112, id: 219442462)
[2021-06-10 12:01:22] (resource_pk: 1117, id: 219442467), (resource_pk: 1118, id: 219442468)
[2021-06-10 12:01:22] (resource_pk: 1119, id: 219442469), (resource_pk: 1120, id: 219442471)
[2021-06-10 12:01:22] (resource_pk: 1123, id: 219442474), (resource_pk: 1124, id: 219442475)
[2021-06-10 12:01:22] (resource_pk: 1126, id: 219442477), (resource_pk: 1127, id: 219442478)
[2021-06-10 12:01:22] (resource_pk: 1129, id: 219442480), (resource_pk: 1130, id: 219442482)
[2021-06-10 12:01:22] (resource_pk: 1129, id: 219442480), (resource_pk: 1131, id: 219442483)
[2021-06-10 12:01:22] (resource_pk: 1130, id: 219442482), (resource_pk: 1131, id: 219442483)
[2021-06-10 12:01:22] (resource_pk: 1140, id: 219442493), (resource_pk: 1141, id: 219442494)
[2021-06-10 12:01:22] (resource_pk: 1142, id: 219442495), (resource_pk: 1143, id: 219442496)
[2021-06-10 12:01:22] (resource_pk: 1142, id: 219442495), (resource_pk: 1144, id: 219442497)
[2021-06-10 12:01:22] (resource_pk: 1143, id: 219442496), (resource_pk: 1144, id: 219442497)
[2021-06-10 12:01:22] (resource_pk: 1196, id: 219442554), (resource_pk: 1197, id: 219442555)
[2021-06-10 12:01:22] (resource_pk: 1200, id: 219442560), (resource_pk: 1201, id: 219442561)
[2021-06-10 12:01:22] (resource_pk: 1218, id: 219442579), (resource_pk: 1219, id: 219442580)
[2021-06-10 12:01:22] (resource_pk: 1228, id: 219442590), (resource_pk: 1229, id: 219442591)
[2021-06-10 12:01:22] (resource_pk: 1253, id: 219442618), (resource_pk: 1254, id: 219442619)
[2021-06-10 12:01:22] (resource_pk: 1272, id: 219442639), (resource_pk: 1273, id: 219442640)
[2021-06-10 12:01:22] (resource_pk: 1275, id: 219442642), (resource_pk: 1276, id: 219442643)
[2021-06-10 12:01:22] (resource_pk: 1275, id: 219442642), (resource_pk: 1277, id: 219442644)
[2021-06-10 12:01:22] (resource_pk: 1276, id: 219442643), (resource_pk: 1277, id: 219442644)
[2021-06-10 12:01:22] (resource_pk: 1320, id: 219442693), (resource_pk: 1321, id: 219442694)
[2021-06-10 12:01:22] (resource_pk: 1323, id: 219442696), (resource_pk: 1324, id: 219442697)
[2021-06-10 12:01:22] (resource_pk: 1339, id: 219442713), (resource_pk: 1340, id: 219442715)
[2021-06-10 12:01:22] (resource_pk: 1346, id: 219442721), (resource_pk: 1347, id: 219442722)
[2021-06-10 12:01:22] (resource_pk: 1351, id: 219442727), (resource_pk: 1352, id: 219442728)
[2021-06-10 12:01:22] (resource_pk: 1369, id: 219442746), (resource_pk: 1370, id: 219442748)
[2021-06-10 12:01:22] (resource_pk: 1376, id: 219442754), (resource_pk: 1377, id: 219442755)
[2021-06-10 12:01:22] (resource_pk: 1415, id: 219442798), (resource_pk: 1416, id: 219442799)
[2021-06-10 12:01:22] (resource_pk: 1463, id: 219442851), (resource_pk: 1464, id: 219442852)
[2021-06-10 12:01:22] (resource_pk: 1465, id: 219442853), (resource_pk: 1466, id: 219442854)
[2021-06-10 12:01:22] (resource_pk: 1488, id: 219442878), (resource_pk: 1489, id: 219442879)
[2021-06-10 12:01:22] (resource_pk: 1493, id: 219442884), (resource_pk: 1494, id: 219442885)
[2021-06-10 12:01:22] (resource_pk: 1503, id: 219442896), (resource_pk: 1504, id: 219442897)
[2021-06-10 12:01:22] (resource_pk: 1530, id: 219442926), (resource_pk: 1531, id: 219442927)
[2021-06-10 12:01:22] (resource_pk: 1530, id: 219442926), (resource_pk: 1532, id: 219442928)
[2021-06-10 12:01:22] (resource_pk: 1531, id: 219442927), (resource_pk: 1532, id: 219442928)
[2021-06-10 12:01:22] (resource_pk: 1550, id: 219442948), (resource_pk: 1551, id: 219442949)
[2021-06-10 12:01:22] (resource_pk: 1561, id: 219442960), (resource_pk: 1562, id: 219442961)
[2021-06-10 12:01:22] (resource_pk: 1561, id: 219442960), (resource_pk: 1563, id: 219442962)
[2021-06-10 12:01:22] (resource_pk: 1562, id: 219442961), (resource_pk: 1563, id: 219442962)
[2021-06-10 12:01:22] (resource_pk: 1577, id: 219442977), (resource_pk: 1578, id: 219442978)
[2021-06-10 12:01:22] (resource_pk: 1581, id: 219442982), (resource_pk: 1582, id: 219442983)
[2021-06-10 12:01:22] (resource_pk: 1603, id: 219443007), (resource_pk: 1604, id: 219443008)
[2021-06-10 12:01:22] (resource_pk: 1614, id: 219443019), (resource_pk: 1615, id: 219443020)
[2021-06-10 12:01:22] (resource_pk: 1632, id: 219443039), (resource_pk: 1633, id: 219443040)
[2021-06-10 12:01:22] (resource_pk: 1634, id: 219443041), (resource_pk: 1635, id: 219443042)
[2021-06-10 12:01:22] (resource_pk: 1634, id: 219443041), (resource_pk: 1636, id: 219443043)
[2021-06-10 12:01:22] (resource_pk: 1635, id: 219443042), (resource_pk: 1636, id: 219443043)
[2021-06-10 12:01:22] (resource_pk: 1650, id: 219443059), (resource_pk: 1651, id: 219443060)
[2021-06-10 12:01:22] (resource_pk: 1684, id: 219443096), (resource_pk: 1685, id: 219443097)
[2021-06-10 12:01:22] (resource_pk: 1686, id: 219443098), (resource_pk: 1687, id: 219443099)
[2021-06-10 12:01:22] (resource_pk: 1774, id: 219443196), (resource_pk: 1775, id: 219443197)
[2021-06-10 12:01:22] (resource_pk: 1776, id: 219443198), (resource_pk: 1777, id: 219443199)
[2021-06-10 12:01:22] (resource_pk: 1792, id: 219443216), (resource_pk: 1793, id: 219443217)
[2021-06-10 12:01:22] (resource_pk: 1808, id: 219443234), (resource_pk: 1809, id: 219443235)
[2021-06-10 12:01:22] (resource_pk: 1812, id: 219443239), (resource_pk: 1813, id: 219443240)
[2021-06-10 12:01:22] (resource_pk: 1812, id: 219443239), (resource_pk: 1814, id: 219443241)
[2021-06-10 12:01:22] (resource_pk: 1813, id: 219443240), (resource_pk: 1814, id: 219443241)
[2021-06-10 12:01:22] (resource_pk: 1834, id: 219443263), (resource_pk: 1841, id: 219443271)
[2021-06-10 12:01:22] (resource_pk: 1839, id: 219443268), (resource_pk: 1840, id: 219443270)
[2021-06-10 12:01:22] (resource_pk: 1884, id: 219443318), (resource_pk: 1885, id: 219443319)
[2021-06-10 12:01:22] (resource_pk: 1892, id: 219443327), (resource_pk: 1893, id: 219443328)
[2021-06-10 12:01:22] (resource_pk: 1897, id: 219443332), (resource_pk: 1898, id: 219443333)
[2021-06-10 12:01:22] (resource_pk: 1921, id: 219443360), (resource_pk: 1922, id: 219443361)
[2021-06-10 12:01:22] (resource_pk: 1943, id: 219443384), (resource_pk: 1944, id: 219443385)
[2021-06-10 12:01:22] (resource_pk: 1949, id: 219443390), (resource_pk: 1950, id: 219443392)
[2021-06-10 12:01:22] (resource_pk: 1951, id: 219443393), (resource_pk: 1952, id: 219443394)
[2021-06-10 12:01:22] (resource_pk: 1955, id: 219443397), (resource_pk: 1956, id: 219443398)
[2021-06-10 12:01:22] (resource_pk: 1958, id: 219443400), (resource_pk: 1959, id: 219443401)
[2021-06-10 12:01:22] (resource_pk: 1961, id: 219443404), (resource_pk: 1962, id: 219443405)
[2021-06-10 12:01:22] (resource_pk: 1961, id: 219443404), (resource_pk: 1963, id: 219443406)
[2021-06-10 12:01:22] (resource_pk: 1962, id: 219443405), (resource_pk: 1963, id: 219443406)
[2021-06-10 12:01:22] (resource_pk: 1974, id: 219443418), (resource_pk: 1975, id: 219443419)
[2021-06-10 12:01:22] (resource_pk: 1974, id: 219443418), (resource_pk: 1976, id: 219443420)
[2021-06-10 12:01:22] (resource_pk: 1975, id: 219443419), (resource_pk: 1976, id: 219443420)
[2021-06-10 12:01:22] (resource_pk: 2028, id: 219443479), (resource_pk: 2029, id: 219443480)
[2021-06-10 12:01:22] (resource_pk: 2032, id: 219443484), (resource_pk: 2033, id: 219443485)
[2021-06-10 12:01:22] (resource_pk: 2050, id: 219443504), (resource_pk: 2051, id: 219443505)
[2021-06-10 12:01:22] (resource_pk: 2060, id: 219443515), (resource_pk: 2061, id: 219443516)
[2021-06-10 12:01:22] (resource_pk: 2085, id: 219443542), (resource_pk: 2086, id: 219443543)
[2021-06-10 12:01:22] (resource_pk: 2104, id: 219443564), (resource_pk: 2105, id: 219443565)
[2021-06-10 12:01:22] (resource_pk: 2107, id: 219443567), (resource_pk: 2108, id: 219443568)
[2021-06-10 12:01:22] (resource_pk: 2107, id: 219443567), (resource_pk: 2109, id: 219443569)
[2021-06-10 12:01:22] (resource_pk: 2108, id: 219443568), (resource_pk: 2109, id: 219443569)
[2021-06-10 12:01:22] (resource_pk: 2152, id: 219443617), (resource_pk: 2153, id: 219443618)
[2021-06-10 12:01:22] (resource_pk: 2155, id: 219443620), (resource_pk: 2156, id: 219443621)
[2021-06-10 12:01:22] (resource_pk: 2171, id: 219443638), (resource_pk: 2172, id: 219443639)
[2021-06-10 12:01:22] (resource_pk: 2178, id: 219443645), (resource_pk: 2179, id: 219443646)
[2021-06-10 12:01:22] (resource_pk: 2183, id: 219443651), (resource_pk: 2184, id: 219443652)
[2021-06-10 12:01:22] (resource_pk: 2208, id: 219443679), (resource_pk: 2209, id: 219443680)
[2021-06-10 12:01:22] (resource_pk: 2247, id: 219443722), (resource_pk: 2248, id: 219443723)
[2021-06-10 12:01:22] (resource_pk: 2295, id: 219443775), (resource_pk: 2296, id: 219443776)
[2021-06-10 12:01:22] (resource_pk: 2297, id: 219443777), (resource_pk: 2298, id: 219443778)
[2021-06-10 12:01:22] (resource_pk: 2315, id: 219443798), (resource_pk: 2379, id: 219443868)
[2021-06-10 12:01:22] (resource_pk: 2316, id: 219443799), (resource_pk: 2317, id: 219443800)
[STOP] [2021-06-10 12:01:22] calculate_statistics
[START] [2021-06-10 12:01:22] complete_harvest_instance
[START] [2021-06-10 12:01:22] overall_tsv_creation
[INFO] [2021-06-10 12:01:22] Processing group of 11116 in 2 batches of 10000
[INFO] [2021-06-10 12:02:27] 7145 Traits (unfiltered)...
[INFO] [2021-06-10 12:03:45] 7145 Traits (filtered)...
[INFO] [2021-06-10 12:03:48] 0 Associations (filtered)...
[INFO] [2021-06-10 12:03:49] 6941 metadata added.
[INFO] [2021-06-10 12:03:49] 0 metadata added.
[INFO] [2021-06-10 12:04:57] 735 Traits (unfiltered)...
[INFO] [2021-06-10 12:05:32] 735 Traits (filtered)...
[INFO] [2021-06-10 12:05:32] 0 Associations (filtered)...
[INFO] [2021-06-10 12:05:32] 685 metadata added.
[INFO] [2021-06-10 12:05:32] 0 metadata added.
[INFO] [2021-06-10 12:06:00] Average Time: 113.83
[INFO] [2021-06-10 12:06:00] Total Time: 4m38s
[STOP] [2021-06-10 12:06:00] overall_tsv_creation
[INFO] [2021-06-10 12:06:00] Done. Check your files:
[INFO] [2021-06-10 12:06:00] (11063 lines) /app/public/data/Alaskan_Arthropo/publish_nodes.tsv
[INFO] [2021-06-10 12:06:01] (50687 lines) /app/public/data/Alaskan_Arthropo/publish_node_ancestors.tsv
[INFO] [2021-06-10 12:06:01] (11116 lines) /app/public/data/Alaskan_Arthropo/publish_scientific_names.tsv
[INFO] [2021-06-10 12:06:02] (7881 lines) /app/public/data/Alaskan_Arthropo/publish_traits.tsv
[INFO] [2021-06-10 12:06:02] (7627 lines) /app/public/data/Alaskan_Arthropo/publish_metadata.tsv
[STOP] [2021-06-10 12:06:03] complete_harvest_instance
[START] [2021-06-10 12:06:03] completed
[STOP] [2021-06-10 12:06:03] completed
[STOP] [2021-06-10 12:06:03] logged process, took 492.62

Latest Process

10 Jun 11:57 EXITED currently processing 0