Harvest for Wright Et Al 2014 Created 31 May 14:23

Stage: completed
Fetched: 31 May 14:23
Validated: 31 May 14:23
Deltas Created 31 May 14:23
Units Normalized: 31 May 14:24
Ancestry Built: 31 May 14:23
Nodes Matched: 31 May 14:24
Names Parsed: 31 May 14:23
New Models Stored: 31 May 14:23
Indexed: 31 May 14:24
Completed: 31 May 14:25
Time to Harvest: less than a minute

Harvesting Log

(215 lines)
[INFO] [2021-05-31 14:23:18] Created harvest instance #3932
[STOP] [2021-05-31 14:23:18] create_harvest_instance
[START] [2021-05-31 14:23:18] fetch_files
[STOP] [2021-05-31 14:23:18] fetch_files
[START] [2021-05-31 14:23:18] validate_each_file
[INFO] [2021-05-31 14:23:18] Looping over 8 formats...
[INFO] [2021-05-31 14:23:18] ...agents (/app/public/data/wright-et-al/agents.txt)
[INFO] [2021-05-31 14:23:18] Valid: /app/public/converted_csv/wright-et-al_agents_3932.csv (0 lines)
[INFO] [2021-05-31 14:23:18] ...refs (/app/public/data/wright-et-al/references.txt)
[INFO] [2021-05-31 14:23:18] Valid: /app/public/converted_csv/wright-et-al_refs_3932.csv (0 lines)
[INFO] [2021-05-31 14:23:18] ...nodes (/app/public/data/wright-et-al/taxa.txt)
[INFO] [2021-05-31 14:23:18] Valid: /app/public/converted_csv/wright-et-al_nodes_3932.csv (501 lines)
[INFO] [2021-05-31 14:23:18] ...media (/app/public/data/wright-et-al/media.txt)
[INFO] [2021-05-31 14:23:18] Valid: /app/public/converted_csv/wright-et-al_media_3932.csv (0 lines)
[INFO] [2021-05-31 14:23:18] ...vernaculars (/app/public/data/wright-et-al/common names.txt)
[INFO] [2021-05-31 14:23:18] Valid: /app/public/converted_csv/wright-et-al_vernaculars_3932.csv (0 lines)
[INFO] [2021-05-31 14:23:18] ...occurrences (/app/public/data/wright-et-al/occurrences.txt)
[INFO] [2021-05-31 14:23:18] Valid: /app/public/converted_csv/wright-et-al_occurrences_3932.csv (753 lines)
[INFO] [2021-05-31 14:23:18] ...assocs (/app/public/data/wright-et-al/associations.txt)
[INFO] [2021-05-31 14:23:18] Valid: /app/public/converted_csv/wright-et-al_assocs_3932.csv (0 lines)
[INFO] [2021-05-31 14:23:18] ...measurements (/app/public/data/wright-et-al/measurement_or_fact.txt)
[INFO] [2021-05-31 14:23:18] Valid: /app/public/converted_csv/wright-et-al_measurements_3932.csv (994 lines)
[STOP] [2021-05-31 14:23:18] validate_each_file
[START] [2021-05-31 14:23:18] convert_to_csv
[INFO] [2021-05-31 14:23:18] Looping over 8 formats...
[INFO] [2021-05-31 14:23:18] ...agents (/app/public/data/wright-et-al/agents.txt)
[CMD] [2021-05-31 14:23:18] /usr/bin/sort /app/public/converted_csv/wright-et-al_agents_3932.csv > /app/public/converted_csv/wright-et-al_agents_3932.csv_sorted
[INFO] [2021-05-31 14:23:19] Converted: /app/public/converted_csv/wright-et-al_agents_3932.csv (0 lines)
[INFO] [2021-05-31 14:23:19] ...refs (/app/public/data/wright-et-al/references.txt)
[CMD] [2021-05-31 14:23:19] /usr/bin/sort /app/public/converted_csv/wright-et-al_refs_3932.csv > /app/public/converted_csv/wright-et-al_refs_3932.csv_sorted
[INFO] [2021-05-31 14:23:19] Converted: /app/public/converted_csv/wright-et-al_refs_3932.csv (0 lines)
[INFO] [2021-05-31 14:23:19] ...nodes (/app/public/data/wright-et-al/taxa.txt)
[CMD] [2021-05-31 14:23:19] /usr/bin/sort /app/public/converted_csv/wright-et-al_nodes_3932.csv > /app/public/converted_csv/wright-et-al_nodes_3932.csv_sorted
[INFO] [2021-05-31 14:23:19] Converted: /app/public/converted_csv/wright-et-al_nodes_3932.csv (501 lines)
[INFO] [2021-05-31 14:23:19] ...media (/app/public/data/wright-et-al/media.txt)
[CMD] [2021-05-31 14:23:19] /usr/bin/sort /app/public/converted_csv/wright-et-al_media_3932.csv > /app/public/converted_csv/wright-et-al_media_3932.csv_sorted
[INFO] [2021-05-31 14:23:20] Converted: /app/public/converted_csv/wright-et-al_media_3932.csv (0 lines)
[INFO] [2021-05-31 14:23:20] ...vernaculars (/app/public/data/wright-et-al/common names.txt)
[CMD] [2021-05-31 14:23:20] /usr/bin/sort /app/public/converted_csv/wright-et-al_vernaculars_3932.csv > /app/public/converted_csv/wright-et-al_vernaculars_3932.csv_sorted
[INFO] [2021-05-31 14:23:20] Converted: /app/public/converted_csv/wright-et-al_vernaculars_3932.csv (0 lines)
[INFO] [2021-05-31 14:23:20] ...occurrences (/app/public/data/wright-et-al/occurrences.txt)
[CMD] [2021-05-31 14:23:20] /usr/bin/sort /app/public/converted_csv/wright-et-al_occurrences_3932.csv > /app/public/converted_csv/wright-et-al_occurrences_3932.csv_sorted
[INFO] [2021-05-31 14:23:21] Converted: /app/public/converted_csv/wright-et-al_occurrences_3932.csv (753 lines)
[INFO] [2021-05-31 14:23:21] ...assocs (/app/public/data/wright-et-al/associations.txt)
[CMD] [2021-05-31 14:23:21] /usr/bin/sort /app/public/converted_csv/wright-et-al_assocs_3932.csv > /app/public/converted_csv/wright-et-al_assocs_3932.csv_sorted
[INFO] [2021-05-31 14:23:21] Converted: /app/public/converted_csv/wright-et-al_assocs_3932.csv (0 lines)
[INFO] [2021-05-31 14:23:21] ...measurements (/app/public/data/wright-et-al/measurement_or_fact.txt)
[CMD] [2021-05-31 14:23:21] /usr/bin/sort /app/public/converted_csv/wright-et-al_measurements_3932.csv > /app/public/converted_csv/wright-et-al_measurements_3932.csv_sorted
[INFO] [2021-05-31 14:23:21] Converted: /app/public/converted_csv/wright-et-al_measurements_3932.csv (994 lines)
[STOP] [2021-05-31 14:23:21] convert_to_csv
[START] [2021-05-31 14:23:21] calculate_delta
[INFO] [2021-05-31 14:23:21] Looping over 8 formats...
[INFO] [2021-05-31 14:23:21] ...agents (/app/public/data/wright-et-al/agents.txt)
[CMD] [2021-05-31 14:23:21] echo "0a" > /app/public/diff/wright-et-al_agents_3932.diff
[CMD] [2021-05-31 14:23:22] tail -n +1 /app/public/converted_csv/wright-et-al_agents_3932.csv >> /app/public/diff/wright-et-al_agents_3932.diff
[CMD] [2021-05-31 14:23:22] echo "." >> /app/public/diff/wright-et-al_agents_3932.diff
[INFO] [2021-05-31 14:23:22] Created diff: /app/public/diff/wright-et-al_agents_3932.diff (2 lines)
[INFO] [2021-05-31 14:23:22] ...refs (/app/public/data/wright-et-al/references.txt)
[CMD] [2021-05-31 14:23:22] echo "0a" > /app/public/diff/wright-et-al_refs_3932.diff
[CMD] [2021-05-31 14:23:23] tail -n +1 /app/public/converted_csv/wright-et-al_refs_3932.csv >> /app/public/diff/wright-et-al_refs_3932.diff
[CMD] [2021-05-31 14:23:23] echo "." >> /app/public/diff/wright-et-al_refs_3932.diff
[INFO] [2021-05-31 14:23:24] Created diff: /app/public/diff/wright-et-al_refs_3932.diff (2 lines)
[INFO] [2021-05-31 14:23:24] ...nodes (/app/public/data/wright-et-al/taxa.txt)
[CMD] [2021-05-31 14:23:24] echo "0a" > /app/public/diff/wright-et-al_nodes_3932.diff
[CMD] [2021-05-31 14:23:24] tail -n +1 /app/public/converted_csv/wright-et-al_nodes_3932.csv >> /app/public/diff/wright-et-al_nodes_3932.diff
[CMD] [2021-05-31 14:23:24] echo "." >> /app/public/diff/wright-et-al_nodes_3932.diff
[INFO] [2021-05-31 14:23:25] Created diff: /app/public/diff/wright-et-al_nodes_3932.diff (503 lines)
[INFO] [2021-05-31 14:23:25] ...media (/app/public/data/wright-et-al/media.txt)
[CMD] [2021-05-31 14:23:25] echo "0a" > /app/public/diff/wright-et-al_media_3932.diff
[CMD] [2021-05-31 14:23:25] tail -n +1 /app/public/converted_csv/wright-et-al_media_3932.csv >> /app/public/diff/wright-et-al_media_3932.diff
[CMD] [2021-05-31 14:23:26] echo "." >> /app/public/diff/wright-et-al_media_3932.diff
[INFO] [2021-05-31 14:23:26] Created diff: /app/public/diff/wright-et-al_media_3932.diff (2 lines)
[INFO] [2021-05-31 14:23:26] ...vernaculars (/app/public/data/wright-et-al/common names.txt)
[CMD] [2021-05-31 14:23:26] echo "0a" > /app/public/diff/wright-et-al_vernaculars_3932.diff
[CMD] [2021-05-31 14:23:26] tail -n +1 /app/public/converted_csv/wright-et-al_vernaculars_3932.csv >> /app/public/diff/wright-et-al_vernaculars_3932.diff
[CMD] [2021-05-31 14:23:27] echo "." >> /app/public/diff/wright-et-al_vernaculars_3932.diff
[INFO] [2021-05-31 14:23:27] Created diff: /app/public/diff/wright-et-al_vernaculars_3932.diff (2 lines)
[INFO] [2021-05-31 14:23:27] ...occurrences (/app/public/data/wright-et-al/occurrences.txt)
[CMD] [2021-05-31 14:23:27] echo "0a" > /app/public/diff/wright-et-al_occurrences_3932.diff
[CMD] [2021-05-31 14:23:27] tail -n +1 /app/public/converted_csv/wright-et-al_occurrences_3932.csv >> /app/public/diff/wright-et-al_occurrences_3932.diff
[CMD] [2021-05-31 14:23:28] echo "." >> /app/public/diff/wright-et-al_occurrences_3932.diff
[INFO] [2021-05-31 14:23:28] Created diff: /app/public/diff/wright-et-al_occurrences_3932.diff (755 lines)
[INFO] [2021-05-31 14:23:28] ...assocs (/app/public/data/wright-et-al/associations.txt)
[CMD] [2021-05-31 14:23:28] echo "0a" > /app/public/diff/wright-et-al_assocs_3932.diff
[CMD] [2021-05-31 14:23:28] tail -n +1 /app/public/converted_csv/wright-et-al_assocs_3932.csv >> /app/public/diff/wright-et-al_assocs_3932.diff
[CMD] [2021-05-31 14:23:29] echo "." >> /app/public/diff/wright-et-al_assocs_3932.diff
[INFO] [2021-05-31 14:23:29] Created diff: /app/public/diff/wright-et-al_assocs_3932.diff (2 lines)
[INFO] [2021-05-31 14:23:29] ...measurements (/app/public/data/wright-et-al/measurement_or_fact.txt)
[CMD] [2021-05-31 14:23:29] echo "0a" > /app/public/diff/wright-et-al_measurements_3932.diff
[CMD] [2021-05-31 14:23:30] tail -n +1 /app/public/converted_csv/wright-et-al_measurements_3932.csv >> /app/public/diff/wright-et-al_measurements_3932.diff
[CMD] [2021-05-31 14:23:30] echo "." >> /app/public/diff/wright-et-al_measurements_3932.diff
[INFO] [2021-05-31 14:23:30] Created diff: /app/public/diff/wright-et-al_measurements_3932.diff (996 lines)
[STOP] [2021-05-31 14:23:30] calculate_delta
[START] [2021-05-31 14:23:30] parse_diff_and_store
[INFO] [2021-05-31 14:23:30] Handling diff: /app/public/diff/wright-et-al_agents_3932.diff (2 lines)
[INFO] [2021-05-31 14:23:31] Loading agents diff file into memory (2 /app/public/diff/wright-et-al_agents_3932.diff lines)...
[INFO] [2021-05-31 14:23:31] Handling diff: /app/public/diff/wright-et-al_refs_3932.diff (2 lines)
[INFO] [2021-05-31 14:23:31] Loading refs diff file into memory (2 /app/public/diff/wright-et-al_refs_3932.diff lines)...
[INFO] [2021-05-31 14:23:32] Handling diff: /app/public/diff/wright-et-al_nodes_3932.diff (503 lines)
[INFO] [2021-05-31 14:23:32] Loading nodes diff file into memory (503 /app/public/diff/wright-et-al_nodes_3932.diff lines)...
[INFO] [2021-05-31 14:23:33] Handling diff: /app/public/diff/wright-et-al_media_3932.diff (2 lines)
[INFO] [2021-05-31 14:23:33] Loading media diff file into memory (2 /app/public/diff/wright-et-al_media_3932.diff lines)...
[INFO] [2021-05-31 14:23:34] Handling diff: /app/public/diff/wright-et-al_vernaculars_3932.diff (2 lines)
[INFO] [2021-05-31 14:23:34] Loading vernaculars diff file into memory (2 /app/public/diff/wright-et-al_vernaculars_3932.diff lines)...
[INFO] [2021-05-31 14:23:34] Handling diff: /app/public/diff/wright-et-al_occurrences_3932.diff (755 lines)
[INFO] [2021-05-31 14:23:35] Loading occurrences diff file into memory (755 /app/public/diff/wright-et-al_occurrences_3932.diff lines)...
[INFO] [2021-05-31 14:23:35] Handling diff: /app/public/diff/wright-et-al_assocs_3932.diff (2 lines)
[INFO] [2021-05-31 14:23:36] Loading assocs diff file into memory (2 /app/public/diff/wright-et-al_assocs_3932.diff lines)...
[INFO] [2021-05-31 14:23:36] Handling diff: /app/public/diff/wright-et-al_measurements_3932.diff (996 lines)
[INFO] [2021-05-31 14:23:36] Loading measurements diff file into memory (996 /app/public/diff/wright-et-al_measurements_3932.diff lines)...
[INFO] [2021-05-31 14:23:38] Storing 982 ScientificNames
[INFO] [2021-05-31 14:23:38] Processing group of 982 in 1 groups of 1000
[INFO] [2021-05-31 14:23:38] Average Time: 0.32
[INFO] [2021-05-31 14:23:38] Total Time: 1s
[INFO] [2021-05-31 14:23:38] Storing 982 Nodes
[INFO] [2021-05-31 14:23:38] Processing group of 982 in 1 groups of 1000
[INFO] [2021-05-31 14:23:38] Average Time: 0.24
[INFO] [2021-05-31 14:23:38] Total Time: 1s
[INFO] [2021-05-31 14:23:38] Storing 753 Occurrences
[INFO] [2021-05-31 14:23:38] Processing group of 753 in 1 groups of 1000
[INFO] [2021-05-31 14:23:38] Average Time: 0.08
[INFO] [2021-05-31 14:23:38] Total Time: 1s
[INFO] [2021-05-31 14:23:38] Storing 915 Traits
[INFO] [2021-05-31 14:23:38] Processing group of 915 in 1 groups of 1000
[INFO] [2021-05-31 14:23:39] Average Time: 0.28
[INFO] [2021-05-31 14:23:39] Total Time: 1s
[INFO] [2021-05-31 14:23:39] Storing 1585 MetaTraits
[INFO] [2021-05-31 14:23:39] Processing group of 1585 in 2 groups of 1000
[INFO] [2021-05-31 14:23:39] Average Time: 0.095
[INFO] [2021-05-31 14:23:39] Total Time: 1s
[INFO] [2021-05-31 14:23:39] Storing 79 OccurrenceMetadata
[INFO] [2021-05-31 14:23:39] Processing group of 79 in 1 groups of 1000
[INFO] [2021-05-31 14:23:39] Average Time: 0.01
[INFO] [2021-05-31 14:23:39] Total Time: 1s
[STOP] [2021-05-31 14:23:39] parse_diff_and_store
[START] [2021-05-31 14:23:39] resolve_keys
[INFO] [2021-05-31 14:23:45] Occurrences to nodes (through scientific_names)...
[INFO] [2021-05-31 14:23:45] traits to occurrences...
[INFO] [2021-05-31 14:23:45] traits to nodes (through occurrences)...
[INFO] [2021-05-31 14:23:45] Traits to sex term...
[INFO] [2021-05-31 14:23:45] Traits to lifestage term...
[INFO] [2021-05-31 14:23:45] MetaTraits to traits...
[INFO] [2021-05-31 14:23:45] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-05-31 14:23:45] Assocs to occurrences...
[INFO] [2021-05-31 14:23:45] Assocs to nodes...
[INFO] [2021-05-31 14:23:45] Assoc to sex term...
[INFO] [2021-05-31 14:23:45] Assoc to lifestage term...
[INFO] [2021-05-31 14:23:45] MetaAssoc to assocs...
[STOP] [2021-05-31 14:23:45] resolve_keys
[START] [2021-05-31 14:23:45] hold_for_later_1
[STOP] [2021-05-31 14:23:45] hold_for_later_1
[START] [2021-05-31 14:23:45] hold_for_later_2
[STOP] [2021-05-31 14:23:45] hold_for_later_2
[START] [2021-05-31 14:23:45] resolve_missing_parents
[STOP] [2021-05-31 14:23:45] resolve_missing_parents
[START] [2021-05-31 14:23:45] rebuild_nodes
[START] [2021-05-31 14:23:45] Flattener#flatten
[START] [2021-05-31 14:23:45] Flattener#study_resource
[START] [2021-05-31 14:23:45] Flattener#build_ancestry
[STOP] [2021-05-31 14:23:45] Flattener#build_ancestry
[INFO] [2021-05-31 14:23:45] 982 ancestry keys
[START] [2021-05-31 14:23:45] build_node_ancestors
[INFO] [2021-05-31 14:23:45] old ancestors deleted.
[STOP] [2021-05-31 14:23:45] build_node_ancestors
[START] [2021-05-31 14:23:45] Flattener#propagate_ancestor_ids
[STOP] [2021-05-31 14:23:46] Flattener#propagate_ancestor_ids
[STOP] [2021-05-31 14:23:46] Flattener#flatten
[STOP] [2021-05-31 14:23:46] rebuild_nodes
[START] [2021-05-31 14:23:46] resolve_missing_media_owners
[STOP] [2021-05-31 14:23:46] resolve_missing_media_owners
[START] [2021-05-31 14:23:46] sanitize_media_verbatims
[STOP] [2021-05-31 14:23:46] sanitize_media_verbatims
[START] [2021-05-31 14:23:46] queue_downloads
[STOP] [2021-05-31 14:23:46] queue_downloads
[START] [2021-05-31 14:23:46] parse_names
[WARN] [2021-05-31 14:23:46] I see 982 names which still need to be parsed.
[WARN] [2021-05-31 14:23:47] I see 170 names which still need to be parsed.
[WARN] [2021-05-31 14:23:49] I see 78 names which still need to be parsed.
[STOP] [2021-05-31 14:23:50] parse_names
[START] [2021-05-31 14:23:50] denormalize_canonical_names_to_nodes
[STOP] [2021-05-31 14:23:50] denormalize_canonical_names_to_nodes
[START] [2021-05-31 14:23:50] match_nodes
[START] [2021-05-31 14:23:50] map_all_nodes_to_pages
[STOP] [2021-05-31 14:24:01] map_all_nodes_to_pages
[INFO] [2021-05-31 14:24:01] 32 Unmatched nodes (of 982)! That's too many to output. Full list in /app/public/data/wright-et-al/unmatched_nodes.txt ; First 10: Canonical: Euscarthmus; Node#95119144; ResourceID: Euscarthmus; Canonical: Euscarthmus meloryphus; Node#95119145; ResourceID: Euscarthmus meloryphus; Canonical: Ochthoeca frontalis; Node#95119397; ResourceID: Ochthoeca frontalis; Canonical: Ochthoeca jelskii; Node#95119399; ResourceID: Ochthoeca jelskii; Canonical: Automolus rubiginosus; Node#95118874; ResourceID: Automolus rubiginosus; Canonical: Glyphorynchus; Node#95119183; ResourceID: Glyphorynchus; Canonical: Syndactyla subularis; Node#95119635; ResourceID: Syndactyla subularis; Canonical: Parus cinerascens; Node#95119421; ResourceID: Parus cinerascens; Canonical: Bradornis; Node#95118898; ResourceID: Bradornis; Canonical: Bradornis mariquensis; Node#95118899; ResourceID: Bradornis mariquensis
[START] [2021-05-31 14:24:01] update_nodes
[STOP] [2021-05-31 14:24:01] update_nodes
[STOP] [2021-05-31 14:24:01] match_nodes
[START] [2021-05-31 14:24:02] reindex_search
[STOP] [2021-05-31 14:24:02] reindex_search
[START] [2021-05-31 14:24:02] normalize_units
[STOP] [2021-05-31 14:24:05] normalize_units
[START] [2021-05-31 14:24:05] calculate_statistics
[STOP] [2021-05-31 14:24:05] calculate_statistics
[START] [2021-05-31 14:24:05] complete_harvest_instance
[START] [2021-05-31 14:24:05] overall_tsv_creation
[INFO] [2021-05-31 14:24:05] Processing group of 982 in 1 batches of 10000
[INFO] [2021-05-31 14:24:40] 753 Traits (unfiltered)...
[INFO] [2021-05-31 14:25:07] 753 Traits (filtered)...
[INFO] [2021-05-31 14:25:07] 0 Associations (filtered)...
[INFO] [2021-05-31 14:25:07] 162 metadata added.
[INFO] [2021-05-31 14:25:07] 0 metadata added.
[INFO] [2021-05-31 14:25:30] Average Time: 63.06
[INFO] [2021-05-31 14:25:30] Total Time: 1m26s
[STOP] [2021-05-31 14:25:30] overall_tsv_creation
[INFO] [2021-05-31 14:25:30] Done. Check your files:
[INFO] [2021-05-31 14:25:31] (812 lines) /app/public/data/wright-et-al/publish_nodes.tsv
[INFO] [2021-05-31 14:25:31] (2520 lines) /app/public/data/wright-et-al/publish_node_ancestors.tsv
[INFO] [2021-05-31 14:25:31] (982 lines) /app/public/data/wright-et-al/publish_scientific_names.tsv
[INFO] [2021-05-31 14:25:32] (754 lines) /app/public/data/wright-et-al/publish_traits.tsv
[INFO] [2021-05-31 14:25:32] (163 lines) /app/public/data/wright-et-al/publish_metadata.tsv
[STOP] [2021-05-31 14:25:32] complete_harvest_instance
[START] [2021-05-31 14:25:32] completed
[STOP] [2021-05-31 14:25:32] completed
[STOP] [2021-05-31 14:25:32] logged process, took 134.54

Latest Process