Harvest for
Webb 2019
Created
19 Apr 14:47
Stage:
completed
Fetched:
19 Apr 14:47
Validated:
19 Apr 14:47
Deltas Created
19 Apr 14:47
Units Normalized:
19 Apr 14:48
Ancestry Built:
19 Apr 14:48
Nodes Matched:
19 Apr 14:48
Names Parsed:
19 Apr 14:48
New Models Stored:
19 Apr 14:48
Indexed:
19 Apr 14:48
Completed:
19 Apr 14:49
Time to Harvest:
less than a minute
Harvesting Log
(868 lines)
# Logfile created on 2020-04-03 15:06:11 -0400 by logger.rb/56815
[START] [2020-04-03 15:06:11] logged process
[START] [2020-04-03 15:06:11] create_harvest_instance
[STOP] [2020-04-03 15:06:12] create_harvest_instance
[START] [2020-04-03 15:06:12] fetch_files
[STOP] [2020-04-03 15:06:12] fetch_files
[START] [2020-04-03 15:06:12] validate_each_file
[STOP] [2020-04-03 15:06:12] validate_each_file
[START] [2020-04-03 15:06:12] convert_to_csv
[CMD] [2020-04-03 15:06:12] /usr/bin/sort /app/public/converted_csv/webb_webb_2019_nodes_20586.csv > /app/public/converted_csv/webb_webb_2019_nodes_20586.csv_sorted
[CMD] [2020-04-03 15:06:12] /usr/bin/sort /app/public/converted_csv/webb_webb_2019_occurrences_20587.csv > /app/public/converted_csv/webb_webb_2019_occurrences_20587.csv_sorted
[CMD] [2020-04-03 15:06:12] /usr/bin/sort /app/public/converted_csv/webb_webb_2019_measurements_20588.csv > /app/public/converted_csv/webb_webb_2019_measurements_20588.csv_sorted
[STOP] [2020-04-03 15:06:12] convert_to_csv
[START] [2020-04-03 15:06:12] calculate_delta
[CMD] [2020-04-03 15:06:12] echo "0a" > /app/public/diff/webb_webb_2019_nodes_20586.diff
[CMD] [2020-04-03 15:06:12] tail -n +1 /app/public/converted_csv/webb_webb_2019_nodes_20586.csv >> /app/public/diff/webb_webb_2019_nodes_20586.diff
[CMD] [2020-04-03 15:06:12] echo "." >> /app/public/diff/webb_webb_2019_nodes_20586.diff
[CMD] [2020-04-03 15:06:12] echo "0a" > /app/public/diff/webb_webb_2019_occurrences_20587.diff
[CMD] [2020-04-03 15:06:12] tail -n +1 /app/public/converted_csv/webb_webb_2019_occurrences_20587.csv >> /app/public/diff/webb_webb_2019_occurrences_20587.diff
[CMD] [2020-04-03 15:06:12] echo "." >> /app/public/diff/webb_webb_2019_occurrences_20587.diff
[CMD] [2020-04-03 15:06:12] echo "0a" > /app/public/diff/webb_webb_2019_measurements_20588.diff
[CMD] [2020-04-03 15:06:12] tail -n +1 /app/public/converted_csv/webb_webb_2019_measurements_20588.csv >> /app/public/diff/webb_webb_2019_measurements_20588.diff
[CMD] [2020-04-03 15:06:12] echo "." >> /app/public/diff/webb_webb_2019_measurements_20588.diff
[STOP] [2020-04-03 15:06:12] calculate_delta
[START] [2020-04-03 15:06:12] parse_diff_and_store
[INFO] [2020-04-03 15:06:12] Loading nodes diff file into memory (true lines)...
[INFO] [2020-04-03 15:06:12] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-04-03 15:06:12] Loading measurements diff file into memory (true lines)...
[INFO] [2020-04-03 15:06:12] Storing 4 ScientificNames
[INFO] [2020-04-03 15:06:12] Processing group of 4 in 1 groups of 1000
[INFO] [2020-04-03 15:06:12] Average Time: 0.0
[INFO] [2020-04-03 15:06:12] Total Time: 1s
[INFO] [2020-04-03 15:06:12] Storing 4 Nodes
[INFO] [2020-04-03 15:06:12] Processing group of 4 in 1 groups of 1000
[INFO] [2020-04-03 15:06:12] Average Time: 0.0
[INFO] [2020-04-03 15:06:12] Total Time: 1s
[INFO] [2020-04-03 15:06:12] Storing 4 Occurrences
[INFO] [2020-04-03 15:06:12] Processing group of 4 in 1 groups of 1000
[INFO] [2020-04-03 15:06:12] Average Time: 0.0
[INFO] [2020-04-03 15:06:12] Total Time: 1s
[INFO] [2020-04-03 15:06:12] Storing 15 Traits
[INFO] [2020-04-03 15:06:12] Processing group of 15 in 1 groups of 1000
[INFO] [2020-04-03 15:06:12] Average Time: 0.01
[INFO] [2020-04-03 15:06:12] Total Time: 1s
[INFO] [2020-04-03 15:06:12] Storing 14 MetaTraits
[INFO] [2020-04-03 15:06:12] Processing group of 14 in 1 groups of 1000
[INFO] [2020-04-03 15:06:12] Average Time: 0.0
[INFO] [2020-04-03 15:06:12] Total Time: 1s
[STOP] [2020-04-03 15:06:12] parse_diff_and_store
[START] [2020-04-03 15:06:12] resolve_keys
[INFO] [2020-04-03 15:06:17] Occurrences to nodes (through scientific_names)...
[INFO] [2020-04-03 15:06:17] traits to occurrences...
[INFO] [2020-04-03 15:06:17] traits to nodes (through occurrences)...
[INFO] [2020-04-03 15:06:17] Traits to sex term...
[INFO] [2020-04-03 15:06:17] Traits to lifestage term...
[INFO] [2020-04-03 15:06:17] MetaTraits to traits...
[INFO] [2020-04-03 15:06:17] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-04-03 15:06:17] Assocs to occurrences...
[INFO] [2020-04-03 15:06:17] Assocs to nodes...
[INFO] [2020-04-03 15:06:17] Assoc to sex term...
[INFO] [2020-04-03 15:06:17] Assoc to lifestage term...
[STOP] [2020-04-03 15:06:17] resolve_keys
[START] [2020-04-03 15:06:17] hold_for_later_1
[STOP] [2020-04-03 15:06:17] hold_for_later_1
[START] [2020-04-03 15:06:17] hold_for_later_2
[STOP] [2020-04-03 15:06:17] hold_for_later_2
[START] [2020-04-03 15:06:17] resolve_missing_parents
[STOP] [2020-04-03 15:06:17] resolve_missing_parents
[START] [2020-04-03 15:06:17] rebuild_nodes
[START] [2020-04-03 15:06:17] Flattener#flatten
[START] [2020-04-03 15:06:17] Flattener#study_resource
[START] [2020-04-03 15:06:17] Flattener#build_ancestry
[STOP] [2020-04-03 15:06:17] Flattener#build_ancestry
[INFO] [2020-04-03 15:06:17] 4 ancestry keys
[START] [2020-04-03 15:06:17] build_node_ancestors
[INFO] [2020-04-03 15:06:17] old ancestors deleted.
[STOP] [2020-04-03 15:06:17] build_node_ancestors
[WARN] [2020-04-03 15:06:17] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-04-03 15:06:17] Flattener#flatten
[STOP] [2020-04-03 15:06:17] rebuild_nodes
[START] [2020-04-03 15:06:18] resolve_missing_media_owners
[STOP] [2020-04-03 15:06:18] resolve_missing_media_owners
[START] [2020-04-03 15:06:18] sanitize_media_verbatims
[STOP] [2020-04-03 15:06:18] sanitize_media_verbatims
[START] [2020-04-03 15:06:18] queue_downloads
[STOP] [2020-04-03 15:06:18] queue_downloads
[START] [2020-04-03 15:06:18] parse_names
[WARN] [2020-04-03 15:06:18] I see 4 names which still need to be parsed.
[STOP] [2020-04-03 15:06:19] parse_names
[START] [2020-04-03 15:06:19] denormalize_canonical_names_to_nodes
[STOP] [2020-04-03 15:06:19] denormalize_canonical_names_to_nodes
[START] [2020-04-03 15:06:19] match_nodes
[START] [2020-04-03 15:06:19] map_all_nodes_to_pages
[STOP] [2020-04-03 15:06:19] map_all_nodes_to_pages
[INFO] [2020-04-03 15:06:19] ZERO unmatched nodes (of 4)! Nicely done.
[START] [2020-04-03 15:06:19] update_nodes
[STOP] [2020-04-03 15:06:19] update_nodes
[STOP] [2020-04-03 15:06:19] match_nodes
[START] [2020-04-03 15:06:19] reindex_search
[STOP] [2020-04-03 15:06:19] reindex_search
[START] [2020-04-03 15:06:19] normalize_units
[STOP] [2020-04-03 15:06:19] normalize_units
[START] [2020-04-03 15:06:19] calculate_statistics
[2020-04-03 15:06:19] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-04-03 15:06:19] calculate_statistics
[START] [2020-04-03 15:06:19] complete_harvest_instance
[START] [2020-04-03 15:06:19] overall_tsv_creation
[INFO] [2020-04-03 15:06:19] Processing group of 4 in 1 batches of 10000
[INFO] [2020-04-03 15:07:09] 4 Traits (unfiltered)...
[INFO] [2020-04-03 15:07:21] 4 Traits (filtered)...
[INFO] [2020-04-03 15:07:21] 0 Associations (filtered)...
[INFO] [2020-04-03 15:07:57] 25 metadata added.
[INFO] [2020-04-03 15:07:57] 0 metadata added.
[INFO] [2020-04-03 15:07:57] Average Time: 69.92
[INFO] [2020-04-03 15:07:57] Total Time: 1m39s
[STOP] [2020-04-03 15:07:57] overall_tsv_creation
[INFO] [2020-04-03 15:07:57] Done. Check your files:
[INFO] [2020-04-03 15:07:57] (4 lines) /app/public/data/webb_webb_2019/publish_nodes.tsv
[INFO] [2020-04-03 15:07:57] (4 lines) /app/public/data/webb_webb_2019/publish_scientific_names.tsv
[INFO] [2020-04-03 15:07:57] (5 lines) /app/public/data/webb_webb_2019/publish_traits.tsv
[INFO] [2020-04-03 15:07:57] (26 lines) /app/public/data/webb_webb_2019/publish_metadata.tsv
[STOP] [2020-04-03 15:07:57] complete_harvest_instance
[START] [2020-04-03 15:07:57] completed
[STOP] [2020-04-03 15:07:57] completed
[STOP] [2020-04-03 15:07:57] logged process, took 105.81
[INFO] [2020-11-13 11:44:05] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-11-13 11:44:10] ## remove_type: ScientificName
[INFO] [2020-11-13 11:44:10] ++ Calling delete_all on 4 instances...
[INFO] [2020-11-13 11:44:10] [11:44:10.063] Removed 4 Scientificnames
[INFO] [2020-11-13 11:44:10] ## remove_type: Vernacular
[INFO] [2020-11-13 11:44:10] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 11:44:10] [11:44:10.066] Removed 0 Vernaculars
[INFO] [2020-11-13 11:44:10] ## remove_type: Article
[INFO] [2020-11-13 11:44:10] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 11:44:10] [11:44:10.069] Removed 0 Articles
[INFO] [2020-11-13 11:44:10] ## remove_type: Medium
[INFO] [2020-11-13 11:44:10] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 11:44:10] [11:44:10.072] Removed 0 Media
[INFO] [2020-11-13 11:44:10] ## remove_type: Trait
[INFO] [2020-11-13 11:44:10] ++ Calling delete_all on 15 instances...
[INFO] [2020-11-13 11:44:10] [11:44:10.082] Removed 15 Traits
[INFO] [2020-11-13 11:44:10] ## remove_type: MetaTrait
[INFO] [2020-11-13 11:44:10] ++ Calling delete_all on 14 instances...
[INFO] [2020-11-13 11:44:10] [11:44:10.110] Removed 14 Metatraits
[INFO] [2020-11-13 11:44:10] ## remove_type: OccurrenceMetadatum
[INFO] [2020-11-13 11:44:10] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 11:44:10] [11:44:10.113] Removed 0 Occurrencemetadata
[INFO] [2020-11-13 11:44:10] ## remove_type: Assoc
[INFO] [2020-11-13 11:44:10] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 11:44:10] [11:44:10.116] Removed 0 Assocs
[INFO] [2020-11-13 11:44:10] ## remove_type: MetaAssoc
[INFO] [2020-11-13 11:44:10] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 11:44:10] [11:44:10.119] Removed 0 Metaassocs
[INFO] [2020-11-13 11:44:10] ## remove_type: Identifier
[INFO] [2020-11-13 11:44:10] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 11:44:10] [11:44:10.122] Removed 0 Identifiers
[INFO] [2020-11-13 11:44:10] ## remove_type: Reference
[INFO] [2020-11-13 11:44:10] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 11:44:10] [11:44:10.124] Removed 0 References
[INFO] [2020-11-13 11:44:10] Starting batch with ID 68185328...
[INFO] [2020-11-13 11:44:10] Starting batch with ID 68185328...
[INFO] [2020-11-13 11:44:10] Starting batch with ID 68185328...
[INFO] [2020-11-13 11:44:10] Starting batch with ID 68185328...
[INFO] [2020-11-13 11:44:10] Starting batch with ID 68185328...
[INFO] [2020-11-13 11:44:10] Starting batch with ID 68185328...
[INFO] [2020-11-13 11:44:10] Starting batch with ID 68185328...
[INFO] [2020-11-13 11:44:10] Starting batch with ID 68185328...
[INFO] [2020-11-13 11:44:10] Starting batch with ID 68185328...
[INFO] [2020-11-13 11:44:10] Starting batch with ID 68185328...
[INFO] [2020-11-13 11:44:10] Starting batch with ID 68185328...
[INFO] [2020-11-13 11:44:10] Starting batch with ID 68185328...
[INFO] [2020-11-13 11:44:10] Starting batch with ID 68185328...
[INFO] [2020-11-13 11:44:10] Starting batch with ID 68185328...
[INFO] [2020-11-13 11:44:10] Starting batch with ID 68185328...
[INFO] [2020-11-13 11:44:10] Starting batch with ID 68185328...
[INFO] [2020-11-13 11:44:10] Starting batch with ID 68185328...
[INFO] [2020-11-13 11:44:10] Starting batch with ID 68185328...
[INFO] [2020-11-13 11:44:10] Starting batch with ID 68185328...
[INFO] [2020-11-13 11:44:10] Starting batch with ID 68185328...
[INFO] [2020-11-13 11:44:10] ## remove_type: Node
[INFO] [2020-11-13 11:44:10] ++ Calling delete_all on 4 instances...
[INFO] [2020-11-13 11:44:10] [11:44:10.843] Removed 4 Nodes
[START] [2020-11-13 11:44:11] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979
[START] [2020-11-13 11:44:11] Creating resource from OpenData
[START] [2020-11-13 11:44:11] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979
[START] [2020-11-13 11:44:11] Parse meta.xml file and create formats with fields
[STOP] [2020-11-13 11:44:11] Parse meta.xml file and create formats with fields
[STOP] [2020-11-13 11:44:11] Creating resource from OpenData
[START] [2020-11-13 11:44:11] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979
[START] [2020-11-13 11:44:11] create_harvest_instance
[STOP] [2020-11-13 11:44:12] create_harvest_instance
[START] [2020-11-13 11:44:12] fetch_files
[STOP] [2020-11-13 11:44:12] fetch_files
[START] [2020-11-13 11:44:12] validate_each_file
[STOP] [2020-11-13 11:44:12] validate_each_file
[START] [2020-11-13 11:44:12] convert_to_csv
[CMD] [2020-11-13 11:44:12] /usr/bin/sort /app/public/converted_csv/webb_webb_2019_nodes_24183.csv > /app/public/converted_csv/webb_webb_2019_nodes_24183.csv_sorted
[CMD] [2020-11-13 11:44:12] /usr/bin/sort /app/public/converted_csv/webb_webb_2019_occurrences_24184.csv > /app/public/converted_csv/webb_webb_2019_occurrences_24184.csv_sorted
[CMD] [2020-11-13 11:44:12] /usr/bin/sort /app/public/converted_csv/webb_webb_2019_measurements_24185.csv > /app/public/converted_csv/webb_webb_2019_measurements_24185.csv_sorted
[STOP] [2020-11-13 11:44:12] convert_to_csv
[START] [2020-11-13 11:44:12] calculate_delta
[CMD] [2020-11-13 11:44:12] echo "0a" > /app/public/diff/webb_webb_2019_nodes_24183.diff
[CMD] [2020-11-13 11:44:12] tail -n +1 /app/public/converted_csv/webb_webb_2019_nodes_24183.csv >> /app/public/diff/webb_webb_2019_nodes_24183.diff
[CMD] [2020-11-13 11:44:12] echo "." >> /app/public/diff/webb_webb_2019_nodes_24183.diff
[CMD] [2020-11-13 11:44:12] echo "0a" > /app/public/diff/webb_webb_2019_occurrences_24184.diff
[CMD] [2020-11-13 11:44:12] tail -n +1 /app/public/converted_csv/webb_webb_2019_occurrences_24184.csv >> /app/public/diff/webb_webb_2019_occurrences_24184.diff
[CMD] [2020-11-13 11:44:13] echo "." >> /app/public/diff/webb_webb_2019_occurrences_24184.diff
[CMD] [2020-11-13 11:44:13] echo "0a" > /app/public/diff/webb_webb_2019_measurements_24185.diff
[CMD] [2020-11-13 11:44:13] tail -n +1 /app/public/converted_csv/webb_webb_2019_measurements_24185.csv >> /app/public/diff/webb_webb_2019_measurements_24185.diff
[CMD] [2020-11-13 11:44:13] echo "." >> /app/public/diff/webb_webb_2019_measurements_24185.diff
[STOP] [2020-11-13 11:44:13] calculate_delta
[START] [2020-11-13 11:44:13] parse_diff_and_store
[INFO] [2020-11-13 11:44:13] Loading nodes diff file into memory (true lines)...
[INFO] [2020-11-13 11:44:13] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-11-13 11:44:13] Loading measurements diff file into memory (true lines)...
[INFO] [2020-11-13 11:44:14] Storing 4 ScientificNames
[INFO] [2020-11-13 11:44:14] Processing group of 4 in 1 groups of 1000
[INFO] [2020-11-13 11:44:14] Average Time: 0.0
[INFO] [2020-11-13 11:44:14] Total Time: 1s
[INFO] [2020-11-13 11:44:14] Storing 4 Nodes
[INFO] [2020-11-13 11:44:14] Processing group of 4 in 1 groups of 1000
[INFO] [2020-11-13 11:44:14] Average Time: 0.0
[INFO] [2020-11-13 11:44:14] Total Time: 1s
[INFO] [2020-11-13 11:44:14] Storing 4 Occurrences
[INFO] [2020-11-13 11:44:14] Processing group of 4 in 1 groups of 1000
[INFO] [2020-11-13 11:44:14] Average Time: 0.0
[INFO] [2020-11-13 11:44:14] Total Time: 1s
[INFO] [2020-11-13 11:44:14] Storing 15 Traits
[INFO] [2020-11-13 11:44:14] Processing group of 15 in 1 groups of 1000
[INFO] [2020-11-13 11:44:14] Average Time: 0.01
[INFO] [2020-11-13 11:44:14] Total Time: 1s
[INFO] [2020-11-13 11:44:14] Storing 10 MetaTraits
[INFO] [2020-11-13 11:44:14] Processing group of 10 in 1 groups of 1000
[INFO] [2020-11-13 11:44:14] Average Time: 0.0
[INFO] [2020-11-13 11:44:14] Total Time: 1s
[STOP] [2020-11-13 11:44:14] parse_diff_and_store
[START] [2020-11-13 11:44:14] resolve_keys
[INFO] [2020-11-13 11:44:19] Occurrences to nodes (through scientific_names)...
[INFO] [2020-11-13 11:44:19] traits to occurrences...
[INFO] [2020-11-13 11:44:19] traits to nodes (through occurrences)...
[INFO] [2020-11-13 11:44:19] Traits to sex term...
[INFO] [2020-11-13 11:44:19] Traits to lifestage term...
[INFO] [2020-11-13 11:44:19] MetaTraits to traits...
[INFO] [2020-11-13 11:44:19] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-11-13 11:44:19] Assocs to occurrences...
[INFO] [2020-11-13 11:44:19] Assocs to nodes...
[INFO] [2020-11-13 11:44:19] Assoc to sex term...
[INFO] [2020-11-13 11:44:19] Assoc to lifestage term...
[INFO] [2020-11-13 11:44:19] MetaAssoc to assocs...
[STOP] [2020-11-13 11:44:19] resolve_keys
[START] [2020-11-13 11:44:19] hold_for_later_1
[STOP] [2020-11-13 11:44:19] hold_for_later_1
[START] [2020-11-13 11:44:19] hold_for_later_2
[STOP] [2020-11-13 11:44:19] hold_for_later_2
[START] [2020-11-13 11:44:19] resolve_missing_parents
[STOP] [2020-11-13 11:44:19] resolve_missing_parents
[START] [2020-11-13 11:44:19] rebuild_nodes
[START] [2020-11-13 11:44:19] Flattener#flatten
[START] [2020-11-13 11:44:19] Flattener#study_resource
[START] [2020-11-13 11:44:19] Flattener#build_ancestry
[STOP] [2020-11-13 11:44:19] Flattener#build_ancestry
[INFO] [2020-11-13 11:44:19] 4 ancestry keys
[START] [2020-11-13 11:44:19] build_node_ancestors
[INFO] [2020-11-13 11:44:19] old ancestors deleted.
[STOP] [2020-11-13 11:44:19] build_node_ancestors
[WARN] [2020-11-13 11:44:19] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-11-13 11:44:19] Flattener#flatten
[STOP] [2020-11-13 11:44:19] rebuild_nodes
[START] [2020-11-13 11:44:19] resolve_missing_media_owners
[STOP] [2020-11-13 11:44:19] resolve_missing_media_owners
[START] [2020-11-13 11:44:19] sanitize_media_verbatims
[STOP] [2020-11-13 11:44:19] sanitize_media_verbatims
[START] [2020-11-13 11:44:19] queue_downloads
[STOP] [2020-11-13 11:44:19] queue_downloads
[START] [2020-11-13 11:44:19] parse_names
[WARN] [2020-11-13 11:44:19] I see 4 names which still need to be parsed.
[STOP] [2020-11-13 11:44:21] parse_names
[START] [2020-11-13 11:44:21] denormalize_canonical_names_to_nodes
[STOP] [2020-11-13 11:44:21] denormalize_canonical_names_to_nodes
[START] [2020-11-13 11:44:21] match_nodes
[START] [2020-11-13 11:44:21] map_all_nodes_to_pages
[STOP] [2020-11-13 11:44:21] map_all_nodes_to_pages
[INFO] [2020-11-13 11:44:21] ZERO unmatched nodes (of 4)! Nicely done.
[START] [2020-11-13 11:44:21] update_nodes
[STOP] [2020-11-13 11:44:21] update_nodes
[STOP] [2020-11-13 11:44:21] match_nodes
[START] [2020-11-13 11:44:21] reindex_search
[STOP] [2020-11-13 11:44:21] reindex_search
[START] [2020-11-13 11:44:21] normalize_units
[STOP] [2020-11-13 11:44:21] normalize_units
[START] [2020-11-13 11:44:21] calculate_statistics
[2020-11-13 11:44:21] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-11-13 11:44:21] calculate_statistics
[START] [2020-11-13 11:44:21] complete_harvest_instance
[START] [2020-11-13 11:44:21] overall_tsv_creation
[INFO] [2020-11-13 11:44:21] Processing group of 4 in 1 batches of 10000
[INFO] [2020-11-13 11:45:00] 4 Traits (unfiltered)...
[INFO] [2020-11-13 11:45:39] 4 Traits (filtered)...
[INFO] [2020-11-13 11:45:39] 0 Associations (filtered)...
[INFO] [2020-11-13 11:45:39] 21 metadata added.
[INFO] [2020-11-13 11:45:39] 0 metadata added.
[INFO] [2020-11-13 11:45:39] Average Time: 53.89
[INFO] [2020-11-13 11:45:39] Total Time: 1m19s
[STOP] [2020-11-13 11:45:39] overall_tsv_creation
[INFO] [2020-11-13 11:45:39] Done. Check your files:
[INFO] [2020-11-13 11:45:39] (4 lines) /app/public/data/webb_webb_2019/publish_nodes.tsv
[INFO] [2020-11-13 11:45:39] (4 lines) /app/public/data/webb_webb_2019/publish_scientific_names.tsv
[INFO] [2020-11-13 11:45:39] (5 lines) /app/public/data/webb_webb_2019/publish_traits.tsv
[INFO] [2020-11-13 11:45:39] (18 lines) /app/public/data/webb_webb_2019/publish_metadata.tsv
[STOP] [2020-11-13 11:45:39] complete_harvest_instance
[START] [2020-11-13 11:45:39] completed
[STOP] [2020-11-13 11:45:39] completed
[STOP] [2020-11-13 11:45:39] logged process, took 88.19
[INFO] [2020-11-13 13:25:27] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-11-13 13:25:28] ## remove_type: ScientificName
[INFO] [2020-11-13 13:25:28] ++ Calling delete_all on 4 instances...
[INFO] [2020-11-13 13:25:28] [13:25:28.353] Removed 4 Scientificnames
[INFO] [2020-11-13 13:25:28] ## remove_type: Vernacular
[INFO] [2020-11-13 13:25:28] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 13:25:28] [13:25:28.356] Removed 0 Vernaculars
[INFO] [2020-11-13 13:25:28] ## remove_type: Article
[INFO] [2020-11-13 13:25:28] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 13:25:28] [13:25:28.358] Removed 0 Articles
[INFO] [2020-11-13 13:25:28] ## remove_type: Medium
[INFO] [2020-11-13 13:25:28] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 13:25:28] [13:25:28.362] Removed 0 Media
[INFO] [2020-11-13 13:25:28] ## remove_type: Trait
[INFO] [2020-11-13 13:25:28] ++ Calling delete_all on 15 instances...
[INFO] [2020-11-13 13:25:28] [13:25:28.365] Removed 15 Traits
[INFO] [2020-11-13 13:25:28] ## remove_type: MetaTrait
[INFO] [2020-11-13 13:25:28] ++ Calling delete_all on 10 instances...
[INFO] [2020-11-13 13:25:28] [13:25:28.368] Removed 10 Metatraits
[INFO] [2020-11-13 13:25:28] ## remove_type: OccurrenceMetadatum
[INFO] [2020-11-13 13:25:28] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 13:25:28] [13:25:28.371] Removed 0 Occurrencemetadata
[INFO] [2020-11-13 13:25:28] ## remove_type: Assoc
[INFO] [2020-11-13 13:25:28] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 13:25:28] [13:25:28.373] Removed 0 Assocs
[INFO] [2020-11-13 13:25:28] ## remove_type: MetaAssoc
[INFO] [2020-11-13 13:25:28] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 13:25:28] [13:25:28.376] Removed 0 Metaassocs
[INFO] [2020-11-13 13:25:28] ## remove_type: Identifier
[INFO] [2020-11-13 13:25:28] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 13:25:28] [13:25:28.378] Removed 0 Identifiers
[INFO] [2020-11-13 13:25:28] ## remove_type: Reference
[INFO] [2020-11-13 13:25:28] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 13:25:28] [13:25:28.380] Removed 0 References
[INFO] [2020-11-13 13:25:28] Starting batch with ID 81659592...
[INFO] [2020-11-13 13:25:28] Starting batch with ID 81659592...
[INFO] [2020-11-13 13:25:28] Starting batch with ID 81659592...
[INFO] [2020-11-13 13:25:28] Starting batch with ID 81659592...
[INFO] [2020-11-13 13:25:28] Starting batch with ID 81659592...
[INFO] [2020-11-13 13:25:28] Starting batch with ID 81659592...
[INFO] [2020-11-13 13:25:28] Starting batch with ID 81659592...
[INFO] [2020-11-13 13:25:28] Starting batch with ID 81659592...
[INFO] [2020-11-13 13:25:28] Starting batch with ID 81659592...
[INFO] [2020-11-13 13:25:28] Starting batch with ID 81659592...
[INFO] [2020-11-13 13:25:28] Starting batch with ID 81659592...
[INFO] [2020-11-13 13:25:28] Starting batch with ID 81659592...
[INFO] [2020-11-13 13:25:28] Starting batch with ID 81659592...
[INFO] [2020-11-13 13:25:28] Starting batch with ID 81659592...
[INFO] [2020-11-13 13:25:28] Starting batch with ID 81659592...
[INFO] [2020-11-13 13:25:28] Starting batch with ID 81659592...
[INFO] [2020-11-13 13:25:28] Starting batch with ID 81659592...
[INFO] [2020-11-13 13:25:28] Starting batch with ID 81659592...
[INFO] [2020-11-13 13:25:29] Starting batch with ID 81659592...
[INFO] [2020-11-13 13:25:29] Starting batch with ID 81659592...
[INFO] [2020-11-13 13:25:29] ## remove_type: Node
[INFO] [2020-11-13 13:25:29] ++ Calling delete_all on 4 instances...
[INFO] [2020-11-13 13:25:29] [13:25:29.140] Removed 4 Nodes
[START] [2020-11-13 13:25:29] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979
[START] [2020-11-13 13:25:29] Creating resource from OpenData
[START] [2020-11-13 13:25:29] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979
[START] [2020-11-13 13:25:29] Parse meta.xml file and create formats with fields
[STOP] [2020-11-13 13:25:29] Parse meta.xml file and create formats with fields
[STOP] [2020-11-13 13:25:29] Creating resource from OpenData
[START] [2020-11-13 13:25:29] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979
[START] [2020-11-13 13:25:29] create_harvest_instance
[STOP] [2020-11-13 13:25:30] create_harvest_instance
[START] [2020-11-13 13:25:30] fetch_files
[STOP] [2020-11-13 13:25:30] fetch_files
[START] [2020-11-13 13:25:30] validate_each_file
[STOP] [2020-11-13 13:25:31] validate_each_file
[START] [2020-11-13 13:25:31] convert_to_csv
[CMD] [2020-11-13 13:25:31] /usr/bin/sort /app/public/converted_csv/webb_webb_2019_nodes_24215.csv > /app/public/converted_csv/webb_webb_2019_nodes_24215.csv_sorted
[CMD] [2020-11-13 13:25:31] /usr/bin/sort /app/public/converted_csv/webb_webb_2019_occurrences_24216.csv > /app/public/converted_csv/webb_webb_2019_occurrences_24216.csv_sorted
[CMD] [2020-11-13 13:25:31] /usr/bin/sort /app/public/converted_csv/webb_webb_2019_measurements_24217.csv > /app/public/converted_csv/webb_webb_2019_measurements_24217.csv_sorted
[STOP] [2020-11-13 13:25:31] convert_to_csv
[START] [2020-11-13 13:25:31] calculate_delta
[CMD] [2020-11-13 13:25:31] echo "0a" > /app/public/diff/webb_webb_2019_nodes_24215.diff
[CMD] [2020-11-13 13:25:31] tail -n +1 /app/public/converted_csv/webb_webb_2019_nodes_24215.csv >> /app/public/diff/webb_webb_2019_nodes_24215.diff
[CMD] [2020-11-13 13:25:31] echo "." >> /app/public/diff/webb_webb_2019_nodes_24215.diff
[CMD] [2020-11-13 13:25:31] echo "0a" > /app/public/diff/webb_webb_2019_occurrences_24216.diff
[CMD] [2020-11-13 13:25:31] tail -n +1 /app/public/converted_csv/webb_webb_2019_occurrences_24216.csv >> /app/public/diff/webb_webb_2019_occurrences_24216.diff
[CMD] [2020-11-13 13:25:31] echo "." >> /app/public/diff/webb_webb_2019_occurrences_24216.diff
[CMD] [2020-11-13 13:25:31] echo "0a" > /app/public/diff/webb_webb_2019_measurements_24217.diff
[CMD] [2020-11-13 13:25:31] tail -n +1 /app/public/converted_csv/webb_webb_2019_measurements_24217.csv >> /app/public/diff/webb_webb_2019_measurements_24217.diff
[CMD] [2020-11-13 13:25:31] echo "." >> /app/public/diff/webb_webb_2019_measurements_24217.diff
[STOP] [2020-11-13 13:25:31] calculate_delta
[START] [2020-11-13 13:25:31] parse_diff_and_store
[INFO] [2020-11-13 13:25:31] Loading nodes diff file into memory (true lines)...
[INFO] [2020-11-13 13:25:31] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-11-13 13:25:32] Loading measurements diff file into memory (true lines)...
[INFO] [2020-11-13 13:25:32] Storing 4 ScientificNames
[INFO] [2020-11-13 13:25:32] Processing group of 4 in 1 groups of 1000
[INFO] [2020-11-13 13:25:32] Average Time: 0.0
[INFO] [2020-11-13 13:25:32] Total Time: 1s
[INFO] [2020-11-13 13:25:32] Storing 4 Nodes
[INFO] [2020-11-13 13:25:32] Processing group of 4 in 1 groups of 1000
[INFO] [2020-11-13 13:25:32] Average Time: 0.0
[INFO] [2020-11-13 13:25:32] Total Time: 1s
[INFO] [2020-11-13 13:25:32] Storing 4 Occurrences
[INFO] [2020-11-13 13:25:32] Processing group of 4 in 1 groups of 1000
[INFO] [2020-11-13 13:25:32] Average Time: 0.0
[INFO] [2020-11-13 13:25:32] Total Time: 1s
[INFO] [2020-11-13 13:25:32] Storing 4 OccurrenceMetadata
[INFO] [2020-11-13 13:25:32] Processing group of 4 in 1 groups of 1000
[INFO] [2020-11-13 13:25:32] Average Time: 0.0
[INFO] [2020-11-13 13:25:32] Total Time: 1s
[INFO] [2020-11-13 13:25:32] Storing 15 Traits
[INFO] [2020-11-13 13:25:32] Processing group of 15 in 1 groups of 1000
[INFO] [2020-11-13 13:25:32] Average Time: 0.01
[INFO] [2020-11-13 13:25:32] Total Time: 1s
[INFO] [2020-11-13 13:25:32] Storing 6 MetaTraits
[INFO] [2020-11-13 13:25:32] Processing group of 6 in 1 groups of 1000
[INFO] [2020-11-13 13:25:32] Average Time: 0.0
[INFO] [2020-11-13 13:25:32] Total Time: 1s
[STOP] [2020-11-13 13:25:32] parse_diff_and_store
[START] [2020-11-13 13:25:32] resolve_keys
[INFO] [2020-11-13 13:25:37] Occurrences to nodes (through scientific_names)...
[INFO] [2020-11-13 13:25:37] traits to occurrences...
[INFO] [2020-11-13 13:25:37] traits to nodes (through occurrences)...
[INFO] [2020-11-13 13:25:37] Traits to sex term...
[INFO] [2020-11-13 13:25:37] Traits to lifestage term...
[INFO] [2020-11-13 13:25:37] MetaTraits to traits...
[INFO] [2020-11-13 13:25:38] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-11-13 13:25:38] Assocs to occurrences...
[INFO] [2020-11-13 13:25:38] Assocs to nodes...
[INFO] [2020-11-13 13:25:38] Assoc to sex term...
[INFO] [2020-11-13 13:25:38] Assoc to lifestage term...
[INFO] [2020-11-13 13:25:38] MetaAssoc to assocs...
[STOP] [2020-11-13 13:25:38] resolve_keys
[START] [2020-11-13 13:25:38] hold_for_later_1
[STOP] [2020-11-13 13:25:38] hold_for_later_1
[START] [2020-11-13 13:25:38] hold_for_later_2
[STOP] [2020-11-13 13:25:38] hold_for_later_2
[START] [2020-11-13 13:25:38] resolve_missing_parents
[STOP] [2020-11-13 13:25:38] resolve_missing_parents
[START] [2020-11-13 13:25:38] rebuild_nodes
[START] [2020-11-13 13:25:38] Flattener#flatten
[START] [2020-11-13 13:25:38] Flattener#study_resource
[START] [2020-11-13 13:25:38] Flattener#build_ancestry
[STOP] [2020-11-13 13:25:38] Flattener#build_ancestry
[INFO] [2020-11-13 13:25:38] 4 ancestry keys
[START] [2020-11-13 13:25:38] build_node_ancestors
[INFO] [2020-11-13 13:25:38] old ancestors deleted.
[STOP] [2020-11-13 13:25:38] build_node_ancestors
[WARN] [2020-11-13 13:25:38] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-11-13 13:25:38] Flattener#flatten
[STOP] [2020-11-13 13:25:38] rebuild_nodes
[START] [2020-11-13 13:25:38] resolve_missing_media_owners
[STOP] [2020-11-13 13:25:38] resolve_missing_media_owners
[START] [2020-11-13 13:25:38] sanitize_media_verbatims
[STOP] [2020-11-13 13:25:38] sanitize_media_verbatims
[START] [2020-11-13 13:25:38] queue_downloads
[STOP] [2020-11-13 13:25:38] queue_downloads
[START] [2020-11-13 13:25:38] parse_names
[WARN] [2020-11-13 13:25:38] I see 4 names which still need to be parsed.
[STOP] [2020-11-13 13:25:39] parse_names
[START] [2020-11-13 13:25:39] denormalize_canonical_names_to_nodes
[STOP] [2020-11-13 13:25:39] denormalize_canonical_names_to_nodes
[START] [2020-11-13 13:25:39] match_nodes
[START] [2020-11-13 13:25:39] map_all_nodes_to_pages
[STOP] [2020-11-13 13:25:39] map_all_nodes_to_pages
[INFO] [2020-11-13 13:25:39] ZERO unmatched nodes (of 4)! Nicely done.
[START] [2020-11-13 13:25:39] update_nodes
[STOP] [2020-11-13 13:25:39] update_nodes
[STOP] [2020-11-13 13:25:39] match_nodes
[START] [2020-11-13 13:25:39] reindex_search
[STOP] [2020-11-13 13:25:39] reindex_search
[START] [2020-11-13 13:25:39] normalize_units
[STOP] [2020-11-13 13:25:39] normalize_units
[START] [2020-11-13 13:25:39] calculate_statistics
[2020-11-13 13:25:39] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-11-13 13:25:39] calculate_statistics
[START] [2020-11-13 13:25:39] complete_harvest_instance
[START] [2020-11-13 13:25:39] overall_tsv_creation
[INFO] [2020-11-13 13:25:39] Processing group of 4 in 1 batches of 10000
[INFO] [2020-11-13 13:26:18] 4 Traits (unfiltered)...
[INFO] [2020-11-13 13:26:56] 4 Traits (filtered)...
[INFO] [2020-11-13 13:26:56] 0 Associations (filtered)...
[INFO] [2020-11-13 13:26:56] 21 metadata added.
[INFO] [2020-11-13 13:26:56] 0 metadata added.
[INFO] [2020-11-13 13:26:56] Average Time: 52.44
[INFO] [2020-11-13 13:26:56] Total Time: 1m17s
[STOP] [2020-11-13 13:26:56] overall_tsv_creation
[INFO] [2020-11-13 13:26:56] Done. Check your files:
[INFO] [2020-11-13 13:26:56] (4 lines) /app/public/data/webb_webb_2019/publish_nodes.tsv
[INFO] [2020-11-13 13:26:56] (4 lines) /app/public/data/webb_webb_2019/publish_scientific_names.tsv
[INFO] [2020-11-13 13:26:56] (5 lines) /app/public/data/webb_webb_2019/publish_traits.tsv
[INFO] [2020-11-13 13:26:56] (18 lines) /app/public/data/webb_webb_2019/publish_metadata.tsv
[STOP] [2020-11-13 13:26:56] complete_harvest_instance
[START] [2020-11-13 13:26:56] completed
[STOP] [2020-11-13 13:26:56] completed
[STOP] [2020-11-13 13:26:56] logged process, took 86.6
[INFO] [2021-04-19 14:39:25] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-19 14:41:03] ## remove_type: ScientificName
[INFO] [2021-04-19 14:41:03] ++ Calling delete_all on 4 instances...
[INFO] [2021-04-19 14:41:03] [14:41:03.805] Removed 4 Scientificnames
[INFO] [2021-04-19 14:41:03] ## remove_type: Vernacular
[INFO] [2021-04-19 14:41:03] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 14:41:03] [14:41:03.807] Removed 0 Vernaculars
[INFO] [2021-04-19 14:41:03] ## remove_type: Article
[INFO] [2021-04-19 14:41:03] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 14:41:03] [14:41:03.808] Removed 0 Articles
[INFO] [2021-04-19 14:41:03] ## remove_type: Medium
[INFO] [2021-04-19 14:41:03] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 14:41:03] [14:41:03.809] Removed 0 Media
[INFO] [2021-04-19 14:41:03] ## remove_type: Trait
[INFO] [2021-04-19 14:41:03] ++ Calling delete_all on 15 instances...
[INFO] [2021-04-19 14:41:03] [14:41:03.812] Removed 15 Traits
[INFO] [2021-04-19 14:41:03] ## remove_type: MetaTrait
[INFO] [2021-04-19 14:41:03] ++ Calling delete_all on 6 instances...
[INFO] [2021-04-19 14:41:03] [14:41:03.813] Removed 6 Metatraits
[INFO] [2021-04-19 14:41:03] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-19 14:41:03] ++ Calling delete_all on 4 instances...
[INFO] [2021-04-19 14:41:03] [14:41:03.815] Removed 4 Occurrencemetadata
[INFO] [2021-04-19 14:41:03] ## remove_type: Assoc
[INFO] [2021-04-19 14:41:03] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 14:41:03] [14:41:03.817] Removed 0 Assocs
[INFO] [2021-04-19 14:41:03] ## remove_type: MetaAssoc
[INFO] [2021-04-19 14:41:03] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 14:41:03] [14:41:03.818] Removed 0 Metaassocs
[INFO] [2021-04-19 14:41:03] ## remove_type: Identifier
[INFO] [2021-04-19 14:41:03] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 14:41:03] [14:41:03.819] Removed 0 Identifiers
[INFO] [2021-04-19 14:41:03] ## remove_type: Reference
[INFO] [2021-04-19 14:41:03] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 14:41:03] [14:41:03.821] Removed 0 References
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:03] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:04] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:04] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:04] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:04] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:04] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:04] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:04] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:04] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:04] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:04] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:04] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:04] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:04] Starting batch with ID 81659615...
[INFO] [2021-04-19 14:41:04] ## remove_type: Node
[INFO] [2021-04-19 14:41:04] ++ Calling delete_all on 4 instances...
[INFO] [2021-04-19 14:41:04] [14:41:04.097] Removed 4 Nodes
[START] [2021-04-19 14:41:04] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 14:41:04] Creating resource from OpenData
[START] [2021-04-19 14:41:04] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 14:41:04] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 14:41:04] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 14:41:04] Creating resource from OpenData
[START] [2021-04-19 14:41:04] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 14:41:04] create_harvest_instance
[INFO] [2021-04-19 14:41:04] Created harvest instance #3746
[STOP] [2021-04-19 14:41:04] create_harvest_instance
[START] [2021-04-19 14:41:04] fetch_files
[STOP] [2021-04-19 14:41:04] fetch_files
[START] [2021-04-19 14:41:04] validate_each_file
[INFO] [2021-04-19 14:41:04] Looping over 3 formats...
[INFO] [2021-04-19 14:41:04] ...nodes (/app/public/data/webb_webb_2019/taxa.txt)
[INFO] [2021-04-19 14:41:04] Valid: /app/public/converted_csv/webb_webb_2019_nodes_3746.csv (4 lines)
[INFO] [2021-04-19 14:41:04] ...occurrences (/app/public/data/webb_webb_2019/occurrences.txt)
[INFO] [2021-04-19 14:41:04] Valid: /app/public/converted_csv/webb_webb_2019_occurrences_3746.csv (4 lines)
[INFO] [2021-04-19 14:41:04] ...measurements (/app/public/data/webb_webb_2019/measurementsorfacts.txt)
[INFO] [2021-04-19 14:41:04] Valid: /app/public/converted_csv/webb_webb_2019_measurements_3746.csv (15 lines)
[STOP] [2021-04-19 14:41:04] validate_each_file
[START] [2021-04-19 14:41:04] convert_to_csv
[INFO] [2021-04-19 14:41:04] Looping over 3 formats...
[INFO] [2021-04-19 14:41:04] ...nodes (/app/public/data/webb_webb_2019/taxa.txt)
[CMD] [2021-04-19 14:41:04] /usr/bin/sort /app/public/converted_csv/webb_webb_2019_nodes_3746.csv > /app/public/converted_csv/webb_webb_2019_nodes_3746.csv_sorted
[INFO] [2021-04-19 14:41:04] Converted: /app/public/converted_csv/webb_webb_2019_nodes_3746.csv (4 lines)
[INFO] [2021-04-19 14:41:04] ...occurrences (/app/public/data/webb_webb_2019/occurrences.txt)
[CMD] [2021-04-19 14:41:04] /usr/bin/sort /app/public/converted_csv/webb_webb_2019_occurrences_3746.csv > /app/public/converted_csv/webb_webb_2019_occurrences_3746.csv_sorted
[INFO] [2021-04-19 14:41:04] Converted: /app/public/converted_csv/webb_webb_2019_occurrences_3746.csv (4 lines)
[INFO] [2021-04-19 14:41:04] ...measurements (/app/public/data/webb_webb_2019/measurementsorfacts.txt)
[CMD] [2021-04-19 14:41:04] /usr/bin/sort /app/public/converted_csv/webb_webb_2019_measurements_3746.csv > /app/public/converted_csv/webb_webb_2019_measurements_3746.csv_sorted
[INFO] [2021-04-19 14:41:04] Converted: /app/public/converted_csv/webb_webb_2019_measurements_3746.csv (15 lines)
[STOP] [2021-04-19 14:41:04] convert_to_csv
[START] [2021-04-19 14:41:04] calculate_delta
[INFO] [2021-04-19 14:41:04] Looping over 3 formats...
[INFO] [2021-04-19 14:41:04] ...nodes (/app/public/data/webb_webb_2019/taxa.txt)
[CMD] [2021-04-19 14:41:04] echo "0a" > /app/public/diff/webb_webb_2019_nodes_3746.diff
[CMD] [2021-04-19 14:41:04] tail -n +1 /app/public/converted_csv/webb_webb_2019_nodes_3746.csv >> /app/public/diff/webb_webb_2019_nodes_3746.diff
[CMD] [2021-04-19 14:41:04] echo "." >> /app/public/diff/webb_webb_2019_nodes_3746.diff
[INFO] [2021-04-19 14:41:04] Created diff: /app/public/diff/webb_webb_2019_nodes_3746.diff (6 lines)
[INFO] [2021-04-19 14:41:04] ...occurrences (/app/public/data/webb_webb_2019/occurrences.txt)
[CMD] [2021-04-19 14:41:04] echo "0a" > /app/public/diff/webb_webb_2019_occurrences_3746.diff
[CMD] [2021-04-19 14:41:04] tail -n +1 /app/public/converted_csv/webb_webb_2019_occurrences_3746.csv >> /app/public/diff/webb_webb_2019_occurrences_3746.diff
[CMD] [2021-04-19 14:41:04] echo "." >> /app/public/diff/webb_webb_2019_occurrences_3746.diff
[INFO] [2021-04-19 14:41:04] Created diff: /app/public/diff/webb_webb_2019_occurrences_3746.diff (6 lines)
[INFO] [2021-04-19 14:41:04] ...measurements (/app/public/data/webb_webb_2019/measurementsorfacts.txt)
[CMD] [2021-04-19 14:41:04] echo "0a" > /app/public/diff/webb_webb_2019_measurements_3746.diff
[CMD] [2021-04-19 14:41:04] tail -n +1 /app/public/converted_csv/webb_webb_2019_measurements_3746.csv >> /app/public/diff/webb_webb_2019_measurements_3746.diff
[CMD] [2021-04-19 14:41:04] echo "." >> /app/public/diff/webb_webb_2019_measurements_3746.diff
[INFO] [2021-04-19 14:41:04] Created diff: /app/public/diff/webb_webb_2019_measurements_3746.diff (17 lines)
[STOP] [2021-04-19 14:41:04] calculate_delta
[START] [2021-04-19 14:41:04] parse_diff_and_store
[INFO] [2021-04-19 14:41:04] Handling diff: /app/public/diff/webb_webb_2019_nodes_3746.diff (6 lines)
[INFO] [2021-04-19 14:41:04] Loading nodes diff file into memory (6 /app/public/diff/webb_webb_2019_nodes_3746.diff lines)...
[INFO] [2021-04-19 14:41:04] Handling diff: /app/public/diff/webb_webb_2019_occurrences_3746.diff (6 lines)
[INFO] [2021-04-19 14:41:04] Loading occurrences diff file into memory (6 /app/public/diff/webb_webb_2019_occurrences_3746.diff lines)...
[INFO] [2021-04-19 14:41:04] Handling diff: /app/public/diff/webb_webb_2019_measurements_3746.diff (17 lines)
[INFO] [2021-04-19 14:41:04] Loading measurements diff file into memory (17 /app/public/diff/webb_webb_2019_measurements_3746.diff lines)...
[STOP] [2021-04-19 14:41:04] parse_diff_and_store
[ERR] [2021-04-19 14:41:04] RuntimeError
[ERR] [2021-04-19 14:41:04] Missing Term for URI `http://eol.org/schema/terms/lowMgCalcite`, must be added!
[ERR] [2021-04-19 14:41:04] ../models/store/model_builder.rb:640:in `fail_on_bad_uri'
[ERR] [2021-04-19 14:41:04] ../models/store/model_builder.rb:594:in `convert_trait_value'
[ERR] [2021-04-19 14:41:04] ../models/store/model_builder.rb:415:in `build_trait'
[ERR] [2021-04-19 14:41:04] ../models/store/model_builder.rb:28:in `build_models'
[ERR] [2021-04-19 14:41:04] ../models/resource_harvester.rb:352:in `block (3 levels) in parse_diff_and_store'
[ERR] [2021-04-19 14:41:04] ../models/csv_parser.rb:111:in `block in diff_as_hashes'
[ERR] [2021-04-19 14:41:04] ../models/csv_parser.rb:28:in `block in line_at_a_time'
[ERR] [2021-04-19 14:41:04] ../models/csv_parser.rb:25:in `line_at_a_time'
[ERR] [2021-04-19 14:41:04] ../models/csv_parser.rb:96:in `diff_as_hashes'
[ERR] [2021-04-19 14:41:04] ../models/resource_harvester.rb:306:in `block (2 levels) in parse_diff_and_store'
[ERR] [2021-04-19 14:41:04] ../models/logged_process.rb:77:in `enter_group'
[ERR] [2021-04-19 14:41:04] ../models/resource_harvester.rb:305:in `block in parse_diff_and_store'
[ERR] [2021-04-19 14:41:04] ../models/resource_harvester.rb:702:in `block in each_diff'
[ERR] [2021-04-19 14:41:04] ../models/resource_harvester.rb:689:in `each_diff'
[ERR] [2021-04-19 14:41:04] ../models/resource_harvester.rb:300:in `parse_diff_and_store'
[ERR] [2021-04-19 14:41:04] ../models/resource_harvester.rb:86:in `block (2 levels) in start'
[ERR] [2021-04-19 14:41:04] ../models/logged_process.rb:34:in `run_step'
[ERR] [2021-04-19 14:41:04] ../models/resource_harvester.rb:86:in `block in start'
[ERR] [2021-04-19 14:41:04] ../models/resource_harvester.rb:75:in `each_key'
[ERR] [2021-04-19 14:41:04] ../models/resource_harvester.rb:75:in `start'
[ERR] [2021-04-19 14:41:04] ../models/resource.rb:261:in `harvest'
[ERR] [2021-04-19 14:41:04] ../models/resource.rb:237:in `re_download_opendata_and_harvest'
[ERR] [2021-04-19 14:41:04] bin/rails:4:in `require'
[ERR] [2021-04-19 14:41:04] bin/rails:4:in `<main>'
[STOP] [2021-04-19 14:41:04] logged process, took 0.29
[INFO] [2021-04-19 14:43:49] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-19 14:47:59] ## remove_type: ScientificName
[INFO] [2021-04-19 14:47:59] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 14:47:59] [14:47:59.311] Removed 0 Scientificnames
[INFO] [2021-04-19 14:47:59] ## remove_type: Vernacular
[INFO] [2021-04-19 14:47:59] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 14:47:59] [14:47:59.312] Removed 0 Vernaculars
[INFO] [2021-04-19 14:47:59] ## remove_type: Article
[INFO] [2021-04-19 14:47:59] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 14:47:59] [14:47:59.314] Removed 0 Articles
[INFO] [2021-04-19 14:47:59] ## remove_type: Medium
[INFO] [2021-04-19 14:47:59] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 14:47:59] [14:47:59.315] Removed 0 Media
[INFO] [2021-04-19 14:47:59] ## remove_type: Trait
[INFO] [2021-04-19 14:47:59] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 14:47:59] [14:47:59.317] Removed 0 Traits
[INFO] [2021-04-19 14:47:59] ## remove_type: MetaTrait
[INFO] [2021-04-19 14:47:59] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 14:47:59] [14:47:59.318] Removed 0 Metatraits
[INFO] [2021-04-19 14:47:59] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-19 14:47:59] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 14:47:59] [14:47:59.319] Removed 0 Occurrencemetadata
[INFO] [2021-04-19 14:47:59] ## remove_type: Assoc
[INFO] [2021-04-19 14:47:59] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 14:47:59] [14:47:59.321] Removed 0 Assocs
[INFO] [2021-04-19 14:47:59] ## remove_type: MetaAssoc
[INFO] [2021-04-19 14:47:59] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 14:47:59] [14:47:59.322] Removed 0 Metaassocs
[INFO] [2021-04-19 14:47:59] ## remove_type: Identifier
[INFO] [2021-04-19 14:47:59] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 14:47:59] [14:47:59.324] Removed 0 Identifiers
[INFO] [2021-04-19 14:47:59] ## remove_type: Reference
[INFO] [2021-04-19 14:47:59] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 14:47:59] [14:47:59.325] Removed 0 References
[INFO] [2021-04-19 14:47:59] ## remove_type: Node
[INFO] [2021-04-19 14:47:59] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 14:47:59] [14:47:59.336] Removed 0 Nodes
[START] [2021-04-19 14:47:59] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 14:47:59] Creating resource from OpenData
[START] [2021-04-19 14:47:59] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 14:47:59] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 14:47:59] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 14:47:59] Creating resource from OpenData
[START] [2021-04-19 14:47:59] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 14:47:59] create_harvest_instance
[INFO] [2021-04-19 14:47:59] Created harvest instance #3751
[STOP] [2021-04-19 14:47:59] create_harvest_instance
[START] [2021-04-19 14:47:59] fetch_files
[STOP] [2021-04-19 14:47:59] fetch_files
[START] [2021-04-19 14:47:59] validate_each_file
[INFO] [2021-04-19 14:47:59] Looping over 3 formats...
[INFO] [2021-04-19 14:47:59] ...nodes (/app/public/data/webb_webb_2019/taxa.txt)
[INFO] [2021-04-19 14:47:59] Valid: /app/public/converted_csv/webb_webb_2019_nodes_3751.csv (4 lines)
[INFO] [2021-04-19 14:47:59] ...occurrences (/app/public/data/webb_webb_2019/occurrences.txt)
[INFO] [2021-04-19 14:47:59] Valid: /app/public/converted_csv/webb_webb_2019_occurrences_3751.csv (4 lines)
[INFO] [2021-04-19 14:47:59] ...measurements (/app/public/data/webb_webb_2019/measurementsorfacts.txt)
[INFO] [2021-04-19 14:47:59] Valid: /app/public/converted_csv/webb_webb_2019_measurements_3751.csv (15 lines)
[STOP] [2021-04-19 14:47:59] validate_each_file
[START] [2021-04-19 14:47:59] convert_to_csv
[INFO] [2021-04-19 14:47:59] Looping over 3 formats...
[INFO] [2021-04-19 14:47:59] ...nodes (/app/public/data/webb_webb_2019/taxa.txt)
[CMD] [2021-04-19 14:47:59] /usr/bin/sort /app/public/converted_csv/webb_webb_2019_nodes_3751.csv > /app/public/converted_csv/webb_webb_2019_nodes_3751.csv_sorted
[INFO] [2021-04-19 14:47:59] Converted: /app/public/converted_csv/webb_webb_2019_nodes_3751.csv (4 lines)
[INFO] [2021-04-19 14:47:59] ...occurrences (/app/public/data/webb_webb_2019/occurrences.txt)
[CMD] [2021-04-19 14:47:59] /usr/bin/sort /app/public/converted_csv/webb_webb_2019_occurrences_3751.csv > /app/public/converted_csv/webb_webb_2019_occurrences_3751.csv_sorted
[INFO] [2021-04-19 14:47:59] Converted: /app/public/converted_csv/webb_webb_2019_occurrences_3751.csv (4 lines)
[INFO] [2021-04-19 14:47:59] ...measurements (/app/public/data/webb_webb_2019/measurementsorfacts.txt)
[CMD] [2021-04-19 14:47:59] /usr/bin/sort /app/public/converted_csv/webb_webb_2019_measurements_3751.csv > /app/public/converted_csv/webb_webb_2019_measurements_3751.csv_sorted
[INFO] [2021-04-19 14:47:59] Converted: /app/public/converted_csv/webb_webb_2019_measurements_3751.csv (15 lines)
[STOP] [2021-04-19 14:47:59] convert_to_csv
[START] [2021-04-19 14:47:59] calculate_delta
[INFO] [2021-04-19 14:47:59] Looping over 3 formats...
[INFO] [2021-04-19 14:47:59] ...nodes (/app/public/data/webb_webb_2019/taxa.txt)
[CMD] [2021-04-19 14:47:59] echo "0a" > /app/public/diff/webb_webb_2019_nodes_3751.diff
[CMD] [2021-04-19 14:47:59] tail -n +1 /app/public/converted_csv/webb_webb_2019_nodes_3751.csv >> /app/public/diff/webb_webb_2019_nodes_3751.diff
[CMD] [2021-04-19 14:47:59] echo "." >> /app/public/diff/webb_webb_2019_nodes_3751.diff
[INFO] [2021-04-19 14:47:59] Created diff: /app/public/diff/webb_webb_2019_nodes_3751.diff (6 lines)
[INFO] [2021-04-19 14:47:59] ...occurrences (/app/public/data/webb_webb_2019/occurrences.txt)
[CMD] [2021-04-19 14:47:59] echo "0a" > /app/public/diff/webb_webb_2019_occurrences_3751.diff
[CMD] [2021-04-19 14:47:59] tail -n +1 /app/public/converted_csv/webb_webb_2019_occurrences_3751.csv >> /app/public/diff/webb_webb_2019_occurrences_3751.diff
[CMD] [2021-04-19 14:47:59] echo "." >> /app/public/diff/webb_webb_2019_occurrences_3751.diff
[INFO] [2021-04-19 14:47:59] Created diff: /app/public/diff/webb_webb_2019_occurrences_3751.diff (6 lines)
[INFO] [2021-04-19 14:47:59] ...measurements (/app/public/data/webb_webb_2019/measurementsorfacts.txt)
[CMD] [2021-04-19 14:47:59] echo "0a" > /app/public/diff/webb_webb_2019_measurements_3751.diff
[CMD] [2021-04-19 14:47:59] tail -n +1 /app/public/converted_csv/webb_webb_2019_measurements_3751.csv >> /app/public/diff/webb_webb_2019_measurements_3751.diff
[CMD] [2021-04-19 14:47:59] echo "." >> /app/public/diff/webb_webb_2019_measurements_3751.diff
[INFO] [2021-04-19 14:47:59] Created diff: /app/public/diff/webb_webb_2019_measurements_3751.diff (17 lines)
[STOP] [2021-04-19 14:47:59] calculate_delta
[START] [2021-04-19 14:47:59] parse_diff_and_store
[INFO] [2021-04-19 14:47:59] Handling diff: /app/public/diff/webb_webb_2019_nodes_3751.diff (6 lines)
[INFO] [2021-04-19 14:47:59] Loading nodes diff file into memory (6 /app/public/diff/webb_webb_2019_nodes_3751.diff lines)...
[INFO] [2021-04-19 14:47:59] Handling diff: /app/public/diff/webb_webb_2019_occurrences_3751.diff (6 lines)
[INFO] [2021-04-19 14:47:59] Loading occurrences diff file into memory (6 /app/public/diff/webb_webb_2019_occurrences_3751.diff lines)...
[INFO] [2021-04-19 14:47:59] Handling diff: /app/public/diff/webb_webb_2019_measurements_3751.diff (17 lines)
[INFO] [2021-04-19 14:47:59] Loading measurements diff file into memory (17 /app/public/diff/webb_webb_2019_measurements_3751.diff lines)...
[INFO] [2021-04-19 14:47:59] Storing 4 ScientificNames
[INFO] [2021-04-19 14:47:59] Processing group of 4 in 1 groups of 1000
[INFO] [2021-04-19 14:47:59] Average Time: 0.0
[INFO] [2021-04-19 14:47:59] Total Time: 1s
[INFO] [2021-04-19 14:47:59] Storing 4 Nodes
[INFO] [2021-04-19 14:47:59] Processing group of 4 in 1 groups of 1000
[INFO] [2021-04-19 14:47:59] Average Time: 0.0
[INFO] [2021-04-19 14:47:59] Total Time: 1s
[INFO] [2021-04-19 14:47:59] Storing 4 Occurrences
[INFO] [2021-04-19 14:47:59] Processing group of 4 in 1 groups of 1000
[INFO] [2021-04-19 14:47:59] Average Time: 0.0
[INFO] [2021-04-19 14:47:59] Total Time: 1s
[INFO] [2021-04-19 14:47:59] Storing 4 OccurrenceMetadata
[INFO] [2021-04-19 14:47:59] Processing group of 4 in 1 groups of 1000
[INFO] [2021-04-19 14:48:00] Average Time: 0.0
[INFO] [2021-04-19 14:48:00] Total Time: 1s
[INFO] [2021-04-19 14:48:00] Storing 15 Traits
[INFO] [2021-04-19 14:48:00] Processing group of 15 in 1 groups of 1000
[INFO] [2021-04-19 14:48:00] Average Time: 0.01
[INFO] [2021-04-19 14:48:00] Total Time: 1s
[INFO] [2021-04-19 14:48:00] Storing 6 MetaTraits
[INFO] [2021-04-19 14:48:00] Processing group of 6 in 1 groups of 1000
[INFO] [2021-04-19 14:48:00] Average Time: 0.0
[INFO] [2021-04-19 14:48:00] Total Time: 1s
[STOP] [2021-04-19 14:48:00] parse_diff_and_store
[START] [2021-04-19 14:48:00] resolve_keys
[INFO] [2021-04-19 14:48:05] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-19 14:48:05] traits to occurrences...
[INFO] [2021-04-19 14:48:05] traits to nodes (through occurrences)...
[INFO] [2021-04-19 14:48:05] Traits to sex term...
[INFO] [2021-04-19 14:48:05] Traits to lifestage term...
[INFO] [2021-04-19 14:48:05] MetaTraits to traits...
[INFO] [2021-04-19 14:48:05] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-19 14:48:05] Assocs to occurrences...
[INFO] [2021-04-19 14:48:05] Assocs to nodes...
[INFO] [2021-04-19 14:48:05] Assoc to sex term...
[INFO] [2021-04-19 14:48:05] Assoc to lifestage term...
[INFO] [2021-04-19 14:48:05] MetaAssoc to assocs...
[STOP] [2021-04-19 14:48:05] resolve_keys
[START] [2021-04-19 14:48:05] hold_for_later_1
[STOP] [2021-04-19 14:48:05] hold_for_later_1
[START] [2021-04-19 14:48:05] hold_for_later_2
[STOP] [2021-04-19 14:48:05] hold_for_later_2
[START] [2021-04-19 14:48:05] resolve_missing_parents
[STOP] [2021-04-19 14:48:05] resolve_missing_parents
[START] [2021-04-19 14:48:05] rebuild_nodes
[START] [2021-04-19 14:48:05] Flattener#flatten
[START] [2021-04-19 14:48:05] Flattener#study_resource
[START] [2021-04-19 14:48:05] Flattener#build_ancestry
[STOP] [2021-04-19 14:48:05] Flattener#build_ancestry
[INFO] [2021-04-19 14:48:05] 4 ancestry keys
[START] [2021-04-19 14:48:05] build_node_ancestors
[INFO] [2021-04-19 14:48:05] old ancestors deleted.
[STOP] [2021-04-19 14:48:05] build_node_ancestors
[WARN] [2021-04-19 14:48:05] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2021-04-19 14:48:05] Flattener#flatten
[STOP] [2021-04-19 14:48:05] rebuild_nodes
[START] [2021-04-19 14:48:05] resolve_missing_media_owners
[STOP] [2021-04-19 14:48:05] resolve_missing_media_owners
[START] [2021-04-19 14:48:05] sanitize_media_verbatims
[STOP] [2021-04-19 14:48:05] sanitize_media_verbatims
[START] [2021-04-19 14:48:06] queue_downloads
[STOP] [2021-04-19 14:48:06] queue_downloads
[START] [2021-04-19 14:48:06] parse_names
[WARN] [2021-04-19 14:48:06] I see 4 names which still need to be parsed.
[STOP] [2021-04-19 14:48:07] parse_names
[START] [2021-04-19 14:48:07] denormalize_canonical_names_to_nodes
[STOP] [2021-04-19 14:48:07] denormalize_canonical_names_to_nodes
[START] [2021-04-19 14:48:07] match_nodes
[START] [2021-04-19 14:48:07] map_all_nodes_to_pages
[STOP] [2021-04-19 14:48:07] map_all_nodes_to_pages
[INFO] [2021-04-19 14:48:07] ZERO unmatched nodes (of 4)! Nicely done.
[START] [2021-04-19 14:48:07] update_nodes
[STOP] [2021-04-19 14:48:07] update_nodes
[STOP] [2021-04-19 14:48:07] match_nodes
[START] [2021-04-19 14:48:07] reindex_search
[STOP] [2021-04-19 14:48:07] reindex_search
[START] [2021-04-19 14:48:07] normalize_units
[STOP] [2021-04-19 14:48:07] normalize_units
[START] [2021-04-19 14:48:07] calculate_statistics
[2021-04-19 14:48:07] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2021-04-19 14:48:07] calculate_statistics
[START] [2021-04-19 14:48:07] complete_harvest_instance
[START] [2021-04-19 14:48:07] overall_tsv_creation
[INFO] [2021-04-19 14:48:07] Processing group of 4 in 1 batches of 10000
[INFO] [2021-04-19 14:48:43] 4 Traits (unfiltered)...
[INFO] [2021-04-19 14:49:17] 4 Traits (filtered)...
[INFO] [2021-04-19 14:49:17] 0 Associations (filtered)...
[INFO] [2021-04-19 14:49:17] 11 metadata added.
[INFO] [2021-04-19 14:49:17] 0 metadata added.
[INFO] [2021-04-19 14:49:45] Average Time: 74.02
[INFO] [2021-04-19 14:49:45] Total Time: 1m38s
[STOP] [2021-04-19 14:49:45] overall_tsv_creation
[INFO] [2021-04-19 14:49:45] Done. Check your files:
[INFO] [2021-04-19 14:49:45] (4 lines) /app/public/data/webb_webb_2019/publish_nodes.tsv
[INFO] [2021-04-19 14:49:45] (4 lines) /app/public/data/webb_webb_2019/publish_scientific_names.tsv
[INFO] [2021-04-19 14:49:45] (5 lines) /app/public/data/webb_webb_2019/publish_traits.tsv
[INFO] [2021-04-19 14:49:45] (12 lines) /app/public/data/webb_webb_2019/publish_metadata.tsv
[STOP] [2021-04-19 14:49:45] complete_harvest_instance
[START] [2021-04-19 14:49:45] completed
[STOP] [2021-04-19 14:49:45] completed
[STOP] [2021-04-19 14:49:45] logged process, took 105.49
Latest Process