Harvest for Borges et al 2016 Created 06 Apr 17:54

Stage: completed
Fetched: 06 Apr 17:54
Validated: 06 Apr 17:54
Deltas Created 06 Apr 17:54
Units Normalized: 06 Apr 17:54
Ancestry Built: 06 Apr 17:54
Nodes Matched: 06 Apr 17:54
Names Parsed: 06 Apr 17:54
New Models Stored: 06 Apr 17:54
Indexed: 06 Apr 17:54
Completed: 06 Apr 17:56
Time to Harvest: less than a minute

Harvesting Log

(505 lines)
# Logfile created on 2021-02-03 09:46:18 -0500 by logger.rb/v1.4.2
[START] [2021-02-03 09:46:18] logged process: 16b22834be7ac1492cba86047bb0f5dbfa370977

[START] [2021-02-03 09:46:18] Creating resource from OpenData
[START] [2021-02-03 09:46:18] logged process: 16b22834be7ac1492cba86047bb0f5dbfa370977

[START] [2021-02-03 09:46:18] Parse meta.xml file and create formats with fields
[STOP] [2021-02-03 09:46:18] Parse meta.xml file and create formats with fields
[STOP] [2021-02-03 09:46:18] Creating resource from OpenData
[INFO] [2021-02-03 09:46:32] ## HARVEST: type = -harvest
[START] [2021-02-03 09:46:33] logged process: 16b22834be7ac1492cba86047bb0f5dbfa370977

[START] [2021-02-03 09:46:33] create_harvest_instance
[STOP] [2021-02-03 09:46:35] create_harvest_instance
[START] [2021-02-03 09:46:35] fetch_files
[STOP] [2021-02-03 09:46:35] fetch_files
[START] [2021-02-03 09:46:35] validate_each_file
[STOP] [2021-02-03 09:46:35] validate_each_file
[ERR] [2021-02-03 09:46:35] Exceptions::ColumnMissing
[ERR] [2021-02-03 09:46:35] MISSING COLUMN: nodes: taxonRank
[ERR] [2021-02-03 09:46:35] ../models/resource_harvester.rb:154:in `block in check_each_column'
[ERR] [2021-02-03 09:46:35] ../models/resource_harvester.rb:153:in `each_with_index'
[ERR] [2021-02-03 09:46:35] ../models/resource_harvester.rb:153:in `check_each_column'
[ERR] [2021-02-03 09:46:35] ../models/resource_harvester.rb:132:in `block in validate_each_file'
[ERR] [2021-02-03 09:46:35] ../models/resource_harvester.rb:684:in `block in each_format'
[ERR] [2021-02-03 09:46:35] ../models/resource_harvester.rb:669:in `each_format'
[ERR] [2021-02-03 09:46:35] ../models/resource_harvester.rb:129:in `validate_each_file'
[ERR] [2021-02-03 09:46:35] ../models/resource_harvester.rb:85:in `block (3 levels) in start'
[ERR] [2021-02-03 09:46:35] ../models/logged_process.rb:27:in `run_step'
[ERR] [2021-02-03 09:46:35] ../models/resource_harvester.rb:85:in `block (2 levels) in start'
[ERR] [2021-02-03 09:46:35] ../models/resource_harvester.rb:74:in `each_key'
[ERR] [2021-02-03 09:46:35] ../models/resource_harvester.rb:74:in `block in start'
[ERR] [2021-02-03 09:46:35] ../models/resource.rb:159:in `lock'
[ERR] [2021-02-03 09:46:35] ../models/resource_harvester.rb:72:in `start'
[ERR] [2021-02-03 09:46:35] ../models/resource.rb:245:in `harvest'
[ERR] [2021-02-03 09:46:35] bin/rails:4:in `require'
[ERR] [2021-02-03 09:46:35] bin/rails:4:in `<main>'
[STOP] [2021-02-03 09:46:35] logged process, took 2.28
[INFO] [2021-02-03 09:47:41] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-02-03 09:47:46] ## remove_type: ScientificName
[INFO] [2021-02-03 09:47:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-02-03 09:47:46] [09:47:46.277] Removed 0 Scientificnames
[INFO] [2021-02-03 09:47:46] ## remove_type: Vernacular
[INFO] [2021-02-03 09:47:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-02-03 09:47:46] [09:47:46.279] Removed 0 Vernaculars
[INFO] [2021-02-03 09:47:46] ## remove_type: Article
[INFO] [2021-02-03 09:47:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-02-03 09:47:46] [09:47:46.282] Removed 0 Articles
[INFO] [2021-02-03 09:47:46] ## remove_type: Medium
[INFO] [2021-02-03 09:47:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-02-03 09:47:46] [09:47:46.286] Removed 0 Media
[INFO] [2021-02-03 09:47:46] ## remove_type: Trait
[INFO] [2021-02-03 09:47:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-02-03 09:47:46] [09:47:46.289] Removed 0 Traits
[INFO] [2021-02-03 09:47:46] ## remove_type: MetaTrait
[INFO] [2021-02-03 09:47:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-02-03 09:47:46] [09:47:46.292] Removed 0 Metatraits
[INFO] [2021-02-03 09:47:46] ## remove_type: OccurrenceMetadatum
[INFO] [2021-02-03 09:47:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-02-03 09:47:46] [09:47:46.295] Removed 0 Occurrencemetadata
[INFO] [2021-02-03 09:47:46] ## remove_type: Assoc
[INFO] [2021-02-03 09:47:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-02-03 09:47:46] [09:47:46.298] Removed 0 Assocs
[INFO] [2021-02-03 09:47:46] ## remove_type: MetaAssoc
[INFO] [2021-02-03 09:47:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-02-03 09:47:46] [09:47:46.301] Removed 0 Metaassocs
[INFO] [2021-02-03 09:47:46] ## remove_type: Identifier
[INFO] [2021-02-03 09:47:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-02-03 09:47:46] [09:47:46.303] Removed 0 Identifiers
[INFO] [2021-02-03 09:47:46] ## remove_type: Reference
[INFO] [2021-02-03 09:47:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-02-03 09:47:46] [09:47:46.306] Removed 0 References
[INFO] [2021-02-03 09:47:46] ## remove_type: Node
[INFO] [2021-02-03 09:47:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-02-03 09:47:46] [09:47:46.327] Removed 0 Nodes
[START] [2021-02-03 09:47:46] logged process: 16b22834be7ac1492cba86047bb0f5dbfa370977

[START] [2021-02-03 09:47:46] Creating resource from OpenData
[START] [2021-02-03 09:47:46] logged process: 16b22834be7ac1492cba86047bb0f5dbfa370977

[START] [2021-02-03 09:47:46] Parse meta.xml file and create formats with fields
[STOP] [2021-02-03 09:47:47] Parse meta.xml file and create formats with fields
[STOP] [2021-02-03 09:47:47] Creating resource from OpenData
[START] [2021-02-03 09:47:47] logged process: 16b22834be7ac1492cba86047bb0f5dbfa370977

[START] [2021-02-03 09:47:47] create_harvest_instance
[STOP] [2021-02-03 09:47:48] create_harvest_instance
[START] [2021-02-03 09:47:48] fetch_files
[STOP] [2021-02-03 09:47:48] fetch_files
[START] [2021-02-03 09:47:48] validate_each_file
[STOP] [2021-02-03 09:47:49] validate_each_file
[START] [2021-02-03 09:47:49] convert_to_csv
[CMD] [2021-02-03 09:47:49] /usr/bin/sort /app/public/converted_csv/borges_et_al_bor_refs_26458.csv > /app/public/converted_csv/borges_et_al_bor_refs_26458.csv_sorted
[CMD] [2021-02-03 09:47:49] /usr/bin/sort /app/public/converted_csv/borges_et_al_bor_nodes_26459.csv > /app/public/converted_csv/borges_et_al_bor_nodes_26459.csv_sorted
[CMD] [2021-02-03 09:47:49] /usr/bin/sort /app/public/converted_csv/borges_et_al_bor_occurrences_26460.csv > /app/public/converted_csv/borges_et_al_bor_occurrences_26460.csv_sorted
[CMD] [2021-02-03 09:47:49] /usr/bin/sort /app/public/converted_csv/borges_et_al_bor_measurements_26461.csv > /app/public/converted_csv/borges_et_al_bor_measurements_26461.csv_sorted
[STOP] [2021-02-03 09:47:49] convert_to_csv
[START] [2021-02-03 09:47:49] calculate_delta
[CMD] [2021-02-03 09:47:49] echo "0a" > /app/public/diff/borges_et_al_bor_refs_26458.diff
[CMD] [2021-02-03 09:47:49] tail -n +1 /app/public/converted_csv/borges_et_al_bor_refs_26458.csv >> /app/public/diff/borges_et_al_bor_refs_26458.diff
[CMD] [2021-02-03 09:47:49] echo "." >> /app/public/diff/borges_et_al_bor_refs_26458.diff
[CMD] [2021-02-03 09:47:49] echo "0a" > /app/public/diff/borges_et_al_bor_nodes_26459.diff
[CMD] [2021-02-03 09:47:49] tail -n +1 /app/public/converted_csv/borges_et_al_bor_nodes_26459.csv >> /app/public/diff/borges_et_al_bor_nodes_26459.diff
[CMD] [2021-02-03 09:47:49] echo "." >> /app/public/diff/borges_et_al_bor_nodes_26459.diff
[CMD] [2021-02-03 09:47:49] echo "0a" > /app/public/diff/borges_et_al_bor_occurrences_26460.diff
[CMD] [2021-02-03 09:47:49] tail -n +1 /app/public/converted_csv/borges_et_al_bor_occurrences_26460.csv >> /app/public/diff/borges_et_al_bor_occurrences_26460.diff
[CMD] [2021-02-03 09:47:49] echo "." >> /app/public/diff/borges_et_al_bor_occurrences_26460.diff
[CMD] [2021-02-03 09:47:49] echo "0a" > /app/public/diff/borges_et_al_bor_measurements_26461.diff
[CMD] [2021-02-03 09:47:49] tail -n +1 /app/public/converted_csv/borges_et_al_bor_measurements_26461.csv >> /app/public/diff/borges_et_al_bor_measurements_26461.diff
[CMD] [2021-02-03 09:47:49] echo "." >> /app/public/diff/borges_et_al_bor_measurements_26461.diff
[STOP] [2021-02-03 09:47:49] calculate_delta
[START] [2021-02-03 09:47:49] parse_diff_and_store
[INFO] [2021-02-03 09:47:49] Loading refs diff file into memory (true lines)...
[INFO] [2021-02-03 09:47:49] Loading nodes diff file into memory (true lines)...
[INFO] [2021-02-03 09:47:49] Loading occurrences diff file into memory (true lines)...
[INFO] [2021-02-03 09:47:50] Loading measurements diff file into memory (true lines)...
[INFO] [2021-02-03 09:47:50] Storing 2 References
[INFO] [2021-02-03 09:47:50] Processing group of 2 in 1 groups of 1000
[INFO] [2021-02-03 09:47:50] Average Time: 0.0
[INFO] [2021-02-03 09:47:50] Total Time: 1s
[INFO] [2021-02-03 09:47:50] Storing 9 ScientificNames
[INFO] [2021-02-03 09:47:50] Processing group of 9 in 1 groups of 1000
[INFO] [2021-02-03 09:47:50] Average Time: 0.01
[INFO] [2021-02-03 09:47:50] Total Time: 1s
[INFO] [2021-02-03 09:47:50] Storing 9 Nodes
[INFO] [2021-02-03 09:47:50] Processing group of 9 in 1 groups of 1000
[INFO] [2021-02-03 09:47:50] Average Time: 0.01
[INFO] [2021-02-03 09:47:50] Total Time: 1s
[INFO] [2021-02-03 09:47:50] Storing 9 Occurrences
[INFO] [2021-02-03 09:47:50] Processing group of 9 in 1 groups of 1000
[INFO] [2021-02-03 09:47:50] Average Time: 0.0
[INFO] [2021-02-03 09:47:50] Total Time: 1s
[INFO] [2021-02-03 09:47:50] Storing 9 OccurrenceMetadata
[INFO] [2021-02-03 09:47:50] Processing group of 9 in 1 groups of 1000
[INFO] [2021-02-03 09:47:50] Average Time: 0.0
[INFO] [2021-02-03 09:47:50] Total Time: 1s
[INFO] [2021-02-03 09:47:50] Storing 9 TraitsReferences
[INFO] [2021-02-03 09:47:50] Processing group of 9 in 1 groups of 1000
[INFO] [2021-02-03 09:47:50] Average Time: 0.0
[INFO] [2021-02-03 09:47:50] Total Time: 1s
[INFO] [2021-02-03 09:47:50] Storing 10 Traits
[INFO] [2021-02-03 09:47:50] Processing group of 10 in 1 groups of 1000
[INFO] [2021-02-03 09:47:50] Average Time: 0.0
[INFO] [2021-02-03 09:47:50] Total Time: 1s
[INFO] [2021-02-03 09:47:50] Storing 9 MetaTraits
[INFO] [2021-02-03 09:47:50] Processing group of 9 in 1 groups of 1000
[INFO] [2021-02-03 09:47:50] Average Time: 0.0
[INFO] [2021-02-03 09:47:50] Total Time: 1s
[STOP] [2021-02-03 09:47:50] parse_diff_and_store
[START] [2021-02-03 09:47:50] resolve_keys
[INFO] [2021-02-03 09:47:56] Occurrences to nodes (through scientific_names)...
[INFO] [2021-02-03 09:47:56] traits to occurrences...
[INFO] [2021-02-03 09:47:56] traits to nodes (through occurrences)...
[INFO] [2021-02-03 09:47:56] Traits to sex term...
[INFO] [2021-02-03 09:47:56] Traits to lifestage term...
[INFO] [2021-02-03 09:47:56] MetaTraits to traits...
[INFO] [2021-02-03 09:47:56] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-02-03 09:47:56] Assocs to occurrences...
[INFO] [2021-02-03 09:47:56] Assocs to nodes...
[INFO] [2021-02-03 09:47:56] Assoc to sex term...
[INFO] [2021-02-03 09:47:56] Assoc to lifestage term...
[INFO] [2021-02-03 09:47:56] MetaAssoc to assocs...
[STOP] [2021-02-03 09:47:56] resolve_keys
[START] [2021-02-03 09:47:56] hold_for_later_1
[STOP] [2021-02-03 09:47:56] hold_for_later_1
[START] [2021-02-03 09:47:56] hold_for_later_2
[STOP] [2021-02-03 09:47:56] hold_for_later_2
[START] [2021-02-03 09:47:56] resolve_missing_parents
[STOP] [2021-02-03 09:47:56] resolve_missing_parents
[START] [2021-02-03 09:47:56] rebuild_nodes
[START] [2021-02-03 09:47:56] Flattener#flatten
[START] [2021-02-03 09:47:56] Flattener#study_resource
[START] [2021-02-03 09:47:56] Flattener#build_ancestry
[STOP] [2021-02-03 09:47:56] Flattener#build_ancestry
[INFO] [2021-02-03 09:47:56] 9 ancestry keys
[START] [2021-02-03 09:47:56] build_node_ancestors
[INFO] [2021-02-03 09:47:56] old ancestors deleted.
[STOP] [2021-02-03 09:47:56] build_node_ancestors
[WARN] [2021-02-03 09:47:56] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2021-02-03 09:47:56] Flattener#flatten
[STOP] [2021-02-03 09:47:56] rebuild_nodes
[START] [2021-02-03 09:47:56] resolve_missing_media_owners
[STOP] [2021-02-03 09:47:56] resolve_missing_media_owners
[START] [2021-02-03 09:47:56] sanitize_media_verbatims
[STOP] [2021-02-03 09:47:56] sanitize_media_verbatims
[START] [2021-02-03 09:47:56] queue_downloads
[STOP] [2021-02-03 09:47:56] queue_downloads
[START] [2021-02-03 09:47:56] parse_names
[WARN] [2021-02-03 09:47:56] I see 9 names which still need to be parsed.
[STOP] [2021-02-03 09:47:57] parse_names
[START] [2021-02-03 09:47:57] denormalize_canonical_names_to_nodes
[STOP] [2021-02-03 09:47:57] denormalize_canonical_names_to_nodes
[START] [2021-02-03 09:47:57] match_nodes
[START] [2021-02-03 09:47:57] map_all_nodes_to_pages
[STOP] [2021-02-03 09:47:57] map_all_nodes_to_pages
[INFO] [2021-02-03 09:47:57] ZERO unmatched nodes (of 9)! Nicely done.
[START] [2021-02-03 09:47:57] update_nodes
[STOP] [2021-02-03 09:47:57] update_nodes
[STOP] [2021-02-03 09:47:57] match_nodes
[START] [2021-02-03 09:47:57] reindex_search
[STOP] [2021-02-03 09:47:57] reindex_search
[START] [2021-02-03 09:47:57] normalize_units
[STOP] [2021-02-03 09:47:57] normalize_units
[START] [2021-02-03 09:47:57] calculate_statistics
[2021-02-03 09:47:57] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2021-02-03 09:47:57] calculate_statistics
[START] [2021-02-03 09:47:57] complete_harvest_instance
[START] [2021-02-03 09:47:57] overall_tsv_creation
[INFO] [2021-02-03 09:47:57] Processing group of 9 in 1 batches of 10000
[INFO] [2021-02-03 09:48:36] 9 Traits (unfiltered)...
[INFO] [2021-02-03 09:49:14] 9 Traits (filtered)...
[INFO] [2021-02-03 09:49:14] 0 Associations (filtered)...
[INFO] [2021-02-03 09:49:14] 19 metadata added.
[INFO] [2021-02-03 09:49:14] 0 metadata added.
[INFO] [2021-02-03 09:49:47] Average Time: 84.14
[INFO] [2021-02-03 09:49:47] Total Time: 1m50s
[STOP] [2021-02-03 09:49:47] overall_tsv_creation
[INFO] [2021-02-03 09:49:47] Done. Check your files:
[INFO] [2021-02-03 09:49:47] (9 lines) /app/public/data/borges_et_al_bor/publish_nodes.tsv
[INFO] [2021-02-03 09:49:47] (9 lines) /app/public/data/borges_et_al_bor/publish_scientific_names.tsv
[INFO] [2021-02-03 09:49:47] (10 lines) /app/public/data/borges_et_al_bor/publish_traits.tsv
[INFO] [2021-02-03 09:49:47] (20 lines) /app/public/data/borges_et_al_bor/publish_metadata.tsv
[STOP] [2021-02-03 09:49:47] complete_harvest_instance
[START] [2021-02-03 09:49:47] completed
[STOP] [2021-02-03 09:49:47] completed
[STOP] [2021-02-03 09:49:47] logged process, took 120.26
[INFO] [2021-04-06 17:52:38] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-06 17:54:23] ## remove_type: ScientificName
[INFO] [2021-04-06 17:54:23] ++ Calling delete_all on 9 instances...
[INFO] [2021-04-06 17:54:23] [17:54:23.627] Removed 9 Scientificnames
[INFO] [2021-04-06 17:54:23] ## remove_type: Vernacular
[INFO] [2021-04-06 17:54:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 17:54:23] [17:54:23.629] Removed 0 Vernaculars
[INFO] [2021-04-06 17:54:23] ## remove_type: Article
[INFO] [2021-04-06 17:54:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 17:54:23] [17:54:23.630] Removed 0 Articles
[INFO] [2021-04-06 17:54:23] ## remove_type: Medium
[INFO] [2021-04-06 17:54:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 17:54:23] [17:54:23.632] Removed 0 Media
[INFO] [2021-04-06 17:54:23] ## remove_type: Trait
[INFO] [2021-04-06 17:54:23] ++ Calling delete_all on 10 instances...
[INFO] [2021-04-06 17:54:23] [17:54:23.634] Removed 10 Traits
[INFO] [2021-04-06 17:54:23] ## remove_type: MetaTrait
[INFO] [2021-04-06 17:54:23] ++ Calling delete_all on 9 instances...
[INFO] [2021-04-06 17:54:23] [17:54:23.636] Removed 9 Metatraits
[INFO] [2021-04-06 17:54:23] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-06 17:54:23] ++ Calling delete_all on 9 instances...
[INFO] [2021-04-06 17:54:23] [17:54:23.638] Removed 9 Occurrencemetadata
[INFO] [2021-04-06 17:54:23] ## remove_type: Assoc
[INFO] [2021-04-06 17:54:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 17:54:23] [17:54:23.640] Removed 0 Assocs
[INFO] [2021-04-06 17:54:23] ## remove_type: MetaAssoc
[INFO] [2021-04-06 17:54:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 17:54:23] [17:54:23.641] Removed 0 Metaassocs
[INFO] [2021-04-06 17:54:23] ## remove_type: Identifier
[INFO] [2021-04-06 17:54:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 17:54:23] [17:54:23.642] Removed 0 Identifiers
[INFO] [2021-04-06 17:54:23] ## remove_type: Reference
[INFO] [2021-04-06 17:54:23] ++ Calling delete_all on 2 instances...
[INFO] [2021-04-06 17:54:23] [17:54:23.644] Removed 2 References
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:23] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:24] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:24] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:24] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:24] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:24] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:24] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:24] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:24] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:24] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:24] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:24] Starting batch with ID 87728212...
[INFO] [2021-04-06 17:54:24] Starting batch with ID 87728216...
[INFO] [2021-04-06 17:54:24] ## remove_type: Node
[INFO] [2021-04-06 17:54:24] ++ Calling delete_all on 9 instances...
[INFO] [2021-04-06 17:54:24] [17:54:24.090] Removed 9 Nodes
[START] [2021-04-06 17:54:24] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-06 17:54:24] Creating resource from OpenData
[START] [2021-04-06 17:54:24] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-06 17:54:24] Parse meta.xml file and create formats with fields
[STOP] [2021-04-06 17:54:24] Parse meta.xml file and create formats with fields
[STOP] [2021-04-06 17:54:24] Creating resource from OpenData
[START] [2021-04-06 17:54:24] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-06 17:54:24] create_harvest_instance
[INFO] [2021-04-06 17:54:24] Created harvest instance #3653
[STOP] [2021-04-06 17:54:24] create_harvest_instance
[START] [2021-04-06 17:54:24] fetch_files
[STOP] [2021-04-06 17:54:24] fetch_files
[START] [2021-04-06 17:54:24] validate_each_file
[INFO] [2021-04-06 17:54:24] Looping over 4 formats...
[INFO] [2021-04-06 17:54:24] ...refs (/app/public/data/borges_et_al_bor/references.txt)
[INFO] [2021-04-06 17:54:24] Valid: /app/public/converted_csv/borges_et_al_bor_refs_3653.csv (2 lines)
[INFO] [2021-04-06 17:54:24] ...nodes (/app/public/data/borges_et_al_bor/taxa.txt)
[INFO] [2021-04-06 17:54:24] Valid: /app/public/converted_csv/borges_et_al_bor_nodes_3653.csv (9 lines)
[INFO] [2021-04-06 17:54:24] ...occurrences (/app/public/data/borges_et_al_bor/occurrences.txt)
[INFO] [2021-04-06 17:54:24] Valid: /app/public/converted_csv/borges_et_al_bor_occurrences_3653.csv (9 lines)
[INFO] [2021-04-06 17:54:24] ...measurements (/app/public/data/borges_et_al_bor/measurementsorfacts.txt)
[INFO] [2021-04-06 17:54:24] Valid: /app/public/converted_csv/borges_et_al_bor_measurements_3653.csv (10 lines)
[STOP] [2021-04-06 17:54:24] validate_each_file
[START] [2021-04-06 17:54:24] convert_to_csv
[INFO] [2021-04-06 17:54:24] Looping over 4 formats...
[INFO] [2021-04-06 17:54:24] ...refs (/app/public/data/borges_et_al_bor/references.txt)
[CMD] [2021-04-06 17:54:24] /usr/bin/sort /app/public/converted_csv/borges_et_al_bor_refs_3653.csv > /app/public/converted_csv/borges_et_al_bor_refs_3653.csv_sorted
[INFO] [2021-04-06 17:54:24] Converted: /app/public/converted_csv/borges_et_al_bor_refs_3653.csv (2 lines)
[INFO] [2021-04-06 17:54:24] ...nodes (/app/public/data/borges_et_al_bor/taxa.txt)
[CMD] [2021-04-06 17:54:24] /usr/bin/sort /app/public/converted_csv/borges_et_al_bor_nodes_3653.csv > /app/public/converted_csv/borges_et_al_bor_nodes_3653.csv_sorted
[INFO] [2021-04-06 17:54:24] Converted: /app/public/converted_csv/borges_et_al_bor_nodes_3653.csv (9 lines)
[INFO] [2021-04-06 17:54:24] ...occurrences (/app/public/data/borges_et_al_bor/occurrences.txt)
[CMD] [2021-04-06 17:54:24] /usr/bin/sort /app/public/converted_csv/borges_et_al_bor_occurrences_3653.csv > /app/public/converted_csv/borges_et_al_bor_occurrences_3653.csv_sorted
[INFO] [2021-04-06 17:54:24] Converted: /app/public/converted_csv/borges_et_al_bor_occurrences_3653.csv (9 lines)
[INFO] [2021-04-06 17:54:24] ...measurements (/app/public/data/borges_et_al_bor/measurementsorfacts.txt)
[CMD] [2021-04-06 17:54:24] /usr/bin/sort /app/public/converted_csv/borges_et_al_bor_measurements_3653.csv > /app/public/converted_csv/borges_et_al_bor_measurements_3653.csv_sorted
[INFO] [2021-04-06 17:54:24] Converted: /app/public/converted_csv/borges_et_al_bor_measurements_3653.csv (10 lines)
[STOP] [2021-04-06 17:54:24] convert_to_csv
[START] [2021-04-06 17:54:24] calculate_delta
[INFO] [2021-04-06 17:54:24] Looping over 4 formats...
[INFO] [2021-04-06 17:54:24] ...refs (/app/public/data/borges_et_al_bor/references.txt)
[CMD] [2021-04-06 17:54:24] echo "0a" > /app/public/diff/borges_et_al_bor_refs_3653.diff
[CMD] [2021-04-06 17:54:24] tail -n +1 /app/public/converted_csv/borges_et_al_bor_refs_3653.csv >> /app/public/diff/borges_et_al_bor_refs_3653.diff
[CMD] [2021-04-06 17:54:24] echo "." >> /app/public/diff/borges_et_al_bor_refs_3653.diff
[INFO] [2021-04-06 17:54:24] Created diff: /app/public/diff/borges_et_al_bor_refs_3653.diff (4 lines)
[INFO] [2021-04-06 17:54:24] ...nodes (/app/public/data/borges_et_al_bor/taxa.txt)
[CMD] [2021-04-06 17:54:24] echo "0a" > /app/public/diff/borges_et_al_bor_nodes_3653.diff
[CMD] [2021-04-06 17:54:24] tail -n +1 /app/public/converted_csv/borges_et_al_bor_nodes_3653.csv >> /app/public/diff/borges_et_al_bor_nodes_3653.diff
[CMD] [2021-04-06 17:54:24] echo "." >> /app/public/diff/borges_et_al_bor_nodes_3653.diff
[INFO] [2021-04-06 17:54:24] Created diff: /app/public/diff/borges_et_al_bor_nodes_3653.diff (11 lines)
[INFO] [2021-04-06 17:54:24] ...occurrences (/app/public/data/borges_et_al_bor/occurrences.txt)
[CMD] [2021-04-06 17:54:24] echo "0a" > /app/public/diff/borges_et_al_bor_occurrences_3653.diff
[CMD] [2021-04-06 17:54:24] tail -n +1 /app/public/converted_csv/borges_et_al_bor_occurrences_3653.csv >> /app/public/diff/borges_et_al_bor_occurrences_3653.diff
[CMD] [2021-04-06 17:54:24] echo "." >> /app/public/diff/borges_et_al_bor_occurrences_3653.diff
[INFO] [2021-04-06 17:54:24] Created diff: /app/public/diff/borges_et_al_bor_occurrences_3653.diff (11 lines)
[INFO] [2021-04-06 17:54:24] ...measurements (/app/public/data/borges_et_al_bor/measurementsorfacts.txt)
[CMD] [2021-04-06 17:54:24] echo "0a" > /app/public/diff/borges_et_al_bor_measurements_3653.diff
[CMD] [2021-04-06 17:54:24] tail -n +1 /app/public/converted_csv/borges_et_al_bor_measurements_3653.csv >> /app/public/diff/borges_et_al_bor_measurements_3653.diff
[CMD] [2021-04-06 17:54:24] echo "." >> /app/public/diff/borges_et_al_bor_measurements_3653.diff
[INFO] [2021-04-06 17:54:24] Created diff: /app/public/diff/borges_et_al_bor_measurements_3653.diff (12 lines)
[STOP] [2021-04-06 17:54:24] calculate_delta
[START] [2021-04-06 17:54:24] parse_diff_and_store
[INFO] [2021-04-06 17:54:24] Handling diff: /app/public/diff/borges_et_al_bor_refs_3653.diff (4 lines)
[INFO] [2021-04-06 17:54:24] Loading refs diff file into memory (4 /app/public/diff/borges_et_al_bor_refs_3653.diff lines)...
[INFO] [2021-04-06 17:54:24] Handling diff: /app/public/diff/borges_et_al_bor_nodes_3653.diff (11 lines)
[INFO] [2021-04-06 17:54:24] Loading nodes diff file into memory (11 /app/public/diff/borges_et_al_bor_nodes_3653.diff lines)...
[INFO] [2021-04-06 17:54:24] Handling diff: /app/public/diff/borges_et_al_bor_occurrences_3653.diff (11 lines)
[INFO] [2021-04-06 17:54:24] Loading occurrences diff file into memory (11 /app/public/diff/borges_et_al_bor_occurrences_3653.diff lines)...
[INFO] [2021-04-06 17:54:24] Handling diff: /app/public/diff/borges_et_al_bor_measurements_3653.diff (12 lines)
[INFO] [2021-04-06 17:54:24] Loading measurements diff file into memory (12 /app/public/diff/borges_et_al_bor_measurements_3653.diff lines)...
[INFO] [2021-04-06 17:54:24] Storing 2 References
[INFO] [2021-04-06 17:54:24] Processing group of 2 in 1 groups of 1000
[INFO] [2021-04-06 17:54:24] Average Time: 0.0
[INFO] [2021-04-06 17:54:24] Total Time: 1s
[INFO] [2021-04-06 17:54:24] Storing 9 ScientificNames
[INFO] [2021-04-06 17:54:24] Processing group of 9 in 1 groups of 1000
[INFO] [2021-04-06 17:54:24] Average Time: 0.0
[INFO] [2021-04-06 17:54:24] Total Time: 1s
[INFO] [2021-04-06 17:54:24] Storing 9 Nodes
[INFO] [2021-04-06 17:54:24] Processing group of 9 in 1 groups of 1000
[INFO] [2021-04-06 17:54:24] Average Time: 0.0
[INFO] [2021-04-06 17:54:24] Total Time: 1s
[INFO] [2021-04-06 17:54:24] Storing 9 Occurrences
[INFO] [2021-04-06 17:54:24] Processing group of 9 in 1 groups of 1000
[INFO] [2021-04-06 17:54:25] Average Time: 0.0
[INFO] [2021-04-06 17:54:25] Total Time: 1s
[INFO] [2021-04-06 17:54:25] Storing 9 OccurrenceMetadata
[INFO] [2021-04-06 17:54:25] Processing group of 9 in 1 groups of 1000
[INFO] [2021-04-06 17:54:25] Average Time: 0.0
[INFO] [2021-04-06 17:54:25] Total Time: 1s
[INFO] [2021-04-06 17:54:25] Storing 9 TraitsReferences
[INFO] [2021-04-06 17:54:25] Processing group of 9 in 1 groups of 1000
[INFO] [2021-04-06 17:54:25] Average Time: 0.0
[INFO] [2021-04-06 17:54:25] Total Time: 1s
[INFO] [2021-04-06 17:54:25] Storing 10 Traits
[INFO] [2021-04-06 17:54:25] Processing group of 10 in 1 groups of 1000
[INFO] [2021-04-06 17:54:25] Average Time: 0.0
[INFO] [2021-04-06 17:54:25] Total Time: 1s
[INFO] [2021-04-06 17:54:25] Storing 9 MetaTraits
[INFO] [2021-04-06 17:54:25] Processing group of 9 in 1 groups of 1000
[INFO] [2021-04-06 17:54:25] Average Time: 0.0
[INFO] [2021-04-06 17:54:25] Total Time: 1s
[STOP] [2021-04-06 17:54:25] parse_diff_and_store
[START] [2021-04-06 17:54:25] resolve_keys
[INFO] [2021-04-06 17:54:30] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-06 17:54:30] traits to occurrences...
[INFO] [2021-04-06 17:54:30] traits to nodes (through occurrences)...
[INFO] [2021-04-06 17:54:30] Traits to sex term...
[INFO] [2021-04-06 17:54:30] Traits to lifestage term...
[INFO] [2021-04-06 17:54:30] MetaTraits to traits...
[INFO] [2021-04-06 17:54:30] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-06 17:54:30] Assocs to occurrences...
[INFO] [2021-04-06 17:54:30] Assocs to nodes...
[INFO] [2021-04-06 17:54:30] Assoc to sex term...
[INFO] [2021-04-06 17:54:30] Assoc to lifestage term...
[INFO] [2021-04-06 17:54:30] MetaAssoc to assocs...
[STOP] [2021-04-06 17:54:30] resolve_keys
[START] [2021-04-06 17:54:30] hold_for_later_1
[STOP] [2021-04-06 17:54:30] hold_for_later_1
[START] [2021-04-06 17:54:30] hold_for_later_2
[STOP] [2021-04-06 17:54:30] hold_for_later_2
[START] [2021-04-06 17:54:30] resolve_missing_parents
[STOP] [2021-04-06 17:54:30] resolve_missing_parents
[START] [2021-04-06 17:54:30] rebuild_nodes
[START] [2021-04-06 17:54:30] Flattener#flatten
[START] [2021-04-06 17:54:30] Flattener#study_resource
[START] [2021-04-06 17:54:30] Flattener#build_ancestry
[STOP] [2021-04-06 17:54:30] Flattener#build_ancestry
[INFO] [2021-04-06 17:54:30] 9 ancestry keys
[START] [2021-04-06 17:54:30] build_node_ancestors
[INFO] [2021-04-06 17:54:30] old ancestors deleted.
[STOP] [2021-04-06 17:54:30] build_node_ancestors
[WARN] [2021-04-06 17:54:30] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2021-04-06 17:54:30] Flattener#flatten
[STOP] [2021-04-06 17:54:30] rebuild_nodes
[START] [2021-04-06 17:54:30] resolve_missing_media_owners
[STOP] [2021-04-06 17:54:30] resolve_missing_media_owners
[START] [2021-04-06 17:54:30] sanitize_media_verbatims
[STOP] [2021-04-06 17:54:30] sanitize_media_verbatims
[START] [2021-04-06 17:54:30] queue_downloads
[STOP] [2021-04-06 17:54:30] queue_downloads
[START] [2021-04-06 17:54:31] parse_names
[WARN] [2021-04-06 17:54:31] I see 9 names which still need to be parsed.
[STOP] [2021-04-06 17:54:32] parse_names
[START] [2021-04-06 17:54:32] denormalize_canonical_names_to_nodes
[STOP] [2021-04-06 17:54:32] denormalize_canonical_names_to_nodes
[START] [2021-04-06 17:54:32] match_nodes
[START] [2021-04-06 17:54:32] map_all_nodes_to_pages
[STOP] [2021-04-06 17:54:32] map_all_nodes_to_pages
[INFO] [2021-04-06 17:54:32] ZERO unmatched nodes (of 9)! Nicely done.
[START] [2021-04-06 17:54:32] update_nodes
[STOP] [2021-04-06 17:54:32] update_nodes
[STOP] [2021-04-06 17:54:32] match_nodes
[START] [2021-04-06 17:54:32] reindex_search
[STOP] [2021-04-06 17:54:32] reindex_search
[START] [2021-04-06 17:54:32] normalize_units
[STOP] [2021-04-06 17:54:32] normalize_units
[START] [2021-04-06 17:54:32] calculate_statistics
[2021-04-06 17:54:32] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2021-04-06 17:54:32] calculate_statistics
[START] [2021-04-06 17:54:32] complete_harvest_instance
[START] [2021-04-06 17:54:32] overall_tsv_creation
[INFO] [2021-04-06 17:54:32] Processing group of 9 in 1 batches of 10000
[INFO] [2021-04-06 17:55:08] 9 Traits (unfiltered)...
[INFO] [2021-04-06 17:55:42] 9 Traits (filtered)...
[INFO] [2021-04-06 17:55:42] 0 Associations (filtered)...
[INFO] [2021-04-06 17:55:42] 10 metadata added.
[INFO] [2021-04-06 17:55:42] 0 metadata added.
[INFO] [2021-04-06 17:56:09] Average Time: 72.5
[INFO] [2021-04-06 17:56:09] Total Time: 1m38s
[STOP] [2021-04-06 17:56:09] overall_tsv_creation
[INFO] [2021-04-06 17:56:09] Done. Check your files:
[INFO] [2021-04-06 17:56:09] (9 lines) /app/public/data/borges_et_al_bor/publish_nodes.tsv
[INFO] [2021-04-06 17:56:09] (9 lines) /app/public/data/borges_et_al_bor/publish_scientific_names.tsv
[INFO] [2021-04-06 17:56:09] (10 lines) /app/public/data/borges_et_al_bor/publish_traits.tsv
[INFO] [2021-04-06 17:56:09] (11 lines) /app/public/data/borges_et_al_bor/publish_metadata.tsv
[STOP] [2021-04-06 17:56:09] complete_harvest_instance
[START] [2021-04-06 17:56:09] completed
[STOP] [2021-04-06 17:56:09] completed
[STOP] [2021-04-06 17:56:09] logged process, took 104.79

Latest Process