Harvest for Olenina et al 2006 Created 03 Apr 18:05

Harvesting Log

(1789 lines) (showing only the last 1000 lines, see /app/public/data/Olenina_et_al_20/process.log for the full file)

[CMD] [2020-12-01 13:47:07] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_agents_24686.csv > /app/public/converted_csv/Olenina_et_al_20_agents_24686.csv_sorted
[CMD] [2020-12-01 13:47:07] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_refs_24687.csv > /app/public/converted_csv/Olenina_et_al_20_refs_24687.csv_sorted
[CMD] [2020-12-01 13:47:07] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_nodes_24688.csv > /app/public/converted_csv/Olenina_et_al_20_nodes_24688.csv_sorted
[CMD] [2020-12-01 13:47:07] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_media_24689.csv > /app/public/converted_csv/Olenina_et_al_20_media_24689.csv_sorted
[CMD] [2020-12-01 13:47:07] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_vernaculars_24690.csv > /app/public/converted_csv/Olenina_et_al_20_vernaculars_24690.csv_sorted
[CMD] [2020-12-01 13:47:07] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_occurrences_24691.csv > /app/public/converted_csv/Olenina_et_al_20_occurrences_24691.csv_sorted
[CMD] [2020-12-01 13:47:07] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_assocs_24692.csv > /app/public/converted_csv/Olenina_et_al_20_assocs_24692.csv_sorted
[CMD] [2020-12-01 13:47:07] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_measurements_24693.csv > /app/public/converted_csv/Olenina_et_al_20_measurements_24693.csv_sorted
[STOP] [2020-12-01 13:47:07] convert_to_csv
[START] [2020-12-01 13:47:07] calculate_delta
[CMD] [2020-12-01 13:47:07] echo "0a" > /app/public/diff/Olenina_et_al_20_agents_24686.diff
[CMD] [2020-12-01 13:47:07] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_agents_24686.csv >> /app/public/diff/Olenina_et_al_20_agents_24686.diff
[CMD] [2020-12-01 13:47:07] echo "." >> /app/public/diff/Olenina_et_al_20_agents_24686.diff
[CMD] [2020-12-01 13:47:07] echo "0a" > /app/public/diff/Olenina_et_al_20_refs_24687.diff
[CMD] [2020-12-01 13:47:07] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_refs_24687.csv >> /app/public/diff/Olenina_et_al_20_refs_24687.diff
[CMD] [2020-12-01 13:47:07] echo "." >> /app/public/diff/Olenina_et_al_20_refs_24687.diff
[CMD] [2020-12-01 13:47:07] echo "0a" > /app/public/diff/Olenina_et_al_20_nodes_24688.diff
[CMD] [2020-12-01 13:47:07] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_nodes_24688.csv >> /app/public/diff/Olenina_et_al_20_nodes_24688.diff
[CMD] [2020-12-01 13:47:07] echo "." >> /app/public/diff/Olenina_et_al_20_nodes_24688.diff
[CMD] [2020-12-01 13:47:07] echo "0a" > /app/public/diff/Olenina_et_al_20_media_24689.diff
[CMD] [2020-12-01 13:47:07] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_media_24689.csv >> /app/public/diff/Olenina_et_al_20_media_24689.diff
[CMD] [2020-12-01 13:47:07] echo "." >> /app/public/diff/Olenina_et_al_20_media_24689.diff
[CMD] [2020-12-01 13:47:07] echo "0a" > /app/public/diff/Olenina_et_al_20_vernaculars_24690.diff
[CMD] [2020-12-01 13:47:07] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_vernaculars_24690.csv >> /app/public/diff/Olenina_et_al_20_vernaculars_24690.diff
[CMD] [2020-12-01 13:47:07] echo "." >> /app/public/diff/Olenina_et_al_20_vernaculars_24690.diff
[CMD] [2020-12-01 13:47:07] echo "0a" > /app/public/diff/Olenina_et_al_20_occurrences_24691.diff
[CMD] [2020-12-01 13:47:07] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_occurrences_24691.csv >> /app/public/diff/Olenina_et_al_20_occurrences_24691.diff
[CMD] [2020-12-01 13:47:07] echo "." >> /app/public/diff/Olenina_et_al_20_occurrences_24691.diff
[CMD] [2020-12-01 13:47:07] echo "0a" > /app/public/diff/Olenina_et_al_20_assocs_24692.diff
[CMD] [2020-12-01 13:47:07] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_assocs_24692.csv >> /app/public/diff/Olenina_et_al_20_assocs_24692.diff
[CMD] [2020-12-01 13:47:07] echo "." >> /app/public/diff/Olenina_et_al_20_assocs_24692.diff
[CMD] [2020-12-01 13:47:07] echo "0a" > /app/public/diff/Olenina_et_al_20_measurements_24693.diff
[CMD] [2020-12-01 13:47:07] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_measurements_24693.csv >> /app/public/diff/Olenina_et_al_20_measurements_24693.diff
[CMD] [2020-12-01 13:47:07] echo "." >> /app/public/diff/Olenina_et_al_20_measurements_24693.diff
[STOP] [2020-12-01 13:47:07] calculate_delta
[START] [2020-12-01 13:47:07] parse_diff_and_store
[INFO] [2020-12-01 13:47:07] Loading agents diff file into memory (true lines)...
[INFO] [2020-12-01 13:47:07] Loading refs diff file into memory (true lines)...
[INFO] [2020-12-01 13:47:07] Loading nodes diff file into memory (true lines)...
[WARN] [2020-12-01 13:47:07] Filtered Scientific Name `Acanthoceras zachariasii                     ` to `Acanthoceras zachariasii `
[WARN] [2020-12-01 13:47:07] Filtered Scientific Name `Carteria                           ` to `Carteria `
[WARN] [2020-12-01 13:47:07] Filtered Scientific Name `Tetraëdron caudatum  ` to `Tetraëdron caudatum `
[INFO] [2020-12-01 13:47:07] Loading media diff file into memory (true lines)...
[INFO] [2020-12-01 13:47:07] Loading vernaculars diff file into memory (true lines)...
[INFO] [2020-12-01 13:47:07] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-12-01 13:47:08] Loading assocs diff file into memory (true lines)...
[INFO] [2020-12-01 13:47:08] Loading measurements diff file into memory (true lines)...
[STOP] [2020-12-01 13:47:09] parse_diff_and_store
[ERR] [2020-12-01 13:47:09] RuntimeError
[ERR] [2020-12-01 13:47:09] Missing Term for URI `http://eol.org/schema/terms/coenobium`, must be added!
[ERR] [2020-12-01 13:47:09] ../models/store/model_builder.rb:635:in `fail_on_bad_uri'
[ERR] [2020-12-01 13:47:09] ../models/store/model_builder.rb:589:in `convert_trait_value'
[ERR] [2020-12-01 13:47:09] ../models/store/model_builder.rb:415:in `build_trait'
[ERR] [2020-12-01 13:47:09] ../models/store/model_builder.rb:28:in `build_models'
[ERR] [2020-12-01 13:47:09] ../models/resource_harvester.rb:357:in `block (3 levels) in parse_diff_and_store'
[ERR] [2020-12-01 13:47:09] ../models/csv_parser.rb:111:in `block in diff_as_hashes'
[ERR] [2020-12-01 13:47:09] ../models/csv_parser.rb:28:in `block in line_at_a_time'
[ERR] [2020-12-01 13:47:09] ../models/csv_parser.rb:25:in `line_at_a_time'
[ERR] [2020-12-01 13:47:09] ../models/csv_parser.rb:96:in `diff_as_hashes'
[ERR] [2020-12-01 13:47:09] ../models/resource_harvester.rb:313:in `block (2 levels) in parse_diff_and_store'
[ERR] [2020-12-01 13:47:09] ../models/logged_process.rb:62:in `enter_group'
[ERR] [2020-12-01 13:47:09] ../models/resource_harvester.rb:312:in `block in parse_diff_and_store'
[ERR] [2020-12-01 13:47:09] ../models/resource_harvester.rb:700:in `block in each_diff'
[ERR] [2020-12-01 13:47:09] ../models/resource_harvester.rb:689:in `each_diff'
[ERR] [2020-12-01 13:47:09] ../models/resource_harvester.rb:307:in `parse_diff_and_store'
[ERR] [2020-12-01 13:47:09] ../models/resource_harvester.rb:85:in `block (3 levels) in start'
[ERR] [2020-12-01 13:47:09] ../models/logged_process.rb:19:in `run_step'
[ERR] [2020-12-01 13:47:09] ../models/resource_harvester.rb:85:in `block (2 levels) in start'
[ERR] [2020-12-01 13:47:09] ../models/resource_harvester.rb:74:in `each_key'
[ERR] [2020-12-01 13:47:09] ../models/resource_harvester.rb:74:in `block in start'
[ERR] [2020-12-01 13:47:09] ../models/resource.rb:151:in `lock'
[ERR] [2020-12-01 13:47:09] ../models/resource_harvester.rb:72:in `start'
[ERR] [2020-12-01 13:47:09] ../models/resource.rb:232:in `harvest'
[ERR] [2020-12-01 13:47:09] ../models/resource.rb:208:in `re_download_opendata_and_harvest'
[ERR] [2020-12-01 13:47:09] bin/rails:4:in `require'
[ERR] [2020-12-01 13:47:09] bin/rails:4:in `<main>'
[STOP] [2020-12-01 13:47:09] logged process, took 13.84
[INFO] [2020-12-02 14:05:23] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-12-02 14:05:24] ## remove_type: ScientificName
[INFO] [2020-12-02 14:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:05:24] [14:05:24.496] Removed 0 Scientificnames
[INFO] [2020-12-02 14:05:24] ## remove_type: Vernacular
[INFO] [2020-12-02 14:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:05:24] [14:05:24.499] Removed 0 Vernaculars
[INFO] [2020-12-02 14:05:24] ## remove_type: Article
[INFO] [2020-12-02 14:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:05:24] [14:05:24.502] Removed 0 Articles
[INFO] [2020-12-02 14:05:24] ## remove_type: Medium
[INFO] [2020-12-02 14:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:05:24] [14:05:24.506] Removed 0 Media
[INFO] [2020-12-02 14:05:24] ## remove_type: Trait
[INFO] [2020-12-02 14:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:05:24] [14:05:24.510] Removed 0 Traits
[INFO] [2020-12-02 14:05:24] ## remove_type: MetaTrait
[INFO] [2020-12-02 14:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:05:24] [14:05:24.513] Removed 0 Metatraits
[INFO] [2020-12-02 14:05:24] ## remove_type: OccurrenceMetadatum
[INFO] [2020-12-02 14:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:05:24] [14:05:24.518] Removed 0 Occurrencemetadata
[INFO] [2020-12-02 14:05:24] ## remove_type: Assoc
[INFO] [2020-12-02 14:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:05:24] [14:05:24.521] Removed 0 Assocs
[INFO] [2020-12-02 14:05:24] ## remove_type: MetaAssoc
[INFO] [2020-12-02 14:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:05:24] [14:05:24.524] Removed 0 Metaassocs
[INFO] [2020-12-02 14:05:24] ## remove_type: Identifier
[INFO] [2020-12-02 14:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:05:24] [14:05:24.527] Removed 0 Identifiers
[INFO] [2020-12-02 14:05:24] ## remove_type: Reference
[INFO] [2020-12-02 14:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:05:24] [14:05:24.530] Removed 0 References
[INFO] [2020-12-02 14:05:25] ## remove_type: Node
[INFO] [2020-12-02 14:05:25] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:05:25] [14:05:25.791] Removed 0 Nodes
[START] [2020-12-02 14:05:26] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-12-02 14:05:26] Creating resource from OpenData
[START] [2020-12-02 14:05:26] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-12-02 14:05:26] Parse meta.xml file and create formats with fields
[WARN] [2020-12-02 14:05:27] (common) IGNORED  (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2020-12-02 14:05:27] (common) IGNORED  (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2020-12-02 14:05:27] (common) IGNORED  (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2020-12-02 14:05:27] (common) IGNORED  (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2020-12-02 14:05:27] (common) IGNORED  (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2020-12-02 14:05:27] (common) IGNORED  (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2020-12-02 14:05:27] (common) IGNORED  (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[WARN] [2020-12-02 14:05:27] SKIPPING http://rs.tdwg.org/dwc/terms/Event config (events.txt)...
[STOP] [2020-12-02 14:05:27] Parse meta.xml file and create formats with fields
[STOP] [2020-12-02 14:05:27] Creating resource from OpenData
[START] [2020-12-02 14:05:27] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-12-02 14:05:27] create_harvest_instance
[STOP] [2020-12-02 14:05:36] create_harvest_instance
[START] [2020-12-02 14:05:36] fetch_files
[STOP] [2020-12-02 14:05:36] fetch_files
[START] [2020-12-02 14:05:36] validate_each_file
[STOP] [2020-12-02 14:05:38] validate_each_file
[START] [2020-12-02 14:05:38] convert_to_csv
[CMD] [2020-12-02 14:05:39] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_agents_24724.csv > /app/public/converted_csv/Olenina_et_al_20_agents_24724.csv_sorted
[CMD] [2020-12-02 14:05:39] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_refs_24725.csv > /app/public/converted_csv/Olenina_et_al_20_refs_24725.csv_sorted
[CMD] [2020-12-02 14:05:39] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_nodes_24726.csv > /app/public/converted_csv/Olenina_et_al_20_nodes_24726.csv_sorted
[CMD] [2020-12-02 14:05:39] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_media_24727.csv > /app/public/converted_csv/Olenina_et_al_20_media_24727.csv_sorted
[CMD] [2020-12-02 14:05:39] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_vernaculars_24728.csv > /app/public/converted_csv/Olenina_et_al_20_vernaculars_24728.csv_sorted
[CMD] [2020-12-02 14:05:39] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_occurrences_24729.csv > /app/public/converted_csv/Olenina_et_al_20_occurrences_24729.csv_sorted
[CMD] [2020-12-02 14:05:39] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_assocs_24730.csv > /app/public/converted_csv/Olenina_et_al_20_assocs_24730.csv_sorted
[CMD] [2020-12-02 14:05:39] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_measurements_24731.csv > /app/public/converted_csv/Olenina_et_al_20_measurements_24731.csv_sorted
[STOP] [2020-12-02 14:05:39] convert_to_csv
[START] [2020-12-02 14:05:39] calculate_delta
[CMD] [2020-12-02 14:05:39] echo "0a" > /app/public/diff/Olenina_et_al_20_agents_24724.diff
[CMD] [2020-12-02 14:05:39] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_agents_24724.csv >> /app/public/diff/Olenina_et_al_20_agents_24724.diff
[CMD] [2020-12-02 14:05:39] echo "." >> /app/public/diff/Olenina_et_al_20_agents_24724.diff
[CMD] [2020-12-02 14:05:39] echo "0a" > /app/public/diff/Olenina_et_al_20_refs_24725.diff
[CMD] [2020-12-02 14:05:39] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_refs_24725.csv >> /app/public/diff/Olenina_et_al_20_refs_24725.diff
[CMD] [2020-12-02 14:05:39] echo "." >> /app/public/diff/Olenina_et_al_20_refs_24725.diff
[CMD] [2020-12-02 14:05:39] echo "0a" > /app/public/diff/Olenina_et_al_20_nodes_24726.diff
[CMD] [2020-12-02 14:05:39] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_nodes_24726.csv >> /app/public/diff/Olenina_et_al_20_nodes_24726.diff
[CMD] [2020-12-02 14:05:39] echo "." >> /app/public/diff/Olenina_et_al_20_nodes_24726.diff
[CMD] [2020-12-02 14:05:39] echo "0a" > /app/public/diff/Olenina_et_al_20_media_24727.diff
[CMD] [2020-12-02 14:05:39] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_media_24727.csv >> /app/public/diff/Olenina_et_al_20_media_24727.diff
[CMD] [2020-12-02 14:05:39] echo "." >> /app/public/diff/Olenina_et_al_20_media_24727.diff
[CMD] [2020-12-02 14:05:39] echo "0a" > /app/public/diff/Olenina_et_al_20_vernaculars_24728.diff
[CMD] [2020-12-02 14:05:39] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_vernaculars_24728.csv >> /app/public/diff/Olenina_et_al_20_vernaculars_24728.diff
[CMD] [2020-12-02 14:05:39] echo "." >> /app/public/diff/Olenina_et_al_20_vernaculars_24728.diff
[CMD] [2020-12-02 14:05:39] echo "0a" > /app/public/diff/Olenina_et_al_20_occurrences_24729.diff
[CMD] [2020-12-02 14:05:39] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_occurrences_24729.csv >> /app/public/diff/Olenina_et_al_20_occurrences_24729.diff
[CMD] [2020-12-02 14:05:39] echo "." >> /app/public/diff/Olenina_et_al_20_occurrences_24729.diff
[CMD] [2020-12-02 14:05:39] echo "0a" > /app/public/diff/Olenina_et_al_20_assocs_24730.diff
[CMD] [2020-12-02 14:05:39] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_assocs_24730.csv >> /app/public/diff/Olenina_et_al_20_assocs_24730.diff
[CMD] [2020-12-02 14:05:39] echo "." >> /app/public/diff/Olenina_et_al_20_assocs_24730.diff
[CMD] [2020-12-02 14:05:39] echo "0a" > /app/public/diff/Olenina_et_al_20_measurements_24731.diff
[CMD] [2020-12-02 14:05:39] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_measurements_24731.csv >> /app/public/diff/Olenina_et_al_20_measurements_24731.diff
[CMD] [2020-12-02 14:05:39] echo "." >> /app/public/diff/Olenina_et_al_20_measurements_24731.diff
[STOP] [2020-12-02 14:05:39] calculate_delta
[START] [2020-12-02 14:05:39] parse_diff_and_store
[INFO] [2020-12-02 14:05:39] Loading agents diff file into memory (true lines)...
[INFO] [2020-12-02 14:05:39] Loading refs diff file into memory (true lines)...
[INFO] [2020-12-02 14:05:39] Loading nodes diff file into memory (true lines)...
[WARN] [2020-12-02 14:05:39] Filtered Scientific Name `Acanthoceras zachariasii                     ` to `Acanthoceras zachariasii `
[WARN] [2020-12-02 14:05:39] Filtered Scientific Name `Carteria                           ` to `Carteria `
[WARN] [2020-12-02 14:05:39] Filtered Scientific Name `Tetraëdron caudatum  ` to `Tetraëdron caudatum `
[INFO] [2020-12-02 14:05:39] Loading media diff file into memory (true lines)...
[INFO] [2020-12-02 14:05:39] Loading vernaculars diff file into memory (true lines)...
[INFO] [2020-12-02 14:05:39] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-12-02 14:05:40] Loading assocs diff file into memory (true lines)...
[INFO] [2020-12-02 14:05:40] Loading measurements diff file into memory (true lines)...
[STOP] [2020-12-02 14:05:41] parse_diff_and_store
[ERR] [2020-12-02 14:05:41] RuntimeError
[ERR] [2020-12-02 14:05:41] Missing Term for URI `http://eol.org/schema/terms/coenobium`, must be added!
[ERR] [2020-12-02 14:05:41] ../models/store/model_builder.rb:635:in `fail_on_bad_uri'
[ERR] [2020-12-02 14:05:41] ../models/store/model_builder.rb:589:in `convert_trait_value'
[ERR] [2020-12-02 14:05:41] ../models/store/model_builder.rb:415:in `build_trait'
[ERR] [2020-12-02 14:05:41] ../models/store/model_builder.rb:28:in `build_models'
[ERR] [2020-12-02 14:05:41] ../models/resource_harvester.rb:357:in `block (3 levels) in parse_diff_and_store'
[ERR] [2020-12-02 14:05:41] ../models/csv_parser.rb:111:in `block in diff_as_hashes'
[ERR] [2020-12-02 14:05:41] ../models/csv_parser.rb:28:in `block in line_at_a_time'
[ERR] [2020-12-02 14:05:41] ../models/csv_parser.rb:25:in `line_at_a_time'
[ERR] [2020-12-02 14:05:41] ../models/csv_parser.rb:96:in `diff_as_hashes'
[ERR] [2020-12-02 14:05:41] ../models/resource_harvester.rb:313:in `block (2 levels) in parse_diff_and_store'
[ERR] [2020-12-02 14:05:41] ../models/logged_process.rb:62:in `enter_group'
[ERR] [2020-12-02 14:05:41] ../models/resource_harvester.rb:312:in `block in parse_diff_and_store'
[ERR] [2020-12-02 14:05:41] ../models/resource_harvester.rb:700:in `block in each_diff'
[ERR] [2020-12-02 14:05:41] ../models/resource_harvester.rb:689:in `each_diff'
[ERR] [2020-12-02 14:05:41] ../models/resource_harvester.rb:307:in `parse_diff_and_store'
[ERR] [2020-12-02 14:05:41] ../models/resource_harvester.rb:85:in `block (3 levels) in start'
[ERR] [2020-12-02 14:05:41] ../models/logged_process.rb:19:in `run_step'
[ERR] [2020-12-02 14:05:41] ../models/resource_harvester.rb:85:in `block (2 levels) in start'
[ERR] [2020-12-02 14:05:41] ../models/resource_harvester.rb:74:in `each_key'
[ERR] [2020-12-02 14:05:41] ../models/resource_harvester.rb:74:in `block in start'
[ERR] [2020-12-02 14:05:41] ../models/resource.rb:151:in `lock'
[ERR] [2020-12-02 14:05:41] ../models/resource_harvester.rb:72:in `start'
[ERR] [2020-12-02 14:05:41] ../models/resource.rb:232:in `harvest'
[ERR] [2020-12-02 14:05:41] ../models/resource.rb:208:in `re_download_opendata_and_harvest'
[ERR] [2020-12-02 14:05:41] bin/rails:4:in `require'
[ERR] [2020-12-02 14:05:41] bin/rails:4:in `<main>'
[STOP] [2020-12-02 14:05:41] logged process, took 14.59
[INFO] [2020-12-02 14:59:50] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-12-02 14:59:52] ## remove_type: ScientificName
[INFO] [2020-12-02 14:59:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:59:52] [14:59:52.636] Removed 0 Scientificnames
[INFO] [2020-12-02 14:59:52] ## remove_type: Vernacular
[INFO] [2020-12-02 14:59:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:59:52] [14:59:52.639] Removed 0 Vernaculars
[INFO] [2020-12-02 14:59:52] ## remove_type: Article
[INFO] [2020-12-02 14:59:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:59:52] [14:59:52.642] Removed 0 Articles
[INFO] [2020-12-02 14:59:52] ## remove_type: Medium
[INFO] [2020-12-02 14:59:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:59:52] [14:59:52.645] Removed 0 Media
[INFO] [2020-12-02 14:59:52] ## remove_type: Trait
[INFO] [2020-12-02 14:59:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:59:52] [14:59:52.649] Removed 0 Traits
[INFO] [2020-12-02 14:59:52] ## remove_type: MetaTrait
[INFO] [2020-12-02 14:59:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:59:52] [14:59:52.656] Removed 0 Metatraits
[INFO] [2020-12-02 14:59:52] ## remove_type: OccurrenceMetadatum
[INFO] [2020-12-02 14:59:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:59:52] [14:59:52.659] Removed 0 Occurrencemetadata
[INFO] [2020-12-02 14:59:52] ## remove_type: Assoc
[INFO] [2020-12-02 14:59:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:59:52] [14:59:52.662] Removed 0 Assocs
[INFO] [2020-12-02 14:59:52] ## remove_type: MetaAssoc
[INFO] [2020-12-02 14:59:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:59:52] [14:59:52.665] Removed 0 Metaassocs
[INFO] [2020-12-02 14:59:52] ## remove_type: Identifier
[INFO] [2020-12-02 14:59:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:59:52] [14:59:52.667] Removed 0 Identifiers
[INFO] [2020-12-02 14:59:52] ## remove_type: Reference
[INFO] [2020-12-02 14:59:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:59:52] [14:59:52.670] Removed 0 References
[INFO] [2020-12-02 14:59:52] ## remove_type: Node
[INFO] [2020-12-02 14:59:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-02 14:59:52] [14:59:52.691] Removed 0 Nodes
[START] [2020-12-02 14:59:53] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-12-02 14:59:53] Creating resource from OpenData
[START] [2020-12-02 14:59:53] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-12-02 14:59:53] Parse meta.xml file and create formats with fields
[WARN] [2020-12-02 14:59:53] (common) IGNORED  (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2020-12-02 14:59:53] (common) IGNORED  (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2020-12-02 14:59:53] (common) IGNORED  (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2020-12-02 14:59:53] (common) IGNORED  (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2020-12-02 14:59:53] (common) IGNORED  (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2020-12-02 14:59:53] (common) IGNORED  (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2020-12-02 14:59:53] (common) IGNORED  (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[WARN] [2020-12-02 14:59:53] SKIPPING http://rs.tdwg.org/dwc/terms/Event config (events.txt)...
[STOP] [2020-12-02 14:59:53] Parse meta.xml file and create formats with fields
[STOP] [2020-12-02 14:59:53] Creating resource from OpenData
[START] [2020-12-02 14:59:53] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-12-02 14:59:53] create_harvest_instance
[STOP] [2020-12-02 15:00:03] create_harvest_instance
[START] [2020-12-02 15:00:03] fetch_files
[STOP] [2020-12-02 15:00:03] fetch_files
[START] [2020-12-02 15:00:03] validate_each_file
[STOP] [2020-12-02 15:00:05] validate_each_file
[START] [2020-12-02 15:00:05] convert_to_csv
[CMD] [2020-12-02 15:00:05] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_agents_24746.csv > /app/public/converted_csv/Olenina_et_al_20_agents_24746.csv_sorted
[CMD] [2020-12-02 15:00:05] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_refs_24747.csv > /app/public/converted_csv/Olenina_et_al_20_refs_24747.csv_sorted
[CMD] [2020-12-02 15:00:05] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_nodes_24748.csv > /app/public/converted_csv/Olenina_et_al_20_nodes_24748.csv_sorted
[CMD] [2020-12-02 15:00:05] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_media_24749.csv > /app/public/converted_csv/Olenina_et_al_20_media_24749.csv_sorted
[CMD] [2020-12-02 15:00:05] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_vernaculars_24750.csv > /app/public/converted_csv/Olenina_et_al_20_vernaculars_24750.csv_sorted
[CMD] [2020-12-02 15:00:05] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_occurrences_24751.csv > /app/public/converted_csv/Olenina_et_al_20_occurrences_24751.csv_sorted
[CMD] [2020-12-02 15:00:05] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_assocs_24752.csv > /app/public/converted_csv/Olenina_et_al_20_assocs_24752.csv_sorted
[CMD] [2020-12-02 15:00:05] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_measurements_24753.csv > /app/public/converted_csv/Olenina_et_al_20_measurements_24753.csv_sorted
[STOP] [2020-12-02 15:00:05] convert_to_csv
[START] [2020-12-02 15:00:05] calculate_delta
[CMD] [2020-12-02 15:00:05] echo "0a" > /app/public/diff/Olenina_et_al_20_agents_24746.diff
[CMD] [2020-12-02 15:00:05] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_agents_24746.csv >> /app/public/diff/Olenina_et_al_20_agents_24746.diff
[CMD] [2020-12-02 15:00:05] echo "." >> /app/public/diff/Olenina_et_al_20_agents_24746.diff
[CMD] [2020-12-02 15:00:05] echo "0a" > /app/public/diff/Olenina_et_al_20_refs_24747.diff
[CMD] [2020-12-02 15:00:05] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_refs_24747.csv >> /app/public/diff/Olenina_et_al_20_refs_24747.diff
[CMD] [2020-12-02 15:00:05] echo "." >> /app/public/diff/Olenina_et_al_20_refs_24747.diff
[CMD] [2020-12-02 15:00:05] echo "0a" > /app/public/diff/Olenina_et_al_20_nodes_24748.diff
[CMD] [2020-12-02 15:00:05] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_nodes_24748.csv >> /app/public/diff/Olenina_et_al_20_nodes_24748.diff
[CMD] [2020-12-02 15:00:05] echo "." >> /app/public/diff/Olenina_et_al_20_nodes_24748.diff
[CMD] [2020-12-02 15:00:05] echo "0a" > /app/public/diff/Olenina_et_al_20_media_24749.diff
[CMD] [2020-12-02 15:00:05] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_media_24749.csv >> /app/public/diff/Olenina_et_al_20_media_24749.diff
[CMD] [2020-12-02 15:00:05] echo "." >> /app/public/diff/Olenina_et_al_20_media_24749.diff
[CMD] [2020-12-02 15:00:05] echo "0a" > /app/public/diff/Olenina_et_al_20_vernaculars_24750.diff
[CMD] [2020-12-02 15:00:05] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_vernaculars_24750.csv >> /app/public/diff/Olenina_et_al_20_vernaculars_24750.diff
[CMD] [2020-12-02 15:00:05] echo "." >> /app/public/diff/Olenina_et_al_20_vernaculars_24750.diff
[CMD] [2020-12-02 15:00:05] echo "0a" > /app/public/diff/Olenina_et_al_20_occurrences_24751.diff
[CMD] [2020-12-02 15:00:05] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_occurrences_24751.csv >> /app/public/diff/Olenina_et_al_20_occurrences_24751.diff
[CMD] [2020-12-02 15:00:05] echo "." >> /app/public/diff/Olenina_et_al_20_occurrences_24751.diff
[CMD] [2020-12-02 15:00:05] echo "0a" > /app/public/diff/Olenina_et_al_20_assocs_24752.diff
[CMD] [2020-12-02 15:00:05] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_assocs_24752.csv >> /app/public/diff/Olenina_et_al_20_assocs_24752.diff
[CMD] [2020-12-02 15:00:05] echo "." >> /app/public/diff/Olenina_et_al_20_assocs_24752.diff
[CMD] [2020-12-02 15:00:05] echo "0a" > /app/public/diff/Olenina_et_al_20_measurements_24753.diff
[CMD] [2020-12-02 15:00:05] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_measurements_24753.csv >> /app/public/diff/Olenina_et_al_20_measurements_24753.diff
[CMD] [2020-12-02 15:00:05] echo "." >> /app/public/diff/Olenina_et_al_20_measurements_24753.diff
[STOP] [2020-12-02 15:00:05] calculate_delta
[START] [2020-12-02 15:00:05] parse_diff_and_store
[INFO] [2020-12-02 15:00:05] Loading agents diff file into memory (true lines)...
[INFO] [2020-12-02 15:00:05] Loading refs diff file into memory (true lines)...
[INFO] [2020-12-02 15:00:05] Loading nodes diff file into memory (true lines)...
[WARN] [2020-12-02 15:00:06] Filtered Scientific Name `Acanthoceras zachariasii                     ` to `Acanthoceras zachariasii `
[WARN] [2020-12-02 15:00:06] Filtered Scientific Name `Carteria                           ` to `Carteria `
[WARN] [2020-12-02 15:00:06] Filtered Scientific Name `Tetraëdron caudatum  ` to `Tetraëdron caudatum `
[INFO] [2020-12-02 15:00:06] Loading media diff file into memory (true lines)...
[INFO] [2020-12-02 15:00:06] Loading vernaculars diff file into memory (true lines)...
[INFO] [2020-12-02 15:00:06] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-12-02 15:00:07] Loading assocs diff file into memory (true lines)...
[INFO] [2020-12-02 15:00:07] Loading measurements diff file into memory (true lines)...
[STOP] [2020-12-02 15:00:08] parse_diff_and_store
[ERR] [2020-12-02 15:00:08] RuntimeError
[ERR] [2020-12-02 15:00:08] Missing Term for URI `http://eol.org/schema/terms/SizeClass`, must be added!
[ERR] [2020-12-02 15:00:08] ../models/store/model_builder.rb:635:in `fail_on_bad_uri'
[ERR] [2020-12-02 15:00:08] ../models/store/model_builder.rb:408:in `build_trait'
[ERR] [2020-12-02 15:00:08] ../models/store/model_builder.rb:28:in `build_models'
[ERR] [2020-12-02 15:00:08] ../models/resource_harvester.rb:357:in `block (3 levels) in parse_diff_and_store'
[ERR] [2020-12-02 15:00:08] ../models/csv_parser.rb:111:in `block in diff_as_hashes'
[ERR] [2020-12-02 15:00:08] ../models/csv_parser.rb:28:in `block in line_at_a_time'
[ERR] [2020-12-02 15:00:08] ../models/csv_parser.rb:25:in `line_at_a_time'
[ERR] [2020-12-02 15:00:08] ../models/csv_parser.rb:96:in `diff_as_hashes'
[ERR] [2020-12-02 15:00:08] ../models/resource_harvester.rb:313:in `block (2 levels) in parse_diff_and_store'
[ERR] [2020-12-02 15:00:08] ../models/logged_process.rb:62:in `enter_group'
[ERR] [2020-12-02 15:00:08] ../models/resource_harvester.rb:312:in `block in parse_diff_and_store'
[ERR] [2020-12-02 15:00:08] ../models/resource_harvester.rb:700:in `block in each_diff'
[ERR] [2020-12-02 15:00:08] ../models/resource_harvester.rb:689:in `each_diff'
[ERR] [2020-12-02 15:00:08] ../models/resource_harvester.rb:307:in `parse_diff_and_store'
[ERR] [2020-12-02 15:00:08] ../models/resource_harvester.rb:85:in `block (3 levels) in start'
[ERR] [2020-12-02 15:00:08] ../models/logged_process.rb:19:in `run_step'
[ERR] [2020-12-02 15:00:08] ../models/resource_harvester.rb:85:in `block (2 levels) in start'
[ERR] [2020-12-02 15:00:08] ../models/resource_harvester.rb:74:in `each_key'
[ERR] [2020-12-02 15:00:08] ../models/resource_harvester.rb:74:in `block in start'
[ERR] [2020-12-02 15:00:08] ../models/resource.rb:151:in `lock'
[ERR] [2020-12-02 15:00:08] ../models/resource_harvester.rb:72:in `start'
[ERR] [2020-12-02 15:00:08] ../models/resource.rb:232:in `harvest'
[ERR] [2020-12-02 15:00:08] ../models/resource.rb:208:in `re_download_opendata_and_harvest'
[ERR] [2020-12-02 15:00:08] bin/rails:4:in `require'
[ERR] [2020-12-02 15:00:08] bin/rails:4:in `<main>'
[STOP] [2020-12-02 15:00:08] logged process, took 14.77
[INFO] [2020-12-07 15:54:49] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-12-07 15:54:53] ## remove_type: ScientificName
[INFO] [2020-12-07 15:54:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 15:54:53] [15:54:53.582] Removed 0 Scientificnames
[INFO] [2020-12-07 15:54:53] ## remove_type: Vernacular
[INFO] [2020-12-07 15:54:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 15:54:53] [15:54:53.585] Removed 0 Vernaculars
[INFO] [2020-12-07 15:54:53] ## remove_type: Article
[INFO] [2020-12-07 15:54:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 15:54:53] [15:54:53.588] Removed 0 Articles
[INFO] [2020-12-07 15:54:53] ## remove_type: Medium
[INFO] [2020-12-07 15:54:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 15:54:53] [15:54:53.592] Removed 0 Media
[INFO] [2020-12-07 15:54:53] ## remove_type: Trait
[INFO] [2020-12-07 15:54:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 15:54:53] [15:54:53.595] Removed 0 Traits
[INFO] [2020-12-07 15:54:53] ## remove_type: MetaTrait
[INFO] [2020-12-07 15:54:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 15:54:53] [15:54:53.609] Removed 0 Metatraits
[INFO] [2020-12-07 15:54:53] ## remove_type: OccurrenceMetadatum
[INFO] [2020-12-07 15:54:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 15:54:53] [15:54:53.615] Removed 0 Occurrencemetadata
[INFO] [2020-12-07 15:54:53] ## remove_type: Assoc
[INFO] [2020-12-07 15:54:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 15:54:53] [15:54:53.618] Removed 0 Assocs
[INFO] [2020-12-07 15:54:53] ## remove_type: MetaAssoc
[INFO] [2020-12-07 15:54:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 15:54:53] [15:54:53.621] Removed 0 Metaassocs
[INFO] [2020-12-07 15:54:53] ## remove_type: Identifier
[INFO] [2020-12-07 15:54:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 15:54:53] [15:54:53.635] Removed 0 Identifiers
[INFO] [2020-12-07 15:54:53] ## remove_type: Reference
[INFO] [2020-12-07 15:54:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 15:54:53] [15:54:53.638] Removed 0 References
[INFO] [2020-12-07 15:54:57] ## remove_type: Node
[INFO] [2020-12-07 15:54:57] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 15:54:57] [15:54:57.020] Removed 0 Nodes
[START] [2020-12-07 15:54:57] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-12-07 15:54:57] Creating resource from OpenData
[START] [2020-12-07 15:54:58] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-12-07 15:54:58] Parse meta.xml file and create formats with fields
[WARN] [2020-12-07 15:54:58] (common) IGNORED  (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2020-12-07 15:54:58] (common) IGNORED  (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2020-12-07 15:54:58] (common) IGNORED  (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2020-12-07 15:54:58] (common) IGNORED  (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2020-12-07 15:54:58] (common) IGNORED  (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2020-12-07 15:54:58] (common) IGNORED  (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2020-12-07 15:54:58] (common) IGNORED  (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[WARN] [2020-12-07 15:54:58] SKIPPING http://rs.tdwg.org/dwc/terms/Event config (events.txt)...
[STOP] [2020-12-07 15:54:58] Parse meta.xml file and create formats with fields
[STOP] [2020-12-07 15:54:58] Creating resource from OpenData
[START] [2020-12-07 15:54:58] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-12-07 15:54:58] create_harvest_instance
[STOP] [2020-12-07 15:55:07] create_harvest_instance
[START] [2020-12-07 15:55:07] fetch_files
[STOP] [2020-12-07 15:55:07] fetch_files
[START] [2020-12-07 15:55:07] validate_each_file
[STOP] [2020-12-07 15:55:09] validate_each_file
[START] [2020-12-07 15:55:09] convert_to_csv
[CMD] [2020-12-07 15:55:09] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_agents_25244.csv > /app/public/converted_csv/Olenina_et_al_20_agents_25244.csv_sorted
[CMD] [2020-12-07 15:55:09] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_refs_25245.csv > /app/public/converted_csv/Olenina_et_al_20_refs_25245.csv_sorted
[CMD] [2020-12-07 15:55:09] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_nodes_25246.csv > /app/public/converted_csv/Olenina_et_al_20_nodes_25246.csv_sorted
[CMD] [2020-12-07 15:55:09] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_media_25247.csv > /app/public/converted_csv/Olenina_et_al_20_media_25247.csv_sorted
[CMD] [2020-12-07 15:55:09] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_vernaculars_25248.csv > /app/public/converted_csv/Olenina_et_al_20_vernaculars_25248.csv_sorted
[CMD] [2020-12-07 15:55:09] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_occurrences_25249.csv > /app/public/converted_csv/Olenina_et_al_20_occurrences_25249.csv_sorted
[CMD] [2020-12-07 15:55:09] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_assocs_25250.csv > /app/public/converted_csv/Olenina_et_al_20_assocs_25250.csv_sorted
[CMD] [2020-12-07 15:55:09] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_measurements_25251.csv > /app/public/converted_csv/Olenina_et_al_20_measurements_25251.csv_sorted
[STOP] [2020-12-07 15:55:09] convert_to_csv
[START] [2020-12-07 15:55:09] calculate_delta
[CMD] [2020-12-07 15:55:09] echo "0a" > /app/public/diff/Olenina_et_al_20_agents_25244.diff
[CMD] [2020-12-07 15:55:09] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_agents_25244.csv >> /app/public/diff/Olenina_et_al_20_agents_25244.diff
[CMD] [2020-12-07 15:55:09] echo "." >> /app/public/diff/Olenina_et_al_20_agents_25244.diff
[CMD] [2020-12-07 15:55:09] echo "0a" > /app/public/diff/Olenina_et_al_20_refs_25245.diff
[CMD] [2020-12-07 15:55:09] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_refs_25245.csv >> /app/public/diff/Olenina_et_al_20_refs_25245.diff
[CMD] [2020-12-07 15:55:09] echo "." >> /app/public/diff/Olenina_et_al_20_refs_25245.diff
[CMD] [2020-12-07 15:55:09] echo "0a" > /app/public/diff/Olenina_et_al_20_nodes_25246.diff
[CMD] [2020-12-07 15:55:09] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_nodes_25246.csv >> /app/public/diff/Olenina_et_al_20_nodes_25246.diff
[CMD] [2020-12-07 15:55:09] echo "." >> /app/public/diff/Olenina_et_al_20_nodes_25246.diff
[CMD] [2020-12-07 15:55:09] echo "0a" > /app/public/diff/Olenina_et_al_20_media_25247.diff
[CMD] [2020-12-07 15:55:09] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_media_25247.csv >> /app/public/diff/Olenina_et_al_20_media_25247.diff
[CMD] [2020-12-07 15:55:09] echo "." >> /app/public/diff/Olenina_et_al_20_media_25247.diff
[CMD] [2020-12-07 15:55:09] echo "0a" > /app/public/diff/Olenina_et_al_20_vernaculars_25248.diff
[CMD] [2020-12-07 15:55:09] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_vernaculars_25248.csv >> /app/public/diff/Olenina_et_al_20_vernaculars_25248.diff
[CMD] [2020-12-07 15:55:09] echo "." >> /app/public/diff/Olenina_et_al_20_vernaculars_25248.diff
[CMD] [2020-12-07 15:55:09] echo "0a" > /app/public/diff/Olenina_et_al_20_occurrences_25249.diff
[CMD] [2020-12-07 15:55:09] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_occurrences_25249.csv >> /app/public/diff/Olenina_et_al_20_occurrences_25249.diff
[CMD] [2020-12-07 15:55:09] echo "." >> /app/public/diff/Olenina_et_al_20_occurrences_25249.diff
[CMD] [2020-12-07 15:55:09] echo "0a" > /app/public/diff/Olenina_et_al_20_assocs_25250.diff
[CMD] [2020-12-07 15:55:09] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_assocs_25250.csv >> /app/public/diff/Olenina_et_al_20_assocs_25250.diff
[CMD] [2020-12-07 15:55:09] echo "." >> /app/public/diff/Olenina_et_al_20_assocs_25250.diff
[CMD] [2020-12-07 15:55:09] echo "0a" > /app/public/diff/Olenina_et_al_20_measurements_25251.diff
[CMD] [2020-12-07 15:55:09] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_measurements_25251.csv >> /app/public/diff/Olenina_et_al_20_measurements_25251.diff
[CMD] [2020-12-07 15:55:09] echo "." >> /app/public/diff/Olenina_et_al_20_measurements_25251.diff
[STOP] [2020-12-07 15:55:09] calculate_delta
[START] [2020-12-07 15:55:09] parse_diff_and_store
[INFO] [2020-12-07 15:55:09] Loading agents diff file into memory (true lines)...
[INFO] [2020-12-07 15:55:09] Loading refs diff file into memory (true lines)...
[INFO] [2020-12-07 15:55:09] Loading nodes diff file into memory (true lines)...
[WARN] [2020-12-07 15:55:09] Filtered Scientific Name `Acanthoceras zachariasii                     ` to `Acanthoceras zachariasii `
[WARN] [2020-12-07 15:55:09] Filtered Scientific Name `Carteria                           ` to `Carteria `
[WARN] [2020-12-07 15:55:10] Filtered Scientific Name `Tetraëdron caudatum  ` to `Tetraëdron caudatum `
[INFO] [2020-12-07 15:55:10] Loading media diff file into memory (true lines)...
[INFO] [2020-12-07 15:55:10] Loading vernaculars diff file into memory (true lines)...
[INFO] [2020-12-07 15:55:10] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-12-07 15:55:11] Loading assocs diff file into memory (true lines)...
[INFO] [2020-12-07 15:55:11] Loading measurements diff file into memory (true lines)...
[STOP] [2020-12-07 15:55:12] parse_diff_and_store
[ERR] [2020-12-07 15:55:12] RuntimeError
[ERR] [2020-12-07 15:55:12] Missing Term for URI `http://eol.org/schema/terms/CellsPerCountingUnit`, must be added!
[ERR] [2020-12-07 15:55:12] ../models/store/model_builder.rb:635:in `fail_on_bad_uri'
[ERR] [2020-12-07 15:55:12] ../models/store/model_builder.rb:408:in `build_trait'
[ERR] [2020-12-07 15:55:12] ../models/store/model_builder.rb:28:in `build_models'
[ERR] [2020-12-07 15:55:12] ../models/resource_harvester.rb:357:in `block (3 levels) in parse_diff_and_store'
[ERR] [2020-12-07 15:55:12] ../models/csv_parser.rb:111:in `block in diff_as_hashes'
[ERR] [2020-12-07 15:55:12] ../models/csv_parser.rb:28:in `block in line_at_a_time'
[ERR] [2020-12-07 15:55:12] ../models/csv_parser.rb:25:in `line_at_a_time'
[ERR] [2020-12-07 15:55:12] ../models/csv_parser.rb:96:in `diff_as_hashes'
[ERR] [2020-12-07 15:55:12] ../models/resource_harvester.rb:313:in `block (2 levels) in parse_diff_and_store'
[ERR] [2020-12-07 15:55:12] ../models/logged_process.rb:62:in `enter_group'
[ERR] [2020-12-07 15:55:12] ../models/resource_harvester.rb:312:in `block in parse_diff_and_store'
[ERR] [2020-12-07 15:55:12] ../models/resource_harvester.rb:700:in `block in each_diff'
[ERR] [2020-12-07 15:55:12] ../models/resource_harvester.rb:689:in `each_diff'
[ERR] [2020-12-07 15:55:12] ../models/resource_harvester.rb:307:in `parse_diff_and_store'
[ERR] [2020-12-07 15:55:12] ../models/resource_harvester.rb:85:in `block (3 levels) in start'
[ERR] [2020-12-07 15:55:12] ../models/logged_process.rb:19:in `run_step'
[ERR] [2020-12-07 15:55:12] ../models/resource_harvester.rb:85:in `block (2 levels) in start'
[ERR] [2020-12-07 15:55:12] ../models/resource_harvester.rb:74:in `each_key'
[ERR] [2020-12-07 15:55:12] ../models/resource_harvester.rb:74:in `block in start'
[ERR] [2020-12-07 15:55:12] ../models/resource.rb:151:in `lock'
[ERR] [2020-12-07 15:55:12] ../models/resource_harvester.rb:72:in `start'
[ERR] [2020-12-07 15:55:12] ../models/resource.rb:232:in `harvest'
[ERR] [2020-12-07 15:55:12] ../models/resource.rb:208:in `re_download_opendata_and_harvest'
[ERR] [2020-12-07 15:55:12] bin/rails:4:in `require'
[ERR] [2020-12-07 15:55:12] bin/rails:4:in `<main>'
[STOP] [2020-12-07 15:55:12] logged process, took 14.52
[INFO] [2020-12-07 16:05:20] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-12-07 16:05:24] ## remove_type: ScientificName
[INFO] [2020-12-07 16:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 16:05:24] [16:05:24.948] Removed 0 Scientificnames
[INFO] [2020-12-07 16:05:24] ## remove_type: Vernacular
[INFO] [2020-12-07 16:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 16:05:24] [16:05:24.952] Removed 0 Vernaculars
[INFO] [2020-12-07 16:05:24] ## remove_type: Article
[INFO] [2020-12-07 16:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 16:05:24] [16:05:24.955] Removed 0 Articles
[INFO] [2020-12-07 16:05:24] ## remove_type: Medium
[INFO] [2020-12-07 16:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 16:05:24] [16:05:24.959] Removed 0 Media
[INFO] [2020-12-07 16:05:24] ## remove_type: Trait
[INFO] [2020-12-07 16:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 16:05:24] [16:05:24.962] Removed 0 Traits
[INFO] [2020-12-07 16:05:24] ## remove_type: MetaTrait
[INFO] [2020-12-07 16:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 16:05:24] [16:05:24.966] Removed 0 Metatraits
[INFO] [2020-12-07 16:05:24] ## remove_type: OccurrenceMetadatum
[INFO] [2020-12-07 16:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 16:05:24] [16:05:24.968] Removed 0 Occurrencemetadata
[INFO] [2020-12-07 16:05:24] ## remove_type: Assoc
[INFO] [2020-12-07 16:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 16:05:24] [16:05:24.971] Removed 0 Assocs
[INFO] [2020-12-07 16:05:24] ## remove_type: MetaAssoc
[INFO] [2020-12-07 16:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 16:05:24] [16:05:24.974] Removed 0 Metaassocs
[INFO] [2020-12-07 16:05:24] ## remove_type: Identifier
[INFO] [2020-12-07 16:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 16:05:24] [16:05:24.976] Removed 0 Identifiers
[INFO] [2020-12-07 16:05:24] ## remove_type: Reference
[INFO] [2020-12-07 16:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 16:05:24] [16:05:24.979] Removed 0 References
[INFO] [2020-12-07 16:05:24] ## remove_type: Node
[INFO] [2020-12-07 16:05:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-07 16:05:24] [16:05:24.999] Removed 0 Nodes
[START] [2020-12-07 16:05:25] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-12-07 16:05:25] Creating resource from OpenData
[START] [2020-12-07 16:05:25] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-12-07 16:05:25] Parse meta.xml file and create formats with fields
[WARN] [2020-12-07 16:05:26] (common) IGNORED  (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2020-12-07 16:05:26] (common) IGNORED  (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2020-12-07 16:05:26] (common) IGNORED  (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2020-12-07 16:05:26] (common) IGNORED  (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2020-12-07 16:05:26] (common) IGNORED  (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2020-12-07 16:05:26] (common) IGNORED  (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2020-12-07 16:05:26] (common) IGNORED  (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[WARN] [2020-12-07 16:05:26] SKIPPING http://rs.tdwg.org/dwc/terms/Event config (events.txt)...
[STOP] [2020-12-07 16:05:26] Parse meta.xml file and create formats with fields
[STOP] [2020-12-07 16:05:26] Creating resource from OpenData
[START] [2020-12-07 16:05:26] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-12-07 16:05:26] create_harvest_instance
[STOP] [2020-12-07 16:05:35] create_harvest_instance
[START] [2020-12-07 16:05:35] fetch_files
[STOP] [2020-12-07 16:05:35] fetch_files
[START] [2020-12-07 16:05:35] validate_each_file
[STOP] [2020-12-07 16:05:37] validate_each_file
[START] [2020-12-07 16:05:37] convert_to_csv
[CMD] [2020-12-07 16:05:37] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_agents_25260.csv > /app/public/converted_csv/Olenina_et_al_20_agents_25260.csv_sorted
[CMD] [2020-12-07 16:05:37] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_refs_25261.csv > /app/public/converted_csv/Olenina_et_al_20_refs_25261.csv_sorted
[CMD] [2020-12-07 16:05:37] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_nodes_25262.csv > /app/public/converted_csv/Olenina_et_al_20_nodes_25262.csv_sorted
[CMD] [2020-12-07 16:05:37] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_media_25263.csv > /app/public/converted_csv/Olenina_et_al_20_media_25263.csv_sorted
[CMD] [2020-12-07 16:05:37] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_vernaculars_25264.csv > /app/public/converted_csv/Olenina_et_al_20_vernaculars_25264.csv_sorted
[CMD] [2020-12-07 16:05:37] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_occurrences_25265.csv > /app/public/converted_csv/Olenina_et_al_20_occurrences_25265.csv_sorted
[CMD] [2020-12-07 16:05:37] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_assocs_25266.csv > /app/public/converted_csv/Olenina_et_al_20_assocs_25266.csv_sorted
[CMD] [2020-12-07 16:05:37] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_measurements_25267.csv > /app/public/converted_csv/Olenina_et_al_20_measurements_25267.csv_sorted
[STOP] [2020-12-07 16:05:37] convert_to_csv
[START] [2020-12-07 16:05:37] calculate_delta
[CMD] [2020-12-07 16:05:37] echo "0a" > /app/public/diff/Olenina_et_al_20_agents_25260.diff
[CMD] [2020-12-07 16:05:37] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_agents_25260.csv >> /app/public/diff/Olenina_et_al_20_agents_25260.diff
[CMD] [2020-12-07 16:05:37] echo "." >> /app/public/diff/Olenina_et_al_20_agents_25260.diff
[CMD] [2020-12-07 16:05:37] echo "0a" > /app/public/diff/Olenina_et_al_20_refs_25261.diff
[CMD] [2020-12-07 16:05:37] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_refs_25261.csv >> /app/public/diff/Olenina_et_al_20_refs_25261.diff
[CMD] [2020-12-07 16:05:37] echo "." >> /app/public/diff/Olenina_et_al_20_refs_25261.diff
[CMD] [2020-12-07 16:05:37] echo "0a" > /app/public/diff/Olenina_et_al_20_nodes_25262.diff
[CMD] [2020-12-07 16:05:37] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_nodes_25262.csv >> /app/public/diff/Olenina_et_al_20_nodes_25262.diff
[CMD] [2020-12-07 16:05:37] echo "." >> /app/public/diff/Olenina_et_al_20_nodes_25262.diff
[CMD] [2020-12-07 16:05:37] echo "0a" > /app/public/diff/Olenina_et_al_20_media_25263.diff
[CMD] [2020-12-07 16:05:37] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_media_25263.csv >> /app/public/diff/Olenina_et_al_20_media_25263.diff
[CMD] [2020-12-07 16:05:37] echo "." >> /app/public/diff/Olenina_et_al_20_media_25263.diff
[CMD] [2020-12-07 16:05:37] echo "0a" > /app/public/diff/Olenina_et_al_20_vernaculars_25264.diff
[CMD] [2020-12-07 16:05:37] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_vernaculars_25264.csv >> /app/public/diff/Olenina_et_al_20_vernaculars_25264.diff
[CMD] [2020-12-07 16:05:37] echo "." >> /app/public/diff/Olenina_et_al_20_vernaculars_25264.diff
[CMD] [2020-12-07 16:05:37] echo "0a" > /app/public/diff/Olenina_et_al_20_occurrences_25265.diff
[CMD] [2020-12-07 16:05:37] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_occurrences_25265.csv >> /app/public/diff/Olenina_et_al_20_occurrences_25265.diff
[CMD] [2020-12-07 16:05:37] echo "." >> /app/public/diff/Olenina_et_al_20_occurrences_25265.diff
[CMD] [2020-12-07 16:05:37] echo "0a" > /app/public/diff/Olenina_et_al_20_assocs_25266.diff
[CMD] [2020-12-07 16:05:37] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_assocs_25266.csv >> /app/public/diff/Olenina_et_al_20_assocs_25266.diff
[CMD] [2020-12-07 16:05:37] echo "." >> /app/public/diff/Olenina_et_al_20_assocs_25266.diff
[CMD] [2020-12-07 16:05:37] echo "0a" > /app/public/diff/Olenina_et_al_20_measurements_25267.diff
[CMD] [2020-12-07 16:05:37] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_measurements_25267.csv >> /app/public/diff/Olenina_et_al_20_measurements_25267.diff
[CMD] [2020-12-07 16:05:37] echo "." >> /app/public/diff/Olenina_et_al_20_measurements_25267.diff
[STOP] [2020-12-07 16:05:37] calculate_delta
[START] [2020-12-07 16:05:37] parse_diff_and_store
[INFO] [2020-12-07 16:05:37] Loading agents diff file into memory (true lines)...
[INFO] [2020-12-07 16:05:37] Loading refs diff file into memory (true lines)...
[INFO] [2020-12-07 16:05:37] Loading nodes diff file into memory (true lines)...
[WARN] [2020-12-07 16:05:37] Filtered Scientific Name `Acanthoceras zachariasii                     ` to `Acanthoceras zachariasii `
[WARN] [2020-12-07 16:05:37] Filtered Scientific Name `Carteria                           ` to `Carteria `
[WARN] [2020-12-07 16:05:37] Filtered Scientific Name `Tetraëdron caudatum  ` to `Tetraëdron caudatum `
[INFO] [2020-12-07 16:05:37] Loading media diff file into memory (true lines)...
[INFO] [2020-12-07 16:05:37] Loading vernaculars diff file into memory (true lines)...
[INFO] [2020-12-07 16:05:37] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-12-07 16:05:38] Loading assocs diff file into memory (true lines)...
[INFO] [2020-12-07 16:05:38] Loading measurements diff file into memory (true lines)...
[INFO] [2020-12-07 16:05:45] Storing 904 ScientificNames
[INFO] [2020-12-07 16:05:45] Processing group of 904 in 1 groups of 1000
[INFO] [2020-12-07 16:05:46] Average Time: 0.54
[INFO] [2020-12-07 16:05:46] Total Time: 1s
[INFO] [2020-12-07 16:05:46] Storing 904 Nodes
[INFO] [2020-12-07 16:05:46] Processing group of 904 in 1 groups of 1000
[INFO] [2020-12-07 16:05:46] Average Time: 0.41
[INFO] [2020-12-07 16:05:46] Total Time: 1s
[INFO] [2020-12-07 16:05:46] Storing 900 Occurrences
[INFO] [2020-12-07 16:05:46] Processing group of 900 in 1 groups of 1000
[INFO] [2020-12-07 16:05:47] Average Time: 0.19
[INFO] [2020-12-07 16:05:47] Total Time: 1s
[INFO] [2020-12-07 16:05:47] Storing 3600 OccurrenceMetadata
[INFO] [2020-12-07 16:05:47] Processing group of 3600 in 4 groups of 1000
[INFO] [2020-12-07 16:05:47] Average Time: 0.11
[INFO] [2020-12-07 16:05:47] Total Time: 1s
[INFO] [2020-12-07 16:05:47] Storing 20077 Traits
[INFO] [2020-12-07 16:05:47] Processing group of 20077 in 21 groups of 1000
[INFO] [2020-12-07 16:05:54] Average Time: 0.306
[INFO] [2020-12-07 16:05:54] Total Time: 7s
[INFO] [2020-12-07 16:05:54] last 3 / first 3: 0.87
[INFO] [2020-12-07 16:05:54] Std.Dev: 0.07071067811865475; Max: 0.38
[INFO] [2020-12-07 16:05:54] Storing 21897 MetaTraits
[INFO] [2020-12-07 16:05:54] Processing group of 21897 in 22 groups of 1000
[INFO] [2020-12-07 16:05:56] Average Time: 0.131
[INFO] [2020-12-07 16:05:56] Total Time: 3s
[INFO] [2020-12-07 16:05:56] last 3 / first 3: 0.8
[INFO] [2020-12-07 16:05:56] Std.Dev: 0.03162277660168379; Max: 0.27
[STOP] [2020-12-07 16:05:57] parse_diff_and_store
[START] [2020-12-07 16:05:57] resolve_keys
[INFO] [2020-12-07 16:06:04] Occurrences to nodes (through scientific_names)...
[INFO] [2020-12-07 16:06:04] traits to occurrences...
[INFO] [2020-12-07 16:06:04] traits to nodes (through occurrences)...
[INFO] [2020-12-07 16:06:04] Traits to sex term...
[INFO] [2020-12-07 16:06:04] Traits to lifestage term...
[INFO] [2020-12-07 16:06:04] MetaTraits to traits...
[INFO] [2020-12-07 16:06:05] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-12-07 16:06:05] Assocs to occurrences...
[INFO] [2020-12-07 16:06:05] Assocs to nodes...
[INFO] [2020-12-07 16:06:05] Assoc to sex term...
[INFO] [2020-12-07 16:06:05] Assoc to lifestage term...
[INFO] [2020-12-07 16:06:05] MetaAssoc to assocs...
[STOP] [2020-12-07 16:06:05] resolve_keys
[START] [2020-12-07 16:06:05] hold_for_later_1
[STOP] [2020-12-07 16:06:05] hold_for_later_1
[START] [2020-12-07 16:06:05] hold_for_later_2
[STOP] [2020-12-07 16:06:05] hold_for_later_2
[START] [2020-12-07 16:06:05] resolve_missing_parents
[STOP] [2020-12-07 16:06:05] resolve_missing_parents
[START] [2020-12-07 16:06:05] rebuild_nodes
[START] [2020-12-07 16:06:05] Flattener#flatten
[START] [2020-12-07 16:06:05] Flattener#study_resource
[START] [2020-12-07 16:06:05] Flattener#build_ancestry
[STOP] [2020-12-07 16:06:05] Flattener#build_ancestry
[INFO] [2020-12-07 16:06:05] 904 ancestry keys
[START] [2020-12-07 16:06:05] build_node_ancestors
[INFO] [2020-12-07 16:06:05] old ancestors deleted.
[STOP] [2020-12-07 16:06:05] build_node_ancestors
[START] [2020-12-07 16:06:05] Flattener#propagate_ancestor_ids
[STOP] [2020-12-07 16:06:05] Flattener#propagate_ancestor_ids
[STOP] [2020-12-07 16:06:05] Flattener#flatten
[STOP] [2020-12-07 16:06:05] rebuild_nodes
[START] [2020-12-07 16:06:05] resolve_missing_media_owners
[STOP] [2020-12-07 16:06:05] resolve_missing_media_owners
[START] [2020-12-07 16:06:05] sanitize_media_verbatims
[STOP] [2020-12-07 16:06:05] sanitize_media_verbatims
[START] [2020-12-07 16:06:05] queue_downloads
[STOP] [2020-12-07 16:06:05] queue_downloads
[START] [2020-12-07 16:06:05] parse_names
[WARN] [2020-12-07 16:06:05] I see 904 names which still need to be parsed.
[WARN] [2020-12-07 16:06:07] I see 7 names which still need to be parsed.
[STOP] [2020-12-07 16:06:08] parse_names
[START] [2020-12-07 16:06:08] denormalize_canonical_names_to_nodes
[STOP] [2020-12-07 16:06:08] denormalize_canonical_names_to_nodes
[START] [2020-12-07 16:06:08] match_nodes
[START] [2020-12-07 16:06:08] map_all_nodes_to_pages
[STOP] [2020-12-07 16:07:18] map_all_nodes_to_pages
[INFO] [2020-12-07 16:07:18] 180 Unmatched nodes (of 904)! That's too many to output. First 10: Acanthoica quattrospina (#86116854); Achnanthes taeniata (#86116855); Actiniscus pentasterias (#86116862); Actinocyclus octonarius crassus (#86116865); Actinocyclus octonarius octonarius (#86116866); Actinocyclus octonarius octonarius (#86116867); Actinocyclus (#86116869); Alexandrium tamarense (#86116875); Amphidinium acutissimum (#86116877); Amphidinium sphenoides (#86116884)
[START] [2020-12-07 16:07:18] update_nodes
[STOP] [2020-12-07 16:07:19] update_nodes
[STOP] [2020-12-07 16:07:19] match_nodes
[START] [2020-12-07 16:07:19] reindex_search
[STOP] [2020-12-07 16:07:20] reindex_search
[START] [2020-12-07 16:07:20] normalize_units
[STOP] [2020-12-07 16:07:20] normalize_units
[START] [2020-12-07 16:07:20] calculate_statistics
[STOP] [2020-12-07 16:07:20] calculate_statistics
[START] [2020-12-07 16:07:20] complete_harvest_instance
[START] [2020-12-07 16:07:20] overall_tsv_creation
[INFO] [2020-12-07 16:07:20] Processing group of 904 in 1 batches of 10000
[INFO] [2020-12-07 16:08:03] 1826 Traits (unfiltered)...
[INFO] [2020-12-07 16:08:45] 1826 Traits (filtered)...
[INFO] [2020-12-07 16:08:45] 0 Associations (filtered)...
[INFO] [2020-12-07 16:08:46] 9230 metadata added.
[INFO] [2020-12-07 16:08:46] 0 metadata added.
[INFO] [2020-12-07 16:08:46] Average Time: 62.96
[INFO] [2020-12-07 16:08:46] Total Time: 1m27s
[STOP] [2020-12-07 16:08:46] overall_tsv_creation
[INFO] [2020-12-07 16:08:46] Done. Check your files:
[INFO] [2020-12-07 16:08:46] (904 lines) /app/public/data/Olenina_et_al_20/publish_nodes.tsv
[INFO] [2020-12-07 16:08:46] (888 lines) /app/public/data/Olenina_et_al_20/publish_node_ancestors.tsv
[INFO] [2020-12-07 16:08:46] (904 lines) /app/public/data/Olenina_et_al_20/publish_scientific_names.tsv
[INFO] [2020-12-07 16:08:46] (1827 lines) /app/public/data/Olenina_et_al_20/publish_traits.tsv
[INFO] [2020-12-07 16:08:46] (9231 lines) /app/public/data/Olenina_et_al_20/publish_metadata.tsv
[STOP] [2020-12-07 16:08:46] complete_harvest_instance
[START] [2020-12-07 16:08:46] completed
[STOP] [2020-12-07 16:08:46] completed
[STOP] [2020-12-07 16:08:46] logged process, took 200.56
[INFO] [2021-04-03 18:05:08] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-03 18:05:08] ## remove_type: ScientificName
[INFO] [2021-04-03 18:05:08] ++ Calling delete_all on 904 instances...
[INFO] [2021-04-03 18:05:08] [18:05:08.852] Removed 904 Scientificnames
[INFO] [2021-04-03 18:05:08] ## remove_type: Vernacular
[INFO] [2021-04-03 18:05:08] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-03 18:05:08] [18:05:08.854] Removed 0 Vernaculars
[INFO] [2021-04-03 18:05:08] ## remove_type: Article
[INFO] [2021-04-03 18:05:08] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-03 18:05:08] [18:05:08.855] Removed 0 Articles
[INFO] [2021-04-03 18:05:08] ## remove_type: Medium
[INFO] [2021-04-03 18:05:08] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-03 18:05:08] [18:05:08.857] Removed 0 Media
[INFO] [2021-04-03 18:05:08] ## remove_type: Trait
[INFO] [2021-04-03 18:05:08] ++ Calling delete_all on 20077 instances...
[INFO] [2021-04-03 18:05:13] [18:05:13.318] Removed 20077 Traits
[INFO] [2021-04-03 18:05:13] ## remove_type: MetaTrait
[INFO] [2021-04-03 18:05:13] ++ Calling delete_all on 21897 instances...
[INFO] [2021-04-03 18:05:14] [18:05:14.599] Removed 21897 Metatraits
[INFO] [2021-04-03 18:05:14] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-03 18:05:14] ++ Calling delete_all on 3600 instances...
[INFO] [2021-04-03 18:05:14] [18:05:14.723] Removed 3600 Occurrencemetadata
[INFO] [2021-04-03 18:05:14] ## remove_type: Assoc
[INFO] [2021-04-03 18:05:14] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-03 18:05:14] [18:05:14.725] Removed 0 Assocs
[INFO] [2021-04-03 18:05:14] ## remove_type: MetaAssoc
[INFO] [2021-04-03 18:05:14] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-03 18:05:14] [18:05:14.726] Removed 0 Metaassocs
[INFO] [2021-04-03 18:05:14] ## remove_type: Identifier
[INFO] [2021-04-03 18:05:14] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-03 18:05:14] [18:05:14.728] Removed 0 Identifiers
[INFO] [2021-04-03 18:05:14] ## remove_type: Reference
[INFO] [2021-04-03 18:05:14] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-03 18:05:14] [18:05:14.729] Removed 0 References
[INFO] [2021-04-03 18:05:14] Starting batch with ID 86117210...
[INFO] [2021-04-03 18:05:14] Starting batch with ID 86117210...
[INFO] [2021-04-03 18:05:14] Starting batch with ID 86117210...
[INFO] [2021-04-03 18:05:15] Starting batch with ID 86117210...
[INFO] [2021-04-03 18:05:15] ## remove_type: Node
[INFO] [2021-04-03 18:05:15] ++ Calling delete_all on 904 instances...
[INFO] [2021-04-03 18:05:15] [18:05:15.904] Removed 904 Nodes
[START] [2021-04-03 18:05:16] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-03 18:05:16] Creating resource from OpenData
[START] [2021-04-03 18:05:18] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-03 18:05:18] Parse meta.xml file and create formats with fields
[WARN] [2021-04-03 18:05:37] (common) IGNORED  (media) field header: ThumbnailURL term: http://eol.org/schema/media/thumbnailURL
[WARN] [2021-04-03 18:05:37] (common) IGNORED  (media) field header: DerivedFrom term: http://rs.tdwg.org/ac/terms/derivedFrom
[WARN] [2021-04-03 18:05:37] (common) IGNORED  (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2021-04-03 18:05:37] (common) IGNORED  (media) field header: Modified term: http://purl.org/dc/terms/modified
[WARN] [2021-04-03 18:05:37] (common) IGNORED  (media) field header: Rating term: http://ns.adobe.com/xap/1.0/Rating
[WARN] [2021-04-03 18:05:37] (common) IGNORED  (media) field header: Audience term: http://purl.org/dc/terms/audience
[WARN] [2021-04-03 18:05:37] (common) IGNORED  (media) field header: Rights term: http://purl.org/dc/terms/rights
[WARN] [2021-04-03 18:05:37] (common) IGNORED  (media) field header: Publisher term: http://purl.org/dc/terms/publisher
[WARN] [2021-04-03 18:05:37] (common) IGNORED  (media) field header: Contributor term: http://purl.org/dc/terms/contributor
[WARN] [2021-04-03 18:05:37] (common) IGNORED  (media) field header: Creator term: http://purl.org/dc/terms/creator
[WARN] [2021-04-03 18:05:37] (common) IGNORED  (media) field header: GenericLocation term: http://purl.org/dc/terms/spatial
[WARN] [2021-04-03 18:05:37] (common) IGNORED  (media) field header: Latitude term: http://www.w3.org/2003/01/geo/wgs84_pos#lat
[WARN] [2021-04-03 18:05:37] (common) IGNORED  (media) field header: Longitude term: http://www.w3.org/2003/01/geo/wgs84_pos#long
[WARN] [2021-04-03 18:05:37] (common) IGNORED  (media) field header: Altitude term: http://www.w3.org/2003/01/geo/wgs84_pos#alt
[WARN] [2021-04-03 18:05:37] (common) IGNORED  (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2021-04-03 18:05:37] (common) IGNORED  (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2021-04-03 18:05:37] (common) IGNORED  (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2021-04-03 18:05:37] (common) IGNORED  (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2021-04-03 18:05:37] (common) IGNORED  (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2021-04-03 18:05:37] (common) IGNORED  (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[WARN] [2021-04-03 18:05:37] SKIPPING http://rs.tdwg.org/dwc/terms/Event config (events.txt)...
[STOP] [2021-04-03 18:05:37] Parse meta.xml file and create formats with fields
[STOP] [2021-04-03 18:05:37] Creating resource from OpenData
[START] [2021-04-03 18:05:38] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-03 18:05:38] create_harvest_instance
[INFO] [2021-04-03 18:05:38] Created harvest instance #3607
[STOP] [2021-04-03 18:05:38] create_harvest_instance
[START] [2021-04-03 18:05:38] fetch_files
[STOP] [2021-04-03 18:05:38] fetch_files
[START] [2021-04-03 18:05:38] validate_each_file
[INFO] [2021-04-03 18:05:38] Looping over 8 formats...
[INFO] [2021-04-03 18:05:38] ...agents (/app/public/data/Olenina_et_al_20/agents.txt)
[INFO] [2021-04-03 18:05:38] Valid: /app/public/converted_csv/Olenina_et_al_20_agents_3607.csv (0 lines)
[INFO] [2021-04-03 18:05:38] ...refs (/app/public/data/Olenina_et_al_20/references.txt)
[INFO] [2021-04-03 18:05:38] Valid: /app/public/converted_csv/Olenina_et_al_20_refs_3607.csv (0 lines)
[INFO] [2021-04-03 18:05:38] ...nodes (/app/public/data/Olenina_et_al_20/taxa.txt)
[INFO] [2021-04-03 18:05:38] Valid: /app/public/converted_csv/Olenina_et_al_20_nodes_3607.csv (899 lines)
[INFO] [2021-04-03 18:05:38] ...media (/app/public/data/Olenina_et_al_20/media.txt)
[INFO] [2021-04-03 18:05:38] Valid: /app/public/converted_csv/Olenina_et_al_20_media_3607.csv (0 lines)
[INFO] [2021-04-03 18:05:38] ...vernaculars (/app/public/data/Olenina_et_al_20/common names.txt)
[INFO] [2021-04-03 18:05:38] Valid: /app/public/converted_csv/Olenina_et_al_20_vernaculars_3607.csv (0 lines)
[INFO] [2021-04-03 18:05:38] ...occurrences (/app/public/data/Olenina_et_al_20/occurrences.txt)
[INFO] [2021-04-03 18:05:38] Valid: /app/public/converted_csv/Olenina_et_al_20_occurrences_3607.csv (900 lines)
[INFO] [2021-04-03 18:05:38] ...assocs (/app/public/data/Olenina_et_al_20/associations.txt)
[INFO] [2021-04-03 18:05:38] Valid: /app/public/converted_csv/Olenina_et_al_20_assocs_3607.csv (0 lines)
[INFO] [2021-04-03 18:05:38] ...measurements (/app/public/data/Olenina_et_al_20/measurements-or-facts.txt)
[INFO] [2021-04-03 18:05:40] Valid: /app/public/converted_csv/Olenina_et_al_20_measurements_3607.csv (21877 lines)
[STOP] [2021-04-03 18:05:40] validate_each_file
[START] [2021-04-03 18:05:40] convert_to_csv
[INFO] [2021-04-03 18:05:40] Looping over 8 formats...
[INFO] [2021-04-03 18:05:40] ...agents (/app/public/data/Olenina_et_al_20/agents.txt)
[CMD] [2021-04-03 18:05:40] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_agents_3607.csv > /app/public/converted_csv/Olenina_et_al_20_agents_3607.csv_sorted
[INFO] [2021-04-03 18:05:41] Converted: /app/public/converted_csv/Olenina_et_al_20_agents_3607.csv (0 lines)
[INFO] [2021-04-03 18:05:41] ...refs (/app/public/data/Olenina_et_al_20/references.txt)
[CMD] [2021-04-03 18:05:41] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_refs_3607.csv > /app/public/converted_csv/Olenina_et_al_20_refs_3607.csv_sorted
[INFO] [2021-04-03 18:05:41] Converted: /app/public/converted_csv/Olenina_et_al_20_refs_3607.csv (0 lines)
[INFO] [2021-04-03 18:05:41] ...nodes (/app/public/data/Olenina_et_al_20/taxa.txt)
[CMD] [2021-04-03 18:05:41] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_nodes_3607.csv > /app/public/converted_csv/Olenina_et_al_20_nodes_3607.csv_sorted
[INFO] [2021-04-03 18:05:42] Converted: /app/public/converted_csv/Olenina_et_al_20_nodes_3607.csv (899 lines)
[INFO] [2021-04-03 18:05:42] ...media (/app/public/data/Olenina_et_al_20/media.txt)
[CMD] [2021-04-03 18:05:42] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_media_3607.csv > /app/public/converted_csv/Olenina_et_al_20_media_3607.csv_sorted
[INFO] [2021-04-03 18:05:43] Converted: /app/public/converted_csv/Olenina_et_al_20_media_3607.csv (0 lines)
[INFO] [2021-04-03 18:05:43] ...vernaculars (/app/public/data/Olenina_et_al_20/common names.txt)
[CMD] [2021-04-03 18:05:43] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_vernaculars_3607.csv > /app/public/converted_csv/Olenina_et_al_20_vernaculars_3607.csv_sorted
[INFO] [2021-04-03 18:05:44] Converted: /app/public/converted_csv/Olenina_et_al_20_vernaculars_3607.csv (0 lines)
[INFO] [2021-04-03 18:05:44] ...occurrences (/app/public/data/Olenina_et_al_20/occurrences.txt)
[CMD] [2021-04-03 18:05:44] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_occurrences_3607.csv > /app/public/converted_csv/Olenina_et_al_20_occurrences_3607.csv_sorted
[INFO] [2021-04-03 18:05:44] Converted: /app/public/converted_csv/Olenina_et_al_20_occurrences_3607.csv (900 lines)
[INFO] [2021-04-03 18:05:45] ...assocs (/app/public/data/Olenina_et_al_20/associations.txt)
[CMD] [2021-04-03 18:05:45] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_assocs_3607.csv > /app/public/converted_csv/Olenina_et_al_20_assocs_3607.csv_sorted
[INFO] [2021-04-03 18:05:45] Converted: /app/public/converted_csv/Olenina_et_al_20_assocs_3607.csv (0 lines)
[INFO] [2021-04-03 18:05:45] ...measurements (/app/public/data/Olenina_et_al_20/measurements-or-facts.txt)
[CMD] [2021-04-03 18:05:45] /usr/bin/sort /app/public/converted_csv/Olenina_et_al_20_measurements_3607.csv > /app/public/converted_csv/Olenina_et_al_20_measurements_3607.csv_sorted
[INFO] [2021-04-03 18:05:46] Converted: /app/public/converted_csv/Olenina_et_al_20_measurements_3607.csv (21877 lines)
[STOP] [2021-04-03 18:05:46] convert_to_csv
[START] [2021-04-03 18:05:46] calculate_delta
[INFO] [2021-04-03 18:05:46] Looping over 8 formats...
[INFO] [2021-04-03 18:05:46] ...agents (/app/public/data/Olenina_et_al_20/agents.txt)
[CMD] [2021-04-03 18:05:46] echo "0a" > /app/public/diff/Olenina_et_al_20_agents_3607.diff
[CMD] [2021-04-03 18:05:47] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_agents_3607.csv >> /app/public/diff/Olenina_et_al_20_agents_3607.diff
[CMD] [2021-04-03 18:05:48] echo "." >> /app/public/diff/Olenina_et_al_20_agents_3607.diff
[INFO] [2021-04-03 18:05:48] Created diff: /app/public/diff/Olenina_et_al_20_agents_3607.diff (2 lines)
[INFO] [2021-04-03 18:05:48] ...refs (/app/public/data/Olenina_et_al_20/references.txt)
[CMD] [2021-04-03 18:05:48] echo "0a" > /app/public/diff/Olenina_et_al_20_refs_3607.diff
[CMD] [2021-04-03 18:05:49] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_refs_3607.csv >> /app/public/diff/Olenina_et_al_20_refs_3607.diff
[CMD] [2021-04-03 18:05:50] echo "." >> /app/public/diff/Olenina_et_al_20_refs_3607.diff
[INFO] [2021-04-03 18:05:51] Created diff: /app/public/diff/Olenina_et_al_20_refs_3607.diff (2 lines)
[INFO] [2021-04-03 18:05:51] ...nodes (/app/public/data/Olenina_et_al_20/taxa.txt)
[CMD] [2021-04-03 18:05:51] echo "0a" > /app/public/diff/Olenina_et_al_20_nodes_3607.diff
[CMD] [2021-04-03 18:05:52] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_nodes_3607.csv >> /app/public/diff/Olenina_et_al_20_nodes_3607.diff
[CMD] [2021-04-03 18:05:52] echo "." >> /app/public/diff/Olenina_et_al_20_nodes_3607.diff
[INFO] [2021-04-03 18:05:53] Created diff: /app/public/diff/Olenina_et_al_20_nodes_3607.diff (901 lines)
[INFO] [2021-04-03 18:05:53] ...media (/app/public/data/Olenina_et_al_20/media.txt)
[CMD] [2021-04-03 18:05:53] echo "0a" > /app/public/diff/Olenina_et_al_20_media_3607.diff
[CMD] [2021-04-03 18:05:54] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_media_3607.csv >> /app/public/diff/Olenina_et_al_20_media_3607.diff
[CMD] [2021-04-03 18:05:55] echo "." >> /app/public/diff/Olenina_et_al_20_media_3607.diff
[INFO] [2021-04-03 18:05:55] Created diff: /app/public/diff/Olenina_et_al_20_media_3607.diff (2 lines)
[INFO] [2021-04-03 18:05:55] ...vernaculars (/app/public/data/Olenina_et_al_20/common names.txt)
[CMD] [2021-04-03 18:05:55] echo "0a" > /app/public/diff/Olenina_et_al_20_vernaculars_3607.diff
[CMD] [2021-04-03 18:05:56] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_vernaculars_3607.csv >> /app/public/diff/Olenina_et_al_20_vernaculars_3607.diff
[CMD] [2021-04-03 18:05:57] echo "." >> /app/public/diff/Olenina_et_al_20_vernaculars_3607.diff
[INFO] [2021-04-03 18:05:58] Created diff: /app/public/diff/Olenina_et_al_20_vernaculars_3607.diff (2 lines)
[INFO] [2021-04-03 18:05:58] ...occurrences (/app/public/data/Olenina_et_al_20/occurrences.txt)
[CMD] [2021-04-03 18:05:58] echo "0a" > /app/public/diff/Olenina_et_al_20_occurrences_3607.diff
[CMD] [2021-04-03 18:05:59] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_occurrences_3607.csv >> /app/public/diff/Olenina_et_al_20_occurrences_3607.diff
[CMD] [2021-04-03 18:05:59] echo "." >> /app/public/diff/Olenina_et_al_20_occurrences_3607.diff
[INFO] [2021-04-03 18:06:00] Created diff: /app/public/diff/Olenina_et_al_20_occurrences_3607.diff (902 lines)
[INFO] [2021-04-03 18:06:00] ...assocs (/app/public/data/Olenina_et_al_20/associations.txt)
[CMD] [2021-04-03 18:06:00] echo "0a" > /app/public/diff/Olenina_et_al_20_assocs_3607.diff
[CMD] [2021-04-03 18:06:01] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_assocs_3607.csv >> /app/public/diff/Olenina_et_al_20_assocs_3607.diff
[CMD] [2021-04-03 18:06:02] echo "." >> /app/public/diff/Olenina_et_al_20_assocs_3607.diff
[INFO] [2021-04-03 18:06:02] Created diff: /app/public/diff/Olenina_et_al_20_assocs_3607.diff (2 lines)
[INFO] [2021-04-03 18:06:02] ...measurements (/app/public/data/Olenina_et_al_20/measurements-or-facts.txt)
[CMD] [2021-04-03 18:06:02] echo "0a" > /app/public/diff/Olenina_et_al_20_measurements_3607.diff
[CMD] [2021-04-03 18:06:03] tail -n +1 /app/public/converted_csv/Olenina_et_al_20_measurements_3607.csv >> /app/public/diff/Olenina_et_al_20_measurements_3607.diff
[CMD] [2021-04-03 18:06:04] echo "." >> /app/public/diff/Olenina_et_al_20_measurements_3607.diff
[INFO] [2021-04-03 18:06:05] Created diff: /app/public/diff/Olenina_et_al_20_measurements_3607.diff (21879 lines)
[STOP] [2021-04-03 18:06:05] calculate_delta
[START] [2021-04-03 18:06:05] parse_diff_and_store
[INFO] [2021-04-03 18:06:05] Handling diff: /app/public/diff/Olenina_et_al_20_agents_3607.diff (2 lines)
[INFO] [2021-04-03 18:06:05] Loading agents diff file into memory (2 /app/public/diff/Olenina_et_al_20_agents_3607.diff lines)...
[INFO] [2021-04-03 18:06:06] Handling diff: /app/public/diff/Olenina_et_al_20_refs_3607.diff (2 lines)
[INFO] [2021-04-03 18:06:07] Loading refs diff file into memory (2 /app/public/diff/Olenina_et_al_20_refs_3607.diff lines)...
[INFO] [2021-04-03 18:06:08] Handling diff: /app/public/diff/Olenina_et_al_20_nodes_3607.diff (901 lines)
[INFO] [2021-04-03 18:06:09] Loading nodes diff file into memory (901 /app/public/diff/Olenina_et_al_20_nodes_3607.diff lines)...
[WARN] [2021-04-03 18:06:09] Filtered Scientific Name `Acanthoceras zachariasii                     ` to `Acanthoceras zachariasii `
[WARN] [2021-04-03 18:06:09] Filtered Scientific Name `Carteria                           ` to `Carteria `
[WARN] [2021-04-03 18:06:10] Filtered Scientific Name `Tetraëdron caudatum  ` to `Tetraëdron caudatum `
[INFO] [2021-04-03 18:06:10] Handling diff: /app/public/diff/Olenina_et_al_20_media_3607.diff (2 lines)
[INFO] [2021-04-03 18:06:10] Loading media diff file into memory (2 /app/public/diff/Olenina_et_al_20_media_3607.diff lines)...
[INFO] [2021-04-03 18:06:11] Handling diff: /app/public/diff/Olenina_et_al_20_vernaculars_3607.diff (2 lines)
[INFO] [2021-04-03 18:06:12] Loading vernaculars diff file into memory (2 /app/public/diff/Olenina_et_al_20_vernaculars_3607.diff lines)...
[INFO] [2021-04-03 18:06:13] Handling diff: /app/public/diff/Olenina_et_al_20_occurrences_3607.diff (902 lines)
[INFO] [2021-04-03 18:06:14] Loading occurrences diff file into memory (902 /app/public/diff/Olenina_et_al_20_occurrences_3607.diff lines)...
[INFO] [2021-04-03 18:06:15] Handling diff: /app/public/diff/Olenina_et_al_20_assocs_3607.diff (2 lines)
[INFO] [2021-04-03 18:06:16] Loading assocs diff file into memory (2 /app/public/diff/Olenina_et_al_20_assocs_3607.diff lines)...
[INFO] [2021-04-03 18:06:16] Handling diff: /app/public/diff/Olenina_et_al_20_measurements_3607.diff (21879 lines)
[INFO] [2021-04-03 18:06:17] Loading measurements diff file into memory (21879 /app/public/diff/Olenina_et_al_20_measurements_3607.diff lines)...
[INFO] [2021-04-03 18:06:27] Storing 904 ScientificNames
[INFO] [2021-04-03 18:06:27] Processing group of 904 in 1 groups of 1000
[INFO] [2021-04-03 18:06:27] Average Time: 0.32
[INFO] [2021-04-03 18:06:27] Total Time: 1s
[INFO] [2021-04-03 18:06:27] Storing 904 Nodes
[INFO] [2021-04-03 18:06:27] Processing group of 904 in 1 groups of 1000
[INFO] [2021-04-03 18:06:28] Average Time: 0.38
[INFO] [2021-04-03 18:06:28] Total Time: 1s
[INFO] [2021-04-03 18:06:28] Storing 900 Occurrences
[INFO] [2021-04-03 18:06:28] Processing group of 900 in 1 groups of 1000
[INFO] [2021-04-03 18:06:28] Average Time: 0.19
[INFO] [2021-04-03 18:06:28] Total Time: 1s
[INFO] [2021-04-03 18:06:28] Storing 3600 OccurrenceMetadata
[INFO] [2021-04-03 18:06:28] Processing group of 3600 in 4 groups of 1000
[INFO] [2021-04-03 18:06:28] Average Time: 0.16
[INFO] [2021-04-03 18:06:28] Total Time: 1s
[INFO] [2021-04-03 18:06:28] Storing 20077 Traits
[INFO] [2021-04-03 18:06:28] Processing group of 20077 in 21 groups of 1000
[INFO] [2021-04-03 18:06:36] Average Time: 0.343
[INFO] [2021-04-03 18:06:36] Total Time: 8s
[INFO] [2021-04-03 18:06:36] last 3 / first 3: 0.52
[INFO] [2021-04-03 18:06:36] Std.Dev: 0.12649110640673517; Max: 0.72
[INFO] [2021-04-03 18:06:36] Storing 21897 MetaTraits
[INFO] [2021-04-03 18:06:36] Processing group of 21897 in 22 groups of 1000
[INFO] [2021-04-03 18:06:39] Average Time: 0.143
[INFO] [2021-04-03 18:06:39] Total Time: 4s
[INFO] [2021-04-03 18:06:39] last 3 / first 3: 1.0
[INFO] [2021-04-03 18:06:39] Std.Dev: 0.03162277660168379; Max: 0.26
[STOP] [2021-04-03 18:06:39] parse_diff_and_store
[START] [2021-04-03 18:06:39] resolve_keys
[INFO] [2021-04-03 18:06:45] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-03 18:06:45] traits to occurrences...
[INFO] [2021-04-03 18:06:45] traits to nodes (through occurrences)...
[INFO] [2021-04-03 18:06:45] Traits to sex term...
[INFO] [2021-04-03 18:06:45] Traits to lifestage term...
[INFO] [2021-04-03 18:06:45] MetaTraits to traits...
[INFO] [2021-04-03 18:06:46] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-03 18:06:46] Assocs to occurrences...
[INFO] [2021-04-03 18:06:46] Assocs to nodes...
[INFO] [2021-04-03 18:06:46] Assoc to sex term...
[INFO] [2021-04-03 18:06:46] Assoc to lifestage term...
[INFO] [2021-04-03 18:06:46] MetaAssoc to assocs...
[STOP] [2021-04-03 18:06:46] resolve_keys
[START] [2021-04-03 18:06:46] hold_for_later_1
[STOP] [2021-04-03 18:06:46] hold_for_later_1
[START] [2021-04-03 18:06:46] hold_for_later_2
[STOP] [2021-04-03 18:06:46] hold_for_later_2
[START] [2021-04-03 18:06:46] resolve_missing_parents
[STOP] [2021-04-03 18:06:46] resolve_missing_parents
[START] [2021-04-03 18:06:46] rebuild_nodes
[START] [2021-04-03 18:06:46] Flattener#flatten
[START] [2021-04-03 18:06:46] Flattener#study_resource
[START] [2021-04-03 18:06:46] Flattener#build_ancestry
[STOP] [2021-04-03 18:06:46] Flattener#build_ancestry
[INFO] [2021-04-03 18:06:46] 904 ancestry keys
[START] [2021-04-03 18:06:46] build_node_ancestors
[INFO] [2021-04-03 18:06:46] old ancestors deleted.
[STOP] [2021-04-03 18:06:46] build_node_ancestors
[START] [2021-04-03 18:06:46] Flattener#propagate_ancestor_ids
[STOP] [2021-04-03 18:06:46] Flattener#propagate_ancestor_ids
[STOP] [2021-04-03 18:06:46] Flattener#flatten
[STOP] [2021-04-03 18:06:46] rebuild_nodes
[START] [2021-04-03 18:06:46] resolve_missing_media_owners
[STOP] [2021-04-03 18:06:46] resolve_missing_media_owners
[START] [2021-04-03 18:06:46] sanitize_media_verbatims
[STOP] [2021-04-03 18:06:46] sanitize_media_verbatims
[START] [2021-04-03 18:06:46] queue_downloads
[STOP] [2021-04-03 18:06:46] queue_downloads
[START] [2021-04-03 18:06:46] parse_names
[WARN] [2021-04-03 18:06:46] I see 904 names which still need to be parsed.
[WARN] [2021-04-03 18:06:48] I see 7 names which still need to be parsed.
[STOP] [2021-04-03 18:06:49] parse_names
[START] [2021-04-03 18:06:49] denormalize_canonical_names_to_nodes
[STOP] [2021-04-03 18:06:49] denormalize_canonical_names_to_nodes
[START] [2021-04-03 18:06:49] match_nodes
[START] [2021-04-03 18:06:49] map_all_nodes_to_pages
[STOP] [2021-04-03 18:07:00] map_all_nodes_to_pages
[INFO] [2021-04-03 18:07:01] 183 Unmatched nodes (of 904)! That's too many to output. Full list in /app/public/data/Olenina_et_al_20/unmatched_nodes.txt ; First 10: Acanthoica quattrospina (#91477586); Achnanthes taeniata (#91477587); Actiniscus pentasterias (#91477594); Actinocyclus octonarius crassus (#91477597); Actinocyclus octonarius octonarius (#91477598); Actinocyclus octonarius octonarius (#91477599); Actinocyclus (#91477601); Alexandrium tamarense (#91477607); Amphidinium acutissimum (#91477609); Amphidinium sphenoides (#91477616)
[START] [2021-04-03 18:07:01] update_nodes
[STOP] [2021-04-03 18:07:01] update_nodes
[STOP] [2021-04-03 18:07:01] match_nodes
[START] [2021-04-03 18:07:01] reindex_search
[STOP] [2021-04-03 18:07:01] reindex_search
[START] [2021-04-03 18:07:01] normalize_units
[STOP] [2021-04-03 18:07:49] normalize_units
[START] [2021-04-03 18:07:49] calculate_statistics
[STOP] [2021-04-03 18:07:49] calculate_statistics
[START] [2021-04-03 18:07:49] complete_harvest_instance
[START] [2021-04-03 18:07:49] overall_tsv_creation
[INFO] [2021-04-03 18:07:49] Processing group of 904 in 1 batches of 10000
[INFO] [2021-04-03 18:08:28] 1826 Traits (unfiltered)...
[INFO] [2021-04-03 18:09:10] 1826 Traits (filtered)...
[INFO] [2021-04-03 18:09:10] 0 Associations (filtered)...
[INFO] [2021-04-03 18:09:11] 5553 metadata added.
[INFO] [2021-04-03 18:09:11] 0 metadata added.
[INFO] [2021-04-03 18:09:39] Average Time: 85.42
[INFO] [2021-04-03 18:09:39] Total Time: 1m50s
[STOP] [2021-04-03 18:09:39] overall_tsv_creation
[INFO] [2021-04-03 18:09:39] Done. Check your files:
[INFO] [2021-04-03 18:09:39] (904 lines) /app/public/data/Olenina_et_al_20/publish_nodes.tsv
[INFO] [2021-04-03 18:09:40] (888 lines) /app/public/data/Olenina_et_al_20/publish_node_ancestors.tsv
[INFO] [2021-04-03 18:09:41] (904 lines) /app/public/data/Olenina_et_al_20/publish_scientific_names.tsv
[INFO] [2021-04-03 18:09:42] (1827 lines) /app/public/data/Olenina_et_al_20/publish_traits.tsv
[INFO] [2021-04-03 18:09:42] (5554 lines) /app/public/data/Olenina_et_al_20/publish_metadata.tsv
[STOP] [2021-04-03 18:09:42] complete_harvest_instance
[START] [2021-04-03 18:09:42] completed
[STOP] [2021-04-03 18:09:42] completed
[STOP] [2021-04-03 18:09:42] logged process, took 245.46

Latest Process

03 Apr 18:05 EXITED currently processing 0