Harvest for
Reid et al
Created
07 Apr 10:56
Stage:
completed
Fetched:
07 Apr 10:56
Validated:
07 Apr 10:56
Deltas Created
07 Apr 10:56
Units Normalized:
07 Apr 10:56
Ancestry Built:
07 Apr 10:56
Nodes Matched:
07 Apr 10:56
Names Parsed:
07 Apr 10:56
New Models Stored:
07 Apr 10:56
Indexed:
07 Apr 10:56
Completed:
07 Apr 10:58
Time to Harvest:
less than a minute
Harvesting Log
(1376 lines)
(showing only the last 1000 lines, see /app/public/data/reid_et_al/process.log for the full file)
[ERR] [2020-11-05 13:49:49] ../models/csv_parser.rb:28:in `block in line_at_a_time'
[ERR] [2020-11-05 13:49:49] ../models/csv_parser.rb:25:in `line_at_a_time'
[ERR] [2020-11-05 13:49:49] ../models/csv_parser.rb:96:in `diff_as_hashes'
[ERR] [2020-11-05 13:49:49] ../models/resource_harvester.rb:313:in `block (2 levels) in parse_diff_and_store'
[ERR] [2020-11-05 13:49:49] ../models/logged_process.rb:62:in `enter_group'
[ERR] [2020-11-05 13:49:49] ../models/resource_harvester.rb:312:in `block in parse_diff_and_store'
[ERR] [2020-11-05 13:49:49] ../models/resource_harvester.rb:700:in `block in each_diff'
[ERR] [2020-11-05 13:49:49] ../models/resource_harvester.rb:689:in `each_diff'
[ERR] [2020-11-05 13:49:49] ../models/resource_harvester.rb:307:in `parse_diff_and_store'
[ERR] [2020-11-05 13:49:49] ../models/resource_harvester.rb:85:in `block (3 levels) in start'
[ERR] [2020-11-05 13:49:49] ../models/logged_process.rb:19:in `run_step'
[ERR] [2020-11-05 13:49:49] ../models/resource_harvester.rb:85:in `block (2 levels) in start'
[ERR] [2020-11-05 13:49:49] ../models/resource_harvester.rb:74:in `each_key'
[ERR] [2020-11-05 13:49:49] ../models/resource_harvester.rb:74:in `block in start'
[ERR] [2020-11-05 13:49:49] ../models/resource.rb:151:in `lock'
[ERR] [2020-11-05 13:49:49] ../models/resource_harvester.rb:72:in `start'
[ERR] [2020-11-05 13:49:49] ../models/resource.rb:232:in `harvest'
[ERR] [2020-11-05 13:49:49] ../models/resource.rb:208:in `re_download_opendata_and_harvest'
[ERR] [2020-11-05 13:49:49] bin/rails:4:in `require'
[ERR] [2020-11-05 13:49:49] bin/rails:4:in `<main>'
[STOP] [2020-11-05 13:49:49] logged process, took 10.67
[INFO] [2020-11-09 13:46:42] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-11-09 13:46:43] ## remove_type: ScientificName
[INFO] [2020-11-09 13:46:43] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 13:46:43] [13:46:43.886] Removed 0 Scientificnames
[INFO] [2020-11-09 13:46:43] ## remove_type: Vernacular
[INFO] [2020-11-09 13:46:43] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 13:46:43] [13:46:43.889] Removed 0 Vernaculars
[INFO] [2020-11-09 13:46:43] ## remove_type: Article
[INFO] [2020-11-09 13:46:43] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 13:46:43] [13:46:43.892] Removed 0 Articles
[INFO] [2020-11-09 13:46:43] ## remove_type: Medium
[INFO] [2020-11-09 13:46:43] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 13:46:43] [13:46:43.895] Removed 0 Media
[INFO] [2020-11-09 13:46:43] ## remove_type: Trait
[INFO] [2020-11-09 13:46:43] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 13:46:43] [13:46:43.899] Removed 0 Traits
[INFO] [2020-11-09 13:46:43] ## remove_type: MetaTrait
[INFO] [2020-11-09 13:46:43] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 13:46:43] [13:46:43.906] Removed 0 Metatraits
[INFO] [2020-11-09 13:46:43] ## remove_type: OccurrenceMetadatum
[INFO] [2020-11-09 13:46:43] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 13:46:43] [13:46:43.919] Removed 0 Occurrencemetadata
[INFO] [2020-11-09 13:46:43] ## remove_type: Assoc
[INFO] [2020-11-09 13:46:43] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 13:46:43] [13:46:43.922] Removed 0 Assocs
[INFO] [2020-11-09 13:46:43] ## remove_type: MetaAssoc
[INFO] [2020-11-09 13:46:43] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 13:46:43] [13:46:43.925] Removed 0 Metaassocs
[INFO] [2020-11-09 13:46:43] ## remove_type: Identifier
[INFO] [2020-11-09 13:46:43] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 13:46:43] [13:46:43.927] Removed 0 Identifiers
[INFO] [2020-11-09 13:46:43] ## remove_type: Reference
[INFO] [2020-11-09 13:46:43] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 13:46:43] [13:46:43.929] Removed 0 References
[INFO] [2020-11-09 13:46:43] ## remove_type: Node
[INFO] [2020-11-09 13:46:43] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 13:46:43] [13:46:43.963] Removed 0 Nodes
[START] [2020-11-09 13:46:44] logged process: 9719e9b2443e9b331c213f6700b6e76a1b7c8548
[START] [2020-11-09 13:46:44] Creating resource from OpenData
[START] [2020-11-09 13:46:44] logged process: 9719e9b2443e9b331c213f6700b6e76a1b7c8548
[START] [2020-11-09 13:46:44] Parse meta.xml file and create formats with fields
[WARN] [2020-11-09 13:46:45] (common) IGNORED (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2020-11-09 13:46:45] (common) IGNORED (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2020-11-09 13:46:45] (common) IGNORED (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2020-11-09 13:46:45] (common) IGNORED (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2020-11-09 13:46:45] (common) IGNORED (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2020-11-09 13:46:45] (common) IGNORED (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2020-11-09 13:46:45] (common) IGNORED (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[WARN] [2020-11-09 13:46:45] SKIPPING http://rs.tdwg.org/dwc/terms/Event config (events.txt)...
[STOP] [2020-11-09 13:46:45] Parse meta.xml file and create formats with fields
[STOP] [2020-11-09 13:46:45] Creating resource from OpenData
[START] [2020-11-09 13:46:45] logged process: 9719e9b2443e9b331c213f6700b6e76a1b7c8548
[START] [2020-11-09 13:46:45] create_harvest_instance
[STOP] [2020-11-09 13:46:54] create_harvest_instance
[START] [2020-11-09 13:46:54] fetch_files
[STOP] [2020-11-09 13:46:54] fetch_files
[START] [2020-11-09 13:46:54] validate_each_file
[STOP] [2020-11-09 13:46:54] validate_each_file
[START] [2020-11-09 13:46:54] convert_to_csv
[CMD] [2020-11-09 13:46:54] /usr/bin/sort /app/public/converted_csv/reid_et_al_agents_23782.csv > /app/public/converted_csv/reid_et_al_agents_23782.csv_sorted
[CMD] [2020-11-09 13:46:54] /usr/bin/sort /app/public/converted_csv/reid_et_al_refs_23783.csv > /app/public/converted_csv/reid_et_al_refs_23783.csv_sorted
[CMD] [2020-11-09 13:46:54] /usr/bin/sort /app/public/converted_csv/reid_et_al_nodes_23784.csv > /app/public/converted_csv/reid_et_al_nodes_23784.csv_sorted
[CMD] [2020-11-09 13:46:54] /usr/bin/sort /app/public/converted_csv/reid_et_al_media_23785.csv > /app/public/converted_csv/reid_et_al_media_23785.csv_sorted
[CMD] [2020-11-09 13:46:54] /usr/bin/sort /app/public/converted_csv/reid_et_al_vernaculars_23786.csv > /app/public/converted_csv/reid_et_al_vernaculars_23786.csv_sorted
[CMD] [2020-11-09 13:46:54] /usr/bin/sort /app/public/converted_csv/reid_et_al_occurrences_23787.csv > /app/public/converted_csv/reid_et_al_occurrences_23787.csv_sorted
[CMD] [2020-11-09 13:46:54] /usr/bin/sort /app/public/converted_csv/reid_et_al_assocs_23788.csv > /app/public/converted_csv/reid_et_al_assocs_23788.csv_sorted
[CMD] [2020-11-09 13:46:54] /usr/bin/sort /app/public/converted_csv/reid_et_al_measurements_23789.csv > /app/public/converted_csv/reid_et_al_measurements_23789.csv_sorted
[STOP] [2020-11-09 13:46:54] convert_to_csv
[START] [2020-11-09 13:46:54] calculate_delta
[CMD] [2020-11-09 13:46:54] echo "0a" > /app/public/diff/reid_et_al_agents_23782.diff
[CMD] [2020-11-09 13:46:54] tail -n +1 /app/public/converted_csv/reid_et_al_agents_23782.csv >> /app/public/diff/reid_et_al_agents_23782.diff
[CMD] [2020-11-09 13:46:54] echo "." >> /app/public/diff/reid_et_al_agents_23782.diff
[CMD] [2020-11-09 13:46:54] echo "0a" > /app/public/diff/reid_et_al_refs_23783.diff
[CMD] [2020-11-09 13:46:54] tail -n +1 /app/public/converted_csv/reid_et_al_refs_23783.csv >> /app/public/diff/reid_et_al_refs_23783.diff
[CMD] [2020-11-09 13:46:54] echo "." >> /app/public/diff/reid_et_al_refs_23783.diff
[CMD] [2020-11-09 13:46:54] echo "0a" > /app/public/diff/reid_et_al_nodes_23784.diff
[CMD] [2020-11-09 13:46:54] tail -n +1 /app/public/converted_csv/reid_et_al_nodes_23784.csv >> /app/public/diff/reid_et_al_nodes_23784.diff
[CMD] [2020-11-09 13:46:54] echo "." >> /app/public/diff/reid_et_al_nodes_23784.diff
[CMD] [2020-11-09 13:46:54] echo "0a" > /app/public/diff/reid_et_al_media_23785.diff
[CMD] [2020-11-09 13:46:54] tail -n +1 /app/public/converted_csv/reid_et_al_media_23785.csv >> /app/public/diff/reid_et_al_media_23785.diff
[CMD] [2020-11-09 13:46:54] echo "." >> /app/public/diff/reid_et_al_media_23785.diff
[CMD] [2020-11-09 13:46:54] echo "0a" > /app/public/diff/reid_et_al_vernaculars_23786.diff
[CMD] [2020-11-09 13:46:54] tail -n +1 /app/public/converted_csv/reid_et_al_vernaculars_23786.csv >> /app/public/diff/reid_et_al_vernaculars_23786.diff
[CMD] [2020-11-09 13:46:54] echo "." >> /app/public/diff/reid_et_al_vernaculars_23786.diff
[CMD] [2020-11-09 13:46:54] echo "0a" > /app/public/diff/reid_et_al_occurrences_23787.diff
[CMD] [2020-11-09 13:46:54] tail -n +1 /app/public/converted_csv/reid_et_al_occurrences_23787.csv >> /app/public/diff/reid_et_al_occurrences_23787.diff
[CMD] [2020-11-09 13:46:54] echo "." >> /app/public/diff/reid_et_al_occurrences_23787.diff
[CMD] [2020-11-09 13:46:54] echo "0a" > /app/public/diff/reid_et_al_assocs_23788.diff
[CMD] [2020-11-09 13:46:54] tail -n +1 /app/public/converted_csv/reid_et_al_assocs_23788.csv >> /app/public/diff/reid_et_al_assocs_23788.diff
[CMD] [2020-11-09 13:46:54] echo "." >> /app/public/diff/reid_et_al_assocs_23788.diff
[CMD] [2020-11-09 13:46:54] echo "0a" > /app/public/diff/reid_et_al_measurements_23789.diff
[CMD] [2020-11-09 13:46:54] tail -n +1 /app/public/converted_csv/reid_et_al_measurements_23789.csv >> /app/public/diff/reid_et_al_measurements_23789.diff
[CMD] [2020-11-09 13:46:54] echo "." >> /app/public/diff/reid_et_al_measurements_23789.diff
[STOP] [2020-11-09 13:46:54] calculate_delta
[START] [2020-11-09 13:46:54] parse_diff_and_store
[INFO] [2020-11-09 13:46:54] Loading agents diff file into memory (true lines)...
[INFO] [2020-11-09 13:46:54] Loading refs diff file into memory (true lines)...
[INFO] [2020-11-09 13:46:54] Loading nodes diff file into memory (true lines)...
[INFO] [2020-11-09 13:46:54] Loading media diff file into memory (true lines)...
[INFO] [2020-11-09 13:46:54] Loading vernaculars diff file into memory (true lines)...
[INFO] [2020-11-09 13:46:55] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-11-09 13:46:56] Loading assocs diff file into memory (true lines)...
[INFO] [2020-11-09 13:46:56] Loading measurements diff file into memory (true lines)...
[STOP] [2020-11-09 13:46:56] parse_diff_and_store
[ERR] [2020-11-09 13:46:56] RuntimeError
[ERR] [2020-11-09 13:46:56] Missing Term for URI `https://eol.org/schema/terms/arms`, must be added!
[ERR] [2020-11-09 13:46:56] ../models/store/model_builder.rb:635:in `fail_on_bad_uri'
[ERR] [2020-11-09 13:46:56] ../models/store/model_builder.rb:589:in `convert_trait_value'
[ERR] [2020-11-09 13:46:56] ../models/store/model_builder.rb:415:in `build_trait'
[ERR] [2020-11-09 13:46:56] ../models/store/model_builder.rb:28:in `build_models'
[ERR] [2020-11-09 13:46:56] ../models/resource_harvester.rb:357:in `block (3 levels) in parse_diff_and_store'
[ERR] [2020-11-09 13:46:56] ../models/csv_parser.rb:111:in `block in diff_as_hashes'
[ERR] [2020-11-09 13:46:56] ../models/csv_parser.rb:28:in `block in line_at_a_time'
[ERR] [2020-11-09 13:46:56] ../models/csv_parser.rb:25:in `line_at_a_time'
[ERR] [2020-11-09 13:46:56] ../models/csv_parser.rb:96:in `diff_as_hashes'
[ERR] [2020-11-09 13:46:56] ../models/resource_harvester.rb:313:in `block (2 levels) in parse_diff_and_store'
[ERR] [2020-11-09 13:46:56] ../models/logged_process.rb:62:in `enter_group'
[ERR] [2020-11-09 13:46:56] ../models/resource_harvester.rb:312:in `block in parse_diff_and_store'
[ERR] [2020-11-09 13:46:56] ../models/resource_harvester.rb:700:in `block in each_diff'
[ERR] [2020-11-09 13:46:56] ../models/resource_harvester.rb:689:in `each_diff'
[ERR] [2020-11-09 13:46:56] ../models/resource_harvester.rb:307:in `parse_diff_and_store'
[ERR] [2020-11-09 13:46:56] ../models/resource_harvester.rb:85:in `block (3 levels) in start'
[ERR] [2020-11-09 13:46:56] ../models/logged_process.rb:19:in `run_step'
[ERR] [2020-11-09 13:46:56] ../models/resource_harvester.rb:85:in `block (2 levels) in start'
[ERR] [2020-11-09 13:46:56] ../models/resource_harvester.rb:74:in `each_key'
[ERR] [2020-11-09 13:46:56] ../models/resource_harvester.rb:74:in `block in start'
[ERR] [2020-11-09 13:46:56] ../models/resource.rb:151:in `lock'
[ERR] [2020-11-09 13:46:56] ../models/resource_harvester.rb:72:in `start'
[ERR] [2020-11-09 13:46:56] ../models/resource.rb:232:in `harvest'
[ERR] [2020-11-09 13:46:56] ../models/resource.rb:208:in `re_download_opendata_and_harvest'
[ERR] [2020-11-09 13:46:56] bin/rails:4:in `require'
[ERR] [2020-11-09 13:46:56] bin/rails:4:in `<main>'
[STOP] [2020-11-09 13:46:56] logged process, took 11.02
[INFO] [2020-11-09 15:11:14] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-11-09 15:11:18] ## remove_type: ScientificName
[INFO] [2020-11-09 15:11:18] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 15:11:18] [15:11:18.944] Removed 0 Scientificnames
[INFO] [2020-11-09 15:11:18] ## remove_type: Vernacular
[INFO] [2020-11-09 15:11:18] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 15:11:18] [15:11:18.947] Removed 0 Vernaculars
[INFO] [2020-11-09 15:11:18] ## remove_type: Article
[INFO] [2020-11-09 15:11:18] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 15:11:18] [15:11:18.950] Removed 0 Articles
[INFO] [2020-11-09 15:11:18] ## remove_type: Medium
[INFO] [2020-11-09 15:11:18] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 15:11:18] [15:11:18.953] Removed 0 Media
[INFO] [2020-11-09 15:11:18] ## remove_type: Trait
[INFO] [2020-11-09 15:11:18] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 15:11:18] [15:11:18.957] Removed 0 Traits
[INFO] [2020-11-09 15:11:18] ## remove_type: MetaTrait
[INFO] [2020-11-09 15:11:18] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 15:11:18] [15:11:18.960] Removed 0 Metatraits
[INFO] [2020-11-09 15:11:18] ## remove_type: OccurrenceMetadatum
[INFO] [2020-11-09 15:11:18] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 15:11:18] [15:11:18.963] Removed 0 Occurrencemetadata
[INFO] [2020-11-09 15:11:18] ## remove_type: Assoc
[INFO] [2020-11-09 15:11:18] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 15:11:18] [15:11:18.965] Removed 0 Assocs
[INFO] [2020-11-09 15:11:18] ## remove_type: MetaAssoc
[INFO] [2020-11-09 15:11:18] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 15:11:18] [15:11:18.968] Removed 0 Metaassocs
[INFO] [2020-11-09 15:11:18] ## remove_type: Identifier
[INFO] [2020-11-09 15:11:18] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 15:11:18] [15:11:18.971] Removed 0 Identifiers
[INFO] [2020-11-09 15:11:18] ## remove_type: Reference
[INFO] [2020-11-09 15:11:18] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 15:11:18] [15:11:18.974] Removed 0 References
[INFO] [2020-11-09 15:11:19] ## remove_type: Node
[INFO] [2020-11-09 15:11:19] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 15:11:19] [15:11:19.246] Removed 0 Nodes
[START] [2020-11-09 15:11:19] logged process: 9719e9b2443e9b331c213f6700b6e76a1b7c8548
[START] [2020-11-09 15:11:19] Creating resource from OpenData
[START] [2020-11-09 15:11:20] logged process: 9719e9b2443e9b331c213f6700b6e76a1b7c8548
[START] [2020-11-09 15:11:20] Parse meta.xml file and create formats with fields
[WARN] [2020-11-09 15:11:20] (common) IGNORED (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2020-11-09 15:11:20] (common) IGNORED (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2020-11-09 15:11:20] (common) IGNORED (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2020-11-09 15:11:20] (common) IGNORED (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2020-11-09 15:11:20] (common) IGNORED (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2020-11-09 15:11:20] (common) IGNORED (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2020-11-09 15:11:20] (common) IGNORED (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[WARN] [2020-11-09 15:11:20] SKIPPING http://rs.tdwg.org/dwc/terms/Event config (events.txt)...
[STOP] [2020-11-09 15:11:20] Parse meta.xml file and create formats with fields
[STOP] [2020-11-09 15:11:20] Creating resource from OpenData
[START] [2020-11-09 15:11:20] logged process: 9719e9b2443e9b331c213f6700b6e76a1b7c8548
[START] [2020-11-09 15:11:20] create_harvest_instance
[STOP] [2020-11-09 15:11:29] create_harvest_instance
[START] [2020-11-09 15:11:29] fetch_files
[STOP] [2020-11-09 15:11:29] fetch_files
[START] [2020-11-09 15:11:29] validate_each_file
[STOP] [2020-11-09 15:11:29] validate_each_file
[START] [2020-11-09 15:11:29] convert_to_csv
[CMD] [2020-11-09 15:11:29] /usr/bin/sort /app/public/converted_csv/reid_et_al_agents_23868.csv > /app/public/converted_csv/reid_et_al_agents_23868.csv_sorted
[CMD] [2020-11-09 15:11:29] /usr/bin/sort /app/public/converted_csv/reid_et_al_refs_23869.csv > /app/public/converted_csv/reid_et_al_refs_23869.csv_sorted
[CMD] [2020-11-09 15:11:29] /usr/bin/sort /app/public/converted_csv/reid_et_al_nodes_23870.csv > /app/public/converted_csv/reid_et_al_nodes_23870.csv_sorted
[CMD] [2020-11-09 15:11:29] /usr/bin/sort /app/public/converted_csv/reid_et_al_media_23871.csv > /app/public/converted_csv/reid_et_al_media_23871.csv_sorted
[CMD] [2020-11-09 15:11:29] /usr/bin/sort /app/public/converted_csv/reid_et_al_vernaculars_23872.csv > /app/public/converted_csv/reid_et_al_vernaculars_23872.csv_sorted
[CMD] [2020-11-09 15:11:29] /usr/bin/sort /app/public/converted_csv/reid_et_al_occurrences_23873.csv > /app/public/converted_csv/reid_et_al_occurrences_23873.csv_sorted
[CMD] [2020-11-09 15:11:29] /usr/bin/sort /app/public/converted_csv/reid_et_al_assocs_23874.csv > /app/public/converted_csv/reid_et_al_assocs_23874.csv_sorted
[CMD] [2020-11-09 15:11:29] /usr/bin/sort /app/public/converted_csv/reid_et_al_measurements_23875.csv > /app/public/converted_csv/reid_et_al_measurements_23875.csv_sorted
[STOP] [2020-11-09 15:11:29] convert_to_csv
[START] [2020-11-09 15:11:29] calculate_delta
[CMD] [2020-11-09 15:11:29] echo "0a" > /app/public/diff/reid_et_al_agents_23868.diff
[CMD] [2020-11-09 15:11:29] tail -n +1 /app/public/converted_csv/reid_et_al_agents_23868.csv >> /app/public/diff/reid_et_al_agents_23868.diff
[CMD] [2020-11-09 15:11:29] echo "." >> /app/public/diff/reid_et_al_agents_23868.diff
[CMD] [2020-11-09 15:11:29] echo "0a" > /app/public/diff/reid_et_al_refs_23869.diff
[CMD] [2020-11-09 15:11:29] tail -n +1 /app/public/converted_csv/reid_et_al_refs_23869.csv >> /app/public/diff/reid_et_al_refs_23869.diff
[CMD] [2020-11-09 15:11:29] echo "." >> /app/public/diff/reid_et_al_refs_23869.diff
[CMD] [2020-11-09 15:11:29] echo "0a" > /app/public/diff/reid_et_al_nodes_23870.diff
[CMD] [2020-11-09 15:11:29] tail -n +1 /app/public/converted_csv/reid_et_al_nodes_23870.csv >> /app/public/diff/reid_et_al_nodes_23870.diff
[CMD] [2020-11-09 15:11:29] echo "." >> /app/public/diff/reid_et_al_nodes_23870.diff
[CMD] [2020-11-09 15:11:29] echo "0a" > /app/public/diff/reid_et_al_media_23871.diff
[CMD] [2020-11-09 15:11:29] tail -n +1 /app/public/converted_csv/reid_et_al_media_23871.csv >> /app/public/diff/reid_et_al_media_23871.diff
[CMD] [2020-11-09 15:11:29] echo "." >> /app/public/diff/reid_et_al_media_23871.diff
[CMD] [2020-11-09 15:11:29] echo "0a" > /app/public/diff/reid_et_al_vernaculars_23872.diff
[CMD] [2020-11-09 15:11:29] tail -n +1 /app/public/converted_csv/reid_et_al_vernaculars_23872.csv >> /app/public/diff/reid_et_al_vernaculars_23872.diff
[CMD] [2020-11-09 15:11:29] echo "." >> /app/public/diff/reid_et_al_vernaculars_23872.diff
[CMD] [2020-11-09 15:11:29] echo "0a" > /app/public/diff/reid_et_al_occurrences_23873.diff
[CMD] [2020-11-09 15:11:29] tail -n +1 /app/public/converted_csv/reid_et_al_occurrences_23873.csv >> /app/public/diff/reid_et_al_occurrences_23873.diff
[CMD] [2020-11-09 15:11:29] echo "." >> /app/public/diff/reid_et_al_occurrences_23873.diff
[CMD] [2020-11-09 15:11:30] echo "0a" > /app/public/diff/reid_et_al_assocs_23874.diff
[CMD] [2020-11-09 15:11:30] tail -n +1 /app/public/converted_csv/reid_et_al_assocs_23874.csv >> /app/public/diff/reid_et_al_assocs_23874.diff
[CMD] [2020-11-09 15:11:30] echo "." >> /app/public/diff/reid_et_al_assocs_23874.diff
[CMD] [2020-11-09 15:11:30] echo "0a" > /app/public/diff/reid_et_al_measurements_23875.diff
[CMD] [2020-11-09 15:11:30] tail -n +1 /app/public/converted_csv/reid_et_al_measurements_23875.csv >> /app/public/diff/reid_et_al_measurements_23875.diff
[CMD] [2020-11-09 15:11:30] echo "." >> /app/public/diff/reid_et_al_measurements_23875.diff
[STOP] [2020-11-09 15:11:30] calculate_delta
[START] [2020-11-09 15:11:30] parse_diff_and_store
[INFO] [2020-11-09 15:11:30] Loading agents diff file into memory (true lines)...
[INFO] [2020-11-09 15:11:30] Loading refs diff file into memory (true lines)...
[INFO] [2020-11-09 15:11:30] Loading nodes diff file into memory (true lines)...
[INFO] [2020-11-09 15:11:30] Loading media diff file into memory (true lines)...
[INFO] [2020-11-09 15:11:30] Loading vernaculars diff file into memory (true lines)...
[INFO] [2020-11-09 15:11:30] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-11-09 15:11:31] Loading assocs diff file into memory (true lines)...
[INFO] [2020-11-09 15:11:31] Loading measurements diff file into memory (true lines)...
[INFO] [2020-11-09 15:11:31] Storing 164 ScientificNames
[INFO] [2020-11-09 15:11:31] Processing group of 164 in 1 groups of 1000
[INFO] [2020-11-09 15:11:31] Average Time: 0.09
[INFO] [2020-11-09 15:11:31] Total Time: 1s
[INFO] [2020-11-09 15:11:31] Storing 164 Nodes
[INFO] [2020-11-09 15:11:31] Processing group of 164 in 1 groups of 1000
[INFO] [2020-11-09 15:11:31] Average Time: 0.1
[INFO] [2020-11-09 15:11:31] Total Time: 1s
[INFO] [2020-11-09 15:11:31] Storing 80 Occurrences
[INFO] [2020-11-09 15:11:31] Processing group of 80 in 1 groups of 1000
[INFO] [2020-11-09 15:11:31] Average Time: 0.01
[INFO] [2020-11-09 15:11:31] Total Time: 1s
[INFO] [2020-11-09 15:11:31] Storing 559 OccurrenceMetadata
[INFO] [2020-11-09 15:11:31] Processing group of 559 in 1 groups of 1000
[INFO] [2020-11-09 15:11:31] Average Time: 0.06
[INFO] [2020-11-09 15:11:31] Total Time: 1s
[INFO] [2020-11-09 15:11:31] Storing 86 Traits
[INFO] [2020-11-09 15:11:31] Processing group of 86 in 1 groups of 1000
[INFO] [2020-11-09 15:11:31] Average Time: 0.02
[INFO] [2020-11-09 15:11:31] Total Time: 1s
[INFO] [2020-11-09 15:11:31] Storing 86 MetaTraits
[INFO] [2020-11-09 15:11:31] Processing group of 86 in 1 groups of 1000
[INFO] [2020-11-09 15:11:31] Average Time: 0.01
[INFO] [2020-11-09 15:11:31] Total Time: 1s
[STOP] [2020-11-09 15:11:31] parse_diff_and_store
[START] [2020-11-09 15:11:31] resolve_keys
[INFO] [2020-11-09 15:11:38] Occurrences to nodes (through scientific_names)...
[INFO] [2020-11-09 15:11:38] traits to occurrences...
[INFO] [2020-11-09 15:11:38] traits to nodes (through occurrences)...
[INFO] [2020-11-09 15:11:38] Traits to sex term...
[INFO] [2020-11-09 15:11:38] Traits to lifestage term...
[INFO] [2020-11-09 15:11:38] MetaTraits to traits...
[INFO] [2020-11-09 15:11:38] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-11-09 15:11:38] Assocs to occurrences...
[INFO] [2020-11-09 15:11:38] Assocs to nodes...
[INFO] [2020-11-09 15:11:38] Assoc to sex term...
[INFO] [2020-11-09 15:11:38] Assoc to lifestage term...
[INFO] [2020-11-09 15:11:38] MetaAssoc to assocs...
[STOP] [2020-11-09 15:11:38] resolve_keys
[START] [2020-11-09 15:11:38] hold_for_later_1
[STOP] [2020-11-09 15:11:38] hold_for_later_1
[START] [2020-11-09 15:11:38] hold_for_later_2
[STOP] [2020-11-09 15:11:38] hold_for_later_2
[START] [2020-11-09 15:11:38] resolve_missing_parents
[STOP] [2020-11-09 15:11:38] resolve_missing_parents
[START] [2020-11-09 15:11:38] rebuild_nodes
[START] [2020-11-09 15:11:38] Flattener#flatten
[START] [2020-11-09 15:11:38] Flattener#study_resource
[START] [2020-11-09 15:11:38] Flattener#build_ancestry
[STOP] [2020-11-09 15:11:38] Flattener#build_ancestry
[INFO] [2020-11-09 15:11:38] 164 ancestry keys
[START] [2020-11-09 15:11:38] build_node_ancestors
[INFO] [2020-11-09 15:11:38] old ancestors deleted.
[STOP] [2020-11-09 15:11:38] build_node_ancestors
[START] [2020-11-09 15:11:38] Flattener#propagate_ancestor_ids
[STOP] [2020-11-09 15:11:38] Flattener#propagate_ancestor_ids
[STOP] [2020-11-09 15:11:38] Flattener#flatten
[STOP] [2020-11-09 15:11:38] rebuild_nodes
[START] [2020-11-09 15:11:38] resolve_missing_media_owners
[STOP] [2020-11-09 15:11:38] resolve_missing_media_owners
[START] [2020-11-09 15:11:38] sanitize_media_verbatims
[STOP] [2020-11-09 15:11:38] sanitize_media_verbatims
[START] [2020-11-09 15:11:38] queue_downloads
[STOP] [2020-11-09 15:11:38] queue_downloads
[START] [2020-11-09 15:11:38] parse_names
[WARN] [2020-11-09 15:11:38] I see 164 names which still need to be parsed.
[WARN] [2020-11-09 15:11:40] I see 8 names which still need to be parsed.
[STOP] [2020-11-09 15:11:41] parse_names
[START] [2020-11-09 15:11:41] denormalize_canonical_names_to_nodes
[STOP] [2020-11-09 15:11:41] denormalize_canonical_names_to_nodes
[START] [2020-11-09 15:11:41] match_nodes
[START] [2020-11-09 15:11:41] map_all_nodes_to_pages
[STOP] [2020-11-09 15:12:01] map_all_nodes_to_pages
[INFO] [2020-11-09 15:12:01] Unmatched nodes (10 of 164): Acanthephyridae (#81597337); Munidopsidae (#81597435); Maxillopoda (#81597443); Histiobranchius (#81597420); Histiobranchius bathybius (#81597421); Porcellanaster cerulus (#81597461); Hymenaster cf_coccinatus (#81597427); Aspidochirotida (#81597377); Golfingiida (#81597475); Polythalamea (#81597468)
[START] [2020-11-09 15:12:01] update_nodes
[STOP] [2020-11-09 15:12:01] update_nodes
[STOP] [2020-11-09 15:12:01] match_nodes
[START] [2020-11-09 15:12:01] reindex_search
[STOP] [2020-11-09 15:12:02] reindex_search
[START] [2020-11-09 15:12:02] normalize_units
[STOP] [2020-11-09 15:12:02] normalize_units
[START] [2020-11-09 15:12:02] calculate_statistics
[STOP] [2020-11-09 15:12:02] calculate_statistics
[START] [2020-11-09 15:12:02] complete_harvest_instance
[START] [2020-11-09 15:12:02] overall_tsv_creation
[INFO] [2020-11-09 15:12:02] Processing group of 164 in 1 batches of 10000
[INFO] [2020-11-09 15:12:44] 86 Traits (unfiltered)...
[INFO] [2020-11-09 15:13:22] 86 Traits (filtered)...
[INFO] [2020-11-09 15:13:22] 0 Associations (filtered)...
[INFO] [2020-11-09 15:13:22] 687 metadata added.
[INFO] [2020-11-09 15:13:22] 0 metadata added.
[INFO] [2020-11-09 15:13:22] Average Time: 54.02
[INFO] [2020-11-09 15:13:22] Total Time: 1m21s
[STOP] [2020-11-09 15:13:22] overall_tsv_creation
[INFO] [2020-11-09 15:13:22] Done. Check your files:
[INFO] [2020-11-09 15:13:22] (156 lines) /app/public/data/reid_et_al/publish_nodes.tsv
[INFO] [2020-11-09 15:13:22] (667 lines) /app/public/data/reid_et_al/publish_node_ancestors.tsv
[INFO] [2020-11-09 15:13:22] (164 lines) /app/public/data/reid_et_al/publish_scientific_names.tsv
[INFO] [2020-11-09 15:13:22] (87 lines) /app/public/data/reid_et_al/publish_traits.tsv
[INFO] [2020-11-09 15:13:22] (688 lines) /app/public/data/reid_et_al/publish_metadata.tsv
[STOP] [2020-11-09 15:13:22] complete_harvest_instance
[START] [2020-11-09 15:13:22] completed
[STOP] [2020-11-09 15:13:22] completed
[STOP] [2020-11-09 15:13:22] logged process, took 122.12
[INFO] [2021-04-07 10:25:51] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-07 10:25:54] ## remove_type: ScientificName
[INFO] [2021-04-07 10:25:54] ++ Calling delete_all on 164 instances...
[INFO] [2021-04-07 10:25:54] [10:25:54.451] Removed 164 Scientificnames
[INFO] [2021-04-07 10:25:54] ## remove_type: Vernacular
[INFO] [2021-04-07 10:25:54] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:25:54] [10:25:54.453] Removed 0 Vernaculars
[INFO] [2021-04-07 10:25:54] ## remove_type: Article
[INFO] [2021-04-07 10:25:54] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:25:54] [10:25:54.455] Removed 0 Articles
[INFO] [2021-04-07 10:25:54] ## remove_type: Medium
[INFO] [2021-04-07 10:25:54] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:25:54] [10:25:54.457] Removed 0 Media
[INFO] [2021-04-07 10:25:54] ## remove_type: Trait
[INFO] [2021-04-07 10:25:54] ++ Calling delete_all on 86 instances...
[INFO] [2021-04-07 10:25:54] [10:25:54.463] Removed 86 Traits
[INFO] [2021-04-07 10:25:54] ## remove_type: MetaTrait
[INFO] [2021-04-07 10:25:54] ++ Calling delete_all on 86 instances...
[INFO] [2021-04-07 10:25:54] [10:25:54.498] Removed 86 Metatraits
[INFO] [2021-04-07 10:25:54] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-07 10:25:54] ++ Calling delete_all on 559 instances...
[INFO] [2021-04-07 10:25:54] [10:25:54.533] Removed 559 Occurrencemetadata
[INFO] [2021-04-07 10:25:54] ## remove_type: Assoc
[INFO] [2021-04-07 10:25:54] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:25:54] [10:25:54.535] Removed 0 Assocs
[INFO] [2021-04-07 10:25:54] ## remove_type: MetaAssoc
[INFO] [2021-04-07 10:25:54] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:25:54] [10:25:54.537] Removed 0 Metaassocs
[INFO] [2021-04-07 10:25:54] ## remove_type: Identifier
[INFO] [2021-04-07 10:25:54] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:25:54] [10:25:54.538] Removed 0 Identifiers
[INFO] [2021-04-07 10:25:54] ## remove_type: Reference
[INFO] [2021-04-07 10:25:54] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:25:54] [10:25:54.540] Removed 0 References
[INFO] [2021-04-07 10:25:54] Starting batch with ID 81597346...
[INFO] [2021-04-07 10:25:55] ## remove_type: Node
[INFO] [2021-04-07 10:25:55] ++ Calling delete_all on 164 instances...
[INFO] [2021-04-07 10:25:55] [10:25:55.137] Removed 164 Nodes
[START] [2021-04-07 10:25:55] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 10:25:55] Creating resource from OpenData
[START] [2021-04-07 10:25:57] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 10:25:57] Parse meta.xml file and create formats with fields
[STOP] [2021-04-07 10:26:09] Parse meta.xml file and create formats with fields
[STOP] [2021-04-07 10:26:09] Creating resource from OpenData
[INFO] [2021-04-07 10:32:05] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-07 10:40:01] ## remove_type: ScientificName
[INFO] [2021-04-07 10:40:01] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:40:01] [10:40:01.018] Removed 0 Scientificnames
[INFO] [2021-04-07 10:40:01] ## remove_type: Vernacular
[INFO] [2021-04-07 10:40:01] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:40:01] [10:40:01.019] Removed 0 Vernaculars
[INFO] [2021-04-07 10:40:01] ## remove_type: Article
[INFO] [2021-04-07 10:40:01] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:40:01] [10:40:01.020] Removed 0 Articles
[INFO] [2021-04-07 10:40:01] ## remove_type: Medium
[INFO] [2021-04-07 10:40:01] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:40:01] [10:40:01.022] Removed 0 Media
[INFO] [2021-04-07 10:40:01] ## remove_type: Trait
[INFO] [2021-04-07 10:40:01] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:40:01] [10:40:01.023] Removed 0 Traits
[INFO] [2021-04-07 10:40:01] ## remove_type: MetaTrait
[INFO] [2021-04-07 10:40:01] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:40:01] [10:40:01.024] Removed 0 Metatraits
[INFO] [2021-04-07 10:40:01] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-07 10:40:01] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:40:01] [10:40:01.025] Removed 0 Occurrencemetadata
[INFO] [2021-04-07 10:40:01] ## remove_type: Assoc
[INFO] [2021-04-07 10:40:01] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:40:01] [10:40:01.026] Removed 0 Assocs
[INFO] [2021-04-07 10:40:01] ## remove_type: MetaAssoc
[INFO] [2021-04-07 10:40:01] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:40:01] [10:40:01.028] Removed 0 Metaassocs
[INFO] [2021-04-07 10:40:01] ## remove_type: Identifier
[INFO] [2021-04-07 10:40:01] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:40:01] [10:40:01.029] Removed 0 Identifiers
[INFO] [2021-04-07 10:40:01] ## remove_type: Reference
[INFO] [2021-04-07 10:40:01] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:40:01] [10:40:01.030] Removed 0 References
[INFO] [2021-04-07 10:40:01] ## remove_type: Node
[INFO] [2021-04-07 10:40:01] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:40:01] [10:40:01.040] Removed 0 Nodes
[START] [2021-04-07 10:40:01] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 10:40:01] Creating resource from OpenData
[START] [2021-04-07 10:40:02] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 10:40:02] Parse meta.xml file and create formats with fields
[STOP] [2021-04-07 10:40:16] Parse meta.xml file and create formats with fields
[STOP] [2021-04-07 10:40:16] Creating resource from OpenData
[INFO] [2021-04-07 10:43:50] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-07 10:43:50] ## remove_type: ScientificName
[INFO] [2021-04-07 10:43:50] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:43:50] [10:43:50.826] Removed 0 Scientificnames
[INFO] [2021-04-07 10:43:50] ## remove_type: Vernacular
[INFO] [2021-04-07 10:43:50] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:43:50] [10:43:50.827] Removed 0 Vernaculars
[INFO] [2021-04-07 10:43:50] ## remove_type: Article
[INFO] [2021-04-07 10:43:50] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:43:50] [10:43:50.828] Removed 0 Articles
[INFO] [2021-04-07 10:43:50] ## remove_type: Medium
[INFO] [2021-04-07 10:43:50] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:43:50] [10:43:50.830] Removed 0 Media
[INFO] [2021-04-07 10:43:50] ## remove_type: Trait
[INFO] [2021-04-07 10:43:50] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:43:50] [10:43:50.831] Removed 0 Traits
[INFO] [2021-04-07 10:43:50] ## remove_type: MetaTrait
[INFO] [2021-04-07 10:43:50] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:43:50] [10:43:50.833] Removed 0 Metatraits
[INFO] [2021-04-07 10:43:50] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-07 10:43:50] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:43:50] [10:43:50.834] Removed 0 Occurrencemetadata
[INFO] [2021-04-07 10:43:50] ## remove_type: Assoc
[INFO] [2021-04-07 10:43:50] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:43:50] [10:43:50.836] Removed 0 Assocs
[INFO] [2021-04-07 10:43:50] ## remove_type: MetaAssoc
[INFO] [2021-04-07 10:43:50] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:43:50] [10:43:50.837] Removed 0 Metaassocs
[INFO] [2021-04-07 10:43:50] ## remove_type: Identifier
[INFO] [2021-04-07 10:43:50] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:43:50] [10:43:50.838] Removed 0 Identifiers
[INFO] [2021-04-07 10:43:50] ## remove_type: Reference
[INFO] [2021-04-07 10:43:50] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:43:50] [10:43:50.840] Removed 0 References
[INFO] [2021-04-07 10:43:50] ## remove_type: Node
[INFO] [2021-04-07 10:43:50] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:43:50] [10:43:50.850] Removed 0 Nodes
[START] [2021-04-07 10:43:51] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 10:43:51] Creating resource from OpenData
[START] [2021-04-07 10:43:52] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 10:43:52] Parse meta.xml file and create formats with fields
[WARN] [2021-04-07 10:44:05] (common) IGNORED (media) field header: ThumbnailURL term: http://eol.org/schema/media/thumbnailURL
[WARN] [2021-04-07 10:44:05] (common) IGNORED (media) field header: DerivedFrom term: http://rs.tdwg.org/ac/terms/derivedFrom
[WARN] [2021-04-07 10:44:05] (common) IGNORED (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2021-04-07 10:44:05] (common) IGNORED (media) field header: Modified term: http://purl.org/dc/terms/modified
[WARN] [2021-04-07 10:44:05] (common) IGNORED (media) field header: Rating term: http://ns.adobe.com/xap/1.0/Rating
[WARN] [2021-04-07 10:44:05] (common) IGNORED (media) field header: Audience term: http://purl.org/dc/terms/audience
[WARN] [2021-04-07 10:44:05] (common) IGNORED (media) field header: Rights term: http://purl.org/dc/terms/rights
[WARN] [2021-04-07 10:44:05] (common) IGNORED (media) field header: Publisher term: http://purl.org/dc/terms/publisher
[WARN] [2021-04-07 10:44:05] (common) IGNORED (media) field header: Contributor term: http://purl.org/dc/terms/contributor
[WARN] [2021-04-07 10:44:05] (common) IGNORED (media) field header: Creator term: http://purl.org/dc/terms/creator
[WARN] [2021-04-07 10:44:05] (common) IGNORED (media) field header: GenericLocation term: http://purl.org/dc/terms/spatial
[WARN] [2021-04-07 10:44:05] (common) IGNORED (media) field header: Latitude term: http://www.w3.org/2003/01/geo/wgs84_pos#lat
[WARN] [2021-04-07 10:44:05] (common) IGNORED (media) field header: Longitude term: http://www.w3.org/2003/01/geo/wgs84_pos#long
[WARN] [2021-04-07 10:44:05] (common) IGNORED (media) field header: Altitude term: http://www.w3.org/2003/01/geo/wgs84_pos#alt
[WARN] [2021-04-07 10:44:05] (common) IGNORED (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2021-04-07 10:44:05] (common) IGNORED (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2021-04-07 10:44:05] (common) IGNORED (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2021-04-07 10:44:05] (common) IGNORED (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2021-04-07 10:44:05] (common) IGNORED (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2021-04-07 10:44:05] (common) IGNORED (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[WARN] [2021-04-07 10:44:05] SKIPPING http://rs.tdwg.org/dwc/terms/Event config (events.txt)...
[STOP] [2021-04-07 10:44:05] Parse meta.xml file and create formats with fields
[STOP] [2021-04-07 10:44:05] Creating resource from OpenData
[START] [2021-04-07 10:44:06] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 10:44:06] create_harvest_instance
[INFO] [2021-04-07 10:44:06] Created harvest instance #3668
[STOP] [2021-04-07 10:44:06] create_harvest_instance
[START] [2021-04-07 10:44:06] fetch_files
[STOP] [2021-04-07 10:44:06] fetch_files
[START] [2021-04-07 10:44:06] validate_each_file
[INFO] [2021-04-07 10:44:06] Looping over 8 formats...
[INFO] [2021-04-07 10:44:06] ...agents (/app/public/data/reid_et_al/agents.txt)
[INFO] [2021-04-07 10:44:06] Valid: /app/public/converted_csv/reid_et_al_agents_3668.csv (0 lines)
[INFO] [2021-04-07 10:44:06] ...refs (/app/public/data/reid_et_al/references.txt)
[INFO] [2021-04-07 10:44:06] Valid: /app/public/converted_csv/reid_et_al_refs_3668.csv (0 lines)
[INFO] [2021-04-07 10:44:06] ...nodes (/app/public/data/reid_et_al/taxa.txt)
[INFO] [2021-04-07 10:44:06] Valid: /app/public/converted_csv/reid_et_al_nodes_3668.csv (53 lines)
[INFO] [2021-04-07 10:44:06] ...media (/app/public/data/reid_et_al/media.txt)
[INFO] [2021-04-07 10:44:06] Valid: /app/public/converted_csv/reid_et_al_media_3668.csv (0 lines)
[INFO] [2021-04-07 10:44:06] ...vernaculars (/app/public/data/reid_et_al/common names.txt)
[INFO] [2021-04-07 10:44:06] Valid: /app/public/converted_csv/reid_et_al_vernaculars_3668.csv (0 lines)
[INFO] [2021-04-07 10:44:06] ...occurrences (/app/public/data/reid_et_al/occurrences.txt)
[INFO] [2021-04-07 10:44:06] Valid: /app/public/converted_csv/reid_et_al_occurrences_3668.csv (80 lines)
[INFO] [2021-04-07 10:44:06] ...assocs (/app/public/data/reid_et_al/associations.txt)
[STOP] [2021-04-07 10:44:06] validate_each_file
[ERR] [2021-04-07 10:44:06] Exceptions::ColumnUnmatched
[ERR] [2021-04-07 10:44:06] TOO MANY COLUMNS: assocs: ,,,,,,
[ERR] [2021-04-07 10:44:06] ../models/resource_harvester.rb:136:in `block in validate_each_file'
[ERR] [2021-04-07 10:44:06] ../models/resource_harvester.rb:681:in `block in each_format'
[ERR] [2021-04-07 10:44:06] ../models/resource_harvester.rb:666:in `each_format'
[ERR] [2021-04-07 10:44:06] ../models/resource_harvester.rb:130:in `validate_each_file'
[ERR] [2021-04-07 10:44:06] ../models/resource_harvester.rb:86:in `block (2 levels) in start'
[ERR] [2021-04-07 10:44:06] ../models/logged_process.rb:34:in `run_step'
[ERR] [2021-04-07 10:44:06] ../models/resource_harvester.rb:86:in `block in start'
[ERR] [2021-04-07 10:44:06] ../models/resource_harvester.rb:75:in `each_key'
[ERR] [2021-04-07 10:44:06] ../models/resource_harvester.rb:75:in `start'
[ERR] [2021-04-07 10:44:06] ../models/resource.rb:261:in `harvest'
[ERR] [2021-04-07 10:44:06] ../models/resource.rb:237:in `re_download_opendata_and_harvest'
[ERR] [2021-04-07 10:44:06] bin/rails:4:in `require'
[ERR] [2021-04-07 10:44:06] bin/rails:4:in `<main>'
[STOP] [2021-04-07 10:44:06] logged process, took 0.69
[INFO] [2021-04-07 10:47:26] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-07 10:47:26] ## remove_type: ScientificName
[INFO] [2021-04-07 10:47:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:47:26] [10:47:26.758] Removed 0 Scientificnames
[INFO] [2021-04-07 10:47:26] ## remove_type: Vernacular
[INFO] [2021-04-07 10:47:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:47:26] [10:47:26.760] Removed 0 Vernaculars
[INFO] [2021-04-07 10:47:26] ## remove_type: Article
[INFO] [2021-04-07 10:47:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:47:26] [10:47:26.762] Removed 0 Articles
[INFO] [2021-04-07 10:47:26] ## remove_type: Medium
[INFO] [2021-04-07 10:47:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:47:26] [10:47:26.763] Removed 0 Media
[INFO] [2021-04-07 10:47:26] ## remove_type: Trait
[INFO] [2021-04-07 10:47:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:47:26] [10:47:26.765] Removed 0 Traits
[INFO] [2021-04-07 10:47:26] ## remove_type: MetaTrait
[INFO] [2021-04-07 10:47:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:47:26] [10:47:26.767] Removed 0 Metatraits
[INFO] [2021-04-07 10:47:26] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-07 10:47:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:47:26] [10:47:26.768] Removed 0 Occurrencemetadata
[INFO] [2021-04-07 10:47:26] ## remove_type: Assoc
[INFO] [2021-04-07 10:47:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:47:26] [10:47:26.770] Removed 0 Assocs
[INFO] [2021-04-07 10:47:26] ## remove_type: MetaAssoc
[INFO] [2021-04-07 10:47:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:47:26] [10:47:26.771] Removed 0 Metaassocs
[INFO] [2021-04-07 10:47:26] ## remove_type: Identifier
[INFO] [2021-04-07 10:47:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:47:26] [10:47:26.772] Removed 0 Identifiers
[INFO] [2021-04-07 10:47:26] ## remove_type: Reference
[INFO] [2021-04-07 10:47:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:47:26] [10:47:26.774] Removed 0 References
[INFO] [2021-04-07 10:47:26] ## remove_type: Node
[INFO] [2021-04-07 10:47:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:47:26] [10:47:26.784] Removed 0 Nodes
[START] [2021-04-07 10:47:27] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 10:47:27] Creating resource from OpenData
[START] [2021-04-07 10:47:28] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 10:47:28] Parse meta.xml file and create formats with fields
[WARN] [2021-04-07 10:47:35] (common) IGNORED (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2021-04-07 10:47:35] (common) IGNORED (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2021-04-07 10:47:35] (common) IGNORED (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2021-04-07 10:47:35] (common) IGNORED (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2021-04-07 10:47:35] (common) IGNORED (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[STOP] [2021-04-07 10:47:35] Parse meta.xml file and create formats with fields
[STOP] [2021-04-07 10:47:35] Creating resource from OpenData
[START] [2021-04-07 10:47:35] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 10:47:35] create_harvest_instance
[INFO] [2021-04-07 10:47:35] Created harvest instance #3669
[STOP] [2021-04-07 10:47:35] create_harvest_instance
[START] [2021-04-07 10:47:35] fetch_files
[STOP] [2021-04-07 10:47:35] fetch_files
[START] [2021-04-07 10:47:35] validate_each_file
[INFO] [2021-04-07 10:47:35] Looping over 4 formats...
[INFO] [2021-04-07 10:47:35] ...refs (/app/public/data/reid_et_al/references.txt)
[INFO] [2021-04-07 10:47:35] Valid: /app/public/converted_csv/reid_et_al_refs_3669.csv (0 lines)
[INFO] [2021-04-07 10:47:35] ...nodes (/app/public/data/reid_et_al/taxa.txt)
[INFO] [2021-04-07 10:47:35] Valid: /app/public/converted_csv/reid_et_al_nodes_3669.csv (53 lines)
[INFO] [2021-04-07 10:47:35] ...occurrences (/app/public/data/reid_et_al/occurrences.txt)
[INFO] [2021-04-07 10:47:35] Valid: /app/public/converted_csv/reid_et_al_occurrences_3669.csv (80 lines)
[INFO] [2021-04-07 10:47:35] ...measurements (/app/public/data/reid_et_al/measurements or facts.txt)
[STOP] [2021-04-07 10:47:35] validate_each_file
[ERR] [2021-04-07 10:47:35] Exceptions::ColumnUnmatched
[ERR] [2021-04-07 10:47:35] TOO MANY COLUMNS: measurements: ReferenceID
[ERR] [2021-04-07 10:47:35] ../models/resource_harvester.rb:136:in `block in validate_each_file'
[ERR] [2021-04-07 10:47:35] ../models/resource_harvester.rb:681:in `block in each_format'
[ERR] [2021-04-07 10:47:35] ../models/resource_harvester.rb:666:in `each_format'
[ERR] [2021-04-07 10:47:35] ../models/resource_harvester.rb:130:in `validate_each_file'
[ERR] [2021-04-07 10:47:35] ../models/resource_harvester.rb:86:in `block (2 levels) in start'
[ERR] [2021-04-07 10:47:35] ../models/logged_process.rb:34:in `run_step'
[ERR] [2021-04-07 10:47:35] ../models/resource_harvester.rb:86:in `block in start'
[ERR] [2021-04-07 10:47:35] ../models/resource_harvester.rb:75:in `each_key'
[ERR] [2021-04-07 10:47:35] ../models/resource_harvester.rb:75:in `start'
[ERR] [2021-04-07 10:47:35] ../models/resource.rb:261:in `harvest'
[ERR] [2021-04-07 10:47:35] ../models/resource.rb:237:in `re_download_opendata_and_harvest'
[ERR] [2021-04-07 10:47:35] bin/rails:4:in `require'
[ERR] [2021-04-07 10:47:35] bin/rails:4:in `<main>'
[STOP] [2021-04-07 10:47:35] logged process, took 0.64
[INFO] [2021-04-07 10:48:44] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-07 10:48:46] ## remove_type: ScientificName
[INFO] [2021-04-07 10:48:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:48:46] [10:48:46.110] Removed 0 Scientificnames
[INFO] [2021-04-07 10:48:46] ## remove_type: Vernacular
[INFO] [2021-04-07 10:48:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:48:46] [10:48:46.112] Removed 0 Vernaculars
[INFO] [2021-04-07 10:48:46] ## remove_type: Article
[INFO] [2021-04-07 10:48:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:48:46] [10:48:46.113] Removed 0 Articles
[INFO] [2021-04-07 10:48:46] ## remove_type: Medium
[INFO] [2021-04-07 10:48:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:48:46] [10:48:46.114] Removed 0 Media
[INFO] [2021-04-07 10:48:46] ## remove_type: Trait
[INFO] [2021-04-07 10:48:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:48:46] [10:48:46.116] Removed 0 Traits
[INFO] [2021-04-07 10:48:46] ## remove_type: MetaTrait
[INFO] [2021-04-07 10:48:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:48:46] [10:48:46.117] Removed 0 Metatraits
[INFO] [2021-04-07 10:48:46] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-07 10:48:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:48:46] [10:48:46.119] Removed 0 Occurrencemetadata
[INFO] [2021-04-07 10:48:46] ## remove_type: Assoc
[INFO] [2021-04-07 10:48:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:48:46] [10:48:46.120] Removed 0 Assocs
[INFO] [2021-04-07 10:48:46] ## remove_type: MetaAssoc
[INFO] [2021-04-07 10:48:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:48:46] [10:48:46.122] Removed 0 Metaassocs
[INFO] [2021-04-07 10:48:46] ## remove_type: Identifier
[INFO] [2021-04-07 10:48:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:48:46] [10:48:46.123] Removed 0 Identifiers
[INFO] [2021-04-07 10:48:46] ## remove_type: Reference
[INFO] [2021-04-07 10:48:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:48:46] [10:48:46.125] Removed 0 References
[INFO] [2021-04-07 10:48:46] ## remove_type: Node
[INFO] [2021-04-07 10:48:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:48:46] [10:48:46.135] Removed 0 Nodes
[START] [2021-04-07 10:48:46] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 10:48:46] Creating resource from OpenData
[START] [2021-04-07 10:48:48] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 10:48:48] Parse meta.xml file and create formats with fields
[WARN] [2021-04-07 10:48:53] (common) IGNORED (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[STOP] [2021-04-07 10:48:53] Parse meta.xml file and create formats with fields
[STOP] [2021-04-07 10:48:53] Creating resource from OpenData
[START] [2021-04-07 10:48:53] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 10:48:53] create_harvest_instance
[INFO] [2021-04-07 10:48:53] Created harvest instance #3670
[STOP] [2021-04-07 10:48:53] create_harvest_instance
[START] [2021-04-07 10:48:53] fetch_files
[STOP] [2021-04-07 10:48:53] fetch_files
[START] [2021-04-07 10:48:53] validate_each_file
[INFO] [2021-04-07 10:48:53] Looping over 3 formats...
[INFO] [2021-04-07 10:48:53] ...nodes (/app/public/data/reid_et_al/taxa.txt)
[INFO] [2021-04-07 10:48:53] Valid: /app/public/converted_csv/reid_et_al_nodes_3670.csv (53 lines)
[INFO] [2021-04-07 10:48:53] ...occurrences (/app/public/data/reid_et_al/occurrences.txt)
[INFO] [2021-04-07 10:48:53] Valid: /app/public/converted_csv/reid_et_al_occurrences_3670.csv (80 lines)
[INFO] [2021-04-07 10:48:53] ...measurements (/app/public/data/reid_et_al/measurements or facts.txt)
[STOP] [2021-04-07 10:48:53] validate_each_file
[ERR] [2021-04-07 10:48:53] Exceptions::ColumnUnmatched
[ERR] [2021-04-07 10:48:53] TOO MANY COLUMNS: measurements: ReferenceID
[ERR] [2021-04-07 10:48:53] ../models/resource_harvester.rb:136:in `block in validate_each_file'
[ERR] [2021-04-07 10:48:53] ../models/resource_harvester.rb:681:in `block in each_format'
[ERR] [2021-04-07 10:48:53] ../models/resource_harvester.rb:666:in `each_format'
[ERR] [2021-04-07 10:48:53] ../models/resource_harvester.rb:130:in `validate_each_file'
[ERR] [2021-04-07 10:48:53] ../models/resource_harvester.rb:86:in `block (2 levels) in start'
[ERR] [2021-04-07 10:48:53] ../models/logged_process.rb:34:in `run_step'
[ERR] [2021-04-07 10:48:53] ../models/resource_harvester.rb:86:in `block in start'
[ERR] [2021-04-07 10:48:53] ../models/resource_harvester.rb:75:in `each_key'
[ERR] [2021-04-07 10:48:53] ../models/resource_harvester.rb:75:in `start'
[ERR] [2021-04-07 10:48:53] ../models/resource.rb:261:in `harvest'
[ERR] [2021-04-07 10:48:53] ../models/resource.rb:237:in `re_download_opendata_and_harvest'
[ERR] [2021-04-07 10:48:53] bin/rails:4:in `require'
[ERR] [2021-04-07 10:48:53] bin/rails:4:in `<main>'
[STOP] [2021-04-07 10:48:53] logged process, took 0.6
[INFO] [2021-04-07 10:53:11] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-07 10:53:13] ## remove_type: ScientificName
[INFO] [2021-04-07 10:53:13] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:53:13] [10:53:13.987] Removed 0 Scientificnames
[INFO] [2021-04-07 10:53:13] ## remove_type: Vernacular
[INFO] [2021-04-07 10:53:13] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:53:13] [10:53:13.988] Removed 0 Vernaculars
[INFO] [2021-04-07 10:53:13] ## remove_type: Article
[INFO] [2021-04-07 10:53:13] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:53:13] [10:53:13.990] Removed 0 Articles
[INFO] [2021-04-07 10:53:13] ## remove_type: Medium
[INFO] [2021-04-07 10:53:13] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:53:13] [10:53:13.992] Removed 0 Media
[INFO] [2021-04-07 10:53:13] ## remove_type: Trait
[INFO] [2021-04-07 10:53:13] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:53:13] [10:53:13.993] Removed 0 Traits
[INFO] [2021-04-07 10:53:13] ## remove_type: MetaTrait
[INFO] [2021-04-07 10:53:13] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:53:13] [10:53:13.995] Removed 0 Metatraits
[INFO] [2021-04-07 10:53:13] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-07 10:53:13] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:53:13] [10:53:13.996] Removed 0 Occurrencemetadata
[INFO] [2021-04-07 10:53:13] ## remove_type: Assoc
[INFO] [2021-04-07 10:53:13] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:53:13] [10:53:13.998] Removed 0 Assocs
[INFO] [2021-04-07 10:53:13] ## remove_type: MetaAssoc
[INFO] [2021-04-07 10:53:13] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:53:13] [10:53:13.999] Removed 0 Metaassocs
[INFO] [2021-04-07 10:53:13] ## remove_type: Identifier
[INFO] [2021-04-07 10:53:14] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:53:14] [10:53:14.001] Removed 0 Identifiers
[INFO] [2021-04-07 10:53:14] ## remove_type: Reference
[INFO] [2021-04-07 10:53:14] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:53:14] [10:53:14.004] Removed 0 References
[INFO] [2021-04-07 10:53:14] ## remove_type: Node
[INFO] [2021-04-07 10:53:14] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:53:14] [10:53:14.015] Removed 0 Nodes
[START] [2021-04-07 10:53:14] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 10:53:14] Creating resource from OpenData
[START] [2021-04-07 10:53:15] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 10:53:15] Parse meta.xml file and create formats with fields
[WARN] [2021-04-07 10:53:22] (common) IGNORED (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2021-04-07 10:53:22] (common) IGNORED (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[STOP] [2021-04-07 10:53:22] Parse meta.xml file and create formats with fields
[STOP] [2021-04-07 10:53:22] Creating resource from OpenData
[START] [2021-04-07 10:53:22] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 10:53:22] create_harvest_instance
[INFO] [2021-04-07 10:53:22] Created harvest instance #3671
[STOP] [2021-04-07 10:53:22] create_harvest_instance
[START] [2021-04-07 10:53:22] fetch_files
[STOP] [2021-04-07 10:53:22] fetch_files
[START] [2021-04-07 10:53:22] validate_each_file
[INFO] [2021-04-07 10:53:22] Looping over 4 formats...
[INFO] [2021-04-07 10:53:22] ...refs (/app/public/data/reid_et_al/references.txt)
[INFO] [2021-04-07 10:53:22] Valid: /app/public/converted_csv/reid_et_al_refs_3671.csv (0 lines)
[INFO] [2021-04-07 10:53:22] ...nodes (/app/public/data/reid_et_al/taxa.txt)
[INFO] [2021-04-07 10:53:22] Valid: /app/public/converted_csv/reid_et_al_nodes_3671.csv (53 lines)
[INFO] [2021-04-07 10:53:22] ...occurrences (/app/public/data/reid_et_al/occurrences.txt)
[INFO] [2021-04-07 10:53:22] Valid: /app/public/converted_csv/reid_et_al_occurrences_3671.csv (80 lines)
[INFO] [2021-04-07 10:53:22] ...measurements (/app/public/data/reid_et_al/measurements or facts.txt)
[STOP] [2021-04-07 10:53:22] validate_each_file
[ERR] [2021-04-07 10:53:22] Exceptions::ColumnUnmatched
[ERR] [2021-04-07 10:53:22] TOO MANY COLUMNS: measurements: ReferenceID
[ERR] [2021-04-07 10:53:22] ../models/resource_harvester.rb:136:in `block in validate_each_file'
[ERR] [2021-04-07 10:53:22] ../models/resource_harvester.rb:681:in `block in each_format'
[ERR] [2021-04-07 10:53:22] ../models/resource_harvester.rb:666:in `each_format'
[ERR] [2021-04-07 10:53:22] ../models/resource_harvester.rb:130:in `validate_each_file'
[ERR] [2021-04-07 10:53:22] ../models/resource_harvester.rb:86:in `block (2 levels) in start'
[ERR] [2021-04-07 10:53:22] ../models/logged_process.rb:34:in `run_step'
[ERR] [2021-04-07 10:53:22] ../models/resource_harvester.rb:86:in `block in start'
[ERR] [2021-04-07 10:53:22] ../models/resource_harvester.rb:75:in `each_key'
[ERR] [2021-04-07 10:53:22] ../models/resource_harvester.rb:75:in `start'
[ERR] [2021-04-07 10:53:22] ../models/resource.rb:261:in `harvest'
[ERR] [2021-04-07 10:53:22] ../models/resource.rb:237:in `re_download_opendata_and_harvest'
[ERR] [2021-04-07 10:53:22] bin/rails:4:in `require'
[ERR] [2021-04-07 10:53:22] bin/rails:4:in `<main>'
[STOP] [2021-04-07 10:53:22] logged process, took 0.61
[INFO] [2021-04-07 10:55:54] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-07 10:55:58] ## remove_type: ScientificName
[INFO] [2021-04-07 10:55:58] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:55:58] [10:55:58.206] Removed 0 Scientificnames
[INFO] [2021-04-07 10:55:58] ## remove_type: Vernacular
[INFO] [2021-04-07 10:55:58] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:55:58] [10:55:58.208] Removed 0 Vernaculars
[INFO] [2021-04-07 10:55:58] ## remove_type: Article
[INFO] [2021-04-07 10:55:58] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:55:58] [10:55:58.209] Removed 0 Articles
[INFO] [2021-04-07 10:55:58] ## remove_type: Medium
[INFO] [2021-04-07 10:55:58] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:55:58] [10:55:58.211] Removed 0 Media
[INFO] [2021-04-07 10:55:58] ## remove_type: Trait
[INFO] [2021-04-07 10:55:58] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:55:58] [10:55:58.213] Removed 0 Traits
[INFO] [2021-04-07 10:55:58] ## remove_type: MetaTrait
[INFO] [2021-04-07 10:55:58] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:55:58] [10:55:58.214] Removed 0 Metatraits
[INFO] [2021-04-07 10:55:58] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-07 10:55:58] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:55:58] [10:55:58.216] Removed 0 Occurrencemetadata
[INFO] [2021-04-07 10:55:58] ## remove_type: Assoc
[INFO] [2021-04-07 10:55:58] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:55:58] [10:55:58.217] Removed 0 Assocs
[INFO] [2021-04-07 10:55:58] ## remove_type: MetaAssoc
[INFO] [2021-04-07 10:55:58] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:55:58] [10:55:58.219] Removed 0 Metaassocs
[INFO] [2021-04-07 10:55:58] ## remove_type: Identifier
[INFO] [2021-04-07 10:55:58] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:55:58] [10:55:58.221] Removed 0 Identifiers
[INFO] [2021-04-07 10:55:58] ## remove_type: Reference
[INFO] [2021-04-07 10:55:58] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:55:58] [10:55:58.222] Removed 0 References
[INFO] [2021-04-07 10:55:58] ## remove_type: Node
[INFO] [2021-04-07 10:55:58] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 10:55:58] [10:55:58.232] Removed 0 Nodes
[START] [2021-04-07 10:55:58] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 10:55:58] Creating resource from OpenData
[START] [2021-04-07 10:56:00] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 10:56:00] Parse meta.xml file and create formats with fields
[WARN] [2021-04-07 10:56:06] (common) IGNORED (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2021-04-07 10:56:06] (common) IGNORED (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[STOP] [2021-04-07 10:56:06] Parse meta.xml file and create formats with fields
[STOP] [2021-04-07 10:56:06] Creating resource from OpenData
[START] [2021-04-07 10:56:07] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 10:56:07] create_harvest_instance
[INFO] [2021-04-07 10:56:07] Created harvest instance #3672
[STOP] [2021-04-07 10:56:07] create_harvest_instance
[START] [2021-04-07 10:56:07] fetch_files
[STOP] [2021-04-07 10:56:07] fetch_files
[START] [2021-04-07 10:56:07] validate_each_file
[INFO] [2021-04-07 10:56:07] Looping over 4 formats...
[INFO] [2021-04-07 10:56:07] ...refs (/app/public/data/reid_et_al/references.txt)
[INFO] [2021-04-07 10:56:07] Valid: /app/public/converted_csv/reid_et_al_refs_3672.csv (0 lines)
[INFO] [2021-04-07 10:56:07] ...nodes (/app/public/data/reid_et_al/taxa.txt)
[INFO] [2021-04-07 10:56:07] Valid: /app/public/converted_csv/reid_et_al_nodes_3672.csv (53 lines)
[INFO] [2021-04-07 10:56:07] ...occurrences (/app/public/data/reid_et_al/occurrences.txt)
[INFO] [2021-04-07 10:56:07] Valid: /app/public/converted_csv/reid_et_al_occurrences_3672.csv (80 lines)
[INFO] [2021-04-07 10:56:07] ...measurements (/app/public/data/reid_et_al/measurements or facts.txt)
[INFO] [2021-04-07 10:56:07] Valid: /app/public/converted_csv/reid_et_al_measurements_3672.csv (405 lines)
[STOP] [2021-04-07 10:56:07] validate_each_file
[START] [2021-04-07 10:56:07] convert_to_csv
[INFO] [2021-04-07 10:56:07] Looping over 4 formats...
[INFO] [2021-04-07 10:56:07] ...refs (/app/public/data/reid_et_al/references.txt)
[CMD] [2021-04-07 10:56:07] /usr/bin/sort /app/public/converted_csv/reid_et_al_refs_3672.csv > /app/public/converted_csv/reid_et_al_refs_3672.csv_sorted
[INFO] [2021-04-07 10:56:07] Converted: /app/public/converted_csv/reid_et_al_refs_3672.csv (0 lines)
[INFO] [2021-04-07 10:56:07] ...nodes (/app/public/data/reid_et_al/taxa.txt)
[CMD] [2021-04-07 10:56:07] /usr/bin/sort /app/public/converted_csv/reid_et_al_nodes_3672.csv > /app/public/converted_csv/reid_et_al_nodes_3672.csv_sorted
[INFO] [2021-04-07 10:56:08] Converted: /app/public/converted_csv/reid_et_al_nodes_3672.csv (53 lines)
[INFO] [2021-04-07 10:56:08] ...occurrences (/app/public/data/reid_et_al/occurrences.txt)
[CMD] [2021-04-07 10:56:08] /usr/bin/sort /app/public/converted_csv/reid_et_al_occurrences_3672.csv > /app/public/converted_csv/reid_et_al_occurrences_3672.csv_sorted
[INFO] [2021-04-07 10:56:09] Converted: /app/public/converted_csv/reid_et_al_occurrences_3672.csv (80 lines)
[INFO] [2021-04-07 10:56:09] ...measurements (/app/public/data/reid_et_al/measurements or facts.txt)
[CMD] [2021-04-07 10:56:09] /usr/bin/sort /app/public/converted_csv/reid_et_al_measurements_3672.csv > /app/public/converted_csv/reid_et_al_measurements_3672.csv_sorted
[INFO] [2021-04-07 10:56:09] Converted: /app/public/converted_csv/reid_et_al_measurements_3672.csv (405 lines)
[STOP] [2021-04-07 10:56:09] convert_to_csv
[START] [2021-04-07 10:56:09] calculate_delta
[INFO] [2021-04-07 10:56:09] Looping over 4 formats...
[INFO] [2021-04-07 10:56:09] ...refs (/app/public/data/reid_et_al/references.txt)
[CMD] [2021-04-07 10:56:09] echo "0a" > /app/public/diff/reid_et_al_refs_3672.diff
[CMD] [2021-04-07 10:56:10] tail -n +1 /app/public/converted_csv/reid_et_al_refs_3672.csv >> /app/public/diff/reid_et_al_refs_3672.diff
[CMD] [2021-04-07 10:56:10] echo "." >> /app/public/diff/reid_et_al_refs_3672.diff
[INFO] [2021-04-07 10:56:11] Created diff: /app/public/diff/reid_et_al_refs_3672.diff (2 lines)
[INFO] [2021-04-07 10:56:11] ...nodes (/app/public/data/reid_et_al/taxa.txt)
[CMD] [2021-04-07 10:56:11] echo "0a" > /app/public/diff/reid_et_al_nodes_3672.diff
[CMD] [2021-04-07 10:56:11] tail -n +1 /app/public/converted_csv/reid_et_al_nodes_3672.csv >> /app/public/diff/reid_et_al_nodes_3672.diff
[CMD] [2021-04-07 10:56:12] echo "." >> /app/public/diff/reid_et_al_nodes_3672.diff
[INFO] [2021-04-07 10:56:12] Created diff: /app/public/diff/reid_et_al_nodes_3672.diff (55 lines)
[INFO] [2021-04-07 10:56:12] ...occurrences (/app/public/data/reid_et_al/occurrences.txt)
[CMD] [2021-04-07 10:56:12] echo "0a" > /app/public/diff/reid_et_al_occurrences_3672.diff
[CMD] [2021-04-07 10:56:13] tail -n +1 /app/public/converted_csv/reid_et_al_occurrences_3672.csv >> /app/public/diff/reid_et_al_occurrences_3672.diff
[CMD] [2021-04-07 10:56:13] echo "." >> /app/public/diff/reid_et_al_occurrences_3672.diff
[INFO] [2021-04-07 10:56:14] Created diff: /app/public/diff/reid_et_al_occurrences_3672.diff (82 lines)
[INFO] [2021-04-07 10:56:14] ...measurements (/app/public/data/reid_et_al/measurements or facts.txt)
[CMD] [2021-04-07 10:56:14] echo "0a" > /app/public/diff/reid_et_al_measurements_3672.diff
[CMD] [2021-04-07 10:56:15] tail -n +1 /app/public/converted_csv/reid_et_al_measurements_3672.csv >> /app/public/diff/reid_et_al_measurements_3672.diff
[CMD] [2021-04-07 10:56:15] echo "." >> /app/public/diff/reid_et_al_measurements_3672.diff
[INFO] [2021-04-07 10:56:16] Created diff: /app/public/diff/reid_et_al_measurements_3672.diff (407 lines)
[STOP] [2021-04-07 10:56:16] calculate_delta
[START] [2021-04-07 10:56:16] parse_diff_and_store
[INFO] [2021-04-07 10:56:16] Handling diff: /app/public/diff/reid_et_al_refs_3672.diff (2 lines)
[INFO] [2021-04-07 10:56:16] Loading refs diff file into memory (2 /app/public/diff/reid_et_al_refs_3672.diff lines)...
[INFO] [2021-04-07 10:56:17] Handling diff: /app/public/diff/reid_et_al_nodes_3672.diff (55 lines)
[INFO] [2021-04-07 10:56:17] Loading nodes diff file into memory (55 /app/public/diff/reid_et_al_nodes_3672.diff lines)...
[INFO] [2021-04-07 10:56:18] Handling diff: /app/public/diff/reid_et_al_occurrences_3672.diff (82 lines)
[INFO] [2021-04-07 10:56:18] Loading occurrences diff file into memory (82 /app/public/diff/reid_et_al_occurrences_3672.diff lines)...
[INFO] [2021-04-07 10:56:19] Handling diff: /app/public/diff/reid_et_al_measurements_3672.diff (407 lines)
[INFO] [2021-04-07 10:56:20] Loading measurements diff file into memory (407 /app/public/diff/reid_et_al_measurements_3672.diff lines)...
[INFO] [2021-04-07 10:56:20] Storing 164 ScientificNames
[INFO] [2021-04-07 10:56:20] Processing group of 164 in 1 groups of 1000
[INFO] [2021-04-07 10:56:20] Average Time: 0.05
[INFO] [2021-04-07 10:56:20] Total Time: 1s
[INFO] [2021-04-07 10:56:20] Storing 164 Nodes
[INFO] [2021-04-07 10:56:20] Processing group of 164 in 1 groups of 1000
[INFO] [2021-04-07 10:56:20] Average Time: 0.04
[INFO] [2021-04-07 10:56:20] Total Time: 1s
[INFO] [2021-04-07 10:56:20] Storing 80 Occurrences
[INFO] [2021-04-07 10:56:20] Processing group of 80 in 1 groups of 1000
[INFO] [2021-04-07 10:56:20] Average Time: 0.01
[INFO] [2021-04-07 10:56:20] Total Time: 1s
[INFO] [2021-04-07 10:56:20] Storing 559 OccurrenceMetadata
[INFO] [2021-04-07 10:56:20] Processing group of 559 in 1 groups of 1000
[INFO] [2021-04-07 10:56:21] Average Time: 0.06
[INFO] [2021-04-07 10:56:21] Total Time: 1s
[INFO] [2021-04-07 10:56:21] Storing 86 Traits
[INFO] [2021-04-07 10:56:21] Processing group of 86 in 1 groups of 1000
[INFO] [2021-04-07 10:56:21] Average Time: 0.03
[INFO] [2021-04-07 10:56:21] Total Time: 1s
[INFO] [2021-04-07 10:56:21] Storing 86 MetaTraits
[INFO] [2021-04-07 10:56:21] Processing group of 86 in 1 groups of 1000
[INFO] [2021-04-07 10:56:21] Average Time: 0.01
[INFO] [2021-04-07 10:56:21] Total Time: 1s
[STOP] [2021-04-07 10:56:21] parse_diff_and_store
[START] [2021-04-07 10:56:21] resolve_keys
[INFO] [2021-04-07 10:56:27] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-07 10:56:27] traits to occurrences...
[INFO] [2021-04-07 10:56:27] traits to nodes (through occurrences)...
[INFO] [2021-04-07 10:56:27] Traits to sex term...
[INFO] [2021-04-07 10:56:27] Traits to lifestage term...
[INFO] [2021-04-07 10:56:27] MetaTraits to traits...
[INFO] [2021-04-07 10:56:27] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-07 10:56:27] Assocs to occurrences...
[INFO] [2021-04-07 10:56:27] Assocs to nodes...
[INFO] [2021-04-07 10:56:27] Assoc to sex term...
[INFO] [2021-04-07 10:56:27] Assoc to lifestage term...
[INFO] [2021-04-07 10:56:27] MetaAssoc to assocs...
[STOP] [2021-04-07 10:56:27] resolve_keys
[START] [2021-04-07 10:56:27] hold_for_later_1
[STOP] [2021-04-07 10:56:27] hold_for_later_1
[START] [2021-04-07 10:56:27] hold_for_later_2
[STOP] [2021-04-07 10:56:27] hold_for_later_2
[START] [2021-04-07 10:56:27] resolve_missing_parents
[STOP] [2021-04-07 10:56:27] resolve_missing_parents
[START] [2021-04-07 10:56:27] rebuild_nodes
[START] [2021-04-07 10:56:27] Flattener#flatten
[START] [2021-04-07 10:56:27] Flattener#study_resource
[START] [2021-04-07 10:56:27] Flattener#build_ancestry
[STOP] [2021-04-07 10:56:28] Flattener#build_ancestry
[INFO] [2021-04-07 10:56:28] 164 ancestry keys
[START] [2021-04-07 10:56:28] build_node_ancestors
[INFO] [2021-04-07 10:56:28] old ancestors deleted.
[STOP] [2021-04-07 10:56:28] build_node_ancestors
[START] [2021-04-07 10:56:28] Flattener#propagate_ancestor_ids
[STOP] [2021-04-07 10:56:28] Flattener#propagate_ancestor_ids
[STOP] [2021-04-07 10:56:28] Flattener#flatten
[STOP] [2021-04-07 10:56:28] rebuild_nodes
[START] [2021-04-07 10:56:28] resolve_missing_media_owners
[STOP] [2021-04-07 10:56:28] resolve_missing_media_owners
[START] [2021-04-07 10:56:28] sanitize_media_verbatims
[STOP] [2021-04-07 10:56:28] sanitize_media_verbatims
[START] [2021-04-07 10:56:28] queue_downloads
[STOP] [2021-04-07 10:56:28] queue_downloads
[START] [2021-04-07 10:56:28] parse_names
[WARN] [2021-04-07 10:56:28] I see 164 names which still need to be parsed.
[WARN] [2021-04-07 10:56:29] I see 8 names which still need to be parsed.
[STOP] [2021-04-07 10:56:30] parse_names
[START] [2021-04-07 10:56:30] denormalize_canonical_names_to_nodes
[STOP] [2021-04-07 10:56:30] denormalize_canonical_names_to_nodes
[START] [2021-04-07 10:56:30] match_nodes
[START] [2021-04-07 10:56:30] map_all_nodes_to_pages
[STOP] [2021-04-07 10:56:36] map_all_nodes_to_pages
[INFO] [2021-04-07 10:56:36] 11 Unmatched nodes (of 164)! That's too many to output. Full list in /app/public/data/reid_et_al/unmatched_nodes.txt ; First 10: Acanthephyridae (#92628484); Munidopsidae (#92628582); Maxillopoda (#92628590); Pedunculata (#92628591); Pedunculata (#92628592); Histiobranchius (#92628567); Histiobranchius bathybius (#92628568); Porcellanaster cerulus (#92628608); Hymenaster cf_coccinatus (#92628574); Aspidochirotida (#92628524)
[START] [2021-04-07 10:56:36] update_nodes
[STOP] [2021-04-07 10:56:36] update_nodes
[STOP] [2021-04-07 10:56:36] match_nodes
[START] [2021-04-07 10:56:36] reindex_search
[STOP] [2021-04-07 10:56:36] reindex_search
[START] [2021-04-07 10:56:36] normalize_units
[STOP] [2021-04-07 10:56:36] normalize_units
[START] [2021-04-07 10:56:36] calculate_statistics
[STOP] [2021-04-07 10:56:36] calculate_statistics
[START] [2021-04-07 10:56:36] complete_harvest_instance
[START] [2021-04-07 10:56:36] overall_tsv_creation
[INFO] [2021-04-07 10:56:36] Processing group of 164 in 1 batches of 10000
[INFO] [2021-04-07 10:57:14] 86 Traits (unfiltered)...
[INFO] [2021-04-07 10:57:48] 86 Traits (filtered)...
[INFO] [2021-04-07 10:57:48] 0 Associations (filtered)...
[INFO] [2021-04-07 10:57:48] 515 metadata added.
[INFO] [2021-04-07 10:57:48] 0 metadata added.
[INFO] [2021-04-07 10:58:15] Average Time: 73.71
[INFO] [2021-04-07 10:58:15] Total Time: 1m39s
[STOP] [2021-04-07 10:58:15] overall_tsv_creation
[INFO] [2021-04-07 10:58:15] Done. Check your files:
[INFO] [2021-04-07 10:58:15] (156 lines) /app/public/data/reid_et_al/publish_nodes.tsv
[INFO] [2021-04-07 10:58:16] (667 lines) /app/public/data/reid_et_al/publish_node_ancestors.tsv
[INFO] [2021-04-07 10:58:16] (164 lines) /app/public/data/reid_et_al/publish_scientific_names.tsv
[INFO] [2021-04-07 10:58:17] (87 lines) /app/public/data/reid_et_al/publish_traits.tsv
[INFO] [2021-04-07 10:58:18] (516 lines) /app/public/data/reid_et_al/publish_metadata.tsv
[STOP] [2021-04-07 10:58:18] complete_harvest_instance
[START] [2021-04-07 10:58:18] completed
[STOP] [2021-04-07 10:58:18] completed
[STOP] [2021-04-07 10:58:18] logged process, took 131.29
Latest Process