harvested resources
sign in

Harvest for Microbe ecomorphological guilds Created 29 Sep 15:22

Stage: parse_diff_and_store
Fetched: 29 Sep 15:22
Validated: 29 Sep 15:22
Deltas Created 29 Sep 15:22
New Models Stored: 29 Sep 15:22
Failed: 29 Sep 15:22
Completed: 29 Sep 15:22
Time to Harvest: less than a minute

Harvesting Log

(76 lines) 
[INFO] [2022-09-29 15:22:22] Created harvest instance #4218
[STOP] [2022-09-29 15:22:22] create_harvest_instance
[START] [2022-09-29 15:22:22] fetch_files
[STOP] [2022-09-29 15:22:22] fetch_files
[START] [2022-09-29 15:22:22] validate_each_file
[INFO] [2022-09-29 15:22:22] Looping over 3 formats...
[INFO] [2022-09-29 15:22:22] ...nodes (/app/public/data/microbe_ecomorph/taxon.tab)
[INFO] [2022-09-29 15:22:22] Valid: /app/public/data/microbe_ecomorph/converted_csv/microbe_ecomorph_nodes_29761.csv (482 lines)
[INFO] [2022-09-29 15:22:22] ...occurrences (/app/public/data/microbe_ecomorph/occurrence_specific.tab)
[INFO] [2022-09-29 15:22:22] Valid: /app/public/data/microbe_ecomorph/converted_csv/microbe_ecomorph_occurrences_29762.csv (572 lines)
[INFO] [2022-09-29 15:22:22] ...measurements (/app/public/data/microbe_ecomorph/measurement_or_fact_specific.tab)
[INFO] [2022-09-29 15:22:22] Valid: /app/public/data/microbe_ecomorph/converted_csv/microbe_ecomorph_measurements_29763.csv (1003 lines)
[STOP] [2022-09-29 15:22:22] validate_each_file
[START] [2022-09-29 15:22:22] convert_to_csv
[INFO] [2022-09-29 15:22:22] Looping over 3 formats...
[INFO] [2022-09-29 15:22:22] ...nodes (/app/public/data/microbe_ecomorph/taxon.tab)
[CMD] [2022-09-29 15:22:22] /usr/bin/sort /app/public/data/microbe_ecomorph/converted_csv/microbe_ecomorph_nodes_29761.csv > /app/public/data/microbe_ecomorph/converted_csv/microbe_ecomorph_nodes_29761.csv_sorted
[INFO] [2022-09-29 15:22:22] Converted: /app/public/data/microbe_ecomorph/converted_csv/microbe_ecomorph_nodes_29761.csv (482 lines)
[INFO] [2022-09-29 15:22:22] ...occurrences (/app/public/data/microbe_ecomorph/occurrence_specific.tab)
[CMD] [2022-09-29 15:22:22] /usr/bin/sort /app/public/data/microbe_ecomorph/converted_csv/microbe_ecomorph_occurrences_29762.csv > /app/public/data/microbe_ecomorph/converted_csv/microbe_ecomorph_occurrences_29762.csv_sorted
[INFO] [2022-09-29 15:22:22] Converted: /app/public/data/microbe_ecomorph/converted_csv/microbe_ecomorph_occurrences_29762.csv (572 lines)
[INFO] [2022-09-29 15:22:22] ...measurements (/app/public/data/microbe_ecomorph/measurement_or_fact_specific.tab)
[CMD] [2022-09-29 15:22:22] /usr/bin/sort /app/public/data/microbe_ecomorph/converted_csv/microbe_ecomorph_measurements_29763.csv > /app/public/data/microbe_ecomorph/converted_csv/microbe_ecomorph_measurements_29763.csv_sorted
[INFO] [2022-09-29 15:22:22] Converted: /app/public/data/microbe_ecomorph/converted_csv/microbe_ecomorph_measurements_29763.csv (1003 lines)
[STOP] [2022-09-29 15:22:22] convert_to_csv
[START] [2022-09-29 15:22:22] calculate_delta
[INFO] [2022-09-29 15:22:22] Looping over 3 formats...
[INFO] [2022-09-29 15:22:22] ...nodes (/app/public/data/microbe_ecomorph/taxon.tab)
[CMD] [2022-09-29 15:22:22] echo "0a" > /app/public/data/microbe_ecomorph/diff/microbe_ecomorph_nodes_29761.diff
[CMD] [2022-09-29 15:22:22] tail -n +1 /app/public/data/microbe_ecomorph/converted_csv/microbe_ecomorph_nodes_29761.csv >> /app/public/data/microbe_ecomorph/diff/microbe_ecomorph_nodes_29761.diff
[CMD] [2022-09-29 15:22:22] echo "." >> /app/public/data/microbe_ecomorph/diff/microbe_ecomorph_nodes_29761.diff
[INFO] [2022-09-29 15:22:22] Created diff: /app/public/data/microbe_ecomorph/diff/microbe_ecomorph_nodes_29761.diff (484 lines)
[INFO] [2022-09-29 15:22:22] ...occurrences (/app/public/data/microbe_ecomorph/occurrence_specific.tab)
[CMD] [2022-09-29 15:22:22] echo "0a" > /app/public/data/microbe_ecomorph/diff/microbe_ecomorph_occurrences_29762.diff
[CMD] [2022-09-29 15:22:22] tail -n +1 /app/public/data/microbe_ecomorph/converted_csv/microbe_ecomorph_occurrences_29762.csv >> /app/public/data/microbe_ecomorph/diff/microbe_ecomorph_occurrences_29762.diff
[CMD] [2022-09-29 15:22:22] echo "." >> /app/public/data/microbe_ecomorph/diff/microbe_ecomorph_occurrences_29762.diff
[INFO] [2022-09-29 15:22:22] Created diff: /app/public/data/microbe_ecomorph/diff/microbe_ecomorph_occurrences_29762.diff (574 lines)
[INFO] [2022-09-29 15:22:22] ...measurements (/app/public/data/microbe_ecomorph/measurement_or_fact_specific.tab)
[CMD] [2022-09-29 15:22:22] echo "0a" > /app/public/data/microbe_ecomorph/diff/microbe_ecomorph_measurements_29763.diff
[CMD] [2022-09-29 15:22:22] tail -n +1 /app/public/data/microbe_ecomorph/converted_csv/microbe_ecomorph_measurements_29763.csv >> /app/public/data/microbe_ecomorph/diff/microbe_ecomorph_measurements_29763.diff
[CMD] [2022-09-29 15:22:22] echo "." >> /app/public/data/microbe_ecomorph/diff/microbe_ecomorph_measurements_29763.diff
[INFO] [2022-09-29 15:22:22] Created diff: /app/public/data/microbe_ecomorph/diff/microbe_ecomorph_measurements_29763.diff (1005 lines)
[STOP] [2022-09-29 15:22:22] calculate_delta
[START] [2022-09-29 15:22:22] parse_diff_and_store
[INFO] [2022-09-29 15:22:22] Handling diff: /app/public/data/microbe_ecomorph/diff/microbe_ecomorph_nodes_29761.diff (484 lines)
[INFO] [2022-09-29 15:22:22] Loading nodes diff file into memory (484 lines)...
[INFO] [2022-09-29 15:22:23] Storing 482 ScientificNames (964/482/484)
[INFO] [2022-09-29 15:22:23] Storing 482 Nodes (964/482/484)
[INFO] [2022-09-29 15:22:23] Handling diff: /app/public/data/microbe_ecomorph/diff/microbe_ecomorph_occurrences_29762.diff (574 lines)
[INFO] [2022-09-29 15:22:23] Loading occurrences diff file into memory (574 lines)...
[STOP] [2022-09-29 15:22:23] parse_diff_and_store
[ERR] [2022-09-29 15:22:23] RuntimeError
[ERR] [2022-09-29 15:22:23] Missing Term for URI `http://purl.obolibrary.org/obo/OPL_0000057`, must be added!
[ERR] [2022-09-29 15:22:23] ../models/store/model_builder.rb:655:in `fail_on_bad_uri'
[ERR] [2022-09-29 15:22:23] ../models/store/model_builder.rb:366:in `build_occurrence'
[ERR] [2022-09-29 15:22:23] ../models/store/model_builder.rb:27:in `build_models'
[ERR] [2022-09-29 15:22:23] ../models/resource_harvester.rb:358:in `block (3 levels) in parse_diff_and_store'
[ERR] [2022-09-29 15:22:23] ../models/csv_parser.rb:111:in `block in diff_as_hashes'
[ERR] [2022-09-29 15:22:23] ../models/csv_parser.rb:28:in `block in line_at_a_time'
[ERR] [2022-09-29 15:22:23] ../models/csv_parser.rb:25:in `line_at_a_time'
[ERR] [2022-09-29 15:22:23] ../models/csv_parser.rb:96:in `diff_as_hashes'
[ERR] [2022-09-29 15:22:23] ../models/resource_harvester.rb:311:in `block (2 levels) in parse_diff_and_store'
[ERR] [2022-09-29 15:22:23] ../models/logged_process.rb:87:in `enter_group'
[ERR] [2022-09-29 15:22:23] ../models/resource_harvester.rb:310:in `block in parse_diff_and_store'
[ERR] [2022-09-29 15:22:23] ../models/resource_harvester.rb:765:in `block in each_diff'
[ERR] [2022-09-29 15:22:23] ../models/resource_harvester.rb:752:in `each_diff'
[ERR] [2022-09-29 15:22:23] ../models/resource_harvester.rb:302:in `parse_diff_and_store'
[ERR] [2022-09-29 15:22:23] ../models/resource_harvester.rb:86:in `block (2 levels) in start'
[ERR] [2022-09-29 15:22:23] ../models/logged_process.rb:43:in `run_step'
[ERR] [2022-09-29 15:22:23] ../models/resource_harvester.rb:86:in `block in start'
[ERR] [2022-09-29 15:22:23] ../models/resource_harvester.rb:75:in `each_key'
[ERR] [2022-09-29 15:22:23] ../models/resource_harvester.rb:75:in `start'
[ERR] [2022-09-29 15:22:23] ../models/resource.rb:300:in `harvest'
[ERR] [2022-09-29 15:22:23] bin/rails:4:in `require'
[ERR] [2022-09-29 15:22:23] bin/rails:4:in `<main>'
[STOP] [2022-09-29 15:22:23] logged process, took 1.14

Latest Process