Harvest for Habitat data for microbial organisms Created 29 Sep 13:25

Harvesting Log

[INFO] [2022-09-29 13:25:11] Created harvest instance #4217
[STOP] [2022-09-29 13:25:11] create_harvest_instance
[START] [2022-09-29 13:25:11] fetch_files
[STOP] [2022-09-29 13:25:11] fetch_files
[START] [2022-09-29 13:25:11] validate_each_file
[INFO] [2022-09-29 13:25:11] Looping over 3 formats...
[INFO] [2022-09-29 13:25:11] ...nodes (/app/public/data/hab_dat_microbia/taxon.tab)
[INFO] [2022-09-29 13:25:11] Valid: /app/public/data/hab_dat_microbia/converted_csv/hab_dat_microbia_nodes_29758.csv (985 lines)
[INFO] [2022-09-29 13:25:11] ...occurrences (/app/public/data/hab_dat_microbia/occurrence_specific.tab)
[INFO] [2022-09-29 13:25:11] Valid: /app/public/data/hab_dat_microbia/converted_csv/hab_dat_microbia_occurrences_29759.csv (1107 lines)
[INFO] [2022-09-29 13:25:11] ...measurements (/app/public/data/hab_dat_microbia/measurement_or_fact_specific.tab)
[INFO] [2022-09-29 13:25:11] Valid: /app/public/data/hab_dat_microbia/converted_csv/hab_dat_microbia_measurements_29760.csv (1568 lines)
[STOP] [2022-09-29 13:25:11] validate_each_file
[START] [2022-09-29 13:25:11] convert_to_csv
[INFO] [2022-09-29 13:25:11] Looping over 3 formats...
[INFO] [2022-09-29 13:25:11] ...nodes (/app/public/data/hab_dat_microbia/taxon.tab)
[CMD] [2022-09-29 13:25:11] /usr/bin/sort /app/public/data/hab_dat_microbia/converted_csv/hab_dat_microbia_nodes_29758.csv > /app/public/data/hab_dat_microbia/converted_csv/hab_dat_microbia_nodes_29758.csv_sorted
[INFO] [2022-09-29 13:25:11] Converted: /app/public/data/hab_dat_microbia/converted_csv/hab_dat_microbia_nodes_29758.csv (985 lines)
[INFO] [2022-09-29 13:25:11] ...occurrences (/app/public/data/hab_dat_microbia/occurrence_specific.tab)
[CMD] [2022-09-29 13:25:11] /usr/bin/sort /app/public/data/hab_dat_microbia/converted_csv/hab_dat_microbia_occurrences_29759.csv > /app/public/data/hab_dat_microbia/converted_csv/hab_dat_microbia_occurrences_29759.csv_sorted
[INFO] [2022-09-29 13:25:11] Converted: /app/public/data/hab_dat_microbia/converted_csv/hab_dat_microbia_occurrences_29759.csv (1107 lines)
[INFO] [2022-09-29 13:25:11] ...measurements (/app/public/data/hab_dat_microbia/measurement_or_fact_specific.tab)
[CMD] [2022-09-29 13:25:11] /usr/bin/sort /app/public/data/hab_dat_microbia/converted_csv/hab_dat_microbia_measurements_29760.csv > /app/public/data/hab_dat_microbia/converted_csv/hab_dat_microbia_measurements_29760.csv_sorted
[INFO] [2022-09-29 13:25:11] Converted: /app/public/data/hab_dat_microbia/converted_csv/hab_dat_microbia_measurements_29760.csv (1568 lines)
[STOP] [2022-09-29 13:25:11] convert_to_csv
[START] [2022-09-29 13:25:11] calculate_delta
[INFO] [2022-09-29 13:25:11] Looping over 3 formats...
[INFO] [2022-09-29 13:25:11] ...nodes (/app/public/data/hab_dat_microbia/taxon.tab)
[CMD] [2022-09-29 13:25:11] echo "0a" > /app/public/data/hab_dat_microbia/diff/hab_dat_microbia_nodes_29758.diff
[CMD] [2022-09-29 13:25:11] tail -n +1 /app/public/data/hab_dat_microbia/converted_csv/hab_dat_microbia_nodes_29758.csv >> /app/public/data/hab_dat_microbia/diff/hab_dat_microbia_nodes_29758.diff
[CMD] [2022-09-29 13:25:11] echo "." >> /app/public/data/hab_dat_microbia/diff/hab_dat_microbia_nodes_29758.diff
[INFO] [2022-09-29 13:25:12] Created diff: /app/public/data/hab_dat_microbia/diff/hab_dat_microbia_nodes_29758.diff (987 lines)
[INFO] [2022-09-29 13:25:12] ...occurrences (/app/public/data/hab_dat_microbia/occurrence_specific.tab)
[CMD] [2022-09-29 13:25:12] echo "0a" > /app/public/data/hab_dat_microbia/diff/hab_dat_microbia_occurrences_29759.diff
[CMD] [2022-09-29 13:25:12] tail -n +1 /app/public/data/hab_dat_microbia/converted_csv/hab_dat_microbia_occurrences_29759.csv >> /app/public/data/hab_dat_microbia/diff/hab_dat_microbia_occurrences_29759.diff
[CMD] [2022-09-29 13:25:12] echo "." >> /app/public/data/hab_dat_microbia/diff/hab_dat_microbia_occurrences_29759.diff
[INFO] [2022-09-29 13:25:12] Created diff: /app/public/data/hab_dat_microbia/diff/hab_dat_microbia_occurrences_29759.diff (1109 lines)
[INFO] [2022-09-29 13:25:12] ...measurements (/app/public/data/hab_dat_microbia/measurement_or_fact_specific.tab)
[CMD] [2022-09-29 13:25:12] echo "0a" > /app/public/data/hab_dat_microbia/diff/hab_dat_microbia_measurements_29760.diff
[CMD] [2022-09-29 13:25:12] tail -n +1 /app/public/data/hab_dat_microbia/converted_csv/hab_dat_microbia_measurements_29760.csv >> /app/public/data/hab_dat_microbia/diff/hab_dat_microbia_measurements_29760.diff
[CMD] [2022-09-29 13:25:12] echo "." >> /app/public/data/hab_dat_microbia/diff/hab_dat_microbia_measurements_29760.diff
[INFO] [2022-09-29 13:25:12] Created diff: /app/public/data/hab_dat_microbia/diff/hab_dat_microbia_measurements_29760.diff (1570 lines)
[STOP] [2022-09-29 13:25:12] calculate_delta
[START] [2022-09-29 13:25:12] parse_diff_and_store
[INFO] [2022-09-29 13:25:12] Handling diff: /app/public/data/hab_dat_microbia/diff/hab_dat_microbia_nodes_29758.diff (987 lines)
[INFO] [2022-09-29 13:25:12] Loading nodes diff file into memory (987 lines)...
[INFO] [2022-09-29 13:25:12] Storing 985 ScientificNames (1970/985/987)
[INFO] [2022-09-29 13:25:12] Storing 985 Nodes (1970/985/987)
[INFO] [2022-09-29 13:25:12] Handling diff: /app/public/data/hab_dat_microbia/diff/hab_dat_microbia_occurrences_29759.diff (1109 lines)
[INFO] [2022-09-29 13:25:12] Loading occurrences diff file into memory (1109 lines)...
[INFO] [2022-09-29 13:25:13] Storing 1107 Occurrences (1107/1107/1109)
[INFO] [2022-09-29 13:25:13] Handling diff: /app/public/data/hab_dat_microbia/diff/hab_dat_microbia_measurements_29760.diff (1570 lines)
[INFO] [2022-09-29 13:25:13] Loading measurements diff file into memory (1570 lines)...
[STOP] [2022-09-29 13:25:13] parse_diff_and_store
[ERR] [2022-09-29 13:25:13] RuntimeError
[ERR] [2022-09-29 13:25:13] Missing Term for URI `http://eol.org/schema/terms/hypoxicSediment`, must be added!
[ERR] [2022-09-29 13:25:13] ../models/store/model_builder.rb:655:in `fail_on_bad_uri'
[ERR] [2022-09-29 13:25:13] ../models/store/model_builder.rb:609:in `convert_trait_value'
[ERR] [2022-09-29 13:25:13] ../models/store/model_builder.rb:427:in `build_trait'
[ERR] [2022-09-29 13:25:13] ../models/store/model_builder.rb:28:in `build_models'
[ERR] [2022-09-29 13:25:13] ../models/resource_harvester.rb:358:in `block (3 levels) in parse_diff_and_store'
[ERR] [2022-09-29 13:25:13] ../models/csv_parser.rb:111:in `block in diff_as_hashes'
[ERR] [2022-09-29 13:25:13] ../models/csv_parser.rb:28:in `block in line_at_a_time'
[ERR] [2022-09-29 13:25:13] ../models/csv_parser.rb:25:in `line_at_a_time'
[ERR] [2022-09-29 13:25:13] ../models/csv_parser.rb:96:in `diff_as_hashes'
[ERR] [2022-09-29 13:25:13] ../models/resource_harvester.rb:311:in `block (2 levels) in parse_diff_and_store'
[ERR] [2022-09-29 13:25:13] ../models/logged_process.rb:87:in `enter_group'
[ERR] [2022-09-29 13:25:13] ../models/resource_harvester.rb:310:in `block in parse_diff_and_store'
[ERR] [2022-09-29 13:25:13] ../models/resource_harvester.rb:765:in `block in each_diff'
[ERR] [2022-09-29 13:25:13] ../models/resource_harvester.rb:752:in `each_diff'
[ERR] [2022-09-29 13:25:13] ../models/resource_harvester.rb:302:in `parse_diff_and_store'
[ERR] [2022-09-29 13:25:13] ../models/resource_harvester.rb:86:in `block (2 levels) in start'
[ERR] [2022-09-29 13:25:13] ../models/logged_process.rb:43:in `run_step'
[ERR] [2022-09-29 13:25:13] ../models/resource_harvester.rb:86:in `block in start'
[ERR] [2022-09-29 13:25:13] ../models/resource_harvester.rb:75:in `each_key'
[ERR] [2022-09-29 13:25:13] ../models/resource_harvester.rb:75:in `start'
[ERR] [2022-09-29 13:25:13] ../models/resource.rb:300:in `harvest'
[ERR] [2022-09-29 13:25:13] ../models/resource.rb:276:in `re_download_opendata_and_harvest'
[ERR] [2022-09-29 13:25:13] bin/rails:4:in `require'
[ERR] [2022-09-29 13:25:13] bin/rails:4:in `<main>'
[STOP] [2022-09-29 13:25:13] logged process, took 1.59

Latest Process

29 Sep 13:25 EXITED currently processing 1 (0.0%) of 1570