Harvest for Achatz et al 2013 Created 31 Jan 12:00

Harvesting Log

[INFO] [2023-01-31 12:00:15] Created harvest instance #4285
[STOP] [2023-01-31 12:00:15] create_harvest_instance
[START] [2023-01-31 12:00:15] fetch_files
[STOP] [2023-01-31 12:00:15] fetch_files
[START] [2023-01-31 12:00:15] validate_each_file
[INFO] [2023-01-31 12:00:15] Looping over 4 formats...
[INFO] [2023-01-31 12:00:15] ...refs (/app/public/data/achatz_et_al_ach/references.txt)
[INFO] [2023-01-31 12:00:15] Valid: /app/public/data/achatz_et_al_ach/converted_csv/achatz_et_al_ach_refs_30153.csv (7 lines)
[INFO] [2023-01-31 12:00:15] ...nodes (/app/public/data/achatz_et_al_ach/taxa.tsv)
[INFO] [2023-01-31 12:00:15] Valid: /app/public/data/achatz_et_al_ach/converted_csv/achatz_et_al_ach_nodes_30152.csv (4 lines)
[INFO] [2023-01-31 12:00:15] ...occurrences (/app/public/data/achatz_et_al_ach/occurrences.tsv)
[INFO] [2023-01-31 12:00:15] Valid: /app/public/data/achatz_et_al_ach/converted_csv/achatz_et_al_ach_occurrences_30154.csv (4 lines)
[INFO] [2023-01-31 12:00:15] ...measurements (/app/public/data/achatz_et_al_ach/measurementOrFact.tsv)
[INFO] [2023-01-31 12:00:15] Valid: /app/public/data/achatz_et_al_ach/converted_csv/achatz_et_al_ach_measurements_30155.csv (10 lines)
[STOP] [2023-01-31 12:00:15] validate_each_file
[START] [2023-01-31 12:00:15] convert_to_csv
[INFO] [2023-01-31 12:00:15] Looping over 4 formats...
[INFO] [2023-01-31 12:00:15] ...refs (/app/public/data/achatz_et_al_ach/references.txt)
[CMD] [2023-01-31 12:00:15] /usr/bin/sort /app/public/data/achatz_et_al_ach/converted_csv/achatz_et_al_ach_refs_30153.csv > /app/public/data/achatz_et_al_ach/converted_csv/achatz_et_al_ach_refs_30153.csv_sorted
[INFO] [2023-01-31 12:00:15] Converted: /app/public/data/achatz_et_al_ach/converted_csv/achatz_et_al_ach_refs_30153.csv (7 lines)
[INFO] [2023-01-31 12:00:15] ...nodes (/app/public/data/achatz_et_al_ach/taxa.tsv)
[CMD] [2023-01-31 12:00:15] /usr/bin/sort /app/public/data/achatz_et_al_ach/converted_csv/achatz_et_al_ach_nodes_30152.csv > /app/public/data/achatz_et_al_ach/converted_csv/achatz_et_al_ach_nodes_30152.csv_sorted
[INFO] [2023-01-31 12:00:15] Converted: /app/public/data/achatz_et_al_ach/converted_csv/achatz_et_al_ach_nodes_30152.csv (4 lines)
[INFO] [2023-01-31 12:00:15] ...occurrences (/app/public/data/achatz_et_al_ach/occurrences.tsv)
[CMD] [2023-01-31 12:00:15] /usr/bin/sort /app/public/data/achatz_et_al_ach/converted_csv/achatz_et_al_ach_occurrences_30154.csv > /app/public/data/achatz_et_al_ach/converted_csv/achatz_et_al_ach_occurrences_30154.csv_sorted
[INFO] [2023-01-31 12:00:15] Converted: /app/public/data/achatz_et_al_ach/converted_csv/achatz_et_al_ach_occurrences_30154.csv (4 lines)
[INFO] [2023-01-31 12:00:15] ...measurements (/app/public/data/achatz_et_al_ach/measurementOrFact.tsv)
[CMD] [2023-01-31 12:00:15] /usr/bin/sort /app/public/data/achatz_et_al_ach/converted_csv/achatz_et_al_ach_measurements_30155.csv > /app/public/data/achatz_et_al_ach/converted_csv/achatz_et_al_ach_measurements_30155.csv_sorted
[INFO] [2023-01-31 12:00:15] Converted: /app/public/data/achatz_et_al_ach/converted_csv/achatz_et_al_ach_measurements_30155.csv (10 lines)
[STOP] [2023-01-31 12:00:15] convert_to_csv
[START] [2023-01-31 12:00:15] calculate_delta
[INFO] [2023-01-31 12:00:15] Looping over 4 formats...
[INFO] [2023-01-31 12:00:15] ...refs (/app/public/data/achatz_et_al_ach/references.txt)
[CMD] [2023-01-31 12:00:15] echo "0a" > /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_refs_30153.diff
[CMD] [2023-01-31 12:00:15] tail -n +1 /app/public/data/achatz_et_al_ach/converted_csv/achatz_et_al_ach_refs_30153.csv >> /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_refs_30153.diff
[CMD] [2023-01-31 12:00:16] echo "." >> /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_refs_30153.diff
[INFO] [2023-01-31 12:00:16] Created diff: /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_refs_30153.diff (9 lines)
[INFO] [2023-01-31 12:00:16] ...nodes (/app/public/data/achatz_et_al_ach/taxa.tsv)
[CMD] [2023-01-31 12:00:16] echo "0a" > /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_nodes_30152.diff
[CMD] [2023-01-31 12:00:16] tail -n +1 /app/public/data/achatz_et_al_ach/converted_csv/achatz_et_al_ach_nodes_30152.csv >> /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_nodes_30152.diff
[CMD] [2023-01-31 12:00:16] echo "." >> /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_nodes_30152.diff
[INFO] [2023-01-31 12:00:16] Created diff: /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_nodes_30152.diff (6 lines)
[INFO] [2023-01-31 12:00:16] ...occurrences (/app/public/data/achatz_et_al_ach/occurrences.tsv)
[CMD] [2023-01-31 12:00:16] echo "0a" > /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_occurrences_30154.diff
[CMD] [2023-01-31 12:00:16] tail -n +1 /app/public/data/achatz_et_al_ach/converted_csv/achatz_et_al_ach_occurrences_30154.csv >> /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_occurrences_30154.diff
[CMD] [2023-01-31 12:00:16] echo "." >> /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_occurrences_30154.diff
[INFO] [2023-01-31 12:00:16] Created diff: /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_occurrences_30154.diff (6 lines)
[INFO] [2023-01-31 12:00:16] ...measurements (/app/public/data/achatz_et_al_ach/measurementOrFact.tsv)
[CMD] [2023-01-31 12:00:16] echo "0a" > /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_measurements_30155.diff
[CMD] [2023-01-31 12:00:16] tail -n +1 /app/public/data/achatz_et_al_ach/converted_csv/achatz_et_al_ach_measurements_30155.csv >> /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_measurements_30155.diff
[CMD] [2023-01-31 12:00:16] echo "." >> /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_measurements_30155.diff
[INFO] [2023-01-31 12:00:16] Created diff: /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_measurements_30155.diff (12 lines)
[STOP] [2023-01-31 12:00:16] calculate_delta
[START] [2023-01-31 12:00:16] parse_diff_and_store
[INFO] [2023-01-31 12:00:16] Handling diff: /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_refs_30153.diff (9 lines)
[INFO] [2023-01-31 12:00:16] Loading refs diff file into memory (9 lines)...
[INFO] [2023-01-31 12:00:16] Storing 7 References (7/7/9)
[INFO] [2023-01-31 12:00:16] Handling diff: /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_nodes_30152.diff (6 lines)
[INFO] [2023-01-31 12:00:16] Loading nodes diff file into memory (6 lines)...
[INFO] [2023-01-31 12:00:16] Storing 4 ScientificNames (8/4/6)
[INFO] [2023-01-31 12:00:16] Storing 4 Nodes (8/4/6)
[INFO] [2023-01-31 12:00:16] Handling diff: /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_occurrences_30154.diff (6 lines)
[INFO] [2023-01-31 12:00:16] Loading occurrences diff file into memory (6 lines)...
[INFO] [2023-01-31 12:00:16] Storing 4 Occurrences (4/4/6)
[INFO] [2023-01-31 12:00:16] Handling diff: /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_measurements_30155.diff (12 lines)
[INFO] [2023-01-31 12:00:16] Loading measurements diff file into memory (12 lines)...
[STOP] [2023-01-31 12:00:16] parse_diff_and_store
[ERR] [2023-01-31 12:00:16] RuntimeError
[ERR] [2023-01-31 12:00:16] Missing Term for URI `https://www.wikidata.org/entity/Q127006`, must be added!
[ERR] [2023-01-31 12:00:16] ../models/store/model_builder.rb:655:in `fail_on_bad_uri'
[ERR] [2023-01-31 12:00:16] ../models/store/model_builder.rb:609:in `convert_trait_value'
[ERR] [2023-01-31 12:00:16] ../models/store/model_builder.rb:427:in `build_trait'
[ERR] [2023-01-31 12:00:16] ../models/store/model_builder.rb:28:in `build_models'
[ERR] [2023-01-31 12:00:16] ../models/resource_harvester.rb:361:in `block (3 levels) in parse_diff_and_store'
[ERR] [2023-01-31 12:00:16] ../models/csv_parser.rb:111:in `block in diff_as_hashes'
[ERR] [2023-01-31 12:00:16] ../models/csv_parser.rb:28:in `block in line_at_a_time'
[ERR] [2023-01-31 12:00:16] ../models/csv_parser.rb:25:in `line_at_a_time'
[ERR] [2023-01-31 12:00:16] ../models/csv_parser.rb:96:in `diff_as_hashes'
[ERR] [2023-01-31 12:00:16] ../models/resource_harvester.rb:314:in `block (2 levels) in parse_diff_and_store'
[ERR] [2023-01-31 12:00:16] ../models/logged_process.rb:87:in `enter_group'
[ERR] [2023-01-31 12:00:16] ../models/resource_harvester.rb:313:in `block in parse_diff_and_store'
[ERR] [2023-01-31 12:00:16] ../models/resource_harvester.rb:769:in `block in each_diff'
[ERR] [2023-01-31 12:00:16] ../models/resource_harvester.rb:756:in `each_diff'
[ERR] [2023-01-31 12:00:16] ../models/resource_harvester.rb:305:in `parse_diff_and_store'
[ERR] [2023-01-31 12:00:16] ../models/resource_harvester.rb:86:in `block (2 levels) in start'
[ERR] [2023-01-31 12:00:16] ../models/logged_process.rb:43:in `run_step'
[ERR] [2023-01-31 12:00:16] ../models/resource_harvester.rb:86:in `block in start'
[ERR] [2023-01-31 12:00:16] ../models/resource_harvester.rb:75:in `each_key'
[ERR] [2023-01-31 12:00:16] ../models/resource_harvester.rb:75:in `start'
[ERR] [2023-01-31 12:00:16] ../models/resource.rb:305:in `harvest'
[ERR] [2023-01-31 12:00:16] ../models/resource.rb:281:in `re_download_opendata_and_harvest'
[ERR] [2023-01-31 12:00:16] bin/rails:4:in `require'
[ERR] [2023-01-31 12:00:16] bin/rails:4:in `<main>'
[STOP] [2023-01-31 12:00:16] logged process, took 0.69
[INFO] [2023-02-01 08:37:16] ## HARVEST: type = resume_harvest
[START] [2023-02-01 08:37:19] logged process: 6a863f1e73ce2cb1722cb96b0ebbd087a11c4304 Okay, I think this will fix the downloaded images bug.
[INFO] [2023-02-01 08:37:19] Already completed stage create_harvest_instance, skipping...
[INFO] [2023-02-01 08:37:19] Already completed stage fetch_files, skipping...
[INFO] [2023-02-01 08:37:19] Already completed stage validate_each_file, skipping...
[INFO] [2023-02-01 08:37:19] Already completed stage convert_to_csv, skipping...
[INFO] [2023-02-01 08:37:19] Already completed stage calculate_delta, skipping...
[START] [2023-02-01 08:37:19] parse_diff_and_store
[INFO] [2023-02-01 08:37:19] Handling diff: /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_refs_30153.diff (9 lines)
[INFO] [2023-02-01 08:37:19] Loading refs diff file into memory (9 lines)...
[WARN] [2023-02-01 08:37:19] SKIPPED 7 References (7/7/9) with resource_pks already be in the database!
[INFO] [2023-02-01 08:37:19] Storing 0 References (7/7/9)
[WARN] [2023-02-01 08:37:19] No models to import, skipping!
[INFO] [2023-02-01 08:37:19] Handling diff: /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_nodes_30152.diff (6 lines)
[INFO] [2023-02-01 08:37:19] Loading nodes diff file into memory (6 lines)...
[WARN] [2023-02-01 08:37:19] SKIPPED 4 Scientific names (8/4/6) with resource_pks already be in the database!
[WARN] [2023-02-01 08:37:19] SKIPPED 4 Nodes (8/4/6) with resource_pks already be in the database!
[INFO] [2023-02-01 08:37:19] Storing 0 ScientificNames (8/4/6)
[WARN] [2023-02-01 08:37:19] No models to import, skipping!
[INFO] [2023-02-01 08:37:19] Storing 0 Nodes (8/4/6)
[WARN] [2023-02-01 08:37:19] No models to import, skipping!
[INFO] [2023-02-01 08:37:19] Handling diff: /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_occurrences_30154.diff (6 lines)
[INFO] [2023-02-01 08:37:19] Loading occurrences diff file into memory (6 lines)...
[WARN] [2023-02-01 08:37:19] SKIPPED 4 Occurrences (4/4/6) with resource_pks already be in the database!
[INFO] [2023-02-01 08:37:19] Storing 0 Occurrences (4/4/6)
[WARN] [2023-02-01 08:37:19] No models to import, skipping!
[INFO] [2023-02-01 08:37:19] Handling diff: /app/public/data/achatz_et_al_ach/diff/achatz_et_al_ach_measurements_30155.diff (12 lines)
[INFO] [2023-02-01 08:37:19] Loading measurements diff file into memory (12 lines)...
[INFO] [2023-02-01 08:37:20] Storing 9 TraitsReferences (30/10/12)
[INFO] [2023-02-01 08:37:20] Storing 10 Traits (30/10/12)
[INFO] [2023-02-01 08:37:20] Storing 11 MetaTraits (30/10/12)
[STOP] [2023-02-01 08:37:20] parse_diff_and_store
[START] [2023-02-01 08:37:20] resolve_keys
[2023-02-01 08:37:20] Resolving downloaded urls (this is not actually downloading them yet)
[INFO] [2023-02-01 08:37:28] Occurrences to nodes (through scientific_names)...
[INFO] [2023-02-01 08:37:28] traits to occurrences...
[INFO] [2023-02-01 08:37:28] traits to nodes (through occurrences)...
[INFO] [2023-02-01 08:37:28] Traits to sex term...
[INFO] [2023-02-01 08:37:28] Traits to lifestage term...
[INFO] [2023-02-01 08:37:28] MetaTraits to traits...
[INFO] [2023-02-01 08:37:28] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2023-02-01 08:37:28] Assocs to occurrences...
[INFO] [2023-02-01 08:37:28] Assocs to nodes...
[INFO] [2023-02-01 08:37:28] Assoc to sex term...
[INFO] [2023-02-01 08:37:28] Assoc to lifestage term...
[INFO] [2023-02-01 08:37:28] MetaAssoc to assocs...
[STOP] [2023-02-01 08:37:28] resolve_keys
[START] [2023-02-01 08:37:28] hold_for_later_1
[STOP] [2023-02-01 08:37:28] hold_for_later_1
[START] [2023-02-01 08:37:28] hold_for_later_2
[STOP] [2023-02-01 08:37:28] hold_for_later_2
[START] [2023-02-01 08:37:28] resolve_missing_parents
[STOP] [2023-02-01 08:37:28] resolve_missing_parents
[START] [2023-02-01 08:37:28] rebuild_nodes
[START] [2023-02-01 08:37:28] Flattener#flatten
[START] [2023-02-01 08:37:28] Flattener#study_resource
[START] [2023-02-01 08:37:28] Flattener#build_ancestry
[STOP] [2023-02-01 08:37:28] Flattener#build_ancestry
[INFO] [2023-02-01 08:37:28] 4 ancestry keys
[START] [2023-02-01 08:37:28] build_node_ancestors
[INFO] [2023-02-01 08:37:28] old ancestors deleted.
[STOP] [2023-02-01 08:37:28] build_node_ancestors
[WARN] [2023-02-01 08:37:28] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2023-02-01 08:37:28] Flattener#flatten
[STOP] [2023-02-01 08:37:28] rebuild_nodes
[START] [2023-02-01 08:37:28] resolve_missing_media_owners
[STOP] [2023-02-01 08:37:28] resolve_missing_media_owners
[START] [2023-02-01 08:37:28] sanitize_media_verbatims
[STOP] [2023-02-01 08:37:28] sanitize_media_verbatims
[START] [2023-02-01 08:37:28] queue_downloads
[STOP] [2023-02-01 08:37:28] queue_downloads
[START] [2023-02-01 08:37:28] parse_names
[WARN] [2023-02-01 08:37:28] I see 4 names which still need to be parsed.
[WARN] [2023-02-01 08:37:28] Names to parse: 4 formatted: 4 learned: 4 parsed: 4
[STOP] [2023-02-01 08:37:29] parse_names
[START] [2023-02-01 08:37:29] denormalize_canonical_names_to_nodes
[STOP] [2023-02-01 08:37:29] denormalize_canonical_names_to_nodes
[START] [2023-02-01 08:37:29] match_nodes
[START] [2023-02-01 08:37:29] map_all_nodes_to_pages
[STOP] [2023-02-01 08:37:29] map_all_nodes_to_pages
[INFO] [2023-02-01 08:37:29] ZERO unmatched nodes (of 4)! Nicely done.
[START] [2023-02-01 08:37:29] update_nodes
[STOP] [2023-02-01 08:37:29] update_nodes
[STOP] [2023-02-01 08:37:29] match_nodes
[START] [2023-02-01 08:37:29] reindex_search
[STOP] [2023-02-01 08:37:29] reindex_search
[START] [2023-02-01 08:37:29] normalize_units
[STOP] [2023-02-01 08:37:29] normalize_units
[START] [2023-02-01 08:37:29] calculate_statistics
[2023-02-01 08:37:29] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[INFO] [2023-02-01 08:37:29] Duplicate page_id count: 0
[STOP] [2023-02-01 08:37:29] calculate_statistics
[START] [2023-02-01 08:37:29] complete_harvest_instance
[START] [2023-02-01 08:37:29] overall_tsv_creation
[INFO] [2023-02-01 08:37:29] Exporting 4 nodes as TSV in batches of 10000...
[INFO] [2023-02-01 08:37:29] Processing group of 4 in 1 batches of 10000
[INFO] [2023-02-01 08:37:29] 6 Traits (unfiltered) and 0 associations...
[INFO] [2023-02-01 08:37:29] Building Traits map for 4 nodes (this can take a while)...
[INFO] [2023-02-01 08:37:29] Mapped 6 traits (11 meta) for 4 nodes.
[INFO] [2023-02-01 08:37:29] Building Associations map (this can take a while)...
[INFO] [2023-02-01 08:37:29] Done. 0 assocs mapped (0 meta).
[INFO] [2023-02-01 08:37:29] Adding 6 traits...
[INFO] [2023-02-01 08:37:29] 13 metadata added.
[INFO] [2023-02-01 08:37:29] Adding 0 assocs...
[INFO] [2023-02-01 08:37:29] 0 metadata added.
[INFO] [2023-02-01 08:38:13] Processed 4/4 nodes
[INFO] [2023-02-01 08:38:13] Average Time: 43.98
[INFO] [2023-02-01 08:38:13] Total Time: 45s
[STOP] [2023-02-01 08:38:13] overall_tsv_creation
[INFO] [2023-02-01 08:38:13] Done. Check your files:
[INFO] [2023-02-01 08:38:13] (4 lines) /app/public/data/achatz_et_al_ach/publish_nodes.tsv
[INFO] [2023-02-01 08:38:13] (4 lines) /app/public/data/achatz_et_al_ach/publish_scientific_names.tsv
[INFO] [2023-02-01 08:38:13] (7 lines) /app/public/data/achatz_et_al_ach/publish_traits.tsv
[INFO] [2023-02-01 08:38:13] (14 lines) /app/public/data/achatz_et_al_ach/publish_metadata.tsv
[STOP] [2023-02-01 08:38:13] complete_harvest_instance
[START] [2023-02-01 08:38:13] completed
[STOP] [2023-02-01 08:38:13] completed
[STOP] [2023-02-01 08:38:13] logged process, took 55.06

Latest Process

01 Feb 08:37 EXITED currently processing 0