Harvest for Hessler 1981 Created 30 Jul 15:28

Harvesting Log

# Logfile created on 2020-07-30 15:13:10 -0400 by logger.rb/v1.4.2
[START] [2020-07-30 15:13:10] logged process
[START] [2020-07-30 15:13:10] Creating resource from OpenData
[START] [2020-07-30 15:13:10] logged process
[START] [2020-07-30 15:13:10] Parse meta.xml file and create formats with fields
[STOP] [2020-07-30 15:13:11] Parse meta.xml file and create formats with fields
[STOP] [2020-07-30 15:13:11] Creating resource from OpenData
[INFO] [2020-07-30 15:13:27] ## HARVEST: type = -harvest
[START] [2020-07-30 15:13:43] logged process
[START] [2020-07-30 15:13:43] create_harvest_instance
[STOP] [2020-07-30 15:13:45] create_harvest_instance
[START] [2020-07-30 15:13:45] fetch_files
[STOP] [2020-07-30 15:13:45] fetch_files
[START] [2020-07-30 15:13:45] validate_each_file
[STOP] [2020-07-30 15:13:45] validate_each_file
[START] [2020-07-30 15:13:45] convert_to_csv
[CMD] [2020-07-30 15:13:45] /usr/bin/sort /app/public/converted_csv/hessler_hessler__refs_22396.csv > /app/public/converted_csv/hessler_hessler__refs_22396.csv_sorted
[CMD] [2020-07-30 15:13:45] /usr/bin/sort /app/public/converted_csv/hessler_hessler__nodes_22397.csv > /app/public/converted_csv/hessler_hessler__nodes_22397.csv_sorted
[CMD] [2020-07-30 15:13:45] /usr/bin/sort /app/public/converted_csv/hessler_hessler__occurrences_22398.csv > /app/public/converted_csv/hessler_hessler__occurrences_22398.csv_sorted
[CMD] [2020-07-30 15:13:45] /usr/bin/sort /app/public/converted_csv/hessler_hessler__measurements_22399.csv > /app/public/converted_csv/hessler_hessler__measurements_22399.csv_sorted
[STOP] [2020-07-30 15:13:45] convert_to_csv
[START] [2020-07-30 15:13:45] calculate_delta
[CMD] [2020-07-30 15:13:45] echo "0a" > /app/public/diff/hessler_hessler__refs_22396.diff
[CMD] [2020-07-30 15:13:45] tail -n +1 /app/public/converted_csv/hessler_hessler__refs_22396.csv >> /app/public/diff/hessler_hessler__refs_22396.diff
[CMD] [2020-07-30 15:13:45] echo "." >> /app/public/diff/hessler_hessler__refs_22396.diff
[CMD] [2020-07-30 15:13:45] echo "0a" > /app/public/diff/hessler_hessler__nodes_22397.diff
[CMD] [2020-07-30 15:13:45] tail -n +1 /app/public/converted_csv/hessler_hessler__nodes_22397.csv >> /app/public/diff/hessler_hessler__nodes_22397.diff
[CMD] [2020-07-30 15:13:45] echo "." >> /app/public/diff/hessler_hessler__nodes_22397.diff
[CMD] [2020-07-30 15:13:45] echo "0a" > /app/public/diff/hessler_hessler__occurrences_22398.diff
[CMD] [2020-07-30 15:13:45] tail -n +1 /app/public/converted_csv/hessler_hessler__occurrences_22398.csv >> /app/public/diff/hessler_hessler__occurrences_22398.diff
[CMD] [2020-07-30 15:13:45] echo "." >> /app/public/diff/hessler_hessler__occurrences_22398.diff
[CMD] [2020-07-30 15:13:45] echo "0a" > /app/public/diff/hessler_hessler__measurements_22399.diff
[CMD] [2020-07-30 15:13:45] tail -n +1 /app/public/converted_csv/hessler_hessler__measurements_22399.csv >> /app/public/diff/hessler_hessler__measurements_22399.diff
[CMD] [2020-07-30 15:13:45] echo "." >> /app/public/diff/hessler_hessler__measurements_22399.diff
[STOP] [2020-07-30 15:13:45] calculate_delta
[START] [2020-07-30 15:13:45] parse_diff_and_store
[INFO] [2020-07-30 15:13:45] Loading refs diff file into memory (true lines)...
[INFO] [2020-07-30 15:13:45] Loading nodes diff file into memory (true lines)...
[INFO] [2020-07-30 15:13:45] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-07-30 15:13:45] Loading measurements diff file into memory (true lines)...
[INFO] [2020-07-30 15:13:46] Storing 20 ScientificNames
[INFO] [2020-07-30 15:13:46] Processing group of 20 in 1 groups of 1000
[INFO] [2020-07-30 15:13:46] Average Time: 0.02
[INFO] [2020-07-30 15:13:46] Total Time: 1s
[INFO] [2020-07-30 15:13:46] Storing 20 Nodes
[INFO] [2020-07-30 15:13:46] Processing group of 20 in 1 groups of 1000
[INFO] [2020-07-30 15:13:46] Average Time: 0.01
[INFO] [2020-07-30 15:13:46] Total Time: 1s
[INFO] [2020-07-30 15:13:46] Storing 20 Occurrences
[INFO] [2020-07-30 15:13:46] Processing group of 20 in 1 groups of 1000
[INFO] [2020-07-30 15:13:46] Average Time: 0.01
[INFO] [2020-07-30 15:13:46] Total Time: 1s
[INFO] [2020-07-30 15:13:46] Storing 20 OccurrenceMetadata
[INFO] [2020-07-30 15:13:46] Processing group of 20 in 1 groups of 1000
[INFO] [2020-07-30 15:13:46] Average Time: 0.01
[INFO] [2020-07-30 15:13:46] Total Time: 1s
[INFO] [2020-07-30 15:13:46] Storing 135 Traits
[INFO] [2020-07-30 15:13:46] Processing group of 135 in 1 groups of 1000
[INFO] [2020-07-30 15:13:46] Average Time: 0.09
[INFO] [2020-07-30 15:13:46] Total Time: 1s
[INFO] [2020-07-30 15:13:46] Storing 106 MetaTraits
[INFO] [2020-07-30 15:13:46] Processing group of 106 in 1 groups of 1000
[INFO] [2020-07-30 15:13:46] Average Time: 0.01
[INFO] [2020-07-30 15:13:46] Total Time: 1s
[STOP] [2020-07-30 15:13:46] parse_diff_and_store
[START] [2020-07-30 15:13:46] resolve_keys
[STOP] [2020-07-30 15:13:47] resolve_keys
[ERR] [2020-07-30 15:13:47] ActiveRecord::StatementInvalid
[ERR] [2020-07-30 15:13:47] Lockfile::StolenLockError: Lockfile::StolenLockError: SELECT MIN(`articles_sections`.`id`) FROM `articles_sections` WHERE (1=1) AND `articles_sections`.`harvest_id` = 3030
[ERR] [2020-07-30 15:13:47] ./config/initializers/core_extensions.rb:14:in `propagate_id'
[ERR] [2020-07-30 15:13:47] ../models/resource_harvester.rb:576:in `propagate_id'
[ERR] [2020-07-30 15:13:47] ../models/resource_harvester.rb:470:in `resolve_article_keys'
[ERR] [2020-07-30 15:13:47] ../models/resource_harvester.rb:428:in `resolve_keys'
[ERR] [2020-07-30 15:13:47] ../models/resource_harvester.rb:86:in `block (3 levels) in start'
[ERR] [2020-07-30 15:13:47] ../models/logged_process.rb:19:in `run_step'
[ERR] [2020-07-30 15:13:47] ../models/resource_harvester.rb:86:in `block (2 levels) in start'
[ERR] [2020-07-30 15:13:47] ../models/resource_harvester.rb:75:in `each_key'
[ERR] [2020-07-30 15:13:47] ../models/resource_harvester.rb:75:in `block in start'
[ERR] [2020-07-30 15:13:47] ../models/resource.rb:151:in `lock'
[ERR] [2020-07-30 15:13:47] ../models/resource_harvester.rb:72:in `start'
[ERR] [2020-07-30 15:13:47] ../models/resource.rb:232:in `harvest'
[ERR] [2020-07-30 15:13:47] bin/rails:4:in `require'
[ERR] [2020-07-30 15:13:47] bin/rails:4:in `<main>'
[STOP] [2020-07-30 15:13:47] logged process, took 3.38
[INFO] [2020-07-30 15:15:29] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-07-30 15:15:31] ## remove_type: ScientificName
[INFO] [2020-07-30 15:15:31] ++ Calling delete_all on 20 instances...
[INFO] [2020-07-30 15:15:31] [15:15:31.845] Removed 20 Scientificnames
[INFO] [2020-07-30 15:15:31] ## remove_type: Vernacular
[INFO] [2020-07-30 15:15:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-30 15:15:31] [15:15:31.848] Removed 0 Vernaculars
[INFO] [2020-07-30 15:15:31] ## remove_type: Article
[INFO] [2020-07-30 15:15:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-30 15:15:31] [15:15:31.851] Removed 0 Articles
[INFO] [2020-07-30 15:15:31] ## remove_type: Medium
[INFO] [2020-07-30 15:15:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-30 15:15:31] [15:15:31.855] Removed 0 Media
[INFO] [2020-07-30 15:15:31] ## remove_type: Trait
[INFO] [2020-07-30 15:15:31] ++ Calling delete_all on 135 instances...
[INFO] [2020-07-30 15:15:31] [15:15:31.864] Removed 135 Traits
[INFO] [2020-07-30 15:15:31] ## remove_type: MetaTrait
[INFO] [2020-07-30 15:15:31] ++ Calling delete_all on 106 instances...
[INFO] [2020-07-30 15:15:31] [15:15:31.868] Removed 106 Metatraits
[INFO] [2020-07-30 15:15:31] ## remove_type: OccurrenceMetadatum
[INFO] [2020-07-30 15:15:31] ++ Calling delete_all on 20 instances...
[INFO] [2020-07-30 15:15:31] [15:15:31.871] Removed 20 Occurrencemetadata
[INFO] [2020-07-30 15:15:31] ## remove_type: Assoc
[INFO] [2020-07-30 15:15:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-30 15:15:31] [15:15:31.874] Removed 0 Assocs
[INFO] [2020-07-30 15:15:31] ## remove_type: MetaAssoc
[INFO] [2020-07-30 15:15:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-30 15:15:31] [15:15:31.876] Removed 0 Metaassocs
[INFO] [2020-07-30 15:15:31] ## remove_type: Identifier
[INFO] [2020-07-30 15:15:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-30 15:15:31] [15:15:31.879] Removed 0 Identifiers
[INFO] [2020-07-30 15:15:31] ## remove_type: Reference
[INFO] [2020-07-30 15:15:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-30 15:15:31] [15:15:31.882] Removed 0 References
[INFO] [2020-07-30 15:15:31] ## remove_type: Node
[INFO] [2020-07-30 15:15:31] ++ Calling delete_all on 20 instances...
[INFO] [2020-07-30 15:15:31] [15:15:31.904] Removed 20 Nodes
[START] [2020-07-30 15:15:32] logged process
[START] [2020-07-30 15:15:32] Creating resource from OpenData
[START] [2020-07-30 15:15:32] logged process
[START] [2020-07-30 15:15:32] Parse meta.xml file and create formats with fields
[STOP] [2020-07-30 15:15:32] Parse meta.xml file and create formats with fields
[STOP] [2020-07-30 15:15:32] Creating resource from OpenData
[START] [2020-07-30 15:15:32] logged process
[START] [2020-07-30 15:15:32] create_harvest_instance
[STOP] [2020-07-30 15:15:33] create_harvest_instance
[START] [2020-07-30 15:15:33] fetch_files
[STOP] [2020-07-30 15:15:33] fetch_files
[START] [2020-07-30 15:15:33] validate_each_file
[STOP] [2020-07-30 15:15:33] validate_each_file
[START] [2020-07-30 15:15:33] convert_to_csv
[CMD] [2020-07-30 15:15:33] /usr/bin/sort /app/public/converted_csv/hessler_hessler__refs_22404.csv > /app/public/converted_csv/hessler_hessler__refs_22404.csv_sorted
[CMD] [2020-07-30 15:15:33] /usr/bin/sort /app/public/converted_csv/hessler_hessler__nodes_22405.csv > /app/public/converted_csv/hessler_hessler__nodes_22405.csv_sorted
[CMD] [2020-07-30 15:15:33] /usr/bin/sort /app/public/converted_csv/hessler_hessler__occurrences_22406.csv > /app/public/converted_csv/hessler_hessler__occurrences_22406.csv_sorted
[CMD] [2020-07-30 15:15:33] /usr/bin/sort /app/public/converted_csv/hessler_hessler__measurements_22407.csv > /app/public/converted_csv/hessler_hessler__measurements_22407.csv_sorted
[STOP] [2020-07-30 15:15:33] convert_to_csv
[START] [2020-07-30 15:15:33] calculate_delta
[CMD] [2020-07-30 15:15:33] echo "0a" > /app/public/diff/hessler_hessler__refs_22404.diff
[CMD] [2020-07-30 15:15:34] tail -n +1 /app/public/converted_csv/hessler_hessler__refs_22404.csv >> /app/public/diff/hessler_hessler__refs_22404.diff
[CMD] [2020-07-30 15:15:34] echo "." >> /app/public/diff/hessler_hessler__refs_22404.diff
[CMD] [2020-07-30 15:15:34] echo "0a" > /app/public/diff/hessler_hessler__nodes_22405.diff
[CMD] [2020-07-30 15:15:34] tail -n +1 /app/public/converted_csv/hessler_hessler__nodes_22405.csv >> /app/public/diff/hessler_hessler__nodes_22405.diff
[CMD] [2020-07-30 15:15:34] echo "." >> /app/public/diff/hessler_hessler__nodes_22405.diff
[CMD] [2020-07-30 15:15:34] echo "0a" > /app/public/diff/hessler_hessler__occurrences_22406.diff
[CMD] [2020-07-30 15:15:34] tail -n +1 /app/public/converted_csv/hessler_hessler__occurrences_22406.csv >> /app/public/diff/hessler_hessler__occurrences_22406.diff
[CMD] [2020-07-30 15:15:34] echo "." >> /app/public/diff/hessler_hessler__occurrences_22406.diff
[CMD] [2020-07-30 15:15:34] echo "0a" > /app/public/diff/hessler_hessler__measurements_22407.diff
[CMD] [2020-07-30 15:15:34] tail -n +1 /app/public/converted_csv/hessler_hessler__measurements_22407.csv >> /app/public/diff/hessler_hessler__measurements_22407.diff
[CMD] [2020-07-30 15:15:34] echo "." >> /app/public/diff/hessler_hessler__measurements_22407.diff
[STOP] [2020-07-30 15:15:34] calculate_delta
[START] [2020-07-30 15:15:34] parse_diff_and_store
[INFO] [2020-07-30 15:15:34] Loading refs diff file into memory (true lines)...
[INFO] [2020-07-30 15:15:34] Loading nodes diff file into memory (true lines)...
[INFO] [2020-07-30 15:15:34] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-07-30 15:15:34] Loading measurements diff file into memory (true lines)...
[INFO] [2020-07-30 15:15:34] Storing 20 ScientificNames
[INFO] [2020-07-30 15:15:34] Processing group of 20 in 1 groups of 1000
[INFO] [2020-07-30 15:15:34] Average Time: 0.01
[INFO] [2020-07-30 15:15:34] Total Time: 1s
[INFO] [2020-07-30 15:15:34] Storing 20 Nodes
[INFO] [2020-07-30 15:15:34] Processing group of 20 in 1 groups of 1000
[INFO] [2020-07-30 15:15:34] Average Time: 0.01
[INFO] [2020-07-30 15:15:34] Total Time: 1s
[INFO] [2020-07-30 15:15:34] Storing 20 Occurrences
[INFO] [2020-07-30 15:15:34] Processing group of 20 in 1 groups of 1000
[INFO] [2020-07-30 15:15:34] Average Time: 0.0
[INFO] [2020-07-30 15:15:34] Total Time: 1s
[INFO] [2020-07-30 15:15:34] Storing 20 OccurrenceMetadata
[INFO] [2020-07-30 15:15:34] Processing group of 20 in 1 groups of 1000
[INFO] [2020-07-30 15:15:34] Average Time: 0.0
[INFO] [2020-07-30 15:15:34] Total Time: 1s
[INFO] [2020-07-30 15:15:34] Storing 135 Traits
[INFO] [2020-07-30 15:15:34] Processing group of 135 in 1 groups of 1000
[INFO] [2020-07-30 15:15:34] Average Time: 0.04
[INFO] [2020-07-30 15:15:34] Total Time: 1s
[INFO] [2020-07-30 15:15:34] Storing 106 MetaTraits
[INFO] [2020-07-30 15:15:34] Processing group of 106 in 1 groups of 1000
[INFO] [2020-07-30 15:15:34] Average Time: 0.01
[INFO] [2020-07-30 15:15:34] Total Time: 1s
[STOP] [2020-07-30 15:15:34] parse_diff_and_store
[START] [2020-07-30 15:15:34] resolve_keys
[INFO] [2020-07-30 15:15:41] Occurrences to nodes (through scientific_names)...
[INFO] [2020-07-30 15:15:41] traits to occurrences...
[INFO] [2020-07-30 15:15:41] traits to nodes (through occurrences)...
[INFO] [2020-07-30 15:15:41] Traits to sex term...
[INFO] [2020-07-30 15:15:41] Traits to lifestage term...
[INFO] [2020-07-30 15:15:41] MetaTraits to traits...
[INFO] [2020-07-30 15:15:41] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-07-30 15:15:41] Assocs to occurrences...
[INFO] [2020-07-30 15:15:41] Assocs to nodes...
[INFO] [2020-07-30 15:15:41] Assoc to sex term...
[INFO] [2020-07-30 15:15:41] Assoc to lifestage term...
[STOP] [2020-07-30 15:15:41] resolve_keys
[START] [2020-07-30 15:15:41] hold_for_later_1
[STOP] [2020-07-30 15:15:41] hold_for_later_1
[START] [2020-07-30 15:15:41] hold_for_later_2
[STOP] [2020-07-30 15:15:41] hold_for_later_2
[START] [2020-07-30 15:15:41] resolve_missing_parents
[STOP] [2020-07-30 15:15:41] resolve_missing_parents
[START] [2020-07-30 15:15:41] rebuild_nodes
[START] [2020-07-30 15:15:41] Flattener#flatten
[START] [2020-07-30 15:15:41] Flattener#study_resource
[START] [2020-07-30 15:15:41] Flattener#build_ancestry
[STOP] [2020-07-30 15:15:41] Flattener#build_ancestry
[INFO] [2020-07-30 15:15:41] 20 ancestry keys
[START] [2020-07-30 15:15:41] build_node_ancestors
[INFO] [2020-07-30 15:15:41] old ancestors deleted.
[STOP] [2020-07-30 15:15:41] build_node_ancestors
[WARN] [2020-07-30 15:15:41] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-07-30 15:15:41] Flattener#flatten
[STOP] [2020-07-30 15:15:41] rebuild_nodes
[START] [2020-07-30 15:15:41] resolve_missing_media_owners
[STOP] [2020-07-30 15:15:41] resolve_missing_media_owners
[START] [2020-07-30 15:15:41] sanitize_media_verbatims
[STOP] [2020-07-30 15:15:41] sanitize_media_verbatims
[START] [2020-07-30 15:15:41] queue_downloads
[STOP] [2020-07-30 15:15:41] queue_downloads
[START] [2020-07-30 15:15:41] parse_names
[WARN] [2020-07-30 15:15:41] I see 20 names which still need to be parsed.
[STOP] [2020-07-30 15:15:42] parse_names
[START] [2020-07-30 15:15:42] denormalize_canonical_names_to_nodes
[STOP] [2020-07-30 15:15:42] denormalize_canonical_names_to_nodes
[START] [2020-07-30 15:15:42] match_nodes
[START] [2020-07-30 15:15:42] map_all_nodes_to_pages
[STOP] [2020-07-30 15:15:42] map_all_nodes_to_pages
[INFO] [2020-07-30 15:15:42] ZERO unmatched nodes (of 20)! Nicely done.
[START] [2020-07-30 15:15:42] update_nodes
[STOP] [2020-07-30 15:15:42] update_nodes
[STOP] [2020-07-30 15:15:42] match_nodes
[START] [2020-07-30 15:15:42] reindex_search
[STOP] [2020-07-30 15:15:42] reindex_search
[START] [2020-07-30 15:15:42] normalize_units
[STOP] [2020-07-30 15:15:42] normalize_units
[START] [2020-07-30 15:15:42] calculate_statistics
[2020-07-30 15:15:42] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-07-30 15:15:42] calculate_statistics
[START] [2020-07-30 15:15:42] complete_harvest_instance
[START] [2020-07-30 15:15:42] overall_tsv_creation
[INFO] [2020-07-30 15:15:42] Processing group of 20 in 1 batches of 10000
[INFO] [2020-07-30 15:16:31] 41 Traits (unfiltered)...
[INFO] [2020-07-30 15:17:08] 41 Traits (filtered)...
[INFO] [2020-07-30 15:17:08] 0 Associations (filtered)...
[INFO] [2020-07-30 15:17:08] 241 metadata added.
[INFO] [2020-07-30 15:17:08] 0 metadata added.
[INFO] [2020-07-30 15:17:08] Average Time: 58.77
[INFO] [2020-07-30 15:17:08] Total Time: 1m26s
[STOP] [2020-07-30 15:17:08] overall_tsv_creation
[INFO] [2020-07-30 15:17:08] Done. Check your files:
[INFO] [2020-07-30 15:17:08] (20 lines) /app/public/data/hessler_hessler_/publish_nodes.tsv
[INFO] [2020-07-30 15:17:08] (20 lines) /app/public/data/hessler_hessler_/publish_scientific_names.tsv
[INFO] [2020-07-30 15:17:08] (42 lines) /app/public/data/hessler_hessler_/publish_traits.tsv
[INFO] [2020-07-30 15:17:08] (136 lines) /app/public/data/hessler_hessler_/publish_metadata.tsv
[STOP] [2020-07-30 15:17:08] complete_harvest_instance
[START] [2020-07-30 15:17:08] completed
[STOP] [2020-07-30 15:17:08] completed
[STOP] [2020-07-30 15:17:08] logged process, took 95.98
[INFO] [2020-07-30 15:28:19] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-07-30 15:28:24] ## remove_type: ScientificName
[INFO] [2020-07-30 15:28:24] ++ Calling delete_all on 20 instances...
[INFO] [2020-07-30 15:28:24] [15:28:24.174] Removed 20 Scientificnames
[INFO] [2020-07-30 15:28:24] ## remove_type: Vernacular
[INFO] [2020-07-30 15:28:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-30 15:28:24] [15:28:24.177] Removed 0 Vernaculars
[INFO] [2020-07-30 15:28:24] ## remove_type: Article
[INFO] [2020-07-30 15:28:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-30 15:28:24] [15:28:24.180] Removed 0 Articles
[INFO] [2020-07-30 15:28:24] ## remove_type: Medium
[INFO] [2020-07-30 15:28:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-30 15:28:24] [15:28:24.184] Removed 0 Media
[INFO] [2020-07-30 15:28:24] ## remove_type: Trait
[INFO] [2020-07-30 15:28:24] ++ Calling delete_all on 135 instances...
[INFO] [2020-07-30 15:28:24] [15:28:24.193] Removed 135 Traits
[INFO] [2020-07-30 15:28:24] ## remove_type: MetaTrait
[INFO] [2020-07-30 15:28:24] ++ Calling delete_all on 106 instances...
[INFO] [2020-07-30 15:28:24] [15:28:24.197] Removed 106 Metatraits
[INFO] [2020-07-30 15:28:24] ## remove_type: OccurrenceMetadatum
[INFO] [2020-07-30 15:28:24] ++ Calling delete_all on 20 instances...
[INFO] [2020-07-30 15:28:24] [15:28:24.201] Removed 20 Occurrencemetadata
[INFO] [2020-07-30 15:28:24] ## remove_type: Assoc
[INFO] [2020-07-30 15:28:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-30 15:28:24] [15:28:24.204] Removed 0 Assocs
[INFO] [2020-07-30 15:28:24] ## remove_type: MetaAssoc
[INFO] [2020-07-30 15:28:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-30 15:28:24] [15:28:24.206] Removed 0 Metaassocs
[INFO] [2020-07-30 15:28:24] ## remove_type: Identifier
[INFO] [2020-07-30 15:28:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-30 15:28:24] [15:28:24.209] Removed 0 Identifiers
[INFO] [2020-07-30 15:28:24] ## remove_type: Reference
[INFO] [2020-07-30 15:28:24] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-30 15:28:24] [15:28:24.211] Removed 0 References
[INFO] [2020-07-30 15:28:24] Starting batch with ID 80634570...
[INFO] [2020-07-30 15:28:24] Starting batch with ID 80634570...
[INFO] [2020-07-30 15:28:24] Starting batch with ID 80634570...
[INFO] [2020-07-30 15:28:24] Starting batch with ID 80634570...
[INFO] [2020-07-30 15:28:24] Starting batch with ID 80634570...
[INFO] [2020-07-30 15:28:24] Starting batch with ID 80634570...
[INFO] [2020-07-30 15:28:24] Starting batch with ID 80634570...
[INFO] [2020-07-30 15:28:24] Starting batch with ID 80634570...
[INFO] [2020-07-30 15:28:24] Starting batch with ID 80634570...
[INFO] [2020-07-30 15:28:24] Starting batch with ID 80634570...
[INFO] [2020-07-30 15:28:24] Starting batch with ID 80634570...
[INFO] [2020-07-30 15:28:24] Starting batch with ID 80634570...
[INFO] [2020-07-30 15:28:24] Starting batch with ID 80634570...
[INFO] [2020-07-30 15:28:24] Starting batch with ID 80634570...
[INFO] [2020-07-30 15:28:24] Starting batch with ID 80634570...
[INFO] [2020-07-30 15:28:24] Starting batch with ID 80634570...
[INFO] [2020-07-30 15:28:24] Starting batch with ID 80634570...
[INFO] [2020-07-30 15:28:24] Starting batch with ID 80634570...
[INFO] [2020-07-30 15:28:25] Starting batch with ID 80634570...
[INFO] [2020-07-30 15:28:25] ## remove_type: Node
[INFO] [2020-07-30 15:28:25] ++ Calling delete_all on 20 instances...
[INFO] [2020-07-30 15:28:25] [15:28:25.266] Removed 20 Nodes
[START] [2020-07-30 15:28:25] logged process
[START] [2020-07-30 15:28:25] Creating resource from OpenData
[START] [2020-07-30 15:28:25] logged process
[START] [2020-07-30 15:28:25] Parse meta.xml file and create formats with fields
[STOP] [2020-07-30 15:28:25] Parse meta.xml file and create formats with fields
[STOP] [2020-07-30 15:28:25] Creating resource from OpenData
[START] [2020-07-30 15:28:25] logged process
[START] [2020-07-30 15:28:25] create_harvest_instance
[STOP] [2020-07-30 15:28:27] create_harvest_instance
[START] [2020-07-30 15:28:27] fetch_files
[STOP] [2020-07-30 15:28:27] fetch_files
[START] [2020-07-30 15:28:27] validate_each_file
[STOP] [2020-07-30 15:28:27] validate_each_file
[START] [2020-07-30 15:28:27] convert_to_csv
[CMD] [2020-07-30 15:28:27] /usr/bin/sort /app/public/converted_csv/hessler_hessler__refs_22412.csv > /app/public/converted_csv/hessler_hessler__refs_22412.csv_sorted
[CMD] [2020-07-30 15:28:27] /usr/bin/sort /app/public/converted_csv/hessler_hessler__nodes_22413.csv > /app/public/converted_csv/hessler_hessler__nodes_22413.csv_sorted
[CMD] [2020-07-30 15:28:27] /usr/bin/sort /app/public/converted_csv/hessler_hessler__occurrences_22414.csv > /app/public/converted_csv/hessler_hessler__occurrences_22414.csv_sorted
[CMD] [2020-07-30 15:28:27] /usr/bin/sort /app/public/converted_csv/hessler_hessler__measurements_22415.csv > /app/public/converted_csv/hessler_hessler__measurements_22415.csv_sorted
[STOP] [2020-07-30 15:28:27] convert_to_csv
[START] [2020-07-30 15:28:27] calculate_delta
[CMD] [2020-07-30 15:28:27] echo "0a" > /app/public/diff/hessler_hessler__refs_22412.diff
[CMD] [2020-07-30 15:28:27] tail -n +1 /app/public/converted_csv/hessler_hessler__refs_22412.csv >> /app/public/diff/hessler_hessler__refs_22412.diff
[CMD] [2020-07-30 15:28:27] echo "." >> /app/public/diff/hessler_hessler__refs_22412.diff
[CMD] [2020-07-30 15:28:27] echo "0a" > /app/public/diff/hessler_hessler__nodes_22413.diff
[CMD] [2020-07-30 15:28:27] tail -n +1 /app/public/converted_csv/hessler_hessler__nodes_22413.csv >> /app/public/diff/hessler_hessler__nodes_22413.diff
[CMD] [2020-07-30 15:28:27] echo "." >> /app/public/diff/hessler_hessler__nodes_22413.diff
[CMD] [2020-07-30 15:28:27] echo "0a" > /app/public/diff/hessler_hessler__occurrences_22414.diff
[CMD] [2020-07-30 15:28:27] tail -n +1 /app/public/converted_csv/hessler_hessler__occurrences_22414.csv >> /app/public/diff/hessler_hessler__occurrences_22414.diff
[CMD] [2020-07-30 15:28:27] echo "." >> /app/public/diff/hessler_hessler__occurrences_22414.diff
[CMD] [2020-07-30 15:28:27] echo "0a" > /app/public/diff/hessler_hessler__measurements_22415.diff
[CMD] [2020-07-30 15:28:27] tail -n +1 /app/public/converted_csv/hessler_hessler__measurements_22415.csv >> /app/public/diff/hessler_hessler__measurements_22415.diff
[CMD] [2020-07-30 15:28:27] echo "." >> /app/public/diff/hessler_hessler__measurements_22415.diff
[STOP] [2020-07-30 15:28:27] calculate_delta
[START] [2020-07-30 15:28:27] parse_diff_and_store
[INFO] [2020-07-30 15:28:27] Loading refs diff file into memory (true lines)...
[INFO] [2020-07-30 15:28:27] Loading nodes diff file into memory (true lines)...
[INFO] [2020-07-30 15:28:27] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-07-30 15:28:27] Loading measurements diff file into memory (true lines)...
[INFO] [2020-07-30 15:28:28] Storing 20 ScientificNames
[INFO] [2020-07-30 15:28:28] Processing group of 20 in 1 groups of 1000
[INFO] [2020-07-30 15:28:28] Average Time: 0.01
[INFO] [2020-07-30 15:28:28] Total Time: 1s
[INFO] [2020-07-30 15:28:28] Storing 20 Nodes
[INFO] [2020-07-30 15:28:28] Processing group of 20 in 1 groups of 1000
[INFO] [2020-07-30 15:28:28] Average Time: 0.01
[INFO] [2020-07-30 15:28:28] Total Time: 1s
[INFO] [2020-07-30 15:28:28] Storing 20 Occurrences
[INFO] [2020-07-30 15:28:28] Processing group of 20 in 1 groups of 1000
[INFO] [2020-07-30 15:28:28] Average Time: 0.0
[INFO] [2020-07-30 15:28:28] Total Time: 1s
[INFO] [2020-07-30 15:28:28] Storing 20 OccurrenceMetadata
[INFO] [2020-07-30 15:28:28] Processing group of 20 in 1 groups of 1000
[INFO] [2020-07-30 15:28:28] Average Time: 0.0
[INFO] [2020-07-30 15:28:28] Total Time: 1s
[INFO] [2020-07-30 15:28:28] Storing 133 Traits
[INFO] [2020-07-30 15:28:28] Processing group of 133 in 1 groups of 1000
[INFO] [2020-07-30 15:28:28] Average Time: 0.03
[INFO] [2020-07-30 15:28:28] Total Time: 1s
[INFO] [2020-07-30 15:28:28] Storing 102 MetaTraits
[INFO] [2020-07-30 15:28:28] Processing group of 102 in 1 groups of 1000
[INFO] [2020-07-30 15:28:28] Average Time: 0.01
[INFO] [2020-07-30 15:28:28] Total Time: 1s
[STOP] [2020-07-30 15:28:28] parse_diff_and_store
[START] [2020-07-30 15:28:28] resolve_keys
[INFO] [2020-07-30 15:28:34] Occurrences to nodes (through scientific_names)...
[INFO] [2020-07-30 15:28:34] traits to occurrences...
[INFO] [2020-07-30 15:28:34] traits to nodes (through occurrences)...
[INFO] [2020-07-30 15:28:34] Traits to sex term...
[INFO] [2020-07-30 15:28:34] Traits to lifestage term...
[INFO] [2020-07-30 15:28:34] MetaTraits to traits...
[INFO] [2020-07-30 15:28:34] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-07-30 15:28:34] Assocs to occurrences...
[INFO] [2020-07-30 15:28:34] Assocs to nodes...
[INFO] [2020-07-30 15:28:34] Assoc to sex term...
[INFO] [2020-07-30 15:28:34] Assoc to lifestage term...
[STOP] [2020-07-30 15:28:34] resolve_keys
[START] [2020-07-30 15:28:34] hold_for_later_1
[STOP] [2020-07-30 15:28:34] hold_for_later_1
[START] [2020-07-30 15:28:34] hold_for_later_2
[STOP] [2020-07-30 15:28:34] hold_for_later_2
[START] [2020-07-30 15:28:34] resolve_missing_parents
[STOP] [2020-07-30 15:28:34] resolve_missing_parents
[START] [2020-07-30 15:28:34] rebuild_nodes
[START] [2020-07-30 15:28:34] Flattener#flatten
[START] [2020-07-30 15:28:34] Flattener#study_resource
[START] [2020-07-30 15:28:34] Flattener#build_ancestry
[STOP] [2020-07-30 15:28:34] Flattener#build_ancestry
[INFO] [2020-07-30 15:28:34] 20 ancestry keys
[START] [2020-07-30 15:28:34] build_node_ancestors
[INFO] [2020-07-30 15:28:34] old ancestors deleted.
[STOP] [2020-07-30 15:28:34] build_node_ancestors
[WARN] [2020-07-30 15:28:34] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-07-30 15:28:34] Flattener#flatten
[STOP] [2020-07-30 15:28:34] rebuild_nodes
[START] [2020-07-30 15:28:34] resolve_missing_media_owners
[STOP] [2020-07-30 15:28:34] resolve_missing_media_owners
[START] [2020-07-30 15:28:34] sanitize_media_verbatims
[STOP] [2020-07-30 15:28:34] sanitize_media_verbatims
[START] [2020-07-30 15:28:34] queue_downloads
[STOP] [2020-07-30 15:28:34] queue_downloads
[START] [2020-07-30 15:28:34] parse_names
[WARN] [2020-07-30 15:28:34] I see 20 names which still need to be parsed.
[STOP] [2020-07-30 15:28:36] parse_names
[START] [2020-07-30 15:28:36] denormalize_canonical_names_to_nodes
[STOP] [2020-07-30 15:28:36] denormalize_canonical_names_to_nodes
[START] [2020-07-30 15:28:36] match_nodes
[START] [2020-07-30 15:28:36] map_all_nodes_to_pages
[STOP] [2020-07-30 15:28:36] map_all_nodes_to_pages
[INFO] [2020-07-30 15:28:36] ZERO unmatched nodes (of 20)! Nicely done.
[START] [2020-07-30 15:28:36] update_nodes
[STOP] [2020-07-30 15:28:36] update_nodes
[STOP] [2020-07-30 15:28:36] match_nodes
[START] [2020-07-30 15:28:36] reindex_search
[STOP] [2020-07-30 15:28:36] reindex_search
[START] [2020-07-30 15:28:36] normalize_units
[STOP] [2020-07-30 15:28:36] normalize_units
[START] [2020-07-30 15:28:36] calculate_statistics
[2020-07-30 15:28:36] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-07-30 15:28:36] calculate_statistics
[START] [2020-07-30 15:28:36] complete_harvest_instance
[START] [2020-07-30 15:28:36] overall_tsv_creation
[INFO] [2020-07-30 15:28:36] Processing group of 20 in 1 batches of 10000
[INFO] [2020-07-30 15:29:25] 39 Traits (unfiltered)...
[INFO] [2020-07-30 15:30:02] 39 Traits (filtered)...
[INFO] [2020-07-30 15:30:02] 0 Associations (filtered)...
[INFO] [2020-07-30 15:30:02] 233 metadata added.
[INFO] [2020-07-30 15:30:02] 0 metadata added.
[INFO] [2020-07-30 15:30:02] Average Time: 58.87
[INFO] [2020-07-30 15:30:02] Total Time: 1m26s
[STOP] [2020-07-30 15:30:02] overall_tsv_creation
[INFO] [2020-07-30 15:30:02] Done. Check your files:
[INFO] [2020-07-30 15:30:02] (20 lines) /app/public/data/hessler_hessler_/publish_nodes.tsv
[INFO] [2020-07-30 15:30:02] (20 lines) /app/public/data/hessler_hessler_/publish_scientific_names.tsv
[INFO] [2020-07-30 15:30:02] (40 lines) /app/public/data/hessler_hessler_/publish_traits.tsv
[INFO] [2020-07-30 15:30:02] (132 lines) /app/public/data/hessler_hessler_/publish_metadata.tsv
[STOP] [2020-07-30 15:30:02] complete_harvest_instance
[START] [2020-07-30 15:30:02] completed
[STOP] [2020-07-30 15:30:02] completed
[STOP] [2020-07-30 15:30:02] logged process, took 96.4

Latest Process

30 Jul 15:28 EXITED currently processing 0