Harvest for Zieger and Meyer Rochow 2008 Created 19 Apr 10:26

Stage: completed
Fetched: 19 Apr 10:26
Validated: 19 Apr 10:26
Deltas Created 19 Apr 10:26
Units Normalized: 19 Apr 10:26
Ancestry Built: 19 Apr 10:26
Nodes Matched: 19 Apr 10:26
Names Parsed: 19 Apr 10:26
New Models Stored: 19 Apr 10:26
Indexed: 19 Apr 10:26
Completed: 19 Apr 10:28
Time to Harvest: less than a minute

Harvesting Log (most recent first)

# Logfile created on 2020-08-27 15:50:32 -0400 by logger.rb/v1.4.2
[START] [2020-08-27 15:50:32] logged process
[START] [2020-08-27 15:50:32] Creating resource from OpenData
[START] [2020-08-27 15:50:32] logged process
[START] [2020-08-27 15:50:32] Parse meta.xml file and create formats with fields
[STOP] [2020-08-27 15:50:32] Parse meta.xml file and create formats with fields
[STOP] [2020-08-27 15:50:32] Creating resource from OpenData
[INFO] [2020-12-03 12:15:34] ## HARVEST: type = -harvest
[START] [2020-12-03 12:15:35] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-12-03 12:15:35] create_harvest_instance
[STOP] [2020-12-03 12:15:36] create_harvest_instance
[START] [2020-12-03 12:15:36] fetch_files
[STOP] [2020-12-03 12:15:36] fetch_files
[START] [2020-12-03 12:15:36] validate_each_file
[STOP] [2020-12-03 12:15:36] validate_each_file
[ERR] [2020-12-03 12:15:36] CSV::MalformedCSVError
[ERR] [2020-12-03 12:15:36] Unquoted fields do not allow new line <"\r"> in line 2.
[ERR] [2020-12-03 12:15:36] ../models/csv_parser.rb:25:in `line_at_a_time'
[ERR] [2020-12-03 12:15:36] ../models/csv_parser.rb:81:in `rows_as_hashes'
[ERR] [2020-12-03 12:15:36] ../models/resource_harvester.rb:176:in `validate_csv'
[ERR] [2020-12-03 12:15:36] ../models/resource_harvester.rb:143:in `block (2 levels) in validate_each_file'
[ERR] [2020-12-03 12:15:36] ../models/resource_harvester.rb:142:in `block in validate_each_file'
[ERR] [2020-12-03 12:15:36] ../models/resource_harvester.rb:682:in `block in each_format'
[ERR] [2020-12-03 12:15:36] ../models/resource_harvester.rb:667:in `each_format'
[ERR] [2020-12-03 12:15:36] ../models/resource_harvester.rb:129:in `validate_each_file'
[ERR] [2020-12-03 12:15:36] ../models/resource_harvester.rb:85:in `block (3 levels) in start'
[ERR] [2020-12-03 12:15:36] ../models/logged_process.rb:19:in `run_step'
[ERR] [2020-12-03 12:15:36] ../models/resource_harvester.rb:85:in `block (2 levels) in start'
[ERR] [2020-12-03 12:15:36] ../models/resource_harvester.rb:74:in `each_key'
[ERR] [2020-12-03 12:15:36] ../models/resource_harvester.rb:74:in `block in start'
[ERR] [2020-12-03 12:15:36] ../models/resource.rb:151:in `lock'
[ERR] [2020-12-03 12:15:36] ../models/resource_harvester.rb:72:in `start'
[ERR] [2020-12-03 12:15:36] ../models/resource.rb:232:in `harvest'
[ERR] [2020-12-03 12:15:36] bin/rails:4:in `require'
[ERR] [2020-12-03 12:15:36] bin/rails:4:in `<main>'
[STOP] [2020-12-03 12:15:36] logged process, took 1.58
[INFO] [2020-12-03 12:18:25] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-12-03 12:18:28] ## remove_type: ScientificName
[INFO] [2020-12-03 12:18:28] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-03 12:18:28] [12:18:28.284] Removed 0 Scientificnames
[INFO] [2020-12-03 12:18:28] ## remove_type: Vernacular
[INFO] [2020-12-03 12:18:28] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-03 12:18:28] [12:18:28.288] Removed 0 Vernaculars
[INFO] [2020-12-03 12:18:28] ## remove_type: Article
[INFO] [2020-12-03 12:18:28] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-03 12:18:28] [12:18:28.291] Removed 0 Articles
[INFO] [2020-12-03 12:18:28] ## remove_type: Medium
[INFO] [2020-12-03 12:18:28] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-03 12:18:28] [12:18:28.294] Removed 0 Media
[INFO] [2020-12-03 12:18:28] ## remove_type: Trait
[INFO] [2020-12-03 12:18:28] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-03 12:18:28] [12:18:28.298] Removed 0 Traits
[INFO] [2020-12-03 12:18:28] ## remove_type: MetaTrait
[INFO] [2020-12-03 12:18:28] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-03 12:18:28] [12:18:28.301] Removed 0 Metatraits
[INFO] [2020-12-03 12:18:28] ## remove_type: OccurrenceMetadatum
[INFO] [2020-12-03 12:18:28] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-03 12:18:28] [12:18:28.305] Removed 0 Occurrencemetadata
[INFO] [2020-12-03 12:18:28] ## remove_type: Assoc
[INFO] [2020-12-03 12:18:28] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-03 12:18:28] [12:18:28.307] Removed 0 Assocs
[INFO] [2020-12-03 12:18:28] ## remove_type: MetaAssoc
[INFO] [2020-12-03 12:18:28] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-03 12:18:28] [12:18:28.310] Removed 0 Metaassocs
[INFO] [2020-12-03 12:18:28] ## remove_type: Identifier
[INFO] [2020-12-03 12:18:28] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-03 12:18:28] [12:18:28.313] Removed 0 Identifiers
[INFO] [2020-12-03 12:18:28] ## remove_type: Reference
[INFO] [2020-12-03 12:18:28] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-03 12:18:28] [12:18:28.316] Removed 0 References
[INFO] [2020-12-03 12:18:28] ## remove_type: Node
[INFO] [2020-12-03 12:18:28] ++ Calling delete_all on 0 instances...
[INFO] [2020-12-03 12:18:28] [12:18:28.336] Removed 0 Nodes
[START] [2020-12-03 12:18:28] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-12-03 12:18:28] Creating resource from OpenData
[START] [2020-12-03 12:18:28] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-12-03 12:18:28] Parse meta.xml file and create formats with fields
[STOP] [2020-12-03 12:18:28] Parse meta.xml file and create formats with fields
[STOP] [2020-12-03 12:18:28] Creating resource from OpenData
[START] [2020-12-03 12:18:28] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-12-03 12:18:28] create_harvest_instance
[STOP] [2020-12-03 12:18:30] create_harvest_instance
[START] [2020-12-03 12:18:30] fetch_files
[STOP] [2020-12-03 12:18:30] fetch_files
[START] [2020-12-03 12:18:30] validate_each_file
[STOP] [2020-12-03 12:18:30] validate_each_file
[START] [2020-12-03 12:18:30] convert_to_csv
[CMD] [2020-12-03 12:18:30] /usr/bin/sort /app/public/converted_csv/zieger_meyer_roc_refs_24900.csv > /app/public/converted_csv/zieger_meyer_roc_refs_24900.csv_sorted
[CMD] [2020-12-03 12:18:30] /usr/bin/sort /app/public/converted_csv/zieger_meyer_roc_nodes_24901.csv > /app/public/converted_csv/zieger_meyer_roc_nodes_24901.csv_sorted
[CMD] [2020-12-03 12:18:30] /usr/bin/sort /app/public/converted_csv/zieger_meyer_roc_occurrences_24902.csv > /app/public/converted_csv/zieger_meyer_roc_occurrences_24902.csv_sorted
[CMD] [2020-12-03 12:18:30] /usr/bin/sort /app/public/converted_csv/zieger_meyer_roc_measurements_24903.csv > /app/public/converted_csv/zieger_meyer_roc_measurements_24903.csv_sorted
[STOP] [2020-12-03 12:18:30] convert_to_csv
[START] [2020-12-03 12:18:30] calculate_delta
[CMD] [2020-12-03 12:18:30] echo "0a" > /app/public/diff/zieger_meyer_roc_refs_24900.diff
[CMD] [2020-12-03 12:18:30] tail -n +1 /app/public/converted_csv/zieger_meyer_roc_refs_24900.csv >> /app/public/diff/zieger_meyer_roc_refs_24900.diff
[CMD] [2020-12-03 12:18:30] echo "." >> /app/public/diff/zieger_meyer_roc_refs_24900.diff
[CMD] [2020-12-03 12:18:30] echo "0a" > /app/public/diff/zieger_meyer_roc_nodes_24901.diff
[CMD] [2020-12-03 12:18:30] tail -n +1 /app/public/converted_csv/zieger_meyer_roc_nodes_24901.csv >> /app/public/diff/zieger_meyer_roc_nodes_24901.diff
[CMD] [2020-12-03 12:18:30] echo "." >> /app/public/diff/zieger_meyer_roc_nodes_24901.diff
[CMD] [2020-12-03 12:18:30] echo "0a" > /app/public/diff/zieger_meyer_roc_occurrences_24902.diff
[CMD] [2020-12-03 12:18:30] tail -n +1 /app/public/converted_csv/zieger_meyer_roc_occurrences_24902.csv >> /app/public/diff/zieger_meyer_roc_occurrences_24902.diff
[CMD] [2020-12-03 12:18:30] echo "." >> /app/public/diff/zieger_meyer_roc_occurrences_24902.diff
[CMD] [2020-12-03 12:18:30] echo "0a" > /app/public/diff/zieger_meyer_roc_measurements_24903.diff
[CMD] [2020-12-03 12:18:30] tail -n +1 /app/public/converted_csv/zieger_meyer_roc_measurements_24903.csv >> /app/public/diff/zieger_meyer_roc_measurements_24903.diff
[CMD] [2020-12-03 12:18:30] echo "." >> /app/public/diff/zieger_meyer_roc_measurements_24903.diff
[STOP] [2020-12-03 12:18:30] calculate_delta
[START] [2020-12-03 12:18:30] parse_diff_and_store
[INFO] [2020-12-03 12:18:30] Loading refs diff file into memory (true lines)...
[INFO] [2020-12-03 12:18:30] Loading nodes diff file into memory (true lines)...
[INFO] [2020-12-03 12:18:30] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-12-03 12:18:30] Loading measurements diff file into memory (true lines)...
[INFO] [2020-12-03 12:18:31] Storing 2 References
[INFO] [2020-12-03 12:18:31] Processing group of 2 in 1 groups of 1000
[INFO] [2020-12-03 12:18:31] Average Time: 0.0
[INFO] [2020-12-03 12:18:31] Total Time: 1s
[INFO] [2020-12-03 12:18:31] Storing 9 ScientificNames
[INFO] [2020-12-03 12:18:31] Processing group of 9 in 1 groups of 1000
[INFO] [2020-12-03 12:18:31] Average Time: 0.03
[INFO] [2020-12-03 12:18:31] Total Time: 1s
[INFO] [2020-12-03 12:18:31] Storing 9 Nodes
[INFO] [2020-12-03 12:18:31] Processing group of 9 in 1 groups of 1000
[INFO] [2020-12-03 12:18:31] Average Time: 0.01
[INFO] [2020-12-03 12:18:31] Total Time: 1s
[INFO] [2020-12-03 12:18:31] Storing 9 Occurrences
[INFO] [2020-12-03 12:18:31] Processing group of 9 in 1 groups of 1000
[INFO] [2020-12-03 12:18:31] Average Time: 0.0
[INFO] [2020-12-03 12:18:31] Total Time: 1s
[INFO] [2020-12-03 12:18:31] Storing 32 Traits
[INFO] [2020-12-03 12:18:31] Processing group of 32 in 1 groups of 1000
[INFO] [2020-12-03 12:18:31] Average Time: 0.01
[INFO] [2020-12-03 12:18:31] Total Time: 1s
[INFO] [2020-12-03 12:18:31] Storing 16 MetaTraits
[INFO] [2020-12-03 12:18:31] Processing group of 16 in 1 groups of 1000
[INFO] [2020-12-03 12:18:31] Average Time: 0.0
[INFO] [2020-12-03 12:18:31] Total Time: 1s
[INFO] [2020-12-03 12:18:31] Storing 2 TraitsReferences
[INFO] [2020-12-03 12:18:31] Processing group of 2 in 1 groups of 1000
[INFO] [2020-12-03 12:18:31] Average Time: 0.0
[INFO] [2020-12-03 12:18:31] Total Time: 1s
[STOP] [2020-12-03 12:18:31] parse_diff_and_store
[START] [2020-12-03 12:18:31] resolve_keys
[INFO] [2020-12-03 12:18:37] Occurrences to nodes (through scientific_names)...
[INFO] [2020-12-03 12:18:37] traits to occurrences...
[INFO] [2020-12-03 12:18:37] traits to nodes (through occurrences)...
[INFO] [2020-12-03 12:18:37] Traits to sex term...
[INFO] [2020-12-03 12:18:37] Traits to lifestage term...
[INFO] [2020-12-03 12:18:37] MetaTraits to traits...
[INFO] [2020-12-03 12:18:37] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-12-03 12:18:37] Assocs to occurrences...
[INFO] [2020-12-03 12:18:37] Assocs to nodes...
[INFO] [2020-12-03 12:18:37] Assoc to sex term...
[INFO] [2020-12-03 12:18:37] Assoc to lifestage term...
[INFO] [2020-12-03 12:18:37] MetaAssoc to assocs...
[STOP] [2020-12-03 12:18:37] resolve_keys
[START] [2020-12-03 12:18:37] hold_for_later_1
[STOP] [2020-12-03 12:18:37] hold_for_later_1
[START] [2020-12-03 12:18:37] hold_for_later_2
[STOP] [2020-12-03 12:18:37] hold_for_later_2
[START] [2020-12-03 12:18:37] resolve_missing_parents
[STOP] [2020-12-03 12:18:37] resolve_missing_parents
[START] [2020-12-03 12:18:37] rebuild_nodes
[START] [2020-12-03 12:18:37] Flattener#flatten
[START] [2020-12-03 12:18:37] Flattener#study_resource
[START] [2020-12-03 12:18:37] Flattener#build_ancestry
[STOP] [2020-12-03 12:18:37] Flattener#build_ancestry
[INFO] [2020-12-03 12:18:37] 9 ancestry keys
[START] [2020-12-03 12:18:37] build_node_ancestors
[INFO] [2020-12-03 12:18:37] old ancestors deleted.
[STOP] [2020-12-03 12:18:37] build_node_ancestors
[WARN] [2020-12-03 12:18:37] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-12-03 12:18:37] Flattener#flatten
[STOP] [2020-12-03 12:18:37] rebuild_nodes
[START] [2020-12-03 12:18:37] resolve_missing_media_owners
[STOP] [2020-12-03 12:18:37] resolve_missing_media_owners
[START] [2020-12-03 12:18:37] sanitize_media_verbatims
[STOP] [2020-12-03 12:18:37] sanitize_media_verbatims
[START] [2020-12-03 12:18:37] queue_downloads
[STOP] [2020-12-03 12:18:37] queue_downloads
[START] [2020-12-03 12:18:37] parse_names
[WARN] [2020-12-03 12:18:37] I see 9 names which still need to be parsed.
[STOP] [2020-12-03 12:18:38] parse_names
[START] [2020-12-03 12:18:38] denormalize_canonical_names_to_nodes
[STOP] [2020-12-03 12:18:38] denormalize_canonical_names_to_nodes
[START] [2020-12-03 12:18:38] match_nodes
[START] [2020-12-03 12:18:38] map_all_nodes_to_pages
[STOP] [2020-12-03 12:18:38] map_all_nodes_to_pages
[INFO] [2020-12-03 12:18:38] ZERO unmatched nodes (of 9)! Nicely done.
[START] [2020-12-03 12:18:38] update_nodes
[STOP] [2020-12-03 12:18:38] update_nodes
[STOP] [2020-12-03 12:18:38] match_nodes
[START] [2020-12-03 12:18:38] reindex_search
[STOP] [2020-12-03 12:18:38] reindex_search
[START] [2020-12-03 12:18:38] normalize_units
[STOP] [2020-12-03 12:18:38] normalize_units
[START] [2020-12-03 12:18:38] calculate_statistics
[2020-12-03 12:18:38] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-12-03 12:18:38] calculate_statistics
[START] [2020-12-03 12:18:38] complete_harvest_instance
[START] [2020-12-03 12:18:38] overall_tsv_creation
[INFO] [2020-12-03 12:18:38] Processing group of 9 in 1 batches of 10000
[INFO] [2020-12-03 12:19:17] 9 Traits (unfiltered)...
[INFO] [2020-12-03 12:19:56] 9 Traits (filtered)...
[INFO] [2020-12-03 12:19:56] 0 Associations (filtered)...
[INFO] [2020-12-03 12:19:56] 41 metadata added.
[INFO] [2020-12-03 12:19:56] 0 metadata added.
[INFO] [2020-12-03 12:19:56] Average Time: 54.55
[INFO] [2020-12-03 12:19:56] Total Time: 1m18s
[STOP] [2020-12-03 12:19:56] overall_tsv_creation
[INFO] [2020-12-03 12:19:56] Done. Check your files:
[INFO] [2020-12-03 12:19:56] (9 lines) /app/public/data/zieger_meyer_roc/publish_nodes.tsv
[INFO] [2020-12-03 12:19:56] (9 lines) /app/public/data/zieger_meyer_roc/publish_scientific_names.tsv
[INFO] [2020-12-03 12:19:56] (10 lines) /app/public/data/zieger_meyer_roc/publish_traits.tsv
[INFO] [2020-12-03 12:19:56] (42 lines) /app/public/data/zieger_meyer_roc/publish_metadata.tsv
[STOP] [2020-12-03 12:19:56] complete_harvest_instance
[START] [2020-12-03 12:19:56] completed
[STOP] [2020-12-03 12:19:56] completed
[STOP] [2020-12-03 12:19:56] logged process, took 87.26
[INFO] [2021-04-19 10:26:13] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-19 10:26:14] ## remove_type: ScientificName
[INFO] [2021-04-19 10:26:14] ++ Calling delete_all on 9 instances...
[INFO] [2021-04-19 10:26:14] [10:26:14.242] Removed 9 Scientificnames
[INFO] [2021-04-19 10:26:14] ## remove_type: Vernacular
[INFO] [2021-04-19 10:26:14] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:26:14] [10:26:14.244] Removed 0 Vernaculars
[INFO] [2021-04-19 10:26:14] ## remove_type: Article
[INFO] [2021-04-19 10:26:14] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:26:14] [10:26:14.245] Removed 0 Articles
[INFO] [2021-04-19 10:26:14] ## remove_type: Medium
[INFO] [2021-04-19 10:26:14] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:26:14] [10:26:14.247] Removed 0 Media
[INFO] [2021-04-19 10:26:14] ## remove_type: Trait
[INFO] [2021-04-19 10:26:14] ++ Calling delete_all on 32 instances...
[INFO] [2021-04-19 10:26:14] [10:26:14.273] Removed 32 Traits
[INFO] [2021-04-19 10:26:14] ## remove_type: MetaTrait
[INFO] [2021-04-19 10:26:14] ++ Calling delete_all on 16 instances...
[INFO] [2021-04-19 10:26:14] [10:26:14.275] Removed 16 Metatraits
[INFO] [2021-04-19 10:26:14] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-19 10:26:14] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:26:14] [10:26:14.277] Removed 0 Occurrencemetadata
[INFO] [2021-04-19 10:26:14] ## remove_type: Assoc
[INFO] [2021-04-19 10:26:14] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:26:14] [10:26:14.278] Removed 0 Assocs
[INFO] [2021-04-19 10:26:14] ## remove_type: MetaAssoc
[INFO] [2021-04-19 10:26:14] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:26:14] [10:26:14.280] Removed 0 Metaassocs
[INFO] [2021-04-19 10:26:14] ## remove_type: Identifier
[INFO] [2021-04-19 10:26:14] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:26:14] [10:26:14.281] Removed 0 Identifiers
[INFO] [2021-04-19 10:26:14] ## remove_type: Reference
[INFO] [2021-04-19 10:26:14] ++ Calling delete_all on 2 instances...
[INFO] [2021-04-19 10:26:14] [10:26:14.283] Removed 2 References
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] Starting batch with ID 86116227...
[INFO] [2021-04-19 10:26:14] ## remove_type: Node
[INFO] [2021-04-19 10:26:14] ++ Calling delete_all on 9 instances...
[INFO] [2021-04-19 10:26:14] [10:26:14.934] Removed 9 Nodes
[START] [2021-04-19 10:26:15] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 10:26:15] Creating resource from OpenData
[START] [2021-04-19 10:26:16] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 10:26:16] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 10:26:21] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 10:26:21] Creating resource from OpenData
[START] [2021-04-19 10:26:21] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 10:26:21] create_harvest_instance
[INFO] [2021-04-19 10:26:21] Created harvest instance #3726
[STOP] [2021-04-19 10:26:21] create_harvest_instance
[START] [2021-04-19 10:26:21] fetch_files
[STOP] [2021-04-19 10:26:21] fetch_files
[START] [2021-04-19 10:26:21] validate_each_file
[INFO] [2021-04-19 10:26:21] Looping over 4 formats...
[INFO] [2021-04-19 10:26:21] ...refs (/app/public/data/zieger_meyer_roc/references.tsv)
[INFO] [2021-04-19 10:26:21] Valid: /app/public/converted_csv/zieger_meyer_roc_refs_3726.csv (2 lines)
[INFO] [2021-04-19 10:26:21] ...nodes (/app/public/data/zieger_meyer_roc/taxa.txt)
[INFO] [2021-04-19 10:26:21] Valid: /app/public/converted_csv/zieger_meyer_roc_nodes_3726.csv (9 lines)
[INFO] [2021-04-19 10:26:21] ...occurrences (/app/public/data/zieger_meyer_roc/occurrences.txt)
[INFO] [2021-04-19 10:26:21] Valid: /app/public/converted_csv/zieger_meyer_roc_occurrences_3726.csv (9 lines)
[INFO] [2021-04-19 10:26:21] ...measurements (/app/public/data/zieger_meyer_roc/measurementsorfacts.txt)
[INFO] [2021-04-19 10:26:21] Valid: /app/public/converted_csv/zieger_meyer_roc_measurements_3726.csv (32 lines)
[STOP] [2021-04-19 10:26:21] validate_each_file
[START] [2021-04-19 10:26:21] convert_to_csv
[INFO] [2021-04-19 10:26:21] Looping over 4 formats...
[INFO] [2021-04-19 10:26:21] ...refs (/app/public/data/zieger_meyer_roc/references.tsv)
[CMD] [2021-04-19 10:26:21] /usr/bin/sort /app/public/converted_csv/zieger_meyer_roc_refs_3726.csv > /app/public/converted_csv/zieger_meyer_roc_refs_3726.csv_sorted
[INFO] [2021-04-19 10:26:21] Converted: /app/public/converted_csv/zieger_meyer_roc_refs_3726.csv (2 lines)
[INFO] [2021-04-19 10:26:21] ...nodes (/app/public/data/zieger_meyer_roc/taxa.txt)
[CMD] [2021-04-19 10:26:21] /usr/bin/sort /app/public/converted_csv/zieger_meyer_roc_nodes_3726.csv > /app/public/converted_csv/zieger_meyer_roc_nodes_3726.csv_sorted
[INFO] [2021-04-19 10:26:22] Converted: /app/public/converted_csv/zieger_meyer_roc_nodes_3726.csv (9 lines)
[INFO] [2021-04-19 10:26:22] ...occurrences (/app/public/data/zieger_meyer_roc/occurrences.txt)
[CMD] [2021-04-19 10:26:22] /usr/bin/sort /app/public/converted_csv/zieger_meyer_roc_occurrences_3726.csv > /app/public/converted_csv/zieger_meyer_roc_occurrences_3726.csv_sorted
[INFO] [2021-04-19 10:26:22] Converted: /app/public/converted_csv/zieger_meyer_roc_occurrences_3726.csv (9 lines)
[INFO] [2021-04-19 10:26:22] ...measurements (/app/public/data/zieger_meyer_roc/measurementsorfacts.txt)
[CMD] [2021-04-19 10:26:22] /usr/bin/sort /app/public/converted_csv/zieger_meyer_roc_measurements_3726.csv > /app/public/converted_csv/zieger_meyer_roc_measurements_3726.csv_sorted
[INFO] [2021-04-19 10:26:23] Converted: /app/public/converted_csv/zieger_meyer_roc_measurements_3726.csv (32 lines)
[STOP] [2021-04-19 10:26:23] convert_to_csv
[START] [2021-04-19 10:26:23] calculate_delta
[INFO] [2021-04-19 10:26:23] Looping over 4 formats...
[INFO] [2021-04-19 10:26:23] ...refs (/app/public/data/zieger_meyer_roc/references.tsv)
[CMD] [2021-04-19 10:26:23] echo "0a" > /app/public/diff/zieger_meyer_roc_refs_3726.diff
[CMD] [2021-04-19 10:26:23] tail -n +1 /app/public/converted_csv/zieger_meyer_roc_refs_3726.csv >> /app/public/diff/zieger_meyer_roc_refs_3726.diff
[CMD] [2021-04-19 10:26:23] echo "." >> /app/public/diff/zieger_meyer_roc_refs_3726.diff
[INFO] [2021-04-19 10:26:24] Created diff: /app/public/diff/zieger_meyer_roc_refs_3726.diff (4 lines)
[INFO] [2021-04-19 10:26:24] ...nodes (/app/public/data/zieger_meyer_roc/taxa.txt)
[CMD] [2021-04-19 10:26:24] echo "0a" > /app/public/diff/zieger_meyer_roc_nodes_3726.diff
[CMD] [2021-04-19 10:26:24] tail -n +1 /app/public/converted_csv/zieger_meyer_roc_nodes_3726.csv >> /app/public/diff/zieger_meyer_roc_nodes_3726.diff
[CMD] [2021-04-19 10:26:25] echo "." >> /app/public/diff/zieger_meyer_roc_nodes_3726.diff
[INFO] [2021-04-19 10:26:25] Created diff: /app/public/diff/zieger_meyer_roc_nodes_3726.diff (11 lines)
[INFO] [2021-04-19 10:26:25] ...occurrences (/app/public/data/zieger_meyer_roc/occurrences.txt)
[CMD] [2021-04-19 10:26:25] echo "0a" > /app/public/diff/zieger_meyer_roc_occurrences_3726.diff
[CMD] [2021-04-19 10:26:25] tail -n +1 /app/public/converted_csv/zieger_meyer_roc_occurrences_3726.csv >> /app/public/diff/zieger_meyer_roc_occurrences_3726.diff
[CMD] [2021-04-19 10:26:26] echo "." >> /app/public/diff/zieger_meyer_roc_occurrences_3726.diff
[INFO] [2021-04-19 10:26:26] Created diff: /app/public/diff/zieger_meyer_roc_occurrences_3726.diff (11 lines)
[INFO] [2021-04-19 10:26:26] ...measurements (/app/public/data/zieger_meyer_roc/measurementsorfacts.txt)
[CMD] [2021-04-19 10:26:26] echo "0a" > /app/public/diff/zieger_meyer_roc_measurements_3726.diff
[CMD] [2021-04-19 10:26:26] tail -n +1 /app/public/converted_csv/zieger_meyer_roc_measurements_3726.csv >> /app/public/diff/zieger_meyer_roc_measurements_3726.diff
[CMD] [2021-04-19 10:26:27] echo "." >> /app/public/diff/zieger_meyer_roc_measurements_3726.diff
[INFO] [2021-04-19 10:26:27] Created diff: /app/public/diff/zieger_meyer_roc_measurements_3726.diff (34 lines)
[STOP] [2021-04-19 10:26:27] calculate_delta
[START] [2021-04-19 10:26:27] parse_diff_and_store
[INFO] [2021-04-19 10:26:27] Handling diff: /app/public/diff/zieger_meyer_roc_refs_3726.diff (4 lines)
[INFO] [2021-04-19 10:26:28] Loading refs diff file into memory (4 /app/public/diff/zieger_meyer_roc_refs_3726.diff lines)...
[INFO] [2021-04-19 10:26:28] Handling diff: /app/public/diff/zieger_meyer_roc_nodes_3726.diff (11 lines)
[INFO] [2021-04-19 10:26:28] Loading nodes diff file into memory (11 /app/public/diff/zieger_meyer_roc_nodes_3726.diff lines)...
[INFO] [2021-04-19 10:26:29] Handling diff: /app/public/diff/zieger_meyer_roc_occurrences_3726.diff (11 lines)
[INFO] [2021-04-19 10:26:29] Loading occurrences diff file into memory (11 /app/public/diff/zieger_meyer_roc_occurrences_3726.diff lines)...
[INFO] [2021-04-19 10:26:30] Handling diff: /app/public/diff/zieger_meyer_roc_measurements_3726.diff (34 lines)
[INFO] [2021-04-19 10:26:30] Loading measurements diff file into memory (34 /app/public/diff/zieger_meyer_roc_measurements_3726.diff lines)...
[INFO] [2021-04-19 10:26:30] Storing 2 References
[INFO] [2021-04-19 10:26:30] Processing group of 2 in 1 groups of 1000
[INFO] [2021-04-19 10:26:30] Average Time: 0.0
[INFO] [2021-04-19 10:26:30] Total Time: 1s
[INFO] [2021-04-19 10:26:30] Storing 9 ScientificNames
[INFO] [2021-04-19 10:26:30] Processing group of 9 in 1 groups of 1000
[INFO] [2021-04-19 10:26:30] Average Time: 0.0
[INFO] [2021-04-19 10:26:30] Total Time: 1s
[INFO] [2021-04-19 10:26:30] Storing 9 Nodes
[INFO] [2021-04-19 10:26:30] Processing group of 9 in 1 groups of 1000
[INFO] [2021-04-19 10:26:30] Average Time: 0.01
[INFO] [2021-04-19 10:26:30] Total Time: 1s
[INFO] [2021-04-19 10:26:30] Storing 9 Occurrences
[INFO] [2021-04-19 10:26:30] Processing group of 9 in 1 groups of 1000
[INFO] [2021-04-19 10:26:30] Average Time: 0.0
[INFO] [2021-04-19 10:26:30] Total Time: 1s
[INFO] [2021-04-19 10:26:30] Storing 32 Traits
[INFO] [2021-04-19 10:26:30] Processing group of 32 in 1 groups of 1000
[INFO] [2021-04-19 10:26:31] Average Time: 0.01
[INFO] [2021-04-19 10:26:31] Total Time: 1s
[INFO] [2021-04-19 10:26:31] Storing 16 MetaTraits
[INFO] [2021-04-19 10:26:31] Processing group of 16 in 1 groups of 1000
[INFO] [2021-04-19 10:26:31] Average Time: 0.0
[INFO] [2021-04-19 10:26:31] Total Time: 1s
[INFO] [2021-04-19 10:26:31] Storing 2 TraitsReferences
[INFO] [2021-04-19 10:26:31] Processing group of 2 in 1 groups of 1000
[INFO] [2021-04-19 10:26:31] Average Time: 0.0
[INFO] [2021-04-19 10:26:31] Total Time: 1s
[STOP] [2021-04-19 10:26:31] parse_diff_and_store
[START] [2021-04-19 10:26:31] resolve_keys
[INFO] [2021-04-19 10:26:36] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-19 10:26:36] traits to occurrences...
[INFO] [2021-04-19 10:26:36] traits to nodes (through occurrences)...
[INFO] [2021-04-19 10:26:36] Traits to sex term...
[INFO] [2021-04-19 10:26:36] Traits to lifestage term...
[INFO] [2021-04-19 10:26:36] MetaTraits to traits...
[INFO] [2021-04-19 10:26:36] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-19 10:26:36] Assocs to occurrences...
[INFO] [2021-04-19 10:26:36] Assocs to nodes...
[INFO] [2021-04-19 10:26:36] Assoc to sex term...
[INFO] [2021-04-19 10:26:36] Assoc to lifestage term...
[INFO] [2021-04-19 10:26:36] MetaAssoc to assocs...
[STOP] [2021-04-19 10:26:36] resolve_keys
[START] [2021-04-19 10:26:36] hold_for_later_1
[STOP] [2021-04-19 10:26:36] hold_for_later_1
[START] [2021-04-19 10:26:36] hold_for_later_2
[STOP] [2021-04-19 10:26:36] hold_for_later_2
[START] [2021-04-19 10:26:36] resolve_missing_parents
[STOP] [2021-04-19 10:26:36] resolve_missing_parents
[START] [2021-04-19 10:26:36] rebuild_nodes
[START] [2021-04-19 10:26:36] Flattener#flatten
[START] [2021-04-19 10:26:36] Flattener#study_resource
[START] [2021-04-19 10:26:36] Flattener#build_ancestry
[STOP] [2021-04-19 10:26:36] Flattener#build_ancestry
[INFO] [2021-04-19 10:26:36] 9 ancestry keys
[START] [2021-04-19 10:26:36] build_node_ancestors
[INFO] [2021-04-19 10:26:36] old ancestors deleted.
[STOP] [2021-04-19 10:26:36] build_node_ancestors
[WARN] [2021-04-19 10:26:36] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2021-04-19 10:26:36] Flattener#flatten
[STOP] [2021-04-19 10:26:36] rebuild_nodes
[START] [2021-04-19 10:26:36] resolve_missing_media_owners
[STOP] [2021-04-19 10:26:36] resolve_missing_media_owners
[START] [2021-04-19 10:26:36] sanitize_media_verbatims
[STOP] [2021-04-19 10:26:36] sanitize_media_verbatims
[START] [2021-04-19 10:26:37] queue_downloads
[STOP] [2021-04-19 10:26:37] queue_downloads
[START] [2021-04-19 10:26:37] parse_names
[WARN] [2021-04-19 10:26:37] I see 9 names which still need to be parsed.
[STOP] [2021-04-19 10:26:38] parse_names
[START] [2021-04-19 10:26:38] denormalize_canonical_names_to_nodes
[STOP] [2021-04-19 10:26:38] denormalize_canonical_names_to_nodes
[START] [2021-04-19 10:26:38] match_nodes
[START] [2021-04-19 10:26:38] map_all_nodes_to_pages
[STOP] [2021-04-19 10:26:38] map_all_nodes_to_pages
[INFO] [2021-04-19 10:26:38] ZERO unmatched nodes (of 9)! Nicely done.
[START] [2021-04-19 10:26:38] update_nodes
[STOP] [2021-04-19 10:26:38] update_nodes
[STOP] [2021-04-19 10:26:38] match_nodes
[START] [2021-04-19 10:26:38] reindex_search
[STOP] [2021-04-19 10:26:38] reindex_search
[START] [2021-04-19 10:26:38] normalize_units
[STOP] [2021-04-19 10:26:38] normalize_units
[START] [2021-04-19 10:26:38] calculate_statistics
[2021-04-19 10:26:38] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2021-04-19 10:26:38] calculate_statistics
[START] [2021-04-19 10:26:38] complete_harvest_instance
[START] [2021-04-19 10:26:38] overall_tsv_creation
[INFO] [2021-04-19 10:26:38] Processing group of 9 in 1 batches of 10000
[INFO] [2021-04-19 10:27:14] 9 Traits (unfiltered)...
[INFO] [2021-04-19 10:27:49] 9 Traits (filtered)...
[INFO] [2021-04-19 10:27:49] 0 Associations (filtered)...
[INFO] [2021-04-19 10:27:49] 25 metadata added.
[INFO] [2021-04-19 10:27:49] 0 metadata added.
[INFO] [2021-04-19 10:28:16] Average Time: 73.54
[INFO] [2021-04-19 10:28:16] Total Time: 1m38s
[STOP] [2021-04-19 10:28:16] overall_tsv_creation
[INFO] [2021-04-19 10:28:16] Done. Check your files:
[INFO] [2021-04-19 10:28:16] (9 lines) /app/public/data/zieger_meyer_roc/publish_nodes.tsv
[INFO] [2021-04-19 10:28:16] (9 lines) /app/public/data/zieger_meyer_roc/publish_scientific_names.tsv
[INFO] [2021-04-19 10:28:17] (10 lines) /app/public/data/zieger_meyer_roc/publish_traits.tsv
[INFO] [2021-04-19 10:28:17] (26 lines) /app/public/data/zieger_meyer_roc/publish_metadata.tsv
[STOP] [2021-04-19 10:28:17] complete_harvest_instance
[START] [2021-04-19 10:28:17] completed
[STOP] [2021-04-19 10:28:17] completed
[STOP] [2021-04-19 10:28:17] logged process, took 116.75

Latest Process