Stage:
completed
Fetched:
19 Apr 10:03
Validated:
19 Apr 10:03
Deltas Created
19 Apr 10:03
Units Normalized:
19 Apr 10:03
Ancestry Built:
19 Apr 10:03
Nodes Matched:
19 Apr 10:03
Names Parsed:
19 Apr 10:03
New Models Stored:
19 Apr 10:03
Indexed:
19 Apr 10:03
Completed:
19 Apr 10:05
Time to Harvest:
less than a minute
Harvesting Log
(760 lines)
# Logfile created on 2020-08-27 15:38:12 -0400 by logger.rb/v1.4.2
[START] [2020-08-27 15:38:12] logged process
[START] [2020-08-27 15:38:12] Creating resource from OpenData
[START] [2020-08-27 15:38:12] logged process
[START] [2020-08-27 15:38:12] Parse meta.xml file and create formats with fields
[STOP] [2020-08-27 15:38:12] Parse meta.xml file and create formats with fields
[STOP] [2020-08-27 15:38:12] Creating resource from OpenData
[INFO] [2020-11-19 14:04:49] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-11-19 14:04:50] ## remove_type: ScientificName
[INFO] [2020-11-19 14:04:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:04:50] [14:04:50.399] Removed 0 Scientificnames
[INFO] [2020-11-19 14:04:50] ## remove_type: Vernacular
[INFO] [2020-11-19 14:04:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:04:50] [14:04:50.402] Removed 0 Vernaculars
[INFO] [2020-11-19 14:04:50] ## remove_type: Article
[INFO] [2020-11-19 14:04:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:04:50] [14:04:50.405] Removed 0 Articles
[INFO] [2020-11-19 14:04:50] ## remove_type: Medium
[INFO] [2020-11-19 14:04:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:04:50] [14:04:50.408] Removed 0 Media
[INFO] [2020-11-19 14:04:50] ## remove_type: Trait
[INFO] [2020-11-19 14:04:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:04:50] [14:04:50.412] Removed 0 Traits
[INFO] [2020-11-19 14:04:50] ## remove_type: MetaTrait
[INFO] [2020-11-19 14:04:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:04:50] [14:04:50.414] Removed 0 Metatraits
[INFO] [2020-11-19 14:04:50] ## remove_type: OccurrenceMetadatum
[INFO] [2020-11-19 14:04:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:04:50] [14:04:50.417] Removed 0 Occurrencemetadata
[INFO] [2020-11-19 14:04:50] ## remove_type: Assoc
[INFO] [2020-11-19 14:04:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:04:50] [14:04:50.420] Removed 0 Assocs
[INFO] [2020-11-19 14:04:50] ## remove_type: MetaAssoc
[INFO] [2020-11-19 14:04:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:04:50] [14:04:50.422] Removed 0 Metaassocs
[INFO] [2020-11-19 14:04:50] ## remove_type: Identifier
[INFO] [2020-11-19 14:04:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:04:50] [14:04:50.425] Removed 0 Identifiers
[INFO] [2020-11-19 14:04:50] ## remove_type: Reference
[INFO] [2020-11-19 14:04:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:04:50] [14:04:50.427] Removed 0 References
[INFO] [2020-11-19 14:04:50] ## remove_type: Node
[INFO] [2020-11-19 14:04:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:04:50] [14:04:50.448] Removed 0 Nodes
[START] [2020-11-19 14:04:50] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979
[START] [2020-11-19 14:04:50] Creating resource from OpenData
[START] [2020-11-19 14:04:50] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979
[START] [2020-11-19 14:04:50] Parse meta.xml file and create formats with fields
[STOP] [2020-11-19 14:04:50] Parse meta.xml file and create formats with fields
[STOP] [2020-11-19 14:04:50] Creating resource from OpenData
[START] [2020-11-19 14:04:50] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979
[START] [2020-11-19 14:04:50] create_harvest_instance
[STOP] [2020-11-19 14:04:52] create_harvest_instance
[START] [2020-11-19 14:04:52] fetch_files
[STOP] [2020-11-19 14:04:52] fetch_files
[START] [2020-11-19 14:04:52] validate_each_file
[STOP] [2020-11-19 14:04:52] validate_each_file
[ERR] [2020-11-19 14:04:52] CSV::MalformedCSVError
[ERR] [2020-11-19 14:04:52] New line must be <"\n"> not <"\r"> in line 2.
[ERR] [2020-11-19 14:04:52] ../models/csv_parser.rb:25:in `line_at_a_time'
[ERR] [2020-11-19 14:04:52] ../models/csv_parser.rb:81:in `rows_as_hashes'
[ERR] [2020-11-19 14:04:52] ../models/resource_harvester.rb:176:in `validate_csv'
[ERR] [2020-11-19 14:04:52] ../models/resource_harvester.rb:143:in `block (2 levels) in validate_each_file'
[ERR] [2020-11-19 14:04:52] ../models/resource_harvester.rb:142:in `block in validate_each_file'
[ERR] [2020-11-19 14:04:52] ../models/resource_harvester.rb:682:in `block in each_format'
[ERR] [2020-11-19 14:04:52] ../models/resource_harvester.rb:667:in `each_format'
[ERR] [2020-11-19 14:04:52] ../models/resource_harvester.rb:129:in `validate_each_file'
[ERR] [2020-11-19 14:04:52] ../models/resource_harvester.rb:85:in `block (3 levels) in start'
[ERR] [2020-11-19 14:04:52] ../models/logged_process.rb:19:in `run_step'
[ERR] [2020-11-19 14:04:52] ../models/resource_harvester.rb:85:in `block (2 levels) in start'
[ERR] [2020-11-19 14:04:52] ../models/resource_harvester.rb:74:in `each_key'
[ERR] [2020-11-19 14:04:52] ../models/resource_harvester.rb:74:in `block in start'
[ERR] [2020-11-19 14:04:52] ../models/resource.rb:151:in `lock'
[ERR] [2020-11-19 14:04:52] ../models/resource_harvester.rb:72:in `start'
[ERR] [2020-11-19 14:04:52] ../models/resource.rb:232:in `harvest'
[ERR] [2020-11-19 14:04:52] ../models/resource.rb:208:in `re_download_opendata_and_harvest'
[ERR] [2020-11-19 14:04:52] bin/rails:4:in `require'
[ERR] [2020-11-19 14:04:52] bin/rails:4:in `<main>'
[STOP] [2020-11-19 14:04:52] logged process, took 1.64
[INFO] [2020-11-19 14:08:35] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-11-19 14:08:38] ## remove_type: ScientificName
[INFO] [2020-11-19 14:08:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:08:38] [14:08:38.901] Removed 0 Scientificnames
[INFO] [2020-11-19 14:08:38] ## remove_type: Vernacular
[INFO] [2020-11-19 14:08:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:08:38] [14:08:38.904] Removed 0 Vernaculars
[INFO] [2020-11-19 14:08:38] ## remove_type: Article
[INFO] [2020-11-19 14:08:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:08:38] [14:08:38.907] Removed 0 Articles
[INFO] [2020-11-19 14:08:38] ## remove_type: Medium
[INFO] [2020-11-19 14:08:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:08:38] [14:08:38.910] Removed 0 Media
[INFO] [2020-11-19 14:08:38] ## remove_type: Trait
[INFO] [2020-11-19 14:08:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:08:38] [14:08:38.913] Removed 0 Traits
[INFO] [2020-11-19 14:08:38] ## remove_type: MetaTrait
[INFO] [2020-11-19 14:08:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:08:38] [14:08:38.916] Removed 0 Metatraits
[INFO] [2020-11-19 14:08:38] ## remove_type: OccurrenceMetadatum
[INFO] [2020-11-19 14:08:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:08:38] [14:08:38.920] Removed 0 Occurrencemetadata
[INFO] [2020-11-19 14:08:38] ## remove_type: Assoc
[INFO] [2020-11-19 14:08:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:08:38] [14:08:38.923] Removed 0 Assocs
[INFO] [2020-11-19 14:08:38] ## remove_type: MetaAssoc
[INFO] [2020-11-19 14:08:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:08:38] [14:08:38.926] Removed 0 Metaassocs
[INFO] [2020-11-19 14:08:38] ## remove_type: Identifier
[INFO] [2020-11-19 14:08:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:08:38] [14:08:38.929] Removed 0 Identifiers
[INFO] [2020-11-19 14:08:38] ## remove_type: Reference
[INFO] [2020-11-19 14:08:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:08:38] [14:08:38.932] Removed 0 References
[INFO] [2020-11-19 14:08:38] ## remove_type: Node
[INFO] [2020-11-19 14:08:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:08:38] [14:08:38.951] Removed 0 Nodes
[START] [2020-11-19 14:08:39] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979
[START] [2020-11-19 14:08:39] Creating resource from OpenData
[START] [2020-11-19 14:08:39] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979
[START] [2020-11-19 14:08:39] Parse meta.xml file and create formats with fields
[STOP] [2020-11-19 14:08:39] Parse meta.xml file and create formats with fields
[STOP] [2020-11-19 14:08:39] Creating resource from OpenData
[START] [2020-11-19 14:08:39] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979
[START] [2020-11-19 14:08:39] create_harvest_instance
[STOP] [2020-11-19 14:08:41] create_harvest_instance
[START] [2020-11-19 14:08:41] fetch_files
[STOP] [2020-11-19 14:08:41] fetch_files
[START] [2020-11-19 14:08:41] validate_each_file
[STOP] [2020-11-19 14:08:41] validate_each_file
[START] [2020-11-19 14:08:41] convert_to_csv
[CMD] [2020-11-19 14:08:41] /usr/bin/sort /app/public/converted_csv/land_fernald_lan_refs_24534.csv > /app/public/converted_csv/land_fernald_lan_refs_24534.csv_sorted
[CMD] [2020-11-19 14:08:41] /usr/bin/sort /app/public/converted_csv/land_fernald_lan_nodes_24535.csv > /app/public/converted_csv/land_fernald_lan_nodes_24535.csv_sorted
[CMD] [2020-11-19 14:08:41] /usr/bin/sort /app/public/converted_csv/land_fernald_lan_occurrences_24536.csv > /app/public/converted_csv/land_fernald_lan_occurrences_24536.csv_sorted
[CMD] [2020-11-19 14:08:41] /usr/bin/sort /app/public/converted_csv/land_fernald_lan_measurements_24537.csv > /app/public/converted_csv/land_fernald_lan_measurements_24537.csv_sorted
[STOP] [2020-11-19 14:08:41] convert_to_csv
[START] [2020-11-19 14:08:41] calculate_delta
[CMD] [2020-11-19 14:08:41] echo "0a" > /app/public/diff/land_fernald_lan_refs_24534.diff
[CMD] [2020-11-19 14:08:41] tail -n +1 /app/public/converted_csv/land_fernald_lan_refs_24534.csv >> /app/public/diff/land_fernald_lan_refs_24534.diff
[CMD] [2020-11-19 14:08:41] echo "." >> /app/public/diff/land_fernald_lan_refs_24534.diff
[CMD] [2020-11-19 14:08:41] echo "0a" > /app/public/diff/land_fernald_lan_nodes_24535.diff
[CMD] [2020-11-19 14:08:41] tail -n +1 /app/public/converted_csv/land_fernald_lan_nodes_24535.csv >> /app/public/diff/land_fernald_lan_nodes_24535.diff
[CMD] [2020-11-19 14:08:41] echo "." >> /app/public/diff/land_fernald_lan_nodes_24535.diff
[CMD] [2020-11-19 14:08:41] echo "0a" > /app/public/diff/land_fernald_lan_occurrences_24536.diff
[CMD] [2020-11-19 14:08:41] tail -n +1 /app/public/converted_csv/land_fernald_lan_occurrences_24536.csv >> /app/public/diff/land_fernald_lan_occurrences_24536.diff
[CMD] [2020-11-19 14:08:41] echo "." >> /app/public/diff/land_fernald_lan_occurrences_24536.diff
[CMD] [2020-11-19 14:08:41] echo "0a" > /app/public/diff/land_fernald_lan_measurements_24537.diff
[CMD] [2020-11-19 14:08:41] tail -n +1 /app/public/converted_csv/land_fernald_lan_measurements_24537.csv >> /app/public/diff/land_fernald_lan_measurements_24537.diff
[CMD] [2020-11-19 14:08:41] echo "." >> /app/public/diff/land_fernald_lan_measurements_24537.diff
[STOP] [2020-11-19 14:08:41] calculate_delta
[START] [2020-11-19 14:08:41] parse_diff_and_store
[INFO] [2020-11-19 14:08:41] Loading refs diff file into memory (true lines)...
[INFO] [2020-11-19 14:08:41] Loading nodes diff file into memory (true lines)...
[INFO] [2020-11-19 14:08:41] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-11-19 14:08:42] Loading measurements diff file into memory (true lines)...
[INFO] [2020-11-19 14:08:42] Storing 18 ScientificNames
[INFO] [2020-11-19 14:08:42] Processing group of 18 in 1 groups of 1000
[INFO] [2020-11-19 14:08:42] Average Time: 0.01
[INFO] [2020-11-19 14:08:42] Total Time: 1s
[INFO] [2020-11-19 14:08:42] Storing 18 Nodes
[INFO] [2020-11-19 14:08:42] Processing group of 18 in 1 groups of 1000
[INFO] [2020-11-19 14:08:42] Average Time: 0.01
[INFO] [2020-11-19 14:08:42] Total Time: 1s
[INFO] [2020-11-19 14:08:42] Storing 19 Occurrences
[INFO] [2020-11-19 14:08:42] Processing group of 19 in 1 groups of 1000
[INFO] [2020-11-19 14:08:42] Average Time: 0.01
[INFO] [2020-11-19 14:08:42] Total Time: 1s
[INFO] [2020-11-19 14:08:42] Storing 3 OccurrenceMetadata
[INFO] [2020-11-19 14:08:42] Processing group of 3 in 1 groups of 1000
[INFO] [2020-11-19 14:08:42] Average Time: 0.0
[INFO] [2020-11-19 14:08:42] Total Time: 1s
[INFO] [2020-11-19 14:08:42] Storing 319 Traits
[INFO] [2020-11-19 14:08:42] Processing group of 319 in 1 groups of 1000
[INFO] [2020-11-19 14:08:42] Average Time: 0.21
[INFO] [2020-11-19 14:08:42] Total Time: 1s
[INFO] [2020-11-19 14:08:42] Storing 35 MetaTraits
[INFO] [2020-11-19 14:08:42] Processing group of 35 in 1 groups of 1000
[INFO] [2020-11-19 14:08:42] Average Time: 0.01
[INFO] [2020-11-19 14:08:42] Total Time: 1s
[STOP] [2020-11-19 14:08:42] parse_diff_and_store
[START] [2020-11-19 14:08:42] resolve_keys
[INFO] [2020-11-19 14:08:49] Occurrences to nodes (through scientific_names)...
[INFO] [2020-11-19 14:08:49] traits to occurrences...
[INFO] [2020-11-19 14:08:49] traits to nodes (through occurrences)...
[INFO] [2020-11-19 14:08:49] Traits to sex term...
[INFO] [2020-11-19 14:08:49] Traits to lifestage term...
[INFO] [2020-11-19 14:08:49] MetaTraits to traits...
[INFO] [2020-11-19 14:08:49] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-11-19 14:08:49] Assocs to occurrences...
[INFO] [2020-11-19 14:08:49] Assocs to nodes...
[INFO] [2020-11-19 14:08:49] Assoc to sex term...
[INFO] [2020-11-19 14:08:49] Assoc to lifestage term...
[INFO] [2020-11-19 14:08:49] MetaAssoc to assocs...
[STOP] [2020-11-19 14:08:49] resolve_keys
[START] [2020-11-19 14:08:49] hold_for_later_1
[STOP] [2020-11-19 14:08:49] hold_for_later_1
[START] [2020-11-19 14:08:49] hold_for_later_2
[STOP] [2020-11-19 14:08:49] hold_for_later_2
[START] [2020-11-19 14:08:49] resolve_missing_parents
[STOP] [2020-11-19 14:08:49] resolve_missing_parents
[START] [2020-11-19 14:08:49] rebuild_nodes
[START] [2020-11-19 14:08:49] Flattener#flatten
[START] [2020-11-19 14:08:49] Flattener#study_resource
[START] [2020-11-19 14:08:49] Flattener#build_ancestry
[STOP] [2020-11-19 14:08:49] Flattener#build_ancestry
[INFO] [2020-11-19 14:08:49] 18 ancestry keys
[START] [2020-11-19 14:08:49] build_node_ancestors
[INFO] [2020-11-19 14:08:49] old ancestors deleted.
[STOP] [2020-11-19 14:08:49] build_node_ancestors
[WARN] [2020-11-19 14:08:49] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-11-19 14:08:49] Flattener#flatten
[STOP] [2020-11-19 14:08:49] rebuild_nodes
[START] [2020-11-19 14:08:49] resolve_missing_media_owners
[STOP] [2020-11-19 14:08:49] resolve_missing_media_owners
[START] [2020-11-19 14:08:49] sanitize_media_verbatims
[STOP] [2020-11-19 14:08:49] sanitize_media_verbatims
[START] [2020-11-19 14:08:49] queue_downloads
[STOP] [2020-11-19 14:08:49] queue_downloads
[START] [2020-11-19 14:08:49] parse_names
[WARN] [2020-11-19 14:08:49] I see 18 names which still need to be parsed.
[STOP] [2020-11-19 14:08:50] parse_names
[START] [2020-11-19 14:08:50] denormalize_canonical_names_to_nodes
[STOP] [2020-11-19 14:08:50] denormalize_canonical_names_to_nodes
[START] [2020-11-19 14:08:50] match_nodes
[START] [2020-11-19 14:08:50] map_all_nodes_to_pages
[STOP] [2020-11-19 14:08:50] map_all_nodes_to_pages
[INFO] [2020-11-19 14:08:50] ZERO unmatched nodes (of 18)! Nicely done.
[START] [2020-11-19 14:08:50] update_nodes
[STOP] [2020-11-19 14:08:50] update_nodes
[STOP] [2020-11-19 14:08:50] match_nodes
[START] [2020-11-19 14:08:50] reindex_search
[STOP] [2020-11-19 14:08:50] reindex_search
[START] [2020-11-19 14:08:50] normalize_units
[STOP] [2020-11-19 14:08:50] normalize_units
[START] [2020-11-19 14:08:50] calculate_statistics
[2020-11-19 14:08:50] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-11-19 14:08:50] calculate_statistics
[START] [2020-11-19 14:08:50] complete_harvest_instance
[START] [2020-11-19 14:08:50] overall_tsv_creation
[INFO] [2020-11-19 14:08:50] Processing group of 18 in 1 batches of 10000
[INFO] [2020-11-19 14:09:34] 19 Traits (unfiltered)...
[INFO] [2020-11-19 14:10:19] 19 Traits (filtered)...
[INFO] [2020-11-19 14:10:19] 0 Associations (filtered)...
[INFO] [2020-11-19 14:10:19] 311 metadata added.
[INFO] [2020-11-19 14:10:19] 0 metadata added.
[INFO] [2020-11-19 14:10:19] Average Time: 68.81
[INFO] [2020-11-19 14:10:19] Total Time: 1m29s
[STOP] [2020-11-19 14:10:19] overall_tsv_creation
[INFO] [2020-11-19 14:10:19] Done. Check your files:
[INFO] [2020-11-19 14:10:19] (18 lines) /app/public/data/land_fernald_lan/publish_nodes.tsv
[INFO] [2020-11-19 14:10:19] (18 lines) /app/public/data/land_fernald_lan/publish_scientific_names.tsv
[INFO] [2020-11-19 14:10:19] (20 lines) /app/public/data/land_fernald_lan/publish_traits.tsv
[INFO] [2020-11-19 14:10:19] (309 lines) /app/public/data/land_fernald_lan/publish_metadata.tsv
[STOP] [2020-11-19 14:10:19] complete_harvest_instance
[START] [2020-11-19 14:10:19] completed
[STOP] [2020-11-19 14:10:19] completed
[STOP] [2020-11-19 14:10:19] logged process, took 99.88
[INFO] [2020-11-19 14:37:07] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-11-19 14:37:08] ## remove_type: ScientificName
[INFO] [2020-11-19 14:37:08] ++ Calling delete_all on 18 instances...
[INFO] [2020-11-19 14:37:08] [14:37:08.968] Removed 18 Scientificnames
[INFO] [2020-11-19 14:37:08] ## remove_type: Vernacular
[INFO] [2020-11-19 14:37:08] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:37:08] [14:37:08.971] Removed 0 Vernaculars
[INFO] [2020-11-19 14:37:08] ## remove_type: Article
[INFO] [2020-11-19 14:37:08] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:37:08] [14:37:08.974] Removed 0 Articles
[INFO] [2020-11-19 14:37:08] ## remove_type: Medium
[INFO] [2020-11-19 14:37:08] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:37:08] [14:37:08.978] Removed 0 Media
[INFO] [2020-11-19 14:37:08] ## remove_type: Trait
[INFO] [2020-11-19 14:37:08] ++ Calling delete_all on 319 instances...
[INFO] [2020-11-19 14:37:09] [14:37:09.003] Removed 319 Traits
[INFO] [2020-11-19 14:37:09] ## remove_type: MetaTrait
[INFO] [2020-11-19 14:37:09] ++ Calling delete_all on 35 instances...
[INFO] [2020-11-19 14:37:09] [14:37:09.007] Removed 35 Metatraits
[INFO] [2020-11-19 14:37:09] ## remove_type: OccurrenceMetadatum
[INFO] [2020-11-19 14:37:09] ++ Calling delete_all on 3 instances...
[INFO] [2020-11-19 14:37:09] [14:37:09.010] Removed 3 Occurrencemetadata
[INFO] [2020-11-19 14:37:09] ## remove_type: Assoc
[INFO] [2020-11-19 14:37:09] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:37:09] [14:37:09.013] Removed 0 Assocs
[INFO] [2020-11-19 14:37:09] ## remove_type: MetaAssoc
[INFO] [2020-11-19 14:37:09] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:37:09] [14:37:09.015] Removed 0 Metaassocs
[INFO] [2020-11-19 14:37:09] ## remove_type: Identifier
[INFO] [2020-11-19 14:37:09] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:37:09] [14:37:09.018] Removed 0 Identifiers
[INFO] [2020-11-19 14:37:09] ## remove_type: Reference
[INFO] [2020-11-19 14:37:09] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 14:37:09] [14:37:09.021] Removed 0 References
[INFO] [2020-11-19 14:37:09] Starting batch with ID 84620934...
[INFO] [2020-11-19 14:37:09] Starting batch with ID 84620934...
[INFO] [2020-11-19 14:37:09] Starting batch with ID 84620934...
[INFO] [2020-11-19 14:37:09] Starting batch with ID 84620934...
[INFO] [2020-11-19 14:37:09] Starting batch with ID 84620934...
[INFO] [2020-11-19 14:37:09] Starting batch with ID 84620934...
[INFO] [2020-11-19 14:37:09] Starting batch with ID 84620934...
[INFO] [2020-11-19 14:37:09] ## remove_type: Node
[INFO] [2020-11-19 14:37:09] ++ Calling delete_all on 18 instances...
[INFO] [2020-11-19 14:37:09] [14:37:09.503] Removed 18 Nodes
[START] [2020-11-19 14:37:09] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979
[START] [2020-11-19 14:37:09] Creating resource from OpenData
[START] [2020-11-19 14:37:10] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979
[START] [2020-11-19 14:37:10] Parse meta.xml file and create formats with fields
[STOP] [2020-11-19 14:37:10] Parse meta.xml file and create formats with fields
[STOP] [2020-11-19 14:37:10] Creating resource from OpenData
[START] [2020-11-19 14:37:10] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979
[START] [2020-11-19 14:37:10] create_harvest_instance
[STOP] [2020-11-19 14:37:11] create_harvest_instance
[START] [2020-11-19 14:37:11] fetch_files
[STOP] [2020-11-19 14:37:11] fetch_files
[START] [2020-11-19 14:37:11] validate_each_file
[STOP] [2020-11-19 14:37:11] validate_each_file
[START] [2020-11-19 14:37:11] convert_to_csv
[CMD] [2020-11-19 14:37:11] /usr/bin/sort /app/public/converted_csv/land_fernald_lan_refs_24574.csv > /app/public/converted_csv/land_fernald_lan_refs_24574.csv_sorted
[CMD] [2020-11-19 14:37:11] /usr/bin/sort /app/public/converted_csv/land_fernald_lan_nodes_24575.csv > /app/public/converted_csv/land_fernald_lan_nodes_24575.csv_sorted
[CMD] [2020-11-19 14:37:11] /usr/bin/sort /app/public/converted_csv/land_fernald_lan_occurrences_24576.csv > /app/public/converted_csv/land_fernald_lan_occurrences_24576.csv_sorted
[CMD] [2020-11-19 14:37:11] /usr/bin/sort /app/public/converted_csv/land_fernald_lan_measurements_24577.csv > /app/public/converted_csv/land_fernald_lan_measurements_24577.csv_sorted
[STOP] [2020-11-19 14:37:11] convert_to_csv
[START] [2020-11-19 14:37:11] calculate_delta
[CMD] [2020-11-19 14:37:11] echo "0a" > /app/public/diff/land_fernald_lan_refs_24574.diff
[CMD] [2020-11-19 14:37:11] tail -n +1 /app/public/converted_csv/land_fernald_lan_refs_24574.csv >> /app/public/diff/land_fernald_lan_refs_24574.diff
[CMD] [2020-11-19 14:37:11] echo "." >> /app/public/diff/land_fernald_lan_refs_24574.diff
[CMD] [2020-11-19 14:37:11] echo "0a" > /app/public/diff/land_fernald_lan_nodes_24575.diff
[CMD] [2020-11-19 14:37:11] tail -n +1 /app/public/converted_csv/land_fernald_lan_nodes_24575.csv >> /app/public/diff/land_fernald_lan_nodes_24575.diff
[CMD] [2020-11-19 14:37:11] echo "." >> /app/public/diff/land_fernald_lan_nodes_24575.diff
[CMD] [2020-11-19 14:37:11] echo "0a" > /app/public/diff/land_fernald_lan_occurrences_24576.diff
[CMD] [2020-11-19 14:37:11] tail -n +1 /app/public/converted_csv/land_fernald_lan_occurrences_24576.csv >> /app/public/diff/land_fernald_lan_occurrences_24576.diff
[CMD] [2020-11-19 14:37:11] echo "." >> /app/public/diff/land_fernald_lan_occurrences_24576.diff
[CMD] [2020-11-19 14:37:11] echo "0a" > /app/public/diff/land_fernald_lan_measurements_24577.diff
[CMD] [2020-11-19 14:37:11] tail -n +1 /app/public/converted_csv/land_fernald_lan_measurements_24577.csv >> /app/public/diff/land_fernald_lan_measurements_24577.diff
[CMD] [2020-11-19 14:37:11] echo "." >> /app/public/diff/land_fernald_lan_measurements_24577.diff
[STOP] [2020-11-19 14:37:11] calculate_delta
[START] [2020-11-19 14:37:11] parse_diff_and_store
[INFO] [2020-11-19 14:37:11] Loading refs diff file into memory (true lines)...
[INFO] [2020-11-19 14:37:11] Loading nodes diff file into memory (true lines)...
[INFO] [2020-11-19 14:37:11] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-11-19 14:37:13] Loading measurements diff file into memory (true lines)...
[INFO] [2020-11-19 14:37:13] Storing 18 ScientificNames
[INFO] [2020-11-19 14:37:13] Processing group of 18 in 1 groups of 1000
[INFO] [2020-11-19 14:37:13] Average Time: 0.01
[INFO] [2020-11-19 14:37:13] Total Time: 1s
[INFO] [2020-11-19 14:37:13] Storing 18 Nodes
[INFO] [2020-11-19 14:37:13] Processing group of 18 in 1 groups of 1000
[INFO] [2020-11-19 14:37:13] Average Time: 0.01
[INFO] [2020-11-19 14:37:13] Total Time: 1s
[INFO] [2020-11-19 14:37:13] Storing 19 Occurrences
[INFO] [2020-11-19 14:37:13] Processing group of 19 in 1 groups of 1000
[INFO] [2020-11-19 14:37:13] Average Time: 0.0
[INFO] [2020-11-19 14:37:13] Total Time: 1s
[INFO] [2020-11-19 14:37:13] Storing 3 OccurrenceMetadata
[INFO] [2020-11-19 14:37:13] Processing group of 3 in 1 groups of 1000
[INFO] [2020-11-19 14:37:13] Average Time: 0.0
[INFO] [2020-11-19 14:37:13] Total Time: 1s
[INFO] [2020-11-19 14:37:13] Storing 319 Traits
[INFO] [2020-11-19 14:37:13] Processing group of 319 in 1 groups of 1000
[INFO] [2020-11-19 14:37:13] Average Time: 0.11
[INFO] [2020-11-19 14:37:13] Total Time: 1s
[INFO] [2020-11-19 14:37:13] Storing 35 MetaTraits
[INFO] [2020-11-19 14:37:13] Processing group of 35 in 1 groups of 1000
[INFO] [2020-11-19 14:37:13] Average Time: 0.01
[INFO] [2020-11-19 14:37:13] Total Time: 1s
[STOP] [2020-11-19 14:37:13] parse_diff_and_store
[START] [2020-11-19 14:37:13] resolve_keys
[INFO] [2020-11-19 14:37:19] Occurrences to nodes (through scientific_names)...
[INFO] [2020-11-19 14:37:19] traits to occurrences...
[INFO] [2020-11-19 14:37:19] traits to nodes (through occurrences)...
[INFO] [2020-11-19 14:37:19] Traits to sex term...
[INFO] [2020-11-19 14:37:19] Traits to lifestage term...
[INFO] [2020-11-19 14:37:19] MetaTraits to traits...
[INFO] [2020-11-19 14:37:19] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-11-19 14:37:19] Assocs to occurrences...
[INFO] [2020-11-19 14:37:19] Assocs to nodes...
[INFO] [2020-11-19 14:37:19] Assoc to sex term...
[INFO] [2020-11-19 14:37:19] Assoc to lifestage term...
[INFO] [2020-11-19 14:37:19] MetaAssoc to assocs...
[STOP] [2020-11-19 14:37:19] resolve_keys
[START] [2020-11-19 14:37:19] hold_for_later_1
[STOP] [2020-11-19 14:37:19] hold_for_later_1
[START] [2020-11-19 14:37:19] hold_for_later_2
[STOP] [2020-11-19 14:37:19] hold_for_later_2
[START] [2020-11-19 14:37:19] resolve_missing_parents
[STOP] [2020-11-19 14:37:19] resolve_missing_parents
[START] [2020-11-19 14:37:19] rebuild_nodes
[START] [2020-11-19 14:37:19] Flattener#flatten
[START] [2020-11-19 14:37:19] Flattener#study_resource
[START] [2020-11-19 14:37:19] Flattener#build_ancestry
[STOP] [2020-11-19 14:37:19] Flattener#build_ancestry
[INFO] [2020-11-19 14:37:19] 18 ancestry keys
[START] [2020-11-19 14:37:19] build_node_ancestors
[INFO] [2020-11-19 14:37:19] old ancestors deleted.
[STOP] [2020-11-19 14:37:19] build_node_ancestors
[WARN] [2020-11-19 14:37:19] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-11-19 14:37:19] Flattener#flatten
[STOP] [2020-11-19 14:37:19] rebuild_nodes
[START] [2020-11-19 14:37:19] resolve_missing_media_owners
[STOP] [2020-11-19 14:37:19] resolve_missing_media_owners
[START] [2020-11-19 14:37:19] sanitize_media_verbatims
[STOP] [2020-11-19 14:37:19] sanitize_media_verbatims
[START] [2020-11-19 14:37:19] queue_downloads
[STOP] [2020-11-19 14:37:19] queue_downloads
[START] [2020-11-19 14:37:19] parse_names
[WARN] [2020-11-19 14:37:19] I see 18 names which still need to be parsed.
[STOP] [2020-11-19 14:37:20] parse_names
[START] [2020-11-19 14:37:20] denormalize_canonical_names_to_nodes
[STOP] [2020-11-19 14:37:20] denormalize_canonical_names_to_nodes
[START] [2020-11-19 14:37:20] match_nodes
[START] [2020-11-19 14:37:20] map_all_nodes_to_pages
[STOP] [2020-11-19 14:37:20] map_all_nodes_to_pages
[INFO] [2020-11-19 14:37:20] ZERO unmatched nodes (of 18)! Nicely done.
[START] [2020-11-19 14:37:20] update_nodes
[STOP] [2020-11-19 14:37:20] update_nodes
[STOP] [2020-11-19 14:37:20] match_nodes
[START] [2020-11-19 14:37:20] reindex_search
[STOP] [2020-11-19 14:37:20] reindex_search
[START] [2020-11-19 14:37:20] normalize_units
[STOP] [2020-11-19 14:37:20] normalize_units
[START] [2020-11-19 14:37:20] calculate_statistics
[2020-11-19 14:37:20] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-11-19 14:37:20] calculate_statistics
[START] [2020-11-19 14:37:20] complete_harvest_instance
[START] [2020-11-19 14:37:20] overall_tsv_creation
[INFO] [2020-11-19 14:37:20] Processing group of 18 in 1 batches of 10000
[INFO] [2020-11-19 14:38:04] 19 Traits (unfiltered)...
[INFO] [2020-11-19 14:38:48] 19 Traits (filtered)...
[INFO] [2020-11-19 14:38:48] 0 Associations (filtered)...
[INFO] [2020-11-19 14:38:48] 335 metadata added.
[INFO] [2020-11-19 14:38:48] 0 metadata added.
[INFO] [2020-11-19 14:38:48] Average Time: 68.5
[INFO] [2020-11-19 14:38:48] Total Time: 1m29s
[STOP] [2020-11-19 14:38:48] overall_tsv_creation
[INFO] [2020-11-19 14:38:48] Done. Check your files:
[INFO] [2020-11-19 14:38:48] (18 lines) /app/public/data/land_fernald_lan/publish_nodes.tsv
[INFO] [2020-11-19 14:38:48] (18 lines) /app/public/data/land_fernald_lan/publish_scientific_names.tsv
[INFO] [2020-11-19 14:38:48] (20 lines) /app/public/data/land_fernald_lan/publish_traits.tsv
[INFO] [2020-11-19 14:38:48] (333 lines) /app/public/data/land_fernald_lan/publish_metadata.tsv
[STOP] [2020-11-19 14:38:48] complete_harvest_instance
[START] [2020-11-19 14:38:48] completed
[STOP] [2020-11-19 14:38:48] completed
[STOP] [2020-11-19 14:38:48] logged process, took 98.85
[INFO] [2021-04-19 10:03:15] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-19 10:03:19] ## remove_type: ScientificName
[INFO] [2021-04-19 10:03:19] ++ Calling delete_all on 18 instances...
[INFO] [2021-04-19 10:03:19] [10:03:19.249] Removed 18 Scientificnames
[INFO] [2021-04-19 10:03:19] ## remove_type: Vernacular
[INFO] [2021-04-19 10:03:19] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:03:19] [10:03:19.251] Removed 0 Vernaculars
[INFO] [2021-04-19 10:03:19] ## remove_type: Article
[INFO] [2021-04-19 10:03:19] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:03:19] [10:03:19.253] Removed 0 Articles
[INFO] [2021-04-19 10:03:19] ## remove_type: Medium
[INFO] [2021-04-19 10:03:19] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:03:19] [10:03:19.254] Removed 0 Media
[INFO] [2021-04-19 10:03:19] ## remove_type: Trait
[INFO] [2021-04-19 10:03:19] ++ Calling delete_all on 319 instances...
[INFO] [2021-04-19 10:03:19] [10:03:19.264] Removed 319 Traits
[INFO] [2021-04-19 10:03:19] ## remove_type: MetaTrait
[INFO] [2021-04-19 10:03:19] ++ Calling delete_all on 35 instances...
[INFO] [2021-04-19 10:03:19] [10:03:19.266] Removed 35 Metatraits
[INFO] [2021-04-19 10:03:19] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-19 10:03:19] ++ Calling delete_all on 3 instances...
[INFO] [2021-04-19 10:03:19] [10:03:19.268] Removed 3 Occurrencemetadata
[INFO] [2021-04-19 10:03:19] ## remove_type: Assoc
[INFO] [2021-04-19 10:03:19] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:03:19] [10:03:19.270] Removed 0 Assocs
[INFO] [2021-04-19 10:03:19] ## remove_type: MetaAssoc
[INFO] [2021-04-19 10:03:19] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:03:19] [10:03:19.271] Removed 0 Metaassocs
[INFO] [2021-04-19 10:03:19] ## remove_type: Identifier
[INFO] [2021-04-19 10:03:19] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:03:19] [10:03:19.273] Removed 0 Identifiers
[INFO] [2021-04-19 10:03:19] ## remove_type: Reference
[INFO] [2021-04-19 10:03:19] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:03:19] [10:03:19.274] Removed 0 References
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:19] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:20] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:20] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:20] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:20] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:20] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:20] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:20] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:20] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:20] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:20] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:20] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:20] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:20] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:20] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:20] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:20] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:20] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:20] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:20] Starting batch with ID 84620959...
[INFO] [2021-04-19 10:03:20] Starting batch with ID 84620954...
[INFO] [2021-04-19 10:03:20] ## remove_type: Node
[INFO] [2021-04-19 10:03:20] ++ Calling delete_all on 18 instances...
[INFO] [2021-04-19 10:03:20] [10:03:20.175] Removed 18 Nodes
[START] [2021-04-19 10:03:20] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 10:03:20] Creating resource from OpenData
[START] [2021-04-19 10:03:21] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 10:03:21] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 10:03:26] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 10:03:26] Creating resource from OpenData
[START] [2021-04-19 10:03:26] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 10:03:26] create_harvest_instance
[INFO] [2021-04-19 10:03:26] Created harvest instance #3722
[STOP] [2021-04-19 10:03:26] create_harvest_instance
[START] [2021-04-19 10:03:26] fetch_files
[STOP] [2021-04-19 10:03:26] fetch_files
[START] [2021-04-19 10:03:26] validate_each_file
[INFO] [2021-04-19 10:03:26] Looping over 4 formats...
[INFO] [2021-04-19 10:03:26] ...refs (/app/public/data/land_fernald_lan/references.tsv)
[INFO] [2021-04-19 10:03:26] Valid: /app/public/converted_csv/land_fernald_lan_refs_3722.csv (0 lines)
[INFO] [2021-04-19 10:03:26] ...nodes (/app/public/data/land_fernald_lan/taxa.txt)
[INFO] [2021-04-19 10:03:26] Valid: /app/public/converted_csv/land_fernald_lan_nodes_3722.csv (18 lines)
[INFO] [2021-04-19 10:03:26] ...occurrences (/app/public/data/land_fernald_lan/occurrences.txt)
[INFO] [2021-04-19 10:03:26] Valid: /app/public/converted_csv/land_fernald_lan_occurrences_3722.csv (19 lines)
[INFO] [2021-04-19 10:03:26] ...measurements (/app/public/data/land_fernald_lan/measurementsorfacts.txt)
[INFO] [2021-04-19 10:03:26] Valid: /app/public/converted_csv/land_fernald_lan_measurements_3722.csv (319 lines)
[STOP] [2021-04-19 10:03:26] validate_each_file
[START] [2021-04-19 10:03:26] convert_to_csv
[INFO] [2021-04-19 10:03:26] Looping over 4 formats...
[INFO] [2021-04-19 10:03:26] ...refs (/app/public/data/land_fernald_lan/references.tsv)
[CMD] [2021-04-19 10:03:26] /usr/bin/sort /app/public/converted_csv/land_fernald_lan_refs_3722.csv > /app/public/converted_csv/land_fernald_lan_refs_3722.csv_sorted
[INFO] [2021-04-19 10:03:27] Converted: /app/public/converted_csv/land_fernald_lan_refs_3722.csv (0 lines)
[INFO] [2021-04-19 10:03:27] ...nodes (/app/public/data/land_fernald_lan/taxa.txt)
[CMD] [2021-04-19 10:03:27] /usr/bin/sort /app/public/converted_csv/land_fernald_lan_nodes_3722.csv > /app/public/converted_csv/land_fernald_lan_nodes_3722.csv_sorted
[INFO] [2021-04-19 10:03:27] Converted: /app/public/converted_csv/land_fernald_lan_nodes_3722.csv (18 lines)
[INFO] [2021-04-19 10:03:27] ...occurrences (/app/public/data/land_fernald_lan/occurrences.txt)
[CMD] [2021-04-19 10:03:27] /usr/bin/sort /app/public/converted_csv/land_fernald_lan_occurrences_3722.csv > /app/public/converted_csv/land_fernald_lan_occurrences_3722.csv_sorted
[INFO] [2021-04-19 10:03:28] Converted: /app/public/converted_csv/land_fernald_lan_occurrences_3722.csv (19 lines)
[INFO] [2021-04-19 10:03:28] ...measurements (/app/public/data/land_fernald_lan/measurementsorfacts.txt)
[CMD] [2021-04-19 10:03:28] /usr/bin/sort /app/public/converted_csv/land_fernald_lan_measurements_3722.csv > /app/public/converted_csv/land_fernald_lan_measurements_3722.csv_sorted
[INFO] [2021-04-19 10:03:28] Converted: /app/public/converted_csv/land_fernald_lan_measurements_3722.csv (319 lines)
[STOP] [2021-04-19 10:03:28] convert_to_csv
[START] [2021-04-19 10:03:28] calculate_delta
[INFO] [2021-04-19 10:03:28] Looping over 4 formats...
[INFO] [2021-04-19 10:03:28] ...refs (/app/public/data/land_fernald_lan/references.tsv)
[CMD] [2021-04-19 10:03:28] echo "0a" > /app/public/diff/land_fernald_lan_refs_3722.diff
[CMD] [2021-04-19 10:03:28] tail -n +1 /app/public/converted_csv/land_fernald_lan_refs_3722.csv >> /app/public/diff/land_fernald_lan_refs_3722.diff
[CMD] [2021-04-19 10:03:29] echo "." >> /app/public/diff/land_fernald_lan_refs_3722.diff
[INFO] [2021-04-19 10:03:29] Created diff: /app/public/diff/land_fernald_lan_refs_3722.diff (2 lines)
[INFO] [2021-04-19 10:03:29] ...nodes (/app/public/data/land_fernald_lan/taxa.txt)
[CMD] [2021-04-19 10:03:29] echo "0a" > /app/public/diff/land_fernald_lan_nodes_3722.diff
[CMD] [2021-04-19 10:03:30] tail -n +1 /app/public/converted_csv/land_fernald_lan_nodes_3722.csv >> /app/public/diff/land_fernald_lan_nodes_3722.diff
[CMD] [2021-04-19 10:03:30] echo "." >> /app/public/diff/land_fernald_lan_nodes_3722.diff
[INFO] [2021-04-19 10:03:30] Created diff: /app/public/diff/land_fernald_lan_nodes_3722.diff (20 lines)
[INFO] [2021-04-19 10:03:30] ...occurrences (/app/public/data/land_fernald_lan/occurrences.txt)
[CMD] [2021-04-19 10:03:30] echo "0a" > /app/public/diff/land_fernald_lan_occurrences_3722.diff
[CMD] [2021-04-19 10:03:31] tail -n +1 /app/public/converted_csv/land_fernald_lan_occurrences_3722.csv >> /app/public/diff/land_fernald_lan_occurrences_3722.diff
[CMD] [2021-04-19 10:03:31] echo "." >> /app/public/diff/land_fernald_lan_occurrences_3722.diff
[INFO] [2021-04-19 10:03:32] Created diff: /app/public/diff/land_fernald_lan_occurrences_3722.diff (21 lines)
[INFO] [2021-04-19 10:03:32] ...measurements (/app/public/data/land_fernald_lan/measurementsorfacts.txt)
[CMD] [2021-04-19 10:03:32] echo "0a" > /app/public/diff/land_fernald_lan_measurements_3722.diff
[CMD] [2021-04-19 10:03:32] tail -n +1 /app/public/converted_csv/land_fernald_lan_measurements_3722.csv >> /app/public/diff/land_fernald_lan_measurements_3722.diff
[CMD] [2021-04-19 10:03:32] echo "." >> /app/public/diff/land_fernald_lan_measurements_3722.diff
[INFO] [2021-04-19 10:03:33] Created diff: /app/public/diff/land_fernald_lan_measurements_3722.diff (321 lines)
[STOP] [2021-04-19 10:03:33] calculate_delta
[START] [2021-04-19 10:03:33] parse_diff_and_store
[INFO] [2021-04-19 10:03:33] Handling diff: /app/public/diff/land_fernald_lan_refs_3722.diff (2 lines)
[INFO] [2021-04-19 10:03:33] Loading refs diff file into memory (2 /app/public/diff/land_fernald_lan_refs_3722.diff lines)...
[INFO] [2021-04-19 10:03:34] Handling diff: /app/public/diff/land_fernald_lan_nodes_3722.diff (20 lines)
[INFO] [2021-04-19 10:03:34] Loading nodes diff file into memory (20 /app/public/diff/land_fernald_lan_nodes_3722.diff lines)...
[INFO] [2021-04-19 10:03:34] Handling diff: /app/public/diff/land_fernald_lan_occurrences_3722.diff (21 lines)
[INFO] [2021-04-19 10:03:35] Loading occurrences diff file into memory (21 /app/public/diff/land_fernald_lan_occurrences_3722.diff lines)...
[INFO] [2021-04-19 10:03:35] Handling diff: /app/public/diff/land_fernald_lan_measurements_3722.diff (321 lines)
[INFO] [2021-04-19 10:03:35] Loading measurements diff file into memory (321 /app/public/diff/land_fernald_lan_measurements_3722.diff lines)...
[INFO] [2021-04-19 10:03:36] Storing 18 ScientificNames
[INFO] [2021-04-19 10:03:36] Processing group of 18 in 1 groups of 1000
[INFO] [2021-04-19 10:03:36] Average Time: 0.01
[INFO] [2021-04-19 10:03:36] Total Time: 1s
[INFO] [2021-04-19 10:03:36] Storing 18 Nodes
[INFO] [2021-04-19 10:03:36] Processing group of 18 in 1 groups of 1000
[INFO] [2021-04-19 10:03:36] Average Time: 0.09
[INFO] [2021-04-19 10:03:36] Total Time: 1s
[INFO] [2021-04-19 10:03:36] Storing 19 Occurrences
[INFO] [2021-04-19 10:03:36] Processing group of 19 in 1 groups of 1000
[INFO] [2021-04-19 10:03:36] Average Time: 0.0
[INFO] [2021-04-19 10:03:36] Total Time: 1s
[INFO] [2021-04-19 10:03:36] Storing 3 OccurrenceMetadata
[INFO] [2021-04-19 10:03:36] Processing group of 3 in 1 groups of 1000
[INFO] [2021-04-19 10:03:36] Average Time: 0.0
[INFO] [2021-04-19 10:03:36] Total Time: 1s
[INFO] [2021-04-19 10:03:36] Storing 319 Traits
[INFO] [2021-04-19 10:03:36] Processing group of 319 in 1 groups of 1000
[INFO] [2021-04-19 10:03:36] Average Time: 0.08
[INFO] [2021-04-19 10:03:36] Total Time: 1s
[INFO] [2021-04-19 10:03:36] Storing 35 MetaTraits
[INFO] [2021-04-19 10:03:36] Processing group of 35 in 1 groups of 1000
[INFO] [2021-04-19 10:03:36] Average Time: 0.01
[INFO] [2021-04-19 10:03:36] Total Time: 1s
[STOP] [2021-04-19 10:03:36] parse_diff_and_store
[START] [2021-04-19 10:03:36] resolve_keys
[INFO] [2021-04-19 10:03:43] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-19 10:03:43] traits to occurrences...
[INFO] [2021-04-19 10:03:43] traits to nodes (through occurrences)...
[INFO] [2021-04-19 10:03:43] Traits to sex term...
[INFO] [2021-04-19 10:03:43] Traits to lifestage term...
[INFO] [2021-04-19 10:03:43] MetaTraits to traits...
[INFO] [2021-04-19 10:03:43] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-19 10:03:43] Assocs to occurrences...
[INFO] [2021-04-19 10:03:43] Assocs to nodes...
[INFO] [2021-04-19 10:03:43] Assoc to sex term...
[INFO] [2021-04-19 10:03:43] Assoc to lifestage term...
[INFO] [2021-04-19 10:03:43] MetaAssoc to assocs...
[STOP] [2021-04-19 10:03:43] resolve_keys
[START] [2021-04-19 10:03:43] hold_for_later_1
[STOP] [2021-04-19 10:03:43] hold_for_later_1
[START] [2021-04-19 10:03:43] hold_for_later_2
[STOP] [2021-04-19 10:03:43] hold_for_later_2
[START] [2021-04-19 10:03:43] resolve_missing_parents
[STOP] [2021-04-19 10:03:43] resolve_missing_parents
[START] [2021-04-19 10:03:43] rebuild_nodes
[START] [2021-04-19 10:03:43] Flattener#flatten
[START] [2021-04-19 10:03:43] Flattener#study_resource
[START] [2021-04-19 10:03:43] Flattener#build_ancestry
[STOP] [2021-04-19 10:03:43] Flattener#build_ancestry
[INFO] [2021-04-19 10:03:43] 18 ancestry keys
[START] [2021-04-19 10:03:43] build_node_ancestors
[INFO] [2021-04-19 10:03:43] old ancestors deleted.
[STOP] [2021-04-19 10:03:43] build_node_ancestors
[WARN] [2021-04-19 10:03:43] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2021-04-19 10:03:43] Flattener#flatten
[STOP] [2021-04-19 10:03:43] rebuild_nodes
[START] [2021-04-19 10:03:43] resolve_missing_media_owners
[STOP] [2021-04-19 10:03:43] resolve_missing_media_owners
[START] [2021-04-19 10:03:43] sanitize_media_verbatims
[STOP] [2021-04-19 10:03:43] sanitize_media_verbatims
[START] [2021-04-19 10:03:43] queue_downloads
[STOP] [2021-04-19 10:03:43] queue_downloads
[START] [2021-04-19 10:03:43] parse_names
[WARN] [2021-04-19 10:03:43] I see 18 names which still need to be parsed.
[STOP] [2021-04-19 10:03:44] parse_names
[START] [2021-04-19 10:03:44] denormalize_canonical_names_to_nodes
[STOP] [2021-04-19 10:03:44] denormalize_canonical_names_to_nodes
[START] [2021-04-19 10:03:44] match_nodes
[START] [2021-04-19 10:03:44] map_all_nodes_to_pages
[STOP] [2021-04-19 10:03:44] map_all_nodes_to_pages
[INFO] [2021-04-19 10:03:44] ZERO unmatched nodes (of 18)! Nicely done.
[START] [2021-04-19 10:03:44] update_nodes
[STOP] [2021-04-19 10:03:44] update_nodes
[STOP] [2021-04-19 10:03:44] match_nodes
[START] [2021-04-19 10:03:44] reindex_search
[STOP] [2021-04-19 10:03:44] reindex_search
[START] [2021-04-19 10:03:44] normalize_units
[STOP] [2021-04-19 10:03:44] normalize_units
[START] [2021-04-19 10:03:44] calculate_statistics
[2021-04-19 10:03:44] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2021-04-19 10:03:44] calculate_statistics
[START] [2021-04-19 10:03:44] complete_harvest_instance
[START] [2021-04-19 10:03:44] overall_tsv_creation
[INFO] [2021-04-19 10:03:44] Processing group of 18 in 1 batches of 10000
[INFO] [2021-04-19 10:04:21] 19 Traits (unfiltered)...
[INFO] [2021-04-19 10:04:54] 19 Traits (filtered)...
[INFO] [2021-04-19 10:04:54] 0 Associations (filtered)...
[INFO] [2021-04-19 10:04:54] 297 metadata added.
[INFO] [2021-04-19 10:04:54] 0 metadata added.
[INFO] [2021-04-19 10:05:21] Average Time: 72.12
[INFO] [2021-04-19 10:05:21] Total Time: 1m37s
[STOP] [2021-04-19 10:05:21] overall_tsv_creation
[INFO] [2021-04-19 10:05:21] Done. Check your files:
[INFO] [2021-04-19 10:05:21] (18 lines) /app/public/data/land_fernald_lan/publish_nodes.tsv
[INFO] [2021-04-19 10:05:22] (18 lines) /app/public/data/land_fernald_lan/publish_scientific_names.tsv
[INFO] [2021-04-19 10:05:22] (20 lines) /app/public/data/land_fernald_lan/publish_traits.tsv
[INFO] [2021-04-19 10:05:22] (298 lines) /app/public/data/land_fernald_lan/publish_metadata.tsv
[STOP] [2021-04-19 10:05:22] complete_harvest_instance
[START] [2021-04-19 10:05:22] completed
[STOP] [2021-04-19 10:05:22] completed
[STOP] [2021-04-19 10:05:22] logged process, took 116.46
Latest Process