Harvest for Elofsson 1966 Created 19 Apr 15:18

Harvesting Log

# Logfile created on 2020-08-27 15:36:34 -0400 by logger.rb/v1.4.2
[START] [2020-08-27 15:36:34] logged process
[START] [2020-08-27 15:36:34] Creating resource from OpenData
[START] [2020-08-27 15:36:34] logged process
[START] [2020-08-27 15:36:34] Parse meta.xml file and create formats with fields
[STOP] [2020-08-27 15:36:34] Parse meta.xml file and create formats with fields
[STOP] [2020-08-27 15:36:34] Creating resource from OpenData
[INFO] [2020-11-19 13:57:52] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-11-19 13:57:53] ## remove_type: ScientificName
[INFO] [2020-11-19 13:57:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 13:57:53] [13:57:53.153] Removed 0 Scientificnames
[INFO] [2020-11-19 13:57:53] ## remove_type: Vernacular
[INFO] [2020-11-19 13:57:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 13:57:53] [13:57:53.156] Removed 0 Vernaculars
[INFO] [2020-11-19 13:57:53] ## remove_type: Article
[INFO] [2020-11-19 13:57:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 13:57:53] [13:57:53.158] Removed 0 Articles
[INFO] [2020-11-19 13:57:53] ## remove_type: Medium
[INFO] [2020-11-19 13:57:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 13:57:53] [13:57:53.162] Removed 0 Media
[INFO] [2020-11-19 13:57:53] ## remove_type: Trait
[INFO] [2020-11-19 13:57:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 13:57:53] [13:57:53.165] Removed 0 Traits
[INFO] [2020-11-19 13:57:53] ## remove_type: MetaTrait
[INFO] [2020-11-19 13:57:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 13:57:53] [13:57:53.167] Removed 0 Metatraits
[INFO] [2020-11-19 13:57:53] ## remove_type: OccurrenceMetadatum
[INFO] [2020-11-19 13:57:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 13:57:53] [13:57:53.170] Removed 0 Occurrencemetadata
[INFO] [2020-11-19 13:57:53] ## remove_type: Assoc
[INFO] [2020-11-19 13:57:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 13:57:53] [13:57:53.173] Removed 0 Assocs
[INFO] [2020-11-19 13:57:53] ## remove_type: MetaAssoc
[INFO] [2020-11-19 13:57:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 13:57:53] [13:57:53.176] Removed 0 Metaassocs
[INFO] [2020-11-19 13:57:53] ## remove_type: Identifier
[INFO] [2020-11-19 13:57:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 13:57:53] [13:57:53.178] Removed 0 Identifiers
[INFO] [2020-11-19 13:57:53] ## remove_type: Reference
[INFO] [2020-11-19 13:57:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 13:57:53] [13:57:53.180] Removed 0 References
[INFO] [2020-11-19 13:57:53] ## remove_type: Node
[INFO] [2020-11-19 13:57:53] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 13:57:53] [13:57:53.215] Removed 0 Nodes
[START] [2020-11-19 13:57:53] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-11-19 13:57:53] Creating resource from OpenData
[START] [2020-11-19 13:57:53] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-11-19 13:57:53] Parse meta.xml file and create formats with fields
[STOP] [2020-11-19 13:57:53] Parse meta.xml file and create formats with fields
[STOP] [2020-11-19 13:57:53] Creating resource from OpenData
[START] [2020-11-19 13:57:53] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-11-19 13:57:53] create_harvest_instance
[STOP] [2020-11-19 13:57:55] create_harvest_instance
[START] [2020-11-19 13:57:55] fetch_files
[STOP] [2020-11-19 13:57:55] fetch_files
[START] [2020-11-19 13:57:55] validate_each_file
[STOP] [2020-11-19 13:57:55] validate_each_file
[START] [2020-11-19 13:57:55] convert_to_csv
[CMD] [2020-11-19 13:57:55] /usr/bin/sort /app/public/converted_csv/elofsson_elofss2_refs_24494.csv > /app/public/converted_csv/elofsson_elofss2_refs_24494.csv_sorted
[CMD] [2020-11-19 13:57:55] /usr/bin/sort /app/public/converted_csv/elofsson_elofss2_nodes_24495.csv > /app/public/converted_csv/elofsson_elofss2_nodes_24495.csv_sorted
[CMD] [2020-11-19 13:57:55] /usr/bin/sort /app/public/converted_csv/elofsson_elofss2_occurrences_24496.csv > /app/public/converted_csv/elofsson_elofss2_occurrences_24496.csv_sorted
[CMD] [2020-11-19 13:57:55] /usr/bin/sort /app/public/converted_csv/elofsson_elofss2_measurements_24497.csv > /app/public/converted_csv/elofsson_elofss2_measurements_24497.csv_sorted
[STOP] [2020-11-19 13:57:55] convert_to_csv
[START] [2020-11-19 13:57:55] calculate_delta
[CMD] [2020-11-19 13:57:55] echo "0a" > /app/public/diff/elofsson_elofss2_refs_24494.diff
[CMD] [2020-11-19 13:57:55] tail -n +1 /app/public/converted_csv/elofsson_elofss2_refs_24494.csv >> /app/public/diff/elofsson_elofss2_refs_24494.diff
[CMD] [2020-11-19 13:57:55] echo "." >> /app/public/diff/elofsson_elofss2_refs_24494.diff
[CMD] [2020-11-19 13:57:55] echo "0a" > /app/public/diff/elofsson_elofss2_nodes_24495.diff
[CMD] [2020-11-19 13:57:55] tail -n +1 /app/public/converted_csv/elofsson_elofss2_nodes_24495.csv >> /app/public/diff/elofsson_elofss2_nodes_24495.diff
[CMD] [2020-11-19 13:57:55] echo "." >> /app/public/diff/elofsson_elofss2_nodes_24495.diff
[CMD] [2020-11-19 13:57:55] echo "0a" > /app/public/diff/elofsson_elofss2_occurrences_24496.diff
[CMD] [2020-11-19 13:57:55] tail -n +1 /app/public/converted_csv/elofsson_elofss2_occurrences_24496.csv >> /app/public/diff/elofsson_elofss2_occurrences_24496.diff
[CMD] [2020-11-19 13:57:55] echo "." >> /app/public/diff/elofsson_elofss2_occurrences_24496.diff
[CMD] [2020-11-19 13:57:55] echo "0a" > /app/public/diff/elofsson_elofss2_measurements_24497.diff
[CMD] [2020-11-19 13:57:55] tail -n +1 /app/public/converted_csv/elofsson_elofss2_measurements_24497.csv >> /app/public/diff/elofsson_elofss2_measurements_24497.diff
[CMD] [2020-11-19 13:57:55] echo "." >> /app/public/diff/elofsson_elofss2_measurements_24497.diff
[STOP] [2020-11-19 13:57:55] calculate_delta
[START] [2020-11-19 13:57:55] parse_diff_and_store
[INFO] [2020-11-19 13:57:55] Loading refs diff file into memory (true lines)...
[INFO] [2020-11-19 13:57:55] Loading nodes diff file into memory (true lines)...
[INFO] [2020-11-19 13:57:55] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-11-19 13:57:55] Loading measurements diff file into memory (true lines)...
[INFO] [2020-11-19 13:57:56] Storing 8 ScientificNames
[INFO] [2020-11-19 13:57:56] Processing group of 8 in 1 groups of 1000
[INFO] [2020-11-19 13:57:56] Average Time: 0.0
[INFO] [2020-11-19 13:57:56] Total Time: 1s
[INFO] [2020-11-19 13:57:56] Storing 8 Nodes
[INFO] [2020-11-19 13:57:56] Processing group of 8 in 1 groups of 1000
[INFO] [2020-11-19 13:57:56] Average Time: 0.0
[INFO] [2020-11-19 13:57:56] Total Time: 1s
[INFO] [2020-11-19 13:57:56] Storing 8 Occurrences
[INFO] [2020-11-19 13:57:56] Processing group of 8 in 1 groups of 1000
[INFO] [2020-11-19 13:57:56] Average Time: 0.0
[INFO] [2020-11-19 13:57:56] Total Time: 1s
[INFO] [2020-11-19 13:57:56] Storing 29 Traits
[INFO] [2020-11-19 13:57:56] Processing group of 29 in 1 groups of 1000
[INFO] [2020-11-19 13:57:56] Average Time: 0.02
[INFO] [2020-11-19 13:57:56] Total Time: 1s
[INFO] [2020-11-19 13:57:56] Storing 15 MetaTraits
[INFO] [2020-11-19 13:57:56] Processing group of 15 in 1 groups of 1000
[INFO] [2020-11-19 13:57:56] Average Time: 0.0
[INFO] [2020-11-19 13:57:56] Total Time: 1s
[STOP] [2020-11-19 13:57:56] parse_diff_and_store
[START] [2020-11-19 13:57:56] resolve_keys
[INFO] [2020-11-19 13:58:02] Occurrences to nodes (through scientific_names)...
[INFO] [2020-11-19 13:58:02] traits to occurrences...
[INFO] [2020-11-19 13:58:02] traits to nodes (through occurrences)...
[INFO] [2020-11-19 13:58:02] Traits to sex term...
[INFO] [2020-11-19 13:58:02] Traits to lifestage term...
[INFO] [2020-11-19 13:58:02] MetaTraits to traits...
[INFO] [2020-11-19 13:58:02] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-11-19 13:58:02] Assocs to occurrences...
[INFO] [2020-11-19 13:58:02] Assocs to nodes...
[INFO] [2020-11-19 13:58:02] Assoc to sex term...
[INFO] [2020-11-19 13:58:02] Assoc to lifestage term...
[INFO] [2020-11-19 13:58:02] MetaAssoc to assocs...
[STOP] [2020-11-19 13:58:02] resolve_keys
[START] [2020-11-19 13:58:02] hold_for_later_1
[STOP] [2020-11-19 13:58:02] hold_for_later_1
[START] [2020-11-19 13:58:02] hold_for_later_2
[STOP] [2020-11-19 13:58:02] hold_for_later_2
[START] [2020-11-19 13:58:02] resolve_missing_parents
[STOP] [2020-11-19 13:58:02] resolve_missing_parents
[START] [2020-11-19 13:58:02] rebuild_nodes
[START] [2020-11-19 13:58:02] Flattener#flatten
[START] [2020-11-19 13:58:02] Flattener#study_resource
[START] [2020-11-19 13:58:02] Flattener#build_ancestry
[STOP] [2020-11-19 13:58:02] Flattener#build_ancestry
[INFO] [2020-11-19 13:58:02] 8 ancestry keys
[START] [2020-11-19 13:58:02] build_node_ancestors
[INFO] [2020-11-19 13:58:02] old ancestors deleted.
[STOP] [2020-11-19 13:58:02] build_node_ancestors
[WARN] [2020-11-19 13:58:02] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-11-19 13:58:02] Flattener#flatten
[STOP] [2020-11-19 13:58:02] rebuild_nodes
[START] [2020-11-19 13:58:02] resolve_missing_media_owners
[STOP] [2020-11-19 13:58:02] resolve_missing_media_owners
[START] [2020-11-19 13:58:02] sanitize_media_verbatims
[STOP] [2020-11-19 13:58:02] sanitize_media_verbatims
[START] [2020-11-19 13:58:02] queue_downloads
[STOP] [2020-11-19 13:58:02] queue_downloads
[START] [2020-11-19 13:58:02] parse_names
[WARN] [2020-11-19 13:58:02] I see 8 names which still need to be parsed.
[STOP] [2020-11-19 13:58:03] parse_names
[START] [2020-11-19 13:58:03] denormalize_canonical_names_to_nodes
[STOP] [2020-11-19 13:58:03] denormalize_canonical_names_to_nodes
[START] [2020-11-19 13:58:03] match_nodes
[START] [2020-11-19 13:58:03] map_all_nodes_to_pages
[STOP] [2020-11-19 13:58:03] map_all_nodes_to_pages
[INFO] [2020-11-19 13:58:03] ZERO unmatched nodes (of 8)! Nicely done.
[START] [2020-11-19 13:58:03] update_nodes
[STOP] [2020-11-19 13:58:03] update_nodes
[STOP] [2020-11-19 13:58:03] match_nodes
[START] [2020-11-19 13:58:03] reindex_search
[STOP] [2020-11-19 13:58:03] reindex_search
[START] [2020-11-19 13:58:03] normalize_units
[STOP] [2020-11-19 13:58:03] normalize_units
[START] [2020-11-19 13:58:03] calculate_statistics
[2020-11-19 13:58:03] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-11-19 13:58:03] calculate_statistics
[START] [2020-11-19 13:58:03] complete_harvest_instance
[START] [2020-11-19 13:58:03] overall_tsv_creation
[INFO] [2020-11-19 13:58:03] Processing group of 8 in 1 batches of 10000
[INFO] [2020-11-19 13:58:47] 8 Traits (unfiltered)...
[INFO] [2020-11-19 13:59:32] 8 Traits (filtered)...
[INFO] [2020-11-19 13:59:32] 0 Associations (filtered)...
[INFO] [2020-11-19 13:59:32] 36 metadata added.
[INFO] [2020-11-19 13:59:32] 0 metadata added.
[INFO] [2020-11-19 13:59:32] Average Time: 68.48
[INFO] [2020-11-19 13:59:32] Total Time: 1m29s
[STOP] [2020-11-19 13:59:32] overall_tsv_creation
[INFO] [2020-11-19 13:59:32] Done. Check your files:
[INFO] [2020-11-19 13:59:32] (8 lines) /app/public/data/elofsson_elofss2/publish_nodes.tsv
[INFO] [2020-11-19 13:59:32] (8 lines) /app/public/data/elofsson_elofss2/publish_scientific_names.tsv
[INFO] [2020-11-19 13:59:32] (9 lines) /app/public/data/elofsson_elofss2/publish_traits.tsv
[INFO] [2020-11-19 13:59:32] (37 lines) /app/public/data/elofsson_elofss2/publish_metadata.tsv
[STOP] [2020-11-19 13:59:32] complete_harvest_instance
[START] [2020-11-19 13:59:32] completed
[STOP] [2020-11-19 13:59:32] completed
[STOP] [2020-11-19 13:59:32] logged process, took 98.73
[INFO] [2021-04-19 15:14:40] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-19 15:18:01] ## remove_type: ScientificName
[INFO] [2021-04-19 15:18:01] ++ Calling delete_all on 8 instances...
[INFO] [2021-04-19 15:18:01] [15:18:01.548] Removed 8 Scientificnames
[INFO] [2021-04-19 15:18:01] ## remove_type: Vernacular
[INFO] [2021-04-19 15:18:01] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:18:01] [15:18:01.549] Removed 0 Vernaculars
[INFO] [2021-04-19 15:18:01] ## remove_type: Article
[INFO] [2021-04-19 15:18:01] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:18:01] [15:18:01.551] Removed 0 Articles
[INFO] [2021-04-19 15:18:01] ## remove_type: Medium
[INFO] [2021-04-19 15:18:01] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:18:01] [15:18:01.552] Removed 0 Media
[INFO] [2021-04-19 15:18:01] ## remove_type: Trait
[INFO] [2021-04-19 15:18:01] ++ Calling delete_all on 29 instances...
[INFO] [2021-04-19 15:18:01] [15:18:01.555] Removed 29 Traits
[INFO] [2021-04-19 15:18:01] ## remove_type: MetaTrait
[INFO] [2021-04-19 15:18:01] ++ Calling delete_all on 15 instances...
[INFO] [2021-04-19 15:18:01] [15:18:01.557] Removed 15 Metatraits
[INFO] [2021-04-19 15:18:01] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-19 15:18:01] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:18:01] [15:18:01.559] Removed 0 Occurrencemetadata
[INFO] [2021-04-19 15:18:01] ## remove_type: Assoc
[INFO] [2021-04-19 15:18:01] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:18:01] [15:18:01.560] Removed 0 Assocs
[INFO] [2021-04-19 15:18:01] ## remove_type: MetaAssoc
[INFO] [2021-04-19 15:18:01] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:18:01] [15:18:01.562] Removed 0 Metaassocs
[INFO] [2021-04-19 15:18:01] ## remove_type: Identifier
[INFO] [2021-04-19 15:18:01] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:18:01] [15:18:01.563] Removed 0 Identifiers
[INFO] [2021-04-19 15:18:01] ## remove_type: Reference
[INFO] [2021-04-19 15:18:01] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:18:01] [15:18:01.565] Removed 0 References
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:01] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] Starting batch with ID 84620892...
[INFO] [2021-04-19 15:18:02] ## remove_type: Node
[INFO] [2021-04-19 15:18:02] ++ Calling delete_all on 8 instances...
[INFO] [2021-04-19 15:18:02] [15:18:02.305] Removed 8 Nodes
[START] [2021-04-19 15:18:02] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 15:18:02] Creating resource from OpenData
[START] [2021-04-19 15:18:06] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 15:18:06] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 15:18:06] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 15:18:06] Creating resource from OpenData
[START] [2021-04-19 15:18:06] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 15:18:06] create_harvest_instance
[INFO] [2021-04-19 15:18:06] Created harvest instance #3761
[STOP] [2021-04-19 15:18:06] create_harvest_instance
[START] [2021-04-19 15:18:06] fetch_files
[STOP] [2021-04-19 15:18:06] fetch_files
[START] [2021-04-19 15:18:06] validate_each_file
[INFO] [2021-04-19 15:18:06] Looping over 4 formats...
[INFO] [2021-04-19 15:18:06] ...refs (/app/public/data/elofsson_elofss2/references.tsv)
[INFO] [2021-04-19 15:18:06] Valid: /app/public/converted_csv/elofsson_elofss2_refs_3761.csv (0 lines)
[INFO] [2021-04-19 15:18:06] ...nodes (/app/public/data/elofsson_elofss2/taxa.txt)
[INFO] [2021-04-19 15:18:06] Valid: /app/public/converted_csv/elofsson_elofss2_nodes_3761.csv (8 lines)
[INFO] [2021-04-19 15:18:06] ...occurrences (/app/public/data/elofsson_elofss2/occurrences.txt)
[INFO] [2021-04-19 15:18:06] Valid: /app/public/converted_csv/elofsson_elofss2_occurrences_3761.csv (8 lines)
[INFO] [2021-04-19 15:18:06] ...measurements (/app/public/data/elofsson_elofss2/measurementsorfacts.txt)
[INFO] [2021-04-19 15:18:06] Valid: /app/public/converted_csv/elofsson_elofss2_measurements_3761.csv (29 lines)
[STOP] [2021-04-19 15:18:06] validate_each_file
[START] [2021-04-19 15:18:06] convert_to_csv
[INFO] [2021-04-19 15:18:06] Looping over 4 formats...
[INFO] [2021-04-19 15:18:06] ...refs (/app/public/data/elofsson_elofss2/references.tsv)
[CMD] [2021-04-19 15:18:06] /usr/bin/sort /app/public/converted_csv/elofsson_elofss2_refs_3761.csv > /app/public/converted_csv/elofsson_elofss2_refs_3761.csv_sorted
[INFO] [2021-04-19 15:18:06] Converted: /app/public/converted_csv/elofsson_elofss2_refs_3761.csv (0 lines)
[INFO] [2021-04-19 15:18:06] ...nodes (/app/public/data/elofsson_elofss2/taxa.txt)
[CMD] [2021-04-19 15:18:06] /usr/bin/sort /app/public/converted_csv/elofsson_elofss2_nodes_3761.csv > /app/public/converted_csv/elofsson_elofss2_nodes_3761.csv_sorted
[INFO] [2021-04-19 15:18:06] Converted: /app/public/converted_csv/elofsson_elofss2_nodes_3761.csv (8 lines)
[INFO] [2021-04-19 15:18:06] ...occurrences (/app/public/data/elofsson_elofss2/occurrences.txt)
[CMD] [2021-04-19 15:18:06] /usr/bin/sort /app/public/converted_csv/elofsson_elofss2_occurrences_3761.csv > /app/public/converted_csv/elofsson_elofss2_occurrences_3761.csv_sorted
[INFO] [2021-04-19 15:18:06] Converted: /app/public/converted_csv/elofsson_elofss2_occurrences_3761.csv (8 lines)
[INFO] [2021-04-19 15:18:06] ...measurements (/app/public/data/elofsson_elofss2/measurementsorfacts.txt)
[CMD] [2021-04-19 15:18:06] /usr/bin/sort /app/public/converted_csv/elofsson_elofss2_measurements_3761.csv > /app/public/converted_csv/elofsson_elofss2_measurements_3761.csv_sorted
[INFO] [2021-04-19 15:18:06] Converted: /app/public/converted_csv/elofsson_elofss2_measurements_3761.csv (29 lines)
[STOP] [2021-04-19 15:18:06] convert_to_csv
[START] [2021-04-19 15:18:06] calculate_delta
[INFO] [2021-04-19 15:18:06] Looping over 4 formats...
[INFO] [2021-04-19 15:18:06] ...refs (/app/public/data/elofsson_elofss2/references.tsv)
[CMD] [2021-04-19 15:18:06] echo "0a" > /app/public/diff/elofsson_elofss2_refs_3761.diff
[CMD] [2021-04-19 15:18:06] tail -n +1 /app/public/converted_csv/elofsson_elofss2_refs_3761.csv >> /app/public/diff/elofsson_elofss2_refs_3761.diff
[CMD] [2021-04-19 15:18:06] echo "." >> /app/public/diff/elofsson_elofss2_refs_3761.diff
[INFO] [2021-04-19 15:18:06] Created diff: /app/public/diff/elofsson_elofss2_refs_3761.diff (2 lines)
[INFO] [2021-04-19 15:18:06] ...nodes (/app/public/data/elofsson_elofss2/taxa.txt)
[CMD] [2021-04-19 15:18:06] echo "0a" > /app/public/diff/elofsson_elofss2_nodes_3761.diff
[CMD] [2021-04-19 15:18:06] tail -n +1 /app/public/converted_csv/elofsson_elofss2_nodes_3761.csv >> /app/public/diff/elofsson_elofss2_nodes_3761.diff
[CMD] [2021-04-19 15:18:06] echo "." >> /app/public/diff/elofsson_elofss2_nodes_3761.diff
[INFO] [2021-04-19 15:18:06] Created diff: /app/public/diff/elofsson_elofss2_nodes_3761.diff (10 lines)
[INFO] [2021-04-19 15:18:06] ...occurrences (/app/public/data/elofsson_elofss2/occurrences.txt)
[CMD] [2021-04-19 15:18:06] echo "0a" > /app/public/diff/elofsson_elofss2_occurrences_3761.diff
[CMD] [2021-04-19 15:18:06] tail -n +1 /app/public/converted_csv/elofsson_elofss2_occurrences_3761.csv >> /app/public/diff/elofsson_elofss2_occurrences_3761.diff
[CMD] [2021-04-19 15:18:06] echo "." >> /app/public/diff/elofsson_elofss2_occurrences_3761.diff
[INFO] [2021-04-19 15:18:06] Created diff: /app/public/diff/elofsson_elofss2_occurrences_3761.diff (10 lines)
[INFO] [2021-04-19 15:18:06] ...measurements (/app/public/data/elofsson_elofss2/measurementsorfacts.txt)
[CMD] [2021-04-19 15:18:06] echo "0a" > /app/public/diff/elofsson_elofss2_measurements_3761.diff
[CMD] [2021-04-19 15:18:06] tail -n +1 /app/public/converted_csv/elofsson_elofss2_measurements_3761.csv >> /app/public/diff/elofsson_elofss2_measurements_3761.diff
[CMD] [2021-04-19 15:18:06] echo "." >> /app/public/diff/elofsson_elofss2_measurements_3761.diff
[INFO] [2021-04-19 15:18:06] Created diff: /app/public/diff/elofsson_elofss2_measurements_3761.diff (31 lines)
[STOP] [2021-04-19 15:18:06] calculate_delta
[START] [2021-04-19 15:18:06] parse_diff_and_store
[INFO] [2021-04-19 15:18:06] Handling diff: /app/public/diff/elofsson_elofss2_refs_3761.diff (2 lines)
[INFO] [2021-04-19 15:18:06] Loading refs diff file into memory (2 /app/public/diff/elofsson_elofss2_refs_3761.diff lines)...
[INFO] [2021-04-19 15:18:06] Handling diff: /app/public/diff/elofsson_elofss2_nodes_3761.diff (10 lines)
[INFO] [2021-04-19 15:18:06] Loading nodes diff file into memory (10 /app/public/diff/elofsson_elofss2_nodes_3761.diff lines)...
[INFO] [2021-04-19 15:18:06] Handling diff: /app/public/diff/elofsson_elofss2_occurrences_3761.diff (10 lines)
[INFO] [2021-04-19 15:18:06] Loading occurrences diff file into memory (10 /app/public/diff/elofsson_elofss2_occurrences_3761.diff lines)...
[INFO] [2021-04-19 15:18:07] Handling diff: /app/public/diff/elofsson_elofss2_measurements_3761.diff (31 lines)
[INFO] [2021-04-19 15:18:07] Loading measurements diff file into memory (31 /app/public/diff/elofsson_elofss2_measurements_3761.diff lines)...
[INFO] [2021-04-19 15:18:07] Storing 8 ScientificNames
[INFO] [2021-04-19 15:18:07] Processing group of 8 in 1 groups of 1000
[INFO] [2021-04-19 15:18:07] Average Time: 0.01
[INFO] [2021-04-19 15:18:07] Total Time: 1s
[INFO] [2021-04-19 15:18:07] Storing 8 Nodes
[INFO] [2021-04-19 15:18:07] Processing group of 8 in 1 groups of 1000
[INFO] [2021-04-19 15:18:07] Average Time: 0.0
[INFO] [2021-04-19 15:18:07] Total Time: 1s
[INFO] [2021-04-19 15:18:07] Storing 8 Occurrences
[INFO] [2021-04-19 15:18:07] Processing group of 8 in 1 groups of 1000
[INFO] [2021-04-19 15:18:07] Average Time: 0.0
[INFO] [2021-04-19 15:18:07] Total Time: 1s
[INFO] [2021-04-19 15:18:07] Storing 29 Traits
[INFO] [2021-04-19 15:18:07] Processing group of 29 in 1 groups of 1000
[INFO] [2021-04-19 15:18:07] Average Time: 0.01
[INFO] [2021-04-19 15:18:07] Total Time: 1s
[INFO] [2021-04-19 15:18:07] Storing 15 MetaTraits
[INFO] [2021-04-19 15:18:07] Processing group of 15 in 1 groups of 1000
[INFO] [2021-04-19 15:18:07] Average Time: 0.0
[INFO] [2021-04-19 15:18:07] Total Time: 1s
[STOP] [2021-04-19 15:18:07] parse_diff_and_store
[START] [2021-04-19 15:18:07] resolve_keys
[INFO] [2021-04-19 15:18:13] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-19 15:18:13] traits to occurrences...
[INFO] [2021-04-19 15:18:13] traits to nodes (through occurrences)...
[INFO] [2021-04-19 15:18:13] Traits to sex term...
[INFO] [2021-04-19 15:18:13] Traits to lifestage term...
[INFO] [2021-04-19 15:18:13] MetaTraits to traits...
[INFO] [2021-04-19 15:18:13] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-19 15:18:13] Assocs to occurrences...
[INFO] [2021-04-19 15:18:13] Assocs to nodes...
[INFO] [2021-04-19 15:18:13] Assoc to sex term...
[INFO] [2021-04-19 15:18:13] Assoc to lifestage term...
[INFO] [2021-04-19 15:18:13] MetaAssoc to assocs...
[STOP] [2021-04-19 15:18:13] resolve_keys
[START] [2021-04-19 15:18:13] hold_for_later_1
[STOP] [2021-04-19 15:18:13] hold_for_later_1
[START] [2021-04-19 15:18:13] hold_for_later_2
[STOP] [2021-04-19 15:18:13] hold_for_later_2
[START] [2021-04-19 15:18:13] resolve_missing_parents
[STOP] [2021-04-19 15:18:13] resolve_missing_parents
[START] [2021-04-19 15:18:13] rebuild_nodes
[START] [2021-04-19 15:18:13] Flattener#flatten
[START] [2021-04-19 15:18:13] Flattener#study_resource
[START] [2021-04-19 15:18:13] Flattener#build_ancestry
[STOP] [2021-04-19 15:18:13] Flattener#build_ancestry
[INFO] [2021-04-19 15:18:13] 8 ancestry keys
[START] [2021-04-19 15:18:13] build_node_ancestors
[INFO] [2021-04-19 15:18:13] old ancestors deleted.
[STOP] [2021-04-19 15:18:13] build_node_ancestors
[WARN] [2021-04-19 15:18:13] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2021-04-19 15:18:13] Flattener#flatten
[STOP] [2021-04-19 15:18:13] rebuild_nodes
[START] [2021-04-19 15:18:13] resolve_missing_media_owners
[STOP] [2021-04-19 15:18:13] resolve_missing_media_owners
[START] [2021-04-19 15:18:13] sanitize_media_verbatims
[STOP] [2021-04-19 15:18:13] sanitize_media_verbatims
[START] [2021-04-19 15:18:13] queue_downloads
[STOP] [2021-04-19 15:18:13] queue_downloads
[START] [2021-04-19 15:18:13] parse_names
[WARN] [2021-04-19 15:18:13] I see 8 names which still need to be parsed.
[STOP] [2021-04-19 15:18:14] parse_names
[START] [2021-04-19 15:18:14] denormalize_canonical_names_to_nodes
[STOP] [2021-04-19 15:18:14] denormalize_canonical_names_to_nodes
[START] [2021-04-19 15:18:14] match_nodes
[START] [2021-04-19 15:18:14] map_all_nodes_to_pages
[STOP] [2021-04-19 15:18:14] map_all_nodes_to_pages
[INFO] [2021-04-19 15:18:14] ZERO unmatched nodes (of 8)! Nicely done.
[START] [2021-04-19 15:18:14] update_nodes
[STOP] [2021-04-19 15:18:14] update_nodes
[STOP] [2021-04-19 15:18:14] match_nodes
[START] [2021-04-19 15:18:14] reindex_search
[STOP] [2021-04-19 15:18:14] reindex_search
[START] [2021-04-19 15:18:14] normalize_units
[STOP] [2021-04-19 15:18:14] normalize_units
[START] [2021-04-19 15:18:14] calculate_statistics
[2021-04-19 15:18:14] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2021-04-19 15:18:14] calculate_statistics
[START] [2021-04-19 15:18:14] complete_harvest_instance
[START] [2021-04-19 15:18:14] overall_tsv_creation
[INFO] [2021-04-19 15:18:14] Processing group of 8 in 1 batches of 10000
[INFO] [2021-04-19 15:18:50] 8 Traits (unfiltered)...
[INFO] [2021-04-19 15:19:23] 8 Traits (filtered)...
[INFO] [2021-04-19 15:19:23] 0 Associations (filtered)...
[INFO] [2021-04-19 15:19:23] 21 metadata added.
[INFO] [2021-04-19 15:19:23] 0 metadata added.
[INFO] [2021-04-19 15:19:49] Average Time: 71.38
[INFO] [2021-04-19 15:19:49] Total Time: 1m36s
[STOP] [2021-04-19 15:19:49] overall_tsv_creation
[INFO] [2021-04-19 15:19:49] Done. Check your files:
[INFO] [2021-04-19 15:19:49] (8 lines) /app/public/data/elofsson_elofss2/publish_nodes.tsv
[INFO] [2021-04-19 15:19:49] (8 lines) /app/public/data/elofsson_elofss2/publish_scientific_names.tsv
[INFO] [2021-04-19 15:19:49] (9 lines) /app/public/data/elofsson_elofss2/publish_traits.tsv
[INFO] [2021-04-19 15:19:49] (22 lines) /app/public/data/elofsson_elofss2/publish_metadata.tsv
[STOP] [2021-04-19 15:19:49] complete_harvest_instance
[START] [2021-04-19 15:19:49] completed
[STOP] [2021-04-19 15:19:49] completed
[STOP] [2021-04-19 15:19:49] logged process, took 102.84

Latest Process

19 Apr 15:18 EXITED currently processing 0