Harvest for Bitsch 2005 Created 19 Apr 09:38

Stage: completed
Fetched: 19 Apr 09:38
Validated: 19 Apr 09:38
Deltas Created 19 Apr 09:39
Units Normalized: 19 Apr 09:39
Ancestry Built: 19 Apr 09:39
Nodes Matched: 19 Apr 09:39
Names Parsed: 19 Apr 09:39
New Models Stored: 19 Apr 09:39
Indexed: 19 Apr 09:39
Completed: 19 Apr 09:40
Time to Harvest: less than a minute

Harvesting Log

(3414 lines) (showing only the last 1000 lines, see /app/public/data/bitsch_bitsch_20/process.log for the full file)
[START] [2020-11-19 10:52:57] sanitize_media_verbatims
[STOP] [2020-11-19 10:52:57] sanitize_media_verbatims
[START] [2020-11-19 10:52:57] queue_downloads
[STOP] [2020-11-19 10:52:57] queue_downloads
[START] [2020-11-19 10:52:57] parse_names
[WARN] [2020-11-19 10:52:57] I see 31 names which still need to be parsed.
[STOP] [2020-11-19 10:52:58] parse_names
[START] [2020-11-19 10:52:58] denormalize_canonical_names_to_nodes
[STOP] [2020-11-19 10:52:58] denormalize_canonical_names_to_nodes
[START] [2020-11-19 10:52:58] match_nodes
[START] [2020-11-19 10:52:58] map_all_nodes_to_pages
[STOP] [2020-11-19 10:52:58] map_all_nodes_to_pages
[INFO] [2020-11-19 10:52:58] ZERO unmatched nodes (of 31)! Nicely done.
[START] [2020-11-19 10:52:58] update_nodes
[STOP] [2020-11-19 10:52:58] update_nodes
[STOP] [2020-11-19 10:52:58] match_nodes
[START] [2020-11-19 10:52:58] reindex_search
[STOP] [2020-11-19 10:52:58] reindex_search
[START] [2020-11-19 10:52:58] normalize_units
[STOP] [2020-11-19 10:52:58] normalize_units
[START] [2020-11-19 10:52:58] calculate_statistics
[2020-11-19 10:52:58] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-11-19 10:52:58] calculate_statistics
[START] [2020-11-19 10:52:58] complete_harvest_instance
[START] [2020-11-19 10:52:58] overall_tsv_creation
[INFO] [2020-11-19 10:52:58] Processing group of 31 in 1 batches of 10000
[INFO] [2020-11-19 10:53:42] 29 Traits (unfiltered)...
[INFO] [2020-11-19 10:54:27] 29 Traits (filtered)...
[INFO] [2020-11-19 10:54:27] 0 Associations (filtered)...
[INFO] [2020-11-19 10:54:27] 262 metadata added.
[INFO] [2020-11-19 10:54:27] 0 metadata added.
[INFO] [2020-11-19 10:54:27] Average Time: 69.11
[INFO] [2020-11-19 10:54:27] Total Time: 1m30s
[STOP] [2020-11-19 10:54:27] overall_tsv_creation
[INFO] [2020-11-19 10:54:27] Done. Check your files:
[INFO] [2020-11-19 10:54:27] (31 lines) /app/public/data/bitsch_bitsch_20/publish_nodes.tsv
[INFO] [2020-11-19 10:54:27] (31 lines) /app/public/data/bitsch_bitsch_20/publish_scientific_names.tsv
[INFO] [2020-11-19 10:54:27] (30 lines) /app/public/data/bitsch_bitsch_20/publish_traits.tsv
[INFO] [2020-11-19 10:54:27] (255 lines) /app/public/data/bitsch_bitsch_20/publish_metadata.tsv
[STOP] [2020-11-19 10:54:27] complete_harvest_instance
[START] [2020-11-19 10:54:27] completed
[STOP] [2020-11-19 10:54:27] completed
[STOP] [2020-11-19 10:54:27] logged process, took 99.83
[INFO] [2020-11-19 11:34:10] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-11-19 11:34:15] ## remove_type: ScientificName
[INFO] [2020-11-19 11:34:15] ++ Calling delete_all on 31 instances...
[INFO] [2020-11-19 11:34:15] [11:34:15.197] Removed 31 Scientificnames
[INFO] [2020-11-19 11:34:15] ## remove_type: Vernacular
[INFO] [2020-11-19 11:34:15] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 11:34:15] [11:34:15.200] Removed 0 Vernaculars
[INFO] [2020-11-19 11:34:15] ## remove_type: Article
[INFO] [2020-11-19 11:34:15] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 11:34:15] [11:34:15.204] Removed 0 Articles
[INFO] [2020-11-19 11:34:15] ## remove_type: Medium
[INFO] [2020-11-19 11:34:15] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 11:34:15] [11:34:15.208] Removed 0 Media
[INFO] [2020-11-19 11:34:15] ## remove_type: Trait
[INFO] [2020-11-19 11:34:15] ++ Calling delete_all on 268 instances...
[INFO] [2020-11-19 11:34:15] [11:34:15.226] Removed 268 Traits
[INFO] [2020-11-19 11:34:15] ## remove_type: MetaTrait
[INFO] [2020-11-19 11:34:15] ++ Calling delete_all on 38 instances...
[INFO] [2020-11-19 11:34:15] [11:34:15.229] Removed 38 Metatraits
[INFO] [2020-11-19 11:34:15] ## remove_type: OccurrenceMetadatum
[INFO] [2020-11-19 11:34:15] ++ Calling delete_all on 14 instances...
[INFO] [2020-11-19 11:34:15] [11:34:15.233] Removed 14 Occurrencemetadata
[INFO] [2020-11-19 11:34:15] ## remove_type: Assoc
[INFO] [2020-11-19 11:34:15] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 11:34:15] [11:34:15.236] Removed 0 Assocs
[INFO] [2020-11-19 11:34:15] ## remove_type: MetaAssoc
[INFO] [2020-11-19 11:34:15] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 11:34:15] [11:34:15.238] Removed 0 Metaassocs
[INFO] [2020-11-19 11:34:15] ## remove_type: Identifier
[INFO] [2020-11-19 11:34:15] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 11:34:15] [11:34:15.240] Removed 0 Identifiers
[INFO] [2020-11-19 11:34:15] ## remove_type: Reference
[INFO] [2020-11-19 11:34:15] ++ Calling delete_all on 2 instances...
[INFO] [2020-11-19 11:34:15] [11:34:15.243] Removed 2 References
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:15] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:16] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:16] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:16] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:16] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:16] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:16] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:16] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:16] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:16] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:16] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:16] Starting batch with ID 84620662...
[INFO] [2020-11-19 11:34:16] ## remove_type: Node
[INFO] [2020-11-19 11:34:16] ++ Calling delete_all on 31 instances...
[INFO] [2020-11-19 11:34:16] [11:34:16.196] Removed 31 Nodes
[START] [2020-11-19 11:34:16] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-11-19 11:34:16] Creating resource from OpenData
[START] [2020-11-19 11:34:16] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-11-19 11:34:16] Parse meta.xml file and create formats with fields
[STOP] [2020-11-19 11:34:16] Parse meta.xml file and create formats with fields
[STOP] [2020-11-19 11:34:16] Creating resource from OpenData
[START] [2020-11-19 11:34:16] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-11-19 11:34:16] create_harvest_instance
[STOP] [2020-11-19 11:34:18] create_harvest_instance
[START] [2020-11-19 11:34:18] fetch_files
[STOP] [2020-11-19 11:34:18] fetch_files
[START] [2020-11-19 11:34:18] validate_each_file
[STOP] [2020-11-19 11:34:18] validate_each_file
[START] [2020-11-19 11:34:18] convert_to_csv
[CMD] [2020-11-19 11:34:18] /usr/bin/sort /app/public/converted_csv/bitsch_bitsch_20_refs_24374.csv > /app/public/converted_csv/bitsch_bitsch_20_refs_24374.csv_sorted
[CMD] [2020-11-19 11:34:18] /usr/bin/sort /app/public/converted_csv/bitsch_bitsch_20_nodes_24375.csv > /app/public/converted_csv/bitsch_bitsch_20_nodes_24375.csv_sorted
[CMD] [2020-11-19 11:34:18] /usr/bin/sort /app/public/converted_csv/bitsch_bitsch_20_occurrences_24376.csv > /app/public/converted_csv/bitsch_bitsch_20_occurrences_24376.csv_sorted
[CMD] [2020-11-19 11:34:18] /usr/bin/sort /app/public/converted_csv/bitsch_bitsch_20_measurements_24377.csv > /app/public/converted_csv/bitsch_bitsch_20_measurements_24377.csv_sorted
[STOP] [2020-11-19 11:34:18] convert_to_csv
[START] [2020-11-19 11:34:18] calculate_delta
[CMD] [2020-11-19 11:34:18] echo "0a" > /app/public/diff/bitsch_bitsch_20_refs_24374.diff
[CMD] [2020-11-19 11:34:18] tail -n +1 /app/public/converted_csv/bitsch_bitsch_20_refs_24374.csv >> /app/public/diff/bitsch_bitsch_20_refs_24374.diff
[CMD] [2020-11-19 11:34:18] echo "." >> /app/public/diff/bitsch_bitsch_20_refs_24374.diff
[CMD] [2020-11-19 11:34:18] echo "0a" > /app/public/diff/bitsch_bitsch_20_nodes_24375.diff
[CMD] [2020-11-19 11:34:18] tail -n +1 /app/public/converted_csv/bitsch_bitsch_20_nodes_24375.csv >> /app/public/diff/bitsch_bitsch_20_nodes_24375.diff
[CMD] [2020-11-19 11:34:18] echo "." >> /app/public/diff/bitsch_bitsch_20_nodes_24375.diff
[CMD] [2020-11-19 11:34:18] echo "0a" > /app/public/diff/bitsch_bitsch_20_occurrences_24376.diff
[CMD] [2020-11-19 11:34:18] tail -n +1 /app/public/converted_csv/bitsch_bitsch_20_occurrences_24376.csv >> /app/public/diff/bitsch_bitsch_20_occurrences_24376.diff
[CMD] [2020-11-19 11:34:18] echo "." >> /app/public/diff/bitsch_bitsch_20_occurrences_24376.diff
[CMD] [2020-11-19 11:34:18] echo "0a" > /app/public/diff/bitsch_bitsch_20_measurements_24377.diff
[CMD] [2020-11-19 11:34:18] tail -n +1 /app/public/converted_csv/bitsch_bitsch_20_measurements_24377.csv >> /app/public/diff/bitsch_bitsch_20_measurements_24377.diff
[CMD] [2020-11-19 11:34:18] echo "." >> /app/public/diff/bitsch_bitsch_20_measurements_24377.diff
[STOP] [2020-11-19 11:34:18] calculate_delta
[START] [2020-11-19 11:34:18] parse_diff_and_store
[INFO] [2020-11-19 11:34:18] Loading refs diff file into memory (true lines)...
[INFO] [2020-11-19 11:34:18] Loading nodes diff file into memory (true lines)...
[INFO] [2020-11-19 11:34:18] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-11-19 11:34:19] Loading measurements diff file into memory (true lines)...
[INFO] [2020-11-19 11:34:19] Storing 2 References
[INFO] [2020-11-19 11:34:19] Processing group of 2 in 1 groups of 1000
[INFO] [2020-11-19 11:34:19] Average Time: 0.0
[INFO] [2020-11-19 11:34:19] Total Time: 1s
[INFO] [2020-11-19 11:34:19] Storing 31 ScientificNames
[INFO] [2020-11-19 11:34:19] Processing group of 31 in 1 groups of 1000
[INFO] [2020-11-19 11:34:19] Average Time: 0.01
[INFO] [2020-11-19 11:34:19] Total Time: 1s
[INFO] [2020-11-19 11:34:19] Storing 31 Nodes
[INFO] [2020-11-19 11:34:19] Processing group of 31 in 1 groups of 1000
[INFO] [2020-11-19 11:34:19] Average Time: 0.01
[INFO] [2020-11-19 11:34:19] Total Time: 1s
[INFO] [2020-11-19 11:34:19] Storing 34 Occurrences
[INFO] [2020-11-19 11:34:19] Processing group of 34 in 1 groups of 1000
[INFO] [2020-11-19 11:34:19] Average Time: 0.01
[INFO] [2020-11-19 11:34:19] Total Time: 1s
[INFO] [2020-11-19 11:34:19] Storing 14 OccurrenceMetadata
[INFO] [2020-11-19 11:34:19] Processing group of 14 in 1 groups of 1000
[INFO] [2020-11-19 11:34:19] Average Time: 0.0
[INFO] [2020-11-19 11:34:19] Total Time: 1s
[INFO] [2020-11-19 11:34:19] Storing 236 Traits
[INFO] [2020-11-19 11:34:19] Processing group of 236 in 1 groups of 1000
[INFO] [2020-11-19 11:34:20] Average Time: 0.07
[INFO] [2020-11-19 11:34:20] Total Time: 1s
[INFO] [2020-11-19 11:34:20] Storing 38 MetaTraits
[INFO] [2020-11-19 11:34:20] Processing group of 38 in 1 groups of 1000
[INFO] [2020-11-19 11:34:20] Average Time: 0.01
[INFO] [2020-11-19 11:34:20] Total Time: 1s
[INFO] [2020-11-19 11:34:20] Storing 2 TraitsReferences
[INFO] [2020-11-19 11:34:20] Processing group of 2 in 1 groups of 1000
[INFO] [2020-11-19 11:34:20] Average Time: 0.0
[INFO] [2020-11-19 11:34:20] Total Time: 1s
[STOP] [2020-11-19 11:34:20] parse_diff_and_store
[START] [2020-11-19 11:34:20] resolve_keys
[INFO] [2020-11-19 11:34:25] Occurrences to nodes (through scientific_names)...
[INFO] [2020-11-19 11:34:25] traits to occurrences...
[INFO] [2020-11-19 11:34:25] traits to nodes (through occurrences)...
[INFO] [2020-11-19 11:34:25] Traits to sex term...
[INFO] [2020-11-19 11:34:25] Traits to lifestage term...
[INFO] [2020-11-19 11:34:26] MetaTraits to traits...
[INFO] [2020-11-19 11:34:26] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-11-19 11:34:26] Assocs to occurrences...
[INFO] [2020-11-19 11:34:26] Assocs to nodes...
[INFO] [2020-11-19 11:34:26] Assoc to sex term...
[INFO] [2020-11-19 11:34:26] Assoc to lifestage term...
[INFO] [2020-11-19 11:34:26] MetaAssoc to assocs...
[STOP] [2020-11-19 11:34:26] resolve_keys
[START] [2020-11-19 11:34:26] hold_for_later_1
[STOP] [2020-11-19 11:34:26] hold_for_later_1
[START] [2020-11-19 11:34:26] hold_for_later_2
[STOP] [2020-11-19 11:34:26] hold_for_later_2
[START] [2020-11-19 11:34:26] resolve_missing_parents
[STOP] [2020-11-19 11:34:26] resolve_missing_parents
[START] [2020-11-19 11:34:26] rebuild_nodes
[START] [2020-11-19 11:34:26] Flattener#flatten
[START] [2020-11-19 11:34:26] Flattener#study_resource
[START] [2020-11-19 11:34:26] Flattener#build_ancestry
[STOP] [2020-11-19 11:34:26] Flattener#build_ancestry
[INFO] [2020-11-19 11:34:26] 31 ancestry keys
[START] [2020-11-19 11:34:26] build_node_ancestors
[INFO] [2020-11-19 11:34:26] old ancestors deleted.
[STOP] [2020-11-19 11:34:26] build_node_ancestors
[WARN] [2020-11-19 11:34:26] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-11-19 11:34:26] Flattener#flatten
[STOP] [2020-11-19 11:34:26] rebuild_nodes
[START] [2020-11-19 11:34:26] resolve_missing_media_owners
[STOP] [2020-11-19 11:34:26] resolve_missing_media_owners
[START] [2020-11-19 11:34:26] sanitize_media_verbatims
[STOP] [2020-11-19 11:34:26] sanitize_media_verbatims
[START] [2020-11-19 11:34:26] queue_downloads
[STOP] [2020-11-19 11:34:26] queue_downloads
[START] [2020-11-19 11:34:26] parse_names
[WARN] [2020-11-19 11:34:26] I see 31 names which still need to be parsed.
[STOP] [2020-11-19 11:34:27] parse_names
[START] [2020-11-19 11:34:27] denormalize_canonical_names_to_nodes
[STOP] [2020-11-19 11:34:27] denormalize_canonical_names_to_nodes
[START] [2020-11-19 11:34:27] match_nodes
[START] [2020-11-19 11:34:27] map_all_nodes_to_pages
[STOP] [2020-11-19 11:34:27] map_all_nodes_to_pages
[INFO] [2020-11-19 11:34:27] ZERO unmatched nodes (of 31)! Nicely done.
[START] [2020-11-19 11:34:27] update_nodes
[STOP] [2020-11-19 11:34:27] update_nodes
[STOP] [2020-11-19 11:34:27] match_nodes
[START] [2020-11-19 11:34:27] reindex_search
[STOP] [2020-11-19 11:34:27] reindex_search
[START] [2020-11-19 11:34:27] normalize_units
[STOP] [2020-11-19 11:34:27] normalize_units
[START] [2020-11-19 11:34:27] calculate_statistics
[2020-11-19 11:34:27] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-11-19 11:34:27] calculate_statistics
[START] [2020-11-19 11:34:27] complete_harvest_instance
[START] [2020-11-19 11:34:27] overall_tsv_creation
[INFO] [2020-11-19 11:34:27] Processing group of 31 in 1 batches of 10000
[INFO] [2020-11-19 11:35:11] 29 Traits (unfiltered)...
[INFO] [2020-11-19 11:35:55] 29 Traits (filtered)...
[INFO] [2020-11-19 11:35:55] 0 Associations (filtered)...
[INFO] [2020-11-19 11:35:55] 230 metadata added.
[INFO] [2020-11-19 11:35:55] 0 metadata added.
[INFO] [2020-11-19 11:35:55] Average Time: 68.34
[INFO] [2020-11-19 11:35:55] Total Time: 1m29s
[STOP] [2020-11-19 11:35:55] overall_tsv_creation
[INFO] [2020-11-19 11:35:55] Done. Check your files:
[INFO] [2020-11-19 11:35:55] (31 lines) /app/public/data/bitsch_bitsch_20/publish_nodes.tsv
[INFO] [2020-11-19 11:35:55] (31 lines) /app/public/data/bitsch_bitsch_20/publish_scientific_names.tsv
[INFO] [2020-11-19 11:35:55] (30 lines) /app/public/data/bitsch_bitsch_20/publish_traits.tsv
[INFO] [2020-11-19 11:35:55] (223 lines) /app/public/data/bitsch_bitsch_20/publish_metadata.tsv
[STOP] [2020-11-19 11:35:55] complete_harvest_instance
[START] [2020-11-19 11:35:55] completed
[STOP] [2020-11-19 11:35:55] completed
[STOP] [2020-11-19 11:35:55] logged process, took 98.9
[INFO] [2020-11-19 12:20:39] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-11-19 12:20:44] ## remove_type: ScientificName
[INFO] [2020-11-19 12:20:44] ++ Calling delete_all on 31 instances...
[INFO] [2020-11-19 12:20:44] [12:20:44.368] Removed 31 Scientificnames
[INFO] [2020-11-19 12:20:44] ## remove_type: Vernacular
[INFO] [2020-11-19 12:20:44] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 12:20:44] [12:20:44.371] Removed 0 Vernaculars
[INFO] [2020-11-19 12:20:44] ## remove_type: Article
[INFO] [2020-11-19 12:20:44] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 12:20:44] [12:20:44.374] Removed 0 Articles
[INFO] [2020-11-19 12:20:44] ## remove_type: Medium
[INFO] [2020-11-19 12:20:44] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 12:20:44] [12:20:44.381] Removed 0 Media
[INFO] [2020-11-19 12:20:44] ## remove_type: Trait
[INFO] [2020-11-19 12:20:44] ++ Calling delete_all on 236 instances...
[INFO] [2020-11-19 12:20:44] [12:20:44.396] Removed 236 Traits
[INFO] [2020-11-19 12:20:44] ## remove_type: MetaTrait
[INFO] [2020-11-19 12:20:44] ++ Calling delete_all on 38 instances...
[INFO] [2020-11-19 12:20:44] [12:20:44.400] Removed 38 Metatraits
[INFO] [2020-11-19 12:20:44] ## remove_type: OccurrenceMetadatum
[INFO] [2020-11-19 12:20:44] ++ Calling delete_all on 14 instances...
[INFO] [2020-11-19 12:20:44] [12:20:44.403] Removed 14 Occurrencemetadata
[INFO] [2020-11-19 12:20:44] ## remove_type: Assoc
[INFO] [2020-11-19 12:20:44] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 12:20:44] [12:20:44.405] Removed 0 Assocs
[INFO] [2020-11-19 12:20:44] ## remove_type: MetaAssoc
[INFO] [2020-11-19 12:20:44] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 12:20:44] [12:20:44.408] Removed 0 Metaassocs
[INFO] [2020-11-19 12:20:44] ## remove_type: Identifier
[INFO] [2020-11-19 12:20:44] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-19 12:20:44] [12:20:44.411] Removed 0 Identifiers
[INFO] [2020-11-19 12:20:44] ## remove_type: Reference
[INFO] [2020-11-19 12:20:44] ++ Calling delete_all on 2 instances...
[INFO] [2020-11-19 12:20:44] [12:20:44.413] Removed 2 References
[INFO] [2020-11-19 12:20:44] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:44] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:44] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:44] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:44] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:44] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:44] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:44] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:44] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:44] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:44] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:44] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:44] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:44] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:44] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:44] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:44] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] Starting batch with ID 84620695...
[INFO] [2020-11-19 12:20:45] ## remove_type: Node
[INFO] [2020-11-19 12:20:45] ++ Calling delete_all on 31 instances...
[INFO] [2020-11-19 12:20:45] [12:20:45.316] Removed 31 Nodes
[START] [2020-11-19 12:20:45] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-11-19 12:20:45] Creating resource from OpenData
[START] [2020-11-19 12:20:45] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-11-19 12:20:45] Parse meta.xml file and create formats with fields
[STOP] [2020-11-19 12:20:45] Parse meta.xml file and create formats with fields
[STOP] [2020-11-19 12:20:45] Creating resource from OpenData
[START] [2020-11-19 12:20:45] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-11-19 12:20:45] create_harvest_instance
[STOP] [2020-11-19 12:20:47] create_harvest_instance
[START] [2020-11-19 12:20:47] fetch_files
[STOP] [2020-11-19 12:20:47] fetch_files
[START] [2020-11-19 12:20:47] validate_each_file
[STOP] [2020-11-19 12:20:47] validate_each_file
[START] [2020-11-19 12:20:47] convert_to_csv
[CMD] [2020-11-19 12:20:47] /usr/bin/sort /app/public/converted_csv/bitsch_bitsch_20_refs_24382.csv > /app/public/converted_csv/bitsch_bitsch_20_refs_24382.csv_sorted
[CMD] [2020-11-19 12:20:47] /usr/bin/sort /app/public/converted_csv/bitsch_bitsch_20_nodes_24383.csv > /app/public/converted_csv/bitsch_bitsch_20_nodes_24383.csv_sorted
[CMD] [2020-11-19 12:20:47] /usr/bin/sort /app/public/converted_csv/bitsch_bitsch_20_occurrences_24384.csv > /app/public/converted_csv/bitsch_bitsch_20_occurrences_24384.csv_sorted
[CMD] [2020-11-19 12:20:47] /usr/bin/sort /app/public/converted_csv/bitsch_bitsch_20_measurements_24385.csv > /app/public/converted_csv/bitsch_bitsch_20_measurements_24385.csv_sorted
[STOP] [2020-11-19 12:20:47] convert_to_csv
[START] [2020-11-19 12:20:47] calculate_delta
[CMD] [2020-11-19 12:20:47] echo "0a" > /app/public/diff/bitsch_bitsch_20_refs_24382.diff
[CMD] [2020-11-19 12:20:47] tail -n +1 /app/public/converted_csv/bitsch_bitsch_20_refs_24382.csv >> /app/public/diff/bitsch_bitsch_20_refs_24382.diff
[CMD] [2020-11-19 12:20:47] echo "." >> /app/public/diff/bitsch_bitsch_20_refs_24382.diff
[CMD] [2020-11-19 12:20:47] echo "0a" > /app/public/diff/bitsch_bitsch_20_nodes_24383.diff
[CMD] [2020-11-19 12:20:47] tail -n +1 /app/public/converted_csv/bitsch_bitsch_20_nodes_24383.csv >> /app/public/diff/bitsch_bitsch_20_nodes_24383.diff
[CMD] [2020-11-19 12:20:47] echo "." >> /app/public/diff/bitsch_bitsch_20_nodes_24383.diff
[CMD] [2020-11-19 12:20:47] echo "0a" > /app/public/diff/bitsch_bitsch_20_occurrences_24384.diff
[CMD] [2020-11-19 12:20:47] tail -n +1 /app/public/converted_csv/bitsch_bitsch_20_occurrences_24384.csv >> /app/public/diff/bitsch_bitsch_20_occurrences_24384.diff
[CMD] [2020-11-19 12:20:47] echo "." >> /app/public/diff/bitsch_bitsch_20_occurrences_24384.diff
[CMD] [2020-11-19 12:20:47] echo "0a" > /app/public/diff/bitsch_bitsch_20_measurements_24385.diff
[CMD] [2020-11-19 12:20:47] tail -n +1 /app/public/converted_csv/bitsch_bitsch_20_measurements_24385.csv >> /app/public/diff/bitsch_bitsch_20_measurements_24385.diff
[CMD] [2020-11-19 12:20:47] echo "." >> /app/public/diff/bitsch_bitsch_20_measurements_24385.diff
[STOP] [2020-11-19 12:20:47] calculate_delta
[START] [2020-11-19 12:20:47] parse_diff_and_store
[INFO] [2020-11-19 12:20:47] Loading refs diff file into memory (true lines)...
[INFO] [2020-11-19 12:20:47] Loading nodes diff file into memory (true lines)...
[INFO] [2020-11-19 12:20:47] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-11-19 12:20:48] Loading measurements diff file into memory (true lines)...
[INFO] [2020-11-19 12:20:48] Storing 2 References
[INFO] [2020-11-19 12:20:48] Processing group of 2 in 1 groups of 1000
[INFO] [2020-11-19 12:20:48] Average Time: 0.0
[INFO] [2020-11-19 12:20:48] Total Time: 1s
[INFO] [2020-11-19 12:20:48] Storing 31 ScientificNames
[INFO] [2020-11-19 12:20:48] Processing group of 31 in 1 groups of 1000
[INFO] [2020-11-19 12:20:49] Average Time: 0.01
[INFO] [2020-11-19 12:20:49] Total Time: 1s
[INFO] [2020-11-19 12:20:49] Storing 31 Nodes
[INFO] [2020-11-19 12:20:49] Processing group of 31 in 1 groups of 1000
[INFO] [2020-11-19 12:20:49] Average Time: 0.01
[INFO] [2020-11-19 12:20:49] Total Time: 1s
[INFO] [2020-11-19 12:20:49] Storing 34 Occurrences
[INFO] [2020-11-19 12:20:49] Processing group of 34 in 1 groups of 1000
[INFO] [2020-11-19 12:20:49] Average Time: 0.01
[INFO] [2020-11-19 12:20:49] Total Time: 1s
[INFO] [2020-11-19 12:20:49] Storing 14 OccurrenceMetadata
[INFO] [2020-11-19 12:20:49] Processing group of 14 in 1 groups of 1000
[INFO] [2020-11-19 12:20:49] Average Time: 0.0
[INFO] [2020-11-19 12:20:49] Total Time: 1s
[INFO] [2020-11-19 12:20:49] Storing 280 Traits
[INFO] [2020-11-19 12:20:49] Processing group of 280 in 1 groups of 1000
[INFO] [2020-11-19 12:20:49] Average Time: 0.08
[INFO] [2020-11-19 12:20:49] Total Time: 1s
[INFO] [2020-11-19 12:20:49] Storing 38 MetaTraits
[INFO] [2020-11-19 12:20:49] Processing group of 38 in 1 groups of 1000
[INFO] [2020-11-19 12:20:49] Average Time: 0.01
[INFO] [2020-11-19 12:20:49] Total Time: 1s
[INFO] [2020-11-19 12:20:49] Storing 2 TraitsReferences
[INFO] [2020-11-19 12:20:49] Processing group of 2 in 1 groups of 1000
[INFO] [2020-11-19 12:20:49] Average Time: 0.01
[INFO] [2020-11-19 12:20:49] Total Time: 1s
[STOP] [2020-11-19 12:20:49] parse_diff_and_store
[START] [2020-11-19 12:20:49] resolve_keys
[INFO] [2020-11-19 12:20:55] Occurrences to nodes (through scientific_names)...
[INFO] [2020-11-19 12:20:55] traits to occurrences...
[INFO] [2020-11-19 12:20:55] traits to nodes (through occurrences)...
[INFO] [2020-11-19 12:20:55] Traits to sex term...
[INFO] [2020-11-19 12:20:55] Traits to lifestage term...
[INFO] [2020-11-19 12:20:55] MetaTraits to traits...
[INFO] [2020-11-19 12:20:55] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-11-19 12:20:55] Assocs to occurrences...
[INFO] [2020-11-19 12:20:55] Assocs to nodes...
[INFO] [2020-11-19 12:20:55] Assoc to sex term...
[INFO] [2020-11-19 12:20:55] Assoc to lifestage term...
[INFO] [2020-11-19 12:20:55] MetaAssoc to assocs...
[STOP] [2020-11-19 12:20:55] resolve_keys
[START] [2020-11-19 12:20:55] hold_for_later_1
[STOP] [2020-11-19 12:20:55] hold_for_later_1
[START] [2020-11-19 12:20:55] hold_for_later_2
[STOP] [2020-11-19 12:20:55] hold_for_later_2
[START] [2020-11-19 12:20:55] resolve_missing_parents
[STOP] [2020-11-19 12:20:55] resolve_missing_parents
[START] [2020-11-19 12:20:55] rebuild_nodes
[START] [2020-11-19 12:20:55] Flattener#flatten
[START] [2020-11-19 12:20:55] Flattener#study_resource
[START] [2020-11-19 12:20:55] Flattener#build_ancestry
[STOP] [2020-11-19 12:20:55] Flattener#build_ancestry
[INFO] [2020-11-19 12:20:55] 31 ancestry keys
[START] [2020-11-19 12:20:55] build_node_ancestors
[INFO] [2020-11-19 12:20:55] old ancestors deleted.
[STOP] [2020-11-19 12:20:55] build_node_ancestors
[WARN] [2020-11-19 12:20:55] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-11-19 12:20:55] Flattener#flatten
[STOP] [2020-11-19 12:20:55] rebuild_nodes
[START] [2020-11-19 12:20:55] resolve_missing_media_owners
[STOP] [2020-11-19 12:20:55] resolve_missing_media_owners
[START] [2020-11-19 12:20:55] sanitize_media_verbatims
[STOP] [2020-11-19 12:20:55] sanitize_media_verbatims
[START] [2020-11-19 12:20:55] queue_downloads
[STOP] [2020-11-19 12:20:55] queue_downloads
[START] [2020-11-19 12:20:55] parse_names
[WARN] [2020-11-19 12:20:55] I see 31 names which still need to be parsed.
[STOP] [2020-11-19 12:20:56] parse_names
[START] [2020-11-19 12:20:56] denormalize_canonical_names_to_nodes
[STOP] [2020-11-19 12:20:56] denormalize_canonical_names_to_nodes
[START] [2020-11-19 12:20:56] match_nodes
[START] [2020-11-19 12:20:56] map_all_nodes_to_pages
[STOP] [2020-11-19 12:20:56] map_all_nodes_to_pages
[INFO] [2020-11-19 12:20:56] ZERO unmatched nodes (of 31)! Nicely done.
[START] [2020-11-19 12:20:56] update_nodes
[STOP] [2020-11-19 12:20:56] update_nodes
[STOP] [2020-11-19 12:20:56] match_nodes
[START] [2020-11-19 12:20:56] reindex_search
[STOP] [2020-11-19 12:20:56] reindex_search
[START] [2020-11-19 12:20:56] normalize_units
[STOP] [2020-11-19 12:20:56] normalize_units
[START] [2020-11-19 12:20:56] calculate_statistics
[2020-11-19 12:20:56] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-11-19 12:20:56] calculate_statistics
[START] [2020-11-19 12:20:56] complete_harvest_instance
[START] [2020-11-19 12:20:56] overall_tsv_creation
[INFO] [2020-11-19 12:20:56] Processing group of 31 in 1 batches of 10000
[INFO] [2020-11-19 12:21:40] 29 Traits (unfiltered)...
[INFO] [2020-11-19 12:22:24] 29 Traits (filtered)...
[INFO] [2020-11-19 12:22:24] 0 Associations (filtered)...
[INFO] [2020-11-19 12:22:24] 274 metadata added.
[INFO] [2020-11-19 12:22:24] 0 metadata added.
[INFO] [2020-11-19 12:22:24] Average Time: 68.25
[INFO] [2020-11-19 12:22:24] Total Time: 1m29s
[STOP] [2020-11-19 12:22:24] overall_tsv_creation
[INFO] [2020-11-19 12:22:24] Done. Check your files:
[INFO] [2020-11-19 12:22:24] (31 lines) /app/public/data/bitsch_bitsch_20/publish_nodes.tsv
[INFO] [2020-11-19 12:22:24] (31 lines) /app/public/data/bitsch_bitsch_20/publish_scientific_names.tsv
[INFO] [2020-11-19 12:22:24] (30 lines) /app/public/data/bitsch_bitsch_20/publish_traits.tsv
[INFO] [2020-11-19 12:22:24] (267 lines) /app/public/data/bitsch_bitsch_20/publish_metadata.tsv
[STOP] [2020-11-19 12:22:24] complete_harvest_instance
[START] [2020-11-19 12:22:24] completed
[STOP] [2020-11-19 12:22:24] completed
[STOP] [2020-11-19 12:22:24] logged process, took 98.95
[INFO] [2020-11-30 14:15:50] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-11-30 14:15:51] ## remove_type: ScientificName
[INFO] [2020-11-30 14:15:51] ++ Calling delete_all on 31 instances...
[INFO] [2020-11-30 14:15:51] [14:15:51.518] Removed 31 Scientificnames
[INFO] [2020-11-30 14:15:51] ## remove_type: Vernacular
[INFO] [2020-11-30 14:15:51] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-30 14:15:51] [14:15:51.521] Removed 0 Vernaculars
[INFO] [2020-11-30 14:15:51] ## remove_type: Article
[INFO] [2020-11-30 14:15:51] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-30 14:15:51] [14:15:51.523] Removed 0 Articles
[INFO] [2020-11-30 14:15:51] ## remove_type: Medium
[INFO] [2020-11-30 14:15:51] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-30 14:15:51] [14:15:51.527] Removed 0 Media
[INFO] [2020-11-30 14:15:51] ## remove_type: Trait
[INFO] [2020-11-30 14:15:51] ++ Calling delete_all on 280 instances...
[INFO] [2020-11-30 14:15:51] [14:15:51.640] Removed 280 Traits
[INFO] [2020-11-30 14:15:51] ## remove_type: MetaTrait
[INFO] [2020-11-30 14:15:51] ++ Calling delete_all on 38 instances...
[INFO] [2020-11-30 14:15:51] [14:15:51.659] Removed 38 Metatraits
[INFO] [2020-11-30 14:15:51] ## remove_type: OccurrenceMetadatum
[INFO] [2020-11-30 14:15:51] ++ Calling delete_all on 14 instances...
[INFO] [2020-11-30 14:15:51] [14:15:51.663] Removed 14 Occurrencemetadata
[INFO] [2020-11-30 14:15:51] ## remove_type: Assoc
[INFO] [2020-11-30 14:15:51] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-30 14:15:51] [14:15:51.665] Removed 0 Assocs
[INFO] [2020-11-30 14:15:51] ## remove_type: MetaAssoc
[INFO] [2020-11-30 14:15:51] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-30 14:15:51] [14:15:51.668] Removed 0 Metaassocs
[INFO] [2020-11-30 14:15:51] ## remove_type: Identifier
[INFO] [2020-11-30 14:15:51] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-30 14:15:51] [14:15:51.687] Removed 0 Identifiers
[INFO] [2020-11-30 14:15:51] ## remove_type: Reference
[INFO] [2020-11-30 14:15:51] ++ Calling delete_all on 2 instances...
[INFO] [2020-11-30 14:15:51] [14:15:51.708] Removed 2 References
[INFO] [2020-11-30 14:15:52] Starting batch with ID 84620725...
[INFO] [2020-11-30 14:15:52] Starting batch with ID 84620725...
[INFO] [2020-11-30 14:15:52] Starting batch with ID 84620725...
[INFO] [2020-11-30 14:15:52] Starting batch with ID 84620725...
[INFO] [2020-11-30 14:15:52] Starting batch with ID 84620725...
[INFO] [2020-11-30 14:15:52] Starting batch with ID 84620725...
[INFO] [2020-11-30 14:15:52] Starting batch with ID 84620725...
[INFO] [2020-11-30 14:15:52] Starting batch with ID 84620725...
[INFO] [2020-11-30 14:15:52] Starting batch with ID 84620725...
[INFO] [2020-11-30 14:15:52] Starting batch with ID 84620725...
[INFO] [2020-11-30 14:15:52] Starting batch with ID 84620725...
[INFO] [2020-11-30 14:15:52] Starting batch with ID 84620725...
[INFO] [2020-11-30 14:15:52] Starting batch with ID 84620725...
[INFO] [2020-11-30 14:15:52] Starting batch with ID 84620738...
[INFO] [2020-11-30 14:15:53] Starting batch with ID 84620726...
[INFO] [2020-11-30 14:15:53] Starting batch with ID 84620726...
[INFO] [2020-11-30 14:15:53] Starting batch with ID 84620726...
[INFO] [2020-11-30 14:15:53] Starting batch with ID 84620726...
[INFO] [2020-11-30 14:15:53] Starting batch with ID 84620726...
[INFO] [2020-11-30 14:15:53] Starting batch with ID 84620726...
[INFO] [2020-11-30 14:15:54] Starting batch with ID 84620728...
[INFO] [2020-11-30 14:15:54] ## remove_type: Node
[INFO] [2020-11-30 14:15:54] ++ Calling delete_all on 31 instances...
[INFO] [2020-11-30 14:15:54] [14:15:54.344] Removed 31 Nodes
[START] [2020-11-30 14:15:54] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-11-30 14:15:54] Creating resource from OpenData
[START] [2020-11-30 14:15:55] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-11-30 14:15:55] Parse meta.xml file and create formats with fields
[STOP] [2020-11-30 14:15:55] Parse meta.xml file and create formats with fields
[STOP] [2020-11-30 14:15:55] Creating resource from OpenData
[START] [2020-11-30 14:15:55] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-11-30 14:15:55] create_harvest_instance
[STOP] [2020-11-30 14:15:56] create_harvest_instance
[START] [2020-11-30 14:15:56] fetch_files
[STOP] [2020-11-30 14:15:56] fetch_files
[START] [2020-11-30 14:15:56] validate_each_file
[STOP] [2020-11-30 14:15:56] validate_each_file
[START] [2020-11-30 14:15:56] convert_to_csv
[CMD] [2020-11-30 14:15:56] /usr/bin/sort /app/public/converted_csv/bitsch_bitsch_20_refs_24650.csv > /app/public/converted_csv/bitsch_bitsch_20_refs_24650.csv_sorted
[CMD] [2020-11-30 14:15:56] /usr/bin/sort /app/public/converted_csv/bitsch_bitsch_20_nodes_24651.csv > /app/public/converted_csv/bitsch_bitsch_20_nodes_24651.csv_sorted
[CMD] [2020-11-30 14:15:56] /usr/bin/sort /app/public/converted_csv/bitsch_bitsch_20_occurrences_24652.csv > /app/public/converted_csv/bitsch_bitsch_20_occurrences_24652.csv_sorted
[CMD] [2020-11-30 14:15:56] /usr/bin/sort /app/public/converted_csv/bitsch_bitsch_20_measurements_24653.csv > /app/public/converted_csv/bitsch_bitsch_20_measurements_24653.csv_sorted
[STOP] [2020-11-30 14:15:56] convert_to_csv
[START] [2020-11-30 14:15:56] calculate_delta
[CMD] [2020-11-30 14:15:56] echo "0a" > /app/public/diff/bitsch_bitsch_20_refs_24650.diff
[CMD] [2020-11-30 14:15:56] tail -n +1 /app/public/converted_csv/bitsch_bitsch_20_refs_24650.csv >> /app/public/diff/bitsch_bitsch_20_refs_24650.diff
[CMD] [2020-11-30 14:15:56] echo "." >> /app/public/diff/bitsch_bitsch_20_refs_24650.diff
[CMD] [2020-11-30 14:15:56] echo "0a" > /app/public/diff/bitsch_bitsch_20_nodes_24651.diff
[CMD] [2020-11-30 14:15:56] tail -n +1 /app/public/converted_csv/bitsch_bitsch_20_nodes_24651.csv >> /app/public/diff/bitsch_bitsch_20_nodes_24651.diff
[CMD] [2020-11-30 14:15:56] echo "." >> /app/public/diff/bitsch_bitsch_20_nodes_24651.diff
[CMD] [2020-11-30 14:15:56] echo "0a" > /app/public/diff/bitsch_bitsch_20_occurrences_24652.diff
[CMD] [2020-11-30 14:15:56] tail -n +1 /app/public/converted_csv/bitsch_bitsch_20_occurrences_24652.csv >> /app/public/diff/bitsch_bitsch_20_occurrences_24652.diff
[CMD] [2020-11-30 14:15:56] echo "." >> /app/public/diff/bitsch_bitsch_20_occurrences_24652.diff
[CMD] [2020-11-30 14:15:56] echo "0a" > /app/public/diff/bitsch_bitsch_20_measurements_24653.diff
[CMD] [2020-11-30 14:15:56] tail -n +1 /app/public/converted_csv/bitsch_bitsch_20_measurements_24653.csv >> /app/public/diff/bitsch_bitsch_20_measurements_24653.diff
[CMD] [2020-11-30 14:15:56] echo "." >> /app/public/diff/bitsch_bitsch_20_measurements_24653.diff
[STOP] [2020-11-30 14:15:56] calculate_delta
[START] [2020-11-30 14:15:56] parse_diff_and_store
[INFO] [2020-11-30 14:15:56] Loading refs diff file into memory (true lines)...
[INFO] [2020-11-30 14:15:56] Loading nodes diff file into memory (true lines)...
[INFO] [2020-11-30 14:15:57] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-11-30 14:15:57] Loading measurements diff file into memory (true lines)...
[INFO] [2020-11-30 14:15:58] Storing 2 References
[INFO] [2020-11-30 14:15:58] Processing group of 2 in 1 groups of 1000
[INFO] [2020-11-30 14:15:58] Average Time: 0.01
[INFO] [2020-11-30 14:15:58] Total Time: 1s
[INFO] [2020-11-30 14:15:58] Storing 31 ScientificNames
[INFO] [2020-11-30 14:15:58] Processing group of 31 in 1 groups of 1000
[INFO] [2020-11-30 14:15:58] Average Time: 0.02
[INFO] [2020-11-30 14:15:58] Total Time: 1s
[INFO] [2020-11-30 14:15:58] Storing 31 Nodes
[INFO] [2020-11-30 14:15:58] Processing group of 31 in 1 groups of 1000
[INFO] [2020-11-30 14:15:58] Average Time: 0.01
[INFO] [2020-11-30 14:15:58] Total Time: 1s
[INFO] [2020-11-30 14:15:58] Storing 34 Occurrences
[INFO] [2020-11-30 14:15:58] Processing group of 34 in 1 groups of 1000
[INFO] [2020-11-30 14:15:58] Average Time: 0.01
[INFO] [2020-11-30 14:15:58] Total Time: 1s
[INFO] [2020-11-30 14:15:58] Storing 14 OccurrenceMetadata
[INFO] [2020-11-30 14:15:58] Processing group of 14 in 1 groups of 1000
[INFO] [2020-11-30 14:15:58] Average Time: 0.01
[INFO] [2020-11-30 14:15:58] Total Time: 1s
[INFO] [2020-11-30 14:15:58] Storing 236 Traits
[INFO] [2020-11-30 14:15:58] Processing group of 236 in 1 groups of 1000
[INFO] [2020-11-30 14:15:58] Average Time: 0.15
[INFO] [2020-11-30 14:15:58] Total Time: 1s
[INFO] [2020-11-30 14:15:58] Storing 38 MetaTraits
[INFO] [2020-11-30 14:15:58] Processing group of 38 in 1 groups of 1000
[INFO] [2020-11-30 14:15:58] Average Time: 0.01
[INFO] [2020-11-30 14:15:58] Total Time: 1s
[INFO] [2020-11-30 14:15:58] Storing 2 TraitsReferences
[INFO] [2020-11-30 14:15:58] Processing group of 2 in 1 groups of 1000
[INFO] [2020-11-30 14:15:58] Average Time: 0.07
[INFO] [2020-11-30 14:15:58] Total Time: 1s
[STOP] [2020-11-30 14:15:58] parse_diff_and_store
[START] [2020-11-30 14:15:58] resolve_keys
[INFO] [2020-11-30 14:16:06] Occurrences to nodes (through scientific_names)...
[INFO] [2020-11-30 14:16:06] traits to occurrences...
[INFO] [2020-11-30 14:16:06] traits to nodes (through occurrences)...
[INFO] [2020-11-30 14:16:06] Traits to sex term...
[INFO] [2020-11-30 14:16:06] Traits to lifestage term...
[INFO] [2020-11-30 14:16:06] MetaTraits to traits...
[INFO] [2020-11-30 14:16:06] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-11-30 14:16:06] Assocs to occurrences...
[INFO] [2020-11-30 14:16:06] Assocs to nodes...
[INFO] [2020-11-30 14:16:06] Assoc to sex term...
[INFO] [2020-11-30 14:16:06] Assoc to lifestage term...
[INFO] [2020-11-30 14:16:06] MetaAssoc to assocs...
[STOP] [2020-11-30 14:16:06] resolve_keys
[START] [2020-11-30 14:16:06] hold_for_later_1
[STOP] [2020-11-30 14:16:06] hold_for_later_1
[START] [2020-11-30 14:16:06] hold_for_later_2
[STOP] [2020-11-30 14:16:06] hold_for_later_2
[START] [2020-11-30 14:16:06] resolve_missing_parents
[STOP] [2020-11-30 14:16:06] resolve_missing_parents
[START] [2020-11-30 14:16:06] rebuild_nodes
[START] [2020-11-30 14:16:06] Flattener#flatten
[START] [2020-11-30 14:16:06] Flattener#study_resource
[START] [2020-11-30 14:16:06] Flattener#build_ancestry
[STOP] [2020-11-30 14:16:06] Flattener#build_ancestry
[INFO] [2020-11-30 14:16:06] 31 ancestry keys
[START] [2020-11-30 14:16:06] build_node_ancestors
[INFO] [2020-11-30 14:16:06] old ancestors deleted.
[STOP] [2020-11-30 14:16:06] build_node_ancestors
[WARN] [2020-11-30 14:16:06] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-11-30 14:16:06] Flattener#flatten
[STOP] [2020-11-30 14:16:06] rebuild_nodes
[START] [2020-11-30 14:16:06] resolve_missing_media_owners
[STOP] [2020-11-30 14:16:06] resolve_missing_media_owners
[START] [2020-11-30 14:16:06] sanitize_media_verbatims
[STOP] [2020-11-30 14:16:06] sanitize_media_verbatims
[START] [2020-11-30 14:16:06] queue_downloads
[STOP] [2020-11-30 14:16:06] queue_downloads
[START] [2020-11-30 14:16:06] parse_names
[WARN] [2020-11-30 14:16:06] I see 31 names which still need to be parsed.
[STOP] [2020-11-30 14:16:07] parse_names
[START] [2020-11-30 14:16:07] denormalize_canonical_names_to_nodes
[STOP] [2020-11-30 14:16:07] denormalize_canonical_names_to_nodes
[START] [2020-11-30 14:16:07] match_nodes
[START] [2020-11-30 14:16:07] map_all_nodes_to_pages
[STOP] [2020-11-30 14:16:07] map_all_nodes_to_pages
[INFO] [2020-11-30 14:16:07] ZERO unmatched nodes (of 31)! Nicely done.
[START] [2020-11-30 14:16:07] update_nodes
[STOP] [2020-11-30 14:16:07] update_nodes
[STOP] [2020-11-30 14:16:07] match_nodes
[START] [2020-11-30 14:16:07] reindex_search
[STOP] [2020-11-30 14:16:07] reindex_search
[START] [2020-11-30 14:16:07] normalize_units
[STOP] [2020-11-30 14:16:07] normalize_units
[START] [2020-11-30 14:16:07] calculate_statistics
[2020-11-30 14:16:07] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-11-30 14:16:07] calculate_statistics
[START] [2020-11-30 14:16:07] complete_harvest_instance
[START] [2020-11-30 14:16:07] overall_tsv_creation
[INFO] [2020-11-30 14:16:07] Processing group of 31 in 1 batches of 10000
[INFO] [2020-11-30 14:16:51] 29 Traits (unfiltered)...
[INFO] [2020-11-30 14:17:29] 29 Traits (filtered)...
[INFO] [2020-11-30 14:17:29] 0 Associations (filtered)...
[INFO] [2020-11-30 14:17:29] 230 metadata added.
[INFO] [2020-11-30 14:17:29] 0 metadata added.
[INFO] [2020-11-30 14:17:29] Average Time: 52.68
[INFO] [2020-11-30 14:17:29] Total Time: 1m22s
[STOP] [2020-11-30 14:17:29] overall_tsv_creation
[INFO] [2020-11-30 14:17:29] Done. Check your files:
[INFO] [2020-11-30 14:17:29] (31 lines) /app/public/data/bitsch_bitsch_20/publish_nodes.tsv
[INFO] [2020-11-30 14:17:29] (31 lines) /app/public/data/bitsch_bitsch_20/publish_scientific_names.tsv
[INFO] [2020-11-30 14:17:29] (30 lines) /app/public/data/bitsch_bitsch_20/publish_traits.tsv
[INFO] [2020-11-30 14:17:29] (223 lines) /app/public/data/bitsch_bitsch_20/publish_metadata.tsv
[STOP] [2020-11-30 14:17:29] complete_harvest_instance
[START] [2020-11-30 14:17:29] completed
[STOP] [2020-11-30 14:17:29] completed
[STOP] [2020-11-30 14:17:29] logged process, took 94.42
[INFO] [2021-04-19 09:38:44] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-19 09:38:45] ## remove_type: ScientificName
[INFO] [2021-04-19 09:38:45] ++ Calling delete_all on 31 instances...
[INFO] [2021-04-19 09:38:45] [09:38:45.672] Removed 31 Scientificnames
[INFO] [2021-04-19 09:38:45] ## remove_type: Vernacular
[INFO] [2021-04-19 09:38:45] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:38:45] [09:38:45.674] Removed 0 Vernaculars
[INFO] [2021-04-19 09:38:45] ## remove_type: Article
[INFO] [2021-04-19 09:38:45] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:38:45] [09:38:45.675] Removed 0 Articles
[INFO] [2021-04-19 09:38:45] ## remove_type: Medium
[INFO] [2021-04-19 09:38:45] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:38:45] [09:38:45.677] Removed 0 Media
[INFO] [2021-04-19 09:38:45] ## remove_type: Trait
[INFO] [2021-04-19 09:38:45] ++ Calling delete_all on 236 instances...
[INFO] [2021-04-19 09:38:45] [09:38:45.688] Removed 236 Traits
[INFO] [2021-04-19 09:38:45] ## remove_type: MetaTrait
[INFO] [2021-04-19 09:38:45] ++ Calling delete_all on 38 instances...
[INFO] [2021-04-19 09:38:45] [09:38:45.706] Removed 38 Metatraits
[INFO] [2021-04-19 09:38:45] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-19 09:38:45] ++ Calling delete_all on 14 instances...
[INFO] [2021-04-19 09:38:45] [09:38:45.713] Removed 14 Occurrencemetadata
[INFO] [2021-04-19 09:38:45] ## remove_type: Assoc
[INFO] [2021-04-19 09:38:45] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:38:45] [09:38:45.715] Removed 0 Assocs
[INFO] [2021-04-19 09:38:45] ## remove_type: MetaAssoc
[INFO] [2021-04-19 09:38:45] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:38:45] [09:38:45.717] Removed 0 Metaassocs
[INFO] [2021-04-19 09:38:45] ## remove_type: Identifier
[INFO] [2021-04-19 09:38:45] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:38:45] [09:38:45.718] Removed 0 Identifiers
[INFO] [2021-04-19 09:38:45] ## remove_type: Reference
[INFO] [2021-04-19 09:38:45] ++ Calling delete_all on 2 instances...
[INFO] [2021-04-19 09:38:45] [09:38:45.726] Removed 2 References
[INFO] [2021-04-19 09:38:45] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] Starting batch with ID 85685320...
[INFO] [2021-04-19 09:38:46] ## remove_type: Node
[INFO] [2021-04-19 09:38:46] ++ Calling delete_all on 31 instances...
[INFO] [2021-04-19 09:38:46] [09:38:46.939] Removed 31 Nodes
[START] [2021-04-19 09:38:47] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 09:38:47] Creating resource from OpenData
[START] [2021-04-19 09:38:48] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 09:38:48] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 09:38:53] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 09:38:53] Creating resource from OpenData
[START] [2021-04-19 09:38:53] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 09:38:53] create_harvest_instance
[INFO] [2021-04-19 09:38:53] Created harvest instance #3718
[STOP] [2021-04-19 09:38:53] create_harvest_instance
[START] [2021-04-19 09:38:53] fetch_files
[STOP] [2021-04-19 09:38:53] fetch_files
[START] [2021-04-19 09:38:53] validate_each_file
[INFO] [2021-04-19 09:38:53] Looping over 4 formats...
[INFO] [2021-04-19 09:38:53] ...refs (/app/public/data/bitsch_bitsch_20/references.tsv)
[INFO] [2021-04-19 09:38:53] Valid: /app/public/converted_csv/bitsch_bitsch_20_refs_3718.csv (2 lines)
[INFO] [2021-04-19 09:38:53] ...nodes (/app/public/data/bitsch_bitsch_20/taxa.txt)
[INFO] [2021-04-19 09:38:53] Valid: /app/public/converted_csv/bitsch_bitsch_20_nodes_3718.csv (31 lines)
[INFO] [2021-04-19 09:38:53] ...occurrences (/app/public/data/bitsch_bitsch_20/occurrences.txt)
[INFO] [2021-04-19 09:38:53] Valid: /app/public/converted_csv/bitsch_bitsch_20_occurrences_3718.csv (34 lines)
[INFO] [2021-04-19 09:38:53] ...measurements (/app/public/data/bitsch_bitsch_20/measurementsorfacts.txt)
[INFO] [2021-04-19 09:38:53] Valid: /app/public/converted_csv/bitsch_bitsch_20_measurements_3718.csv (236 lines)
[STOP] [2021-04-19 09:38:53] validate_each_file
[START] [2021-04-19 09:38:53] convert_to_csv
[INFO] [2021-04-19 09:38:53] Looping over 4 formats...
[INFO] [2021-04-19 09:38:53] ...refs (/app/public/data/bitsch_bitsch_20/references.tsv)
[CMD] [2021-04-19 09:38:53] /usr/bin/sort /app/public/converted_csv/bitsch_bitsch_20_refs_3718.csv > /app/public/converted_csv/bitsch_bitsch_20_refs_3718.csv_sorted
[INFO] [2021-04-19 09:38:54] Converted: /app/public/converted_csv/bitsch_bitsch_20_refs_3718.csv (2 lines)
[INFO] [2021-04-19 09:38:54] ...nodes (/app/public/data/bitsch_bitsch_20/taxa.txt)
[CMD] [2021-04-19 09:38:54] /usr/bin/sort /app/public/converted_csv/bitsch_bitsch_20_nodes_3718.csv > /app/public/converted_csv/bitsch_bitsch_20_nodes_3718.csv_sorted
[INFO] [2021-04-19 09:38:54] Converted: /app/public/converted_csv/bitsch_bitsch_20_nodes_3718.csv (31 lines)
[INFO] [2021-04-19 09:38:54] ...occurrences (/app/public/data/bitsch_bitsch_20/occurrences.txt)
[CMD] [2021-04-19 09:38:54] /usr/bin/sort /app/public/converted_csv/bitsch_bitsch_20_occurrences_3718.csv > /app/public/converted_csv/bitsch_bitsch_20_occurrences_3718.csv_sorted
[INFO] [2021-04-19 09:38:55] Converted: /app/public/converted_csv/bitsch_bitsch_20_occurrences_3718.csv (34 lines)
[INFO] [2021-04-19 09:38:55] ...measurements (/app/public/data/bitsch_bitsch_20/measurementsorfacts.txt)
[CMD] [2021-04-19 09:38:55] /usr/bin/sort /app/public/converted_csv/bitsch_bitsch_20_measurements_3718.csv > /app/public/converted_csv/bitsch_bitsch_20_measurements_3718.csv_sorted
[INFO] [2021-04-19 09:38:55] Converted: /app/public/converted_csv/bitsch_bitsch_20_measurements_3718.csv (236 lines)
[STOP] [2021-04-19 09:38:55] convert_to_csv
[START] [2021-04-19 09:38:55] calculate_delta
[INFO] [2021-04-19 09:38:55] Looping over 4 formats...
[INFO] [2021-04-19 09:38:55] ...refs (/app/public/data/bitsch_bitsch_20/references.tsv)
[CMD] [2021-04-19 09:38:55] echo "0a" > /app/public/diff/bitsch_bitsch_20_refs_3718.diff
[CMD] [2021-04-19 09:38:55] tail -n +1 /app/public/converted_csv/bitsch_bitsch_20_refs_3718.csv >> /app/public/diff/bitsch_bitsch_20_refs_3718.diff
[CMD] [2021-04-19 09:38:56] echo "." >> /app/public/diff/bitsch_bitsch_20_refs_3718.diff
[INFO] [2021-04-19 09:38:56] Created diff: /app/public/diff/bitsch_bitsch_20_refs_3718.diff (4 lines)
[INFO] [2021-04-19 09:38:56] ...nodes (/app/public/data/bitsch_bitsch_20/taxa.txt)
[CMD] [2021-04-19 09:38:56] echo "0a" > /app/public/diff/bitsch_bitsch_20_nodes_3718.diff
[CMD] [2021-04-19 09:38:56] tail -n +1 /app/public/converted_csv/bitsch_bitsch_20_nodes_3718.csv >> /app/public/diff/bitsch_bitsch_20_nodes_3718.diff
[CMD] [2021-04-19 09:38:57] echo "." >> /app/public/diff/bitsch_bitsch_20_nodes_3718.diff
[INFO] [2021-04-19 09:38:57] Created diff: /app/public/diff/bitsch_bitsch_20_nodes_3718.diff (33 lines)
[INFO] [2021-04-19 09:38:57] ...occurrences (/app/public/data/bitsch_bitsch_20/occurrences.txt)
[CMD] [2021-04-19 09:38:57] echo "0a" > /app/public/diff/bitsch_bitsch_20_occurrences_3718.diff
[CMD] [2021-04-19 09:38:58] tail -n +1 /app/public/converted_csv/bitsch_bitsch_20_occurrences_3718.csv >> /app/public/diff/bitsch_bitsch_20_occurrences_3718.diff
[CMD] [2021-04-19 09:38:58] echo "." >> /app/public/diff/bitsch_bitsch_20_occurrences_3718.diff
[INFO] [2021-04-19 09:38:58] Created diff: /app/public/diff/bitsch_bitsch_20_occurrences_3718.diff (36 lines)
[INFO] [2021-04-19 09:38:58] ...measurements (/app/public/data/bitsch_bitsch_20/measurementsorfacts.txt)
[CMD] [2021-04-19 09:38:58] echo "0a" > /app/public/diff/bitsch_bitsch_20_measurements_3718.diff
[CMD] [2021-04-19 09:38:59] tail -n +1 /app/public/converted_csv/bitsch_bitsch_20_measurements_3718.csv >> /app/public/diff/bitsch_bitsch_20_measurements_3718.diff
[CMD] [2021-04-19 09:38:59] echo "." >> /app/public/diff/bitsch_bitsch_20_measurements_3718.diff
[INFO] [2021-04-19 09:39:00] Created diff: /app/public/diff/bitsch_bitsch_20_measurements_3718.diff (238 lines)
[STOP] [2021-04-19 09:39:00] calculate_delta
[START] [2021-04-19 09:39:00] parse_diff_and_store
[INFO] [2021-04-19 09:39:00] Handling diff: /app/public/diff/bitsch_bitsch_20_refs_3718.diff (4 lines)
[INFO] [2021-04-19 09:39:00] Loading refs diff file into memory (4 /app/public/diff/bitsch_bitsch_20_refs_3718.diff lines)...
[INFO] [2021-04-19 09:39:00] Handling diff: /app/public/diff/bitsch_bitsch_20_nodes_3718.diff (33 lines)
[INFO] [2021-04-19 09:39:01] Loading nodes diff file into memory (33 /app/public/diff/bitsch_bitsch_20_nodes_3718.diff lines)...
[INFO] [2021-04-19 09:39:01] Handling diff: /app/public/diff/bitsch_bitsch_20_occurrences_3718.diff (36 lines)
[INFO] [2021-04-19 09:39:02] Loading occurrences diff file into memory (36 /app/public/diff/bitsch_bitsch_20_occurrences_3718.diff lines)...
[INFO] [2021-04-19 09:39:02] Handling diff: /app/public/diff/bitsch_bitsch_20_measurements_3718.diff (238 lines)
[INFO] [2021-04-19 09:39:02] Loading measurements diff file into memory (238 /app/public/diff/bitsch_bitsch_20_measurements_3718.diff lines)...
[INFO] [2021-04-19 09:39:03] Storing 2 References
[INFO] [2021-04-19 09:39:03] Processing group of 2 in 1 groups of 1000
[INFO] [2021-04-19 09:39:03] Average Time: 0.0
[INFO] [2021-04-19 09:39:03] Total Time: 1s
[INFO] [2021-04-19 09:39:03] Storing 31 ScientificNames
[INFO] [2021-04-19 09:39:03] Processing group of 31 in 1 groups of 1000
[INFO] [2021-04-19 09:39:03] Average Time: 0.01
[INFO] [2021-04-19 09:39:03] Total Time: 1s
[INFO] [2021-04-19 09:39:03] Storing 31 Nodes
[INFO] [2021-04-19 09:39:03] Processing group of 31 in 1 groups of 1000
[INFO] [2021-04-19 09:39:03] Average Time: 0.02
[INFO] [2021-04-19 09:39:03] Total Time: 1s
[INFO] [2021-04-19 09:39:03] Storing 34 Occurrences
[INFO] [2021-04-19 09:39:03] Processing group of 34 in 1 groups of 1000
[INFO] [2021-04-19 09:39:03] Average Time: 0.1
[INFO] [2021-04-19 09:39:03] Total Time: 1s
[INFO] [2021-04-19 09:39:03] Storing 14 OccurrenceMetadata
[INFO] [2021-04-19 09:39:03] Processing group of 14 in 1 groups of 1000
[INFO] [2021-04-19 09:39:03] Average Time: 0.0
[INFO] [2021-04-19 09:39:03] Total Time: 1s
[INFO] [2021-04-19 09:39:03] Storing 236 Traits
[INFO] [2021-04-19 09:39:03] Processing group of 236 in 1 groups of 1000
[INFO] [2021-04-19 09:39:03] Average Time: 0.06
[INFO] [2021-04-19 09:39:03] Total Time: 1s
[INFO] [2021-04-19 09:39:03] Storing 38 MetaTraits
[INFO] [2021-04-19 09:39:03] Processing group of 38 in 1 groups of 1000
[INFO] [2021-04-19 09:39:03] Average Time: 0.01
[INFO] [2021-04-19 09:39:03] Total Time: 1s
[INFO] [2021-04-19 09:39:03] Storing 2 TraitsReferences
[INFO] [2021-04-19 09:39:03] Processing group of 2 in 1 groups of 1000
[INFO] [2021-04-19 09:39:03] Average Time: 0.0
[INFO] [2021-04-19 09:39:03] Total Time: 1s
[STOP] [2021-04-19 09:39:03] parse_diff_and_store
[START] [2021-04-19 09:39:03] resolve_keys
[INFO] [2021-04-19 09:39:10] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-19 09:39:10] traits to occurrences...
[INFO] [2021-04-19 09:39:10] traits to nodes (through occurrences)...
[INFO] [2021-04-19 09:39:10] Traits to sex term...
[INFO] [2021-04-19 09:39:10] Traits to lifestage term...
[INFO] [2021-04-19 09:39:10] MetaTraits to traits...
[INFO] [2021-04-19 09:39:10] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-19 09:39:10] Assocs to occurrences...
[INFO] [2021-04-19 09:39:10] Assocs to nodes...
[INFO] [2021-04-19 09:39:10] Assoc to sex term...
[INFO] [2021-04-19 09:39:10] Assoc to lifestage term...
[INFO] [2021-04-19 09:39:10] MetaAssoc to assocs...
[STOP] [2021-04-19 09:39:10] resolve_keys
[START] [2021-04-19 09:39:10] hold_for_later_1
[STOP] [2021-04-19 09:39:10] hold_for_later_1
[START] [2021-04-19 09:39:10] hold_for_later_2
[STOP] [2021-04-19 09:39:10] hold_for_later_2
[START] [2021-04-19 09:39:10] resolve_missing_parents
[STOP] [2021-04-19 09:39:10] resolve_missing_parents
[START] [2021-04-19 09:39:10] rebuild_nodes
[START] [2021-04-19 09:39:10] Flattener#flatten
[START] [2021-04-19 09:39:10] Flattener#study_resource
[START] [2021-04-19 09:39:10] Flattener#build_ancestry
[STOP] [2021-04-19 09:39:10] Flattener#build_ancestry
[INFO] [2021-04-19 09:39:10] 31 ancestry keys
[START] [2021-04-19 09:39:10] build_node_ancestors
[INFO] [2021-04-19 09:39:10] old ancestors deleted.
[STOP] [2021-04-19 09:39:10] build_node_ancestors
[WARN] [2021-04-19 09:39:10] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2021-04-19 09:39:10] Flattener#flatten
[STOP] [2021-04-19 09:39:10] rebuild_nodes
[START] [2021-04-19 09:39:10] resolve_missing_media_owners
[STOP] [2021-04-19 09:39:10] resolve_missing_media_owners
[START] [2021-04-19 09:39:10] sanitize_media_verbatims
[STOP] [2021-04-19 09:39:10] sanitize_media_verbatims
[START] [2021-04-19 09:39:10] queue_downloads
[STOP] [2021-04-19 09:39:10] queue_downloads
[START] [2021-04-19 09:39:10] parse_names
[WARN] [2021-04-19 09:39:10] I see 31 names which still need to be parsed.
[STOP] [2021-04-19 09:39:11] parse_names
[START] [2021-04-19 09:39:11] denormalize_canonical_names_to_nodes
[STOP] [2021-04-19 09:39:11] denormalize_canonical_names_to_nodes
[START] [2021-04-19 09:39:11] match_nodes
[START] [2021-04-19 09:39:11] map_all_nodes_to_pages
[STOP] [2021-04-19 09:39:11] map_all_nodes_to_pages
[INFO] [2021-04-19 09:39:11] ZERO unmatched nodes (of 31)! Nicely done.
[START] [2021-04-19 09:39:11] update_nodes
[STOP] [2021-04-19 09:39:11] update_nodes
[STOP] [2021-04-19 09:39:11] match_nodes
[START] [2021-04-19 09:39:11] reindex_search
[STOP] [2021-04-19 09:39:12] reindex_search
[START] [2021-04-19 09:39:12] normalize_units
[STOP] [2021-04-19 09:39:12] normalize_units
[START] [2021-04-19 09:39:12] calculate_statistics
[2021-04-19 09:39:12] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2021-04-19 09:39:12] calculate_statistics
[START] [2021-04-19 09:39:12] complete_harvest_instance
[START] [2021-04-19 09:39:12] overall_tsv_creation
[INFO] [2021-04-19 09:39:12] Processing group of 31 in 1 batches of 10000
[INFO] [2021-04-19 09:39:48] 29 Traits (unfiltered)...
[INFO] [2021-04-19 09:40:22] 29 Traits (filtered)...
[INFO] [2021-04-19 09:40:22] 0 Associations (filtered)...
[INFO] [2021-04-19 09:40:22] 191 metadata added.
[INFO] [2021-04-19 09:40:22] 0 metadata added.
[INFO] [2021-04-19 09:40:48] Average Time: 72.3
[INFO] [2021-04-19 09:40:48] Total Time: 1m37s
[STOP] [2021-04-19 09:40:48] overall_tsv_creation
[INFO] [2021-04-19 09:40:48] Done. Check your files:
[INFO] [2021-04-19 09:40:49] (31 lines) /app/public/data/bitsch_bitsch_20/publish_nodes.tsv
[INFO] [2021-04-19 09:40:49] (31 lines) /app/public/data/bitsch_bitsch_20/publish_scientific_names.tsv
[INFO] [2021-04-19 09:40:49] (30 lines) /app/public/data/bitsch_bitsch_20/publish_traits.tsv
[INFO] [2021-04-19 09:40:50] (192 lines) /app/public/data/bitsch_bitsch_20/publish_metadata.tsv
[STOP] [2021-04-19 09:40:50] complete_harvest_instance
[START] [2021-04-19 09:40:50] completed
[STOP] [2021-04-19 09:40:50] completed
[STOP] [2021-04-19 09:40:50] logged process, took 116.98

Latest Process