Harvest for
Nilsson 1983
Created
19 Apr 15:47
Stage:
completed
Fetched:
19 Apr 15:47
Validated:
19 Apr 15:47
Deltas Created
19 Apr 15:47
Units Normalized:
19 Apr 15:47
Ancestry Built:
19 Apr 15:47
Nodes Matched:
19 Apr 15:47
Names Parsed:
19 Apr 15:47
New Models Stored:
19 Apr 15:47
Indexed:
19 Apr 15:47
Completed:
19 Apr 15:49
Time to Harvest:
less than a minute
Harvesting Log
(654 lines)
# Logfile created on 2020-08-27 15:46:11 -0400 by logger.rb/v1.4.2
[START] [2020-08-27 15:46:11] logged process
[START] [2020-08-27 15:46:11] Creating resource from OpenData
[START] [2020-08-27 15:46:11] logged process
[START] [2020-08-27 15:46:11] Parse meta.xml file and create formats with fields
[STOP] [2020-08-27 15:46:11] Parse meta.xml file and create formats with fields
[STOP] [2020-08-27 15:46:11] Creating resource from OpenData
[INFO] [2020-12-03 11:53:16] ## HARVEST: type = -harvest
[START] [2020-12-03 11:53:20] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979
[START] [2020-12-03 11:53:20] create_harvest_instance
[STOP] [2020-12-03 11:53:21] create_harvest_instance
[START] [2020-12-03 11:53:21] fetch_files
[STOP] [2020-12-03 11:53:21] fetch_files
[START] [2020-12-03 11:53:21] validate_each_file
[STOP] [2020-12-03 11:53:22] validate_each_file
[START] [2020-12-03 11:53:22] convert_to_csv
[CMD] [2020-12-03 11:53:22] /usr/bin/sort /app/public/converted_csv/nilsson_nilsson2_refs_24852.csv > /app/public/converted_csv/nilsson_nilsson2_refs_24852.csv_sorted
[CMD] [2020-12-03 11:53:22] /usr/bin/sort /app/public/converted_csv/nilsson_nilsson2_nodes_24853.csv > /app/public/converted_csv/nilsson_nilsson2_nodes_24853.csv_sorted
[CMD] [2020-12-03 11:53:22] /usr/bin/sort /app/public/converted_csv/nilsson_nilsson2_occurrences_24854.csv > /app/public/converted_csv/nilsson_nilsson2_occurrences_24854.csv_sorted
[CMD] [2020-12-03 11:53:22] /usr/bin/sort /app/public/converted_csv/nilsson_nilsson2_measurements_24855.csv > /app/public/converted_csv/nilsson_nilsson2_measurements_24855.csv_sorted
[STOP] [2020-12-03 11:53:22] convert_to_csv
[START] [2020-12-03 11:53:22] calculate_delta
[CMD] [2020-12-03 11:53:22] echo "0a" > /app/public/diff/nilsson_nilsson2_refs_24852.diff
[CMD] [2020-12-03 11:53:22] tail -n +1 /app/public/converted_csv/nilsson_nilsson2_refs_24852.csv >> /app/public/diff/nilsson_nilsson2_refs_24852.diff
[CMD] [2020-12-03 11:53:22] echo "." >> /app/public/diff/nilsson_nilsson2_refs_24852.diff
[CMD] [2020-12-03 11:53:22] echo "0a" > /app/public/diff/nilsson_nilsson2_nodes_24853.diff
[CMD] [2020-12-03 11:53:22] tail -n +1 /app/public/converted_csv/nilsson_nilsson2_nodes_24853.csv >> /app/public/diff/nilsson_nilsson2_nodes_24853.diff
[CMD] [2020-12-03 11:53:22] echo "." >> /app/public/diff/nilsson_nilsson2_nodes_24853.diff
[CMD] [2020-12-03 11:53:22] echo "0a" > /app/public/diff/nilsson_nilsson2_occurrences_24854.diff
[CMD] [2020-12-03 11:53:22] tail -n +1 /app/public/converted_csv/nilsson_nilsson2_occurrences_24854.csv >> /app/public/diff/nilsson_nilsson2_occurrences_24854.diff
[CMD] [2020-12-03 11:53:22] echo "." >> /app/public/diff/nilsson_nilsson2_occurrences_24854.diff
[CMD] [2020-12-03 11:53:22] echo "0a" > /app/public/diff/nilsson_nilsson2_measurements_24855.diff
[CMD] [2020-12-03 11:53:22] tail -n +1 /app/public/converted_csv/nilsson_nilsson2_measurements_24855.csv >> /app/public/diff/nilsson_nilsson2_measurements_24855.diff
[CMD] [2020-12-03 11:53:22] echo "." >> /app/public/diff/nilsson_nilsson2_measurements_24855.diff
[STOP] [2020-12-03 11:53:22] calculate_delta
[START] [2020-12-03 11:53:22] parse_diff_and_store
[INFO] [2020-12-03 11:53:22] Loading refs diff file into memory (true lines)...
[INFO] [2020-12-03 11:53:22] Loading nodes diff file into memory (true lines)...
[INFO] [2020-12-03 11:53:22] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-12-03 11:53:23] Loading measurements diff file into memory (true lines)...
[INFO] [2020-12-03 11:53:23] Storing 46 ScientificNames
[INFO] [2020-12-03 11:53:23] Processing group of 46 in 1 groups of 1000
[INFO] [2020-12-03 11:53:23] Average Time: 0.05
[INFO] [2020-12-03 11:53:23] Total Time: 1s
[INFO] [2020-12-03 11:53:23] Storing 46 Nodes
[INFO] [2020-12-03 11:53:23] Processing group of 46 in 1 groups of 1000
[INFO] [2020-12-03 11:53:23] Average Time: 0.03
[INFO] [2020-12-03 11:53:23] Total Time: 1s
[INFO] [2020-12-03 11:53:23] Storing 47 Occurrences
[INFO] [2020-12-03 11:53:23] Processing group of 47 in 1 groups of 1000
[INFO] [2020-12-03 11:53:23] Average Time: 0.02
[INFO] [2020-12-03 11:53:23] Total Time: 1s
[INFO] [2020-12-03 11:53:23] Storing 4 OccurrenceMetadata
[INFO] [2020-12-03 11:53:23] Processing group of 4 in 1 groups of 1000
[INFO] [2020-12-03 11:53:23] Average Time: 0.0
[INFO] [2020-12-03 11:53:23] Total Time: 1s
[INFO] [2020-12-03 11:53:23] Storing 136 Traits
[INFO] [2020-12-03 11:53:23] Processing group of 136 in 1 groups of 1000
[INFO] [2020-12-03 11:53:23] Average Time: 0.07
[INFO] [2020-12-03 11:53:23] Total Time: 1s
[INFO] [2020-12-03 11:53:23] Storing 47 MetaTraits
[INFO] [2020-12-03 11:53:23] Processing group of 47 in 1 groups of 1000
[INFO] [2020-12-03 11:53:23] Average Time: 0.01
[INFO] [2020-12-03 11:53:23] Total Time: 1s
[STOP] [2020-12-03 11:53:23] parse_diff_and_store
[START] [2020-12-03 11:53:23] resolve_keys
[INFO] [2020-12-03 11:53:29] Occurrences to nodes (through scientific_names)...
[INFO] [2020-12-03 11:53:29] traits to occurrences...
[INFO] [2020-12-03 11:53:29] traits to nodes (through occurrences)...
[INFO] [2020-12-03 11:53:29] Traits to sex term...
[INFO] [2020-12-03 11:53:29] Traits to lifestage term...
[INFO] [2020-12-03 11:53:29] MetaTraits to traits...
[INFO] [2020-12-03 11:53:29] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-12-03 11:53:29] Assocs to occurrences...
[INFO] [2020-12-03 11:53:29] Assocs to nodes...
[INFO] [2020-12-03 11:53:29] Assoc to sex term...
[INFO] [2020-12-03 11:53:29] Assoc to lifestage term...
[INFO] [2020-12-03 11:53:29] MetaAssoc to assocs...
[STOP] [2020-12-03 11:53:29] resolve_keys
[START] [2020-12-03 11:53:29] hold_for_later_1
[STOP] [2020-12-03 11:53:29] hold_for_later_1
[START] [2020-12-03 11:53:29] hold_for_later_2
[STOP] [2020-12-03 11:53:29] hold_for_later_2
[START] [2020-12-03 11:53:29] resolve_missing_parents
[STOP] [2020-12-03 11:53:29] resolve_missing_parents
[START] [2020-12-03 11:53:29] rebuild_nodes
[START] [2020-12-03 11:53:29] Flattener#flatten
[START] [2020-12-03 11:53:29] Flattener#study_resource
[START] [2020-12-03 11:53:29] Flattener#build_ancestry
[STOP] [2020-12-03 11:53:29] Flattener#build_ancestry
[INFO] [2020-12-03 11:53:29] 46 ancestry keys
[START] [2020-12-03 11:53:29] build_node_ancestors
[INFO] [2020-12-03 11:53:29] old ancestors deleted.
[STOP] [2020-12-03 11:53:29] build_node_ancestors
[WARN] [2020-12-03 11:53:29] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-12-03 11:53:29] Flattener#flatten
[STOP] [2020-12-03 11:53:29] rebuild_nodes
[START] [2020-12-03 11:53:29] resolve_missing_media_owners
[STOP] [2020-12-03 11:53:29] resolve_missing_media_owners
[START] [2020-12-03 11:53:29] sanitize_media_verbatims
[STOP] [2020-12-03 11:53:29] sanitize_media_verbatims
[START] [2020-12-03 11:53:29] queue_downloads
[STOP] [2020-12-03 11:53:29] queue_downloads
[START] [2020-12-03 11:53:29] parse_names
[WARN] [2020-12-03 11:53:29] I see 46 names which still need to be parsed.
[WARN] [2020-12-03 11:53:30] I see 1 names which still need to be parsed.
[STOP] [2020-12-03 11:53:31] parse_names
[START] [2020-12-03 11:53:31] denormalize_canonical_names_to_nodes
[STOP] [2020-12-03 11:53:31] denormalize_canonical_names_to_nodes
[START] [2020-12-03 11:53:31] match_nodes
[START] [2020-12-03 11:53:31] map_all_nodes_to_pages
[STOP] [2020-12-03 11:53:31] map_all_nodes_to_pages
[INFO] [2020-12-03 11:53:31] ZERO unmatched nodes (of 46)! Nicely done.
[START] [2020-12-03 11:53:31] update_nodes
[STOP] [2020-12-03 11:53:31] update_nodes
[STOP] [2020-12-03 11:53:31] match_nodes
[START] [2020-12-03 11:53:31] reindex_search
[STOP] [2020-12-03 11:53:31] reindex_search
[START] [2020-12-03 11:53:31] normalize_units
[STOP] [2020-12-03 11:53:31] normalize_units
[START] [2020-12-03 11:53:31] calculate_statistics
[2020-12-03 11:53:31] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-12-03 11:53:31] calculate_statistics
[START] [2020-12-03 11:53:31] complete_harvest_instance
[START] [2020-12-03 11:53:31] overall_tsv_creation
[INFO] [2020-12-03 11:53:31] Processing group of 46 in 1 batches of 10000
[INFO] [2020-12-03 11:54:11] 47 Traits (unfiltered)...
[INFO] [2020-12-03 11:54:50] 47 Traits (filtered)...
[INFO] [2020-12-03 11:54:50] 0 Associations (filtered)...
[INFO] [2020-12-03 11:54:50] 142 metadata added.
[INFO] [2020-12-03 11:54:50] 0 metadata added.
[INFO] [2020-12-03 11:54:50] Average Time: 55.19
[INFO] [2020-12-03 11:54:50] Total Time: 1m19s
[STOP] [2020-12-03 11:54:50] overall_tsv_creation
[INFO] [2020-12-03 11:54:50] Done. Check your files:
[INFO] [2020-12-03 11:54:50] (45 lines) /app/public/data/nilsson_nilsson2/publish_nodes.tsv
[INFO] [2020-12-03 11:54:50] (46 lines) /app/public/data/nilsson_nilsson2/publish_scientific_names.tsv
[INFO] [2020-12-03 11:54:50] (48 lines) /app/public/data/nilsson_nilsson2/publish_traits.tsv
[INFO] [2020-12-03 11:54:50] (137 lines) /app/public/data/nilsson_nilsson2/publish_metadata.tsv
[STOP] [2020-12-03 11:54:50] complete_harvest_instance
[START] [2020-12-03 11:54:50] completed
[STOP] [2020-12-03 11:54:50] completed
[STOP] [2020-12-03 11:54:50] logged process, took 89.77
[INFO] [2021-04-19 10:13:20] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-19 10:13:23] ## remove_type: ScientificName
[INFO] [2021-04-19 10:13:23] ++ Calling delete_all on 46 instances...
[INFO] [2021-04-19 10:13:23] [10:13:23.992] Removed 46 Scientificnames
[INFO] [2021-04-19 10:13:23] ## remove_type: Vernacular
[INFO] [2021-04-19 10:13:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:13:23] [10:13:23.993] Removed 0 Vernaculars
[INFO] [2021-04-19 10:13:23] ## remove_type: Article
[INFO] [2021-04-19 10:13:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:13:23] [10:13:23.995] Removed 0 Articles
[INFO] [2021-04-19 10:13:23] ## remove_type: Medium
[INFO] [2021-04-19 10:13:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:13:23] [10:13:23.996] Removed 0 Media
[INFO] [2021-04-19 10:13:23] ## remove_type: Trait
[INFO] [2021-04-19 10:13:23] ++ Calling delete_all on 136 instances...
[INFO] [2021-04-19 10:13:24] [10:13:24.001] Removed 136 Traits
[INFO] [2021-04-19 10:13:24] ## remove_type: MetaTrait
[INFO] [2021-04-19 10:13:24] ++ Calling delete_all on 47 instances...
[INFO] [2021-04-19 10:13:24] [10:13:24.004] Removed 47 Metatraits
[INFO] [2021-04-19 10:13:24] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-19 10:13:24] ++ Calling delete_all on 4 instances...
[INFO] [2021-04-19 10:13:24] [10:13:24.006] Removed 4 Occurrencemetadata
[INFO] [2021-04-19 10:13:24] ## remove_type: Assoc
[INFO] [2021-04-19 10:13:24] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:13:24] [10:13:24.007] Removed 0 Assocs
[INFO] [2021-04-19 10:13:24] ## remove_type: MetaAssoc
[INFO] [2021-04-19 10:13:24] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:13:24] [10:13:24.008] Removed 0 Metaassocs
[INFO] [2021-04-19 10:13:24] ## remove_type: Identifier
[INFO] [2021-04-19 10:13:24] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:13:24] [10:13:24.010] Removed 0 Identifiers
[INFO] [2021-04-19 10:13:24] ## remove_type: Reference
[INFO] [2021-04-19 10:13:24] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:13:24] [10:13:24.012] Removed 0 References
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] Starting batch with ID 86116168...
[INFO] [2021-04-19 10:13:24] ## remove_type: Node
[INFO] [2021-04-19 10:13:24] ++ Calling delete_all on 46 instances...
[INFO] [2021-04-19 10:13:24] [10:13:24.511] Removed 46 Nodes
[START] [2021-04-19 10:13:25] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 10:13:25] Creating resource from OpenData
[START] [2021-04-19 10:13:25] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 10:13:25] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 10:13:30] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 10:13:30] Creating resource from OpenData
[START] [2021-04-19 10:13:30] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 10:13:30] create_harvest_instance
[INFO] [2021-04-19 10:13:30] Created harvest instance #3725
[STOP] [2021-04-19 10:13:30] create_harvest_instance
[START] [2021-04-19 10:13:30] fetch_files
[STOP] [2021-04-19 10:13:30] fetch_files
[START] [2021-04-19 10:13:30] validate_each_file
[INFO] [2021-04-19 10:13:30] Looping over 4 formats...
[INFO] [2021-04-19 10:13:30] ...refs (/app/public/data/nilsson_nilsson2/references.tsv)
[INFO] [2021-04-19 10:13:30] Valid: /app/public/converted_csv/nilsson_nilsson2_refs_3725.csv (0 lines)
[INFO] [2021-04-19 10:13:30] ...nodes (/app/public/data/nilsson_nilsson2/taxa.txt)
[INFO] [2021-04-19 10:13:30] Valid: /app/public/converted_csv/nilsson_nilsson2_nodes_3725.csv (46 lines)
[INFO] [2021-04-19 10:13:30] ...occurrences (/app/public/data/nilsson_nilsson2/occurrences.txt)
[INFO] [2021-04-19 10:13:31] Valid: /app/public/converted_csv/nilsson_nilsson2_occurrences_3725.csv (47 lines)
[INFO] [2021-04-19 10:13:31] ...measurements (/app/public/data/nilsson_nilsson2/measurementsorfacts.txt)
[INFO] [2021-04-19 10:13:31] Valid: /app/public/converted_csv/nilsson_nilsson2_measurements_3725.csv (136 lines)
[STOP] [2021-04-19 10:13:31] validate_each_file
[START] [2021-04-19 10:13:31] convert_to_csv
[INFO] [2021-04-19 10:13:31] Looping over 4 formats...
[INFO] [2021-04-19 10:13:31] ...refs (/app/public/data/nilsson_nilsson2/references.tsv)
[CMD] [2021-04-19 10:13:31] /usr/bin/sort /app/public/converted_csv/nilsson_nilsson2_refs_3725.csv > /app/public/converted_csv/nilsson_nilsson2_refs_3725.csv_sorted
[INFO] [2021-04-19 10:13:31] Converted: /app/public/converted_csv/nilsson_nilsson2_refs_3725.csv (0 lines)
[INFO] [2021-04-19 10:13:31] ...nodes (/app/public/data/nilsson_nilsson2/taxa.txt)
[CMD] [2021-04-19 10:13:31] /usr/bin/sort /app/public/converted_csv/nilsson_nilsson2_nodes_3725.csv > /app/public/converted_csv/nilsson_nilsson2_nodes_3725.csv_sorted
[INFO] [2021-04-19 10:13:31] Converted: /app/public/converted_csv/nilsson_nilsson2_nodes_3725.csv (46 lines)
[INFO] [2021-04-19 10:13:31] ...occurrences (/app/public/data/nilsson_nilsson2/occurrences.txt)
[CMD] [2021-04-19 10:13:31] /usr/bin/sort /app/public/converted_csv/nilsson_nilsson2_occurrences_3725.csv > /app/public/converted_csv/nilsson_nilsson2_occurrences_3725.csv_sorted
[INFO] [2021-04-19 10:13:32] Converted: /app/public/converted_csv/nilsson_nilsson2_occurrences_3725.csv (47 lines)
[INFO] [2021-04-19 10:13:32] ...measurements (/app/public/data/nilsson_nilsson2/measurementsorfacts.txt)
[CMD] [2021-04-19 10:13:32] /usr/bin/sort /app/public/converted_csv/nilsson_nilsson2_measurements_3725.csv > /app/public/converted_csv/nilsson_nilsson2_measurements_3725.csv_sorted
[INFO] [2021-04-19 10:13:32] Converted: /app/public/converted_csv/nilsson_nilsson2_measurements_3725.csv (136 lines)
[STOP] [2021-04-19 10:13:32] convert_to_csv
[START] [2021-04-19 10:13:32] calculate_delta
[INFO] [2021-04-19 10:13:32] Looping over 4 formats...
[INFO] [2021-04-19 10:13:32] ...refs (/app/public/data/nilsson_nilsson2/references.tsv)
[CMD] [2021-04-19 10:13:32] echo "0a" > /app/public/diff/nilsson_nilsson2_refs_3725.diff
[CMD] [2021-04-19 10:13:33] tail -n +1 /app/public/converted_csv/nilsson_nilsson2_refs_3725.csv >> /app/public/diff/nilsson_nilsson2_refs_3725.diff
[CMD] [2021-04-19 10:13:33] echo "." >> /app/public/diff/nilsson_nilsson2_refs_3725.diff
[INFO] [2021-04-19 10:13:33] Created diff: /app/public/diff/nilsson_nilsson2_refs_3725.diff (2 lines)
[INFO] [2021-04-19 10:13:33] ...nodes (/app/public/data/nilsson_nilsson2/taxa.txt)
[CMD] [2021-04-19 10:13:33] echo "0a" > /app/public/diff/nilsson_nilsson2_nodes_3725.diff
[CMD] [2021-04-19 10:13:34] tail -n +1 /app/public/converted_csv/nilsson_nilsson2_nodes_3725.csv >> /app/public/diff/nilsson_nilsson2_nodes_3725.diff
[CMD] [2021-04-19 10:13:34] echo "." >> /app/public/diff/nilsson_nilsson2_nodes_3725.diff
[INFO] [2021-04-19 10:13:34] Created diff: /app/public/diff/nilsson_nilsson2_nodes_3725.diff (48 lines)
[INFO] [2021-04-19 10:13:34] ...occurrences (/app/public/data/nilsson_nilsson2/occurrences.txt)
[CMD] [2021-04-19 10:13:34] echo "0a" > /app/public/diff/nilsson_nilsson2_occurrences_3725.diff
[CMD] [2021-04-19 10:13:35] tail -n +1 /app/public/converted_csv/nilsson_nilsson2_occurrences_3725.csv >> /app/public/diff/nilsson_nilsson2_occurrences_3725.diff
[CMD] [2021-04-19 10:13:35] echo "." >> /app/public/diff/nilsson_nilsson2_occurrences_3725.diff
[INFO] [2021-04-19 10:13:35] Created diff: /app/public/diff/nilsson_nilsson2_occurrences_3725.diff (49 lines)
[INFO] [2021-04-19 10:13:35] ...measurements (/app/public/data/nilsson_nilsson2/measurementsorfacts.txt)
[CMD] [2021-04-19 10:13:35] echo "0a" > /app/public/diff/nilsson_nilsson2_measurements_3725.diff
[CMD] [2021-04-19 10:13:36] tail -n +1 /app/public/converted_csv/nilsson_nilsson2_measurements_3725.csv >> /app/public/diff/nilsson_nilsson2_measurements_3725.diff
[CMD] [2021-04-19 10:13:36] echo "." >> /app/public/diff/nilsson_nilsson2_measurements_3725.diff
[INFO] [2021-04-19 10:13:37] Created diff: /app/public/diff/nilsson_nilsson2_measurements_3725.diff (138 lines)
[STOP] [2021-04-19 10:13:37] calculate_delta
[START] [2021-04-19 10:13:37] parse_diff_and_store
[INFO] [2021-04-19 10:13:37] Handling diff: /app/public/diff/nilsson_nilsson2_refs_3725.diff (2 lines)
[INFO] [2021-04-19 10:13:37] Loading refs diff file into memory (2 /app/public/diff/nilsson_nilsson2_refs_3725.diff lines)...
[INFO] [2021-04-19 10:13:37] Handling diff: /app/public/diff/nilsson_nilsson2_nodes_3725.diff (48 lines)
[INFO] [2021-04-19 10:13:38] Loading nodes diff file into memory (48 /app/public/diff/nilsson_nilsson2_nodes_3725.diff lines)...
[INFO] [2021-04-19 10:13:38] Handling diff: /app/public/diff/nilsson_nilsson2_occurrences_3725.diff (49 lines)
[INFO] [2021-04-19 10:13:39] Loading occurrences diff file into memory (49 /app/public/diff/nilsson_nilsson2_occurrences_3725.diff lines)...
[INFO] [2021-04-19 10:13:39] Handling diff: /app/public/diff/nilsson_nilsson2_measurements_3725.diff (138 lines)
[INFO] [2021-04-19 10:13:39] Loading measurements diff file into memory (138 /app/public/diff/nilsson_nilsson2_measurements_3725.diff lines)...
[INFO] [2021-04-19 10:13:40] Storing 46 ScientificNames
[INFO] [2021-04-19 10:13:40] Processing group of 46 in 1 groups of 1000
[INFO] [2021-04-19 10:13:40] Average Time: 0.02
[INFO] [2021-04-19 10:13:40] Total Time: 1s
[INFO] [2021-04-19 10:13:40] Storing 46 Nodes
[INFO] [2021-04-19 10:13:40] Processing group of 46 in 1 groups of 1000
[INFO] [2021-04-19 10:13:40] Average Time: 0.02
[INFO] [2021-04-19 10:13:40] Total Time: 1s
[INFO] [2021-04-19 10:13:40] Storing 47 Occurrences
[INFO] [2021-04-19 10:13:40] Processing group of 47 in 1 groups of 1000
[INFO] [2021-04-19 10:13:40] Average Time: 0.01
[INFO] [2021-04-19 10:13:40] Total Time: 1s
[INFO] [2021-04-19 10:13:40] Storing 4 OccurrenceMetadata
[INFO] [2021-04-19 10:13:40] Processing group of 4 in 1 groups of 1000
[INFO] [2021-04-19 10:13:40] Average Time: 0.0
[INFO] [2021-04-19 10:13:40] Total Time: 1s
[INFO] [2021-04-19 10:13:40] Storing 136 Traits
[INFO] [2021-04-19 10:13:40] Processing group of 136 in 1 groups of 1000
[INFO] [2021-04-19 10:13:40] Average Time: 0.04
[INFO] [2021-04-19 10:13:40] Total Time: 1s
[INFO] [2021-04-19 10:13:40] Storing 47 MetaTraits
[INFO] [2021-04-19 10:13:40] Processing group of 47 in 1 groups of 1000
[INFO] [2021-04-19 10:13:40] Average Time: 0.01
[INFO] [2021-04-19 10:13:40] Total Time: 1s
[STOP] [2021-04-19 10:13:40] parse_diff_and_store
[START] [2021-04-19 10:13:40] resolve_keys
[INFO] [2021-04-19 10:13:46] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-19 10:13:46] traits to occurrences...
[INFO] [2021-04-19 10:13:46] traits to nodes (through occurrences)...
[INFO] [2021-04-19 10:13:46] Traits to sex term...
[INFO] [2021-04-19 10:13:46] Traits to lifestage term...
[INFO] [2021-04-19 10:13:46] MetaTraits to traits...
[INFO] [2021-04-19 10:13:46] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-19 10:13:46] Assocs to occurrences...
[INFO] [2021-04-19 10:13:46] Assocs to nodes...
[INFO] [2021-04-19 10:13:46] Assoc to sex term...
[INFO] [2021-04-19 10:13:46] Assoc to lifestage term...
[INFO] [2021-04-19 10:13:46] MetaAssoc to assocs...
[STOP] [2021-04-19 10:13:46] resolve_keys
[START] [2021-04-19 10:13:46] hold_for_later_1
[STOP] [2021-04-19 10:13:46] hold_for_later_1
[START] [2021-04-19 10:13:46] hold_for_later_2
[STOP] [2021-04-19 10:13:46] hold_for_later_2
[START] [2021-04-19 10:13:46] resolve_missing_parents
[STOP] [2021-04-19 10:13:46] resolve_missing_parents
[START] [2021-04-19 10:13:46] rebuild_nodes
[START] [2021-04-19 10:13:46] Flattener#flatten
[START] [2021-04-19 10:13:46] Flattener#study_resource
[START] [2021-04-19 10:13:46] Flattener#build_ancestry
[STOP] [2021-04-19 10:13:46] Flattener#build_ancestry
[INFO] [2021-04-19 10:13:46] 46 ancestry keys
[START] [2021-04-19 10:13:46] build_node_ancestors
[INFO] [2021-04-19 10:13:46] old ancestors deleted.
[STOP] [2021-04-19 10:13:46] build_node_ancestors
[WARN] [2021-04-19 10:13:46] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2021-04-19 10:13:46] Flattener#flatten
[STOP] [2021-04-19 10:13:46] rebuild_nodes
[START] [2021-04-19 10:13:46] resolve_missing_media_owners
[STOP] [2021-04-19 10:13:46] resolve_missing_media_owners
[START] [2021-04-19 10:13:46] sanitize_media_verbatims
[STOP] [2021-04-19 10:13:46] sanitize_media_verbatims
[START] [2021-04-19 10:13:46] queue_downloads
[STOP] [2021-04-19 10:13:46] queue_downloads
[START] [2021-04-19 10:13:46] parse_names
[WARN] [2021-04-19 10:13:46] I see 46 names which still need to be parsed.
[WARN] [2021-04-19 10:13:47] I see 1 names which still need to be parsed.
[STOP] [2021-04-19 10:13:48] parse_names
[START] [2021-04-19 10:13:48] denormalize_canonical_names_to_nodes
[STOP] [2021-04-19 10:13:48] denormalize_canonical_names_to_nodes
[START] [2021-04-19 10:13:48] match_nodes
[START] [2021-04-19 10:13:48] map_all_nodes_to_pages
[STOP] [2021-04-19 10:13:49] map_all_nodes_to_pages
[INFO] [2021-04-19 10:13:49] ZERO unmatched nodes (of 46)! Nicely done.
[START] [2021-04-19 10:13:49] update_nodes
[STOP] [2021-04-19 10:13:49] update_nodes
[STOP] [2021-04-19 10:13:49] match_nodes
[START] [2021-04-19 10:13:49] reindex_search
[STOP] [2021-04-19 10:13:49] reindex_search
[START] [2021-04-19 10:13:49] normalize_units
[STOP] [2021-04-19 10:13:49] normalize_units
[START] [2021-04-19 10:13:49] calculate_statistics
[2021-04-19 10:13:49] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2021-04-19 10:13:49] calculate_statistics
[START] [2021-04-19 10:13:49] complete_harvest_instance
[START] [2021-04-19 10:13:49] overall_tsv_creation
[INFO] [2021-04-19 10:13:49] Processing group of 46 in 1 batches of 10000
[INFO] [2021-04-19 10:14:26] 47 Traits (unfiltered)...
[INFO] [2021-04-19 10:15:00] 47 Traits (filtered)...
[INFO] [2021-04-19 10:15:00] 0 Associations (filtered)...
[INFO] [2021-04-19 10:15:00] 89 metadata added.
[INFO] [2021-04-19 10:15:00] 0 metadata added.
[INFO] [2021-04-19 10:15:26] Average Time: 72.89
[INFO] [2021-04-19 10:15:26] Total Time: 1m38s
[STOP] [2021-04-19 10:15:26] overall_tsv_creation
[INFO] [2021-04-19 10:15:26] Done. Check your files:
[INFO] [2021-04-19 10:15:27] (45 lines) /app/public/data/nilsson_nilsson2/publish_nodes.tsv
[INFO] [2021-04-19 10:15:27] (46 lines) /app/public/data/nilsson_nilsson2/publish_scientific_names.tsv
[INFO] [2021-04-19 10:15:27] (48 lines) /app/public/data/nilsson_nilsson2/publish_traits.tsv
[INFO] [2021-04-19 10:15:28] (90 lines) /app/public/data/nilsson_nilsson2/publish_metadata.tsv
[STOP] [2021-04-19 10:15:28] complete_harvest_instance
[START] [2021-04-19 10:15:28] completed
[STOP] [2021-04-19 10:15:28] completed
[STOP] [2021-04-19 10:15:28] logged process, took 117.81
[INFO] [2021-04-19 15:36:24] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-19 15:47:37] ## remove_type: ScientificName
[INFO] [2021-04-19 15:47:37] ++ Calling delete_all on 46 instances...
[INFO] [2021-04-19 15:47:37] [15:47:37.550] Removed 46 Scientificnames
[INFO] [2021-04-19 15:47:37] ## remove_type: Vernacular
[INFO] [2021-04-19 15:47:37] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:47:37] [15:47:37.551] Removed 0 Vernaculars
[INFO] [2021-04-19 15:47:37] ## remove_type: Article
[INFO] [2021-04-19 15:47:37] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:47:37] [15:47:37.553] Removed 0 Articles
[INFO] [2021-04-19 15:47:37] ## remove_type: Medium
[INFO] [2021-04-19 15:47:37] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:47:37] [15:47:37.554] Removed 0 Media
[INFO] [2021-04-19 15:47:37] ## remove_type: Trait
[INFO] [2021-04-19 15:47:37] ++ Calling delete_all on 136 instances...
[INFO] [2021-04-19 15:47:37] [15:47:37.564] Removed 136 Traits
[INFO] [2021-04-19 15:47:37] ## remove_type: MetaTrait
[INFO] [2021-04-19 15:47:37] ++ Calling delete_all on 47 instances...
[INFO] [2021-04-19 15:47:37] [15:47:37.568] Removed 47 Metatraits
[INFO] [2021-04-19 15:47:37] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-19 15:47:37] ++ Calling delete_all on 4 instances...
[INFO] [2021-04-19 15:47:37] [15:47:37.570] Removed 4 Occurrencemetadata
[INFO] [2021-04-19 15:47:37] ## remove_type: Assoc
[INFO] [2021-04-19 15:47:37] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:47:37] [15:47:37.571] Removed 0 Assocs
[INFO] [2021-04-19 15:47:37] ## remove_type: MetaAssoc
[INFO] [2021-04-19 15:47:37] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:47:37] [15:47:37.573] Removed 0 Metaassocs
[INFO] [2021-04-19 15:47:37] ## remove_type: Identifier
[INFO] [2021-04-19 15:47:37] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:47:37] [15:47:37.574] Removed 0 Identifiers
[INFO] [2021-04-19 15:47:37] ## remove_type: Reference
[INFO] [2021-04-19 15:47:37] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:47:37] [15:47:37.575] Removed 0 References
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:37] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:38] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:38] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:38] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:38] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:38] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:38] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:38] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:38] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:38] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:38] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:38] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:38] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:38] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:38] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:38] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:38] Starting batch with ID 92872677...
[INFO] [2021-04-19 15:47:38] ## remove_type: Node
[INFO] [2021-04-19 15:47:38] ++ Calling delete_all on 46 instances...
[INFO] [2021-04-19 15:47:38] [15:47:38.180] Removed 46 Nodes
[START] [2021-04-19 15:47:38] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 15:47:38] Creating resource from OpenData
[START] [2021-04-19 15:47:38] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 15:47:38] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 15:47:38] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 15:47:38] Creating resource from OpenData
[START] [2021-04-19 15:47:38] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 15:47:38] create_harvest_instance
[INFO] [2021-04-19 15:47:38] Created harvest instance #3778
[STOP] [2021-04-19 15:47:38] create_harvest_instance
[START] [2021-04-19 15:47:38] fetch_files
[STOP] [2021-04-19 15:47:38] fetch_files
[START] [2021-04-19 15:47:38] validate_each_file
[INFO] [2021-04-19 15:47:38] Looping over 4 formats...
[INFO] [2021-04-19 15:47:38] ...refs (/app/public/data/nilsson_nilsson2/references.tsv)
[INFO] [2021-04-19 15:47:38] Valid: /app/public/converted_csv/nilsson_nilsson2_refs_3778.csv (0 lines)
[INFO] [2021-04-19 15:47:38] ...nodes (/app/public/data/nilsson_nilsson2/taxa.txt)
[INFO] [2021-04-19 15:47:38] Valid: /app/public/converted_csv/nilsson_nilsson2_nodes_3778.csv (46 lines)
[INFO] [2021-04-19 15:47:38] ...occurrences (/app/public/data/nilsson_nilsson2/occurrences.txt)
[INFO] [2021-04-19 15:47:38] Valid: /app/public/converted_csv/nilsson_nilsson2_occurrences_3778.csv (47 lines)
[INFO] [2021-04-19 15:47:38] ...measurements (/app/public/data/nilsson_nilsson2/measurementsorfacts.txt)
[INFO] [2021-04-19 15:47:38] Valid: /app/public/converted_csv/nilsson_nilsson2_measurements_3778.csv (136 lines)
[STOP] [2021-04-19 15:47:38] validate_each_file
[START] [2021-04-19 15:47:38] convert_to_csv
[INFO] [2021-04-19 15:47:38] Looping over 4 formats...
[INFO] [2021-04-19 15:47:38] ...refs (/app/public/data/nilsson_nilsson2/references.tsv)
[CMD] [2021-04-19 15:47:38] /usr/bin/sort /app/public/converted_csv/nilsson_nilsson2_refs_3778.csv > /app/public/converted_csv/nilsson_nilsson2_refs_3778.csv_sorted
[INFO] [2021-04-19 15:47:38] Converted: /app/public/converted_csv/nilsson_nilsson2_refs_3778.csv (0 lines)
[INFO] [2021-04-19 15:47:38] ...nodes (/app/public/data/nilsson_nilsson2/taxa.txt)
[CMD] [2021-04-19 15:47:38] /usr/bin/sort /app/public/converted_csv/nilsson_nilsson2_nodes_3778.csv > /app/public/converted_csv/nilsson_nilsson2_nodes_3778.csv_sorted
[INFO] [2021-04-19 15:47:38] Converted: /app/public/converted_csv/nilsson_nilsson2_nodes_3778.csv (46 lines)
[INFO] [2021-04-19 15:47:38] ...occurrences (/app/public/data/nilsson_nilsson2/occurrences.txt)
[CMD] [2021-04-19 15:47:38] /usr/bin/sort /app/public/converted_csv/nilsson_nilsson2_occurrences_3778.csv > /app/public/converted_csv/nilsson_nilsson2_occurrences_3778.csv_sorted
[INFO] [2021-04-19 15:47:38] Converted: /app/public/converted_csv/nilsson_nilsson2_occurrences_3778.csv (47 lines)
[INFO] [2021-04-19 15:47:38] ...measurements (/app/public/data/nilsson_nilsson2/measurementsorfacts.txt)
[CMD] [2021-04-19 15:47:38] /usr/bin/sort /app/public/converted_csv/nilsson_nilsson2_measurements_3778.csv > /app/public/converted_csv/nilsson_nilsson2_measurements_3778.csv_sorted
[INFO] [2021-04-19 15:47:38] Converted: /app/public/converted_csv/nilsson_nilsson2_measurements_3778.csv (136 lines)
[STOP] [2021-04-19 15:47:38] convert_to_csv
[START] [2021-04-19 15:47:38] calculate_delta
[INFO] [2021-04-19 15:47:38] Looping over 4 formats...
[INFO] [2021-04-19 15:47:38] ...refs (/app/public/data/nilsson_nilsson2/references.tsv)
[CMD] [2021-04-19 15:47:38] echo "0a" > /app/public/diff/nilsson_nilsson2_refs_3778.diff
[CMD] [2021-04-19 15:47:38] tail -n +1 /app/public/converted_csv/nilsson_nilsson2_refs_3778.csv >> /app/public/diff/nilsson_nilsson2_refs_3778.diff
[CMD] [2021-04-19 15:47:38] echo "." >> /app/public/diff/nilsson_nilsson2_refs_3778.diff
[INFO] [2021-04-19 15:47:38] Created diff: /app/public/diff/nilsson_nilsson2_refs_3778.diff (2 lines)
[INFO] [2021-04-19 15:47:38] ...nodes (/app/public/data/nilsson_nilsson2/taxa.txt)
[CMD] [2021-04-19 15:47:38] echo "0a" > /app/public/diff/nilsson_nilsson2_nodes_3778.diff
[CMD] [2021-04-19 15:47:38] tail -n +1 /app/public/converted_csv/nilsson_nilsson2_nodes_3778.csv >> /app/public/diff/nilsson_nilsson2_nodes_3778.diff
[CMD] [2021-04-19 15:47:38] echo "." >> /app/public/diff/nilsson_nilsson2_nodes_3778.diff
[INFO] [2021-04-19 15:47:38] Created diff: /app/public/diff/nilsson_nilsson2_nodes_3778.diff (48 lines)
[INFO] [2021-04-19 15:47:38] ...occurrences (/app/public/data/nilsson_nilsson2/occurrences.txt)
[CMD] [2021-04-19 15:47:38] echo "0a" > /app/public/diff/nilsson_nilsson2_occurrences_3778.diff
[CMD] [2021-04-19 15:47:38] tail -n +1 /app/public/converted_csv/nilsson_nilsson2_occurrences_3778.csv >> /app/public/diff/nilsson_nilsson2_occurrences_3778.diff
[CMD] [2021-04-19 15:47:38] echo "." >> /app/public/diff/nilsson_nilsson2_occurrences_3778.diff
[INFO] [2021-04-19 15:47:38] Created diff: /app/public/diff/nilsson_nilsson2_occurrences_3778.diff (49 lines)
[INFO] [2021-04-19 15:47:38] ...measurements (/app/public/data/nilsson_nilsson2/measurementsorfacts.txt)
[CMD] [2021-04-19 15:47:38] echo "0a" > /app/public/diff/nilsson_nilsson2_measurements_3778.diff
[CMD] [2021-04-19 15:47:38] tail -n +1 /app/public/converted_csv/nilsson_nilsson2_measurements_3778.csv >> /app/public/diff/nilsson_nilsson2_measurements_3778.diff
[CMD] [2021-04-19 15:47:38] echo "." >> /app/public/diff/nilsson_nilsson2_measurements_3778.diff
[INFO] [2021-04-19 15:47:38] Created diff: /app/public/diff/nilsson_nilsson2_measurements_3778.diff (138 lines)
[STOP] [2021-04-19 15:47:38] calculate_delta
[START] [2021-04-19 15:47:38] parse_diff_and_store
[INFO] [2021-04-19 15:47:38] Handling diff: /app/public/diff/nilsson_nilsson2_refs_3778.diff (2 lines)
[INFO] [2021-04-19 15:47:38] Loading refs diff file into memory (2 /app/public/diff/nilsson_nilsson2_refs_3778.diff lines)...
[INFO] [2021-04-19 15:47:38] Handling diff: /app/public/diff/nilsson_nilsson2_nodes_3778.diff (48 lines)
[INFO] [2021-04-19 15:47:38] Loading nodes diff file into memory (48 /app/public/diff/nilsson_nilsson2_nodes_3778.diff lines)...
[INFO] [2021-04-19 15:47:38] Handling diff: /app/public/diff/nilsson_nilsson2_occurrences_3778.diff (49 lines)
[INFO] [2021-04-19 15:47:38] Loading occurrences diff file into memory (49 /app/public/diff/nilsson_nilsson2_occurrences_3778.diff lines)...
[INFO] [2021-04-19 15:47:38] Handling diff: /app/public/diff/nilsson_nilsson2_measurements_3778.diff (138 lines)
[INFO] [2021-04-19 15:47:38] Loading measurements diff file into memory (138 /app/public/diff/nilsson_nilsson2_measurements_3778.diff lines)...
[INFO] [2021-04-19 15:47:38] Storing 46 ScientificNames
[INFO] [2021-04-19 15:47:38] Processing group of 46 in 1 groups of 1000
[INFO] [2021-04-19 15:47:39] Average Time: 0.01
[INFO] [2021-04-19 15:47:39] Total Time: 1s
[INFO] [2021-04-19 15:47:39] Storing 46 Nodes
[INFO] [2021-04-19 15:47:39] Processing group of 46 in 1 groups of 1000
[INFO] [2021-04-19 15:47:39] Average Time: 0.01
[INFO] [2021-04-19 15:47:39] Total Time: 1s
[INFO] [2021-04-19 15:47:39] Storing 47 Occurrences
[INFO] [2021-04-19 15:47:39] Processing group of 47 in 1 groups of 1000
[INFO] [2021-04-19 15:47:39] Average Time: 0.01
[INFO] [2021-04-19 15:47:39] Total Time: 1s
[INFO] [2021-04-19 15:47:39] Storing 4 OccurrenceMetadata
[INFO] [2021-04-19 15:47:39] Processing group of 4 in 1 groups of 1000
[INFO] [2021-04-19 15:47:39] Average Time: 0.0
[INFO] [2021-04-19 15:47:39] Total Time: 1s
[INFO] [2021-04-19 15:47:39] Storing 136 Traits
[INFO] [2021-04-19 15:47:39] Processing group of 136 in 1 groups of 1000
[INFO] [2021-04-19 15:47:39] Average Time: 0.05
[INFO] [2021-04-19 15:47:39] Total Time: 1s
[INFO] [2021-04-19 15:47:39] Storing 47 MetaTraits
[INFO] [2021-04-19 15:47:39] Processing group of 47 in 1 groups of 1000
[INFO] [2021-04-19 15:47:39] Average Time: 0.01
[INFO] [2021-04-19 15:47:39] Total Time: 1s
[STOP] [2021-04-19 15:47:39] parse_diff_and_store
[START] [2021-04-19 15:47:39] resolve_keys
[INFO] [2021-04-19 15:47:45] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-19 15:47:45] traits to occurrences...
[INFO] [2021-04-19 15:47:45] traits to nodes (through occurrences)...
[INFO] [2021-04-19 15:47:45] Traits to sex term...
[INFO] [2021-04-19 15:47:45] Traits to lifestage term...
[INFO] [2021-04-19 15:47:45] MetaTraits to traits...
[INFO] [2021-04-19 15:47:45] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-19 15:47:45] Assocs to occurrences...
[INFO] [2021-04-19 15:47:45] Assocs to nodes...
[INFO] [2021-04-19 15:47:45] Assoc to sex term...
[INFO] [2021-04-19 15:47:45] Assoc to lifestage term...
[INFO] [2021-04-19 15:47:45] MetaAssoc to assocs...
[STOP] [2021-04-19 15:47:45] resolve_keys
[START] [2021-04-19 15:47:45] hold_for_later_1
[STOP] [2021-04-19 15:47:45] hold_for_later_1
[START] [2021-04-19 15:47:45] hold_for_later_2
[STOP] [2021-04-19 15:47:45] hold_for_later_2
[START] [2021-04-19 15:47:45] resolve_missing_parents
[STOP] [2021-04-19 15:47:45] resolve_missing_parents
[START] [2021-04-19 15:47:45] rebuild_nodes
[START] [2021-04-19 15:47:45] Flattener#flatten
[START] [2021-04-19 15:47:45] Flattener#study_resource
[START] [2021-04-19 15:47:45] Flattener#build_ancestry
[STOP] [2021-04-19 15:47:45] Flattener#build_ancestry
[INFO] [2021-04-19 15:47:45] 46 ancestry keys
[START] [2021-04-19 15:47:45] build_node_ancestors
[INFO] [2021-04-19 15:47:45] old ancestors deleted.
[STOP] [2021-04-19 15:47:45] build_node_ancestors
[WARN] [2021-04-19 15:47:45] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2021-04-19 15:47:45] Flattener#flatten
[STOP] [2021-04-19 15:47:45] rebuild_nodes
[START] [2021-04-19 15:47:45] resolve_missing_media_owners
[STOP] [2021-04-19 15:47:45] resolve_missing_media_owners
[START] [2021-04-19 15:47:45] sanitize_media_verbatims
[STOP] [2021-04-19 15:47:45] sanitize_media_verbatims
[START] [2021-04-19 15:47:45] queue_downloads
[STOP] [2021-04-19 15:47:45] queue_downloads
[START] [2021-04-19 15:47:45] parse_names
[WARN] [2021-04-19 15:47:45] I see 46 names which still need to be parsed.
[WARN] [2021-04-19 15:47:46] I see 1 names which still need to be parsed.
[STOP] [2021-04-19 15:47:47] parse_names
[START] [2021-04-19 15:47:47] denormalize_canonical_names_to_nodes
[STOP] [2021-04-19 15:47:47] denormalize_canonical_names_to_nodes
[START] [2021-04-19 15:47:47] match_nodes
[START] [2021-04-19 15:47:47] map_all_nodes_to_pages
[STOP] [2021-04-19 15:47:47] map_all_nodes_to_pages
[INFO] [2021-04-19 15:47:47] ZERO unmatched nodes (of 46)! Nicely done.
[START] [2021-04-19 15:47:47] update_nodes
[STOP] [2021-04-19 15:47:47] update_nodes
[STOP] [2021-04-19 15:47:47] match_nodes
[START] [2021-04-19 15:47:47] reindex_search
[STOP] [2021-04-19 15:47:47] reindex_search
[START] [2021-04-19 15:47:47] normalize_units
[STOP] [2021-04-19 15:47:47] normalize_units
[START] [2021-04-19 15:47:47] calculate_statistics
[2021-04-19 15:47:47] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2021-04-19 15:47:47] calculate_statistics
[START] [2021-04-19 15:47:47] complete_harvest_instance
[START] [2021-04-19 15:47:47] overall_tsv_creation
[INFO] [2021-04-19 15:47:47] Processing group of 46 in 1 batches of 10000
[INFO] [2021-04-19 15:48:23] 47 Traits (unfiltered)...
[INFO] [2021-04-19 15:48:57] 47 Traits (filtered)...
[INFO] [2021-04-19 15:48:57] 0 Associations (filtered)...
[INFO] [2021-04-19 15:48:57] 89 metadata added.
[INFO] [2021-04-19 15:48:57] 0 metadata added.
[INFO] [2021-04-19 15:49:24] Average Time: 72.6
[INFO] [2021-04-19 15:49:24] Total Time: 1m37s
[STOP] [2021-04-19 15:49:24] overall_tsv_creation
[INFO] [2021-04-19 15:49:24] Done. Check your files:
[INFO] [2021-04-19 15:49:24] (45 lines) /app/public/data/nilsson_nilsson2/publish_nodes.tsv
[INFO] [2021-04-19 15:49:24] (46 lines) /app/public/data/nilsson_nilsson2/publish_scientific_names.tsv
[INFO] [2021-04-19 15:49:24] (48 lines) /app/public/data/nilsson_nilsson2/publish_traits.tsv
[INFO] [2021-04-19 15:49:24] (90 lines) /app/public/data/nilsson_nilsson2/publish_metadata.tsv
[STOP] [2021-04-19 15:49:24] complete_harvest_instance
[START] [2021-04-19 15:49:24] completed
[STOP] [2021-04-19 15:49:24] completed
[STOP] [2021-04-19 15:49:24] logged process, took 105.63
Latest Process