Harvest for Higgins 1960 Created 19 Apr 16:06

Harvesting Log

# Logfile created on 2020-08-27 15:52:01 -0400 by logger.rb/v1.4.2
[START] [2020-08-27 15:52:01] logged process
[START] [2020-08-27 15:52:01] Creating resource from OpenData
[START] [2020-08-27 15:52:02] logged process
[START] [2020-08-27 15:52:02] Parse meta.xml file and create formats with fields
[STOP] [2020-08-27 15:52:02] Parse meta.xml file and create formats with fields
[STOP] [2020-08-27 15:52:02] Creating resource from OpenData
[INFO] [2020-12-03 12:29:10] ## HARVEST: type = -harvest
[START] [2020-12-03 12:29:10] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-12-03 12:29:10] create_harvest_instance
[STOP] [2020-12-03 12:29:12] create_harvest_instance
[START] [2020-12-03 12:29:12] fetch_files
[STOP] [2020-12-03 12:29:12] fetch_files
[START] [2020-12-03 12:29:12] validate_each_file
[STOP] [2020-12-03 12:29:12] validate_each_file
[START] [2020-12-03 12:29:12] convert_to_csv
[CMD] [2020-12-03 12:29:12] /usr/bin/sort /app/public/converted_csv/higgins_higgins__refs_24912.csv > /app/public/converted_csv/higgins_higgins__refs_24912.csv_sorted
[CMD] [2020-12-03 12:29:12] /usr/bin/sort /app/public/converted_csv/higgins_higgins__nodes_24913.csv > /app/public/converted_csv/higgins_higgins__nodes_24913.csv_sorted
[CMD] [2020-12-03 12:29:12] /usr/bin/sort /app/public/converted_csv/higgins_higgins__occurrences_24914.csv > /app/public/converted_csv/higgins_higgins__occurrences_24914.csv_sorted
[CMD] [2020-12-03 12:29:12] /usr/bin/sort /app/public/converted_csv/higgins_higgins__measurements_24915.csv > /app/public/converted_csv/higgins_higgins__measurements_24915.csv_sorted
[STOP] [2020-12-03 12:29:12] convert_to_csv
[START] [2020-12-03 12:29:12] calculate_delta
[CMD] [2020-12-03 12:29:12] echo "0a" > /app/public/diff/higgins_higgins__refs_24912.diff
[CMD] [2020-12-03 12:29:12] tail -n +1 /app/public/converted_csv/higgins_higgins__refs_24912.csv >> /app/public/diff/higgins_higgins__refs_24912.diff
[CMD] [2020-12-03 12:29:12] echo "." >> /app/public/diff/higgins_higgins__refs_24912.diff
[CMD] [2020-12-03 12:29:12] echo "0a" > /app/public/diff/higgins_higgins__nodes_24913.diff
[CMD] [2020-12-03 12:29:12] tail -n +1 /app/public/converted_csv/higgins_higgins__nodes_24913.csv >> /app/public/diff/higgins_higgins__nodes_24913.diff
[CMD] [2020-12-03 12:29:12] echo "." >> /app/public/diff/higgins_higgins__nodes_24913.diff
[CMD] [2020-12-03 12:29:12] echo "0a" > /app/public/diff/higgins_higgins__occurrences_24914.diff
[CMD] [2020-12-03 12:29:12] tail -n +1 /app/public/converted_csv/higgins_higgins__occurrences_24914.csv >> /app/public/diff/higgins_higgins__occurrences_24914.diff
[CMD] [2020-12-03 12:29:12] echo "." >> /app/public/diff/higgins_higgins__occurrences_24914.diff
[CMD] [2020-12-03 12:29:12] echo "0a" > /app/public/diff/higgins_higgins__measurements_24915.diff
[CMD] [2020-12-03 12:29:12] tail -n +1 /app/public/converted_csv/higgins_higgins__measurements_24915.csv >> /app/public/diff/higgins_higgins__measurements_24915.diff
[CMD] [2020-12-03 12:29:12] echo "." >> /app/public/diff/higgins_higgins__measurements_24915.diff
[STOP] [2020-12-03 12:29:12] calculate_delta
[START] [2020-12-03 12:29:12] parse_diff_and_store
[INFO] [2020-12-03 12:29:12] Loading refs diff file into memory (true lines)...
[INFO] [2020-12-03 12:29:12] Loading nodes diff file into memory (true lines)...
[INFO] [2020-12-03 12:29:12] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-12-03 12:29:12] Loading measurements diff file into memory (true lines)...
[INFO] [2020-12-03 12:29:13] Storing 1 ScientificNames
[INFO] [2020-12-03 12:29:13] Processing group of 1 in 1 groups of 1000
[INFO] [2020-12-03 12:29:13] Average Time: 0.01
[INFO] [2020-12-03 12:29:13] Total Time: 1s
[INFO] [2020-12-03 12:29:13] Storing 1 Nodes
[INFO] [2020-12-03 12:29:13] Processing group of 1 in 1 groups of 1000
[INFO] [2020-12-03 12:29:13] Average Time: 0.0
[INFO] [2020-12-03 12:29:13] Total Time: 1s
[INFO] [2020-12-03 12:29:13] Storing 1 Occurrences
[INFO] [2020-12-03 12:29:13] Processing group of 1 in 1 groups of 1000
[INFO] [2020-12-03 12:29:13] Average Time: 0.0
[INFO] [2020-12-03 12:29:13] Total Time: 1s
[INFO] [2020-12-03 12:29:13] Storing 5 Traits
[INFO] [2020-12-03 12:29:13] Processing group of 5 in 1 groups of 1000
[INFO] [2020-12-03 12:29:13] Average Time: 0.0
[INFO] [2020-12-03 12:29:13] Total Time: 1s
[INFO] [2020-12-03 12:29:13] Storing 1 MetaTraits
[INFO] [2020-12-03 12:29:13] Processing group of 1 in 1 groups of 1000
[INFO] [2020-12-03 12:29:13] Average Time: 0.0
[INFO] [2020-12-03 12:29:13] Total Time: 1s
[STOP] [2020-12-03 12:29:13] parse_diff_and_store
[START] [2020-12-03 12:29:13] resolve_keys
[INFO] [2020-12-03 12:29:19] Occurrences to nodes (through scientific_names)...
[INFO] [2020-12-03 12:29:19] traits to occurrences...
[INFO] [2020-12-03 12:29:19] traits to nodes (through occurrences)...
[INFO] [2020-12-03 12:29:19] Traits to sex term...
[INFO] [2020-12-03 12:29:19] Traits to lifestage term...
[INFO] [2020-12-03 12:29:19] MetaTraits to traits...
[INFO] [2020-12-03 12:29:19] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-12-03 12:29:19] Assocs to occurrences...
[INFO] [2020-12-03 12:29:19] Assocs to nodes...
[INFO] [2020-12-03 12:29:19] Assoc to sex term...
[INFO] [2020-12-03 12:29:19] Assoc to lifestage term...
[INFO] [2020-12-03 12:29:19] MetaAssoc to assocs...
[STOP] [2020-12-03 12:29:19] resolve_keys
[START] [2020-12-03 12:29:19] hold_for_later_1
[STOP] [2020-12-03 12:29:19] hold_for_later_1
[START] [2020-12-03 12:29:19] hold_for_later_2
[STOP] [2020-12-03 12:29:19] hold_for_later_2
[START] [2020-12-03 12:29:19] resolve_missing_parents
[STOP] [2020-12-03 12:29:19] resolve_missing_parents
[START] [2020-12-03 12:29:19] rebuild_nodes
[START] [2020-12-03 12:29:19] Flattener#flatten
[START] [2020-12-03 12:29:19] Flattener#study_resource
[START] [2020-12-03 12:29:19] Flattener#build_ancestry
[STOP] [2020-12-03 12:29:19] Flattener#build_ancestry
[INFO] [2020-12-03 12:29:19] 1 ancestry keys
[START] [2020-12-03 12:29:19] build_node_ancestors
[INFO] [2020-12-03 12:29:19] old ancestors deleted.
[STOP] [2020-12-03 12:29:19] build_node_ancestors
[WARN] [2020-12-03 12:29:19] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-12-03 12:29:19] Flattener#flatten
[STOP] [2020-12-03 12:29:19] rebuild_nodes
[START] [2020-12-03 12:29:19] resolve_missing_media_owners
[STOP] [2020-12-03 12:29:19] resolve_missing_media_owners
[START] [2020-12-03 12:29:19] sanitize_media_verbatims
[STOP] [2020-12-03 12:29:19] sanitize_media_verbatims
[START] [2020-12-03 12:29:19] queue_downloads
[STOP] [2020-12-03 12:29:19] queue_downloads
[START] [2020-12-03 12:29:19] parse_names
[WARN] [2020-12-03 12:29:19] I see 1 names which still need to be parsed.
[STOP] [2020-12-03 12:29:20] parse_names
[START] [2020-12-03 12:29:20] denormalize_canonical_names_to_nodes
[STOP] [2020-12-03 12:29:20] denormalize_canonical_names_to_nodes
[START] [2020-12-03 12:29:20] match_nodes
[START] [2020-12-03 12:29:20] map_all_nodes_to_pages
[STOP] [2020-12-03 12:29:20] map_all_nodes_to_pages
[INFO] [2020-12-03 12:29:20] ZERO unmatched nodes (of 1)! Nicely done.
[START] [2020-12-03 12:29:20] update_nodes
[STOP] [2020-12-03 12:29:20] update_nodes
[STOP] [2020-12-03 12:29:20] match_nodes
[START] [2020-12-03 12:29:20] reindex_search
[STOP] [2020-12-03 12:29:20] reindex_search
[START] [2020-12-03 12:29:20] normalize_units
[STOP] [2020-12-03 12:29:20] normalize_units
[START] [2020-12-03 12:29:20] calculate_statistics
[2020-12-03 12:29:20] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-12-03 12:29:20] calculate_statistics
[START] [2020-12-03 12:29:20] complete_harvest_instance
[START] [2020-12-03 12:29:20] overall_tsv_creation
[INFO] [2020-12-03 12:29:20] Processing group of 1 in 1 batches of 10000
[INFO] [2020-12-03 12:29:58] 1 Traits (unfiltered)...
[INFO] [2020-12-03 12:30:35] 1 Traits (filtered)...
[INFO] [2020-12-03 12:30:35] 0 Associations (filtered)...
[INFO] [2020-12-03 12:30:35] 5 metadata added.
[INFO] [2020-12-03 12:30:35] 0 metadata added.
[INFO] [2020-12-03 12:30:35] Average Time: 52.11
[INFO] [2020-12-03 12:30:35] Total Time: 1m15s
[STOP] [2020-12-03 12:30:35] overall_tsv_creation
[INFO] [2020-12-03 12:30:35] Done. Check your files:
[INFO] [2020-12-03 12:30:35] (1 lines) /app/public/data/higgins_higgins_/publish_nodes.tsv
[INFO] [2020-12-03 12:30:35] (1 lines) /app/public/data/higgins_higgins_/publish_scientific_names.tsv
[INFO] [2020-12-03 12:30:35] (2 lines) /app/public/data/higgins_higgins_/publish_traits.tsv
[INFO] [2020-12-03 12:30:35] (6 lines) /app/public/data/higgins_higgins_/publish_metadata.tsv
[STOP] [2020-12-03 12:30:35] complete_harvest_instance
[START] [2020-12-03 12:30:35] completed
[STOP] [2020-12-03 12:30:35] completed
[STOP] [2020-12-03 12:30:35] logged process, took 84.4
[INFO] [2021-04-19 15:58:14] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-19 16:05:59] ## remove_type: ScientificName
[INFO] [2021-04-19 16:05:59] ++ Calling delete_all on 1 instances...
[INFO] [2021-04-19 16:05:59] [16:05:59.884] Removed 1 Scientificnames
[INFO] [2021-04-19 16:05:59] ## remove_type: Vernacular
[INFO] [2021-04-19 16:05:59] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 16:05:59] [16:05:59.885] Removed 0 Vernaculars
[INFO] [2021-04-19 16:05:59] ## remove_type: Article
[INFO] [2021-04-19 16:05:59] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 16:05:59] [16:05:59.887] Removed 0 Articles
[INFO] [2021-04-19 16:05:59] ## remove_type: Medium
[INFO] [2021-04-19 16:05:59] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 16:05:59] [16:05:59.888] Removed 0 Media
[INFO] [2021-04-19 16:05:59] ## remove_type: Trait
[INFO] [2021-04-19 16:05:59] ++ Calling delete_all on 5 instances...
[INFO] [2021-04-19 16:05:59] [16:05:59.890] Removed 5 Traits
[INFO] [2021-04-19 16:05:59] ## remove_type: MetaTrait
[INFO] [2021-04-19 16:05:59] ++ Calling delete_all on 1 instances...
[INFO] [2021-04-19 16:05:59] [16:05:59.892] Removed 1 Metatraits
[INFO] [2021-04-19 16:05:59] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-19 16:05:59] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 16:05:59] [16:05:59.894] Removed 0 Occurrencemetadata
[INFO] [2021-04-19 16:05:59] ## remove_type: Assoc
[INFO] [2021-04-19 16:05:59] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 16:05:59] [16:05:59.895] Removed 0 Assocs
[INFO] [2021-04-19 16:05:59] ## remove_type: MetaAssoc
[INFO] [2021-04-19 16:05:59] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 16:05:59] [16:05:59.896] Removed 0 Metaassocs
[INFO] [2021-04-19 16:05:59] ## remove_type: Identifier
[INFO] [2021-04-19 16:05:59] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 16:05:59] [16:05:59.898] Removed 0 Identifiers
[INFO] [2021-04-19 16:05:59] ## remove_type: Reference
[INFO] [2021-04-19 16:05:59] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 16:05:59] [16:05:59.899] Removed 0 References
[INFO] [2021-04-19 16:05:59] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:05:59] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:05:59] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:05:59] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:05:59] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:05:59] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:05:59] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:05:59] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:05:59] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:05:59] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:05:59] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:05:59] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:05:59] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:05:59] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:05:59] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] Starting batch with ID 86116271...
[INFO] [2021-04-19 16:06:00] ## remove_type: Node
[INFO] [2021-04-19 16:06:00] ++ Calling delete_all on 1 instances...
[INFO] [2021-04-19 16:06:00] [16:06:00.328] Removed 1 Nodes
[START] [2021-04-19 16:06:00] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 16:06:00] Creating resource from OpenData
[START] [2021-04-19 16:06:00] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 16:06:00] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 16:06:00] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 16:06:00] Creating resource from OpenData
[START] [2021-04-19 16:06:00] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 16:06:00] create_harvest_instance
[INFO] [2021-04-19 16:06:00] Created harvest instance #3787
[STOP] [2021-04-19 16:06:00] create_harvest_instance
[START] [2021-04-19 16:06:00] fetch_files
[STOP] [2021-04-19 16:06:00] fetch_files
[START] [2021-04-19 16:06:00] validate_each_file
[INFO] [2021-04-19 16:06:00] Looping over 4 formats...
[INFO] [2021-04-19 16:06:00] ...refs (/app/public/data/higgins_higgins_/references.tsv)
[INFO] [2021-04-19 16:06:00] Valid: /app/public/converted_csv/higgins_higgins__refs_3787.csv (0 lines)
[INFO] [2021-04-19 16:06:00] ...nodes (/app/public/data/higgins_higgins_/taxa.txt)
[INFO] [2021-04-19 16:06:00] Valid: /app/public/converted_csv/higgins_higgins__nodes_3787.csv (1 lines)
[INFO] [2021-04-19 16:06:00] ...occurrences (/app/public/data/higgins_higgins_/occurrences.txt)
[INFO] [2021-04-19 16:06:00] Valid: /app/public/converted_csv/higgins_higgins__occurrences_3787.csv (1 lines)
[INFO] [2021-04-19 16:06:00] ...measurements (/app/public/data/higgins_higgins_/measurementsorfacts.txt)
[INFO] [2021-04-19 16:06:00] Valid: /app/public/converted_csv/higgins_higgins__measurements_3787.csv (5 lines)
[STOP] [2021-04-19 16:06:00] validate_each_file
[START] [2021-04-19 16:06:00] convert_to_csv
[INFO] [2021-04-19 16:06:00] Looping over 4 formats...
[INFO] [2021-04-19 16:06:00] ...refs (/app/public/data/higgins_higgins_/references.tsv)
[CMD] [2021-04-19 16:06:00] /usr/bin/sort /app/public/converted_csv/higgins_higgins__refs_3787.csv > /app/public/converted_csv/higgins_higgins__refs_3787.csv_sorted
[INFO] [2021-04-19 16:06:00] Converted: /app/public/converted_csv/higgins_higgins__refs_3787.csv (0 lines)
[INFO] [2021-04-19 16:06:00] ...nodes (/app/public/data/higgins_higgins_/taxa.txt)
[CMD] [2021-04-19 16:06:00] /usr/bin/sort /app/public/converted_csv/higgins_higgins__nodes_3787.csv > /app/public/converted_csv/higgins_higgins__nodes_3787.csv_sorted
[INFO] [2021-04-19 16:06:00] Converted: /app/public/converted_csv/higgins_higgins__nodes_3787.csv (1 lines)
[INFO] [2021-04-19 16:06:00] ...occurrences (/app/public/data/higgins_higgins_/occurrences.txt)
[CMD] [2021-04-19 16:06:00] /usr/bin/sort /app/public/converted_csv/higgins_higgins__occurrences_3787.csv > /app/public/converted_csv/higgins_higgins__occurrences_3787.csv_sorted
[INFO] [2021-04-19 16:06:00] Converted: /app/public/converted_csv/higgins_higgins__occurrences_3787.csv (1 lines)
[INFO] [2021-04-19 16:06:00] ...measurements (/app/public/data/higgins_higgins_/measurementsorfacts.txt)
[CMD] [2021-04-19 16:06:00] /usr/bin/sort /app/public/converted_csv/higgins_higgins__measurements_3787.csv > /app/public/converted_csv/higgins_higgins__measurements_3787.csv_sorted
[INFO] [2021-04-19 16:06:00] Converted: /app/public/converted_csv/higgins_higgins__measurements_3787.csv (5 lines)
[STOP] [2021-04-19 16:06:00] convert_to_csv
[START] [2021-04-19 16:06:00] calculate_delta
[INFO] [2021-04-19 16:06:00] Looping over 4 formats...
[INFO] [2021-04-19 16:06:00] ...refs (/app/public/data/higgins_higgins_/references.tsv)
[CMD] [2021-04-19 16:06:00] echo "0a" > /app/public/diff/higgins_higgins__refs_3787.diff
[CMD] [2021-04-19 16:06:00] tail -n +1 /app/public/converted_csv/higgins_higgins__refs_3787.csv >> /app/public/diff/higgins_higgins__refs_3787.diff
[CMD] [2021-04-19 16:06:00] echo "." >> /app/public/diff/higgins_higgins__refs_3787.diff
[INFO] [2021-04-19 16:06:00] Created diff: /app/public/diff/higgins_higgins__refs_3787.diff (2 lines)
[INFO] [2021-04-19 16:06:00] ...nodes (/app/public/data/higgins_higgins_/taxa.txt)
[CMD] [2021-04-19 16:06:00] echo "0a" > /app/public/diff/higgins_higgins__nodes_3787.diff
[CMD] [2021-04-19 16:06:00] tail -n +1 /app/public/converted_csv/higgins_higgins__nodes_3787.csv >> /app/public/diff/higgins_higgins__nodes_3787.diff
[CMD] [2021-04-19 16:06:00] echo "." >> /app/public/diff/higgins_higgins__nodes_3787.diff
[INFO] [2021-04-19 16:06:00] Created diff: /app/public/diff/higgins_higgins__nodes_3787.diff (3 lines)
[INFO] [2021-04-19 16:06:00] ...occurrences (/app/public/data/higgins_higgins_/occurrences.txt)
[CMD] [2021-04-19 16:06:00] echo "0a" > /app/public/diff/higgins_higgins__occurrences_3787.diff
[CMD] [2021-04-19 16:06:00] tail -n +1 /app/public/converted_csv/higgins_higgins__occurrences_3787.csv >> /app/public/diff/higgins_higgins__occurrences_3787.diff
[CMD] [2021-04-19 16:06:00] echo "." >> /app/public/diff/higgins_higgins__occurrences_3787.diff
[INFO] [2021-04-19 16:06:00] Created diff: /app/public/diff/higgins_higgins__occurrences_3787.diff (3 lines)
[INFO] [2021-04-19 16:06:00] ...measurements (/app/public/data/higgins_higgins_/measurementsorfacts.txt)
[CMD] [2021-04-19 16:06:00] echo "0a" > /app/public/diff/higgins_higgins__measurements_3787.diff
[CMD] [2021-04-19 16:06:00] tail -n +1 /app/public/converted_csv/higgins_higgins__measurements_3787.csv >> /app/public/diff/higgins_higgins__measurements_3787.diff
[CMD] [2021-04-19 16:06:00] echo "." >> /app/public/diff/higgins_higgins__measurements_3787.diff
[INFO] [2021-04-19 16:06:00] Created diff: /app/public/diff/higgins_higgins__measurements_3787.diff (7 lines)
[STOP] [2021-04-19 16:06:00] calculate_delta
[START] [2021-04-19 16:06:00] parse_diff_and_store
[INFO] [2021-04-19 16:06:00] Handling diff: /app/public/diff/higgins_higgins__refs_3787.diff (2 lines)
[INFO] [2021-04-19 16:06:00] Loading refs diff file into memory (2 /app/public/diff/higgins_higgins__refs_3787.diff lines)...
[INFO] [2021-04-19 16:06:00] Handling diff: /app/public/diff/higgins_higgins__nodes_3787.diff (3 lines)
[INFO] [2021-04-19 16:06:00] Loading nodes diff file into memory (3 /app/public/diff/higgins_higgins__nodes_3787.diff lines)...
[INFO] [2021-04-19 16:06:00] Handling diff: /app/public/diff/higgins_higgins__occurrences_3787.diff (3 lines)
[INFO] [2021-04-19 16:06:00] Loading occurrences diff file into memory (3 /app/public/diff/higgins_higgins__occurrences_3787.diff lines)...
[INFO] [2021-04-19 16:06:01] Handling diff: /app/public/diff/higgins_higgins__measurements_3787.diff (7 lines)
[INFO] [2021-04-19 16:06:01] Loading measurements diff file into memory (7 /app/public/diff/higgins_higgins__measurements_3787.diff lines)...
[INFO] [2021-04-19 16:06:01] Storing 1 ScientificNames
[INFO] [2021-04-19 16:06:01] Processing group of 1 in 1 groups of 1000
[INFO] [2021-04-19 16:06:01] Average Time: 0.0
[INFO] [2021-04-19 16:06:01] Total Time: 1s
[INFO] [2021-04-19 16:06:01] Storing 1 Nodes
[INFO] [2021-04-19 16:06:01] Processing group of 1 in 1 groups of 1000
[INFO] [2021-04-19 16:06:01] Average Time: 0.0
[INFO] [2021-04-19 16:06:01] Total Time: 1s
[INFO] [2021-04-19 16:06:01] Storing 1 Occurrences
[INFO] [2021-04-19 16:06:01] Processing group of 1 in 1 groups of 1000
[INFO] [2021-04-19 16:06:01] Average Time: 0.0
[INFO] [2021-04-19 16:06:01] Total Time: 1s
[INFO] [2021-04-19 16:06:01] Storing 5 Traits
[INFO] [2021-04-19 16:06:01] Processing group of 5 in 1 groups of 1000
[INFO] [2021-04-19 16:06:01] Average Time: 0.0
[INFO] [2021-04-19 16:06:01] Total Time: 1s
[INFO] [2021-04-19 16:06:01] Storing 1 MetaTraits
[INFO] [2021-04-19 16:06:01] Processing group of 1 in 1 groups of 1000
[INFO] [2021-04-19 16:06:01] Average Time: 0.0
[INFO] [2021-04-19 16:06:01] Total Time: 1s
[STOP] [2021-04-19 16:06:01] parse_diff_and_store
[START] [2021-04-19 16:06:01] resolve_keys
[INFO] [2021-04-19 16:06:06] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-19 16:06:06] traits to occurrences...
[INFO] [2021-04-19 16:06:06] traits to nodes (through occurrences)...
[INFO] [2021-04-19 16:06:06] Traits to sex term...
[INFO] [2021-04-19 16:06:06] Traits to lifestage term...
[INFO] [2021-04-19 16:06:06] MetaTraits to traits...
[INFO] [2021-04-19 16:06:06] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-19 16:06:06] Assocs to occurrences...
[INFO] [2021-04-19 16:06:06] Assocs to nodes...
[INFO] [2021-04-19 16:06:06] Assoc to sex term...
[INFO] [2021-04-19 16:06:06] Assoc to lifestage term...
[INFO] [2021-04-19 16:06:06] MetaAssoc to assocs...
[STOP] [2021-04-19 16:06:06] resolve_keys
[START] [2021-04-19 16:06:06] hold_for_later_1
[STOP] [2021-04-19 16:06:06] hold_for_later_1
[START] [2021-04-19 16:06:06] hold_for_later_2
[STOP] [2021-04-19 16:06:06] hold_for_later_2
[START] [2021-04-19 16:06:06] resolve_missing_parents
[STOP] [2021-04-19 16:06:07] resolve_missing_parents
[START] [2021-04-19 16:06:07] rebuild_nodes
[START] [2021-04-19 16:06:07] Flattener#flatten
[START] [2021-04-19 16:06:07] Flattener#study_resource
[START] [2021-04-19 16:06:07] Flattener#build_ancestry
[STOP] [2021-04-19 16:06:07] Flattener#build_ancestry
[INFO] [2021-04-19 16:06:07] 1 ancestry keys
[START] [2021-04-19 16:06:07] build_node_ancestors
[INFO] [2021-04-19 16:06:07] old ancestors deleted.
[STOP] [2021-04-19 16:06:07] build_node_ancestors
[WARN] [2021-04-19 16:06:07] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2021-04-19 16:06:07] Flattener#flatten
[STOP] [2021-04-19 16:06:07] rebuild_nodes
[START] [2021-04-19 16:06:07] resolve_missing_media_owners
[STOP] [2021-04-19 16:06:07] resolve_missing_media_owners
[START] [2021-04-19 16:06:07] sanitize_media_verbatims
[STOP] [2021-04-19 16:06:07] sanitize_media_verbatims
[START] [2021-04-19 16:06:07] queue_downloads
[STOP] [2021-04-19 16:06:07] queue_downloads
[START] [2021-04-19 16:06:07] parse_names
[WARN] [2021-04-19 16:06:07] I see 1 names which still need to be parsed.
[STOP] [2021-04-19 16:06:08] parse_names
[START] [2021-04-19 16:06:08] denormalize_canonical_names_to_nodes
[STOP] [2021-04-19 16:06:08] denormalize_canonical_names_to_nodes
[START] [2021-04-19 16:06:08] match_nodes
[START] [2021-04-19 16:06:08] map_all_nodes_to_pages
[STOP] [2021-04-19 16:06:08] map_all_nodes_to_pages
[INFO] [2021-04-19 16:06:08] ZERO unmatched nodes (of 1)! Nicely done.
[START] [2021-04-19 16:06:08] update_nodes
[STOP] [2021-04-19 16:06:08] update_nodes
[STOP] [2021-04-19 16:06:08] match_nodes
[START] [2021-04-19 16:06:08] reindex_search
[STOP] [2021-04-19 16:06:08] reindex_search
[START] [2021-04-19 16:06:08] normalize_units
[STOP] [2021-04-19 16:06:08] normalize_units
[START] [2021-04-19 16:06:08] calculate_statistics
[2021-04-19 16:06:08] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2021-04-19 16:06:08] calculate_statistics
[START] [2021-04-19 16:06:08] complete_harvest_instance
[START] [2021-04-19 16:06:08] overall_tsv_creation
[INFO] [2021-04-19 16:06:08] Processing group of 1 in 1 batches of 10000
[INFO] [2021-04-19 16:06:43] 1 Traits (unfiltered)...
[INFO] [2021-04-19 16:07:15] 1 Traits (filtered)...
[INFO] [2021-04-19 16:07:15] 0 Associations (filtered)...
[INFO] [2021-04-19 16:07:15] 4 metadata added.
[INFO] [2021-04-19 16:07:15] 0 metadata added.
[INFO] [2021-04-19 16:07:41] Average Time: 69.64
[INFO] [2021-04-19 16:07:41] Total Time: 1m34s
[STOP] [2021-04-19 16:07:41] overall_tsv_creation
[INFO] [2021-04-19 16:07:41] Done. Check your files:
[INFO] [2021-04-19 16:07:41] (1 lines) /app/public/data/higgins_higgins_/publish_nodes.tsv
[INFO] [2021-04-19 16:07:41] (1 lines) /app/public/data/higgins_higgins_/publish_scientific_names.tsv
[INFO] [2021-04-19 16:07:41] (2 lines) /app/public/data/higgins_higgins_/publish_traits.tsv
[INFO] [2021-04-19 16:07:41] (5 lines) /app/public/data/higgins_higgins_/publish_metadata.tsv
[STOP] [2021-04-19 16:07:41] complete_harvest_instance
[START] [2021-04-19 16:07:41] completed
[STOP] [2021-04-19 16:07:41] completed
[STOP] [2021-04-19 16:07:41] logged process, took 101.05

Latest Process

19 Apr 16:06 EXITED currently processing 0