Harvest for Du et al, 2016 Created 19 Apr 11:29

Stage: completed
Fetched: 19 Apr 11:29
Validated: 19 Apr 11:29
Deltas Created 19 Apr 11:29
Units Normalized: 19 Apr 11:29
Ancestry Built: 19 Apr 11:29
Nodes Matched: 19 Apr 11:29
Names Parsed: 19 Apr 11:29
New Models Stored: 19 Apr 11:29
Indexed: 19 Apr 11:29
Completed: 19 Apr 11:31
Time to Harvest: less than a minute
Harvesting Log

(417 lines)
# Logfile created on 2020-12-08 15:36:25 -0500 by logger.rb/v1.4.2
[START] [2020-12-08 15:36:25] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-12-08 15:36:25] Creating resource from OpenData
[START] [2020-12-08 15:36:26] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-12-08 15:36:26] Parse meta.xml file and create formats with fields
[STOP] [2020-12-08 15:36:26] Parse meta.xml file and create formats with fields
[STOP] [2020-12-08 15:36:26] Creating resource from OpenData
[INFO] [2020-12-08 15:36:45] ## HARVEST: type = -harvest
[START] [2020-12-08 15:36:48] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-12-08 15:36:48] create_harvest_instance
[STOP] [2020-12-08 15:36:49] create_harvest_instance
[START] [2020-12-08 15:36:49] fetch_files
[STOP] [2020-12-08 15:36:49] fetch_files
[START] [2020-12-08 15:36:49] validate_each_file
[STOP] [2020-12-08 15:36:49] validate_each_file
[START] [2020-12-08 15:36:49] convert_to_csv
[CMD] [2020-12-08 15:36:49] /usr/bin/sort /app/public/converted_csv/du_et_al_du_et_a_nodes_25315.csv > /app/public/converted_csv/du_et_al_du_et_a_nodes_25315.csv_sorted
[CMD] [2020-12-08 15:36:49] /usr/bin/sort /app/public/converted_csv/du_et_al_du_et_a_occurrences_25316.csv > /app/public/converted_csv/du_et_al_du_et_a_occurrences_25316.csv_sorted
[CMD] [2020-12-08 15:36:49] /usr/bin/sort /app/public/converted_csv/du_et_al_du_et_a_measurements_25317.csv > /app/public/converted_csv/du_et_al_du_et_a_measurements_25317.csv_sorted
[STOP] [2020-12-08 15:36:49] convert_to_csv
[START] [2020-12-08 15:36:49] calculate_delta
[CMD] [2020-12-08 15:36:49] echo "0a" > /app/public/diff/du_et_al_du_et_a_nodes_25315.diff
[CMD] [2020-12-08 15:36:49] tail -n +1 /app/public/converted_csv/du_et_al_du_et_a_nodes_25315.csv >> /app/public/diff/du_et_al_du_et_a_nodes_25315.diff
[CMD] [2020-12-08 15:36:49] echo "." >> /app/public/diff/du_et_al_du_et_a_nodes_25315.diff
[CMD] [2020-12-08 15:36:49] echo "0a" > /app/public/diff/du_et_al_du_et_a_occurrences_25316.diff
[CMD] [2020-12-08 15:36:49] tail -n +1 /app/public/converted_csv/du_et_al_du_et_a_occurrences_25316.csv >> /app/public/diff/du_et_al_du_et_a_occurrences_25316.diff
[CMD] [2020-12-08 15:36:49] echo "." >> /app/public/diff/du_et_al_du_et_a_occurrences_25316.diff
[CMD] [2020-12-08 15:36:49] echo "0a" > /app/public/diff/du_et_al_du_et_a_measurements_25317.diff
[CMD] [2020-12-08 15:36:49] tail -n +1 /app/public/converted_csv/du_et_al_du_et_a_measurements_25317.csv >> /app/public/diff/du_et_al_du_et_a_measurements_25317.diff
[CMD] [2020-12-08 15:36:49] echo "." >> /app/public/diff/du_et_al_du_et_a_measurements_25317.diff
[STOP] [2020-12-08 15:36:49] calculate_delta
[START] [2020-12-08 15:36:49] parse_diff_and_store
[INFO] [2020-12-08 15:36:49] Loading nodes diff file into memory (true lines)...
[INFO] [2020-12-08 15:36:49] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-12-08 15:36:49] Loading measurements diff file into memory (true lines)...
[INFO] [2020-12-08 15:36:50] Storing 11 ScientificNames
[INFO] [2020-12-08 15:36:50] Processing group of 11 in 1 groups of 1000
[INFO] [2020-12-08 15:36:50] Average Time: 0.02
[INFO] [2020-12-08 15:36:50] Total Time: 1s
[INFO] [2020-12-08 15:36:50] Storing 11 Nodes
[INFO] [2020-12-08 15:36:50] Processing group of 11 in 1 groups of 1000
[INFO] [2020-12-08 15:36:50] Average Time: 0.01
[INFO] [2020-12-08 15:36:50] Total Time: 1s
[INFO] [2020-12-08 15:36:50] Storing 11 Occurrences
[INFO] [2020-12-08 15:36:50] Processing group of 11 in 1 groups of 1000
[INFO] [2020-12-08 15:36:50] Average Time: 0.01
[INFO] [2020-12-08 15:36:50] Total Time: 1s
[INFO] [2020-12-08 15:36:50] Storing 25 Traits
[INFO] [2020-12-08 15:36:50] Processing group of 25 in 1 groups of 1000
[INFO] [2020-12-08 15:36:50] Average Time: 0.01
[INFO] [2020-12-08 15:36:50] Total Time: 1s
[INFO] [2020-12-08 15:36:50] Storing 11 MetaTraits
[INFO] [2020-12-08 15:36:50] Processing group of 11 in 1 groups of 1000
[INFO] [2020-12-08 15:36:50] Average Time: 0.0
[INFO] [2020-12-08 15:36:50] Total Time: 1s
[STOP] [2020-12-08 15:36:50] parse_diff_and_store
[START] [2020-12-08 15:36:50] resolve_keys
[INFO] [2020-12-08 15:36:56] Occurrences to nodes (through scientific_names)...
[INFO] [2020-12-08 15:36:56] traits to occurrences...
[INFO] [2020-12-08 15:36:56] traits to nodes (through occurrences)...
[INFO] [2020-12-08 15:36:56] Traits to sex term...
[INFO] [2020-12-08 15:36:56] Traits to lifestage term...
[INFO] [2020-12-08 15:36:56] MetaTraits to traits...
[INFO] [2020-12-08 15:36:56] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-12-08 15:36:56] Assocs to occurrences...
[INFO] [2020-12-08 15:36:56] Assocs to nodes...
[INFO] [2020-12-08 15:36:56] Assoc to sex term...
[INFO] [2020-12-08 15:36:56] Assoc to lifestage term...
[INFO] [2020-12-08 15:36:56] MetaAssoc to assocs...
[STOP] [2020-12-08 15:36:56] resolve_keys
[START] [2020-12-08 15:36:56] hold_for_later_1
[STOP] [2020-12-08 15:36:56] hold_for_later_1
[START] [2020-12-08 15:36:56] hold_for_later_2
[STOP] [2020-12-08 15:36:56] hold_for_later_2
[START] [2020-12-08 15:36:56] resolve_missing_parents
[STOP] [2020-12-08 15:36:56] resolve_missing_parents
[START] [2020-12-08 15:36:56] rebuild_nodes
[START] [2020-12-08 15:36:56] Flattener#flatten
[START] [2020-12-08 15:36:56] Flattener#study_resource
[START] [2020-12-08 15:36:56] Flattener#build_ancestry
[STOP] [2020-12-08 15:36:56] Flattener#build_ancestry
[INFO] [2020-12-08 15:36:56] 11 ancestry keys
[START] [2020-12-08 15:36:56] build_node_ancestors
[INFO] [2020-12-08 15:36:56] old ancestors deleted.
[STOP] [2020-12-08 15:36:56] build_node_ancestors
[WARN] [2020-12-08 15:36:56] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-12-08 15:36:56] Flattener#flatten
[STOP] [2020-12-08 15:36:56] rebuild_nodes
[START] [2020-12-08 15:36:56] resolve_missing_media_owners
[STOP] [2020-12-08 15:36:56] resolve_missing_media_owners
[START] [2020-12-08 15:36:56] sanitize_media_verbatims
[STOP] [2020-12-08 15:36:56] sanitize_media_verbatims
[START] [2020-12-08 15:36:56] queue_downloads
[STOP] [2020-12-08 15:36:56] queue_downloads
[START] [2020-12-08 15:36:56] parse_names
[WARN] [2020-12-08 15:36:56] I see 11 names which still need to be parsed.
[STOP] [2020-12-08 15:36:57] parse_names
[START] [2020-12-08 15:36:57] denormalize_canonical_names_to_nodes
[STOP] [2020-12-08 15:36:57] denormalize_canonical_names_to_nodes
[START] [2020-12-08 15:36:57] match_nodes
[START] [2020-12-08 15:36:57] map_all_nodes_to_pages
[STOP] [2020-12-08 15:36:57] map_all_nodes_to_pages
[INFO] [2020-12-08 15:36:57] ZERO unmatched nodes (of 11)! Nicely done.
[START] [2020-12-08 15:36:57] update_nodes
[STOP] [2020-12-08 15:36:57] update_nodes
[STOP] [2020-12-08 15:36:57] match_nodes
[START] [2020-12-08 15:36:57] reindex_search
[STOP] [2020-12-08 15:36:57] reindex_search
[START] [2020-12-08 15:36:57] normalize_units
[STOP] [2020-12-08 15:36:57] normalize_units
[START] [2020-12-08 15:36:57] calculate_statistics
[2020-12-08 15:36:57] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-12-08 15:36:57] calculate_statistics
[START] [2020-12-08 15:36:57] complete_harvest_instance
[START] [2020-12-08 15:36:57] overall_tsv_creation
[INFO] [2020-12-08 15:36:57] Processing group of 11 in 1 batches of 10000
[INFO] [2020-12-08 15:37:37] 11 Traits (unfiltered)...
[INFO] [2020-12-08 15:38:15] 11 Traits (filtered)...
[INFO] [2020-12-08 15:38:15] 0 Associations (filtered)...
[INFO] [2020-12-08 15:38:15] 25 metadata added.
[INFO] [2020-12-08 15:38:15] 0 metadata added.
[INFO] [2020-12-08 15:38:15] Average Time: 54.99
[INFO] [2020-12-08 15:38:15] Total Time: 1m18s
[STOP] [2020-12-08 15:38:15] overall_tsv_creation
[INFO] [2020-12-08 15:38:15] Done. Check your files:
[INFO] [2020-12-08 15:38:15] (11 lines) /app/public/data/du_et_al_du_et_a/publish_nodes.tsv
[INFO] [2020-12-08 15:38:15] (11 lines) /app/public/data/du_et_al_du_et_a/publish_scientific_names.tsv
[INFO] [2020-12-08 15:38:15] (12 lines) /app/public/data/du_et_al_du_et_a/publish_traits.tsv
[INFO] [2020-12-08 15:38:15] (26 lines) /app/public/data/du_et_al_du_et_a/publish_metadata.tsv
[STOP] [2020-12-08 15:38:15] complete_harvest_instance
[START] [2020-12-08 15:38:15] completed
[STOP] [2020-12-08 15:38:15] completed
[STOP] [2020-12-08 15:38:15] logged process, took 87.64
[INFO] [2021-04-19 11:19:37] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-19 11:29:30] ## remove_type: ScientificName
[INFO] [2021-04-19 11:29:30] ++ Calling delete_all on 11 instances...
[INFO] [2021-04-19 11:29:30] [11:29:30.195] Removed 11 Scientificnames
[INFO] [2021-04-19 11:29:30] ## remove_type: Vernacular
[INFO] [2021-04-19 11:29:30] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 11:29:30] [11:29:30.197] Removed 0 Vernaculars
[INFO] [2021-04-19 11:29:30] ## remove_type: Article
[INFO] [2021-04-19 11:29:30] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 11:29:30] [11:29:30.198] Removed 0 Articles
[INFO] [2021-04-19 11:29:30] ## remove_type: Medium
[INFO] [2021-04-19 11:29:30] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 11:29:30] [11:29:30.200] Removed 0 Media
[INFO] [2021-04-19 11:29:30] ## remove_type: Trait
[INFO] [2021-04-19 11:29:30] ++ Calling delete_all on 25 instances...
[INFO] [2021-04-19 11:29:30] [11:29:30.202] Removed 25 Traits
[INFO] [2021-04-19 11:29:30] ## remove_type: MetaTrait
[INFO] [2021-04-19 11:29:30] ++ Calling delete_all on 11 instances...
[INFO] [2021-04-19 11:29:30] [11:29:30.205] Removed 11 Metatraits
[INFO] [2021-04-19 11:29:30] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-19 11:29:30] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 11:29:30] [11:29:30.207] Removed 0 Occurrencemetadata
[INFO] [2021-04-19 11:29:30] ## remove_type: Assoc
[INFO] [2021-04-19 11:29:30] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 11:29:30] [11:29:30.208] Removed 0 Assocs
[INFO] [2021-04-19 11:29:30] ## remove_type: MetaAssoc
[INFO] [2021-04-19 11:29:30] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 11:29:30] [11:29:30.210] Removed 0 Metaassocs
[INFO] [2021-04-19 11:29:30] ## remove_type: Identifier
[INFO] [2021-04-19 11:29:30] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 11:29:30] [11:29:30.211] Removed 0 Identifiers
[INFO] [2021-04-19 11:29:30] ## remove_type: Reference
[INFO] [2021-04-19 11:29:30] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 11:29:30] [11:29:30.213] Removed 0 References
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] Starting batch with ID 86121274...
[INFO] [2021-04-19 11:29:30] ## remove_type: Node
[INFO] [2021-04-19 11:29:30] ++ Calling delete_all on 11 instances...
[INFO] [2021-04-19 11:29:30] [11:29:30.928] Removed 11 Nodes
[START] [2021-04-19 11:29:31] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 11:29:31] Creating resource from OpenData
[START] [2021-04-19 11:29:32] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 11:29:32] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 11:29:35] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 11:29:35] Creating resource from OpenData
[START] [2021-04-19 11:29:36] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 11:29:36] create_harvest_instance
[INFO] [2021-04-19 11:29:36] Created harvest instance #3740
[STOP] [2021-04-19 11:29:36] create_harvest_instance
[START] [2021-04-19 11:29:36] fetch_files
[STOP] [2021-04-19 11:29:36] fetch_files
[START] [2021-04-19 11:29:36] validate_each_file
[INFO] [2021-04-19 11:29:36] Looping over 3 formats...
[INFO] [2021-04-19 11:29:36] ...nodes (/app/public/data/du_et_al_du_et_a/taxa.txt)
[INFO] [2021-04-19 11:29:36] Valid: /app/public/converted_csv/du_et_al_du_et_a_nodes_3740.csv (11 lines)
[INFO] [2021-04-19 11:29:36] ...occurrences (/app/public/data/du_et_al_du_et_a/occurrences.txt)
[INFO] [2021-04-19 11:29:36] Valid: /app/public/converted_csv/du_et_al_du_et_a_occurrences_3740.csv (11 lines)
[INFO] [2021-04-19 11:29:36] ...measurements (/app/public/data/du_et_al_du_et_a/measurementsorfacts.txt)
[INFO] [2021-04-19 11:29:36] Valid: /app/public/converted_csv/du_et_al_du_et_a_measurements_3740.csv (25 lines)
[STOP] [2021-04-19 11:29:36] validate_each_file
[START] [2021-04-19 11:29:36] convert_to_csv
[INFO] [2021-04-19 11:29:36] Looping over 3 formats...
[INFO] [2021-04-19 11:29:36] ...nodes (/app/public/data/du_et_al_du_et_a/taxa.txt)
[CMD] [2021-04-19 11:29:36] /usr/bin/sort /app/public/converted_csv/du_et_al_du_et_a_nodes_3740.csv > /app/public/converted_csv/du_et_al_du_et_a_nodes_3740.csv_sorted
[INFO] [2021-04-19 11:29:36] Converted: /app/public/converted_csv/du_et_al_du_et_a_nodes_3740.csv (11 lines)
[INFO] [2021-04-19 11:29:36] ...occurrences (/app/public/data/du_et_al_du_et_a/occurrences.txt)
[CMD] [2021-04-19 11:29:36] /usr/bin/sort /app/public/converted_csv/du_et_al_du_et_a_occurrences_3740.csv > /app/public/converted_csv/du_et_al_du_et_a_occurrences_3740.csv_sorted
[INFO] [2021-04-19 11:29:37] Converted: /app/public/converted_csv/du_et_al_du_et_a_occurrences_3740.csv (11 lines)
[INFO] [2021-04-19 11:29:37] ...measurements (/app/public/data/du_et_al_du_et_a/measurementsorfacts.txt)
[CMD] [2021-04-19 11:29:37] /usr/bin/sort /app/public/converted_csv/du_et_al_du_et_a_measurements_3740.csv > /app/public/converted_csv/du_et_al_du_et_a_measurements_3740.csv_sorted
[INFO] [2021-04-19 11:29:37] Converted: /app/public/converted_csv/du_et_al_du_et_a_measurements_3740.csv (25 lines)
[STOP] [2021-04-19 11:29:37] convert_to_csv
[START] [2021-04-19 11:29:37] calculate_delta
[INFO] [2021-04-19 11:29:37] Looping over 3 formats...
[INFO] [2021-04-19 11:29:37] ...nodes (/app/public/data/du_et_al_du_et_a/taxa.txt)
[CMD] [2021-04-19 11:29:37] echo "0a" > /app/public/diff/du_et_al_du_et_a_nodes_3740.diff
[CMD] [2021-04-19 11:29:37] tail -n +1 /app/public/converted_csv/du_et_al_du_et_a_nodes_3740.csv >> /app/public/diff/du_et_al_du_et_a_nodes_3740.diff
[CMD] [2021-04-19 11:29:38] echo "." >> /app/public/diff/du_et_al_du_et_a_nodes_3740.diff
[INFO] [2021-04-19 11:29:38] Created diff: /app/public/diff/du_et_al_du_et_a_nodes_3740.diff (13 lines)
[INFO] [2021-04-19 11:29:38] ...occurrences (/app/public/data/du_et_al_du_et_a/occurrences.txt)
[CMD] [2021-04-19 11:29:38] echo "0a" > /app/public/diff/du_et_al_du_et_a_occurrences_3740.diff
[CMD] [2021-04-19 11:29:38] tail -n +1 /app/public/converted_csv/du_et_al_du_et_a_occurrences_3740.csv >> /app/public/diff/du_et_al_du_et_a_occurrences_3740.diff
[CMD] [2021-04-19 11:29:39] echo "." >> /app/public/diff/du_et_al_du_et_a_occurrences_3740.diff
[INFO] [2021-04-19 11:29:39] Created diff: /app/public/diff/du_et_al_du_et_a_occurrences_3740.diff (13 lines)
[INFO] [2021-04-19 11:29:39] ...measurements (/app/public/data/du_et_al_du_et_a/measurementsorfacts.txt)
[CMD] [2021-04-19 11:29:39] echo "0a" > /app/public/diff/du_et_al_du_et_a_measurements_3740.diff
[CMD] [2021-04-19 11:29:40] tail -n +1 /app/public/converted_csv/du_et_al_du_et_a_measurements_3740.csv >> /app/public/diff/du_et_al_du_et_a_measurements_3740.diff
[CMD] [2021-04-19 11:29:40] echo "." >> /app/public/diff/du_et_al_du_et_a_measurements_3740.diff
[INFO] [2021-04-19 11:29:40] Created diff: /app/public/diff/du_et_al_du_et_a_measurements_3740.diff (27 lines)
[STOP] [2021-04-19 11:29:40] calculate_delta
[START] [2021-04-19 11:29:40] parse_diff_and_store
[INFO] [2021-04-19 11:29:40] Handling diff: /app/public/diff/du_et_al_du_et_a_nodes_3740.diff (13 lines)
[INFO] [2021-04-19 11:29:41] Loading nodes diff file into memory (13 /app/public/diff/du_et_al_du_et_a_nodes_3740.diff lines)...
[INFO] [2021-04-19 11:29:41] Handling diff: /app/public/diff/du_et_al_du_et_a_occurrences_3740.diff (13 lines)
[INFO] [2021-04-19 11:29:42] Loading occurrences diff file into memory (13 /app/public/diff/du_et_al_du_et_a_occurrences_3740.diff lines)...
[INFO] [2021-04-19 11:29:42] Handling diff: /app/public/diff/du_et_al_du_et_a_measurements_3740.diff (27 lines)
[INFO] [2021-04-19 11:29:42] Loading measurements diff file into memory (27 /app/public/diff/du_et_al_du_et_a_measurements_3740.diff lines)...
[INFO] [2021-04-19 11:29:43] Storing 11 ScientificNames
[INFO] [2021-04-19 11:29:43] Processing group of 11 in 1 groups of 1000
[INFO] [2021-04-19 11:29:43] Average Time: 0.0
[INFO] [2021-04-19 11:29:43] Total Time: 1s
[INFO] [2021-04-19 11:29:43] Storing 11 Nodes
[INFO] [2021-04-19 11:29:43] Processing group of 11 in 1 groups of 1000
[INFO] [2021-04-19 11:29:43] Average Time: 0.0
[INFO] [2021-04-19 11:29:43] Total Time: 1s
[INFO] [2021-04-19 11:29:43] Storing 11 Occurrences
[INFO] [2021-04-19 11:29:43] Processing group of 11 in 1 groups of 1000
[INFO] [2021-04-19 11:29:43] Average Time: 0.0
[INFO] [2021-04-19 11:29:43] Total Time: 1s
[INFO] [2021-04-19 11:29:43] Storing 25 Traits
[INFO] [2021-04-19 11:29:43] Processing group of 25 in 1 groups of 1000
[INFO] [2021-04-19 11:29:43] Average Time: 0.01
[INFO] [2021-04-19 11:29:43] Total Time: 1s
[INFO] [2021-04-19 11:29:43] Storing 11 MetaTraits
[INFO] [2021-04-19 11:29:43] Processing group of 11 in 1 groups of 1000
[INFO] [2021-04-19 11:29:43] Average Time: 0.0
[INFO] [2021-04-19 11:29:43] Total Time: 1s
[STOP] [2021-04-19 11:29:43] parse_diff_and_store
[START] [2021-04-19 11:29:43] resolve_keys
[INFO] [2021-04-19 11:29:49] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-19 11:29:49] traits to occurrences...
[INFO] [2021-04-19 11:29:49] traits to nodes (through occurrences)...
[INFO] [2021-04-19 11:29:49] Traits to sex term...
[INFO] [2021-04-19 11:29:49] Traits to lifestage term...
[INFO] [2021-04-19 11:29:49] MetaTraits to traits...
[INFO] [2021-04-19 11:29:49] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-19 11:29:49] Assocs to occurrences...
[INFO] [2021-04-19 11:29:49] Assocs to nodes...
[INFO] [2021-04-19 11:29:49] Assoc to sex term...
[INFO] [2021-04-19 11:29:49] Assoc to lifestage term...
[INFO] [2021-04-19 11:29:49] MetaAssoc to assocs...
[STOP] [2021-04-19 11:29:49] resolve_keys
[START] [2021-04-19 11:29:49] hold_for_later_1
[STOP] [2021-04-19 11:29:49] hold_for_later_1
[START] [2021-04-19 11:29:49] hold_for_later_2
[STOP] [2021-04-19 11:29:49] hold_for_later_2
[START] [2021-04-19 11:29:49] resolve_missing_parents
[STOP] [2021-04-19 11:29:49] resolve_missing_parents
[START] [2021-04-19 11:29:49] rebuild_nodes
[START] [2021-04-19 11:29:49] Flattener#flatten
[START] [2021-04-19 11:29:49] Flattener#study_resource
[START] [2021-04-19 11:29:49] Flattener#build_ancestry
[STOP] [2021-04-19 11:29:49] Flattener#build_ancestry
[INFO] [2021-04-19 11:29:49] 11 ancestry keys
[START] [2021-04-19 11:29:49] build_node_ancestors
[INFO] [2021-04-19 11:29:49] old ancestors deleted.
[STOP] [2021-04-19 11:29:49] build_node_ancestors
[WARN] [2021-04-19 11:29:49] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2021-04-19 11:29:49] Flattener#flatten
[STOP] [2021-04-19 11:29:49] rebuild_nodes
[START] [2021-04-19 11:29:49] resolve_missing_media_owners
[STOP] [2021-04-19 11:29:49] resolve_missing_media_owners
[START] [2021-04-19 11:29:49] sanitize_media_verbatims
[STOP] [2021-04-19 11:29:49] sanitize_media_verbatims
[START] [2021-04-19 11:29:49] queue_downloads
[STOP] [2021-04-19 11:29:49] queue_downloads
[START] [2021-04-19 11:29:49] parse_names
[WARN] [2021-04-19 11:29:49] I see 11 names which still need to be parsed.
[STOP] [2021-04-19 11:29:50] parse_names
[START] [2021-04-19 11:29:50] denormalize_canonical_names_to_nodes
[STOP] [2021-04-19 11:29:50] denormalize_canonical_names_to_nodes
[START] [2021-04-19 11:29:50] match_nodes
[START] [2021-04-19 11:29:50] map_all_nodes_to_pages
[STOP] [2021-04-19 11:29:50] map_all_nodes_to_pages
[INFO] [2021-04-19 11:29:50] ZERO unmatched nodes (of 11)! Nicely done.
[START] [2021-04-19 11:29:50] update_nodes
[STOP] [2021-04-19 11:29:50] update_nodes
[STOP] [2021-04-19 11:29:50] match_nodes
[START] [2021-04-19 11:29:50] reindex_search
[STOP] [2021-04-19 11:29:50] reindex_search
[START] [2021-04-19 11:29:50] normalize_units
[STOP] [2021-04-19 11:29:50] normalize_units
[START] [2021-04-19 11:29:50] calculate_statistics
[2021-04-19 11:29:50] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2021-04-19 11:29:50] calculate_statistics
[START] [2021-04-19 11:29:51] complete_harvest_instance
[START] [2021-04-19 11:29:51] overall_tsv_creation
[INFO] [2021-04-19 11:29:51] Processing group of 11 in 1 batches of 10000
[INFO] [2021-04-19 11:30:27] 11 Traits (unfiltered)...
[INFO] [2021-04-19 11:31:01] 11 Traits (filtered)...
[INFO] [2021-04-19 11:31:01] 0 Associations (filtered)...
[INFO] [2021-04-19 11:31:01] 14 metadata added.
[INFO] [2021-04-19 11:31:01] 0 metadata added.
[INFO] [2021-04-19 11:31:27] Average Time: 72.44
[INFO] [2021-04-19 11:31:27] Total Time: 1m37s
[STOP] [2021-04-19 11:31:27] overall_tsv_creation
[INFO] [2021-04-19 11:31:27] Done. Check your files:
[INFO] [2021-04-19 11:31:28] (11 lines) /app/public/data/du_et_al_du_et_a/publish_nodes.tsv
[INFO] [2021-04-19 11:31:28] (11 lines) /app/public/data/du_et_al_du_et_a/publish_scientific_names.tsv
[INFO] [2021-04-19 11:31:28] (12 lines) /app/public/data/du_et_al_du_et_a/publish_traits.tsv
[INFO] [2021-04-19 11:31:29] (15 lines) /app/public/data/du_et_al_du_et_a/publish_metadata.tsv
[STOP] [2021-04-19 11:31:29] complete_harvest_instance
[START] [2021-04-19 11:31:29] completed
[STOP] [2021-04-19 11:31:29] completed
[STOP] [2021-04-19 11:31:29] logged process, took 113.42
Latest Process

19 Apr 11:29 EXITED currently processing 0