Harvest for Renous et al 2007 Created 19 Apr 09:26

Stage: completed
Fetched: 19 Apr 09:26
Validated: 19 Apr 09:26
Deltas Created 19 Apr 09:26
Units Normalized: 19 Apr 09:27
Ancestry Built: 19 Apr 09:27
Nodes Matched: 19 Apr 09:27
Names Parsed: 19 Apr 09:27
New Models Stored: 19 Apr 09:26
Indexed: 19 Apr 09:27
Completed: 19 Apr 09:28
Time to Harvest: less than a minute

Harvesting Log

(751 lines)
# Logfile created on 2020-07-22 15:08:38 -0400 by logger.rb/v1.4.2
[START] [2020-07-22 15:08:38] logged process
[START] [2020-07-22 15:08:38] Creating resource from OpenData
[START] [2020-07-22 15:08:39] logged process
[START] [2020-07-22 15:08:39] Parse meta.xml file and create formats with fields
[STOP] [2020-07-22 15:08:39] Parse meta.xml file and create formats with fields
[STOP] [2020-07-22 15:08:39] Creating resource from OpenData
[INFO] [2020-07-22 15:12:47] ## HARVEST: type = -harvest
[START] [2020-07-22 15:12:48] logged process
[START] [2020-07-22 15:12:48] create_harvest_instance
[STOP] [2020-07-22 15:12:49] create_harvest_instance
[START] [2020-07-22 15:12:49] fetch_files
[STOP] [2020-07-22 15:12:49] fetch_files
[START] [2020-07-22 15:12:49] validate_each_file
[STOP] [2020-07-22 15:12:50] validate_each_file
[START] [2020-07-22 15:12:50] convert_to_csv
[CMD] [2020-07-22 15:12:50] /usr/bin/sort /app/public/converted_csv/renous_et_al_ren_nodes_22054.csv > /app/public/converted_csv/renous_et_al_ren_nodes_22054.csv_sorted
[CMD] [2020-07-22 15:12:50] /usr/bin/sort /app/public/converted_csv/renous_et_al_ren_occurrences_22055.csv > /app/public/converted_csv/renous_et_al_ren_occurrences_22055.csv_sorted
[CMD] [2020-07-22 15:12:50] /usr/bin/sort /app/public/converted_csv/renous_et_al_ren_measurements_22056.csv > /app/public/converted_csv/renous_et_al_ren_measurements_22056.csv_sorted
[STOP] [2020-07-22 15:12:50] convert_to_csv
[START] [2020-07-22 15:12:50] calculate_delta
[CMD] [2020-07-22 15:12:50] echo "0a" > /app/public/diff/renous_et_al_ren_nodes_22054.diff
[CMD] [2020-07-22 15:12:50] tail -n +1 /app/public/converted_csv/renous_et_al_ren_nodes_22054.csv >> /app/public/diff/renous_et_al_ren_nodes_22054.diff
[CMD] [2020-07-22 15:12:50] echo "." >> /app/public/diff/renous_et_al_ren_nodes_22054.diff
[CMD] [2020-07-22 15:12:50] echo "0a" > /app/public/diff/renous_et_al_ren_occurrences_22055.diff
[CMD] [2020-07-22 15:12:50] tail -n +1 /app/public/converted_csv/renous_et_al_ren_occurrences_22055.csv >> /app/public/diff/renous_et_al_ren_occurrences_22055.diff
[CMD] [2020-07-22 15:12:50] echo "." >> /app/public/diff/renous_et_al_ren_occurrences_22055.diff
[CMD] [2020-07-22 15:12:50] echo "0a" > /app/public/diff/renous_et_al_ren_measurements_22056.diff
[CMD] [2020-07-22 15:12:50] tail -n +1 /app/public/converted_csv/renous_et_al_ren_measurements_22056.csv >> /app/public/diff/renous_et_al_ren_measurements_22056.diff
[CMD] [2020-07-22 15:12:50] echo "." >> /app/public/diff/renous_et_al_ren_measurements_22056.diff
[STOP] [2020-07-22 15:12:50] calculate_delta
[START] [2020-07-22 15:12:50] parse_diff_and_store
[INFO] [2020-07-22 15:12:50] Loading nodes diff file into memory (true lines)...
[INFO] [2020-07-22 15:12:50] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-07-22 15:12:50] Loading measurements diff file into memory (true lines)...
[WARN] [2020-07-22 15:12:50] IGNORING a measurement NOT of a taxon (10) with NO parent and NO occurrence ID.
[WARN] [2020-07-22 15:12:50] IGNORING a measurement NOT of a taxon (12) with NO parent and NO occurrence ID.
[WARN] [2020-07-22 15:12:50] IGNORING a measurement NOT of a taxon (14) with NO parent and NO occurrence ID.
[WARN] [2020-07-22 15:12:50] IGNORING a measurement NOT of a taxon (16) with NO parent and NO occurrence ID.
[WARN] [2020-07-22 15:12:50] IGNORING a measurement NOT of a taxon (18) with NO parent and NO occurrence ID.
[WARN] [2020-07-22 15:12:50] IGNORING a measurement NOT of a taxon (2) with NO parent and NO occurrence ID.
[WARN] [2020-07-22 15:12:50] IGNORING a measurement NOT of a taxon (20) with NO parent and NO occurrence ID.
[WARN] [2020-07-22 15:12:50] IGNORING a measurement NOT of a taxon (22) with NO parent and NO occurrence ID.
[WARN] [2020-07-22 15:12:50] IGNORING a measurement NOT of a taxon (24) with NO parent and NO occurrence ID.
[WARN] [2020-07-22 15:12:50] IGNORING a measurement NOT of a taxon (26) with NO parent and NO occurrence ID.
[WARN] [2020-07-22 15:12:50] IGNORING a measurement NOT of a taxon (28) with NO parent and NO occurrence ID.
[WARN] [2020-07-22 15:12:50] IGNORING a measurement NOT of a taxon (4) with NO parent and NO occurrence ID.
[WARN] [2020-07-22 15:12:50] IGNORING a measurement NOT of a taxon (6) with NO parent and NO occurrence ID.
[WARN] [2020-07-22 15:12:50] IGNORING a measurement NOT of a taxon (8) with NO parent and NO occurrence ID.
[INFO] [2020-07-22 15:12:50] Storing 7 ScientificNames
[INFO] [2020-07-22 15:12:50] Processing group of 7 in 1 groups of 1000
[INFO] [2020-07-22 15:12:50] Average Time: 0.01
[INFO] [2020-07-22 15:12:50] Total Time: 1s
[INFO] [2020-07-22 15:12:50] Storing 7 Nodes
[INFO] [2020-07-22 15:12:50] Processing group of 7 in 1 groups of 1000
[INFO] [2020-07-22 15:12:50] Average Time: 0.0
[INFO] [2020-07-22 15:12:50] Total Time: 1s
[INFO] [2020-07-22 15:12:50] Storing 7 Occurrences
[INFO] [2020-07-22 15:12:50] Processing group of 7 in 1 groups of 1000
[INFO] [2020-07-22 15:12:50] Average Time: 0.0
[INFO] [2020-07-22 15:12:50] Total Time: 1s
[INFO] [2020-07-22 15:12:50] Storing 13 Traits
[INFO] [2020-07-22 15:12:50] Processing group of 13 in 1 groups of 1000
[INFO] [2020-07-22 15:12:50] Average Time: 0.01
[INFO] [2020-07-22 15:12:50] Total Time: 1s
[INFO] [2020-07-22 15:12:50] Storing 26 MetaTraits
[INFO] [2020-07-22 15:12:50] Processing group of 26 in 1 groups of 1000
[INFO] [2020-07-22 15:12:50] Average Time: 0.0
[INFO] [2020-07-22 15:12:50] Total Time: 1s
[STOP] [2020-07-22 15:12:50] parse_diff_and_store
[START] [2020-07-22 15:12:50] resolve_keys
[INFO] [2020-07-22 15:12:57] Occurrences to nodes (through scientific_names)...
[INFO] [2020-07-22 15:12:57] traits to occurrences...
[INFO] [2020-07-22 15:12:57] traits to nodes (through occurrences)...
[INFO] [2020-07-22 15:12:57] Traits to sex term...
[INFO] [2020-07-22 15:12:57] Traits to lifestage term...
[INFO] [2020-07-22 15:12:57] MetaTraits to traits...
[INFO] [2020-07-22 15:12:57] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-07-22 15:12:57] Assocs to occurrences...
[INFO] [2020-07-22 15:12:57] Assocs to nodes...
[INFO] [2020-07-22 15:12:57] Assoc to sex term...
[INFO] [2020-07-22 15:12:57] Assoc to lifestage term...
[STOP] [2020-07-22 15:12:57] resolve_keys
[START] [2020-07-22 15:12:57] hold_for_later_1
[STOP] [2020-07-22 15:12:57] hold_for_later_1
[START] [2020-07-22 15:12:57] hold_for_later_2
[STOP] [2020-07-22 15:12:57] hold_for_later_2
[START] [2020-07-22 15:12:57] resolve_missing_parents
[STOP] [2020-07-22 15:12:57] resolve_missing_parents
[START] [2020-07-22 15:12:57] rebuild_nodes
[START] [2020-07-22 15:12:57] Flattener#flatten
[START] [2020-07-22 15:12:57] Flattener#study_resource
[START] [2020-07-22 15:12:57] Flattener#build_ancestry
[STOP] [2020-07-22 15:12:57] Flattener#build_ancestry
[INFO] [2020-07-22 15:12:57] 7 ancestry keys
[START] [2020-07-22 15:12:57] build_node_ancestors
[INFO] [2020-07-22 15:12:57] old ancestors deleted.
[STOP] [2020-07-22 15:12:57] build_node_ancestors
[WARN] [2020-07-22 15:12:57] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-07-22 15:12:57] Flattener#flatten
[STOP] [2020-07-22 15:12:57] rebuild_nodes
[START] [2020-07-22 15:12:57] resolve_missing_media_owners
[STOP] [2020-07-22 15:12:57] resolve_missing_media_owners
[START] [2020-07-22 15:12:57] sanitize_media_verbatims
[STOP] [2020-07-22 15:12:57] sanitize_media_verbatims
[START] [2020-07-22 15:12:57] queue_downloads
[STOP] [2020-07-22 15:12:57] queue_downloads
[START] [2020-07-22 15:12:57] parse_names
[WARN] [2020-07-22 15:12:57] I see 7 names which still need to be parsed.
[STOP] [2020-07-22 15:12:58] parse_names
[START] [2020-07-22 15:12:58] denormalize_canonical_names_to_nodes
[STOP] [2020-07-22 15:12:58] denormalize_canonical_names_to_nodes
[START] [2020-07-22 15:12:58] match_nodes
[START] [2020-07-22 15:12:58] map_all_nodes_to_pages
[STOP] [2020-07-22 15:12:58] map_all_nodes_to_pages
[INFO] [2020-07-22 15:12:58] ZERO unmatched nodes (of 7)! Nicely done.
[START] [2020-07-22 15:12:58] update_nodes
[STOP] [2020-07-22 15:12:58] update_nodes
[STOP] [2020-07-22 15:12:58] match_nodes
[START] [2020-07-22 15:12:58] reindex_search
[STOP] [2020-07-22 15:12:58] reindex_search
[START] [2020-07-22 15:12:58] normalize_units
[STOP] [2020-07-22 15:12:58] normalize_units
[START] [2020-07-22 15:12:58] calculate_statistics
[2020-07-22 15:12:58] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-07-22 15:12:58] calculate_statistics
[START] [2020-07-22 15:12:58] complete_harvest_instance
[START] [2020-07-22 15:12:58] overall_tsv_creation
[INFO] [2020-07-22 15:12:58] Processing group of 7 in 1 batches of 10000
[INFO] [2020-07-22 15:13:47] 13 Traits (unfiltered)...
[INFO] [2020-07-22 15:14:23] 13 Traits (filtered)...
[INFO] [2020-07-22 15:14:23] 0 Associations (filtered)...
[INFO] [2020-07-22 15:14:23] 26 metadata added.
[INFO] [2020-07-22 15:14:23] 0 metadata added.
[INFO] [2020-07-22 15:14:23] Average Time: 57.67
[INFO] [2020-07-22 15:14:23] Total Time: 1m25s
[STOP] [2020-07-22 15:14:23] overall_tsv_creation
[INFO] [2020-07-22 15:14:23] Done. Check your files:
[INFO] [2020-07-22 15:14:23] (7 lines) /app/public/data/renous_et_al_ren/publish_nodes.tsv
[INFO] [2020-07-22 15:14:23] (7 lines) /app/public/data/renous_et_al_ren/publish_scientific_names.tsv
[INFO] [2020-07-22 15:14:23] (14 lines) /app/public/data/renous_et_al_ren/publish_traits.tsv
[INFO] [2020-07-22 15:14:23] (1 lines) /app/public/data/renous_et_al_ren/publish_metadata.tsv
[STOP] [2020-07-22 15:14:23] complete_harvest_instance
[START] [2020-07-22 15:14:23] completed
[STOP] [2020-07-22 15:14:23] completed
[STOP] [2020-07-22 15:14:23] logged process, took 94.51
[INFO] [2020-08-20 14:47:13] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-08-20 14:47:14] ## remove_type: ScientificName
[INFO] [2020-08-20 14:47:14] ++ Calling delete_all on 7 instances...
[INFO] [2020-08-20 14:47:14] [14:47:14.080] Removed 7 Scientificnames
[INFO] [2020-08-20 14:47:14] ## remove_type: Vernacular
[INFO] [2020-08-20 14:47:14] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-20 14:47:14] [14:47:14.083] Removed 0 Vernaculars
[INFO] [2020-08-20 14:47:14] ## remove_type: Article
[INFO] [2020-08-20 14:47:14] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-20 14:47:14] [14:47:14.085] Removed 0 Articles
[INFO] [2020-08-20 14:47:14] ## remove_type: Medium
[INFO] [2020-08-20 14:47:14] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-20 14:47:14] [14:47:14.089] Removed 0 Media
[INFO] [2020-08-20 14:47:14] ## remove_type: Trait
[INFO] [2020-08-20 14:47:14] ++ Calling delete_all on 13 instances...
[INFO] [2020-08-20 14:47:14] [14:47:14.093] Removed 13 Traits
[INFO] [2020-08-20 14:47:14] ## remove_type: MetaTrait
[INFO] [2020-08-20 14:47:14] ++ Calling delete_all on 26 instances...
[INFO] [2020-08-20 14:47:14] [14:47:14.147] Removed 26 Metatraits
[INFO] [2020-08-20 14:47:14] ## remove_type: OccurrenceMetadatum
[INFO] [2020-08-20 14:47:14] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-20 14:47:14] [14:47:14.151] Removed 0 Occurrencemetadata
[INFO] [2020-08-20 14:47:14] ## remove_type: Assoc
[INFO] [2020-08-20 14:47:14] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-20 14:47:14] [14:47:14.154] Removed 0 Assocs
[INFO] [2020-08-20 14:47:14] ## remove_type: MetaAssoc
[INFO] [2020-08-20 14:47:14] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-20 14:47:14] [14:47:14.160] Removed 0 Metaassocs
[INFO] [2020-08-20 14:47:14] ## remove_type: Identifier
[INFO] [2020-08-20 14:47:14] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-20 14:47:14] [14:47:14.179] Removed 0 Identifiers
[INFO] [2020-08-20 14:47:14] ## remove_type: Reference
[INFO] [2020-08-20 14:47:14] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-20 14:47:14] [14:47:14.182] Removed 0 References
[INFO] [2020-08-20 14:47:14] Starting batch with ID 80633643...
[INFO] [2020-08-20 14:47:14] Starting batch with ID 80633643...
[INFO] [2020-08-20 14:47:14] Starting batch with ID 80633643...
[INFO] [2020-08-20 14:47:14] Starting batch with ID 80633643...
[INFO] [2020-08-20 14:47:14] Starting batch with ID 80633643...
[INFO] [2020-08-20 14:47:14] Starting batch with ID 80633643...
[INFO] [2020-08-20 14:47:15] Starting batch with ID 80633643...
[INFO] [2020-08-20 14:47:15] Starting batch with ID 80633643...
[INFO] [2020-08-20 14:47:15] Starting batch with ID 80633643...
[INFO] [2020-08-20 14:47:16] Starting batch with ID 80633643...
[INFO] [2020-08-20 14:47:16] Starting batch with ID 80633643...
[INFO] [2020-08-20 14:47:16] Starting batch with ID 80633643...
[INFO] [2020-08-20 14:47:16] Starting batch with ID 80633643...
[INFO] [2020-08-20 14:47:16] Starting batch with ID 80633643...
[INFO] [2020-08-20 14:47:16] Starting batch with ID 80633643...
[INFO] [2020-08-20 14:47:16] Starting batch with ID 80633643...
[INFO] [2020-08-20 14:47:16] Starting batch with ID 80633643...
[INFO] [2020-08-20 14:47:16] Starting batch with ID 80633643...
[INFO] [2020-08-20 14:47:16] Starting batch with ID 80633643...
[INFO] [2020-08-20 14:47:16] Starting batch with ID 80633643...
[INFO] [2020-08-20 14:47:16] Starting batch with ID 80633643...
[INFO] [2020-08-20 14:47:16] Starting batch with ID 80633648...
[INFO] [2020-08-20 14:47:16] Starting batch with ID 80633648...
[INFO] [2020-08-20 14:47:17] Starting batch with ID 80633648...
[INFO] [2020-08-20 14:47:17] Starting batch with ID 80633648...
[INFO] [2020-08-20 14:47:17] Starting batch with ID 80633648...
[INFO] [2020-08-20 14:47:17] Starting batch with ID 80633648...
[INFO] [2020-08-20 14:47:17] Starting batch with ID 80633648...
[INFO] [2020-08-20 14:47:17] Starting batch with ID 80633648...
[INFO] [2020-08-20 14:47:17] Starting batch with ID 80633648...
[INFO] [2020-08-20 14:47:17] Starting batch with ID 80633648...
[INFO] [2020-08-20 14:47:17] Starting batch with ID 80633648...
[INFO] [2020-08-20 14:47:17] Starting batch with ID 80633648...
[INFO] [2020-08-20 14:47:17] Starting batch with ID 80633648...
[INFO] [2020-08-20 14:47:17] Starting batch with ID 80633648...
[INFO] [2020-08-20 14:47:17] Starting batch with ID 80633648...
[INFO] [2020-08-20 14:47:18] Starting batch with ID 80633648...
[INFO] [2020-08-20 14:47:18] Starting batch with ID 80633648...
[INFO] [2020-08-20 14:47:18] Starting batch with ID 80633648...
[INFO] [2020-08-20 14:47:18] Starting batch with ID 80633648...
[INFO] [2020-08-20 14:47:18] Starting batch with ID 80633648...
[INFO] [2020-08-20 14:47:18] Starting batch with ID 80633648...
[INFO] [2020-08-20 14:47:18] Starting batch with ID 80633644...
[INFO] [2020-08-20 14:47:18] Starting batch with ID 80633644...
[INFO] [2020-08-20 14:47:18] Starting batch with ID 80633644...
[INFO] [2020-08-20 14:47:18] Starting batch with ID 80633644...
[INFO] [2020-08-20 14:47:18] Starting batch with ID 80633644...
[INFO] [2020-08-20 14:47:19] Starting batch with ID 80633644...
[INFO] [2020-08-20 14:47:19] Starting batch with ID 80633644...
[INFO] [2020-08-20 14:47:19] Starting batch with ID 80633644...
[INFO] [2020-08-20 14:47:19] Starting batch with ID 80633644...
[INFO] [2020-08-20 14:47:19] ## remove_type: Node
[INFO] [2020-08-20 14:47:19] ++ Calling delete_all on 7 instances...
[INFO] [2020-08-20 14:47:19] [14:47:19.308] Removed 7 Nodes
[START] [2020-08-20 14:47:19] logged process
[START] [2020-08-20 14:47:19] Creating resource from OpenData
[START] [2020-08-20 14:47:20] logged process
[START] [2020-08-20 14:47:20] Parse meta.xml file and create formats with fields
[STOP] [2020-08-20 14:47:20] Parse meta.xml file and create formats with fields
[STOP] [2020-08-20 14:47:20] Creating resource from OpenData
[START] [2020-08-20 14:47:20] logged process
[START] [2020-08-20 14:47:20] create_harvest_instance
[STOP] [2020-08-20 14:47:21] create_harvest_instance
[START] [2020-08-20 14:47:21] fetch_files
[STOP] [2020-08-20 14:47:21] fetch_files
[START] [2020-08-20 14:47:21] validate_each_file
[STOP] [2020-08-20 14:47:21] validate_each_file
[START] [2020-08-20 14:47:21] convert_to_csv
[CMD] [2020-08-20 14:47:21] /usr/bin/sort /app/public/converted_csv/renous_et_al_ren_nodes_22500.csv > /app/public/converted_csv/renous_et_al_ren_nodes_22500.csv_sorted
[CMD] [2020-08-20 14:47:21] /usr/bin/sort /app/public/converted_csv/renous_et_al_ren_occurrences_22501.csv > /app/public/converted_csv/renous_et_al_ren_occurrences_22501.csv_sorted
[CMD] [2020-08-20 14:47:21] /usr/bin/sort /app/public/converted_csv/renous_et_al_ren_measurements_22502.csv > /app/public/converted_csv/renous_et_al_ren_measurements_22502.csv_sorted
[STOP] [2020-08-20 14:47:21] convert_to_csv
[START] [2020-08-20 14:47:21] calculate_delta
[CMD] [2020-08-20 14:47:21] echo "0a" > /app/public/diff/renous_et_al_ren_nodes_22500.diff
[CMD] [2020-08-20 14:47:21] tail -n +1 /app/public/converted_csv/renous_et_al_ren_nodes_22500.csv >> /app/public/diff/renous_et_al_ren_nodes_22500.diff
[CMD] [2020-08-20 14:47:21] echo "." >> /app/public/diff/renous_et_al_ren_nodes_22500.diff
[CMD] [2020-08-20 14:47:21] echo "0a" > /app/public/diff/renous_et_al_ren_occurrences_22501.diff
[CMD] [2020-08-20 14:47:21] tail -n +1 /app/public/converted_csv/renous_et_al_ren_occurrences_22501.csv >> /app/public/diff/renous_et_al_ren_occurrences_22501.diff
[CMD] [2020-08-20 14:47:21] echo "." >> /app/public/diff/renous_et_al_ren_occurrences_22501.diff
[CMD] [2020-08-20 14:47:21] echo "0a" > /app/public/diff/renous_et_al_ren_measurements_22502.diff
[CMD] [2020-08-20 14:47:21] tail -n +1 /app/public/converted_csv/renous_et_al_ren_measurements_22502.csv >> /app/public/diff/renous_et_al_ren_measurements_22502.diff
[CMD] [2020-08-20 14:47:21] echo "." >> /app/public/diff/renous_et_al_ren_measurements_22502.diff
[STOP] [2020-08-20 14:47:21] calculate_delta
[START] [2020-08-20 14:47:21] parse_diff_and_store
[INFO] [2020-08-20 14:47:21] Loading nodes diff file into memory (true lines)...
[INFO] [2020-08-20 14:47:21] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-08-20 14:47:21] Loading measurements diff file into memory (true lines)...
[INFO] [2020-08-20 14:47:21] Storing 7 ScientificNames
[INFO] [2020-08-20 14:47:21] Processing group of 7 in 1 groups of 1000
[INFO] [2020-08-20 14:47:21] Average Time: 0.0
[INFO] [2020-08-20 14:47:21] Total Time: 1s
[INFO] [2020-08-20 14:47:21] Storing 7 Nodes
[INFO] [2020-08-20 14:47:21] Processing group of 7 in 1 groups of 1000
[INFO] [2020-08-20 14:47:21] Average Time: 0.0
[INFO] [2020-08-20 14:47:21] Total Time: 1s
[INFO] [2020-08-20 14:47:21] Storing 7 Occurrences
[INFO] [2020-08-20 14:47:21] Processing group of 7 in 1 groups of 1000
[INFO] [2020-08-20 14:47:21] Average Time: 0.0
[INFO] [2020-08-20 14:47:21] Total Time: 1s
[INFO] [2020-08-20 14:47:21] Storing 28 Traits
[INFO] [2020-08-20 14:47:21] Processing group of 28 in 1 groups of 1000
[INFO] [2020-08-20 14:47:22] Average Time: 0.01
[INFO] [2020-08-20 14:47:22] Total Time: 1s
[INFO] [2020-08-20 14:47:22] Storing 28 MetaTraits
[INFO] [2020-08-20 14:47:22] Processing group of 28 in 1 groups of 1000
[INFO] [2020-08-20 14:47:22] Average Time: 0.01
[INFO] [2020-08-20 14:47:22] Total Time: 1s
[STOP] [2020-08-20 14:47:22] parse_diff_and_store
[START] [2020-08-20 14:47:22] resolve_keys
[INFO] [2020-08-20 14:47:29] Occurrences to nodes (through scientific_names)...
[INFO] [2020-08-20 14:47:29] traits to occurrences...
[INFO] [2020-08-20 14:47:29] traits to nodes (through occurrences)...
[INFO] [2020-08-20 14:47:29] Traits to sex term...
[INFO] [2020-08-20 14:47:29] Traits to lifestage term...
[INFO] [2020-08-20 14:47:29] MetaTraits to traits...
[INFO] [2020-08-20 14:47:29] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-08-20 14:47:29] Assocs to occurrences...
[INFO] [2020-08-20 14:47:29] Assocs to nodes...
[INFO] [2020-08-20 14:47:29] Assoc to sex term...
[INFO] [2020-08-20 14:47:29] Assoc to lifestage term...
[STOP] [2020-08-20 14:47:29] resolve_keys
[START] [2020-08-20 14:47:29] hold_for_later_1
[STOP] [2020-08-20 14:47:29] hold_for_later_1
[START] [2020-08-20 14:47:29] hold_for_later_2
[STOP] [2020-08-20 14:47:29] hold_for_later_2
[START] [2020-08-20 14:47:29] resolve_missing_parents
[STOP] [2020-08-20 14:47:29] resolve_missing_parents
[START] [2020-08-20 14:47:29] rebuild_nodes
[START] [2020-08-20 14:47:29] Flattener#flatten
[START] [2020-08-20 14:47:29] Flattener#study_resource
[START] [2020-08-20 14:47:29] Flattener#build_ancestry
[STOP] [2020-08-20 14:47:29] Flattener#build_ancestry
[INFO] [2020-08-20 14:47:29] 7 ancestry keys
[START] [2020-08-20 14:47:29] build_node_ancestors
[INFO] [2020-08-20 14:47:29] old ancestors deleted.
[STOP] [2020-08-20 14:47:29] build_node_ancestors
[WARN] [2020-08-20 14:47:29] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-08-20 14:47:29] Flattener#flatten
[STOP] [2020-08-20 14:47:29] rebuild_nodes
[START] [2020-08-20 14:47:29] resolve_missing_media_owners
[STOP] [2020-08-20 14:47:29] resolve_missing_media_owners
[START] [2020-08-20 14:47:29] sanitize_media_verbatims
[STOP] [2020-08-20 14:47:29] sanitize_media_verbatims
[START] [2020-08-20 14:47:29] queue_downloads
[STOP] [2020-08-20 14:47:29] queue_downloads
[START] [2020-08-20 14:47:29] parse_names
[WARN] [2020-08-20 14:47:29] I see 7 names which still need to be parsed.
[STOP] [2020-08-20 14:47:30] parse_names
[START] [2020-08-20 14:47:30] denormalize_canonical_names_to_nodes
[STOP] [2020-08-20 14:47:30] denormalize_canonical_names_to_nodes
[START] [2020-08-20 14:47:30] match_nodes
[START] [2020-08-20 14:47:30] map_all_nodes_to_pages
[STOP] [2020-08-20 14:47:30] map_all_nodes_to_pages
[INFO] [2020-08-20 14:47:30] ZERO unmatched nodes (of 7)! Nicely done.
[START] [2020-08-20 14:47:30] update_nodes
[STOP] [2020-08-20 14:47:30] update_nodes
[STOP] [2020-08-20 14:47:30] match_nodes
[START] [2020-08-20 14:47:30] reindex_search
[STOP] [2020-08-20 14:47:31] reindex_search
[START] [2020-08-20 14:47:31] normalize_units
[STOP] [2020-08-20 14:47:31] normalize_units
[START] [2020-08-20 14:47:31] calculate_statistics
[2020-08-20 14:47:31] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-08-20 14:47:31] calculate_statistics
[START] [2020-08-20 14:47:31] complete_harvest_instance
[START] [2020-08-20 14:47:31] overall_tsv_creation
[INFO] [2020-08-20 14:47:31] Processing group of 7 in 1 batches of 10000
[INFO] [2020-08-20 14:49:18] 14 Traits (unfiltered)...
[INFO] [2020-08-20 14:49:51] 14 Traits (filtered)...
[INFO] [2020-08-20 14:49:51] 0 Associations (filtered)...
[INFO] [2020-08-20 14:49:51] 42 metadata added.
[INFO] [2020-08-20 14:49:51] 0 metadata added.
[INFO] [2020-08-20 14:49:51] Average Time: 51.74
[INFO] [2020-08-20 14:49:51] Total Time: 2m21s
[STOP] [2020-08-20 14:49:51] overall_tsv_creation
[INFO] [2020-08-20 14:49:51] Done. Check your files:
[INFO] [2020-08-20 14:49:51] (7 lines) /app/public/data/renous_et_al_ren/publish_nodes.tsv
[INFO] [2020-08-20 14:49:51] (7 lines) /app/public/data/renous_et_al_ren/publish_scientific_names.tsv
[INFO] [2020-08-20 14:49:51] (15 lines) /app/public/data/renous_et_al_ren/publish_traits.tsv
[INFO] [2020-08-20 14:49:51] (15 lines) /app/public/data/renous_et_al_ren/publish_metadata.tsv
[STOP] [2020-08-20 14:49:51] complete_harvest_instance
[START] [2020-08-20 14:49:51] completed
[STOP] [2020-08-20 14:49:51] completed
[STOP] [2020-08-20 14:49:51] logged process, took 151.47
[INFO] [2020-11-13 10:40:17] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-11-13 10:40:20] ## remove_type: ScientificName
[INFO] [2020-11-13 10:40:20] ++ Calling delete_all on 7 instances...
[INFO] [2020-11-13 10:40:20] [10:40:20.982] Removed 7 Scientificnames
[INFO] [2020-11-13 10:40:20] ## remove_type: Vernacular
[INFO] [2020-11-13 10:40:20] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:40:20] [10:40:20.985] Removed 0 Vernaculars
[INFO] [2020-11-13 10:40:20] ## remove_type: Article
[INFO] [2020-11-13 10:40:20] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:40:20] [10:40:20.987] Removed 0 Articles
[INFO] [2020-11-13 10:40:20] ## remove_type: Medium
[INFO] [2020-11-13 10:40:20] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:40:20] [10:40:20.991] Removed 0 Media
[INFO] [2020-11-13 10:40:20] ## remove_type: Trait
[INFO] [2020-11-13 10:40:20] ++ Calling delete_all on 28 instances...
[INFO] [2020-11-13 10:40:20] [10:40:20.995] Removed 28 Traits
[INFO] [2020-11-13 10:40:20] ## remove_type: MetaTrait
[INFO] [2020-11-13 10:40:20] ++ Calling delete_all on 28 instances...
[INFO] [2020-11-13 10:40:21] [10:40:21.003] Removed 28 Metatraits
[INFO] [2020-11-13 10:40:21] ## remove_type: OccurrenceMetadatum
[INFO] [2020-11-13 10:40:21] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:40:21] [10:40:21.005] Removed 0 Occurrencemetadata
[INFO] [2020-11-13 10:40:21] ## remove_type: Assoc
[INFO] [2020-11-13 10:40:21] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:40:21] [10:40:21.008] Removed 0 Assocs
[INFO] [2020-11-13 10:40:21] ## remove_type: MetaAssoc
[INFO] [2020-11-13 10:40:21] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:40:21] [10:40:21.010] Removed 0 Metaassocs
[INFO] [2020-11-13 10:40:21] ## remove_type: Identifier
[INFO] [2020-11-13 10:40:21] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:40:21] [10:40:21.012] Removed 0 Identifiers
[INFO] [2020-11-13 10:40:21] ## remove_type: Reference
[INFO] [2020-11-13 10:40:21] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:40:21] [10:40:21.015] Removed 0 References
[INFO] [2020-11-13 10:40:21] Starting batch with ID 80729213...
[INFO] [2020-11-13 10:40:21] Starting batch with ID 80729213...
[INFO] [2020-11-13 10:40:21] Starting batch with ID 80729213...
[INFO] [2020-11-13 10:40:21] Starting batch with ID 80729213...
[INFO] [2020-11-13 10:40:21] Starting batch with ID 80729213...
[INFO] [2020-11-13 10:40:21] Starting batch with ID 80729213...
[INFO] [2020-11-13 10:40:21] Starting batch with ID 80729213...
[INFO] [2020-11-13 10:40:21] Starting batch with ID 80729213...
[INFO] [2020-11-13 10:40:21] Starting batch with ID 80729213...
[INFO] [2020-11-13 10:40:21] Starting batch with ID 80729213...
[INFO] [2020-11-13 10:40:21] Starting batch with ID 80729213...
[INFO] [2020-11-13 10:40:21] Starting batch with ID 80729213...
[INFO] [2020-11-13 10:40:21] Starting batch with ID 80729213...
[INFO] [2020-11-13 10:40:21] ## remove_type: Node
[INFO] [2020-11-13 10:40:21] ++ Calling delete_all on 7 instances...
[INFO] [2020-11-13 10:40:21] [10:40:21.581] Removed 7 Nodes
[START] [2020-11-13 10:40:21] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-11-13 10:40:21] Creating resource from OpenData
[START] [2020-11-13 10:40:22] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-11-13 10:40:22] Parse meta.xml file and create formats with fields
[STOP] [2020-11-13 10:40:22] Parse meta.xml file and create formats with fields
[STOP] [2020-11-13 10:40:22] Creating resource from OpenData
[START] [2020-11-13 10:40:22] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-11-13 10:40:22] create_harvest_instance
[STOP] [2020-11-13 10:40:23] create_harvest_instance
[START] [2020-11-13 10:40:23] fetch_files
[STOP] [2020-11-13 10:40:23] fetch_files
[START] [2020-11-13 10:40:23] validate_each_file
[STOP] [2020-11-13 10:40:23] validate_each_file
[START] [2020-11-13 10:40:23] convert_to_csv
[CMD] [2020-11-13 10:40:23] /usr/bin/sort /app/public/converted_csv/renous_et_al_ren_nodes_24133.csv > /app/public/converted_csv/renous_et_al_ren_nodes_24133.csv_sorted
[CMD] [2020-11-13 10:40:23] /usr/bin/sort /app/public/converted_csv/renous_et_al_ren_occurrences_24134.csv > /app/public/converted_csv/renous_et_al_ren_occurrences_24134.csv_sorted
[CMD] [2020-11-13 10:40:23] /usr/bin/sort /app/public/converted_csv/renous_et_al_ren_measurements_24135.csv > /app/public/converted_csv/renous_et_al_ren_measurements_24135.csv_sorted
[STOP] [2020-11-13 10:40:23] convert_to_csv
[START] [2020-11-13 10:40:23] calculate_delta
[CMD] [2020-11-13 10:40:23] echo "0a" > /app/public/diff/renous_et_al_ren_nodes_24133.diff
[CMD] [2020-11-13 10:40:23] tail -n +1 /app/public/converted_csv/renous_et_al_ren_nodes_24133.csv >> /app/public/diff/renous_et_al_ren_nodes_24133.diff
[CMD] [2020-11-13 10:40:23] echo "." >> /app/public/diff/renous_et_al_ren_nodes_24133.diff
[CMD] [2020-11-13 10:40:23] echo "0a" > /app/public/diff/renous_et_al_ren_occurrences_24134.diff
[CMD] [2020-11-13 10:40:23] tail -n +1 /app/public/converted_csv/renous_et_al_ren_occurrences_24134.csv >> /app/public/diff/renous_et_al_ren_occurrences_24134.diff
[CMD] [2020-11-13 10:40:23] echo "." >> /app/public/diff/renous_et_al_ren_occurrences_24134.diff
[CMD] [2020-11-13 10:40:23] echo "0a" > /app/public/diff/renous_et_al_ren_measurements_24135.diff
[CMD] [2020-11-13 10:40:23] tail -n +1 /app/public/converted_csv/renous_et_al_ren_measurements_24135.csv >> /app/public/diff/renous_et_al_ren_measurements_24135.diff
[CMD] [2020-11-13 10:40:23] echo "." >> /app/public/diff/renous_et_al_ren_measurements_24135.diff
[STOP] [2020-11-13 10:40:23] calculate_delta
[START] [2020-11-13 10:40:23] parse_diff_and_store
[INFO] [2020-11-13 10:40:23] Loading nodes diff file into memory (true lines)...
[INFO] [2020-11-13 10:40:23] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-11-13 10:40:23] Loading measurements diff file into memory (true lines)...
[INFO] [2020-11-13 10:40:24] Storing 7 ScientificNames
[INFO] [2020-11-13 10:40:24] Processing group of 7 in 1 groups of 1000
[INFO] [2020-11-13 10:40:24] Average Time: 0.0
[INFO] [2020-11-13 10:40:24] Total Time: 1s
[INFO] [2020-11-13 10:40:24] Storing 7 Nodes
[INFO] [2020-11-13 10:40:24] Processing group of 7 in 1 groups of 1000
[INFO] [2020-11-13 10:40:24] Average Time: 0.0
[INFO] [2020-11-13 10:40:24] Total Time: 1s
[INFO] [2020-11-13 10:40:24] Storing 7 Occurrences
[INFO] [2020-11-13 10:40:24] Processing group of 7 in 1 groups of 1000
[INFO] [2020-11-13 10:40:24] Average Time: 0.0
[INFO] [2020-11-13 10:40:24] Total Time: 1s
[INFO] [2020-11-13 10:40:24] Storing 28 Traits
[INFO] [2020-11-13 10:40:24] Processing group of 28 in 1 groups of 1000
[INFO] [2020-11-13 10:40:24] Average Time: 0.01
[INFO] [2020-11-13 10:40:24] Total Time: 1s
[INFO] [2020-11-13 10:40:24] Storing 14 MetaTraits
[INFO] [2020-11-13 10:40:24] Processing group of 14 in 1 groups of 1000
[INFO] [2020-11-13 10:40:24] Average Time: 0.0
[INFO] [2020-11-13 10:40:24] Total Time: 1s
[STOP] [2020-11-13 10:40:24] parse_diff_and_store
[START] [2020-11-13 10:40:24] resolve_keys
[INFO] [2020-11-13 10:40:30] Occurrences to nodes (through scientific_names)...
[INFO] [2020-11-13 10:40:30] traits to occurrences...
[INFO] [2020-11-13 10:40:30] traits to nodes (through occurrences)...
[INFO] [2020-11-13 10:40:30] Traits to sex term...
[INFO] [2020-11-13 10:40:30] Traits to lifestage term...
[INFO] [2020-11-13 10:40:30] MetaTraits to traits...
[INFO] [2020-11-13 10:40:30] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-11-13 10:40:30] Assocs to occurrences...
[INFO] [2020-11-13 10:40:30] Assocs to nodes...
[INFO] [2020-11-13 10:40:30] Assoc to sex term...
[INFO] [2020-11-13 10:40:30] Assoc to lifestage term...
[INFO] [2020-11-13 10:40:30] MetaAssoc to assocs...
[STOP] [2020-11-13 10:40:30] resolve_keys
[START] [2020-11-13 10:40:30] hold_for_later_1
[STOP] [2020-11-13 10:40:30] hold_for_later_1
[START] [2020-11-13 10:40:30] hold_for_later_2
[STOP] [2020-11-13 10:40:30] hold_for_later_2
[START] [2020-11-13 10:40:30] resolve_missing_parents
[STOP] [2020-11-13 10:40:30] resolve_missing_parents
[START] [2020-11-13 10:40:30] rebuild_nodes
[START] [2020-11-13 10:40:30] Flattener#flatten
[START] [2020-11-13 10:40:30] Flattener#study_resource
[START] [2020-11-13 10:40:30] Flattener#build_ancestry
[STOP] [2020-11-13 10:40:30] Flattener#build_ancestry
[INFO] [2020-11-13 10:40:30] 7 ancestry keys
[START] [2020-11-13 10:40:30] build_node_ancestors
[INFO] [2020-11-13 10:40:30] old ancestors deleted.
[STOP] [2020-11-13 10:40:30] build_node_ancestors
[WARN] [2020-11-13 10:40:30] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-11-13 10:40:30] Flattener#flatten
[STOP] [2020-11-13 10:40:30] rebuild_nodes
[START] [2020-11-13 10:40:30] resolve_missing_media_owners
[STOP] [2020-11-13 10:40:30] resolve_missing_media_owners
[START] [2020-11-13 10:40:30] sanitize_media_verbatims
[STOP] [2020-11-13 10:40:30] sanitize_media_verbatims
[START] [2020-11-13 10:40:30] queue_downloads
[STOP] [2020-11-13 10:40:30] queue_downloads
[START] [2020-11-13 10:40:30] parse_names
[WARN] [2020-11-13 10:40:30] I see 7 names which still need to be parsed.
[STOP] [2020-11-13 10:40:31] parse_names
[START] [2020-11-13 10:40:31] denormalize_canonical_names_to_nodes
[STOP] [2020-11-13 10:40:31] denormalize_canonical_names_to_nodes
[START] [2020-11-13 10:40:31] match_nodes
[START] [2020-11-13 10:40:31] map_all_nodes_to_pages
[STOP] [2020-11-13 10:40:31] map_all_nodes_to_pages
[INFO] [2020-11-13 10:40:31] ZERO unmatched nodes (of 7)! Nicely done.
[START] [2020-11-13 10:40:31] update_nodes
[STOP] [2020-11-13 10:40:31] update_nodes
[STOP] [2020-11-13 10:40:31] match_nodes
[START] [2020-11-13 10:40:31] reindex_search
[STOP] [2020-11-13 10:40:31] reindex_search
[START] [2020-11-13 10:40:31] normalize_units
[STOP] [2020-11-13 10:40:31] normalize_units
[START] [2020-11-13 10:40:31] calculate_statistics
[2020-11-13 10:40:31] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-11-13 10:40:31] calculate_statistics
[START] [2020-11-13 10:40:31] complete_harvest_instance
[START] [2020-11-13 10:40:31] overall_tsv_creation
[INFO] [2020-11-13 10:40:31] Processing group of 7 in 1 batches of 10000
[INFO] [2020-11-13 10:41:11] 14 Traits (unfiltered)...
[INFO] [2020-11-13 10:41:49] 14 Traits (filtered)...
[INFO] [2020-11-13 10:41:49] 0 Associations (filtered)...
[INFO] [2020-11-13 10:41:49] 28 metadata added.
[INFO] [2020-11-13 10:41:49] 0 metadata added.
[INFO] [2020-11-13 10:41:49] Average Time: 52.98
[INFO] [2020-11-13 10:41:49] Total Time: 1m18s
[STOP] [2020-11-13 10:41:49] overall_tsv_creation
[INFO] [2020-11-13 10:41:49] Done. Check your files:
[INFO] [2020-11-13 10:41:49] (7 lines) /app/public/data/renous_et_al_ren/publish_nodes.tsv
[INFO] [2020-11-13 10:41:49] (7 lines) /app/public/data/renous_et_al_ren/publish_scientific_names.tsv
[INFO] [2020-11-13 10:41:49] (15 lines) /app/public/data/renous_et_al_ren/publish_traits.tsv
[INFO] [2020-11-13 10:41:49] (29 lines) /app/public/data/renous_et_al_ren/publish_metadata.tsv
[STOP] [2020-11-13 10:41:49] complete_harvest_instance
[START] [2020-11-13 10:41:49] completed
[STOP] [2020-11-13 10:41:49] completed
[STOP] [2020-11-13 10:41:49] logged process, took 87.06
[INFO] [2021-04-19 09:26:42] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-19 09:26:43] ## remove_type: ScientificName
[INFO] [2021-04-19 09:26:43] ++ Calling delete_all on 7 instances...
[INFO] [2021-04-19 09:26:43] [09:26:43.616] Removed 7 Scientificnames
[INFO] [2021-04-19 09:26:43] ## remove_type: Vernacular
[INFO] [2021-04-19 09:26:43] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:26:43] [09:26:43.618] Removed 0 Vernaculars
[INFO] [2021-04-19 09:26:43] ## remove_type: Article
[INFO] [2021-04-19 09:26:43] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:26:43] [09:26:43.620] Removed 0 Articles
[INFO] [2021-04-19 09:26:43] ## remove_type: Medium
[INFO] [2021-04-19 09:26:43] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:26:43] [09:26:43.621] Removed 0 Media
[INFO] [2021-04-19 09:26:43] ## remove_type: Trait
[INFO] [2021-04-19 09:26:43] ++ Calling delete_all on 28 instances...
[INFO] [2021-04-19 09:26:43] [09:26:43.632] Removed 28 Traits
[INFO] [2021-04-19 09:26:43] ## remove_type: MetaTrait
[INFO] [2021-04-19 09:26:43] ++ Calling delete_all on 14 instances...
[INFO] [2021-04-19 09:26:43] [09:26:43.634] Removed 14 Metatraits
[INFO] [2021-04-19 09:26:43] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-19 09:26:43] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:26:43] [09:26:43.636] Removed 0 Occurrencemetadata
[INFO] [2021-04-19 09:26:43] ## remove_type: Assoc
[INFO] [2021-04-19 09:26:43] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:26:43] [09:26:43.637] Removed 0 Assocs
[INFO] [2021-04-19 09:26:43] ## remove_type: MetaAssoc
[INFO] [2021-04-19 09:26:43] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:26:43] [09:26:43.639] Removed 0 Metaassocs
[INFO] [2021-04-19 09:26:43] ## remove_type: Identifier
[INFO] [2021-04-19 09:26:43] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:26:43] [09:26:43.640] Removed 0 Identifiers
[INFO] [2021-04-19 09:26:43] ## remove_type: Reference
[INFO] [2021-04-19 09:26:43] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:26:43] [09:26:43.642] Removed 0 References
[INFO] [2021-04-19 09:26:43] Starting batch with ID 81658599...
[INFO] [2021-04-19 09:26:43] Starting batch with ID 81658599...
[INFO] [2021-04-19 09:26:43] Starting batch with ID 81658599...
[INFO] [2021-04-19 09:26:43] Starting batch with ID 81658599...
[INFO] [2021-04-19 09:26:43] Starting batch with ID 81658599...
[INFO] [2021-04-19 09:26:43] Starting batch with ID 81658599...
[INFO] [2021-04-19 09:26:43] Starting batch with ID 81658599...
[INFO] [2021-04-19 09:26:43] Starting batch with ID 81658599...
[INFO] [2021-04-19 09:26:43] Starting batch with ID 81658599...
[INFO] [2021-04-19 09:26:44] Starting batch with ID 81658601...
[INFO] [2021-04-19 09:26:44] ## remove_type: Node
[INFO] [2021-04-19 09:26:44] ++ Calling delete_all on 7 instances...
[INFO] [2021-04-19 09:26:44] [09:26:44.020] Removed 7 Nodes
[START] [2021-04-19 09:26:44] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 09:26:44] Creating resource from OpenData
[START] [2021-04-19 09:26:45] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 09:26:45] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 09:26:49] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 09:26:49] Creating resource from OpenData
[START] [2021-04-19 09:26:49] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 09:26:49] create_harvest_instance
[INFO] [2021-04-19 09:26:49] Created harvest instance #3715
[STOP] [2021-04-19 09:26:49] create_harvest_instance
[START] [2021-04-19 09:26:49] fetch_files
[STOP] [2021-04-19 09:26:49] fetch_files
[START] [2021-04-19 09:26:49] validate_each_file
[INFO] [2021-04-19 09:26:49] Looping over 3 formats...
[INFO] [2021-04-19 09:26:49] ...nodes (/app/public/data/renous_et_al_ren/taxa.txt)
[INFO] [2021-04-19 09:26:49] Valid: /app/public/converted_csv/renous_et_al_ren_nodes_3715.csv (7 lines)
[INFO] [2021-04-19 09:26:49] ...occurrences (/app/public/data/renous_et_al_ren/occurrences.txt)
[INFO] [2021-04-19 09:26:49] Valid: /app/public/converted_csv/renous_et_al_ren_occurrences_3715.csv (7 lines)
[INFO] [2021-04-19 09:26:49] ...measurements (/app/public/data/renous_et_al_ren/measurementsorfacts.txt)
[INFO] [2021-04-19 09:26:49] Valid: /app/public/converted_csv/renous_et_al_ren_measurements_3715.csv (28 lines)
[STOP] [2021-04-19 09:26:49] validate_each_file
[START] [2021-04-19 09:26:49] convert_to_csv
[INFO] [2021-04-19 09:26:49] Looping over 3 formats...
[INFO] [2021-04-19 09:26:49] ...nodes (/app/public/data/renous_et_al_ren/taxa.txt)
[CMD] [2021-04-19 09:26:49] /usr/bin/sort /app/public/converted_csv/renous_et_al_ren_nodes_3715.csv > /app/public/converted_csv/renous_et_al_ren_nodes_3715.csv_sorted
[INFO] [2021-04-19 09:26:49] Converted: /app/public/converted_csv/renous_et_al_ren_nodes_3715.csv (7 lines)
[INFO] [2021-04-19 09:26:49] ...occurrences (/app/public/data/renous_et_al_ren/occurrences.txt)
[CMD] [2021-04-19 09:26:49] /usr/bin/sort /app/public/converted_csv/renous_et_al_ren_occurrences_3715.csv > /app/public/converted_csv/renous_et_al_ren_occurrences_3715.csv_sorted
[INFO] [2021-04-19 09:26:50] Converted: /app/public/converted_csv/renous_et_al_ren_occurrences_3715.csv (7 lines)
[INFO] [2021-04-19 09:26:50] ...measurements (/app/public/data/renous_et_al_ren/measurementsorfacts.txt)
[CMD] [2021-04-19 09:26:50] /usr/bin/sort /app/public/converted_csv/renous_et_al_ren_measurements_3715.csv > /app/public/converted_csv/renous_et_al_ren_measurements_3715.csv_sorted
[INFO] [2021-04-19 09:26:50] Converted: /app/public/converted_csv/renous_et_al_ren_measurements_3715.csv (28 lines)
[STOP] [2021-04-19 09:26:50] convert_to_csv
[START] [2021-04-19 09:26:50] calculate_delta
[INFO] [2021-04-19 09:26:50] Looping over 3 formats...
[INFO] [2021-04-19 09:26:50] ...nodes (/app/public/data/renous_et_al_ren/taxa.txt)
[CMD] [2021-04-19 09:26:50] echo "0a" > /app/public/diff/renous_et_al_ren_nodes_3715.diff
[CMD] [2021-04-19 09:26:51] tail -n +1 /app/public/converted_csv/renous_et_al_ren_nodes_3715.csv >> /app/public/diff/renous_et_al_ren_nodes_3715.diff
[CMD] [2021-04-19 09:26:51] echo "." >> /app/public/diff/renous_et_al_ren_nodes_3715.diff
[INFO] [2021-04-19 09:26:51] Created diff: /app/public/diff/renous_et_al_ren_nodes_3715.diff (9 lines)
[INFO] [2021-04-19 09:26:51] ...occurrences (/app/public/data/renous_et_al_ren/occurrences.txt)
[CMD] [2021-04-19 09:26:51] echo "0a" > /app/public/diff/renous_et_al_ren_occurrences_3715.diff
[CMD] [2021-04-19 09:26:52] tail -n +1 /app/public/converted_csv/renous_et_al_ren_occurrences_3715.csv >> /app/public/diff/renous_et_al_ren_occurrences_3715.diff
[CMD] [2021-04-19 09:26:52] echo "." >> /app/public/diff/renous_et_al_ren_occurrences_3715.diff
[INFO] [2021-04-19 09:26:53] Created diff: /app/public/diff/renous_et_al_ren_occurrences_3715.diff (9 lines)
[INFO] [2021-04-19 09:26:53] ...measurements (/app/public/data/renous_et_al_ren/measurementsorfacts.txt)
[CMD] [2021-04-19 09:26:53] echo "0a" > /app/public/diff/renous_et_al_ren_measurements_3715.diff
[CMD] [2021-04-19 09:26:53] tail -n +1 /app/public/converted_csv/renous_et_al_ren_measurements_3715.csv >> /app/public/diff/renous_et_al_ren_measurements_3715.diff
[CMD] [2021-04-19 09:26:53] echo "." >> /app/public/diff/renous_et_al_ren_measurements_3715.diff
[INFO] [2021-04-19 09:26:54] Created diff: /app/public/diff/renous_et_al_ren_measurements_3715.diff (30 lines)
[STOP] [2021-04-19 09:26:54] calculate_delta
[START] [2021-04-19 09:26:54] parse_diff_and_store
[INFO] [2021-04-19 09:26:54] Handling diff: /app/public/diff/renous_et_al_ren_nodes_3715.diff (9 lines)
[INFO] [2021-04-19 09:26:54] Loading nodes diff file into memory (9 /app/public/diff/renous_et_al_ren_nodes_3715.diff lines)...
[INFO] [2021-04-19 09:26:55] Handling diff: /app/public/diff/renous_et_al_ren_occurrences_3715.diff (9 lines)
[INFO] [2021-04-19 09:26:55] Loading occurrences diff file into memory (9 /app/public/diff/renous_et_al_ren_occurrences_3715.diff lines)...
[INFO] [2021-04-19 09:26:55] Handling diff: /app/public/diff/renous_et_al_ren_measurements_3715.diff (30 lines)
[INFO] [2021-04-19 09:26:56] Loading measurements diff file into memory (30 /app/public/diff/renous_et_al_ren_measurements_3715.diff lines)...
[INFO] [2021-04-19 09:26:56] Storing 7 ScientificNames
[INFO] [2021-04-19 09:26:56] Processing group of 7 in 1 groups of 1000
[INFO] [2021-04-19 09:26:56] Average Time: 0.0
[INFO] [2021-04-19 09:26:56] Total Time: 1s
[INFO] [2021-04-19 09:26:56] Storing 7 Nodes
[INFO] [2021-04-19 09:26:56] Processing group of 7 in 1 groups of 1000
[INFO] [2021-04-19 09:26:56] Average Time: 0.0
[INFO] [2021-04-19 09:26:56] Total Time: 1s
[INFO] [2021-04-19 09:26:56] Storing 7 Occurrences
[INFO] [2021-04-19 09:26:56] Processing group of 7 in 1 groups of 1000
[INFO] [2021-04-19 09:26:56] Average Time: 0.0
[INFO] [2021-04-19 09:26:56] Total Time: 1s
[INFO] [2021-04-19 09:26:56] Storing 28 Traits
[INFO] [2021-04-19 09:26:56] Processing group of 28 in 1 groups of 1000
[INFO] [2021-04-19 09:26:56] Average Time: 0.02
[INFO] [2021-04-19 09:26:56] Total Time: 1s
[INFO] [2021-04-19 09:26:56] Storing 14 MetaTraits
[INFO] [2021-04-19 09:26:56] Processing group of 14 in 1 groups of 1000
[INFO] [2021-04-19 09:26:56] Average Time: 0.0
[INFO] [2021-04-19 09:26:56] Total Time: 1s
[STOP] [2021-04-19 09:26:56] parse_diff_and_store
[START] [2021-04-19 09:26:56] resolve_keys
[INFO] [2021-04-19 09:27:02] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-19 09:27:02] traits to occurrences...
[INFO] [2021-04-19 09:27:02] traits to nodes (through occurrences)...
[INFO] [2021-04-19 09:27:02] Traits to sex term...
[INFO] [2021-04-19 09:27:02] Traits to lifestage term...
[INFO] [2021-04-19 09:27:02] MetaTraits to traits...
[INFO] [2021-04-19 09:27:02] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-19 09:27:02] Assocs to occurrences...
[INFO] [2021-04-19 09:27:02] Assocs to nodes...
[INFO] [2021-04-19 09:27:02] Assoc to sex term...
[INFO] [2021-04-19 09:27:02] Assoc to lifestage term...
[INFO] [2021-04-19 09:27:02] MetaAssoc to assocs...
[STOP] [2021-04-19 09:27:02] resolve_keys
[START] [2021-04-19 09:27:02] hold_for_later_1
[STOP] [2021-04-19 09:27:02] hold_for_later_1
[START] [2021-04-19 09:27:02] hold_for_later_2
[STOP] [2021-04-19 09:27:02] hold_for_later_2
[START] [2021-04-19 09:27:02] resolve_missing_parents
[STOP] [2021-04-19 09:27:02] resolve_missing_parents
[START] [2021-04-19 09:27:03] rebuild_nodes
[START] [2021-04-19 09:27:03] Flattener#flatten
[START] [2021-04-19 09:27:03] Flattener#study_resource
[START] [2021-04-19 09:27:03] Flattener#build_ancestry
[STOP] [2021-04-19 09:27:03] Flattener#build_ancestry
[INFO] [2021-04-19 09:27:03] 7 ancestry keys
[START] [2021-04-19 09:27:03] build_node_ancestors
[INFO] [2021-04-19 09:27:03] old ancestors deleted.
[STOP] [2021-04-19 09:27:03] build_node_ancestors
[WARN] [2021-04-19 09:27:03] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2021-04-19 09:27:03] Flattener#flatten
[STOP] [2021-04-19 09:27:03] rebuild_nodes
[START] [2021-04-19 09:27:03] resolve_missing_media_owners
[STOP] [2021-04-19 09:27:03] resolve_missing_media_owners
[START] [2021-04-19 09:27:03] sanitize_media_verbatims
[STOP] [2021-04-19 09:27:03] sanitize_media_verbatims
[START] [2021-04-19 09:27:03] queue_downloads
[STOP] [2021-04-19 09:27:03] queue_downloads
[START] [2021-04-19 09:27:03] parse_names
[WARN] [2021-04-19 09:27:03] I see 7 names which still need to be parsed.
[STOP] [2021-04-19 09:27:04] parse_names
[START] [2021-04-19 09:27:04] denormalize_canonical_names_to_nodes
[STOP] [2021-04-19 09:27:04] denormalize_canonical_names_to_nodes
[START] [2021-04-19 09:27:04] match_nodes
[START] [2021-04-19 09:27:04] map_all_nodes_to_pages
[STOP] [2021-04-19 09:27:04] map_all_nodes_to_pages
[INFO] [2021-04-19 09:27:04] ZERO unmatched nodes (of 7)! Nicely done.
[START] [2021-04-19 09:27:04] update_nodes
[STOP] [2021-04-19 09:27:04] update_nodes
[STOP] [2021-04-19 09:27:04] match_nodes
[START] [2021-04-19 09:27:04] reindex_search
[STOP] [2021-04-19 09:27:04] reindex_search
[START] [2021-04-19 09:27:04] normalize_units
[STOP] [2021-04-19 09:27:04] normalize_units
[START] [2021-04-19 09:27:04] calculate_statistics
[2021-04-19 09:27:04] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2021-04-19 09:27:04] calculate_statistics
[START] [2021-04-19 09:27:04] complete_harvest_instance
[START] [2021-04-19 09:27:04] overall_tsv_creation
[INFO] [2021-04-19 09:27:04] Processing group of 7 in 1 batches of 10000
[INFO] [2021-04-19 09:27:40] 14 Traits (unfiltered)...
[INFO] [2021-04-19 09:28:14] 14 Traits (filtered)...
[INFO] [2021-04-19 09:28:14] 0 Associations (filtered)...
[INFO] [2021-04-19 09:28:14] 14 metadata added.
[INFO] [2021-04-19 09:28:14] 0 metadata added.
[INFO] [2021-04-19 09:28:41] Average Time: 72.35
[INFO] [2021-04-19 09:28:41] Total Time: 1m37s
[STOP] [2021-04-19 09:28:41] overall_tsv_creation
[INFO] [2021-04-19 09:28:41] Done. Check your files:
[INFO] [2021-04-19 09:28:41] (7 lines) /app/public/data/renous_et_al_ren/publish_nodes.tsv
[INFO] [2021-04-19 09:28:42] (7 lines) /app/public/data/renous_et_al_ren/publish_scientific_names.tsv
[INFO] [2021-04-19 09:28:42] (15 lines) /app/public/data/renous_et_al_ren/publish_traits.tsv
[INFO] [2021-04-19 09:28:42] (15 lines) /app/public/data/renous_et_al_ren/publish_metadata.tsv
[STOP] [2021-04-19 09:28:42] complete_harvest_instance
[START] [2021-04-19 09:28:42] completed
[STOP] [2021-04-19 09:28:42] completed
[STOP] [2021-04-19 09:28:42] logged process, took 113.81

Latest Process