Harvest for Travis et al 2017 Created 21 Jul 11:13

Harvesting Log

(1020 lines) (showing only the last 1000 lines, see /app/public/data/travis_et_al_tra/process.log for the full file)

[START] [2020-07-17 16:37:06] calculate_delta
[CMD] [2020-07-17 16:37:06] echo "0a" > /app/public/diff/travis_et_al_tra_measurements_21885.diff
[CMD] [2020-07-17 16:37:06] tail -n +1 /app/public/converted_csv/travis_et_al_tra_measurements_21885.csv >> /app/public/diff/travis_et_al_tra_measurements_21885.diff
[CMD] [2020-07-17 16:37:06] echo "." >> /app/public/diff/travis_et_al_tra_measurements_21885.diff
[STOP] [2020-07-17 16:37:06] calculate_delta
[START] [2020-07-17 16:37:06] parse_diff_and_store
[INFO] [2020-07-17 16:37:06] Loading measurements diff file into memory (true lines)...
[INFO] [2020-07-17 16:37:07] Storing 12 Traits
[INFO] [2020-07-17 16:37:07] Processing group of 12 in 1 groups of 1000
[INFO] [2020-07-17 16:37:07] Average Time: 0.01
[INFO] [2020-07-17 16:37:07] Total Time: 1s
[INFO] [2020-07-17 16:37:07] Storing 36 MetaTraits
[INFO] [2020-07-17 16:37:07] Processing group of 36 in 1 groups of 1000
[INFO] [2020-07-17 16:37:07] Average Time: 0.01
[INFO] [2020-07-17 16:37:07] Total Time: 1s
[STOP] [2020-07-17 16:37:07] parse_diff_and_store
[START] [2020-07-17 16:37:07] resolve_keys
[INFO] [2020-07-17 16:37:13] Occurrences to nodes (through scientific_names)...
[INFO] [2020-07-17 16:37:13] traits to occurrences...
[INFO] [2020-07-17 16:37:13] traits to nodes (through occurrences)...
[INFO] [2020-07-17 16:37:13] Traits to sex term...
[INFO] [2020-07-17 16:37:13] Traits to lifestage term...
[INFO] [2020-07-17 16:37:13] MetaTraits to traits...
[INFO] [2020-07-17 16:37:13] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-07-17 16:37:13] Assocs to occurrences...
[INFO] [2020-07-17 16:37:13] Assocs to nodes...
[INFO] [2020-07-17 16:37:13] Assoc to sex term...
[INFO] [2020-07-17 16:37:13] Assoc to lifestage term...
[STOP] [2020-07-17 16:37:13] resolve_keys
[START] [2020-07-17 16:37:13] hold_for_later_1
[STOP] [2020-07-17 16:37:13] hold_for_later_1
[START] [2020-07-17 16:37:13] hold_for_later_2
[STOP] [2020-07-17 16:37:13] hold_for_later_2
[START] [2020-07-17 16:37:13] resolve_missing_parents
[STOP] [2020-07-17 16:37:13] resolve_missing_parents
[START] [2020-07-17 16:37:13] rebuild_nodes
[START] [2020-07-17 16:37:13] Flattener#flatten
[START] [2020-07-17 16:37:13] Flattener#study_resource
[WARN] [2020-07-17 16:37:13] Flattener: no children found, aborting flatten
[STOP] [2020-07-17 16:37:13] Flattener#flatten
[STOP] [2020-07-17 16:37:13] rebuild_nodes
[START] [2020-07-17 16:37:13] resolve_missing_media_owners
[STOP] [2020-07-17 16:37:13] resolve_missing_media_owners
[START] [2020-07-17 16:37:13] sanitize_media_verbatims
[STOP] [2020-07-17 16:37:13] sanitize_media_verbatims
[START] [2020-07-17 16:37:13] queue_downloads
[STOP] [2020-07-17 16:37:13] queue_downloads
[START] [2020-07-17 16:37:13] parse_names
[STOP] [2020-07-17 16:37:13] parse_names
[START] [2020-07-17 16:37:13] denormalize_canonical_names_to_nodes
[STOP] [2020-07-17 16:37:13] denormalize_canonical_names_to_nodes
[START] [2020-07-17 16:37:13] match_nodes
[START] [2020-07-17 16:37:13] map_all_nodes_to_pages
[STOP] [2020-07-17 16:37:13] map_all_nodes_to_pages
[INFO] [2020-07-17 16:37:13] ZERO unmatched nodes (of 0)! Nicely done.
[START] [2020-07-17 16:37:13] update_nodes
[STOP] [2020-07-17 16:37:13] update_nodes
[STOP] [2020-07-17 16:37:13] match_nodes
[START] [2020-07-17 16:37:13] reindex_search
[STOP] [2020-07-17 16:37:13] reindex_search
[START] [2020-07-17 16:37:13] normalize_units
[STOP] [2020-07-17 16:37:13] normalize_units
[START] [2020-07-17 16:37:13] calculate_statistics
[ERR] [2020-07-17 16:37:13] Exception
[ERR] [2020-07-17 16:37:13] ZERO NODES! That is probably... bad.
[STOP] [2020-07-17 16:37:13] calculate_statistics
[STOP] [2020-07-17 16:37:13] logged process, took 7.79
[INFO] [2020-07-17 17:21:58] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-07-17 17:22:02] ## remove_type: ScientificName
[INFO] [2020-07-17 17:22:02] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-17 17:22:02] [17:22:02.512] Removed 0 Scientificnames
[INFO] [2020-07-17 17:22:02] ## remove_type: Vernacular
[INFO] [2020-07-17 17:22:02] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-17 17:22:02] [17:22:02.515] Removed 0 Vernaculars
[INFO] [2020-07-17 17:22:02] ## remove_type: Article
[INFO] [2020-07-17 17:22:02] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-17 17:22:02] [17:22:02.519] Removed 0 Articles
[INFO] [2020-07-17 17:22:02] ## remove_type: Medium
[INFO] [2020-07-17 17:22:02] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-17 17:22:02] [17:22:02.523] Removed 0 Media
[INFO] [2020-07-17 17:22:02] ## remove_type: Trait
[INFO] [2020-07-17 17:22:02] ++ Calling delete_all on 12 instances...
[INFO] [2020-07-17 17:22:02] [17:22:02.528] Removed 12 Traits
[INFO] [2020-07-17 17:22:02] ## remove_type: MetaTrait
[INFO] [2020-07-17 17:22:02] ++ Calling delete_all on 36 instances...
[INFO] [2020-07-17 17:22:02] [17:22:02.531] Removed 36 Metatraits
[INFO] [2020-07-17 17:22:02] ## remove_type: OccurrenceMetadatum
[INFO] [2020-07-17 17:22:02] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-17 17:22:02] [17:22:02.534] Removed 0 Occurrencemetadata
[INFO] [2020-07-17 17:22:02] ## remove_type: Assoc
[INFO] [2020-07-17 17:22:02] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-17 17:22:02] [17:22:02.537] Removed 0 Assocs
[INFO] [2020-07-17 17:22:02] ## remove_type: MetaAssoc
[INFO] [2020-07-17 17:22:02] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-17 17:22:02] [17:22:02.540] Removed 0 Metaassocs
[INFO] [2020-07-17 17:22:02] ## remove_type: Identifier
[INFO] [2020-07-17 17:22:02] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-17 17:22:02] [17:22:02.543] Removed 0 Identifiers
[INFO] [2020-07-17 17:22:02] ## remove_type: Reference
[INFO] [2020-07-17 17:22:02] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-17 17:22:02] [17:22:02.545] Removed 0 References
[INFO] [2020-07-17 17:22:02] ## remove_type: Node
[INFO] [2020-07-17 17:22:02] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-17 17:22:02] [17:22:02.567] Removed 0 Nodes
[START] [2020-07-17 17:22:02] logged process
[START] [2020-07-17 17:22:02] Creating resource from OpenData
[START] [2020-07-17 17:22:02] logged process
[START] [2020-07-17 17:22:02] Parse meta.xml file and create formats with fields
[STOP] [2020-07-17 17:22:02] Parse meta.xml file and create formats with fields
[STOP] [2020-07-17 17:22:02] Creating resource from OpenData
[START] [2020-07-17 17:22:02] logged process
[START] [2020-07-17 17:22:02] create_harvest_instance
[STOP] [2020-07-17 17:22:03] create_harvest_instance
[START] [2020-07-17 17:22:03] fetch_files
[STOP] [2020-07-17 17:22:03] fetch_files
[START] [2020-07-17 17:22:03] validate_each_file
[STOP] [2020-07-17 17:22:04] validate_each_file
[START] [2020-07-17 17:22:04] convert_to_csv
[CMD] [2020-07-17 17:22:04] /usr/bin/sort /app/public/converted_csv/travis_et_al_tra_nodes_21897.csv > /app/public/converted_csv/travis_et_al_tra_nodes_21897.csv_sorted
[CMD] [2020-07-17 17:22:04] /usr/bin/sort /app/public/converted_csv/travis_et_al_tra_occurrences_21898.csv > /app/public/converted_csv/travis_et_al_tra_occurrences_21898.csv_sorted
[CMD] [2020-07-17 17:22:04] /usr/bin/sort /app/public/converted_csv/travis_et_al_tra_measurements_21899.csv > /app/public/converted_csv/travis_et_al_tra_measurements_21899.csv_sorted
[STOP] [2020-07-17 17:22:04] convert_to_csv
[START] [2020-07-17 17:22:04] calculate_delta
[CMD] [2020-07-17 17:22:04] echo "0a" > /app/public/diff/travis_et_al_tra_nodes_21897.diff
[CMD] [2020-07-17 17:22:04] tail -n +1 /app/public/converted_csv/travis_et_al_tra_nodes_21897.csv >> /app/public/diff/travis_et_al_tra_nodes_21897.diff
[CMD] [2020-07-17 17:22:04] echo "." >> /app/public/diff/travis_et_al_tra_nodes_21897.diff
[CMD] [2020-07-17 17:22:04] echo "0a" > /app/public/diff/travis_et_al_tra_occurrences_21898.diff
[CMD] [2020-07-17 17:22:04] tail -n +1 /app/public/converted_csv/travis_et_al_tra_occurrences_21898.csv >> /app/public/diff/travis_et_al_tra_occurrences_21898.diff
[CMD] [2020-07-17 17:22:04] echo "." >> /app/public/diff/travis_et_al_tra_occurrences_21898.diff
[CMD] [2020-07-17 17:22:04] echo "0a" > /app/public/diff/travis_et_al_tra_measurements_21899.diff
[CMD] [2020-07-17 17:22:04] tail -n +1 /app/public/converted_csv/travis_et_al_tra_measurements_21899.csv >> /app/public/diff/travis_et_al_tra_measurements_21899.diff
[CMD] [2020-07-17 17:22:04] echo "." >> /app/public/diff/travis_et_al_tra_measurements_21899.diff
[STOP] [2020-07-17 17:22:04] calculate_delta
[START] [2020-07-17 17:22:04] parse_diff_and_store
[INFO] [2020-07-17 17:22:04] Loading nodes diff file into memory (true lines)...
[INFO] [2020-07-17 17:22:04] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-07-17 17:22:04] Loading measurements diff file into memory (true lines)...
[INFO] [2020-07-17 17:22:04] Storing 12 ScientificNames
[INFO] [2020-07-17 17:22:04] Processing group of 12 in 1 groups of 1000
[INFO] [2020-07-17 17:22:04] Average Time: 0.01
[INFO] [2020-07-17 17:22:04] Total Time: 1s
[INFO] [2020-07-17 17:22:04] Storing 12 Nodes
[INFO] [2020-07-17 17:22:04] Processing group of 12 in 1 groups of 1000
[INFO] [2020-07-17 17:22:04] Average Time: 0.01
[INFO] [2020-07-17 17:22:04] Total Time: 1s
[INFO] [2020-07-17 17:22:04] Storing 12 Occurrences
[INFO] [2020-07-17 17:22:04] Processing group of 12 in 1 groups of 1000
[INFO] [2020-07-17 17:22:04] Average Time: 0.0
[INFO] [2020-07-17 17:22:04] Total Time: 1s
[INFO] [2020-07-17 17:22:04] Storing 12 Traits
[INFO] [2020-07-17 17:22:04] Processing group of 12 in 1 groups of 1000
[INFO] [2020-07-17 17:22:04] Average Time: 0.0
[INFO] [2020-07-17 17:22:04] Total Time: 1s
[INFO] [2020-07-17 17:22:04] Storing 36 MetaTraits
[INFO] [2020-07-17 17:22:04] Processing group of 36 in 1 groups of 1000
[INFO] [2020-07-17 17:22:04] Average Time: 0.01
[INFO] [2020-07-17 17:22:04] Total Time: 1s
[STOP] [2020-07-17 17:22:04] parse_diff_and_store
[START] [2020-07-17 17:22:04] resolve_keys
[INFO] [2020-07-17 17:22:11] Occurrences to nodes (through scientific_names)...
[INFO] [2020-07-17 17:22:11] traits to occurrences...
[INFO] [2020-07-17 17:22:11] traits to nodes (through occurrences)...
[INFO] [2020-07-17 17:22:11] Traits to sex term...
[INFO] [2020-07-17 17:22:11] Traits to lifestage term...
[INFO] [2020-07-17 17:22:11] MetaTraits to traits...
[INFO] [2020-07-17 17:22:11] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-07-17 17:22:11] Assocs to occurrences...
[INFO] [2020-07-17 17:22:11] Assocs to nodes...
[INFO] [2020-07-17 17:22:11] Assoc to sex term...
[INFO] [2020-07-17 17:22:11] Assoc to lifestage term...
[STOP] [2020-07-17 17:22:11] resolve_keys
[START] [2020-07-17 17:22:11] hold_for_later_1
[STOP] [2020-07-17 17:22:11] hold_for_later_1
[START] [2020-07-17 17:22:11] hold_for_later_2
[STOP] [2020-07-17 17:22:11] hold_for_later_2
[START] [2020-07-17 17:22:11] resolve_missing_parents
[STOP] [2020-07-17 17:22:11] resolve_missing_parents
[START] [2020-07-17 17:22:11] rebuild_nodes
[START] [2020-07-17 17:22:11] Flattener#flatten
[START] [2020-07-17 17:22:11] Flattener#study_resource
[START] [2020-07-17 17:22:11] Flattener#build_ancestry
[STOP] [2020-07-17 17:22:11] Flattener#build_ancestry
[INFO] [2020-07-17 17:22:11] 12 ancestry keys
[START] [2020-07-17 17:22:11] build_node_ancestors
[INFO] [2020-07-17 17:22:11] old ancestors deleted.
[STOP] [2020-07-17 17:22:11] build_node_ancestors
[WARN] [2020-07-17 17:22:11] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-07-17 17:22:11] Flattener#flatten
[STOP] [2020-07-17 17:22:11] rebuild_nodes
[START] [2020-07-17 17:22:11] resolve_missing_media_owners
[STOP] [2020-07-17 17:22:11] resolve_missing_media_owners
[START] [2020-07-17 17:22:11] sanitize_media_verbatims
[STOP] [2020-07-17 17:22:11] sanitize_media_verbatims
[START] [2020-07-17 17:22:11] queue_downloads
[STOP] [2020-07-17 17:22:11] queue_downloads
[START] [2020-07-17 17:22:11] parse_names
[WARN] [2020-07-17 17:22:11] I see 12 names which still need to be parsed.
[STOP] [2020-07-17 17:22:12] parse_names
[START] [2020-07-17 17:22:12] denormalize_canonical_names_to_nodes
[STOP] [2020-07-17 17:22:12] denormalize_canonical_names_to_nodes
[START] [2020-07-17 17:22:12] match_nodes
[START] [2020-07-17 17:22:12] map_all_nodes_to_pages
[STOP] [2020-07-17 17:22:13] map_all_nodes_to_pages
[INFO] [2020-07-17 17:22:13] ZERO unmatched nodes (of 12)! Nicely done.
[START] [2020-07-17 17:22:13] update_nodes
[STOP] [2020-07-17 17:22:13] update_nodes
[STOP] [2020-07-17 17:22:13] match_nodes
[START] [2020-07-17 17:22:13] reindex_search
[STOP] [2020-07-17 17:22:13] reindex_search
[START] [2020-07-17 17:22:13] normalize_units
[STOP] [2020-07-17 17:22:13] normalize_units
[START] [2020-07-17 17:22:13] calculate_statistics
[2020-07-17 17:22:13] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-07-17 17:22:13] calculate_statistics
[START] [2020-07-17 17:22:13] complete_harvest_instance
[START] [2020-07-17 17:22:13] overall_tsv_creation
[INFO] [2020-07-17 17:22:13] Processing group of 12 in 1 batches of 10000
[INFO] [2020-07-17 17:23:02] 12 Traits (unfiltered)...
[INFO] [2020-07-17 17:23:38] 12 Traits (filtered)...
[INFO] [2020-07-17 17:23:38] 0 Associations (filtered)...
[INFO] [2020-07-17 17:23:38] 36 metadata added.
[INFO] [2020-07-17 17:23:38] 0 metadata added.
[INFO] [2020-07-17 17:23:38] Average Time: 57.91
[INFO] [2020-07-17 17:23:38] Total Time: 1m26s
[STOP] [2020-07-17 17:23:38] overall_tsv_creation
[INFO] [2020-07-17 17:23:38] Done. Check your files:
[INFO] [2020-07-17 17:23:38] (12 lines) /app/public/data/travis_et_al_tra/publish_nodes.tsv
[INFO] [2020-07-17 17:23:38] (12 lines) /app/public/data/travis_et_al_tra/publish_scientific_names.tsv
[INFO] [2020-07-17 17:23:38] (13 lines) /app/public/data/travis_et_al_tra/publish_traits.tsv
[INFO] [2020-07-17 17:23:38] (1 lines) /app/public/data/travis_et_al_tra/publish_metadata.tsv
[STOP] [2020-07-17 17:23:38] complete_harvest_instance
[START] [2020-07-17 17:23:38] completed
[STOP] [2020-07-17 17:23:38] completed
[STOP] [2020-07-17 17:23:38] logged process, took 95.64
[INFO] [2020-07-21 10:40:27] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-07-21 10:40:31] ## remove_type: ScientificName
[INFO] [2020-07-21 10:40:31] ++ Calling delete_all on 12 instances...
[INFO] [2020-07-21 10:40:31] [10:40:31.899] Removed 12 Scientificnames
[INFO] [2020-07-21 10:40:31] ## remove_type: Vernacular
[INFO] [2020-07-21 10:40:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 10:40:31] [10:40:31.902] Removed 0 Vernaculars
[INFO] [2020-07-21 10:40:31] ## remove_type: Article
[INFO] [2020-07-21 10:40:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 10:40:31] [10:40:31.905] Removed 0 Articles
[INFO] [2020-07-21 10:40:31] ## remove_type: Medium
[INFO] [2020-07-21 10:40:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 10:40:31] [10:40:31.908] Removed 0 Media
[INFO] [2020-07-21 10:40:31] ## remove_type: Trait
[INFO] [2020-07-21 10:40:31] ++ Calling delete_all on 12 instances...
[INFO] [2020-07-21 10:40:31] [10:40:31.913] Removed 12 Traits
[INFO] [2020-07-21 10:40:31] ## remove_type: MetaTrait
[INFO] [2020-07-21 10:40:31] ++ Calling delete_all on 36 instances...
[INFO] [2020-07-21 10:40:31] [10:40:31.916] Removed 36 Metatraits
[INFO] [2020-07-21 10:40:31] ## remove_type: OccurrenceMetadatum
[INFO] [2020-07-21 10:40:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 10:40:31] [10:40:31.919] Removed 0 Occurrencemetadata
[INFO] [2020-07-21 10:40:31] ## remove_type: Assoc
[INFO] [2020-07-21 10:40:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 10:40:31] [10:40:31.922] Removed 0 Assocs
[INFO] [2020-07-21 10:40:31] ## remove_type: MetaAssoc
[INFO] [2020-07-21 10:40:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 10:40:31] [10:40:31.925] Removed 0 Metaassocs
[INFO] [2020-07-21 10:40:31] ## remove_type: Identifier
[INFO] [2020-07-21 10:40:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 10:40:31] [10:40:31.928] Removed 0 Identifiers
[INFO] [2020-07-21 10:40:31] ## remove_type: Reference
[INFO] [2020-07-21 10:40:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 10:40:31] [10:40:31.931] Removed 0 References
[INFO] [2020-07-21 10:40:31] Starting batch with ID 80576762...
[INFO] [2020-07-21 10:40:32] Starting batch with ID 80576762...
[INFO] [2020-07-21 10:40:32] Starting batch with ID 80576762...
[INFO] [2020-07-21 10:40:32] Starting batch with ID 80576762...
[INFO] [2020-07-21 10:40:32] Starting batch with ID 80576762...
[INFO] [2020-07-21 10:40:32] Starting batch with ID 80576762...
[INFO] [2020-07-21 10:40:32] Starting batch with ID 80576762...
[INFO] [2020-07-21 10:40:32] ## remove_type: Node
[INFO] [2020-07-21 10:40:32] ++ Calling delete_all on 12 instances...
[INFO] [2020-07-21 10:40:32] [10:40:32.416] Removed 12 Nodes
[START] [2020-07-21 10:40:32] logged process
[START] [2020-07-21 10:40:32] Creating resource from OpenData
[START] [2020-07-21 10:40:32] logged process
[START] [2020-07-21 10:40:32] Parse meta.xml file and create formats with fields
[STOP] [2020-07-21 10:40:33] Parse meta.xml file and create formats with fields
[STOP] [2020-07-21 10:40:33] Creating resource from OpenData
[START] [2020-07-21 10:40:33] logged process
[START] [2020-07-21 10:40:33] create_harvest_instance
[STOP] [2020-07-21 10:40:34] create_harvest_instance
[START] [2020-07-21 10:40:34] fetch_files
[STOP] [2020-07-21 10:40:34] fetch_files
[START] [2020-07-21 10:40:34] validate_each_file
[STOP] [2020-07-21 10:40:34] validate_each_file
[START] [2020-07-21 10:40:34] convert_to_csv
[CMD] [2020-07-21 10:40:34] /usr/bin/sort /app/public/converted_csv/travis_et_al_tra_nodes_21927.csv > /app/public/converted_csv/travis_et_al_tra_nodes_21927.csv_sorted
[CMD] [2020-07-21 10:40:34] /usr/bin/sort /app/public/converted_csv/travis_et_al_tra_occurrences_21928.csv > /app/public/converted_csv/travis_et_al_tra_occurrences_21928.csv_sorted
[CMD] [2020-07-21 10:40:34] /usr/bin/sort /app/public/converted_csv/travis_et_al_tra_measurements_21929.csv > /app/public/converted_csv/travis_et_al_tra_measurements_21929.csv_sorted
[STOP] [2020-07-21 10:40:34] convert_to_csv
[START] [2020-07-21 10:40:34] calculate_delta
[CMD] [2020-07-21 10:40:34] echo "0a" > /app/public/diff/travis_et_al_tra_nodes_21927.diff
[CMD] [2020-07-21 10:40:34] tail -n +1 /app/public/converted_csv/travis_et_al_tra_nodes_21927.csv >> /app/public/diff/travis_et_al_tra_nodes_21927.diff
[CMD] [2020-07-21 10:40:34] echo "." >> /app/public/diff/travis_et_al_tra_nodes_21927.diff
[CMD] [2020-07-21 10:40:34] echo "0a" > /app/public/diff/travis_et_al_tra_occurrences_21928.diff
[CMD] [2020-07-21 10:40:34] tail -n +1 /app/public/converted_csv/travis_et_al_tra_occurrences_21928.csv >> /app/public/diff/travis_et_al_tra_occurrences_21928.diff
[CMD] [2020-07-21 10:40:34] echo "." >> /app/public/diff/travis_et_al_tra_occurrences_21928.diff
[CMD] [2020-07-21 10:40:34] echo "0a" > /app/public/diff/travis_et_al_tra_measurements_21929.diff
[CMD] [2020-07-21 10:40:34] tail -n +1 /app/public/converted_csv/travis_et_al_tra_measurements_21929.csv >> /app/public/diff/travis_et_al_tra_measurements_21929.diff
[CMD] [2020-07-21 10:40:34] echo "." >> /app/public/diff/travis_et_al_tra_measurements_21929.diff
[STOP] [2020-07-21 10:40:34] calculate_delta
[START] [2020-07-21 10:40:34] parse_diff_and_store
[INFO] [2020-07-21 10:40:34] Loading nodes diff file into memory (true lines)...
[INFO] [2020-07-21 10:40:34] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-07-21 10:40:34] Loading measurements diff file into memory (true lines)...
[INFO] [2020-07-21 10:40:34] Storing 12 ScientificNames
[INFO] [2020-07-21 10:40:34] Processing group of 12 in 1 groups of 1000
[INFO] [2020-07-21 10:40:34] Average Time: 0.01
[INFO] [2020-07-21 10:40:34] Total Time: 1s
[INFO] [2020-07-21 10:40:34] Storing 12 Nodes
[INFO] [2020-07-21 10:40:34] Processing group of 12 in 1 groups of 1000
[INFO] [2020-07-21 10:40:34] Average Time: 0.0
[INFO] [2020-07-21 10:40:34] Total Time: 1s
[INFO] [2020-07-21 10:40:34] Storing 12 Occurrences
[INFO] [2020-07-21 10:40:34] Processing group of 12 in 1 groups of 1000
[INFO] [2020-07-21 10:40:34] Average Time: 0.0
[INFO] [2020-07-21 10:40:34] Total Time: 1s
[INFO] [2020-07-21 10:40:34] Storing 12 Traits
[INFO] [2020-07-21 10:40:34] Processing group of 12 in 1 groups of 1000
[INFO] [2020-07-21 10:40:34] Average Time: 0.0
[INFO] [2020-07-21 10:40:34] Total Time: 1s
[INFO] [2020-07-21 10:40:34] Storing 36 MetaTraits
[INFO] [2020-07-21 10:40:34] Processing group of 36 in 1 groups of 1000
[INFO] [2020-07-21 10:40:34] Average Time: 0.01
[INFO] [2020-07-21 10:40:34] Total Time: 1s
[STOP] [2020-07-21 10:40:34] parse_diff_and_store
[START] [2020-07-21 10:40:34] resolve_keys
[INFO] [2020-07-21 10:40:41] Occurrences to nodes (through scientific_names)...
[INFO] [2020-07-21 10:40:41] traits to occurrences...
[INFO] [2020-07-21 10:40:41] traits to nodes (through occurrences)...
[INFO] [2020-07-21 10:40:41] Traits to sex term...
[INFO] [2020-07-21 10:40:41] Traits to lifestage term...
[INFO] [2020-07-21 10:40:41] MetaTraits to traits...
[INFO] [2020-07-21 10:40:41] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-07-21 10:40:41] Assocs to occurrences...
[INFO] [2020-07-21 10:40:41] Assocs to nodes...
[INFO] [2020-07-21 10:40:41] Assoc to sex term...
[INFO] [2020-07-21 10:40:41] Assoc to lifestage term...
[STOP] [2020-07-21 10:40:41] resolve_keys
[START] [2020-07-21 10:40:41] hold_for_later_1
[STOP] [2020-07-21 10:40:41] hold_for_later_1
[START] [2020-07-21 10:40:41] hold_for_later_2
[STOP] [2020-07-21 10:40:41] hold_for_later_2
[START] [2020-07-21 10:40:41] resolve_missing_parents
[STOP] [2020-07-21 10:40:41] resolve_missing_parents
[START] [2020-07-21 10:40:41] rebuild_nodes
[START] [2020-07-21 10:40:41] Flattener#flatten
[START] [2020-07-21 10:40:41] Flattener#study_resource
[START] [2020-07-21 10:40:41] Flattener#build_ancestry
[STOP] [2020-07-21 10:40:41] Flattener#build_ancestry
[INFO] [2020-07-21 10:40:41] 12 ancestry keys
[START] [2020-07-21 10:40:41] build_node_ancestors
[INFO] [2020-07-21 10:40:41] old ancestors deleted.
[STOP] [2020-07-21 10:40:41] build_node_ancestors
[WARN] [2020-07-21 10:40:41] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-07-21 10:40:41] Flattener#flatten
[STOP] [2020-07-21 10:40:41] rebuild_nodes
[START] [2020-07-21 10:40:41] resolve_missing_media_owners
[STOP] [2020-07-21 10:40:41] resolve_missing_media_owners
[START] [2020-07-21 10:40:41] sanitize_media_verbatims
[STOP] [2020-07-21 10:40:41] sanitize_media_verbatims
[START] [2020-07-21 10:40:41] queue_downloads
[STOP] [2020-07-21 10:40:41] queue_downloads
[START] [2020-07-21 10:40:41] parse_names
[WARN] [2020-07-21 10:40:41] I see 12 names which still need to be parsed.
[STOP] [2020-07-21 10:40:42] parse_names
[START] [2020-07-21 10:40:42] denormalize_canonical_names_to_nodes
[STOP] [2020-07-21 10:40:42] denormalize_canonical_names_to_nodes
[START] [2020-07-21 10:40:42] match_nodes
[START] [2020-07-21 10:40:42] map_all_nodes_to_pages
[STOP] [2020-07-21 10:40:43] map_all_nodes_to_pages
[INFO] [2020-07-21 10:40:43] ZERO unmatched nodes (of 12)! Nicely done.
[START] [2020-07-21 10:40:43] update_nodes
[STOP] [2020-07-21 10:40:43] update_nodes
[STOP] [2020-07-21 10:40:43] match_nodes
[START] [2020-07-21 10:40:43] reindex_search
[STOP] [2020-07-21 10:40:43] reindex_search
[START] [2020-07-21 10:40:43] normalize_units
[STOP] [2020-07-21 10:40:43] normalize_units
[START] [2020-07-21 10:40:43] calculate_statistics
[2020-07-21 10:40:43] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-07-21 10:40:43] calculate_statistics
[START] [2020-07-21 10:40:43] complete_harvest_instance
[START] [2020-07-21 10:40:43] overall_tsv_creation
[INFO] [2020-07-21 10:40:43] Processing group of 12 in 1 batches of 10000
[INFO] [2020-07-21 10:41:32] 12 Traits (unfiltered)...
[INFO] [2020-07-21 10:42:08] 12 Traits (filtered)...
[INFO] [2020-07-21 10:42:08] 0 Associations (filtered)...
[INFO] [2020-07-21 10:42:08] 36 metadata added.
[INFO] [2020-07-21 10:42:08] 0 metadata added.
[INFO] [2020-07-21 10:42:08] Average Time: 57.74
[INFO] [2020-07-21 10:42:08] Total Time: 1m25s
[STOP] [2020-07-21 10:42:08] overall_tsv_creation
[INFO] [2020-07-21 10:42:08] Done. Check your files:
[INFO] [2020-07-21 10:42:08] (12 lines) /app/public/data/travis_et_al_tra/publish_nodes.tsv
[INFO] [2020-07-21 10:42:08] (12 lines) /app/public/data/travis_et_al_tra/publish_scientific_names.tsv
[INFO] [2020-07-21 10:42:08] (13 lines) /app/public/data/travis_et_al_tra/publish_traits.tsv
[INFO] [2020-07-21 10:42:08] (1 lines) /app/public/data/travis_et_al_tra/publish_metadata.tsv
[STOP] [2020-07-21 10:42:08] complete_harvest_instance
[START] [2020-07-21 10:42:08] completed
[STOP] [2020-07-21 10:42:08] completed
[STOP] [2020-07-21 10:42:08] logged process, took 95.52
[INFO] [2020-07-21 11:04:28] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-07-21 11:04:30] ## remove_type: ScientificName
[INFO] [2020-07-21 11:04:30] ++ Calling delete_all on 12 instances...
[INFO] [2020-07-21 11:04:30] [11:04:30.659] Removed 12 Scientificnames
[INFO] [2020-07-21 11:04:30] ## remove_type: Vernacular
[INFO] [2020-07-21 11:04:30] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:04:30] [11:04:30.661] Removed 0 Vernaculars
[INFO] [2020-07-21 11:04:30] ## remove_type: Article
[INFO] [2020-07-21 11:04:30] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:04:30] [11:04:30.664] Removed 0 Articles
[INFO] [2020-07-21 11:04:30] ## remove_type: Medium
[INFO] [2020-07-21 11:04:30] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:04:30] [11:04:30.667] Removed 0 Media
[INFO] [2020-07-21 11:04:30] ## remove_type: Trait
[INFO] [2020-07-21 11:04:30] ++ Calling delete_all on 12 instances...
[INFO] [2020-07-21 11:04:30] [11:04:30.670] Removed 12 Traits
[INFO] [2020-07-21 11:04:30] ## remove_type: MetaTrait
[INFO] [2020-07-21 11:04:30] ++ Calling delete_all on 36 instances...
[INFO] [2020-07-21 11:04:30] [11:04:30.673] Removed 36 Metatraits
[INFO] [2020-07-21 11:04:30] ## remove_type: OccurrenceMetadatum
[INFO] [2020-07-21 11:04:30] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:04:30] [11:04:30.676] Removed 0 Occurrencemetadata
[INFO] [2020-07-21 11:04:30] ## remove_type: Assoc
[INFO] [2020-07-21 11:04:30] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:04:30] [11:04:30.679] Removed 0 Assocs
[INFO] [2020-07-21 11:04:30] ## remove_type: MetaAssoc
[INFO] [2020-07-21 11:04:30] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:04:30] [11:04:30.681] Removed 0 Metaassocs
[INFO] [2020-07-21 11:04:30] ## remove_type: Identifier
[INFO] [2020-07-21 11:04:30] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:04:30] [11:04:30.683] Removed 0 Identifiers
[INFO] [2020-07-21 11:04:30] ## remove_type: Reference
[INFO] [2020-07-21 11:04:30] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:04:30] [11:04:30.686] Removed 0 References
[INFO] [2020-07-21 11:04:30] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:30] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:30] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:30] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:30] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:30] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:30] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:30] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:30] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:30] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:30] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:30] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:30] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:30] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:30] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:31] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:31] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:31] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:31] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:31] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:31] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:31] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:31] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:31] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:31] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:31] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:31] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:31] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:31] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:31] Starting batch with ID 80632533...
[INFO] [2020-07-21 11:04:31] ## remove_type: Node
[INFO] [2020-07-21 11:04:31] ++ Calling delete_all on 12 instances...
[INFO] [2020-07-21 11:04:31] [11:04:31.868] Removed 12 Nodes
[START] [2020-07-21 11:04:32] logged process
[START] [2020-07-21 11:04:32] Creating resource from OpenData
[START] [2020-07-21 11:04:32] logged process
[START] [2020-07-21 11:04:32] Parse meta.xml file and create formats with fields
[STOP] [2020-07-21 11:04:32] Parse meta.xml file and create formats with fields
[STOP] [2020-07-21 11:04:32] Creating resource from OpenData
[START] [2020-07-21 11:04:32] logged process
[START] [2020-07-21 11:04:32] create_harvest_instance
[STOP] [2020-07-21 11:04:33] create_harvest_instance
[START] [2020-07-21 11:04:33] fetch_files
[STOP] [2020-07-21 11:04:33] fetch_files
[START] [2020-07-21 11:04:33] validate_each_file
[STOP] [2020-07-21 11:04:33] validate_each_file
[START] [2020-07-21 11:04:33] convert_to_csv
[CMD] [2020-07-21 11:04:33] /usr/bin/sort /app/public/converted_csv/travis_et_al_tra_nodes_21933.csv > /app/public/converted_csv/travis_et_al_tra_nodes_21933.csv_sorted
[CMD] [2020-07-21 11:04:33] /usr/bin/sort /app/public/converted_csv/travis_et_al_tra_occurrences_21934.csv > /app/public/converted_csv/travis_et_al_tra_occurrences_21934.csv_sorted
[CMD] [2020-07-21 11:04:33] /usr/bin/sort /app/public/converted_csv/travis_et_al_tra_measurements_21935.csv > /app/public/converted_csv/travis_et_al_tra_measurements_21935.csv_sorted
[STOP] [2020-07-21 11:04:33] convert_to_csv
[START] [2020-07-21 11:04:33] calculate_delta
[CMD] [2020-07-21 11:04:33] echo "0a" > /app/public/diff/travis_et_al_tra_nodes_21933.diff
[CMD] [2020-07-21 11:04:33] tail -n +1 /app/public/converted_csv/travis_et_al_tra_nodes_21933.csv >> /app/public/diff/travis_et_al_tra_nodes_21933.diff
[CMD] [2020-07-21 11:04:33] echo "." >> /app/public/diff/travis_et_al_tra_nodes_21933.diff
[CMD] [2020-07-21 11:04:33] echo "0a" > /app/public/diff/travis_et_al_tra_occurrences_21934.diff
[CMD] [2020-07-21 11:04:33] tail -n +1 /app/public/converted_csv/travis_et_al_tra_occurrences_21934.csv >> /app/public/diff/travis_et_al_tra_occurrences_21934.diff
[CMD] [2020-07-21 11:04:33] echo "." >> /app/public/diff/travis_et_al_tra_occurrences_21934.diff
[CMD] [2020-07-21 11:04:33] echo "0a" > /app/public/diff/travis_et_al_tra_measurements_21935.diff
[CMD] [2020-07-21 11:04:33] tail -n +1 /app/public/converted_csv/travis_et_al_tra_measurements_21935.csv >> /app/public/diff/travis_et_al_tra_measurements_21935.diff
[CMD] [2020-07-21 11:04:33] echo "." >> /app/public/diff/travis_et_al_tra_measurements_21935.diff
[STOP] [2020-07-21 11:04:33] calculate_delta
[START] [2020-07-21 11:04:33] parse_diff_and_store
[INFO] [2020-07-21 11:04:33] Loading nodes diff file into memory (true lines)...
[INFO] [2020-07-21 11:04:33] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-07-21 11:04:33] Loading measurements diff file into memory (true lines)...
[INFO] [2020-07-21 11:04:33] Storing 12 ScientificNames
[INFO] [2020-07-21 11:04:33] Processing group of 12 in 1 groups of 1000
[INFO] [2020-07-21 11:04:33] Average Time: 0.01
[INFO] [2020-07-21 11:04:33] Total Time: 1s
[INFO] [2020-07-21 11:04:33] Storing 12 Nodes
[INFO] [2020-07-21 11:04:33] Processing group of 12 in 1 groups of 1000
[INFO] [2020-07-21 11:04:33] Average Time: 0.0
[INFO] [2020-07-21 11:04:33] Total Time: 1s
[INFO] [2020-07-21 11:04:33] Storing 12 Occurrences
[INFO] [2020-07-21 11:04:33] Processing group of 12 in 1 groups of 1000
[INFO] [2020-07-21 11:04:33] Average Time: 0.0
[INFO] [2020-07-21 11:04:33] Total Time: 1s
[INFO] [2020-07-21 11:04:33] Storing 12 Traits
[INFO] [2020-07-21 11:04:33] Processing group of 12 in 1 groups of 1000
[INFO] [2020-07-21 11:04:33] Average Time: 0.0
[INFO] [2020-07-21 11:04:33] Total Time: 1s
[INFO] [2020-07-21 11:04:33] Storing 36 MetaTraits
[INFO] [2020-07-21 11:04:33] Processing group of 36 in 1 groups of 1000
[INFO] [2020-07-21 11:04:33] Average Time: 0.01
[INFO] [2020-07-21 11:04:33] Total Time: 1s
[STOP] [2020-07-21 11:04:33] parse_diff_and_store
[START] [2020-07-21 11:04:33] resolve_keys
[INFO] [2020-07-21 11:04:40] Occurrences to nodes (through scientific_names)...
[INFO] [2020-07-21 11:04:40] traits to occurrences...
[INFO] [2020-07-21 11:04:40] traits to nodes (through occurrences)...
[INFO] [2020-07-21 11:04:40] Traits to sex term...
[INFO] [2020-07-21 11:04:40] Traits to lifestage term...
[INFO] [2020-07-21 11:04:40] MetaTraits to traits...
[INFO] [2020-07-21 11:04:40] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-07-21 11:04:40] Assocs to occurrences...
[INFO] [2020-07-21 11:04:40] Assocs to nodes...
[INFO] [2020-07-21 11:04:40] Assoc to sex term...
[INFO] [2020-07-21 11:04:40] Assoc to lifestage term...
[STOP] [2020-07-21 11:04:40] resolve_keys
[START] [2020-07-21 11:04:40] hold_for_later_1
[STOP] [2020-07-21 11:04:40] hold_for_later_1
[START] [2020-07-21 11:04:40] hold_for_later_2
[STOP] [2020-07-21 11:04:40] hold_for_later_2
[START] [2020-07-21 11:04:40] resolve_missing_parents
[STOP] [2020-07-21 11:04:40] resolve_missing_parents
[START] [2020-07-21 11:04:40] rebuild_nodes
[START] [2020-07-21 11:04:40] Flattener#flatten
[START] [2020-07-21 11:04:40] Flattener#study_resource
[START] [2020-07-21 11:04:40] Flattener#build_ancestry
[STOP] [2020-07-21 11:04:40] Flattener#build_ancestry
[INFO] [2020-07-21 11:04:40] 12 ancestry keys
[START] [2020-07-21 11:04:40] build_node_ancestors
[INFO] [2020-07-21 11:04:40] old ancestors deleted.
[STOP] [2020-07-21 11:04:40] build_node_ancestors
[WARN] [2020-07-21 11:04:40] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-07-21 11:04:40] Flattener#flatten
[STOP] [2020-07-21 11:04:40] rebuild_nodes
[START] [2020-07-21 11:04:40] resolve_missing_media_owners
[STOP] [2020-07-21 11:04:40] resolve_missing_media_owners
[START] [2020-07-21 11:04:40] sanitize_media_verbatims
[STOP] [2020-07-21 11:04:40] sanitize_media_verbatims
[START] [2020-07-21 11:04:40] queue_downloads
[STOP] [2020-07-21 11:04:40] queue_downloads
[START] [2020-07-21 11:04:40] parse_names
[WARN] [2020-07-21 11:04:40] I see 12 names which still need to be parsed.
[STOP] [2020-07-21 11:04:41] parse_names
[START] [2020-07-21 11:04:41] denormalize_canonical_names_to_nodes
[STOP] [2020-07-21 11:04:41] denormalize_canonical_names_to_nodes
[START] [2020-07-21 11:04:41] match_nodes
[START] [2020-07-21 11:04:41] map_all_nodes_to_pages
[STOP] [2020-07-21 11:04:42] map_all_nodes_to_pages
[INFO] [2020-07-21 11:04:42] ZERO unmatched nodes (of 12)! Nicely done.
[START] [2020-07-21 11:04:42] update_nodes
[STOP] [2020-07-21 11:04:42] update_nodes
[STOP] [2020-07-21 11:04:42] match_nodes
[START] [2020-07-21 11:04:42] reindex_search
[STOP] [2020-07-21 11:04:42] reindex_search
[START] [2020-07-21 11:04:42] normalize_units
[STOP] [2020-07-21 11:04:42] normalize_units
[START] [2020-07-21 11:04:42] calculate_statistics
[2020-07-21 11:04:42] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-07-21 11:04:42] calculate_statistics
[START] [2020-07-21 11:04:42] complete_harvest_instance
[START] [2020-07-21 11:04:42] overall_tsv_creation
[INFO] [2020-07-21 11:04:42] Processing group of 12 in 1 batches of 10000
[INFO] [2020-07-21 11:05:31] 12 Traits (unfiltered)...
[INFO] [2020-07-21 11:06:07] 12 Traits (filtered)...
[INFO] [2020-07-21 11:06:07] 0 Associations (filtered)...
[INFO] [2020-07-21 11:06:07] 36 metadata added.
[INFO] [2020-07-21 11:06:07] 0 metadata added.
[INFO] [2020-07-21 11:06:07] Average Time: 57.59
[INFO] [2020-07-21 11:06:07] Total Time: 1m25s
[STOP] [2020-07-21 11:06:07] overall_tsv_creation
[INFO] [2020-07-21 11:06:07] Done. Check your files:
[INFO] [2020-07-21 11:06:07] (12 lines) /app/public/data/travis_et_al_tra/publish_nodes.tsv
[INFO] [2020-07-21 11:06:07] (12 lines) /app/public/data/travis_et_al_tra/publish_scientific_names.tsv
[INFO] [2020-07-21 11:06:07] (13 lines) /app/public/data/travis_et_al_tra/publish_traits.tsv
[INFO] [2020-07-21 11:06:07] (1 lines) /app/public/data/travis_et_al_tra/publish_metadata.tsv
[STOP] [2020-07-21 11:06:07] complete_harvest_instance
[START] [2020-07-21 11:06:07] completed
[STOP] [2020-07-21 11:06:07] completed
[STOP] [2020-07-21 11:06:07] logged process, took 94.87
[INFO] [2020-07-21 11:10:34] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-07-21 11:10:38] ## remove_type: ScientificName
[INFO] [2020-07-21 11:10:38] ++ Calling delete_all on 12 instances...
[INFO] [2020-07-21 11:10:38] [11:10:38.538] Removed 12 Scientificnames
[INFO] [2020-07-21 11:10:38] ## remove_type: Vernacular
[INFO] [2020-07-21 11:10:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:10:38] [11:10:38.541] Removed 0 Vernaculars
[INFO] [2020-07-21 11:10:38] ## remove_type: Article
[INFO] [2020-07-21 11:10:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:10:38] [11:10:38.544] Removed 0 Articles
[INFO] [2020-07-21 11:10:38] ## remove_type: Medium
[INFO] [2020-07-21 11:10:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:10:38] [11:10:38.548] Removed 0 Media
[INFO] [2020-07-21 11:10:38] ## remove_type: Trait
[INFO] [2020-07-21 11:10:38] ++ Calling delete_all on 12 instances...
[INFO] [2020-07-21 11:10:38] [11:10:38.552] Removed 12 Traits
[INFO] [2020-07-21 11:10:38] ## remove_type: MetaTrait
[INFO] [2020-07-21 11:10:38] ++ Calling delete_all on 36 instances...
[INFO] [2020-07-21 11:10:38] [11:10:38.555] Removed 36 Metatraits
[INFO] [2020-07-21 11:10:38] ## remove_type: OccurrenceMetadatum
[INFO] [2020-07-21 11:10:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:10:38] [11:10:38.558] Removed 0 Occurrencemetadata
[INFO] [2020-07-21 11:10:38] ## remove_type: Assoc
[INFO] [2020-07-21 11:10:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:10:38] [11:10:38.560] Removed 0 Assocs
[INFO] [2020-07-21 11:10:38] ## remove_type: MetaAssoc
[INFO] [2020-07-21 11:10:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:10:38] [11:10:38.563] Removed 0 Metaassocs
[INFO] [2020-07-21 11:10:38] ## remove_type: Identifier
[INFO] [2020-07-21 11:10:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:10:38] [11:10:38.566] Removed 0 Identifiers
[INFO] [2020-07-21 11:10:38] ## remove_type: Reference
[INFO] [2020-07-21 11:10:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:10:38] [11:10:38.569] Removed 0 References
[INFO] [2020-07-21 11:10:38] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:38] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:38] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:38] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:38] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:38] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:38] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:38] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:38] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:38] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:38] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:38] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:38] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:38] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:39] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:39] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:39] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:39] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:39] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:39] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:39] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:39] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:39] Starting batch with ID 80632541...
[INFO] [2020-07-21 11:10:39] Starting batch with ID 80632543...
[INFO] [2020-07-21 11:10:39] ## remove_type: Node
[INFO] [2020-07-21 11:10:39] ++ Calling delete_all on 12 instances...
[INFO] [2020-07-21 11:10:39] [11:10:39.230] Removed 12 Nodes
[START] [2020-07-21 11:10:39] logged process
[START] [2020-07-21 11:10:39] Creating resource from OpenData
[START] [2020-07-21 11:10:39] logged process
[START] [2020-07-21 11:10:39] Parse meta.xml file and create formats with fields
[STOP] [2020-07-21 11:10:39] Parse meta.xml file and create formats with fields
[STOP] [2020-07-21 11:10:39] Creating resource from OpenData
[START] [2020-07-21 11:10:39] logged process
[START] [2020-07-21 11:10:39] create_harvest_instance
[STOP] [2020-07-21 11:10:40] create_harvest_instance
[START] [2020-07-21 11:10:40] fetch_files
[STOP] [2020-07-21 11:10:40] fetch_files
[START] [2020-07-21 11:10:40] validate_each_file
[STOP] [2020-07-21 11:10:40] validate_each_file
[START] [2020-07-21 11:10:40] convert_to_csv
[CMD] [2020-07-21 11:10:40] /usr/bin/sort /app/public/converted_csv/travis_et_al_tra_nodes_21939.csv > /app/public/converted_csv/travis_et_al_tra_nodes_21939.csv_sorted
[CMD] [2020-07-21 11:10:40] /usr/bin/sort /app/public/converted_csv/travis_et_al_tra_occurrences_21940.csv > /app/public/converted_csv/travis_et_al_tra_occurrences_21940.csv_sorted
[CMD] [2020-07-21 11:10:40] /usr/bin/sort /app/public/converted_csv/travis_et_al_tra_measurements_21941.csv > /app/public/converted_csv/travis_et_al_tra_measurements_21941.csv_sorted
[STOP] [2020-07-21 11:10:40] convert_to_csv
[START] [2020-07-21 11:10:40] calculate_delta
[CMD] [2020-07-21 11:10:40] echo "0a" > /app/public/diff/travis_et_al_tra_nodes_21939.diff
[CMD] [2020-07-21 11:10:40] tail -n +1 /app/public/converted_csv/travis_et_al_tra_nodes_21939.csv >> /app/public/diff/travis_et_al_tra_nodes_21939.diff
[CMD] [2020-07-21 11:10:40] echo "." >> /app/public/diff/travis_et_al_tra_nodes_21939.diff
[CMD] [2020-07-21 11:10:40] echo "0a" > /app/public/diff/travis_et_al_tra_occurrences_21940.diff
[CMD] [2020-07-21 11:10:40] tail -n +1 /app/public/converted_csv/travis_et_al_tra_occurrences_21940.csv >> /app/public/diff/travis_et_al_tra_occurrences_21940.diff
[CMD] [2020-07-21 11:10:40] echo "." >> /app/public/diff/travis_et_al_tra_occurrences_21940.diff
[CMD] [2020-07-21 11:10:40] echo "0a" > /app/public/diff/travis_et_al_tra_measurements_21941.diff
[CMD] [2020-07-21 11:10:40] tail -n +1 /app/public/converted_csv/travis_et_al_tra_measurements_21941.csv >> /app/public/diff/travis_et_al_tra_measurements_21941.diff
[CMD] [2020-07-21 11:10:40] echo "." >> /app/public/diff/travis_et_al_tra_measurements_21941.diff
[STOP] [2020-07-21 11:10:40] calculate_delta
[START] [2020-07-21 11:10:40] parse_diff_and_store
[INFO] [2020-07-21 11:10:40] Loading nodes diff file into memory (true lines)...
[INFO] [2020-07-21 11:10:41] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-07-21 11:10:41] Loading measurements diff file into memory (true lines)...
[INFO] [2020-07-21 11:10:41] Storing 12 ScientificNames
[INFO] [2020-07-21 11:10:41] Processing group of 12 in 1 groups of 1000
[INFO] [2020-07-21 11:10:41] Average Time: 0.01
[INFO] [2020-07-21 11:10:41] Total Time: 1s
[INFO] [2020-07-21 11:10:41] Storing 12 Nodes
[INFO] [2020-07-21 11:10:41] Processing group of 12 in 1 groups of 1000
[INFO] [2020-07-21 11:10:41] Average Time: 0.0
[INFO] [2020-07-21 11:10:41] Total Time: 1s
[INFO] [2020-07-21 11:10:41] Storing 12 Occurrences
[INFO] [2020-07-21 11:10:41] Processing group of 12 in 1 groups of 1000
[INFO] [2020-07-21 11:10:41] Average Time: 0.0
[INFO] [2020-07-21 11:10:41] Total Time: 1s
[INFO] [2020-07-21 11:10:41] Storing 12 Traits
[INFO] [2020-07-21 11:10:41] Processing group of 12 in 1 groups of 1000
[INFO] [2020-07-21 11:10:41] Average Time: 0.0
[INFO] [2020-07-21 11:10:41] Total Time: 1s
[INFO] [2020-07-21 11:10:41] Storing 36 MetaTraits
[INFO] [2020-07-21 11:10:41] Processing group of 36 in 1 groups of 1000
[INFO] [2020-07-21 11:10:41] Average Time: 0.01
[INFO] [2020-07-21 11:10:41] Total Time: 1s
[STOP] [2020-07-21 11:10:41] parse_diff_and_store
[START] [2020-07-21 11:10:41] resolve_keys
[INFO] [2020-07-21 11:10:47] Occurrences to nodes (through scientific_names)...
[INFO] [2020-07-21 11:10:47] traits to occurrences...
[INFO] [2020-07-21 11:10:47] traits to nodes (through occurrences)...
[INFO] [2020-07-21 11:10:47] Traits to sex term...
[INFO] [2020-07-21 11:10:47] Traits to lifestage term...
[INFO] [2020-07-21 11:10:47] MetaTraits to traits...
[INFO] [2020-07-21 11:10:47] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-07-21 11:10:47] Assocs to occurrences...
[INFO] [2020-07-21 11:10:47] Assocs to nodes...
[INFO] [2020-07-21 11:10:47] Assoc to sex term...
[INFO] [2020-07-21 11:10:47] Assoc to lifestage term...
[STOP] [2020-07-21 11:10:47] resolve_keys
[START] [2020-07-21 11:10:47] hold_for_later_1
[STOP] [2020-07-21 11:10:47] hold_for_later_1
[START] [2020-07-21 11:10:47] hold_for_later_2
[STOP] [2020-07-21 11:10:47] hold_for_later_2
[START] [2020-07-21 11:10:47] resolve_missing_parents
[STOP] [2020-07-21 11:10:47] resolve_missing_parents
[START] [2020-07-21 11:10:47] rebuild_nodes
[START] [2020-07-21 11:10:47] Flattener#flatten
[START] [2020-07-21 11:10:47] Flattener#study_resource
[START] [2020-07-21 11:10:47] Flattener#build_ancestry
[STOP] [2020-07-21 11:10:47] Flattener#build_ancestry
[INFO] [2020-07-21 11:10:47] 12 ancestry keys
[START] [2020-07-21 11:10:47] build_node_ancestors
[INFO] [2020-07-21 11:10:47] old ancestors deleted.
[STOP] [2020-07-21 11:10:47] build_node_ancestors
[WARN] [2020-07-21 11:10:47] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-07-21 11:10:47] Flattener#flatten
[STOP] [2020-07-21 11:10:47] rebuild_nodes
[START] [2020-07-21 11:10:47] resolve_missing_media_owners
[STOP] [2020-07-21 11:10:47] resolve_missing_media_owners
[START] [2020-07-21 11:10:47] sanitize_media_verbatims
[STOP] [2020-07-21 11:10:47] sanitize_media_verbatims
[START] [2020-07-21 11:10:47] queue_downloads
[STOP] [2020-07-21 11:10:47] queue_downloads
[START] [2020-07-21 11:10:47] parse_names
[WARN] [2020-07-21 11:10:47] I see 12 names which still need to be parsed.
[STOP] [2020-07-21 11:10:49] parse_names
[START] [2020-07-21 11:10:49] denormalize_canonical_names_to_nodes
[STOP] [2020-07-21 11:10:49] denormalize_canonical_names_to_nodes
[START] [2020-07-21 11:10:49] match_nodes
[START] [2020-07-21 11:10:49] map_all_nodes_to_pages
[STOP] [2020-07-21 11:10:49] map_all_nodes_to_pages
[INFO] [2020-07-21 11:10:49] ZERO unmatched nodes (of 12)! Nicely done.
[START] [2020-07-21 11:10:49] update_nodes
[STOP] [2020-07-21 11:10:49] update_nodes
[STOP] [2020-07-21 11:10:49] match_nodes
[START] [2020-07-21 11:10:49] reindex_search
[STOP] [2020-07-21 11:10:49] reindex_search
[START] [2020-07-21 11:10:49] normalize_units
[STOP] [2020-07-21 11:10:49] normalize_units
[START] [2020-07-21 11:10:49] calculate_statistics
[2020-07-21 11:10:49] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-07-21 11:10:49] calculate_statistics
[START] [2020-07-21 11:10:49] complete_harvest_instance
[START] [2020-07-21 11:10:49] overall_tsv_creation
[INFO] [2020-07-21 11:10:49] Processing group of 12 in 1 batches of 10000
[INFO] [2020-07-21 11:11:39] 12 Traits (unfiltered)...
[INFO] [2020-07-21 11:12:15] 12 Traits (filtered)...
[INFO] [2020-07-21 11:12:15] 0 Associations (filtered)...
[INFO] [2020-07-21 11:12:15] 36 metadata added.
[INFO] [2020-07-21 11:12:15] 0 metadata added.
[INFO] [2020-07-21 11:12:15] Average Time: 58.09
[INFO] [2020-07-21 11:12:15] Total Time: 1m26s
[STOP] [2020-07-21 11:12:15] overall_tsv_creation
[INFO] [2020-07-21 11:12:15] Done. Check your files:
[INFO] [2020-07-21 11:12:15] (12 lines) /app/public/data/travis_et_al_tra/publish_nodes.tsv
[INFO] [2020-07-21 11:12:15] (12 lines) /app/public/data/travis_et_al_tra/publish_scientific_names.tsv
[INFO] [2020-07-21 11:12:15] (13 lines) /app/public/data/travis_et_al_tra/publish_traits.tsv
[INFO] [2020-07-21 11:12:15] (1 lines) /app/public/data/travis_et_al_tra/publish_metadata.tsv
[STOP] [2020-07-21 11:12:15] complete_harvest_instance
[START] [2020-07-21 11:12:15] completed
[STOP] [2020-07-21 11:12:15] completed
[STOP] [2020-07-21 11:12:15] logged process, took 95.31
[START] [2020-07-21 11:12:15] logged process
[START] [2020-07-21 11:12:15] overall_tsv_creation
[INFO] [2020-07-21 11:12:15] Processing group of 12 in 1 batches of 10000
[INFO] [2020-07-21 11:12:15] Average Time: NaN
[INFO] [2020-07-21 11:12:15] Total Time: 1s
[STOP] [2020-07-21 11:12:15] overall_tsv_creation
[INFO] [2020-07-21 11:12:24] ## HARVEST: type = re_download_opendata_-harvest
[START] [2020-07-21 11:12:25] logged process
[START] [2020-07-21 11:12:25] overall_tsv_creation
[INFO] [2020-07-21 11:12:25] Processing group of 12 in 1 batches of 10000
[INFO] [2020-07-21 11:13:14] 12 Traits (unfiltered)...
[INFO] [2020-07-21 11:13:50] 12 Traits (filtered)...
[INFO] [2020-07-21 11:13:50] 0 Associations (filtered)...
[INFO] [2020-07-21 11:13:50] 36 metadata added.
[INFO] [2020-07-21 11:13:50] 0 metadata added.
[INFO] [2020-07-21 11:13:50] Average Time: 57.81
[INFO] [2020-07-21 11:13:50] Total Time: 1m25s
[STOP] [2020-07-21 11:13:50] overall_tsv_creation
[INFO] [2020-07-21 11:13:50] Done. Check your files:
[INFO] [2020-07-21 11:13:50] (12 lines) /app/public/data/travis_et_al_tra/publish_nodes.tsv
[INFO] [2020-07-21 11:13:50] (12 lines) /app/public/data/travis_et_al_tra/publish_scientific_names.tsv
[INFO] [2020-07-21 11:13:50] (13 lines) /app/public/data/travis_et_al_tra/publish_traits.tsv
[INFO] [2020-07-21 11:13:50] (1 lines) /app/public/data/travis_et_al_tra/publish_metadata.tsv
[INFO] [2020-07-21 11:13:50] ## remove_type: ScientificName
[INFO] [2020-07-21 11:13:50] ++ Calling delete_all on 12 instances...
[INFO] [2020-07-21 11:13:50] [11:13:50.615] Removed 12 Scientificnames
[INFO] [2020-07-21 11:13:50] ## remove_type: Vernacular
[INFO] [2020-07-21 11:13:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:13:50] [11:13:50.616] Removed 0 Vernaculars
[INFO] [2020-07-21 11:13:50] ## remove_type: Article
[INFO] [2020-07-21 11:13:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:13:50] [11:13:50.618] Removed 0 Articles
[INFO] [2020-07-21 11:13:50] ## remove_type: Medium
[INFO] [2020-07-21 11:13:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:13:50] [11:13:50.619] Removed 0 Media
[INFO] [2020-07-21 11:13:50] ## remove_type: Trait
[INFO] [2020-07-21 11:13:50] ++ Calling delete_all on 12 instances...
[INFO] [2020-07-21 11:13:50] [11:13:50.622] Removed 12 Traits
[INFO] [2020-07-21 11:13:50] ## remove_type: MetaTrait
[INFO] [2020-07-21 11:13:50] ++ Calling delete_all on 36 instances...
[INFO] [2020-07-21 11:13:50] [11:13:50.625] Removed 36 Metatraits
[INFO] [2020-07-21 11:13:50] ## remove_type: OccurrenceMetadatum
[INFO] [2020-07-21 11:13:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:13:50] [11:13:50.627] Removed 0 Occurrencemetadata
[INFO] [2020-07-21 11:13:50] ## remove_type: Assoc
[INFO] [2020-07-21 11:13:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:13:50] [11:13:50.628] Removed 0 Assocs
[INFO] [2020-07-21 11:13:50] ## remove_type: MetaAssoc
[INFO] [2020-07-21 11:13:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:13:50] [11:13:50.629] Removed 0 Metaassocs
[INFO] [2020-07-21 11:13:50] ## remove_type: Identifier
[INFO] [2020-07-21 11:13:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:13:50] [11:13:50.630] Removed 0 Identifiers
[INFO] [2020-07-21 11:13:50] ## remove_type: Reference
[INFO] [2020-07-21 11:13:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-21 11:13:50] [11:13:50.632] Removed 0 References
[INFO] [2020-07-21 11:13:50] Starting batch with ID 80632558...
[INFO] [2020-07-21 11:13:50] Starting batch with ID 80632558...
[INFO] [2020-07-21 11:13:50] Starting batch with ID 80632558...
[INFO] [2020-07-21 11:13:50] Starting batch with ID 80632558...
[INFO] [2020-07-21 11:13:50] Starting batch with ID 80632558...
[INFO] [2020-07-21 11:13:50] Starting batch with ID 80632558...
[INFO] [2020-07-21 11:13:50] Starting batch with ID 80632558...
[INFO] [2020-07-21 11:13:50] Starting batch with ID 80632558...
[INFO] [2020-07-21 11:13:50] Starting batch with ID 80632558...
[INFO] [2020-07-21 11:13:50] Starting batch with ID 80632558...
[INFO] [2020-07-21 11:13:50] Starting batch with ID 80632558...
[INFO] [2020-07-21 11:13:50] Starting batch with ID 80632558...
[INFO] [2020-07-21 11:13:50] Starting batch with ID 80632558...
[INFO] [2020-07-21 11:13:50] Starting batch with ID 80632558...
[INFO] [2020-07-21 11:13:51] Starting batch with ID 80632558...
[INFO] [2020-07-21 11:13:51] ## remove_type: Node
[INFO] [2020-07-21 11:13:51] ++ Calling delete_all on 12 instances...
[INFO] [2020-07-21 11:13:51] [11:13:51.144] Removed 12 Nodes
[START] [2020-07-21 11:13:51] logged process
[START] [2020-07-21 11:13:51] Creating resource from OpenData
[START] [2020-07-21 11:13:51] logged process
[START] [2020-07-21 11:13:51] Parse meta.xml file and create formats with fields
[STOP] [2020-07-21 11:13:51] Parse meta.xml file and create formats with fields
[STOP] [2020-07-21 11:13:51] Creating resource from OpenData
[START] [2020-07-21 11:13:51] logged process
[START] [2020-07-21 11:13:51] create_harvest_instance
[STOP] [2020-07-21 11:13:52] create_harvest_instance
[START] [2020-07-21 11:13:52] fetch_files
[STOP] [2020-07-21 11:13:52] fetch_files
[START] [2020-07-21 11:13:52] validate_each_file
[STOP] [2020-07-21 11:13:52] validate_each_file
[START] [2020-07-21 11:13:52] convert_to_csv
[CMD] [2020-07-21 11:13:52] /usr/bin/sort /app/public/converted_csv/travis_et_al_tra_nodes_21945.csv > /app/public/converted_csv/travis_et_al_tra_nodes_21945.csv_sorted
[CMD] [2020-07-21 11:13:52] /usr/bin/sort /app/public/converted_csv/travis_et_al_tra_occurrences_21946.csv > /app/public/converted_csv/travis_et_al_tra_occurrences_21946.csv_sorted
[CMD] [2020-07-21 11:13:52] /usr/bin/sort /app/public/converted_csv/travis_et_al_tra_measurements_21947.csv > /app/public/converted_csv/travis_et_al_tra_measurements_21947.csv_sorted
[STOP] [2020-07-21 11:13:52] convert_to_csv
[START] [2020-07-21 11:13:52] calculate_delta
[CMD] [2020-07-21 11:13:52] echo "0a" > /app/public/diff/travis_et_al_tra_nodes_21945.diff
[CMD] [2020-07-21 11:13:52] tail -n +1 /app/public/converted_csv/travis_et_al_tra_nodes_21945.csv >> /app/public/diff/travis_et_al_tra_nodes_21945.diff
[CMD] [2020-07-21 11:13:52] echo "." >> /app/public/diff/travis_et_al_tra_nodes_21945.diff
[CMD] [2020-07-21 11:13:52] echo "0a" > /app/public/diff/travis_et_al_tra_occurrences_21946.diff
[CMD] [2020-07-21 11:13:52] tail -n +1 /app/public/converted_csv/travis_et_al_tra_occurrences_21946.csv >> /app/public/diff/travis_et_al_tra_occurrences_21946.diff
[CMD] [2020-07-21 11:13:52] echo "." >> /app/public/diff/travis_et_al_tra_occurrences_21946.diff
[CMD] [2020-07-21 11:13:52] echo "0a" > /app/public/diff/travis_et_al_tra_measurements_21947.diff
[CMD] [2020-07-21 11:13:52] tail -n +1 /app/public/converted_csv/travis_et_al_tra_measurements_21947.csv >> /app/public/diff/travis_et_al_tra_measurements_21947.diff
[CMD] [2020-07-21 11:13:52] echo "." >> /app/public/diff/travis_et_al_tra_measurements_21947.diff
[STOP] [2020-07-21 11:13:52] calculate_delta
[START] [2020-07-21 11:13:52] parse_diff_and_store
[INFO] [2020-07-21 11:13:52] Loading nodes diff file into memory (true lines)...
[INFO] [2020-07-21 11:13:52] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-07-21 11:13:52] Loading measurements diff file into memory (true lines)...
[INFO] [2020-07-21 11:13:52] Storing 12 ScientificNames
[INFO] [2020-07-21 11:13:52] Processing group of 12 in 1 groups of 1000
[INFO] [2020-07-21 11:13:52] Average Time: 0.01
[INFO] [2020-07-21 11:13:52] Total Time: 1s
[INFO] [2020-07-21 11:13:52] Storing 12 Nodes
[INFO] [2020-07-21 11:13:52] Processing group of 12 in 1 groups of 1000
[INFO] [2020-07-21 11:13:52] Average Time: 0.0
[INFO] [2020-07-21 11:13:52] Total Time: 1s
[INFO] [2020-07-21 11:13:52] Storing 12 Occurrences
[INFO] [2020-07-21 11:13:52] Processing group of 12 in 1 groups of 1000
[INFO] [2020-07-21 11:13:52] Average Time: 0.0
[INFO] [2020-07-21 11:13:52] Total Time: 1s
[INFO] [2020-07-21 11:13:52] Storing 12 Traits
[INFO] [2020-07-21 11:13:52] Processing group of 12 in 1 groups of 1000
[INFO] [2020-07-21 11:13:52] Average Time: 0.0
[INFO] [2020-07-21 11:13:52] Total Time: 1s
[INFO] [2020-07-21 11:13:52] Storing 36 MetaTraits
[INFO] [2020-07-21 11:13:52] Processing group of 36 in 1 groups of 1000
[INFO] [2020-07-21 11:13:52] Average Time: 0.01
[INFO] [2020-07-21 11:13:52] Total Time: 1s
[STOP] [2020-07-21 11:13:52] parse_diff_and_store
[START] [2020-07-21 11:13:52] resolve_keys
[INFO] [2020-07-21 11:13:59] Occurrences to nodes (through scientific_names)...
[INFO] [2020-07-21 11:13:59] traits to occurrences...
[INFO] [2020-07-21 11:13:59] traits to nodes (through occurrences)...
[INFO] [2020-07-21 11:13:59] Traits to sex term...
[INFO] [2020-07-21 11:13:59] Traits to lifestage term...
[INFO] [2020-07-21 11:13:59] MetaTraits to traits...
[INFO] [2020-07-21 11:13:59] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-07-21 11:13:59] Assocs to occurrences...
[INFO] [2020-07-21 11:13:59] Assocs to nodes...
[INFO] [2020-07-21 11:13:59] Assoc to sex term...
[INFO] [2020-07-21 11:13:59] Assoc to lifestage term...
[STOP] [2020-07-21 11:13:59] resolve_keys
[START] [2020-07-21 11:13:59] hold_for_later_1
[STOP] [2020-07-21 11:13:59] hold_for_later_1
[START] [2020-07-21 11:13:59] hold_for_later_2
[STOP] [2020-07-21 11:13:59] hold_for_later_2
[START] [2020-07-21 11:13:59] resolve_missing_parents
[STOP] [2020-07-21 11:13:59] resolve_missing_parents
[START] [2020-07-21 11:13:59] rebuild_nodes
[START] [2020-07-21 11:13:59] Flattener#flatten
[START] [2020-07-21 11:13:59] Flattener#study_resource
[START] [2020-07-21 11:13:59] Flattener#build_ancestry
[STOP] [2020-07-21 11:13:59] Flattener#build_ancestry
[INFO] [2020-07-21 11:13:59] 12 ancestry keys
[START] [2020-07-21 11:13:59] build_node_ancestors
[INFO] [2020-07-21 11:13:59] old ancestors deleted.
[STOP] [2020-07-21 11:13:59] build_node_ancestors
[WARN] [2020-07-21 11:13:59] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-07-21 11:13:59] Flattener#flatten
[STOP] [2020-07-21 11:13:59] rebuild_nodes
[START] [2020-07-21 11:13:59] resolve_missing_media_owners
[STOP] [2020-07-21 11:13:59] resolve_missing_media_owners
[START] [2020-07-21 11:13:59] sanitize_media_verbatims
[STOP] [2020-07-21 11:13:59] sanitize_media_verbatims
[START] [2020-07-21 11:13:59] queue_downloads
[STOP] [2020-07-21 11:13:59] queue_downloads
[START] [2020-07-21 11:13:59] parse_names
[WARN] [2020-07-21 11:13:59] I see 12 names which still need to be parsed.
[STOP] [2020-07-21 11:14:00] parse_names
[START] [2020-07-21 11:14:00] denormalize_canonical_names_to_nodes
[STOP] [2020-07-21 11:14:00] denormalize_canonical_names_to_nodes
[START] [2020-07-21 11:14:00] match_nodes
[START] [2020-07-21 11:14:00] map_all_nodes_to_pages
[STOP] [2020-07-21 11:14:01] map_all_nodes_to_pages
[INFO] [2020-07-21 11:14:01] ZERO unmatched nodes (of 12)! Nicely done.
[START] [2020-07-21 11:14:01] update_nodes
[STOP] [2020-07-21 11:14:01] update_nodes
[STOP] [2020-07-21 11:14:01] match_nodes
[START] [2020-07-21 11:14:01] reindex_search
[STOP] [2020-07-21 11:14:01] reindex_search
[START] [2020-07-21 11:14:01] normalize_units
[STOP] [2020-07-21 11:14:01] normalize_units
[START] [2020-07-21 11:14:01] calculate_statistics
[2020-07-21 11:14:01] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-07-21 11:14:01] calculate_statistics
[START] [2020-07-21 11:14:01] complete_harvest_instance
[START] [2020-07-21 11:14:01] overall_tsv_creation
[INFO] [2020-07-21 11:14:01] Processing group of 12 in 1 batches of 10000
[INFO] [2020-07-21 11:14:50] 12 Traits (unfiltered)...
[INFO] [2020-07-21 11:15:26] 12 Traits (filtered)...
[INFO] [2020-07-21 11:15:26] 0 Associations (filtered)...
[INFO] [2020-07-21 11:15:26] 36 metadata added.
[INFO] [2020-07-21 11:15:26] 0 metadata added.
[INFO] [2020-07-21 11:15:26] Average Time: 58.12
[INFO] [2020-07-21 11:15:26] Total Time: 1m26s
[STOP] [2020-07-21 11:15:26] overall_tsv_creation
[INFO] [2020-07-21 11:15:26] Done. Check your files:
[INFO] [2020-07-21 11:15:26] (12 lines) /app/public/data/travis_et_al_tra/publish_nodes.tsv
[INFO] [2020-07-21 11:15:26] (12 lines) /app/public/data/travis_et_al_tra/publish_scientific_names.tsv
[INFO] [2020-07-21 11:15:26] (13 lines) /app/public/data/travis_et_al_tra/publish_traits.tsv
[INFO] [2020-07-21 11:15:26] (1 lines) /app/public/data/travis_et_al_tra/publish_metadata.tsv
[STOP] [2020-07-21 11:15:26] complete_harvest_instance
[START] [2020-07-21 11:15:26] completed
[STOP] [2020-07-21 11:15:26] completed
[STOP] [2020-07-21 11:15:26] logged process, took 95.33

Latest Process

21 Jul 11:13 EXITED currently processing 0