Harvest for
Lamb et al 1987
Created
19 Apr 09:18
Stage:
completed
Fetched:
19 Apr 09:18
Validated:
19 Apr 09:18
Deltas Created
19 Apr 09:18
Units Normalized:
19 Apr 09:18
Ancestry Built:
19 Apr 09:18
Nodes Matched:
19 Apr 09:18
Names Parsed:
19 Apr 09:18
New Models Stored:
19 Apr 09:18
Indexed:
19 Apr 09:18
Completed:
19 Apr 09:20
Time to Harvest:
less than a minute
Harvesting Log
(774 lines)
# Logfile created on 2020-06-16 09:51:02 -0400 by logger.rb/v1.4.2
[INFO] [2020-06-16 09:51:02] ## HARVEST: type = -harvest
[START] [2020-06-16 09:51:03] logged process
[START] [2020-06-16 09:51:03] create_harvest_instance
[STOP] [2020-06-16 09:51:04] create_harvest_instance
[START] [2020-06-16 09:51:04] fetch_files
[STOP] [2020-06-16 09:51:04] fetch_files
[START] [2020-06-16 09:51:04] validate_each_file
[STOP] [2020-06-16 09:51:04] validate_each_file
[START] [2020-06-16 09:51:04] convert_to_csv
[CMD] [2020-06-16 09:51:04] /usr/bin/sort /app/public/converted_csv/lamb_et_al_lamb__nodes_21320.csv > /app/public/converted_csv/lamb_et_al_lamb__nodes_21320.csv_sorted
[CMD] [2020-06-16 09:51:04] /usr/bin/sort /app/public/converted_csv/lamb_et_al_lamb__occurrences_21321.csv > /app/public/converted_csv/lamb_et_al_lamb__occurrences_21321.csv_sorted
[CMD] [2020-06-16 09:51:04] /usr/bin/sort /app/public/converted_csv/lamb_et_al_lamb__measurements_21322.csv > /app/public/converted_csv/lamb_et_al_lamb__measurements_21322.csv_sorted
[STOP] [2020-06-16 09:51:04] convert_to_csv
[START] [2020-06-16 09:51:04] calculate_delta
[CMD] [2020-06-16 09:51:04] echo "0a" > /app/public/diff/lamb_et_al_lamb__nodes_21320.diff
[CMD] [2020-06-16 09:51:04] tail -n +1 /app/public/converted_csv/lamb_et_al_lamb__nodes_21320.csv >> /app/public/diff/lamb_et_al_lamb__nodes_21320.diff
[CMD] [2020-06-16 09:51:04] echo "." >> /app/public/diff/lamb_et_al_lamb__nodes_21320.diff
[CMD] [2020-06-16 09:51:04] echo "0a" > /app/public/diff/lamb_et_al_lamb__occurrences_21321.diff
[CMD] [2020-06-16 09:51:04] tail -n +1 /app/public/converted_csv/lamb_et_al_lamb__occurrences_21321.csv >> /app/public/diff/lamb_et_al_lamb__occurrences_21321.diff
[CMD] [2020-06-16 09:51:04] echo "." >> /app/public/diff/lamb_et_al_lamb__occurrences_21321.diff
[CMD] [2020-06-16 09:51:04] echo "0a" > /app/public/diff/lamb_et_al_lamb__measurements_21322.diff
[CMD] [2020-06-16 09:51:04] tail -n +1 /app/public/converted_csv/lamb_et_al_lamb__measurements_21322.csv >> /app/public/diff/lamb_et_al_lamb__measurements_21322.diff
[CMD] [2020-06-16 09:51:04] echo "." >> /app/public/diff/lamb_et_al_lamb__measurements_21322.diff
[STOP] [2020-06-16 09:51:04] calculate_delta
[START] [2020-06-16 09:51:04] parse_diff_and_store
[INFO] [2020-06-16 09:51:04] Loading nodes diff file into memory (true lines)...
[INFO] [2020-06-16 09:51:04] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-06-16 09:51:04] Loading measurements diff file into memory (true lines)...
[WARN] [2020-06-16 09:51:04] Created predicate term for http://purl.obolibrary.org/obo/RO_0003009!
[WARN] [2020-06-16 09:51:04] Created predicate term for http://purl.obolibrary.org/obo/RO_0003010!
[WARN] [2020-06-16 09:51:04] Created predicate term for http://purl.obolibrary.org/obo/RO_0003001!
[WARN] [2020-06-16 09:51:04] Created predicate term for http://purl.obolibrary.org/obo/RO_0003002!
[WARN] [2020-06-16 09:51:04] Created predicate term for http://purl.obolibrary.org/obo/RO_0003003!
[WARN] [2020-06-16 09:51:04] Created predicate term for http://purl.obolibrary.org/obo/RO_0003004!
[WARN] [2020-06-16 09:51:04] Created predicate term for http://purl.obolibrary.org/obo/RO_0003005!
[WARN] [2020-06-16 09:51:04] Created predicate term for http://purl.obolibrary.org/obo/RO_0003006!
[WARN] [2020-06-16 09:51:04] Created predicate term for http://purl.obolibrary.org/obo/RO_0003007!
[WARN] [2020-06-16 09:51:04] Created predicate term for http://purl.obolibrary.org/obo/RO_0003008!
[INFO] [2020-06-16 09:51:04] Storing 26 ScientificNames
[INFO] [2020-06-16 09:51:04] Processing group of 26 in 1 groups of 1000
[INFO] [2020-06-16 09:51:05] Average Time: 0.01
[INFO] [2020-06-16 09:51:05] Total Time: 1s
[INFO] [2020-06-16 09:51:05] Storing 26 Nodes
[INFO] [2020-06-16 09:51:05] Processing group of 26 in 1 groups of 1000
[INFO] [2020-06-16 09:51:05] Average Time: 0.02
[INFO] [2020-06-16 09:51:05] Total Time: 1s
[INFO] [2020-06-16 09:51:05] Storing 11 Occurrences
[INFO] [2020-06-16 09:51:05] Processing group of 11 in 1 groups of 1000
[INFO] [2020-06-16 09:51:05] Average Time: 0.0
[INFO] [2020-06-16 09:51:05] Total Time: 1s
[INFO] [2020-06-16 09:51:05] Storing 22 Traits
[INFO] [2020-06-16 09:51:05] Processing group of 22 in 1 groups of 1000
[INFO] [2020-06-16 09:51:05] Average Time: 0.01
[INFO] [2020-06-16 09:51:05] Total Time: 1s
[INFO] [2020-06-16 09:51:05] Storing 33 MetaTraits
[INFO] [2020-06-16 09:51:05] Processing group of 33 in 1 groups of 1000
[INFO] [2020-06-16 09:51:05] Average Time: 0.01
[INFO] [2020-06-16 09:51:05] Total Time: 1s
[STOP] [2020-06-16 09:51:05] parse_diff_and_store
[START] [2020-06-16 09:51:05] resolve_keys
[INFO] [2020-06-16 09:51:11] Occurrences to nodes (through scientific_names)...
[INFO] [2020-06-16 09:51:11] traits to occurrences...
[INFO] [2020-06-16 09:51:11] traits to nodes (through occurrences)...
[INFO] [2020-06-16 09:51:11] Traits to sex term...
[INFO] [2020-06-16 09:51:11] Traits to lifestage term...
[INFO] [2020-06-16 09:51:11] MetaTraits to traits...
[INFO] [2020-06-16 09:51:11] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-06-16 09:51:11] Assocs to occurrences...
[INFO] [2020-06-16 09:51:11] Assocs to nodes...
[INFO] [2020-06-16 09:51:11] Assoc to sex term...
[INFO] [2020-06-16 09:51:11] Assoc to lifestage term...
[STOP] [2020-06-16 09:51:11] resolve_keys
[START] [2020-06-16 09:51:11] hold_for_later_1
[STOP] [2020-06-16 09:51:11] hold_for_later_1
[START] [2020-06-16 09:51:11] hold_for_later_2
[STOP] [2020-06-16 09:51:11] hold_for_later_2
[START] [2020-06-16 09:51:11] resolve_missing_parents
[STOP] [2020-06-16 09:51:11] resolve_missing_parents
[START] [2020-06-16 09:51:11] rebuild_nodes
[START] [2020-06-16 09:51:11] Flattener#flatten
[START] [2020-06-16 09:51:11] Flattener#study_resource
[START] [2020-06-16 09:51:11] Flattener#build_ancestry
[STOP] [2020-06-16 09:51:11] Flattener#build_ancestry
[INFO] [2020-06-16 09:51:11] 26 ancestry keys
[START] [2020-06-16 09:51:11] build_node_ancestors
[INFO] [2020-06-16 09:51:11] old ancestors deleted.
[STOP] [2020-06-16 09:51:12] build_node_ancestors
[START] [2020-06-16 09:51:12] Flattener#propagate_ancestor_ids
[STOP] [2020-06-16 09:51:12] Flattener#propagate_ancestor_ids
[STOP] [2020-06-16 09:51:12] Flattener#flatten
[STOP] [2020-06-16 09:51:12] rebuild_nodes
[START] [2020-06-16 09:51:12] resolve_missing_media_owners
[STOP] [2020-06-16 09:51:12] resolve_missing_media_owners
[START] [2020-06-16 09:51:12] sanitize_media_verbatims
[STOP] [2020-06-16 09:51:12] sanitize_media_verbatims
[START] [2020-06-16 09:51:12] queue_downloads
[STOP] [2020-06-16 09:51:12] queue_downloads
[START] [2020-06-16 09:51:12] parse_names
[WARN] [2020-06-16 09:51:12] I see 26 names which still need to be parsed.
[WARN] [2020-06-16 09:51:13] I see 1 names which still need to be parsed.
[STOP] [2020-06-16 09:51:14] parse_names
[START] [2020-06-16 09:51:14] denormalize_canonical_names_to_nodes
[STOP] [2020-06-16 09:51:14] denormalize_canonical_names_to_nodes
[START] [2020-06-16 09:51:14] match_nodes
[START] [2020-06-16 09:51:14] map_all_nodes_to_pages
[STOP] [2020-06-16 09:51:15] map_all_nodes_to_pages
[INFO] [2020-06-16 09:51:15] ZERO unmatched nodes (of 26)! Nicely done.
[START] [2020-06-16 09:51:15] update_nodes
[STOP] [2020-06-16 09:51:15] update_nodes
[STOP] [2020-06-16 09:51:15] match_nodes
[START] [2020-06-16 09:51:15] reindex_search
[STOP] [2020-06-16 09:51:15] reindex_search
[START] [2020-06-16 09:51:15] normalize_units
[STOP] [2020-06-16 09:51:15] normalize_units
[START] [2020-06-16 09:51:15] calculate_statistics
[STOP] [2020-06-16 09:51:15] calculate_statistics
[START] [2020-06-16 09:51:15] complete_harvest_instance
[START] [2020-06-16 09:51:15] overall_tsv_creation
[INFO] [2020-06-16 09:51:15] Processing group of 26 in 1 batches of 10000
[INFO] [2020-06-16 09:52:06] 11 Traits (unfiltered)...
[INFO] [2020-06-16 09:52:20] 11 Traits (filtered)...
[INFO] [2020-06-16 09:52:20] 0 Associations (filtered)...
[INFO] [2020-06-16 09:52:57] 44 metadata added.
[INFO] [2020-06-16 09:52:57] 0 metadata added.
[INFO] [2020-06-16 09:52:57] Average Time: 73.34
[INFO] [2020-06-16 09:52:57] Total Time: 1m43s
[STOP] [2020-06-16 09:52:57] overall_tsv_creation
[INFO] [2020-06-16 09:52:57] Done. Check your files:
[INFO] [2020-06-16 09:52:57] (25 lines) /app/public/data/lamb_et_al_lamb_/publish_nodes.tsv
[INFO] [2020-06-16 09:52:57] (53 lines) /app/public/data/lamb_et_al_lamb_/publish_node_ancestors.tsv
[INFO] [2020-06-16 09:52:57] (26 lines) /app/public/data/lamb_et_al_lamb_/publish_scientific_names.tsv
[INFO] [2020-06-16 09:52:57] (12 lines) /app/public/data/lamb_et_al_lamb_/publish_traits.tsv
[INFO] [2020-06-16 09:52:57] (45 lines) /app/public/data/lamb_et_al_lamb_/publish_metadata.tsv
[STOP] [2020-06-16 09:52:57] complete_harvest_instance
[START] [2020-06-16 09:52:57] completed
[STOP] [2020-06-16 09:52:57] completed
[STOP] [2020-06-16 09:52:57] logged process, took 114.22
[INFO] [2020-11-09 14:36:33] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-11-09 14:36:35] ## remove_type: ScientificName
[INFO] [2020-11-09 14:36:35] ++ Calling delete_all on 26 instances...
[INFO] [2020-11-09 14:36:35] [14:36:35.120] Removed 26 Scientificnames
[INFO] [2020-11-09 14:36:35] ## remove_type: Vernacular
[INFO] [2020-11-09 14:36:35] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 14:36:35] [14:36:35.122] Removed 0 Vernaculars
[INFO] [2020-11-09 14:36:35] ## remove_type: Article
[INFO] [2020-11-09 14:36:35] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 14:36:35] [14:36:35.125] Removed 0 Articles
[INFO] [2020-11-09 14:36:35] ## remove_type: Medium
[INFO] [2020-11-09 14:36:35] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 14:36:35] [14:36:35.129] Removed 0 Media
[INFO] [2020-11-09 14:36:35] ## remove_type: Trait
[INFO] [2020-11-09 14:36:35] ++ Calling delete_all on 22 instances...
[INFO] [2020-11-09 14:36:35] [14:36:35.147] Removed 22 Traits
[INFO] [2020-11-09 14:36:35] ## remove_type: MetaTrait
[INFO] [2020-11-09 14:36:35] ++ Calling delete_all on 33 instances...
[INFO] [2020-11-09 14:36:35] [14:36:35.202] Removed 33 Metatraits
[INFO] [2020-11-09 14:36:35] ## remove_type: OccurrenceMetadatum
[INFO] [2020-11-09 14:36:35] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 14:36:35] [14:36:35.215] Removed 0 Occurrencemetadata
[INFO] [2020-11-09 14:36:35] ## remove_type: Assoc
[INFO] [2020-11-09 14:36:35] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 14:36:35] [14:36:35.218] Removed 0 Assocs
[INFO] [2020-11-09 14:36:35] ## remove_type: MetaAssoc
[INFO] [2020-11-09 14:36:35] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 14:36:35] [14:36:35.221] Removed 0 Metaassocs
[INFO] [2020-11-09 14:36:35] ## remove_type: Identifier
[INFO] [2020-11-09 14:36:35] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 14:36:35] [14:36:35.228] Removed 0 Identifiers
[INFO] [2020-11-09 14:36:35] ## remove_type: Reference
[INFO] [2020-11-09 14:36:35] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-09 14:36:35] [14:36:35.231] Removed 0 References
[INFO] [2020-11-09 14:36:35] Starting batch with ID 80097935...
[INFO] [2020-11-09 14:36:35] Starting batch with ID 80097934...
[INFO] [2020-11-09 14:36:36] ## remove_type: Node
[INFO] [2020-11-09 14:36:36] ++ Calling delete_all on 26 instances...
[INFO] [2020-11-09 14:36:36] [14:36:36.278] Removed 26 Nodes
[START] [2020-11-09 14:36:36] logged process: 9719e9b2443e9b331c213f6700b6e76a1b7c8548
[START] [2020-11-09 14:36:36] Creating resource from OpenData
[START] [2020-11-09 14:36:36] logged process: 9719e9b2443e9b331c213f6700b6e76a1b7c8548
[START] [2020-11-09 14:36:36] Parse meta.xml file and create formats with fields
[STOP] [2020-11-09 14:36:36] Parse meta.xml file and create formats with fields
[STOP] [2020-11-09 14:36:36] Creating resource from OpenData
[START] [2020-11-09 14:36:36] logged process: 9719e9b2443e9b331c213f6700b6e76a1b7c8548
[START] [2020-11-09 14:36:36] create_harvest_instance
[STOP] [2020-11-09 14:36:38] create_harvest_instance
[START] [2020-11-09 14:36:38] fetch_files
[STOP] [2020-11-09 14:36:38] fetch_files
[START] [2020-11-09 14:36:38] validate_each_file
[STOP] [2020-11-09 14:36:38] validate_each_file
[START] [2020-11-09 14:36:38] convert_to_csv
[CMD] [2020-11-09 14:36:38] /usr/bin/sort /app/public/converted_csv/lamb_et_al_lamb__nodes_23849.csv > /app/public/converted_csv/lamb_et_al_lamb__nodes_23849.csv_sorted
[CMD] [2020-11-09 14:36:38] /usr/bin/sort /app/public/converted_csv/lamb_et_al_lamb__occurrences_23850.csv > /app/public/converted_csv/lamb_et_al_lamb__occurrences_23850.csv_sorted
[CMD] [2020-11-09 14:36:38] /usr/bin/sort /app/public/converted_csv/lamb_et_al_lamb__measurements_23851.csv > /app/public/converted_csv/lamb_et_al_lamb__measurements_23851.csv_sorted
[STOP] [2020-11-09 14:36:38] convert_to_csv
[START] [2020-11-09 14:36:38] calculate_delta
[CMD] [2020-11-09 14:36:38] echo "0a" > /app/public/diff/lamb_et_al_lamb__nodes_23849.diff
[CMD] [2020-11-09 14:36:38] tail -n +1 /app/public/converted_csv/lamb_et_al_lamb__nodes_23849.csv >> /app/public/diff/lamb_et_al_lamb__nodes_23849.diff
[CMD] [2020-11-09 14:36:38] echo "." >> /app/public/diff/lamb_et_al_lamb__nodes_23849.diff
[CMD] [2020-11-09 14:36:38] echo "0a" > /app/public/diff/lamb_et_al_lamb__occurrences_23850.diff
[CMD] [2020-11-09 14:36:38] tail -n +1 /app/public/converted_csv/lamb_et_al_lamb__occurrences_23850.csv >> /app/public/diff/lamb_et_al_lamb__occurrences_23850.diff
[CMD] [2020-11-09 14:36:38] echo "." >> /app/public/diff/lamb_et_al_lamb__occurrences_23850.diff
[CMD] [2020-11-09 14:36:38] echo "0a" > /app/public/diff/lamb_et_al_lamb__measurements_23851.diff
[CMD] [2020-11-09 14:36:38] tail -n +1 /app/public/converted_csv/lamb_et_al_lamb__measurements_23851.csv >> /app/public/diff/lamb_et_al_lamb__measurements_23851.diff
[CMD] [2020-11-09 14:36:38] echo "." >> /app/public/diff/lamb_et_al_lamb__measurements_23851.diff
[STOP] [2020-11-09 14:36:38] calculate_delta
[START] [2020-11-09 14:36:38] parse_diff_and_store
[INFO] [2020-11-09 14:36:38] Loading nodes diff file into memory (true lines)...
[INFO] [2020-11-09 14:36:38] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-11-09 14:36:38] Loading measurements diff file into memory (true lines)...
[INFO] [2020-11-09 14:36:39] Storing 26 ScientificNames
[INFO] [2020-11-09 14:36:39] Processing group of 26 in 1 groups of 1000
[INFO] [2020-11-09 14:36:39] Average Time: 0.01
[INFO] [2020-11-09 14:36:39] Total Time: 1s
[INFO] [2020-11-09 14:36:39] Storing 26 Nodes
[INFO] [2020-11-09 14:36:39] Processing group of 26 in 1 groups of 1000
[INFO] [2020-11-09 14:36:39] Average Time: 0.01
[INFO] [2020-11-09 14:36:39] Total Time: 1s
[INFO] [2020-11-09 14:36:39] Storing 11 Occurrences
[INFO] [2020-11-09 14:36:39] Processing group of 11 in 1 groups of 1000
[INFO] [2020-11-09 14:36:39] Average Time: 0.0
[INFO] [2020-11-09 14:36:39] Total Time: 1s
[INFO] [2020-11-09 14:36:39] Storing 22 Traits
[INFO] [2020-11-09 14:36:39] Processing group of 22 in 1 groups of 1000
[INFO] [2020-11-09 14:36:39] Average Time: 0.01
[INFO] [2020-11-09 14:36:39] Total Time: 1s
[INFO] [2020-11-09 14:36:39] Storing 22 MetaTraits
[INFO] [2020-11-09 14:36:39] Processing group of 22 in 1 groups of 1000
[INFO] [2020-11-09 14:36:39] Average Time: 0.0
[INFO] [2020-11-09 14:36:39] Total Time: 1s
[STOP] [2020-11-09 14:36:39] parse_diff_and_store
[START] [2020-11-09 14:36:39] resolve_keys
[INFO] [2020-11-09 14:36:45] Occurrences to nodes (through scientific_names)...
[INFO] [2020-11-09 14:36:45] traits to occurrences...
[INFO] [2020-11-09 14:36:45] traits to nodes (through occurrences)...
[INFO] [2020-11-09 14:36:45] Traits to sex term...
[INFO] [2020-11-09 14:36:45] Traits to lifestage term...
[INFO] [2020-11-09 14:36:45] MetaTraits to traits...
[INFO] [2020-11-09 14:36:45] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-11-09 14:36:45] Assocs to occurrences...
[INFO] [2020-11-09 14:36:45] Assocs to nodes...
[INFO] [2020-11-09 14:36:45] Assoc to sex term...
[INFO] [2020-11-09 14:36:45] Assoc to lifestage term...
[INFO] [2020-11-09 14:36:45] MetaAssoc to assocs...
[STOP] [2020-11-09 14:36:45] resolve_keys
[START] [2020-11-09 14:36:45] hold_for_later_1
[STOP] [2020-11-09 14:36:45] hold_for_later_1
[START] [2020-11-09 14:36:45] hold_for_later_2
[STOP] [2020-11-09 14:36:45] hold_for_later_2
[START] [2020-11-09 14:36:45] resolve_missing_parents
[STOP] [2020-11-09 14:36:45] resolve_missing_parents
[START] [2020-11-09 14:36:45] rebuild_nodes
[START] [2020-11-09 14:36:45] Flattener#flatten
[START] [2020-11-09 14:36:45] Flattener#study_resource
[START] [2020-11-09 14:36:45] Flattener#build_ancestry
[STOP] [2020-11-09 14:36:45] Flattener#build_ancestry
[INFO] [2020-11-09 14:36:45] 26 ancestry keys
[START] [2020-11-09 14:36:45] build_node_ancestors
[INFO] [2020-11-09 14:36:45] old ancestors deleted.
[STOP] [2020-11-09 14:36:45] build_node_ancestors
[START] [2020-11-09 14:36:45] Flattener#propagate_ancestor_ids
[STOP] [2020-11-09 14:36:45] Flattener#propagate_ancestor_ids
[STOP] [2020-11-09 14:36:45] Flattener#flatten
[STOP] [2020-11-09 14:36:45] rebuild_nodes
[START] [2020-11-09 14:36:45] resolve_missing_media_owners
[STOP] [2020-11-09 14:36:45] resolve_missing_media_owners
[START] [2020-11-09 14:36:45] sanitize_media_verbatims
[STOP] [2020-11-09 14:36:45] sanitize_media_verbatims
[START] [2020-11-09 14:36:45] queue_downloads
[STOP] [2020-11-09 14:36:46] queue_downloads
[START] [2020-11-09 14:36:46] parse_names
[WARN] [2020-11-09 14:36:46] I see 26 names which still need to be parsed.
[WARN] [2020-11-09 14:36:47] I see 1 names which still need to be parsed.
[STOP] [2020-11-09 14:36:48] parse_names
[START] [2020-11-09 14:36:48] denormalize_canonical_names_to_nodes
[STOP] [2020-11-09 14:36:48] denormalize_canonical_names_to_nodes
[START] [2020-11-09 14:36:48] match_nodes
[START] [2020-11-09 14:36:48] map_all_nodes_to_pages
[STOP] [2020-11-09 14:36:49] map_all_nodes_to_pages
[INFO] [2020-11-09 14:36:49] ZERO unmatched nodes (of 26)! Nicely done.
[START] [2020-11-09 14:36:49] update_nodes
[STOP] [2020-11-09 14:36:49] update_nodes
[STOP] [2020-11-09 14:36:49] match_nodes
[START] [2020-11-09 14:36:49] reindex_search
[STOP] [2020-11-09 14:36:49] reindex_search
[START] [2020-11-09 14:36:50] normalize_units
[STOP] [2020-11-09 14:36:50] normalize_units
[START] [2020-11-09 14:36:50] calculate_statistics
[STOP] [2020-11-09 14:36:50] calculate_statistics
[START] [2020-11-09 14:36:50] complete_harvest_instance
[START] [2020-11-09 14:36:50] overall_tsv_creation
[INFO] [2020-11-09 14:36:50] Processing group of 26 in 1 batches of 10000
[INFO] [2020-11-09 14:37:31] 11 Traits (unfiltered)...
[INFO] [2020-11-09 14:38:09] 11 Traits (filtered)...
[INFO] [2020-11-09 14:38:09] 0 Associations (filtered)...
[INFO] [2020-11-09 14:38:09] 33 metadata added.
[INFO] [2020-11-09 14:38:09] 0 metadata added.
[INFO] [2020-11-09 14:38:10] Average Time: 53.86
[INFO] [2020-11-09 14:38:10] Total Time: 1m20s
[STOP] [2020-11-09 14:38:10] overall_tsv_creation
[INFO] [2020-11-09 14:38:10] Done. Check your files:
[INFO] [2020-11-09 14:38:10] (25 lines) /app/public/data/lamb_et_al_lamb_/publish_nodes.tsv
[INFO] [2020-11-09 14:38:10] (53 lines) /app/public/data/lamb_et_al_lamb_/publish_node_ancestors.tsv
[INFO] [2020-11-09 14:38:10] (26 lines) /app/public/data/lamb_et_al_lamb_/publish_scientific_names.tsv
[INFO] [2020-11-09 14:38:10] (12 lines) /app/public/data/lamb_et_al_lamb_/publish_traits.tsv
[INFO] [2020-11-09 14:38:10] (34 lines) /app/public/data/lamb_et_al_lamb_/publish_metadata.tsv
[STOP] [2020-11-09 14:38:10] complete_harvest_instance
[START] [2020-11-09 14:38:10] completed
[STOP] [2020-11-09 14:38:10] completed
[STOP] [2020-11-09 14:38:10] logged process, took 93.11
[INFO] [2020-11-13 10:11:18] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-11-13 10:11:23] ## remove_type: ScientificName
[INFO] [2020-11-13 10:11:23] ++ Calling delete_all on 26 instances...
[INFO] [2020-11-13 10:11:23] [10:11:23.517] Removed 26 Scientificnames
[INFO] [2020-11-13 10:11:23] ## remove_type: Vernacular
[INFO] [2020-11-13 10:11:23] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:11:23] [10:11:23.520] Removed 0 Vernaculars
[INFO] [2020-11-13 10:11:23] ## remove_type: Article
[INFO] [2020-11-13 10:11:23] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:11:23] [10:11:23.523] Removed 0 Articles
[INFO] [2020-11-13 10:11:23] ## remove_type: Medium
[INFO] [2020-11-13 10:11:23] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:11:23] [10:11:23.527] Removed 0 Media
[INFO] [2020-11-13 10:11:23] ## remove_type: Trait
[INFO] [2020-11-13 10:11:23] ++ Calling delete_all on 22 instances...
[INFO] [2020-11-13 10:11:23] [10:11:23.539] Removed 22 Traits
[INFO] [2020-11-13 10:11:23] ## remove_type: MetaTrait
[INFO] [2020-11-13 10:11:23] ++ Calling delete_all on 22 instances...
[INFO] [2020-11-13 10:11:23] [10:11:23.556] Removed 22 Metatraits
[INFO] [2020-11-13 10:11:23] ## remove_type: OccurrenceMetadatum
[INFO] [2020-11-13 10:11:23] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:11:23] [10:11:23.562] Removed 0 Occurrencemetadata
[INFO] [2020-11-13 10:11:23] ## remove_type: Assoc
[INFO] [2020-11-13 10:11:23] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:11:23] [10:11:23.565] Removed 0 Assocs
[INFO] [2020-11-13 10:11:23] ## remove_type: MetaAssoc
[INFO] [2020-11-13 10:11:23] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:11:23] [10:11:23.576] Removed 0 Metaassocs
[INFO] [2020-11-13 10:11:23] ## remove_type: Identifier
[INFO] [2020-11-13 10:11:23] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:11:23] [10:11:23.578] Removed 0 Identifiers
[INFO] [2020-11-13 10:11:23] ## remove_type: Reference
[INFO] [2020-11-13 10:11:23] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:11:23] [10:11:23.582] Removed 0 References
[INFO] [2020-11-13 10:11:23] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:23] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:23] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:23] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] Starting batch with ID 81597308...
[INFO] [2020-11-13 10:11:24] ## remove_type: Node
[INFO] [2020-11-13 10:11:24] ++ Calling delete_all on 26 instances...
[INFO] [2020-11-13 10:11:24] [10:11:24.930] Removed 26 Nodes
[START] [2020-11-13 10:11:25] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979
[START] [2020-11-13 10:11:25] Creating resource from OpenData
[START] [2020-11-13 10:11:25] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979
[START] [2020-11-13 10:11:25] Parse meta.xml file and create formats with fields
[STOP] [2020-11-13 10:11:25] Parse meta.xml file and create formats with fields
[STOP] [2020-11-13 10:11:25] Creating resource from OpenData
[START] [2020-11-13 10:11:25] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979
[START] [2020-11-13 10:11:25] create_harvest_instance
[STOP] [2020-11-13 10:11:27] create_harvest_instance
[START] [2020-11-13 10:11:27] fetch_files
[STOP] [2020-11-13 10:11:27] fetch_files
[START] [2020-11-13 10:11:27] validate_each_file
[STOP] [2020-11-13 10:11:27] validate_each_file
[START] [2020-11-13 10:11:27] convert_to_csv
[CMD] [2020-11-13 10:11:27] /usr/bin/sort /app/public/converted_csv/lamb_et_al_lamb__nodes_24075.csv > /app/public/converted_csv/lamb_et_al_lamb__nodes_24075.csv_sorted
[CMD] [2020-11-13 10:11:27] /usr/bin/sort /app/public/converted_csv/lamb_et_al_lamb__occurrences_24076.csv > /app/public/converted_csv/lamb_et_al_lamb__occurrences_24076.csv_sorted
[CMD] [2020-11-13 10:11:27] /usr/bin/sort /app/public/converted_csv/lamb_et_al_lamb__measurements_24077.csv > /app/public/converted_csv/lamb_et_al_lamb__measurements_24077.csv_sorted
[STOP] [2020-11-13 10:11:27] convert_to_csv
[START] [2020-11-13 10:11:27] calculate_delta
[CMD] [2020-11-13 10:11:27] echo "0a" > /app/public/diff/lamb_et_al_lamb__nodes_24075.diff
[CMD] [2020-11-13 10:11:27] tail -n +1 /app/public/converted_csv/lamb_et_al_lamb__nodes_24075.csv >> /app/public/diff/lamb_et_al_lamb__nodes_24075.diff
[CMD] [2020-11-13 10:11:27] echo "." >> /app/public/diff/lamb_et_al_lamb__nodes_24075.diff
[CMD] [2020-11-13 10:11:27] echo "0a" > /app/public/diff/lamb_et_al_lamb__occurrences_24076.diff
[CMD] [2020-11-13 10:11:27] tail -n +1 /app/public/converted_csv/lamb_et_al_lamb__occurrences_24076.csv >> /app/public/diff/lamb_et_al_lamb__occurrences_24076.diff
[CMD] [2020-11-13 10:11:27] echo "." >> /app/public/diff/lamb_et_al_lamb__occurrences_24076.diff
[CMD] [2020-11-13 10:11:27] echo "0a" > /app/public/diff/lamb_et_al_lamb__measurements_24077.diff
[CMD] [2020-11-13 10:11:27] tail -n +1 /app/public/converted_csv/lamb_et_al_lamb__measurements_24077.csv >> /app/public/diff/lamb_et_al_lamb__measurements_24077.diff
[CMD] [2020-11-13 10:11:27] echo "." >> /app/public/diff/lamb_et_al_lamb__measurements_24077.diff
[STOP] [2020-11-13 10:11:27] calculate_delta
[START] [2020-11-13 10:11:27] parse_diff_and_store
[INFO] [2020-11-13 10:11:27] Loading nodes diff file into memory (true lines)...
[INFO] [2020-11-13 10:11:27] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-11-13 10:11:27] Loading measurements diff file into memory (true lines)...
[INFO] [2020-11-13 10:11:28] Storing 26 ScientificNames
[INFO] [2020-11-13 10:11:28] Processing group of 26 in 1 groups of 1000
[INFO] [2020-11-13 10:11:28] Average Time: 0.01
[INFO] [2020-11-13 10:11:28] Total Time: 1s
[INFO] [2020-11-13 10:11:28] Storing 26 Nodes
[INFO] [2020-11-13 10:11:28] Processing group of 26 in 1 groups of 1000
[INFO] [2020-11-13 10:11:28] Average Time: 0.01
[INFO] [2020-11-13 10:11:28] Total Time: 1s
[INFO] [2020-11-13 10:11:28] Storing 11 Occurrences
[INFO] [2020-11-13 10:11:28] Processing group of 11 in 1 groups of 1000
[INFO] [2020-11-13 10:11:28] Average Time: 0.0
[INFO] [2020-11-13 10:11:28] Total Time: 1s
[INFO] [2020-11-13 10:11:28] Storing 22 Traits
[INFO] [2020-11-13 10:11:28] Processing group of 22 in 1 groups of 1000
[INFO] [2020-11-13 10:11:28] Average Time: 0.01
[INFO] [2020-11-13 10:11:28] Total Time: 1s
[INFO] [2020-11-13 10:11:28] Storing 22 MetaTraits
[INFO] [2020-11-13 10:11:28] Processing group of 22 in 1 groups of 1000
[INFO] [2020-11-13 10:11:28] Average Time: 0.01
[INFO] [2020-11-13 10:11:28] Total Time: 1s
[STOP] [2020-11-13 10:11:28] parse_diff_and_store
[START] [2020-11-13 10:11:28] resolve_keys
[INFO] [2020-11-13 10:11:34] Occurrences to nodes (through scientific_names)...
[INFO] [2020-11-13 10:11:34] traits to occurrences...
[INFO] [2020-11-13 10:11:34] traits to nodes (through occurrences)...
[INFO] [2020-11-13 10:11:34] Traits to sex term...
[INFO] [2020-11-13 10:11:34] Traits to lifestage term...
[INFO] [2020-11-13 10:11:34] MetaTraits to traits...
[INFO] [2020-11-13 10:11:34] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-11-13 10:11:34] Assocs to occurrences...
[INFO] [2020-11-13 10:11:34] Assocs to nodes...
[INFO] [2020-11-13 10:11:34] Assoc to sex term...
[INFO] [2020-11-13 10:11:34] Assoc to lifestage term...
[INFO] [2020-11-13 10:11:34] MetaAssoc to assocs...
[STOP] [2020-11-13 10:11:34] resolve_keys
[START] [2020-11-13 10:11:34] hold_for_later_1
[STOP] [2020-11-13 10:11:34] hold_for_later_1
[START] [2020-11-13 10:11:34] hold_for_later_2
[STOP] [2020-11-13 10:11:34] hold_for_later_2
[START] [2020-11-13 10:11:34] resolve_missing_parents
[STOP] [2020-11-13 10:11:34] resolve_missing_parents
[START] [2020-11-13 10:11:34] rebuild_nodes
[START] [2020-11-13 10:11:34] Flattener#flatten
[START] [2020-11-13 10:11:34] Flattener#study_resource
[START] [2020-11-13 10:11:34] Flattener#build_ancestry
[STOP] [2020-11-13 10:11:34] Flattener#build_ancestry
[INFO] [2020-11-13 10:11:34] 26 ancestry keys
[START] [2020-11-13 10:11:34] build_node_ancestors
[INFO] [2020-11-13 10:11:34] old ancestors deleted.
[STOP] [2020-11-13 10:11:34] build_node_ancestors
[START] [2020-11-13 10:11:34] Flattener#propagate_ancestor_ids
[STOP] [2020-11-13 10:11:34] Flattener#propagate_ancestor_ids
[STOP] [2020-11-13 10:11:34] Flattener#flatten
[STOP] [2020-11-13 10:11:34] rebuild_nodes
[START] [2020-11-13 10:11:34] resolve_missing_media_owners
[STOP] [2020-11-13 10:11:34] resolve_missing_media_owners
[START] [2020-11-13 10:11:34] sanitize_media_verbatims
[STOP] [2020-11-13 10:11:34] sanitize_media_verbatims
[START] [2020-11-13 10:11:34] queue_downloads
[STOP] [2020-11-13 10:11:34] queue_downloads
[START] [2020-11-13 10:11:34] parse_names
[WARN] [2020-11-13 10:11:34] I see 26 names which still need to be parsed.
[WARN] [2020-11-13 10:11:35] I see 1 names which still need to be parsed.
[STOP] [2020-11-13 10:11:37] parse_names
[START] [2020-11-13 10:11:37] denormalize_canonical_names_to_nodes
[STOP] [2020-11-13 10:11:37] denormalize_canonical_names_to_nodes
[START] [2020-11-13 10:11:37] match_nodes
[START] [2020-11-13 10:11:37] map_all_nodes_to_pages
[STOP] [2020-11-13 10:11:38] map_all_nodes_to_pages
[INFO] [2020-11-13 10:11:38] ZERO unmatched nodes (of 26)! Nicely done.
[START] [2020-11-13 10:11:38] update_nodes
[STOP] [2020-11-13 10:11:38] update_nodes
[STOP] [2020-11-13 10:11:38] match_nodes
[START] [2020-11-13 10:11:38] reindex_search
[STOP] [2020-11-13 10:11:38] reindex_search
[START] [2020-11-13 10:11:38] normalize_units
[STOP] [2020-11-13 10:11:38] normalize_units
[START] [2020-11-13 10:11:38] calculate_statistics
[STOP] [2020-11-13 10:11:38] calculate_statistics
[START] [2020-11-13 10:11:38] complete_harvest_instance
[START] [2020-11-13 10:11:38] overall_tsv_creation
[INFO] [2020-11-13 10:11:38] Processing group of 26 in 1 batches of 10000
[INFO] [2020-11-13 10:12:33] 11 Traits (unfiltered)...
[INFO] [2020-11-13 10:13:11] 11 Traits (filtered)...
[INFO] [2020-11-13 10:13:11] 0 Associations (filtered)...
[INFO] [2020-11-13 10:13:11] 33 metadata added.
[INFO] [2020-11-13 10:13:11] 0 metadata added.
[INFO] [2020-11-13 10:13:11] Average Time: 54.69
[INFO] [2020-11-13 10:13:11] Total Time: 1m34s
[STOP] [2020-11-13 10:13:11] overall_tsv_creation
[INFO] [2020-11-13 10:13:11] Done. Check your files:
[INFO] [2020-11-13 10:13:11] (25 lines) /app/public/data/lamb_et_al_lamb_/publish_nodes.tsv
[INFO] [2020-11-13 10:13:11] (53 lines) /app/public/data/lamb_et_al_lamb_/publish_node_ancestors.tsv
[INFO] [2020-11-13 10:13:11] (26 lines) /app/public/data/lamb_et_al_lamb_/publish_scientific_names.tsv
[INFO] [2020-11-13 10:13:11] (12 lines) /app/public/data/lamb_et_al_lamb_/publish_traits.tsv
[INFO] [2020-11-13 10:13:11] (34 lines) /app/public/data/lamb_et_al_lamb_/publish_metadata.tsv
[STOP] [2020-11-13 10:13:11] complete_harvest_instance
[START] [2020-11-13 10:13:11] completed
[STOP] [2020-11-13 10:13:11] completed
[STOP] [2020-11-13 10:13:11] logged process, took 105.7
[INFO] [2021-04-19 09:18:15] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-19 09:18:15] ## remove_type: ScientificName
[INFO] [2021-04-19 09:18:15] ++ Calling delete_all on 26 instances...
[INFO] [2021-04-19 09:18:15] [09:18:15.324] Removed 26 Scientificnames
[INFO] [2021-04-19 09:18:15] ## remove_type: Vernacular
[INFO] [2021-04-19 09:18:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:18:15] [09:18:15.326] Removed 0 Vernaculars
[INFO] [2021-04-19 09:18:15] ## remove_type: Article
[INFO] [2021-04-19 09:18:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:18:15] [09:18:15.327] Removed 0 Articles
[INFO] [2021-04-19 09:18:15] ## remove_type: Medium
[INFO] [2021-04-19 09:18:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:18:15] [09:18:15.329] Removed 0 Media
[INFO] [2021-04-19 09:18:15] ## remove_type: Trait
[INFO] [2021-04-19 09:18:15] ++ Calling delete_all on 22 instances...
[INFO] [2021-04-19 09:18:15] [09:18:15.338] Removed 22 Traits
[INFO] [2021-04-19 09:18:15] ## remove_type: MetaTrait
[INFO] [2021-04-19 09:18:15] ++ Calling delete_all on 22 instances...
[INFO] [2021-04-19 09:18:15] [09:18:15.368] Removed 22 Metatraits
[INFO] [2021-04-19 09:18:15] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-19 09:18:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:18:15] [09:18:15.369] Removed 0 Occurrencemetadata
[INFO] [2021-04-19 09:18:15] ## remove_type: Assoc
[INFO] [2021-04-19 09:18:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:18:15] [09:18:15.371] Removed 0 Assocs
[INFO] [2021-04-19 09:18:15] ## remove_type: MetaAssoc
[INFO] [2021-04-19 09:18:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:18:15] [09:18:15.373] Removed 0 Metaassocs
[INFO] [2021-04-19 09:18:15] ## remove_type: Identifier
[INFO] [2021-04-19 09:18:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:18:15] [09:18:15.375] Removed 0 Identifiers
[INFO] [2021-04-19 09:18:15] ## remove_type: Reference
[INFO] [2021-04-19 09:18:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:18:15] [09:18:15.376] Removed 0 References
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:15] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:16] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:16] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:16] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:16] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:16] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:16] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:16] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:16] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:16] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:16] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:16] Starting batch with ID 81657868...
[INFO] [2021-04-19 09:18:16] ## remove_type: Node
[INFO] [2021-04-19 09:18:16] ++ Calling delete_all on 26 instances...
[INFO] [2021-04-19 09:18:16] [09:18:16.194] Removed 26 Nodes
[START] [2021-04-19 09:18:16] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 09:18:16] Creating resource from OpenData
[START] [2021-04-19 09:18:17] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 09:18:17] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 09:18:21] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 09:18:21] Creating resource from OpenData
[START] [2021-04-19 09:18:21] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 09:18:21] create_harvest_instance
[INFO] [2021-04-19 09:18:21] Created harvest instance #3714
[STOP] [2021-04-19 09:18:21] create_harvest_instance
[START] [2021-04-19 09:18:21] fetch_files
[STOP] [2021-04-19 09:18:21] fetch_files
[START] [2021-04-19 09:18:21] validate_each_file
[INFO] [2021-04-19 09:18:21] Looping over 3 formats...
[INFO] [2021-04-19 09:18:21] ...nodes (/app/public/data/lamb_et_al_lamb_/taxa.txt)
[INFO] [2021-04-19 09:18:21] Valid: /app/public/converted_csv/lamb_et_al_lamb__nodes_3714.csv (26 lines)
[INFO] [2021-04-19 09:18:21] ...occurrences (/app/public/data/lamb_et_al_lamb_/occurrences.txt)
[INFO] [2021-04-19 09:18:21] Valid: /app/public/converted_csv/lamb_et_al_lamb__occurrences_3714.csv (11 lines)
[INFO] [2021-04-19 09:18:21] ...measurements (/app/public/data/lamb_et_al_lamb_/measurementsorfacts.txt)
[INFO] [2021-04-19 09:18:21] Valid: /app/public/converted_csv/lamb_et_al_lamb__measurements_3714.csv (22 lines)
[STOP] [2021-04-19 09:18:21] validate_each_file
[START] [2021-04-19 09:18:21] convert_to_csv
[INFO] [2021-04-19 09:18:21] Looping over 3 formats...
[INFO] [2021-04-19 09:18:21] ...nodes (/app/public/data/lamb_et_al_lamb_/taxa.txt)
[CMD] [2021-04-19 09:18:21] /usr/bin/sort /app/public/converted_csv/lamb_et_al_lamb__nodes_3714.csv > /app/public/converted_csv/lamb_et_al_lamb__nodes_3714.csv_sorted
[INFO] [2021-04-19 09:18:22] Converted: /app/public/converted_csv/lamb_et_al_lamb__nodes_3714.csv (26 lines)
[INFO] [2021-04-19 09:18:22] ...occurrences (/app/public/data/lamb_et_al_lamb_/occurrences.txt)
[CMD] [2021-04-19 09:18:22] /usr/bin/sort /app/public/converted_csv/lamb_et_al_lamb__occurrences_3714.csv > /app/public/converted_csv/lamb_et_al_lamb__occurrences_3714.csv_sorted
[INFO] [2021-04-19 09:18:22] Converted: /app/public/converted_csv/lamb_et_al_lamb__occurrences_3714.csv (11 lines)
[INFO] [2021-04-19 09:18:22] ...measurements (/app/public/data/lamb_et_al_lamb_/measurementsorfacts.txt)
[CMD] [2021-04-19 09:18:22] /usr/bin/sort /app/public/converted_csv/lamb_et_al_lamb__measurements_3714.csv > /app/public/converted_csv/lamb_et_al_lamb__measurements_3714.csv_sorted
[INFO] [2021-04-19 09:18:23] Converted: /app/public/converted_csv/lamb_et_al_lamb__measurements_3714.csv (22 lines)
[STOP] [2021-04-19 09:18:23] convert_to_csv
[START] [2021-04-19 09:18:23] calculate_delta
[INFO] [2021-04-19 09:18:23] Looping over 3 formats...
[INFO] [2021-04-19 09:18:23] ...nodes (/app/public/data/lamb_et_al_lamb_/taxa.txt)
[CMD] [2021-04-19 09:18:23] echo "0a" > /app/public/diff/lamb_et_al_lamb__nodes_3714.diff
[CMD] [2021-04-19 09:18:23] tail -n +1 /app/public/converted_csv/lamb_et_al_lamb__nodes_3714.csv >> /app/public/diff/lamb_et_al_lamb__nodes_3714.diff
[CMD] [2021-04-19 09:18:23] echo "." >> /app/public/diff/lamb_et_al_lamb__nodes_3714.diff
[INFO] [2021-04-19 09:18:24] Created diff: /app/public/diff/lamb_et_al_lamb__nodes_3714.diff (28 lines)
[INFO] [2021-04-19 09:18:24] ...occurrences (/app/public/data/lamb_et_al_lamb_/occurrences.txt)
[CMD] [2021-04-19 09:18:24] echo "0a" > /app/public/diff/lamb_et_al_lamb__occurrences_3714.diff
[CMD] [2021-04-19 09:18:24] tail -n +1 /app/public/converted_csv/lamb_et_al_lamb__occurrences_3714.csv >> /app/public/diff/lamb_et_al_lamb__occurrences_3714.diff
[CMD] [2021-04-19 09:18:25] echo "." >> /app/public/diff/lamb_et_al_lamb__occurrences_3714.diff
[INFO] [2021-04-19 09:18:25] Created diff: /app/public/diff/lamb_et_al_lamb__occurrences_3714.diff (13 lines)
[INFO] [2021-04-19 09:18:25] ...measurements (/app/public/data/lamb_et_al_lamb_/measurementsorfacts.txt)
[CMD] [2021-04-19 09:18:25] echo "0a" > /app/public/diff/lamb_et_al_lamb__measurements_3714.diff
[CMD] [2021-04-19 09:18:25] tail -n +1 /app/public/converted_csv/lamb_et_al_lamb__measurements_3714.csv >> /app/public/diff/lamb_et_al_lamb__measurements_3714.diff
[CMD] [2021-04-19 09:18:26] echo "." >> /app/public/diff/lamb_et_al_lamb__measurements_3714.diff
[INFO] [2021-04-19 09:18:26] Created diff: /app/public/diff/lamb_et_al_lamb__measurements_3714.diff (24 lines)
[STOP] [2021-04-19 09:18:26] calculate_delta
[START] [2021-04-19 09:18:26] parse_diff_and_store
[INFO] [2021-04-19 09:18:26] Handling diff: /app/public/diff/lamb_et_al_lamb__nodes_3714.diff (28 lines)
[INFO] [2021-04-19 09:18:27] Loading nodes diff file into memory (28 /app/public/diff/lamb_et_al_lamb__nodes_3714.diff lines)...
[INFO] [2021-04-19 09:18:27] Handling diff: /app/public/diff/lamb_et_al_lamb__occurrences_3714.diff (13 lines)
[INFO] [2021-04-19 09:18:27] Loading occurrences diff file into memory (13 /app/public/diff/lamb_et_al_lamb__occurrences_3714.diff lines)...
[INFO] [2021-04-19 09:18:28] Handling diff: /app/public/diff/lamb_et_al_lamb__measurements_3714.diff (24 lines)
[INFO] [2021-04-19 09:18:28] Loading measurements diff file into memory (24 /app/public/diff/lamb_et_al_lamb__measurements_3714.diff lines)...
[INFO] [2021-04-19 09:18:29] Storing 26 ScientificNames
[INFO] [2021-04-19 09:18:29] Processing group of 26 in 1 groups of 1000
[INFO] [2021-04-19 09:18:29] Average Time: 0.01
[INFO] [2021-04-19 09:18:29] Total Time: 1s
[INFO] [2021-04-19 09:18:29] Storing 26 Nodes
[INFO] [2021-04-19 09:18:29] Processing group of 26 in 1 groups of 1000
[INFO] [2021-04-19 09:18:29] Average Time: 0.01
[INFO] [2021-04-19 09:18:29] Total Time: 1s
[INFO] [2021-04-19 09:18:29] Storing 11 Occurrences
[INFO] [2021-04-19 09:18:29] Processing group of 11 in 1 groups of 1000
[INFO] [2021-04-19 09:18:29] Average Time: 0.0
[INFO] [2021-04-19 09:18:29] Total Time: 1s
[INFO] [2021-04-19 09:18:29] Storing 22 Traits
[INFO] [2021-04-19 09:18:29] Processing group of 22 in 1 groups of 1000
[INFO] [2021-04-19 09:18:29] Average Time: 0.01
[INFO] [2021-04-19 09:18:29] Total Time: 1s
[INFO] [2021-04-19 09:18:29] Storing 22 MetaTraits
[INFO] [2021-04-19 09:18:29] Processing group of 22 in 1 groups of 1000
[INFO] [2021-04-19 09:18:29] Average Time: 0.0
[INFO] [2021-04-19 09:18:29] Total Time: 1s
[STOP] [2021-04-19 09:18:29] parse_diff_and_store
[START] [2021-04-19 09:18:29] resolve_keys
[INFO] [2021-04-19 09:18:35] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-19 09:18:35] traits to occurrences...
[INFO] [2021-04-19 09:18:35] traits to nodes (through occurrences)...
[INFO] [2021-04-19 09:18:35] Traits to sex term...
[INFO] [2021-04-19 09:18:35] Traits to lifestage term...
[INFO] [2021-04-19 09:18:35] MetaTraits to traits...
[INFO] [2021-04-19 09:18:35] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-19 09:18:35] Assocs to occurrences...
[INFO] [2021-04-19 09:18:35] Assocs to nodes...
[INFO] [2021-04-19 09:18:35] Assoc to sex term...
[INFO] [2021-04-19 09:18:35] Assoc to lifestage term...
[INFO] [2021-04-19 09:18:35] MetaAssoc to assocs...
[STOP] [2021-04-19 09:18:35] resolve_keys
[START] [2021-04-19 09:18:35] hold_for_later_1
[STOP] [2021-04-19 09:18:35] hold_for_later_1
[START] [2021-04-19 09:18:35] hold_for_later_2
[STOP] [2021-04-19 09:18:35] hold_for_later_2
[START] [2021-04-19 09:18:35] resolve_missing_parents
[STOP] [2021-04-19 09:18:35] resolve_missing_parents
[START] [2021-04-19 09:18:35] rebuild_nodes
[START] [2021-04-19 09:18:35] Flattener#flatten
[START] [2021-04-19 09:18:35] Flattener#study_resource
[START] [2021-04-19 09:18:35] Flattener#build_ancestry
[STOP] [2021-04-19 09:18:35] Flattener#build_ancestry
[INFO] [2021-04-19 09:18:35] 26 ancestry keys
[START] [2021-04-19 09:18:35] build_node_ancestors
[INFO] [2021-04-19 09:18:35] old ancestors deleted.
[STOP] [2021-04-19 09:18:35] build_node_ancestors
[START] [2021-04-19 09:18:35] Flattener#propagate_ancestor_ids
[STOP] [2021-04-19 09:18:35] Flattener#propagate_ancestor_ids
[STOP] [2021-04-19 09:18:36] Flattener#flatten
[STOP] [2021-04-19 09:18:36] rebuild_nodes
[START] [2021-04-19 09:18:36] resolve_missing_media_owners
[STOP] [2021-04-19 09:18:36] resolve_missing_media_owners
[START] [2021-04-19 09:18:36] sanitize_media_verbatims
[STOP] [2021-04-19 09:18:36] sanitize_media_verbatims
[START] [2021-04-19 09:18:36] queue_downloads
[STOP] [2021-04-19 09:18:36] queue_downloads
[START] [2021-04-19 09:18:36] parse_names
[WARN] [2021-04-19 09:18:36] I see 26 names which still need to be parsed.
[WARN] [2021-04-19 09:18:37] I see 1 names which still need to be parsed.
[STOP] [2021-04-19 09:18:38] parse_names
[START] [2021-04-19 09:18:38] denormalize_canonical_names_to_nodes
[STOP] [2021-04-19 09:18:38] denormalize_canonical_names_to_nodes
[START] [2021-04-19 09:18:38] match_nodes
[START] [2021-04-19 09:18:38] map_all_nodes_to_pages
[STOP] [2021-04-19 09:18:42] map_all_nodes_to_pages
[INFO] [2021-04-19 09:18:42] ZERO unmatched nodes (of 26)! Nicely done.
[START] [2021-04-19 09:18:42] update_nodes
[STOP] [2021-04-19 09:18:42] update_nodes
[STOP] [2021-04-19 09:18:42] match_nodes
[START] [2021-04-19 09:18:42] reindex_search
[STOP] [2021-04-19 09:18:43] reindex_search
[START] [2021-04-19 09:18:43] normalize_units
[STOP] [2021-04-19 09:18:43] normalize_units
[START] [2021-04-19 09:18:43] calculate_statistics
[STOP] [2021-04-19 09:18:43] calculate_statistics
[START] [2021-04-19 09:18:43] complete_harvest_instance
[START] [2021-04-19 09:18:43] overall_tsv_creation
[INFO] [2021-04-19 09:18:43] Processing group of 26 in 1 batches of 10000
[INFO] [2021-04-19 09:19:34] 11 Traits (unfiltered)...
[INFO] [2021-04-19 09:20:08] 11 Traits (filtered)...
[INFO] [2021-04-19 09:20:08] 0 Associations (filtered)...
[INFO] [2021-04-19 09:20:08] 0 metadata added.
[INFO] [2021-04-19 09:20:08] 0 metadata added.
[INFO] [2021-04-19 09:20:36] Average Time: 74.44
[INFO] [2021-04-19 09:20:36] Total Time: 1m53s
[STOP] [2021-04-19 09:20:36] overall_tsv_creation
[INFO] [2021-04-19 09:20:36] Done. Check your files:
[INFO] [2021-04-19 09:20:36] (25 lines) /app/public/data/lamb_et_al_lamb_/publish_nodes.tsv
[INFO] [2021-04-19 09:20:36] (53 lines) /app/public/data/lamb_et_al_lamb_/publish_node_ancestors.tsv
[INFO] [2021-04-19 09:20:37] (26 lines) /app/public/data/lamb_et_al_lamb_/publish_scientific_names.tsv
[INFO] [2021-04-19 09:20:37] (12 lines) /app/public/data/lamb_et_al_lamb_/publish_traits.tsv
[INFO] [2021-04-19 09:20:38] (1 lines) /app/public/data/lamb_et_al_lamb_/publish_metadata.tsv
[STOP] [2021-04-19 09:20:38] complete_harvest_instance
[START] [2021-04-19 09:20:38] completed
[STOP] [2021-04-19 09:20:38] completed
[STOP] [2021-04-19 09:20:38] logged process, took 136.59
Latest Process