Harvest for Lafferty and Shaw 2013 Created 19 Apr 09:36

Harvesting Log

(1340 lines) (showing only the last 1000 lines, see /app/public/data/lafferty_shaw_la/process.log for the full file)

[INFO] [2020-08-04 15:31:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 15:31:38] [15:31:38.301] Removed 0 Occurrencemetadata
[INFO] [2020-08-04 15:31:38] ## remove_type: Assoc
[INFO] [2020-08-04 15:31:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 15:31:38] [15:31:38.303] Removed 0 Assocs
[INFO] [2020-08-04 15:31:38] ## remove_type: MetaAssoc
[INFO] [2020-08-04 15:31:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 15:31:38] [15:31:38.306] Removed 0 Metaassocs
[INFO] [2020-08-04 15:31:38] ## remove_type: Identifier
[INFO] [2020-08-04 15:31:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 15:31:38] [15:31:38.308] Removed 0 Identifiers
[INFO] [2020-08-04 15:31:38] ## remove_type: Reference
[INFO] [2020-08-04 15:31:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 15:31:38] [15:31:38.310] Removed 0 References
[INFO] [2020-08-04 15:31:38] ## remove_type: Node
[INFO] [2020-08-04 15:31:38] ++ Calling delete_all on 53 instances...
[INFO] [2020-08-04 15:31:38] [15:31:38.344] Removed 53 Nodes
[START] [2020-08-04 15:31:38] logged process
[START] [2020-08-04 15:31:38] Creating resource from OpenData
[START] [2020-08-04 15:31:38] logged process
[START] [2020-08-04 15:31:38] Parse meta.xml file and create formats with fields
[STOP] [2020-08-04 15:31:38] Parse meta.xml file and create formats with fields
[STOP] [2020-08-04 15:31:38] Creating resource from OpenData
[START] [2020-08-04 15:31:38] logged process
[START] [2020-08-04 15:31:38] create_harvest_instance
[STOP] [2020-08-04 15:31:39] create_harvest_instance
[START] [2020-08-04 15:31:39] fetch_files
[STOP] [2020-08-04 15:31:39] fetch_files
[START] [2020-08-04 15:31:39] validate_each_file
[STOP] [2020-08-04 15:31:40] validate_each_file
[START] [2020-08-04 15:31:40] convert_to_csv
[CMD] [2020-08-04 15:31:40] /usr/bin/sort /app/public/converted_csv/lafferty_shaw_la_nodes_22458.csv > /app/public/converted_csv/lafferty_shaw_la_nodes_22458.csv_sorted
[CMD] [2020-08-04 15:31:40] /usr/bin/sort /app/public/converted_csv/lafferty_shaw_la_occurrences_22459.csv > /app/public/converted_csv/lafferty_shaw_la_occurrences_22459.csv_sorted
[CMD] [2020-08-04 15:31:40] /usr/bin/sort /app/public/converted_csv/lafferty_shaw_la_measurements_22460.csv > /app/public/converted_csv/lafferty_shaw_la_measurements_22460.csv_sorted
[STOP] [2020-08-04 15:31:40] convert_to_csv
[START] [2020-08-04 15:31:40] calculate_delta
[CMD] [2020-08-04 15:31:40] echo "0a" > /app/public/diff/lafferty_shaw_la_nodes_22458.diff
[CMD] [2020-08-04 15:31:40] tail -n +1 /app/public/converted_csv/lafferty_shaw_la_nodes_22458.csv >> /app/public/diff/lafferty_shaw_la_nodes_22458.diff
[CMD] [2020-08-04 15:31:40] echo "." >> /app/public/diff/lafferty_shaw_la_nodes_22458.diff
[CMD] [2020-08-04 15:31:40] echo "0a" > /app/public/diff/lafferty_shaw_la_occurrences_22459.diff
[CMD] [2020-08-04 15:31:40] tail -n +1 /app/public/converted_csv/lafferty_shaw_la_occurrences_22459.csv >> /app/public/diff/lafferty_shaw_la_occurrences_22459.diff
[CMD] [2020-08-04 15:31:40] echo "." >> /app/public/diff/lafferty_shaw_la_occurrences_22459.diff
[CMD] [2020-08-04 15:31:40] echo "0a" > /app/public/diff/lafferty_shaw_la_measurements_22460.diff
[CMD] [2020-08-04 15:31:40] tail -n +1 /app/public/converted_csv/lafferty_shaw_la_measurements_22460.csv >> /app/public/diff/lafferty_shaw_la_measurements_22460.diff
[CMD] [2020-08-04 15:31:40] echo "." >> /app/public/diff/lafferty_shaw_la_measurements_22460.diff
[STOP] [2020-08-04 15:31:40] calculate_delta
[START] [2020-08-04 15:31:40] parse_diff_and_store
[INFO] [2020-08-04 15:31:40] Loading nodes diff file into memory (true lines)...
[INFO] [2020-08-04 15:31:40] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-08-04 15:31:40] Loading measurements diff file into memory (true lines)...
[INFO] [2020-08-04 15:31:41] Storing 53 ScientificNames
[INFO] [2020-08-04 15:31:41] Processing group of 53 in 1 groups of 1000
[INFO] [2020-08-04 15:31:41] Average Time: 0.02
[INFO] [2020-08-04 15:31:41] Total Time: 1s
[INFO] [2020-08-04 15:31:41] Storing 53 Nodes
[INFO] [2020-08-04 15:31:41] Processing group of 53 in 1 groups of 1000
[INFO] [2020-08-04 15:31:41] Average Time: 0.01
[INFO] [2020-08-04 15:31:41] Total Time: 1s
[INFO] [2020-08-04 15:31:41] Storing 53 Occurrences
[INFO] [2020-08-04 15:31:41] Processing group of 53 in 1 groups of 1000
[INFO] [2020-08-04 15:31:41] Average Time: 0.01
[INFO] [2020-08-04 15:31:41] Total Time: 1s
[INFO] [2020-08-04 15:31:41] Storing 446 Traits
[INFO] [2020-08-04 15:31:41] Processing group of 446 in 1 groups of 1000
[INFO] [2020-08-04 15:31:41] Average Time: 0.11
[INFO] [2020-08-04 15:31:41] Total Time: 1s
[INFO] [2020-08-04 15:31:41] Storing 378 MetaTraits
[INFO] [2020-08-04 15:31:41] Processing group of 378 in 1 groups of 1000
[INFO] [2020-08-04 15:31:41] Average Time: 0.04
[INFO] [2020-08-04 15:31:41] Total Time: 1s
[STOP] [2020-08-04 15:31:41] parse_diff_and_store
[START] [2020-08-04 15:31:41] resolve_keys
[INFO] [2020-08-04 15:31:48] Occurrences to nodes (through scientific_names)...
[INFO] [2020-08-04 15:31:48] traits to occurrences...
[INFO] [2020-08-04 15:31:48] traits to nodes (through occurrences)...
[INFO] [2020-08-04 15:31:48] Traits to sex term...
[INFO] [2020-08-04 15:31:48] Traits to lifestage term...
[INFO] [2020-08-04 15:31:48] MetaTraits to traits...
[INFO] [2020-08-04 15:31:48] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-08-04 15:31:48] Assocs to occurrences...
[INFO] [2020-08-04 15:31:48] Assocs to nodes...
[INFO] [2020-08-04 15:31:48] Assoc to sex term...
[INFO] [2020-08-04 15:31:48] Assoc to lifestage term...
[STOP] [2020-08-04 15:31:48] resolve_keys
[START] [2020-08-04 15:31:48] hold_for_later_1
[STOP] [2020-08-04 15:31:48] hold_for_later_1
[START] [2020-08-04 15:31:48] hold_for_later_2
[STOP] [2020-08-04 15:31:48] hold_for_later_2
[START] [2020-08-04 15:31:48] resolve_missing_parents
[STOP] [2020-08-04 15:31:48] resolve_missing_parents
[START] [2020-08-04 15:31:48] rebuild_nodes
[START] [2020-08-04 15:31:48] Flattener#flatten
[START] [2020-08-04 15:31:48] Flattener#study_resource
[START] [2020-08-04 15:31:48] Flattener#build_ancestry
[STOP] [2020-08-04 15:31:48] Flattener#build_ancestry
[INFO] [2020-08-04 15:31:48] 53 ancestry keys
[START] [2020-08-04 15:31:48] build_node_ancestors
[INFO] [2020-08-04 15:31:48] old ancestors deleted.
[STOP] [2020-08-04 15:31:48] build_node_ancestors
[WARN] [2020-08-04 15:31:48] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-08-04 15:31:48] Flattener#flatten
[STOP] [2020-08-04 15:31:48] rebuild_nodes
[START] [2020-08-04 15:31:48] resolve_missing_media_owners
[STOP] [2020-08-04 15:31:48] resolve_missing_media_owners
[START] [2020-08-04 15:31:48] sanitize_media_verbatims
[STOP] [2020-08-04 15:31:48] sanitize_media_verbatims
[START] [2020-08-04 15:31:48] queue_downloads
[STOP] [2020-08-04 15:31:48] queue_downloads
[START] [2020-08-04 15:31:48] parse_names
[WARN] [2020-08-04 15:31:48] I see 53 names which still need to be parsed.
[STOP] [2020-08-04 15:31:50] parse_names
[START] [2020-08-04 15:31:50] denormalize_canonical_names_to_nodes
[STOP] [2020-08-04 15:31:50] denormalize_canonical_names_to_nodes
[START] [2020-08-04 15:31:50] match_nodes
[START] [2020-08-04 15:31:50] map_all_nodes_to_pages
[STOP] [2020-08-04 15:31:50] map_all_nodes_to_pages
[INFO] [2020-08-04 15:31:50] Unmatched nodes (3 of 53): Crassiphiala (#80704867); Nanophyetes (#80704887); Ornithodiplostomum (#80704890)
[START] [2020-08-04 15:31:50] update_nodes
[STOP] [2020-08-04 15:31:50] update_nodes
[STOP] [2020-08-04 15:31:50] match_nodes
[START] [2020-08-04 15:31:50] reindex_search
[STOP] [2020-08-04 15:31:50] reindex_search
[START] [2020-08-04 15:31:50] normalize_units
[STOP] [2020-08-04 15:31:50] normalize_units
[START] [2020-08-04 15:31:50] calculate_statistics
[2020-08-04 15:31:50] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-08-04 15:31:50] calculate_statistics
[START] [2020-08-04 15:31:50] complete_harvest_instance
[START] [2020-08-04 15:31:50] overall_tsv_creation
[INFO] [2020-08-04 15:31:50] Processing group of 53 in 1 batches of 10000
[INFO] [2020-08-04 15:32:40] 126 Traits (unfiltered)...
[INFO] [2020-08-04 15:33:18] 126 Traits (filtered)...
[INFO] [2020-08-04 15:33:18] 0 Associations (filtered)...
[INFO] [2020-08-04 15:33:18] 698 metadata added.
[INFO] [2020-08-04 15:33:18] 0 metadata added.
[INFO] [2020-08-04 15:33:18] Average Time: 60.65
[INFO] [2020-08-04 15:33:18] Total Time: 1m29s
[STOP] [2020-08-04 15:33:18] overall_tsv_creation
[INFO] [2020-08-04 15:33:18] Done. Check your files:
[INFO] [2020-08-04 15:33:19] (53 lines) /app/public/data/lafferty_shaw_la/publish_nodes.tsv
[INFO] [2020-08-04 15:33:19] (53 lines) /app/public/data/lafferty_shaw_la/publish_scientific_names.tsv
[INFO] [2020-08-04 15:33:19] (127 lines) /app/public/data/lafferty_shaw_la/publish_traits.tsv
[INFO] [2020-08-04 15:33:19] (321 lines) /app/public/data/lafferty_shaw_la/publish_metadata.tsv
[STOP] [2020-08-04 15:33:19] complete_harvest_instance
[START] [2020-08-04 15:33:19] completed
[STOP] [2020-08-04 15:33:19] completed
[STOP] [2020-08-04 15:33:19] logged process, took 100.5
[INFO] [2020-08-04 15:46:29] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-08-04 15:46:33] ## remove_type: ScientificName
[INFO] [2020-08-04 15:46:33] ++ Calling delete_all on 53 instances...
[INFO] [2020-08-04 15:46:33] [15:46:33.875] Removed 53 Scientificnames
[INFO] [2020-08-04 15:46:33] ## remove_type: Vernacular
[INFO] [2020-08-04 15:46:33] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 15:46:33] [15:46:33.878] Removed 0 Vernaculars
[INFO] [2020-08-04 15:46:33] ## remove_type: Article
[INFO] [2020-08-04 15:46:33] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 15:46:33] [15:46:33.881] Removed 0 Articles
[INFO] [2020-08-04 15:46:33] ## remove_type: Medium
[INFO] [2020-08-04 15:46:33] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 15:46:33] [15:46:33.885] Removed 0 Media
[INFO] [2020-08-04 15:46:33] ## remove_type: Trait
[INFO] [2020-08-04 15:46:33] ++ Calling delete_all on 446 instances...
[INFO] [2020-08-04 15:46:33] [15:46:33.919] Removed 446 Traits
[INFO] [2020-08-04 15:46:33] ## remove_type: MetaTrait
[INFO] [2020-08-04 15:46:33] ++ Calling delete_all on 378 instances...
[INFO] [2020-08-04 15:46:33] [15:46:33.925] Removed 378 Metatraits
[INFO] [2020-08-04 15:46:33] ## remove_type: OccurrenceMetadatum
[INFO] [2020-08-04 15:46:33] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 15:46:33] [15:46:33.928] Removed 0 Occurrencemetadata
[INFO] [2020-08-04 15:46:33] ## remove_type: Assoc
[INFO] [2020-08-04 15:46:33] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 15:46:33] [15:46:33.931] Removed 0 Assocs
[INFO] [2020-08-04 15:46:33] ## remove_type: MetaAssoc
[INFO] [2020-08-04 15:46:33] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 15:46:33] [15:46:33.933] Removed 0 Metaassocs
[INFO] [2020-08-04 15:46:33] ## remove_type: Identifier
[INFO] [2020-08-04 15:46:33] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 15:46:33] [15:46:33.936] Removed 0 Identifiers
[INFO] [2020-08-04 15:46:33] ## remove_type: Reference
[INFO] [2020-08-04 15:46:33] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 15:46:33] [15:46:33.938] Removed 0 References
[INFO] [2020-08-04 15:46:33] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:33] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:34] Starting batch with ID 80704862...
[INFO] [2020-08-04 15:46:35] Starting batch with ID 80704866...
[INFO] [2020-08-04 15:46:35] ## remove_type: Node
[INFO] [2020-08-04 15:46:35] ++ Calling delete_all on 53 instances...
[INFO] [2020-08-04 15:46:35] [15:46:35.168] Removed 53 Nodes
[START] [2020-08-04 15:46:35] logged process
[START] [2020-08-04 15:46:35] Creating resource from OpenData
[START] [2020-08-04 15:46:35] logged process
[START] [2020-08-04 15:46:35] Parse meta.xml file and create formats with fields
[STOP] [2020-08-04 15:46:35] Parse meta.xml file and create formats with fields
[STOP] [2020-08-04 15:46:35] Creating resource from OpenData
[START] [2020-08-04 15:46:35] logged process
[START] [2020-08-04 15:46:35] create_harvest_instance
[STOP] [2020-08-04 15:46:36] create_harvest_instance
[START] [2020-08-04 15:46:36] fetch_files
[STOP] [2020-08-04 15:46:36] fetch_files
[START] [2020-08-04 15:46:36] validate_each_file
[STOP] [2020-08-04 15:46:36] validate_each_file
[START] [2020-08-04 15:46:36] convert_to_csv
[CMD] [2020-08-04 15:46:36] /usr/bin/sort /app/public/converted_csv/lafferty_shaw_la_nodes_22464.csv > /app/public/converted_csv/lafferty_shaw_la_nodes_22464.csv_sorted
[CMD] [2020-08-04 15:46:36] /usr/bin/sort /app/public/converted_csv/lafferty_shaw_la_occurrences_22465.csv > /app/public/converted_csv/lafferty_shaw_la_occurrences_22465.csv_sorted
[CMD] [2020-08-04 15:46:36] /usr/bin/sort /app/public/converted_csv/lafferty_shaw_la_measurements_22466.csv > /app/public/converted_csv/lafferty_shaw_la_measurements_22466.csv_sorted
[STOP] [2020-08-04 15:46:36] convert_to_csv
[START] [2020-08-04 15:46:36] calculate_delta
[CMD] [2020-08-04 15:46:36] echo "0a" > /app/public/diff/lafferty_shaw_la_nodes_22464.diff
[CMD] [2020-08-04 15:46:36] tail -n +1 /app/public/converted_csv/lafferty_shaw_la_nodes_22464.csv >> /app/public/diff/lafferty_shaw_la_nodes_22464.diff
[CMD] [2020-08-04 15:46:36] echo "." >> /app/public/diff/lafferty_shaw_la_nodes_22464.diff
[CMD] [2020-08-04 15:46:36] echo "0a" > /app/public/diff/lafferty_shaw_la_occurrences_22465.diff
[CMD] [2020-08-04 15:46:36] tail -n +1 /app/public/converted_csv/lafferty_shaw_la_occurrences_22465.csv >> /app/public/diff/lafferty_shaw_la_occurrences_22465.diff
[CMD] [2020-08-04 15:46:36] echo "." >> /app/public/diff/lafferty_shaw_la_occurrences_22465.diff
[CMD] [2020-08-04 15:46:36] echo "0a" > /app/public/diff/lafferty_shaw_la_measurements_22466.diff
[CMD] [2020-08-04 15:46:36] tail -n +1 /app/public/converted_csv/lafferty_shaw_la_measurements_22466.csv >> /app/public/diff/lafferty_shaw_la_measurements_22466.diff
[CMD] [2020-08-04 15:46:36] echo "." >> /app/public/diff/lafferty_shaw_la_measurements_22466.diff
[STOP] [2020-08-04 15:46:36] calculate_delta
[START] [2020-08-04 15:46:36] parse_diff_and_store
[INFO] [2020-08-04 15:46:36] Loading nodes diff file into memory (true lines)...
[INFO] [2020-08-04 15:46:36] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-08-04 15:46:36] Loading measurements diff file into memory (true lines)...
[INFO] [2020-08-04 15:46:38] Storing 53 ScientificNames
[INFO] [2020-08-04 15:46:38] Processing group of 53 in 1 groups of 1000
[INFO] [2020-08-04 15:46:38] Average Time: 0.02
[INFO] [2020-08-04 15:46:38] Total Time: 1s
[INFO] [2020-08-04 15:46:38] Storing 53 Nodes
[INFO] [2020-08-04 15:46:38] Processing group of 53 in 1 groups of 1000
[INFO] [2020-08-04 15:46:38] Average Time: 0.02
[INFO] [2020-08-04 15:46:38] Total Time: 1s
[INFO] [2020-08-04 15:46:38] Storing 53 Occurrences
[INFO] [2020-08-04 15:46:38] Processing group of 53 in 1 groups of 1000
[INFO] [2020-08-04 15:46:38] Average Time: 0.01
[INFO] [2020-08-04 15:46:38] Total Time: 1s
[INFO] [2020-08-04 15:46:38] Storing 446 Traits
[INFO] [2020-08-04 15:46:38] Processing group of 446 in 1 groups of 1000
[INFO] [2020-08-04 15:46:38] Average Time: 0.11
[INFO] [2020-08-04 15:46:38] Total Time: 1s
[INFO] [2020-08-04 15:46:38] Storing 378 MetaTraits
[INFO] [2020-08-04 15:46:38] Processing group of 378 in 1 groups of 1000
[INFO] [2020-08-04 15:46:38] Average Time: 0.03
[INFO] [2020-08-04 15:46:38] Total Time: 1s
[STOP] [2020-08-04 15:46:38] parse_diff_and_store
[START] [2020-08-04 15:46:38] resolve_keys
[INFO] [2020-08-04 15:46:45] Occurrences to nodes (through scientific_names)...
[INFO] [2020-08-04 15:46:45] traits to occurrences...
[INFO] [2020-08-04 15:46:45] traits to nodes (through occurrences)...
[INFO] [2020-08-04 15:46:45] Traits to sex term...
[INFO] [2020-08-04 15:46:45] Traits to lifestage term...
[INFO] [2020-08-04 15:46:45] MetaTraits to traits...
[INFO] [2020-08-04 15:46:45] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-08-04 15:46:45] Assocs to occurrences...
[INFO] [2020-08-04 15:46:45] Assocs to nodes...
[INFO] [2020-08-04 15:46:45] Assoc to sex term...
[INFO] [2020-08-04 15:46:45] Assoc to lifestage term...
[STOP] [2020-08-04 15:46:45] resolve_keys
[START] [2020-08-04 15:46:45] hold_for_later_1
[STOP] [2020-08-04 15:46:45] hold_for_later_1
[START] [2020-08-04 15:46:45] hold_for_later_2
[STOP] [2020-08-04 15:46:45] hold_for_later_2
[START] [2020-08-04 15:46:45] resolve_missing_parents
[STOP] [2020-08-04 15:46:45] resolve_missing_parents
[START] [2020-08-04 15:46:45] rebuild_nodes
[START] [2020-08-04 15:46:45] Flattener#flatten
[START] [2020-08-04 15:46:45] Flattener#study_resource
[START] [2020-08-04 15:46:45] Flattener#build_ancestry
[STOP] [2020-08-04 15:46:45] Flattener#build_ancestry
[INFO] [2020-08-04 15:46:45] 53 ancestry keys
[START] [2020-08-04 15:46:45] build_node_ancestors
[INFO] [2020-08-04 15:46:45] old ancestors deleted.
[STOP] [2020-08-04 15:46:45] build_node_ancestors
[WARN] [2020-08-04 15:46:45] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-08-04 15:46:45] Flattener#flatten
[STOP] [2020-08-04 15:46:45] rebuild_nodes
[START] [2020-08-04 15:46:45] resolve_missing_media_owners
[STOP] [2020-08-04 15:46:45] resolve_missing_media_owners
[START] [2020-08-04 15:46:45] sanitize_media_verbatims
[STOP] [2020-08-04 15:46:45] sanitize_media_verbatims
[START] [2020-08-04 15:46:45] queue_downloads
[STOP] [2020-08-04 15:46:45] queue_downloads
[START] [2020-08-04 15:46:45] parse_names
[WARN] [2020-08-04 15:46:45] I see 53 names which still need to be parsed.
[STOP] [2020-08-04 15:46:46] parse_names
[START] [2020-08-04 15:46:46] denormalize_canonical_names_to_nodes
[STOP] [2020-08-04 15:46:46] denormalize_canonical_names_to_nodes
[START] [2020-08-04 15:46:46] match_nodes
[START] [2020-08-04 15:46:46] map_all_nodes_to_pages
[STOP] [2020-08-04 15:46:47] map_all_nodes_to_pages
[INFO] [2020-08-04 15:46:47] Unmatched nodes (3 of 53): Crassiphiala (#80704920); Nanophyetes (#80704940); Ornithodiplostomum (#80704943)
[START] [2020-08-04 15:46:47] update_nodes
[STOP] [2020-08-04 15:46:47] update_nodes
[STOP] [2020-08-04 15:46:47] match_nodes
[START] [2020-08-04 15:46:47] reindex_search
[STOP] [2020-08-04 15:46:47] reindex_search
[START] [2020-08-04 15:46:47] normalize_units
[STOP] [2020-08-04 15:46:47] normalize_units
[START] [2020-08-04 15:46:47] calculate_statistics
[2020-08-04 15:46:47] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-08-04 15:46:47] calculate_statistics
[START] [2020-08-04 15:46:47] complete_harvest_instance
[START] [2020-08-04 15:46:47] overall_tsv_creation
[INFO] [2020-08-04 15:46:47] Processing group of 53 in 1 batches of 10000
[INFO] [2020-08-04 15:47:37] 126 Traits (unfiltered)...
[INFO] [2020-08-04 15:48:14] 126 Traits (filtered)...
[INFO] [2020-08-04 15:48:14] 0 Associations (filtered)...
[INFO] [2020-08-04 15:48:14] 698 metadata added.
[INFO] [2020-08-04 15:48:14] 0 metadata added.
[INFO] [2020-08-04 15:48:14] Average Time: 59.67
[INFO] [2020-08-04 15:48:14] Total Time: 1m28s
[STOP] [2020-08-04 15:48:14] overall_tsv_creation
[INFO] [2020-08-04 15:48:14] Done. Check your files:
[INFO] [2020-08-04 15:48:14] (53 lines) /app/public/data/lafferty_shaw_la/publish_nodes.tsv
[INFO] [2020-08-04 15:48:14] (53 lines) /app/public/data/lafferty_shaw_la/publish_scientific_names.tsv
[INFO] [2020-08-04 15:48:14] (127 lines) /app/public/data/lafferty_shaw_la/publish_traits.tsv
[INFO] [2020-08-04 15:48:14] (321 lines) /app/public/data/lafferty_shaw_la/publish_metadata.tsv
[STOP] [2020-08-04 15:48:14] complete_harvest_instance
[START] [2020-08-04 15:48:14] completed
[STOP] [2020-08-04 15:48:14] completed
[STOP] [2020-08-04 15:48:14] logged process, took 98.93
[INFO] [2020-08-04 15:57:18] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-08-04 15:57:20] ## remove_type: ScientificName
[INFO] [2020-08-04 15:57:20] ++ Calling delete_all on 53 instances...
[INFO] [2020-08-04 15:57:20] [15:57:20.193] Removed 53 Scientificnames
[INFO] [2020-08-04 15:57:20] ## remove_type: Vernacular
[INFO] [2020-08-04 15:57:20] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 15:57:20] [15:57:20.196] Removed 0 Vernaculars
[INFO] [2020-08-04 15:57:20] ## remove_type: Article
[INFO] [2020-08-04 15:57:20] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 15:57:20] [15:57:20.199] Removed 0 Articles
[INFO] [2020-08-04 15:57:20] ## remove_type: Medium
[INFO] [2020-08-04 15:57:20] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 15:57:20] [15:57:20.202] Removed 0 Media
[INFO] [2020-08-04 15:57:20] ## remove_type: Trait
[INFO] [2020-08-04 15:57:20] ++ Calling delete_all on 446 instances...
[INFO] [2020-08-04 15:57:20] [15:57:20.230] Removed 446 Traits
[INFO] [2020-08-04 15:57:20] ## remove_type: MetaTrait
[INFO] [2020-08-04 15:57:20] ++ Calling delete_all on 378 instances...
[INFO] [2020-08-04 15:57:20] [15:57:20.237] Removed 378 Metatraits
[INFO] [2020-08-04 15:57:20] ## remove_type: OccurrenceMetadatum
[INFO] [2020-08-04 15:57:20] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 15:57:20] [15:57:20.240] Removed 0 Occurrencemetadata
[INFO] [2020-08-04 15:57:20] ## remove_type: Assoc
[INFO] [2020-08-04 15:57:20] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 15:57:20] [15:57:20.243] Removed 0 Assocs
[INFO] [2020-08-04 15:57:20] ## remove_type: MetaAssoc
[INFO] [2020-08-04 15:57:20] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 15:57:20] [15:57:20.245] Removed 0 Metaassocs
[INFO] [2020-08-04 15:57:20] ## remove_type: Identifier
[INFO] [2020-08-04 15:57:20] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 15:57:20] [15:57:20.247] Removed 0 Identifiers
[INFO] [2020-08-04 15:57:20] ## remove_type: Reference
[INFO] [2020-08-04 15:57:20] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 15:57:20] [15:57:20.250] Removed 0 References
[INFO] [2020-08-04 15:57:20] Starting batch with ID 80704914...
[INFO] [2020-08-04 15:57:20] Starting batch with ID 80704914...
[INFO] [2020-08-04 15:57:20] Starting batch with ID 80704914...
[INFO] [2020-08-04 15:57:20] Starting batch with ID 80704914...
[INFO] [2020-08-04 15:57:20] Starting batch with ID 80704914...
[INFO] [2020-08-04 15:57:20] Starting batch with ID 80704914...
[INFO] [2020-08-04 15:57:20] Starting batch with ID 80704914...
[INFO] [2020-08-04 15:57:20] Starting batch with ID 80704914...
[INFO] [2020-08-04 15:57:20] Starting batch with ID 80704914...
[INFO] [2020-08-04 15:57:20] Starting batch with ID 80704914...
[INFO] [2020-08-04 15:57:20] ## remove_type: Node
[INFO] [2020-08-04 15:57:20] ++ Calling delete_all on 53 instances...
[INFO] [2020-08-04 15:57:20] [15:57:20.798] Removed 53 Nodes
[START] [2020-08-04 15:57:21] logged process
[START] [2020-08-04 15:57:21] Creating resource from OpenData
[START] [2020-08-04 15:57:21] logged process
[START] [2020-08-04 15:57:21] Parse meta.xml file and create formats with fields
[STOP] [2020-08-04 15:57:21] Parse meta.xml file and create formats with fields
[STOP] [2020-08-04 15:57:21] Creating resource from OpenData
[START] [2020-08-04 15:57:21] logged process
[START] [2020-08-04 15:57:21] create_harvest_instance
[STOP] [2020-08-04 15:57:22] create_harvest_instance
[START] [2020-08-04 15:57:22] fetch_files
[STOP] [2020-08-04 15:57:22] fetch_files
[START] [2020-08-04 15:57:22] validate_each_file
[STOP] [2020-08-04 15:57:22] validate_each_file
[START] [2020-08-04 15:57:22] convert_to_csv
[CMD] [2020-08-04 15:57:22] /usr/bin/sort /app/public/converted_csv/lafferty_shaw_la_nodes_22470.csv > /app/public/converted_csv/lafferty_shaw_la_nodes_22470.csv_sorted
[CMD] [2020-08-04 15:57:22] /usr/bin/sort /app/public/converted_csv/lafferty_shaw_la_occurrences_22471.csv > /app/public/converted_csv/lafferty_shaw_la_occurrences_22471.csv_sorted
[CMD] [2020-08-04 15:57:22] /usr/bin/sort /app/public/converted_csv/lafferty_shaw_la_measurements_22472.csv > /app/public/converted_csv/lafferty_shaw_la_measurements_22472.csv_sorted
[STOP] [2020-08-04 15:57:22] convert_to_csv
[START] [2020-08-04 15:57:22] calculate_delta
[CMD] [2020-08-04 15:57:22] echo "0a" > /app/public/diff/lafferty_shaw_la_nodes_22470.diff
[CMD] [2020-08-04 15:57:22] tail -n +1 /app/public/converted_csv/lafferty_shaw_la_nodes_22470.csv >> /app/public/diff/lafferty_shaw_la_nodes_22470.diff
[CMD] [2020-08-04 15:57:22] echo "." >> /app/public/diff/lafferty_shaw_la_nodes_22470.diff
[CMD] [2020-08-04 15:57:22] echo "0a" > /app/public/diff/lafferty_shaw_la_occurrences_22471.diff
[CMD] [2020-08-04 15:57:22] tail -n +1 /app/public/converted_csv/lafferty_shaw_la_occurrences_22471.csv >> /app/public/diff/lafferty_shaw_la_occurrences_22471.diff
[CMD] [2020-08-04 15:57:22] echo "." >> /app/public/diff/lafferty_shaw_la_occurrences_22471.diff
[CMD] [2020-08-04 15:57:22] echo "0a" > /app/public/diff/lafferty_shaw_la_measurements_22472.diff
[CMD] [2020-08-04 15:57:22] tail -n +1 /app/public/converted_csv/lafferty_shaw_la_measurements_22472.csv >> /app/public/diff/lafferty_shaw_la_measurements_22472.diff
[CMD] [2020-08-04 15:57:22] echo "." >> /app/public/diff/lafferty_shaw_la_measurements_22472.diff
[STOP] [2020-08-04 15:57:22] calculate_delta
[START] [2020-08-04 15:57:22] parse_diff_and_store
[INFO] [2020-08-04 15:57:22] Loading nodes diff file into memory (true lines)...
[INFO] [2020-08-04 15:57:22] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-08-04 15:57:22] Loading measurements diff file into memory (true lines)...
[INFO] [2020-08-04 15:57:24] Storing 53 ScientificNames
[INFO] [2020-08-04 15:57:24] Processing group of 53 in 1 groups of 1000
[INFO] [2020-08-04 15:57:24] Average Time: 0.02
[INFO] [2020-08-04 15:57:24] Total Time: 1s
[INFO] [2020-08-04 15:57:24] Storing 53 Nodes
[INFO] [2020-08-04 15:57:24] Processing group of 53 in 1 groups of 1000
[INFO] [2020-08-04 15:57:24] Average Time: 0.01
[INFO] [2020-08-04 15:57:24] Total Time: 1s
[INFO] [2020-08-04 15:57:24] Storing 53 Occurrences
[INFO] [2020-08-04 15:57:24] Processing group of 53 in 1 groups of 1000
[INFO] [2020-08-04 15:57:24] Average Time: 0.01
[INFO] [2020-08-04 15:57:24] Total Time: 1s
[INFO] [2020-08-04 15:57:24] Storing 446 Traits
[INFO] [2020-08-04 15:57:24] Processing group of 446 in 1 groups of 1000
[INFO] [2020-08-04 15:57:24] Average Time: 0.11
[INFO] [2020-08-04 15:57:24] Total Time: 1s
[INFO] [2020-08-04 15:57:24] Storing 378 MetaTraits
[INFO] [2020-08-04 15:57:24] Processing group of 378 in 1 groups of 1000
[INFO] [2020-08-04 15:57:24] Average Time: 0.03
[INFO] [2020-08-04 15:57:24] Total Time: 1s
[STOP] [2020-08-04 15:57:24] parse_diff_and_store
[START] [2020-08-04 15:57:24] resolve_keys
[INFO] [2020-08-04 15:57:30] Occurrences to nodes (through scientific_names)...
[INFO] [2020-08-04 15:57:30] traits to occurrences...
[INFO] [2020-08-04 15:57:30] traits to nodes (through occurrences)...
[INFO] [2020-08-04 15:57:30] Traits to sex term...
[INFO] [2020-08-04 15:57:30] Traits to lifestage term...
[INFO] [2020-08-04 15:57:31] MetaTraits to traits...
[INFO] [2020-08-04 15:57:31] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-08-04 15:57:31] Assocs to occurrences...
[INFO] [2020-08-04 15:57:31] Assocs to nodes...
[INFO] [2020-08-04 15:57:31] Assoc to sex term...
[INFO] [2020-08-04 15:57:31] Assoc to lifestage term...
[STOP] [2020-08-04 15:57:31] resolve_keys
[START] [2020-08-04 15:57:31] hold_for_later_1
[STOP] [2020-08-04 15:57:31] hold_for_later_1
[START] [2020-08-04 15:57:31] hold_for_later_2
[STOP] [2020-08-04 15:57:31] hold_for_later_2
[START] [2020-08-04 15:57:31] resolve_missing_parents
[STOP] [2020-08-04 15:57:31] resolve_missing_parents
[START] [2020-08-04 15:57:31] rebuild_nodes
[START] [2020-08-04 15:57:31] Flattener#flatten
[START] [2020-08-04 15:57:31] Flattener#study_resource
[START] [2020-08-04 15:57:31] Flattener#build_ancestry
[STOP] [2020-08-04 15:57:31] Flattener#build_ancestry
[INFO] [2020-08-04 15:57:31] 53 ancestry keys
[START] [2020-08-04 15:57:31] build_node_ancestors
[INFO] [2020-08-04 15:57:31] old ancestors deleted.
[STOP] [2020-08-04 15:57:31] build_node_ancestors
[WARN] [2020-08-04 15:57:31] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-08-04 15:57:31] Flattener#flatten
[STOP] [2020-08-04 15:57:31] rebuild_nodes
[START] [2020-08-04 15:57:31] resolve_missing_media_owners
[STOP] [2020-08-04 15:57:31] resolve_missing_media_owners
[START] [2020-08-04 15:57:31] sanitize_media_verbatims
[STOP] [2020-08-04 15:57:31] sanitize_media_verbatims
[START] [2020-08-04 15:57:31] queue_downloads
[STOP] [2020-08-04 15:57:31] queue_downloads
[START] [2020-08-04 15:57:31] parse_names
[WARN] [2020-08-04 15:57:31] I see 53 names which still need to be parsed.
[STOP] [2020-08-04 15:57:32] parse_names
[START] [2020-08-04 15:57:32] denormalize_canonical_names_to_nodes
[STOP] [2020-08-04 15:57:32] denormalize_canonical_names_to_nodes
[START] [2020-08-04 15:57:32] match_nodes
[START] [2020-08-04 15:57:32] map_all_nodes_to_pages
[STOP] [2020-08-04 15:57:32] map_all_nodes_to_pages
[INFO] [2020-08-04 15:57:32] Unmatched nodes (3 of 53): Crassiphiala (#80704973); Nanophyetes (#80704993); Ornithodiplostomum (#80704996)
[START] [2020-08-04 15:57:32] update_nodes
[STOP] [2020-08-04 15:57:32] update_nodes
[STOP] [2020-08-04 15:57:32] match_nodes
[START] [2020-08-04 15:57:32] reindex_search
[STOP] [2020-08-04 15:57:32] reindex_search
[START] [2020-08-04 15:57:32] normalize_units
[STOP] [2020-08-04 15:57:32] normalize_units
[START] [2020-08-04 15:57:32] calculate_statistics
[2020-08-04 15:57:32] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-08-04 15:57:32] calculate_statistics
[START] [2020-08-04 15:57:32] complete_harvest_instance
[START] [2020-08-04 15:57:32] overall_tsv_creation
[INFO] [2020-08-04 15:57:32] Processing group of 53 in 1 batches of 10000
[INFO] [2020-08-04 15:58:22] 126 Traits (unfiltered)...
[INFO] [2020-08-04 15:58:59] 126 Traits (filtered)...
[INFO] [2020-08-04 15:58:59] 0 Associations (filtered)...
[INFO] [2020-08-04 15:58:59] 698 metadata added.
[INFO] [2020-08-04 15:58:59] 0 metadata added.
[INFO] [2020-08-04 15:58:59] Average Time: 59.23
[INFO] [2020-08-04 15:58:59] Total Time: 1m27s
[STOP] [2020-08-04 15:58:59] overall_tsv_creation
[INFO] [2020-08-04 15:58:59] Done. Check your files:
[INFO] [2020-08-04 15:58:59] (53 lines) /app/public/data/lafferty_shaw_la/publish_nodes.tsv
[INFO] [2020-08-04 15:58:59] (53 lines) /app/public/data/lafferty_shaw_la/publish_scientific_names.tsv
[INFO] [2020-08-04 15:58:59] (127 lines) /app/public/data/lafferty_shaw_la/publish_traits.tsv
[INFO] [2020-08-04 15:58:59] (321 lines) /app/public/data/lafferty_shaw_la/publish_metadata.tsv
[STOP] [2020-08-04 15:58:59] complete_harvest_instance
[START] [2020-08-04 15:58:59] completed
[STOP] [2020-08-04 15:58:59] completed
[STOP] [2020-08-04 15:58:59] logged process, took 98.11
[INFO] [2020-11-13 10:37:36] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-11-13 10:37:38] ## remove_type: ScientificName
[INFO] [2020-11-13 10:37:38] ++ Calling delete_all on 53 instances...
[INFO] [2020-11-13 10:37:38] [10:37:38.874] Removed 53 Scientificnames
[INFO] [2020-11-13 10:37:38] ## remove_type: Vernacular
[INFO] [2020-11-13 10:37:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:37:38] [10:37:38.877] Removed 0 Vernaculars
[INFO] [2020-11-13 10:37:38] ## remove_type: Article
[INFO] [2020-11-13 10:37:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:37:38] [10:37:38.880] Removed 0 Articles
[INFO] [2020-11-13 10:37:38] ## remove_type: Medium
[INFO] [2020-11-13 10:37:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:37:38] [10:37:38.883] Removed 0 Media
[INFO] [2020-11-13 10:37:38] ## remove_type: Trait
[INFO] [2020-11-13 10:37:38] ++ Calling delete_all on 446 instances...
[INFO] [2020-11-13 10:37:38] [10:37:38.922] Removed 446 Traits
[INFO] [2020-11-13 10:37:38] ## remove_type: MetaTrait
[INFO] [2020-11-13 10:37:38] ++ Calling delete_all on 378 instances...
[INFO] [2020-11-13 10:37:38] [10:37:38.982] Removed 378 Metatraits
[INFO] [2020-11-13 10:37:38] ## remove_type: OccurrenceMetadatum
[INFO] [2020-11-13 10:37:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:37:39] [10:37:39.000] Removed 0 Occurrencemetadata
[INFO] [2020-11-13 10:37:39] ## remove_type: Assoc
[INFO] [2020-11-13 10:37:39] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:37:39] [10:37:39.003] Removed 0 Assocs
[INFO] [2020-11-13 10:37:39] ## remove_type: MetaAssoc
[INFO] [2020-11-13 10:37:39] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:37:39] [10:37:39.006] Removed 0 Metaassocs
[INFO] [2020-11-13 10:37:39] ## remove_type: Identifier
[INFO] [2020-11-13 10:37:39] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:37:39] [10:37:39.008] Removed 0 Identifiers
[INFO] [2020-11-13 10:37:39] ## remove_type: Reference
[INFO] [2020-11-13 10:37:39] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-13 10:37:39] [10:37:39.011] Removed 0 References
[INFO] [2020-11-13 10:37:39] Starting batch with ID 80704966...
[INFO] [2020-11-13 10:37:39] Starting batch with ID 80704966...
[INFO] [2020-11-13 10:37:39] Starting batch with ID 80704966...
[INFO] [2020-11-13 10:37:39] Starting batch with ID 80704966...
[INFO] [2020-11-13 10:37:39] Starting batch with ID 80704966...
[INFO] [2020-11-13 10:37:39] Starting batch with ID 80704966...
[INFO] [2020-11-13 10:37:39] Starting batch with ID 80704966...
[INFO] [2020-11-13 10:37:39] Starting batch with ID 80704966...
[INFO] [2020-11-13 10:37:39] Starting batch with ID 80704966...
[INFO] [2020-11-13 10:37:39] Starting batch with ID 80704966...
[INFO] [2020-11-13 10:37:39] Starting batch with ID 80704966...
[INFO] [2020-11-13 10:37:39] Starting batch with ID 80704966...
[INFO] [2020-11-13 10:37:39] Starting batch with ID 80704966...
[INFO] [2020-11-13 10:37:39] Starting batch with ID 80704977...
[INFO] [2020-11-13 10:37:39] Starting batch with ID 80704967...
[INFO] [2020-11-13 10:37:39] ## remove_type: Node
[INFO] [2020-11-13 10:37:39] ++ Calling delete_all on 53 instances...
[INFO] [2020-11-13 10:37:39] [10:37:39.299] Removed 53 Nodes
[START] [2020-11-13 10:37:39] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-11-13 10:37:39] Creating resource from OpenData
[START] [2020-11-13 10:37:39] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-11-13 10:37:39] Parse meta.xml file and create formats with fields
[STOP] [2020-11-13 10:37:39] Parse meta.xml file and create formats with fields
[STOP] [2020-11-13 10:37:39] Creating resource from OpenData
[START] [2020-11-13 10:37:39] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-11-13 10:37:39] create_harvest_instance
[STOP] [2020-11-13 10:37:41] create_harvest_instance
[START] [2020-11-13 10:37:41] fetch_files
[STOP] [2020-11-13 10:37:41] fetch_files
[START] [2020-11-13 10:37:41] validate_each_file
[STOP] [2020-11-13 10:37:41] validate_each_file
[START] [2020-11-13 10:37:41] convert_to_csv
[CMD] [2020-11-13 10:37:41] /usr/bin/sort /app/public/converted_csv/lafferty_shaw_la_nodes_24127.csv > /app/public/converted_csv/lafferty_shaw_la_nodes_24127.csv_sorted
[CMD] [2020-11-13 10:37:41] /usr/bin/sort /app/public/converted_csv/lafferty_shaw_la_occurrences_24128.csv > /app/public/converted_csv/lafferty_shaw_la_occurrences_24128.csv_sorted
[CMD] [2020-11-13 10:37:41] /usr/bin/sort /app/public/converted_csv/lafferty_shaw_la_measurements_24129.csv > /app/public/converted_csv/lafferty_shaw_la_measurements_24129.csv_sorted
[STOP] [2020-11-13 10:37:41] convert_to_csv
[START] [2020-11-13 10:37:41] calculate_delta
[CMD] [2020-11-13 10:37:41] echo "0a" > /app/public/diff/lafferty_shaw_la_nodes_24127.diff
[CMD] [2020-11-13 10:37:41] tail -n +1 /app/public/converted_csv/lafferty_shaw_la_nodes_24127.csv >> /app/public/diff/lafferty_shaw_la_nodes_24127.diff
[CMD] [2020-11-13 10:37:41] echo "." >> /app/public/diff/lafferty_shaw_la_nodes_24127.diff
[CMD] [2020-11-13 10:37:41] echo "0a" > /app/public/diff/lafferty_shaw_la_occurrences_24128.diff
[CMD] [2020-11-13 10:37:41] tail -n +1 /app/public/converted_csv/lafferty_shaw_la_occurrences_24128.csv >> /app/public/diff/lafferty_shaw_la_occurrences_24128.diff
[CMD] [2020-11-13 10:37:41] echo "." >> /app/public/diff/lafferty_shaw_la_occurrences_24128.diff
[CMD] [2020-11-13 10:37:41] echo "0a" > /app/public/diff/lafferty_shaw_la_measurements_24129.diff
[CMD] [2020-11-13 10:37:41] tail -n +1 /app/public/converted_csv/lafferty_shaw_la_measurements_24129.csv >> /app/public/diff/lafferty_shaw_la_measurements_24129.diff
[CMD] [2020-11-13 10:37:41] echo "." >> /app/public/diff/lafferty_shaw_la_measurements_24129.diff
[STOP] [2020-11-13 10:37:41] calculate_delta
[START] [2020-11-13 10:37:41] parse_diff_and_store
[INFO] [2020-11-13 10:37:41] Loading nodes diff file into memory (true lines)...
[INFO] [2020-11-13 10:37:41] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-11-13 10:37:41] Loading measurements diff file into memory (true lines)...
[INFO] [2020-11-13 10:37:42] Storing 53 ScientificNames
[INFO] [2020-11-13 10:37:42] Processing group of 53 in 1 groups of 1000
[INFO] [2020-11-13 10:37:42] Average Time: 0.02
[INFO] [2020-11-13 10:37:42] Total Time: 1s
[INFO] [2020-11-13 10:37:42] Storing 53 Nodes
[INFO] [2020-11-13 10:37:42] Processing group of 53 in 1 groups of 1000
[INFO] [2020-11-13 10:37:42] Average Time: 0.02
[INFO] [2020-11-13 10:37:42] Total Time: 1s
[INFO] [2020-11-13 10:37:42] Storing 53 Occurrences
[INFO] [2020-11-13 10:37:42] Processing group of 53 in 1 groups of 1000
[INFO] [2020-11-13 10:37:42] Average Time: 0.01
[INFO] [2020-11-13 10:37:42] Total Time: 1s
[INFO] [2020-11-13 10:37:42] Storing 446 Traits
[INFO] [2020-11-13 10:37:42] Processing group of 446 in 1 groups of 1000
[INFO] [2020-11-13 10:37:42] Average Time: 0.15
[INFO] [2020-11-13 10:37:42] Total Time: 1s
[INFO] [2020-11-13 10:37:42] Storing 252 MetaTraits
[INFO] [2020-11-13 10:37:42] Processing group of 252 in 1 groups of 1000
[INFO] [2020-11-13 10:37:42] Average Time: 0.03
[INFO] [2020-11-13 10:37:42] Total Time: 1s
[STOP] [2020-11-13 10:37:42] parse_diff_and_store
[START] [2020-11-13 10:37:42] resolve_keys
[INFO] [2020-11-13 10:37:49] Occurrences to nodes (through scientific_names)...
[INFO] [2020-11-13 10:37:49] traits to occurrences...
[INFO] [2020-11-13 10:37:49] traits to nodes (through occurrences)...
[INFO] [2020-11-13 10:37:49] Traits to sex term...
[INFO] [2020-11-13 10:37:49] Traits to lifestage term...
[INFO] [2020-11-13 10:37:49] MetaTraits to traits...
[INFO] [2020-11-13 10:37:49] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-11-13 10:37:49] Assocs to occurrences...
[INFO] [2020-11-13 10:37:49] Assocs to nodes...
[INFO] [2020-11-13 10:37:49] Assoc to sex term...
[INFO] [2020-11-13 10:37:49] Assoc to lifestage term...
[INFO] [2020-11-13 10:37:49] MetaAssoc to assocs...
[STOP] [2020-11-13 10:37:49] resolve_keys
[START] [2020-11-13 10:37:49] hold_for_later_1
[STOP] [2020-11-13 10:37:49] hold_for_later_1
[START] [2020-11-13 10:37:49] hold_for_later_2
[STOP] [2020-11-13 10:37:49] hold_for_later_2
[START] [2020-11-13 10:37:49] resolve_missing_parents
[STOP] [2020-11-13 10:37:49] resolve_missing_parents
[START] [2020-11-13 10:37:49] rebuild_nodes
[START] [2020-11-13 10:37:49] Flattener#flatten
[START] [2020-11-13 10:37:49] Flattener#study_resource
[START] [2020-11-13 10:37:49] Flattener#build_ancestry
[STOP] [2020-11-13 10:37:49] Flattener#build_ancestry
[INFO] [2020-11-13 10:37:49] 53 ancestry keys
[START] [2020-11-13 10:37:49] build_node_ancestors
[INFO] [2020-11-13 10:37:49] old ancestors deleted.
[STOP] [2020-11-13 10:37:49] build_node_ancestors
[WARN] [2020-11-13 10:37:49] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-11-13 10:37:49] Flattener#flatten
[STOP] [2020-11-13 10:37:49] rebuild_nodes
[START] [2020-11-13 10:37:49] resolve_missing_media_owners
[STOP] [2020-11-13 10:37:49] resolve_missing_media_owners
[START] [2020-11-13 10:37:49] sanitize_media_verbatims
[STOP] [2020-11-13 10:37:49] sanitize_media_verbatims
[START] [2020-11-13 10:37:49] queue_downloads
[STOP] [2020-11-13 10:37:49] queue_downloads
[START] [2020-11-13 10:37:49] parse_names
[WARN] [2020-11-13 10:37:49] I see 53 names which still need to be parsed.
[STOP] [2020-11-13 10:37:50] parse_names
[START] [2020-11-13 10:37:50] denormalize_canonical_names_to_nodes
[STOP] [2020-11-13 10:37:50] denormalize_canonical_names_to_nodes
[START] [2020-11-13 10:37:50] match_nodes
[START] [2020-11-13 10:37:50] map_all_nodes_to_pages
[STOP] [2020-11-13 10:37:50] map_all_nodes_to_pages
[INFO] [2020-11-13 10:37:50] Unmatched nodes (3 of 53): Crassiphiala (#81658552); Nanophyetes (#81658572); Ornithodiplostomum (#81658575)
[START] [2020-11-13 10:37:50] update_nodes
[STOP] [2020-11-13 10:37:50] update_nodes
[STOP] [2020-11-13 10:37:50] match_nodes
[START] [2020-11-13 10:37:50] reindex_search
[STOP] [2020-11-13 10:37:50] reindex_search
[START] [2020-11-13 10:37:50] normalize_units
[STOP] [2020-11-13 10:37:50] normalize_units
[START] [2020-11-13 10:37:50] calculate_statistics
[2020-11-13 10:37:50] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-11-13 10:37:50] calculate_statistics
[START] [2020-11-13 10:37:50] complete_harvest_instance
[START] [2020-11-13 10:37:50] overall_tsv_creation
[INFO] [2020-11-13 10:37:50] Processing group of 53 in 1 batches of 10000
[INFO] [2020-11-13 10:38:30] 126 Traits (unfiltered)...
[INFO] [2020-11-13 10:39:08] 126 Traits (filtered)...
[INFO] [2020-11-13 10:39:08] 0 Associations (filtered)...
[INFO] [2020-11-13 10:39:08] 572 metadata added.
[INFO] [2020-11-13 10:39:08] 0 metadata added.
[INFO] [2020-11-13 10:39:08] Average Time: 52.87
[INFO] [2020-11-13 10:39:08] Total Time: 1m18s
[STOP] [2020-11-13 10:39:08] overall_tsv_creation
[INFO] [2020-11-13 10:39:08] Done. Check your files:
[INFO] [2020-11-13 10:39:08] (53 lines) /app/public/data/lafferty_shaw_la/publish_nodes.tsv
[INFO] [2020-11-13 10:39:08] (53 lines) /app/public/data/lafferty_shaw_la/publish_scientific_names.tsv
[INFO] [2020-11-13 10:39:08] (127 lines) /app/public/data/lafferty_shaw_la/publish_traits.tsv
[INFO] [2020-11-13 10:39:08] (573 lines) /app/public/data/lafferty_shaw_la/publish_metadata.tsv
[STOP] [2020-11-13 10:39:08] complete_harvest_instance
[START] [2020-11-13 10:39:08] completed
[STOP] [2020-11-13 10:39:08] completed
[STOP] [2020-11-13 10:39:08] logged process, took 88.52
[INFO] [2021-04-19 09:32:01] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-19 09:32:03] ## remove_type: ScientificName
[INFO] [2021-04-19 09:32:03] ++ Calling delete_all on 53 instances...
[INFO] [2021-04-19 09:32:03] [09:32:03.286] Removed 53 Scientificnames
[INFO] [2021-04-19 09:32:03] ## remove_type: Vernacular
[INFO] [2021-04-19 09:32:03] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:32:03] [09:32:03.288] Removed 0 Vernaculars
[INFO] [2021-04-19 09:32:03] ## remove_type: Article
[INFO] [2021-04-19 09:32:03] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:32:03] [09:32:03.290] Removed 0 Articles
[INFO] [2021-04-19 09:32:03] ## remove_type: Medium
[INFO] [2021-04-19 09:32:03] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:32:03] [09:32:03.291] Removed 0 Media
[INFO] [2021-04-19 09:32:03] ## remove_type: Trait
[INFO] [2021-04-19 09:32:03] ++ Calling delete_all on 446 instances...
[INFO] [2021-04-19 09:32:03] [09:32:03.393] Removed 446 Traits
[INFO] [2021-04-19 09:32:03] ## remove_type: MetaTrait
[INFO] [2021-04-19 09:32:03] ++ Calling delete_all on 252 instances...
[INFO] [2021-04-19 09:32:03] [09:32:03.400] Removed 252 Metatraits
[INFO] [2021-04-19 09:32:03] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-19 09:32:03] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:32:03] [09:32:03.401] Removed 0 Occurrencemetadata
[INFO] [2021-04-19 09:32:03] ## remove_type: Assoc
[INFO] [2021-04-19 09:32:03] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:32:03] [09:32:03.403] Removed 0 Assocs
[INFO] [2021-04-19 09:32:03] ## remove_type: MetaAssoc
[INFO] [2021-04-19 09:32:03] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:32:03] [09:32:03.404] Removed 0 Metaassocs
[INFO] [2021-04-19 09:32:03] ## remove_type: Identifier
[INFO] [2021-04-19 09:32:03] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:32:03] [09:32:03.406] Removed 0 Identifiers
[INFO] [2021-04-19 09:32:03] ## remove_type: Reference
[INFO] [2021-04-19 09:32:03] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:32:03] [09:32:03.408] Removed 0 References
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:03] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:04] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:04] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:04] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:04] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:04] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:04] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:04] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:04] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:04] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:04] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:04] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:04] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:04] Starting batch with ID 81658546...
[INFO] [2021-04-19 09:32:04] ## remove_type: Node
[INFO] [2021-04-19 09:32:04] ++ Calling delete_all on 53 instances...
[INFO] [2021-04-19 09:32:04] [09:32:04.297] Removed 53 Nodes
[START] [2021-04-19 09:32:04] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 09:32:04] Creating resource from OpenData
[START] [2021-04-19 09:32:05] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 09:32:05] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 09:32:06] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 09:32:06] Creating resource from OpenData
[INFO] [2021-04-19 09:36:22] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-19 09:36:23] ## remove_type: ScientificName
[INFO] [2021-04-19 09:36:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:36:23] [09:36:23.822] Removed 0 Scientificnames
[INFO] [2021-04-19 09:36:23] ## remove_type: Vernacular
[INFO] [2021-04-19 09:36:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:36:23] [09:36:23.823] Removed 0 Vernaculars
[INFO] [2021-04-19 09:36:23] ## remove_type: Article
[INFO] [2021-04-19 09:36:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:36:23] [09:36:23.825] Removed 0 Articles
[INFO] [2021-04-19 09:36:23] ## remove_type: Medium
[INFO] [2021-04-19 09:36:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:36:23] [09:36:23.826] Removed 0 Media
[INFO] [2021-04-19 09:36:23] ## remove_type: Trait
[INFO] [2021-04-19 09:36:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:36:23] [09:36:23.828] Removed 0 Traits
[INFO] [2021-04-19 09:36:23] ## remove_type: MetaTrait
[INFO] [2021-04-19 09:36:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:36:23] [09:36:23.829] Removed 0 Metatraits
[INFO] [2021-04-19 09:36:23] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-19 09:36:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:36:23] [09:36:23.830] Removed 0 Occurrencemetadata
[INFO] [2021-04-19 09:36:23] ## remove_type: Assoc
[INFO] [2021-04-19 09:36:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:36:23] [09:36:23.832] Removed 0 Assocs
[INFO] [2021-04-19 09:36:23] ## remove_type: MetaAssoc
[INFO] [2021-04-19 09:36:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:36:23] [09:36:23.833] Removed 0 Metaassocs
[INFO] [2021-04-19 09:36:23] ## remove_type: Identifier
[INFO] [2021-04-19 09:36:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:36:23] [09:36:23.835] Removed 0 Identifiers
[INFO] [2021-04-19 09:36:23] ## remove_type: Reference
[INFO] [2021-04-19 09:36:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:36:23] [09:36:23.836] Removed 0 References
[INFO] [2021-04-19 09:36:23] ## remove_type: Node
[INFO] [2021-04-19 09:36:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 09:36:23] [09:36:23.846] Removed 0 Nodes
[START] [2021-04-19 09:36:24] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 09:36:24] Creating resource from OpenData
[START] [2021-04-19 09:36:25] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 09:36:25] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 09:36:28] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 09:36:28] Creating resource from OpenData
[START] [2021-04-19 09:36:29] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 09:36:29] create_harvest_instance
[INFO] [2021-04-19 09:36:29] Created harvest instance #3717
[STOP] [2021-04-19 09:36:29] create_harvest_instance
[START] [2021-04-19 09:36:29] fetch_files
[STOP] [2021-04-19 09:36:29] fetch_files
[START] [2021-04-19 09:36:29] validate_each_file
[INFO] [2021-04-19 09:36:29] Looping over 3 formats...
[INFO] [2021-04-19 09:36:29] ...nodes (/app/public/data/lafferty_shaw_la/taxa.txt)
[INFO] [2021-04-19 09:36:29] Valid: /app/public/converted_csv/lafferty_shaw_la_nodes_3717.csv (53 lines)
[INFO] [2021-04-19 09:36:29] ...occurrences (/app/public/data/lafferty_shaw_la/occurrences.txt)
[INFO] [2021-04-19 09:36:29] Valid: /app/public/converted_csv/lafferty_shaw_la_occurrences_3717.csv (53 lines)
[INFO] [2021-04-19 09:36:29] ...measurements (/app/public/data/lafferty_shaw_la/measurementsorfacts.txt)
[INFO] [2021-04-19 09:36:29] Valid: /app/public/converted_csv/lafferty_shaw_la_measurements_3717.csv (446 lines)
[STOP] [2021-04-19 09:36:29] validate_each_file
[START] [2021-04-19 09:36:29] convert_to_csv
[INFO] [2021-04-19 09:36:29] Looping over 3 formats...
[INFO] [2021-04-19 09:36:29] ...nodes (/app/public/data/lafferty_shaw_la/taxa.txt)
[CMD] [2021-04-19 09:36:29] /usr/bin/sort /app/public/converted_csv/lafferty_shaw_la_nodes_3717.csv > /app/public/converted_csv/lafferty_shaw_la_nodes_3717.csv_sorted
[INFO] [2021-04-19 09:36:29] Converted: /app/public/converted_csv/lafferty_shaw_la_nodes_3717.csv (53 lines)
[INFO] [2021-04-19 09:36:29] ...occurrences (/app/public/data/lafferty_shaw_la/occurrences.txt)
[CMD] [2021-04-19 09:36:29] /usr/bin/sort /app/public/converted_csv/lafferty_shaw_la_occurrences_3717.csv > /app/public/converted_csv/lafferty_shaw_la_occurrences_3717.csv_sorted
[INFO] [2021-04-19 09:36:30] Converted: /app/public/converted_csv/lafferty_shaw_la_occurrences_3717.csv (53 lines)
[INFO] [2021-04-19 09:36:30] ...measurements (/app/public/data/lafferty_shaw_la/measurementsorfacts.txt)
[CMD] [2021-04-19 09:36:30] /usr/bin/sort /app/public/converted_csv/lafferty_shaw_la_measurements_3717.csv > /app/public/converted_csv/lafferty_shaw_la_measurements_3717.csv_sorted
[INFO] [2021-04-19 09:36:30] Converted: /app/public/converted_csv/lafferty_shaw_la_measurements_3717.csv (446 lines)
[STOP] [2021-04-19 09:36:30] convert_to_csv
[START] [2021-04-19 09:36:30] calculate_delta
[INFO] [2021-04-19 09:36:30] Looping over 3 formats...
[INFO] [2021-04-19 09:36:30] ...nodes (/app/public/data/lafferty_shaw_la/taxa.txt)
[CMD] [2021-04-19 09:36:30] echo "0a" > /app/public/diff/lafferty_shaw_la_nodes_3717.diff
[CMD] [2021-04-19 09:36:30] tail -n +1 /app/public/converted_csv/lafferty_shaw_la_nodes_3717.csv >> /app/public/diff/lafferty_shaw_la_nodes_3717.diff
[CMD] [2021-04-19 09:36:31] echo "." >> /app/public/diff/lafferty_shaw_la_nodes_3717.diff
[INFO] [2021-04-19 09:36:31] Created diff: /app/public/diff/lafferty_shaw_la_nodes_3717.diff (55 lines)
[INFO] [2021-04-19 09:36:31] ...occurrences (/app/public/data/lafferty_shaw_la/occurrences.txt)
[CMD] [2021-04-19 09:36:31] echo "0a" > /app/public/diff/lafferty_shaw_la_occurrences_3717.diff
[CMD] [2021-04-19 09:36:32] tail -n +1 /app/public/converted_csv/lafferty_shaw_la_occurrences_3717.csv >> /app/public/diff/lafferty_shaw_la_occurrences_3717.diff
[CMD] [2021-04-19 09:36:32] echo "." >> /app/public/diff/lafferty_shaw_la_occurrences_3717.diff
[INFO] [2021-04-19 09:36:33] Created diff: /app/public/diff/lafferty_shaw_la_occurrences_3717.diff (55 lines)
[INFO] [2021-04-19 09:36:33] ...measurements (/app/public/data/lafferty_shaw_la/measurementsorfacts.txt)
[CMD] [2021-04-19 09:36:33] echo "0a" > /app/public/diff/lafferty_shaw_la_measurements_3717.diff
[CMD] [2021-04-19 09:36:33] tail -n +1 /app/public/converted_csv/lafferty_shaw_la_measurements_3717.csv >> /app/public/diff/lafferty_shaw_la_measurements_3717.diff
[CMD] [2021-04-19 09:36:33] echo "." >> /app/public/diff/lafferty_shaw_la_measurements_3717.diff
[INFO] [2021-04-19 09:36:34] Created diff: /app/public/diff/lafferty_shaw_la_measurements_3717.diff (448 lines)
[STOP] [2021-04-19 09:36:34] calculate_delta
[START] [2021-04-19 09:36:34] parse_diff_and_store
[INFO] [2021-04-19 09:36:34] Handling diff: /app/public/diff/lafferty_shaw_la_nodes_3717.diff (55 lines)
[INFO] [2021-04-19 09:36:34] Loading nodes diff file into memory (55 /app/public/diff/lafferty_shaw_la_nodes_3717.diff lines)...
[INFO] [2021-04-19 09:36:35] Handling diff: /app/public/diff/lafferty_shaw_la_occurrences_3717.diff (55 lines)
[INFO] [2021-04-19 09:36:35] Loading occurrences diff file into memory (55 /app/public/diff/lafferty_shaw_la_occurrences_3717.diff lines)...
[INFO] [2021-04-19 09:36:35] Handling diff: /app/public/diff/lafferty_shaw_la_measurements_3717.diff (448 lines)
[INFO] [2021-04-19 09:36:36] Loading measurements diff file into memory (448 /app/public/diff/lafferty_shaw_la_measurements_3717.diff lines)...
[INFO] [2021-04-19 09:36:36] Storing 54 ScientificNames
[INFO] [2021-04-19 09:36:36] Processing group of 54 in 1 groups of 1000
[INFO] [2021-04-19 09:36:36] Average Time: 0.02
[INFO] [2021-04-19 09:36:36] Total Time: 1s
[INFO] [2021-04-19 09:36:36] Storing 54 Nodes
[INFO] [2021-04-19 09:36:36] Processing group of 54 in 1 groups of 1000
[INFO] [2021-04-19 09:36:36] Average Time: 0.04
[INFO] [2021-04-19 09:36:36] Total Time: 1s
[INFO] [2021-04-19 09:36:36] Storing 53 Occurrences
[INFO] [2021-04-19 09:36:36] Processing group of 53 in 1 groups of 1000
[INFO] [2021-04-19 09:36:36] Average Time: 0.01
[INFO] [2021-04-19 09:36:36] Total Time: 1s
[INFO] [2021-04-19 09:36:36] Storing 446 Traits
[INFO] [2021-04-19 09:36:36] Processing group of 446 in 1 groups of 1000
[INFO] [2021-04-19 09:36:37] Average Time: 0.13
[INFO] [2021-04-19 09:36:37] Total Time: 1s
[INFO] [2021-04-19 09:36:37] Storing 252 MetaTraits
[INFO] [2021-04-19 09:36:37] Processing group of 252 in 1 groups of 1000
[INFO] [2021-04-19 09:36:37] Average Time: 0.03
[INFO] [2021-04-19 09:36:37] Total Time: 1s
[STOP] [2021-04-19 09:36:37] parse_diff_and_store
[START] [2021-04-19 09:36:37] resolve_keys
[INFO] [2021-04-19 09:36:43] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-19 09:36:43] traits to occurrences...
[INFO] [2021-04-19 09:36:43] traits to nodes (through occurrences)...
[INFO] [2021-04-19 09:36:43] Traits to sex term...
[INFO] [2021-04-19 09:36:43] Traits to lifestage term...
[INFO] [2021-04-19 09:36:43] MetaTraits to traits...
[INFO] [2021-04-19 09:36:43] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-19 09:36:43] Assocs to occurrences...
[INFO] [2021-04-19 09:36:43] Assocs to nodes...
[INFO] [2021-04-19 09:36:43] Assoc to sex term...
[INFO] [2021-04-19 09:36:43] Assoc to lifestage term...
[INFO] [2021-04-19 09:36:43] MetaAssoc to assocs...
[STOP] [2021-04-19 09:36:43] resolve_keys
[START] [2021-04-19 09:36:43] hold_for_later_1
[STOP] [2021-04-19 09:36:43] hold_for_later_1
[START] [2021-04-19 09:36:43] hold_for_later_2
[STOP] [2021-04-19 09:36:43] hold_for_later_2
[START] [2021-04-19 09:36:43] resolve_missing_parents
[STOP] [2021-04-19 09:36:43] resolve_missing_parents
[START] [2021-04-19 09:36:43] rebuild_nodes
[START] [2021-04-19 09:36:43] Flattener#flatten
[START] [2021-04-19 09:36:43] Flattener#study_resource
[START] [2021-04-19 09:36:43] Flattener#build_ancestry
[STOP] [2021-04-19 09:36:43] Flattener#build_ancestry
[INFO] [2021-04-19 09:36:43] 54 ancestry keys
[START] [2021-04-19 09:36:43] build_node_ancestors
[INFO] [2021-04-19 09:36:43] old ancestors deleted.
[STOP] [2021-04-19 09:36:43] build_node_ancestors
[START] [2021-04-19 09:36:43] Flattener#propagate_ancestor_ids
[STOP] [2021-04-19 09:36:43] Flattener#propagate_ancestor_ids
[STOP] [2021-04-19 09:36:43] Flattener#flatten
[STOP] [2021-04-19 09:36:43] rebuild_nodes
[START] [2021-04-19 09:36:43] resolve_missing_media_owners
[STOP] [2021-04-19 09:36:43] resolve_missing_media_owners
[START] [2021-04-19 09:36:43] sanitize_media_verbatims
[STOP] [2021-04-19 09:36:43] sanitize_media_verbatims
[START] [2021-04-19 09:36:43] queue_downloads
[STOP] [2021-04-19 09:36:43] queue_downloads
[START] [2021-04-19 09:36:43] parse_names
[WARN] [2021-04-19 09:36:43] I see 54 names which still need to be parsed.
[STOP] [2021-04-19 09:36:44] parse_names
[START] [2021-04-19 09:36:44] denormalize_canonical_names_to_nodes
[STOP] [2021-04-19 09:36:44] denormalize_canonical_names_to_nodes
[START] [2021-04-19 09:36:44] match_nodes
[START] [2021-04-19 09:36:44] map_all_nodes_to_pages
[STOP] [2021-04-19 09:36:50] map_all_nodes_to_pages
[INFO] [2021-04-19 09:36:50] Unmatched nodes (3 of 54): Crassiphiala (#92872210); Nanophyetes (#92872230); Ornithodiplostomum (#92872233)
[START] [2021-04-19 09:36:50] update_nodes
[STOP] [2021-04-19 09:36:50] update_nodes
[STOP] [2021-04-19 09:36:50] match_nodes
[START] [2021-04-19 09:36:50] reindex_search
[STOP] [2021-04-19 09:36:50] reindex_search
[START] [2021-04-19 09:36:50] normalize_units
[STOP] [2021-04-19 09:36:50] normalize_units
[START] [2021-04-19 09:36:50] calculate_statistics
[STOP] [2021-04-19 09:36:50] calculate_statistics
[START] [2021-04-19 09:36:50] complete_harvest_instance
[START] [2021-04-19 09:36:50] overall_tsv_creation
[INFO] [2021-04-19 09:36:51] Processing group of 54 in 1 batches of 10000
[INFO] [2021-04-19 09:37:27] 126 Traits (unfiltered)...
[INFO] [2021-04-19 09:38:01] 126 Traits (filtered)...
[INFO] [2021-04-19 09:38:01] 0 Associations (filtered)...
[INFO] [2021-04-19 09:38:02] 320 metadata added.
[INFO] [2021-04-19 09:38:02] 0 metadata added.
[INFO] [2021-04-19 09:38:28] Average Time: 73.08
[INFO] [2021-04-19 09:38:28] Total Time: 1m38s
[STOP] [2021-04-19 09:38:28] overall_tsv_creation
[INFO] [2021-04-19 09:38:28] Done. Check your files:
[INFO] [2021-04-19 09:38:28] (54 lines) /app/public/data/lafferty_shaw_la/publish_nodes.tsv
[INFO] [2021-04-19 09:38:29] (3 lines) /app/public/data/lafferty_shaw_la/publish_node_ancestors.tsv
[INFO] [2021-04-19 09:38:29] (54 lines) /app/public/data/lafferty_shaw_la/publish_scientific_names.tsv
[INFO] [2021-04-19 09:38:30] (127 lines) /app/public/data/lafferty_shaw_la/publish_traits.tsv
[INFO] [2021-04-19 09:38:30] (321 lines) /app/public/data/lafferty_shaw_la/publish_metadata.tsv
[STOP] [2021-04-19 09:38:30] complete_harvest_instance
[START] [2021-04-19 09:38:30] completed
[STOP] [2021-04-19 09:38:30] completed
[STOP] [2021-04-19 09:38:30] logged process, took 121.73

Latest Process

19 Apr 09:32 EXITED currently processing 0