Harvest for Proudlove 2020 Created 19 Apr 15:42

Harvesting Log

# Logfile created on 2020-08-27 15:44:51 -0400 by logger.rb/v1.4.2
[START] [2020-08-27 15:44:51] logged process
[START] [2020-08-27 15:44:51] Creating resource from OpenData
[START] [2020-08-27 15:44:51] logged process
[START] [2020-08-27 15:44:51] Parse meta.xml file and create formats with fields
[STOP] [2020-08-27 15:44:52] Parse meta.xml file and create formats with fields
[STOP] [2020-08-27 15:44:52] Creating resource from OpenData
[INFO] [2020-12-03 11:39:38] ## HARVEST: type = -harvest
[START] [2020-12-03 11:39:41] logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

[START] [2020-12-03 11:39:41] create_harvest_instance
[STOP] [2020-12-03 11:39:42] create_harvest_instance
[START] [2020-12-03 11:39:42] fetch_files
[STOP] [2020-12-03 11:39:42] fetch_files
[START] [2020-12-03 11:39:42] validate_each_file
[STOP] [2020-12-03 11:39:43] validate_each_file
[START] [2020-12-03 11:39:43] convert_to_csv
[CMD] [2020-12-03 11:39:43] /usr/bin/sort /app/public/converted_csv/proudlove_proudl_refs_24832.csv > /app/public/converted_csv/proudlove_proudl_refs_24832.csv_sorted
[CMD] [2020-12-03 11:39:43] /usr/bin/sort /app/public/converted_csv/proudlove_proudl_nodes_24833.csv > /app/public/converted_csv/proudlove_proudl_nodes_24833.csv_sorted
[CMD] [2020-12-03 11:39:43] /usr/bin/sort /app/public/converted_csv/proudlove_proudl_occurrences_24834.csv > /app/public/converted_csv/proudlove_proudl_occurrences_24834.csv_sorted
[CMD] [2020-12-03 11:39:43] /usr/bin/sort /app/public/converted_csv/proudlove_proudl_measurements_24835.csv > /app/public/converted_csv/proudlove_proudl_measurements_24835.csv_sorted
[STOP] [2020-12-03 11:39:43] convert_to_csv
[START] [2020-12-03 11:39:43] calculate_delta
[CMD] [2020-12-03 11:39:43] echo "0a" > /app/public/diff/proudlove_proudl_refs_24832.diff
[CMD] [2020-12-03 11:39:43] tail -n +1 /app/public/converted_csv/proudlove_proudl_refs_24832.csv >> /app/public/diff/proudlove_proudl_refs_24832.diff
[CMD] [2020-12-03 11:39:43] echo "." >> /app/public/diff/proudlove_proudl_refs_24832.diff
[CMD] [2020-12-03 11:39:43] echo "0a" > /app/public/diff/proudlove_proudl_nodes_24833.diff
[CMD] [2020-12-03 11:39:43] tail -n +1 /app/public/converted_csv/proudlove_proudl_nodes_24833.csv >> /app/public/diff/proudlove_proudl_nodes_24833.diff
[CMD] [2020-12-03 11:39:43] echo "." >> /app/public/diff/proudlove_proudl_nodes_24833.diff
[CMD] [2020-12-03 11:39:43] echo "0a" > /app/public/diff/proudlove_proudl_occurrences_24834.diff
[CMD] [2020-12-03 11:39:43] tail -n +1 /app/public/converted_csv/proudlove_proudl_occurrences_24834.csv >> /app/public/diff/proudlove_proudl_occurrences_24834.diff
[CMD] [2020-12-03 11:39:43] echo "." >> /app/public/diff/proudlove_proudl_occurrences_24834.diff
[CMD] [2020-12-03 11:39:43] echo "0a" > /app/public/diff/proudlove_proudl_measurements_24835.diff
[CMD] [2020-12-03 11:39:43] tail -n +1 /app/public/converted_csv/proudlove_proudl_measurements_24835.csv >> /app/public/diff/proudlove_proudl_measurements_24835.diff
[CMD] [2020-12-03 11:39:43] echo "." >> /app/public/diff/proudlove_proudl_measurements_24835.diff
[STOP] [2020-12-03 11:39:43] calculate_delta
[START] [2020-12-03 11:39:43] parse_diff_and_store
[INFO] [2020-12-03 11:39:43] Loading refs diff file into memory (true lines)...
[INFO] [2020-12-03 11:39:43] Loading nodes diff file into memory (true lines)...
[INFO] [2020-12-03 11:39:43] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-12-03 11:39:43] Loading measurements diff file into memory (true lines)...
[INFO] [2020-12-03 11:39:44] Storing 216 ScientificNames
[INFO] [2020-12-03 11:39:44] Processing group of 216 in 1 groups of 1000
[INFO] [2020-12-03 11:39:44] Average Time: 0.13
[INFO] [2020-12-03 11:39:44] Total Time: 1s
[INFO] [2020-12-03 11:39:44] Storing 216 Nodes
[INFO] [2020-12-03 11:39:44] Processing group of 216 in 1 groups of 1000
[INFO] [2020-12-03 11:39:44] Average Time: 0.09
[INFO] [2020-12-03 11:39:44] Total Time: 1s
[INFO] [2020-12-03 11:39:44] Storing 262 Occurrences
[INFO] [2020-12-03 11:39:44] Processing group of 262 in 1 groups of 1000
[INFO] [2020-12-03 11:39:44] Average Time: 0.1
[INFO] [2020-12-03 11:39:44] Total Time: 1s
[INFO] [2020-12-03 11:39:44] Storing 692 Traits
[INFO] [2020-12-03 11:39:44] Processing group of 692 in 1 groups of 1000
[INFO] [2020-12-03 11:39:45] Average Time: 0.3
[INFO] [2020-12-03 11:39:45] Total Time: 1s
[INFO] [2020-12-03 11:39:45] Storing 692 MetaTraits
[INFO] [2020-12-03 11:39:45] Processing group of 692 in 1 groups of 1000
[INFO] [2020-12-03 11:39:45] Average Time: 0.09
[INFO] [2020-12-03 11:39:45] Total Time: 1s
[STOP] [2020-12-03 11:39:45] parse_diff_and_store
[START] [2020-12-03 11:39:45] resolve_keys
[INFO] [2020-12-03 11:39:51] Occurrences to nodes (through scientific_names)...
[INFO] [2020-12-03 11:39:51] traits to occurrences...
[INFO] [2020-12-03 11:39:51] traits to nodes (through occurrences)...
[INFO] [2020-12-03 11:39:51] Traits to sex term...
[INFO] [2020-12-03 11:39:51] Traits to lifestage term...
[INFO] [2020-12-03 11:39:51] MetaTraits to traits...
[INFO] [2020-12-03 11:39:51] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-12-03 11:39:51] Assocs to occurrences...
[INFO] [2020-12-03 11:39:51] Assocs to nodes...
[INFO] [2020-12-03 11:39:51] Assoc to sex term...
[INFO] [2020-12-03 11:39:51] Assoc to lifestage term...
[INFO] [2020-12-03 11:39:51] MetaAssoc to assocs...
[STOP] [2020-12-03 11:39:51] resolve_keys
[START] [2020-12-03 11:39:51] hold_for_later_1
[STOP] [2020-12-03 11:39:51] hold_for_later_1
[START] [2020-12-03 11:39:51] hold_for_later_2
[STOP] [2020-12-03 11:39:51] hold_for_later_2
[START] [2020-12-03 11:39:51] resolve_missing_parents
[STOP] [2020-12-03 11:39:51] resolve_missing_parents
[START] [2020-12-03 11:39:51] rebuild_nodes
[START] [2020-12-03 11:39:51] Flattener#flatten
[START] [2020-12-03 11:39:51] Flattener#study_resource
[START] [2020-12-03 11:39:51] Flattener#build_ancestry
[STOP] [2020-12-03 11:39:51] Flattener#build_ancestry
[INFO] [2020-12-03 11:39:51] 216 ancestry keys
[START] [2020-12-03 11:39:51] build_node_ancestors
[INFO] [2020-12-03 11:39:51] old ancestors deleted.
[STOP] [2020-12-03 11:39:51] build_node_ancestors
[START] [2020-12-03 11:39:51] Flattener#propagate_ancestor_ids
[STOP] [2020-12-03 11:39:51] Flattener#propagate_ancestor_ids
[STOP] [2020-12-03 11:39:51] Flattener#flatten
[STOP] [2020-12-03 11:39:51] rebuild_nodes
[START] [2020-12-03 11:39:51] resolve_missing_media_owners
[STOP] [2020-12-03 11:39:51] resolve_missing_media_owners
[START] [2020-12-03 11:39:51] sanitize_media_verbatims
[STOP] [2020-12-03 11:39:51] sanitize_media_verbatims
[START] [2020-12-03 11:39:51] queue_downloads
[STOP] [2020-12-03 11:39:51] queue_downloads
[START] [2020-12-03 11:39:51] parse_names
[WARN] [2020-12-03 11:39:51] I see 216 names which still need to be parsed.
[WARN] [2020-12-03 11:39:52] I see 2 names which still need to be parsed.
[STOP] [2020-12-03 11:39:54] parse_names
[START] [2020-12-03 11:39:54] denormalize_canonical_names_to_nodes
[STOP] [2020-12-03 11:39:54] denormalize_canonical_names_to_nodes
[START] [2020-12-03 11:39:54] match_nodes
[START] [2020-12-03 11:39:54] map_all_nodes_to_pages
[STOP] [2020-12-03 11:39:56] map_all_nodes_to_pages
[INFO] [2020-12-03 11:39:56] 13 Unmatched nodes (of 216)! That's too many to output. First 10: Aenigmachanna gollum (#86115927); Bibarba wenliuensis (#86115939); Caecogobius personatus (#86115945); Kayahschistura lokalayensis (#86115970); Paralepidocephalus translucens (#86116004); Sinocyclocheilus convexiforeheadus (#86116053); Trichomycterus donascimientoi (#86116085); Trichomycterus spectrum (#86116091); Triplophysa erythraea (#86116098); Triplophysa luochengensis (#86116106)
[START] [2020-12-03 11:39:56] update_nodes
[STOP] [2020-12-03 11:39:56] update_nodes
[STOP] [2020-12-03 11:39:56] match_nodes
[START] [2020-12-03 11:39:56] reindex_search
[STOP] [2020-12-03 11:39:57] reindex_search
[START] [2020-12-03 11:39:57] normalize_units
[STOP] [2020-12-03 11:39:57] normalize_units
[START] [2020-12-03 11:39:57] calculate_statistics
[STOP] [2020-12-03 11:39:57] calculate_statistics
[START] [2020-12-03 11:39:57] complete_harvest_instance
[START] [2020-12-03 11:39:57] overall_tsv_creation
[INFO] [2020-12-03 11:39:57] Processing group of 216 in 1 batches of 10000
[INFO] [2020-12-03 11:40:37] 633 Traits (unfiltered)...
[INFO] [2020-12-03 11:41:15] 633 Traits (filtered)...
[INFO] [2020-12-03 11:41:15] 0 Associations (filtered)...
[INFO] [2020-12-03 11:41:15] 633 metadata added.
[INFO] [2020-12-03 11:41:15] 0 metadata added.
[INFO] [2020-12-03 11:41:15] Average Time: 55.71
[INFO] [2020-12-03 11:41:15] Total Time: 1m19s
[STOP] [2020-12-03 11:41:15] overall_tsv_creation
[INFO] [2020-12-03 11:41:15] Done. Check your files:
[INFO] [2020-12-03 11:41:15] (214 lines) /app/public/data/proudlove_proudl/publish_nodes.tsv
[INFO] [2020-12-03 11:41:15] (90 lines) /app/public/data/proudlove_proudl/publish_node_ancestors.tsv
[INFO] [2020-12-03 11:41:15] (216 lines) /app/public/data/proudlove_proudl/publish_scientific_names.tsv
[INFO] [2020-12-03 11:41:15] (634 lines) /app/public/data/proudlove_proudl/publish_traits.tsv
[INFO] [2020-12-03 11:41:15] (634 lines) /app/public/data/proudlove_proudl/publish_metadata.tsv
[STOP] [2020-12-03 11:41:15] complete_harvest_instance
[START] [2020-12-03 11:41:15] completed
[STOP] [2020-12-03 11:41:15] completed
[STOP] [2020-12-03 11:41:15] logged process, took 94.78
[INFO] [2021-04-19 10:06:06] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-19 10:06:08] ## remove_type: ScientificName
[INFO] [2021-04-19 10:06:08] ++ Calling delete_all on 216 instances...
[INFO] [2021-04-19 10:06:08] [10:06:08.098] Removed 216 Scientificnames
[INFO] [2021-04-19 10:06:08] ## remove_type: Vernacular
[INFO] [2021-04-19 10:06:08] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:06:08] [10:06:08.099] Removed 0 Vernaculars
[INFO] [2021-04-19 10:06:08] ## remove_type: Article
[INFO] [2021-04-19 10:06:08] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:06:08] [10:06:08.101] Removed 0 Articles
[INFO] [2021-04-19 10:06:08] ## remove_type: Medium
[INFO] [2021-04-19 10:06:08] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:06:08] [10:06:08.102] Removed 0 Media
[INFO] [2021-04-19 10:06:08] ## remove_type: Trait
[INFO] [2021-04-19 10:06:08] ++ Calling delete_all on 692 instances...
[INFO] [2021-04-19 10:06:08] [10:06:08.146] Removed 692 Traits
[INFO] [2021-04-19 10:06:08] ## remove_type: MetaTrait
[INFO] [2021-04-19 10:06:08] ++ Calling delete_all on 692 instances...
[INFO] [2021-04-19 10:06:08] [10:06:08.172] Removed 692 Metatraits
[INFO] [2021-04-19 10:06:08] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-19 10:06:08] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:06:08] [10:06:08.173] Removed 0 Occurrencemetadata
[INFO] [2021-04-19 10:06:08] ## remove_type: Assoc
[INFO] [2021-04-19 10:06:08] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:06:08] [10:06:08.174] Removed 0 Assocs
[INFO] [2021-04-19 10:06:08] ## remove_type: MetaAssoc
[INFO] [2021-04-19 10:06:08] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:06:08] [10:06:08.176] Removed 0 Metaassocs
[INFO] [2021-04-19 10:06:08] ## remove_type: Identifier
[INFO] [2021-04-19 10:06:08] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:06:08] [10:06:08.185] Removed 0 Identifiers
[INFO] [2021-04-19 10:06:08] ## remove_type: Reference
[INFO] [2021-04-19 10:06:08] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 10:06:08] [10:06:08.187] Removed 0 References
[INFO] [2021-04-19 10:06:08] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:08] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:08] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:08] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:08] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:08] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:08] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:08] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:08] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:08] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:08] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:08] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:08] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:08] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:08] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:09] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:09] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:09] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:09] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:09] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:09] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:09] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:09] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:09] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:09] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:09] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:09] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:09] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:09] Starting batch with ID 86115925...
[INFO] [2021-04-19 10:06:09] ## remove_type: Node
[INFO] [2021-04-19 10:06:09] ++ Calling delete_all on 216 instances...
[INFO] [2021-04-19 10:06:09] [10:06:09.467] Removed 216 Nodes
[START] [2021-04-19 10:06:09] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 10:06:09] Creating resource from OpenData
[START] [2021-04-19 10:06:10] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 10:06:10] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 10:06:15] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 10:06:15] Creating resource from OpenData
[START] [2021-04-19 10:06:16] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 10:06:16] create_harvest_instance
[INFO] [2021-04-19 10:06:16] Created harvest instance #3723
[STOP] [2021-04-19 10:06:16] create_harvest_instance
[START] [2021-04-19 10:06:16] fetch_files
[STOP] [2021-04-19 10:06:16] fetch_files
[START] [2021-04-19 10:06:16] validate_each_file
[INFO] [2021-04-19 10:06:16] Looping over 4 formats...
[INFO] [2021-04-19 10:06:16] ...refs (/app/public/data/proudlove_proudl/references.tsv)
[INFO] [2021-04-19 10:06:16] Valid: /app/public/converted_csv/proudlove_proudl_refs_3723.csv (0 lines)
[INFO] [2021-04-19 10:06:16] ...nodes (/app/public/data/proudlove_proudl/taxa.txt)
[INFO] [2021-04-19 10:06:16] Valid: /app/public/converted_csv/proudlove_proudl_nodes_3723.csv (213 lines)
[INFO] [2021-04-19 10:06:16] ...occurrences (/app/public/data/proudlove_proudl/occurrences.txt)
[INFO] [2021-04-19 10:06:16] Valid: /app/public/converted_csv/proudlove_proudl_occurrences_3723.csv (262 lines)
[INFO] [2021-04-19 10:06:16] ...measurements (/app/public/data/proudlove_proudl/measurementsorfacts.txt)
[INFO] [2021-04-19 10:06:16] Valid: /app/public/converted_csv/proudlove_proudl_measurements_3723.csv (692 lines)
[STOP] [2021-04-19 10:06:16] validate_each_file
[START] [2021-04-19 10:06:16] convert_to_csv
[INFO] [2021-04-19 10:06:16] Looping over 4 formats...
[INFO] [2021-04-19 10:06:16] ...refs (/app/public/data/proudlove_proudl/references.tsv)
[CMD] [2021-04-19 10:06:16] /usr/bin/sort /app/public/converted_csv/proudlove_proudl_refs_3723.csv > /app/public/converted_csv/proudlove_proudl_refs_3723.csv_sorted
[INFO] [2021-04-19 10:06:16] Converted: /app/public/converted_csv/proudlove_proudl_refs_3723.csv (0 lines)
[INFO] [2021-04-19 10:06:16] ...nodes (/app/public/data/proudlove_proudl/taxa.txt)
[CMD] [2021-04-19 10:06:16] /usr/bin/sort /app/public/converted_csv/proudlove_proudl_nodes_3723.csv > /app/public/converted_csv/proudlove_proudl_nodes_3723.csv_sorted
[INFO] [2021-04-19 10:06:16] Converted: /app/public/converted_csv/proudlove_proudl_nodes_3723.csv (213 lines)
[INFO] [2021-04-19 10:06:16] ...occurrences (/app/public/data/proudlove_proudl/occurrences.txt)
[CMD] [2021-04-19 10:06:16] /usr/bin/sort /app/public/converted_csv/proudlove_proudl_occurrences_3723.csv > /app/public/converted_csv/proudlove_proudl_occurrences_3723.csv_sorted
[INFO] [2021-04-19 10:06:17] Converted: /app/public/converted_csv/proudlove_proudl_occurrences_3723.csv (262 lines)
[INFO] [2021-04-19 10:06:17] ...measurements (/app/public/data/proudlove_proudl/measurementsorfacts.txt)
[CMD] [2021-04-19 10:06:17] /usr/bin/sort /app/public/converted_csv/proudlove_proudl_measurements_3723.csv > /app/public/converted_csv/proudlove_proudl_measurements_3723.csv_sorted
[INFO] [2021-04-19 10:06:17] Converted: /app/public/converted_csv/proudlove_proudl_measurements_3723.csv (692 lines)
[STOP] [2021-04-19 10:06:17] convert_to_csv
[START] [2021-04-19 10:06:17] calculate_delta
[INFO] [2021-04-19 10:06:17] Looping over 4 formats...
[INFO] [2021-04-19 10:06:17] ...refs (/app/public/data/proudlove_proudl/references.tsv)
[CMD] [2021-04-19 10:06:17] echo "0a" > /app/public/diff/proudlove_proudl_refs_3723.diff
[CMD] [2021-04-19 10:06:18] tail -n +1 /app/public/converted_csv/proudlove_proudl_refs_3723.csv >> /app/public/diff/proudlove_proudl_refs_3723.diff
[CMD] [2021-04-19 10:06:18] echo "." >> /app/public/diff/proudlove_proudl_refs_3723.diff
[INFO] [2021-04-19 10:06:18] Created diff: /app/public/diff/proudlove_proudl_refs_3723.diff (2 lines)
[INFO] [2021-04-19 10:06:18] ...nodes (/app/public/data/proudlove_proudl/taxa.txt)
[CMD] [2021-04-19 10:06:18] echo "0a" > /app/public/diff/proudlove_proudl_nodes_3723.diff
[CMD] [2021-04-19 10:06:19] tail -n +1 /app/public/converted_csv/proudlove_proudl_nodes_3723.csv >> /app/public/diff/proudlove_proudl_nodes_3723.diff
[CMD] [2021-04-19 10:06:19] echo "." >> /app/public/diff/proudlove_proudl_nodes_3723.diff
[INFO] [2021-04-19 10:06:19] Created diff: /app/public/diff/proudlove_proudl_nodes_3723.diff (215 lines)
[INFO] [2021-04-19 10:06:19] ...occurrences (/app/public/data/proudlove_proudl/occurrences.txt)
[CMD] [2021-04-19 10:06:19] echo "0a" > /app/public/diff/proudlove_proudl_occurrences_3723.diff
[CMD] [2021-04-19 10:06:20] tail -n +1 /app/public/converted_csv/proudlove_proudl_occurrences_3723.csv >> /app/public/diff/proudlove_proudl_occurrences_3723.diff
[CMD] [2021-04-19 10:06:20] echo "." >> /app/public/diff/proudlove_proudl_occurrences_3723.diff
[INFO] [2021-04-19 10:06:21] Created diff: /app/public/diff/proudlove_proudl_occurrences_3723.diff (264 lines)
[INFO] [2021-04-19 10:06:21] ...measurements (/app/public/data/proudlove_proudl/measurementsorfacts.txt)
[CMD] [2021-04-19 10:06:21] echo "0a" > /app/public/diff/proudlove_proudl_measurements_3723.diff
[CMD] [2021-04-19 10:06:21] tail -n +1 /app/public/converted_csv/proudlove_proudl_measurements_3723.csv >> /app/public/diff/proudlove_proudl_measurements_3723.diff
[CMD] [2021-04-19 10:06:21] echo "." >> /app/public/diff/proudlove_proudl_measurements_3723.diff
[INFO] [2021-04-19 10:06:22] Created diff: /app/public/diff/proudlove_proudl_measurements_3723.diff (694 lines)
[STOP] [2021-04-19 10:06:22] calculate_delta
[START] [2021-04-19 10:06:22] parse_diff_and_store
[INFO] [2021-04-19 10:06:22] Handling diff: /app/public/diff/proudlove_proudl_refs_3723.diff (2 lines)
[INFO] [2021-04-19 10:06:22] Loading refs diff file into memory (2 /app/public/diff/proudlove_proudl_refs_3723.diff lines)...
[INFO] [2021-04-19 10:06:23] Handling diff: /app/public/diff/proudlove_proudl_nodes_3723.diff (215 lines)
[INFO] [2021-04-19 10:06:23] Loading nodes diff file into memory (215 /app/public/diff/proudlove_proudl_nodes_3723.diff lines)...
[INFO] [2021-04-19 10:06:23] Handling diff: /app/public/diff/proudlove_proudl_occurrences_3723.diff (264 lines)
[INFO] [2021-04-19 10:06:24] Loading occurrences diff file into memory (264 /app/public/diff/proudlove_proudl_occurrences_3723.diff lines)...
[INFO] [2021-04-19 10:06:24] Handling diff: /app/public/diff/proudlove_proudl_measurements_3723.diff (694 lines)
[INFO] [2021-04-19 10:06:25] Loading measurements diff file into memory (694 /app/public/diff/proudlove_proudl_measurements_3723.diff lines)...
[INFO] [2021-04-19 10:06:25] Storing 216 ScientificNames
[INFO] [2021-04-19 10:06:25] Processing group of 216 in 1 groups of 1000
[INFO] [2021-04-19 10:06:25] Average Time: 0.07
[INFO] [2021-04-19 10:06:25] Total Time: 1s
[INFO] [2021-04-19 10:06:25] Storing 216 Nodes
[INFO] [2021-04-19 10:06:25] Processing group of 216 in 1 groups of 1000
[INFO] [2021-04-19 10:06:25] Average Time: 0.08
[INFO] [2021-04-19 10:06:25] Total Time: 1s
[INFO] [2021-04-19 10:06:25] Storing 262 Occurrences
[INFO] [2021-04-19 10:06:25] Processing group of 262 in 1 groups of 1000
[INFO] [2021-04-19 10:06:25] Average Time: 0.04
[INFO] [2021-04-19 10:06:25] Total Time: 1s
[INFO] [2021-04-19 10:06:25] Storing 692 Traits
[INFO] [2021-04-19 10:06:25] Processing group of 692 in 1 groups of 1000
[INFO] [2021-04-19 10:06:26] Average Time: 0.19
[INFO] [2021-04-19 10:06:26] Total Time: 1s
[INFO] [2021-04-19 10:06:26] Storing 692 MetaTraits
[INFO] [2021-04-19 10:06:26] Processing group of 692 in 1 groups of 1000
[INFO] [2021-04-19 10:06:26] Average Time: 0.09
[INFO] [2021-04-19 10:06:26] Total Time: 1s
[STOP] [2021-04-19 10:06:26] parse_diff_and_store
[START] [2021-04-19 10:06:26] resolve_keys
[INFO] [2021-04-19 10:06:32] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-19 10:06:32] traits to occurrences...
[INFO] [2021-04-19 10:06:32] traits to nodes (through occurrences)...
[INFO] [2021-04-19 10:06:32] Traits to sex term...
[INFO] [2021-04-19 10:06:32] Traits to lifestage term...
[INFO] [2021-04-19 10:06:32] MetaTraits to traits...
[INFO] [2021-04-19 10:06:32] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-19 10:06:32] Assocs to occurrences...
[INFO] [2021-04-19 10:06:32] Assocs to nodes...
[INFO] [2021-04-19 10:06:32] Assoc to sex term...
[INFO] [2021-04-19 10:06:32] Assoc to lifestage term...
[INFO] [2021-04-19 10:06:32] MetaAssoc to assocs...
[STOP] [2021-04-19 10:06:32] resolve_keys
[START] [2021-04-19 10:06:32] hold_for_later_1
[STOP] [2021-04-19 10:06:32] hold_for_later_1
[START] [2021-04-19 10:06:32] hold_for_later_2
[STOP] [2021-04-19 10:06:32] hold_for_later_2
[START] [2021-04-19 10:06:32] resolve_missing_parents
[STOP] [2021-04-19 10:06:32] resolve_missing_parents
[START] [2021-04-19 10:06:32] rebuild_nodes
[START] [2021-04-19 10:06:32] Flattener#flatten
[START] [2021-04-19 10:06:32] Flattener#study_resource
[START] [2021-04-19 10:06:32] Flattener#build_ancestry
[STOP] [2021-04-19 10:06:32] Flattener#build_ancestry
[INFO] [2021-04-19 10:06:32] 216 ancestry keys
[START] [2021-04-19 10:06:32] build_node_ancestors
[INFO] [2021-04-19 10:06:32] old ancestors deleted.
[STOP] [2021-04-19 10:06:32] build_node_ancestors
[START] [2021-04-19 10:06:32] Flattener#propagate_ancestor_ids
[STOP] [2021-04-19 10:06:32] Flattener#propagate_ancestor_ids
[STOP] [2021-04-19 10:06:32] Flattener#flatten
[STOP] [2021-04-19 10:06:32] rebuild_nodes
[START] [2021-04-19 10:06:32] resolve_missing_media_owners
[STOP] [2021-04-19 10:06:32] resolve_missing_media_owners
[START] [2021-04-19 10:06:32] sanitize_media_verbatims
[STOP] [2021-04-19 10:06:32] sanitize_media_verbatims
[START] [2021-04-19 10:06:32] queue_downloads
[STOP] [2021-04-19 10:06:32] queue_downloads
[START] [2021-04-19 10:06:32] parse_names
[WARN] [2021-04-19 10:06:32] I see 216 names which still need to be parsed.
[WARN] [2021-04-19 10:06:33] I see 2 names which still need to be parsed.
[STOP] [2021-04-19 10:06:34] parse_names
[START] [2021-04-19 10:06:34] denormalize_canonical_names_to_nodes
[STOP] [2021-04-19 10:06:34] denormalize_canonical_names_to_nodes
[START] [2021-04-19 10:06:34] match_nodes
[START] [2021-04-19 10:06:34] map_all_nodes_to_pages
[STOP] [2021-04-19 10:06:40] map_all_nodes_to_pages
[INFO] [2021-04-19 10:06:40] 13 Unmatched nodes (of 216)! That's too many to output. Full list in /app/public/data/proudlove_proudl/unmatched_nodes.txt ; First 10: Aenigmachanna gollum (#92872451); Bibarba wenliuensis (#92872463); Caecogobius personatus (#92872469); Kayahschistura lokalayensis (#92872494); Paralepidocephalus translucens (#92872528); Sinocyclocheilus convexiforeheadus (#92872577); Trichomycterus donascimientoi (#92872609); Trichomycterus spectrum (#92872615); Triplophysa erythraea (#92872622); Triplophysa tianlinensis (#92872638)
[START] [2021-04-19 10:06:40] update_nodes
[STOP] [2021-04-19 10:06:40] update_nodes
[STOP] [2021-04-19 10:06:40] match_nodes
[START] [2021-04-19 10:06:40] reindex_search
[STOP] [2021-04-19 10:06:40] reindex_search
[START] [2021-04-19 10:06:40] normalize_units
[STOP] [2021-04-19 10:06:40] normalize_units
[START] [2021-04-19 10:06:40] calculate_statistics
[STOP] [2021-04-19 10:06:40] calculate_statistics
[START] [2021-04-19 10:06:40] complete_harvest_instance
[START] [2021-04-19 10:06:40] overall_tsv_creation
[INFO] [2021-04-19 10:06:40] Processing group of 216 in 1 batches of 10000
[INFO] [2021-04-19 10:07:18] 633 Traits (unfiltered)...
[INFO] [2021-04-19 10:07:52] 633 Traits (filtered)...
[INFO] [2021-04-19 10:07:52] 0 Associations (filtered)...
[INFO] [2021-04-19 10:07:52] 0 metadata added.
[INFO] [2021-04-19 10:07:52] 0 metadata added.
[INFO] [2021-04-19 10:08:19] Average Time: 73.85
[INFO] [2021-04-19 10:08:19] Total Time: 1m39s
[STOP] [2021-04-19 10:08:19] overall_tsv_creation
[INFO] [2021-04-19 10:08:19] Done. Check your files:
[INFO] [2021-04-19 10:08:19] (214 lines) /app/public/data/proudlove_proudl/publish_nodes.tsv
[INFO] [2021-04-19 10:08:19] (90 lines) /app/public/data/proudlove_proudl/publish_node_ancestors.tsv
[INFO] [2021-04-19 10:08:20] (216 lines) /app/public/data/proudlove_proudl/publish_scientific_names.tsv
[INFO] [2021-04-19 10:08:20] (634 lines) /app/public/data/proudlove_proudl/publish_traits.tsv
[INFO] [2021-04-19 10:08:21] (1 lines) /app/public/data/proudlove_proudl/publish_metadata.tsv
[STOP] [2021-04-19 10:08:21] complete_harvest_instance
[START] [2021-04-19 10:08:21] completed
[STOP] [2021-04-19 10:08:21] completed
[STOP] [2021-04-19 10:08:21] logged process, took 125.44
[INFO] [2021-04-19 15:36:11] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-19 15:42:17] ## remove_type: ScientificName
[INFO] [2021-04-19 15:42:17] ++ Calling delete_all on 216 instances...
[INFO] [2021-04-19 15:42:17] [15:42:17.875] Removed 216 Scientificnames
[INFO] [2021-04-19 15:42:17] ## remove_type: Vernacular
[INFO] [2021-04-19 15:42:17] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:42:17] [15:42:17.877] Removed 0 Vernaculars
[INFO] [2021-04-19 15:42:17] ## remove_type: Article
[INFO] [2021-04-19 15:42:17] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:42:17] [15:42:17.878] Removed 0 Articles
[INFO] [2021-04-19 15:42:17] ## remove_type: Medium
[INFO] [2021-04-19 15:42:17] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:42:17] [15:42:17.880] Removed 0 Media
[INFO] [2021-04-19 15:42:17] ## remove_type: Trait
[INFO] [2021-04-19 15:42:17] ++ Calling delete_all on 692 instances...
[INFO] [2021-04-19 15:42:17] [15:42:17.978] Removed 692 Traits
[INFO] [2021-04-19 15:42:17] ## remove_type: MetaTrait
[INFO] [2021-04-19 15:42:17] ++ Calling delete_all on 692 instances...
[INFO] [2021-04-19 15:42:18] [15:42:18.011] Removed 692 Metatraits
[INFO] [2021-04-19 15:42:18] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-19 15:42:18] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:42:18] [15:42:18.012] Removed 0 Occurrencemetadata
[INFO] [2021-04-19 15:42:18] ## remove_type: Assoc
[INFO] [2021-04-19 15:42:18] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:42:18] [15:42:18.014] Removed 0 Assocs
[INFO] [2021-04-19 15:42:18] ## remove_type: MetaAssoc
[INFO] [2021-04-19 15:42:18] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:42:18] [15:42:18.015] Removed 0 Metaassocs
[INFO] [2021-04-19 15:42:18] ## remove_type: Identifier
[INFO] [2021-04-19 15:42:18] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:42:18] [15:42:18.017] Removed 0 Identifiers
[INFO] [2021-04-19 15:42:18] ## remove_type: Reference
[INFO] [2021-04-19 15:42:18] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-19 15:42:18] [15:42:18.018] Removed 0 References
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] Starting batch with ID 92872452...
[INFO] [2021-04-19 15:42:18] ## remove_type: Node
[INFO] [2021-04-19 15:42:18] ++ Calling delete_all on 216 instances...
[INFO] [2021-04-19 15:42:18] [15:42:18.600] Removed 216 Nodes
[START] [2021-04-19 15:42:18] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 15:42:18] Creating resource from OpenData
[START] [2021-04-19 15:42:18] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 15:42:18] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 15:42:18] Parse meta.xml file and create formats with fields
[STOP] [2021-04-19 15:42:18] Creating resource from OpenData
[START] [2021-04-19 15:42:19] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-19 15:42:19] create_harvest_instance
[INFO] [2021-04-19 15:42:19] Created harvest instance #3775
[STOP] [2021-04-19 15:42:19] create_harvest_instance
[START] [2021-04-19 15:42:19] fetch_files
[STOP] [2021-04-19 15:42:19] fetch_files
[START] [2021-04-19 15:42:19] validate_each_file
[INFO] [2021-04-19 15:42:19] Looping over 4 formats...
[INFO] [2021-04-19 15:42:19] ...refs (/app/public/data/proudlove_proudl/references.tsv)
[INFO] [2021-04-19 15:42:19] Valid: /app/public/converted_csv/proudlove_proudl_refs_3775.csv (0 lines)
[INFO] [2021-04-19 15:42:19] ...nodes (/app/public/data/proudlove_proudl/taxa.txt)
[INFO] [2021-04-19 15:42:19] Valid: /app/public/converted_csv/proudlove_proudl_nodes_3775.csv (213 lines)
[INFO] [2021-04-19 15:42:19] ...occurrences (/app/public/data/proudlove_proudl/occurrences.txt)
[INFO] [2021-04-19 15:42:19] Valid: /app/public/converted_csv/proudlove_proudl_occurrences_3775.csv (262 lines)
[INFO] [2021-04-19 15:42:19] ...measurements (/app/public/data/proudlove_proudl/measurementsorfacts.txt)
[INFO] [2021-04-19 15:42:19] Valid: /app/public/converted_csv/proudlove_proudl_measurements_3775.csv (692 lines)
[STOP] [2021-04-19 15:42:19] validate_each_file
[START] [2021-04-19 15:42:19] convert_to_csv
[INFO] [2021-04-19 15:42:19] Looping over 4 formats...
[INFO] [2021-04-19 15:42:19] ...refs (/app/public/data/proudlove_proudl/references.tsv)
[CMD] [2021-04-19 15:42:19] /usr/bin/sort /app/public/converted_csv/proudlove_proudl_refs_3775.csv > /app/public/converted_csv/proudlove_proudl_refs_3775.csv_sorted
[INFO] [2021-04-19 15:42:19] Converted: /app/public/converted_csv/proudlove_proudl_refs_3775.csv (0 lines)
[INFO] [2021-04-19 15:42:19] ...nodes (/app/public/data/proudlove_proudl/taxa.txt)
[CMD] [2021-04-19 15:42:19] /usr/bin/sort /app/public/converted_csv/proudlove_proudl_nodes_3775.csv > /app/public/converted_csv/proudlove_proudl_nodes_3775.csv_sorted
[INFO] [2021-04-19 15:42:19] Converted: /app/public/converted_csv/proudlove_proudl_nodes_3775.csv (213 lines)
[INFO] [2021-04-19 15:42:19] ...occurrences (/app/public/data/proudlove_proudl/occurrences.txt)
[CMD] [2021-04-19 15:42:19] /usr/bin/sort /app/public/converted_csv/proudlove_proudl_occurrences_3775.csv > /app/public/converted_csv/proudlove_proudl_occurrences_3775.csv_sorted
[INFO] [2021-04-19 15:42:19] Converted: /app/public/converted_csv/proudlove_proudl_occurrences_3775.csv (262 lines)
[INFO] [2021-04-19 15:42:19] ...measurements (/app/public/data/proudlove_proudl/measurementsorfacts.txt)
[CMD] [2021-04-19 15:42:19] /usr/bin/sort /app/public/converted_csv/proudlove_proudl_measurements_3775.csv > /app/public/converted_csv/proudlove_proudl_measurements_3775.csv_sorted
[INFO] [2021-04-19 15:42:19] Converted: /app/public/converted_csv/proudlove_proudl_measurements_3775.csv (692 lines)
[STOP] [2021-04-19 15:42:19] convert_to_csv
[START] [2021-04-19 15:42:19] calculate_delta
[INFO] [2021-04-19 15:42:19] Looping over 4 formats...
[INFO] [2021-04-19 15:42:19] ...refs (/app/public/data/proudlove_proudl/references.tsv)
[CMD] [2021-04-19 15:42:19] echo "0a" > /app/public/diff/proudlove_proudl_refs_3775.diff
[CMD] [2021-04-19 15:42:19] tail -n +1 /app/public/converted_csv/proudlove_proudl_refs_3775.csv >> /app/public/diff/proudlove_proudl_refs_3775.diff
[CMD] [2021-04-19 15:42:19] echo "." >> /app/public/diff/proudlove_proudl_refs_3775.diff
[INFO] [2021-04-19 15:42:19] Created diff: /app/public/diff/proudlove_proudl_refs_3775.diff (2 lines)
[INFO] [2021-04-19 15:42:19] ...nodes (/app/public/data/proudlove_proudl/taxa.txt)
[CMD] [2021-04-19 15:42:19] echo "0a" > /app/public/diff/proudlove_proudl_nodes_3775.diff
[CMD] [2021-04-19 15:42:19] tail -n +1 /app/public/converted_csv/proudlove_proudl_nodes_3775.csv >> /app/public/diff/proudlove_proudl_nodes_3775.diff
[CMD] [2021-04-19 15:42:19] echo "." >> /app/public/diff/proudlove_proudl_nodes_3775.diff
[INFO] [2021-04-19 15:42:19] Created diff: /app/public/diff/proudlove_proudl_nodes_3775.diff (215 lines)
[INFO] [2021-04-19 15:42:19] ...occurrences (/app/public/data/proudlove_proudl/occurrences.txt)
[CMD] [2021-04-19 15:42:19] echo "0a" > /app/public/diff/proudlove_proudl_occurrences_3775.diff
[CMD] [2021-04-19 15:42:19] tail -n +1 /app/public/converted_csv/proudlove_proudl_occurrences_3775.csv >> /app/public/diff/proudlove_proudl_occurrences_3775.diff
[CMD] [2021-04-19 15:42:19] echo "." >> /app/public/diff/proudlove_proudl_occurrences_3775.diff
[INFO] [2021-04-19 15:42:19] Created diff: /app/public/diff/proudlove_proudl_occurrences_3775.diff (264 lines)
[INFO] [2021-04-19 15:42:19] ...measurements (/app/public/data/proudlove_proudl/measurementsorfacts.txt)
[CMD] [2021-04-19 15:42:19] echo "0a" > /app/public/diff/proudlove_proudl_measurements_3775.diff
[CMD] [2021-04-19 15:42:19] tail -n +1 /app/public/converted_csv/proudlove_proudl_measurements_3775.csv >> /app/public/diff/proudlove_proudl_measurements_3775.diff
[CMD] [2021-04-19 15:42:19] echo "." >> /app/public/diff/proudlove_proudl_measurements_3775.diff
[INFO] [2021-04-19 15:42:19] Created diff: /app/public/diff/proudlove_proudl_measurements_3775.diff (694 lines)
[STOP] [2021-04-19 15:42:19] calculate_delta
[START] [2021-04-19 15:42:19] parse_diff_and_store
[INFO] [2021-04-19 15:42:19] Handling diff: /app/public/diff/proudlove_proudl_refs_3775.diff (2 lines)
[INFO] [2021-04-19 15:42:19] Loading refs diff file into memory (2 /app/public/diff/proudlove_proudl_refs_3775.diff lines)...
[INFO] [2021-04-19 15:42:19] Handling diff: /app/public/diff/proudlove_proudl_nodes_3775.diff (215 lines)
[INFO] [2021-04-19 15:42:19] Loading nodes diff file into memory (215 /app/public/diff/proudlove_proudl_nodes_3775.diff lines)...
[INFO] [2021-04-19 15:42:19] Handling diff: /app/public/diff/proudlove_proudl_occurrences_3775.diff (264 lines)
[INFO] [2021-04-19 15:42:19] Loading occurrences diff file into memory (264 /app/public/diff/proudlove_proudl_occurrences_3775.diff lines)...
[INFO] [2021-04-19 15:42:19] Handling diff: /app/public/diff/proudlove_proudl_measurements_3775.diff (694 lines)
[INFO] [2021-04-19 15:42:19] Loading measurements diff file into memory (694 /app/public/diff/proudlove_proudl_measurements_3775.diff lines)...
[INFO] [2021-04-19 15:42:19] Storing 216 ScientificNames
[INFO] [2021-04-19 15:42:19] Processing group of 216 in 1 groups of 1000
[INFO] [2021-04-19 15:42:19] Average Time: 0.06
[INFO] [2021-04-19 15:42:19] Total Time: 1s
[INFO] [2021-04-19 15:42:19] Storing 216 Nodes
[INFO] [2021-04-19 15:42:19] Processing group of 216 in 1 groups of 1000
[INFO] [2021-04-19 15:42:19] Average Time: 0.06
[INFO] [2021-04-19 15:42:19] Total Time: 1s
[INFO] [2021-04-19 15:42:19] Storing 262 Occurrences
[INFO] [2021-04-19 15:42:19] Processing group of 262 in 1 groups of 1000
[INFO] [2021-04-19 15:42:19] Average Time: 0.03
[INFO] [2021-04-19 15:42:19] Total Time: 1s
[INFO] [2021-04-19 15:42:19] Storing 692 Traits
[INFO] [2021-04-19 15:42:19] Processing group of 692 in 1 groups of 1000
[INFO] [2021-04-19 15:42:20] Average Time: 0.19
[INFO] [2021-04-19 15:42:20] Total Time: 1s
[INFO] [2021-04-19 15:42:20] Storing 692 MetaTraits
[INFO] [2021-04-19 15:42:20] Processing group of 692 in 1 groups of 1000
[INFO] [2021-04-19 15:42:20] Average Time: 0.08
[INFO] [2021-04-19 15:42:20] Total Time: 1s
[STOP] [2021-04-19 15:42:20] parse_diff_and_store
[START] [2021-04-19 15:42:20] resolve_keys
[INFO] [2021-04-19 15:42:26] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-19 15:42:26] traits to occurrences...
[INFO] [2021-04-19 15:42:26] traits to nodes (through occurrences)...
[INFO] [2021-04-19 15:42:26] Traits to sex term...
[INFO] [2021-04-19 15:42:26] Traits to lifestage term...
[INFO] [2021-04-19 15:42:26] MetaTraits to traits...
[INFO] [2021-04-19 15:42:26] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-19 15:42:26] Assocs to occurrences...
[INFO] [2021-04-19 15:42:26] Assocs to nodes...
[INFO] [2021-04-19 15:42:26] Assoc to sex term...
[INFO] [2021-04-19 15:42:26] Assoc to lifestage term...
[INFO] [2021-04-19 15:42:26] MetaAssoc to assocs...
[STOP] [2021-04-19 15:42:26] resolve_keys
[START] [2021-04-19 15:42:26] hold_for_later_1
[STOP] [2021-04-19 15:42:26] hold_for_later_1
[START] [2021-04-19 15:42:26] hold_for_later_2
[STOP] [2021-04-19 15:42:26] hold_for_later_2
[START] [2021-04-19 15:42:26] resolve_missing_parents
[STOP] [2021-04-19 15:42:26] resolve_missing_parents
[START] [2021-04-19 15:42:26] rebuild_nodes
[START] [2021-04-19 15:42:26] Flattener#flatten
[START] [2021-04-19 15:42:26] Flattener#study_resource
[START] [2021-04-19 15:42:26] Flattener#build_ancestry
[STOP] [2021-04-19 15:42:26] Flattener#build_ancestry
[INFO] [2021-04-19 15:42:26] 216 ancestry keys
[START] [2021-04-19 15:42:26] build_node_ancestors
[INFO] [2021-04-19 15:42:26] old ancestors deleted.
[STOP] [2021-04-19 15:42:26] build_node_ancestors
[START] [2021-04-19 15:42:26] Flattener#propagate_ancestor_ids
[STOP] [2021-04-19 15:42:26] Flattener#propagate_ancestor_ids
[STOP] [2021-04-19 15:42:26] Flattener#flatten
[STOP] [2021-04-19 15:42:26] rebuild_nodes
[START] [2021-04-19 15:42:26] resolve_missing_media_owners
[STOP] [2021-04-19 15:42:26] resolve_missing_media_owners
[START] [2021-04-19 15:42:26] sanitize_media_verbatims
[STOP] [2021-04-19 15:42:26] sanitize_media_verbatims
[START] [2021-04-19 15:42:26] queue_downloads
[STOP] [2021-04-19 15:42:26] queue_downloads
[START] [2021-04-19 15:42:26] parse_names
[WARN] [2021-04-19 15:42:26] I see 216 names which still need to be parsed.
[WARN] [2021-04-19 15:42:27] I see 2 names which still need to be parsed.
[STOP] [2021-04-19 15:42:28] parse_names
[START] [2021-04-19 15:42:28] denormalize_canonical_names_to_nodes
[STOP] [2021-04-19 15:42:28] denormalize_canonical_names_to_nodes
[START] [2021-04-19 15:42:28] match_nodes
[START] [2021-04-19 15:42:28] map_all_nodes_to_pages
[STOP] [2021-04-19 15:42:30] map_all_nodes_to_pages
[INFO] [2021-04-19 15:42:30] 13 Unmatched nodes (of 216)! That's too many to output. Full list in /app/public/data/proudlove_proudl/unmatched_nodes.txt ; First 10: Aenigmachanna gollum (#92874124); Bibarba wenliuensis (#92874136); Caecogobius personatus (#92874142); Kayahschistura lokalayensis (#92874167); Paralepidocephalus translucens (#92874201); Sinocyclocheilus convexiforeheadus (#92874250); Trichomycterus donascimientoi (#92874282); Trichomycterus spectrum (#92874288); Triplophysa erythraea (#92874295); Triplophysa tianlinensis (#92874311)
[START] [2021-04-19 15:42:30] update_nodes
[STOP] [2021-04-19 15:42:30] update_nodes
[STOP] [2021-04-19 15:42:30] match_nodes
[START] [2021-04-19 15:42:30] reindex_search
[STOP] [2021-04-19 15:42:30] reindex_search
[START] [2021-04-19 15:42:30] normalize_units
[STOP] [2021-04-19 15:42:30] normalize_units
[START] [2021-04-19 15:42:30] calculate_statistics
[STOP] [2021-04-19 15:42:30] calculate_statistics
[START] [2021-04-19 15:42:30] complete_harvest_instance
[START] [2021-04-19 15:42:30] overall_tsv_creation
[INFO] [2021-04-19 15:42:30] Processing group of 216 in 1 batches of 10000
[INFO] [2021-04-19 15:43:07] 633 Traits (unfiltered)...
[INFO] [2021-04-19 15:43:41] 633 Traits (filtered)...
[INFO] [2021-04-19 15:43:41] 0 Associations (filtered)...
[INFO] [2021-04-19 15:43:42] 0 metadata added.
[INFO] [2021-04-19 15:43:42] 0 metadata added.
[INFO] [2021-04-19 15:44:08] Average Time: 74.3
[INFO] [2021-04-19 15:44:08] Total Time: 1m39s
[STOP] [2021-04-19 15:44:08] overall_tsv_creation
[INFO] [2021-04-19 15:44:08] Done. Check your files:
[INFO] [2021-04-19 15:44:08] (214 lines) /app/public/data/proudlove_proudl/publish_nodes.tsv
[INFO] [2021-04-19 15:44:08] (90 lines) /app/public/data/proudlove_proudl/publish_node_ancestors.tsv
[INFO] [2021-04-19 15:44:08] (216 lines) /app/public/data/proudlove_proudl/publish_scientific_names.tsv
[INFO] [2021-04-19 15:44:08] (634 lines) /app/public/data/proudlove_proudl/publish_traits.tsv
[INFO] [2021-04-19 15:44:08] (1 lines) /app/public/data/proudlove_proudl/publish_metadata.tsv
[STOP] [2021-04-19 15:44:08] complete_harvest_instance
[START] [2021-04-19 15:44:08] completed
[STOP] [2021-04-19 15:44:08] completed
[STOP] [2021-04-19 15:44:08] logged process, took 109.78

Latest Process

19 Apr 15:42 EXITED currently processing 0