Harvest for Lovegrove 2014 Created 07 Apr 14:30

Harvesting Log

# Logfile created on 2020-05-05 14:10:25 -0400 by logger.rb/v1.4.2
[INFO] [2020-05-05 14:10:25] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-05-05 14:10:29] ## remove_type: ScientificName
[INFO] [2020-05-05 14:10:29] ++ Calling delete_all on 85 instances...
[INFO] [2020-05-05 14:10:29] [14:10:29.298] Removed 85 Scientificnames
[INFO] [2020-05-05 14:10:29] ## remove_type: Vernacular
[INFO] [2020-05-05 14:10:29] ++ Calling delete_all on 0 instances...
[INFO] [2020-05-05 14:10:29] [14:10:29.300] Removed 0 Vernaculars
[INFO] [2020-05-05 14:10:29] ## remove_type: Article
[INFO] [2020-05-05 14:10:29] ++ Calling delete_all on 0 instances...
[INFO] [2020-05-05 14:10:29] [14:10:29.301] Removed 0 Articles
[INFO] [2020-05-05 14:10:29] ## remove_type: Medium
[INFO] [2020-05-05 14:10:29] ++ Calling delete_all on 0 instances...
[INFO] [2020-05-05 14:10:29] [14:10:29.303] Removed 0 Media
[INFO] [2020-05-05 14:10:29] ## remove_type: Trait
[INFO] [2020-05-05 14:10:29] ++ Calling delete_all on 644 instances...
[INFO] [2020-05-05 14:10:29] [14:10:29.368] Removed 644 Traits
[INFO] [2020-05-05 14:10:29] ## remove_type: MetaTrait
[INFO] [2020-05-05 14:10:29] ++ Calling delete_all on 586 instances...
[INFO] [2020-05-05 14:10:29] [14:10:29.406] Removed 586 Metatraits
[INFO] [2020-05-05 14:10:29] ## remove_type: OccurrenceMetadatum
[INFO] [2020-05-05 14:10:29] ++ Calling delete_all on 332 instances...
[INFO] [2020-05-05 14:10:29] [14:10:29.439] Removed 332 Occurrencemetadata
[INFO] [2020-05-05 14:10:29] ## remove_type: Assoc
[INFO] [2020-05-05 14:10:29] ++ Calling delete_all on 0 instances...
[INFO] [2020-05-05 14:10:29] [14:10:29.441] Removed 0 Assocs
[INFO] [2020-05-05 14:10:29] ## remove_type: MetaAssoc
[INFO] [2020-05-05 14:10:29] ++ Calling delete_all on 0 instances...
[INFO] [2020-05-05 14:10:29] [14:10:29.442] Removed 0 Metaassocs
[INFO] [2020-05-05 14:10:29] ## remove_type: Identifier
[INFO] [2020-05-05 14:10:29] ++ Calling delete_all on 0 instances...
[INFO] [2020-05-05 14:10:29] [14:10:29.444] Removed 0 Identifiers
[INFO] [2020-05-05 14:10:29] ## remove_type: Reference
[INFO] [2020-05-05 14:10:29] ++ Calling delete_all on 6 instances...
[INFO] [2020-05-05 14:10:29] [14:10:29.451] Removed 6 References
[INFO] [2020-05-05 14:10:29] Starting batch with ID 73218390...
[INFO] [2020-05-05 14:10:29] Starting batch with ID 73218390...
[INFO] [2020-05-05 14:10:29] Starting batch with ID 73218390...
[INFO] [2020-05-05 14:10:29] Starting batch with ID 73218390...
[INFO] [2020-05-05 14:10:29] Starting batch with ID 73218390...
[INFO] [2020-05-05 14:10:29] Starting batch with ID 73218390...
[INFO] [2020-05-05 14:10:29] Starting batch with ID 73218390...
[INFO] [2020-05-05 14:10:29] Starting batch with ID 73218390...
[INFO] [2020-05-05 14:10:29] Starting batch with ID 73218390...
[INFO] [2020-05-05 14:10:29] Starting batch with ID 73218390...
[INFO] [2020-05-05 14:10:29] Starting batch with ID 73218390...
[INFO] [2020-05-05 14:10:30] Starting batch with ID 73218390...
[INFO] [2020-05-05 14:10:30] Starting batch with ID 73218390...
[INFO] [2020-05-05 14:10:30] Starting batch with ID 73218390...
[INFO] [2020-05-05 14:10:30] Starting batch with ID 73218390...
[INFO] [2020-05-05 14:10:30] Starting batch with ID 73218390...
[INFO] [2020-05-05 14:10:30] Starting batch with ID 73218390...
[INFO] [2020-05-05 14:10:30] Starting batch with ID 73218390...
[INFO] [2020-05-05 14:10:30] Starting batch with ID 73218390...
[INFO] [2020-05-05 14:10:30] Starting batch with ID 73218400...
[INFO] [2020-05-05 14:10:30] Starting batch with ID 73218389...
[INFO] [2020-05-05 14:10:30] Starting batch with ID 73218389...
[INFO] [2020-05-05 14:10:30] ## remove_type: Node
[INFO] [2020-05-05 14:10:30] ++ Calling delete_all on 85 instances...
[INFO] [2020-05-05 14:10:30] [14:10:30.366] Removed 85 Nodes
[INFO] [2020-05-05 14:10:39] ## HARVEST: type = -harvest
[START] [2020-05-05 14:10:40] logged process
[START] [2020-05-05 14:10:40] create_harvest_instance
[STOP] [2020-05-05 14:10:49] create_harvest_instance
[START] [2020-05-05 14:10:49] fetch_files
[STOP] [2020-05-05 14:10:49] fetch_files
[START] [2020-05-05 14:10:49] validate_each_file
[STOP] [2020-05-05 14:10:50] validate_each_file
[START] [2020-05-05 14:10:50] convert_to_csv
[CMD] [2020-05-05 14:10:50] /usr/bin/sort /app/public/converted_csv/lovegrove_agents_20903.csv > /app/public/converted_csv/lovegrove_agents_20903.csv_sorted
[CMD] [2020-05-05 14:10:50] /usr/bin/sort /app/public/converted_csv/lovegrove_refs_20904.csv > /app/public/converted_csv/lovegrove_refs_20904.csv_sorted
[CMD] [2020-05-05 14:10:50] /usr/bin/sort /app/public/converted_csv/lovegrove_nodes_20905.csv > /app/public/converted_csv/lovegrove_nodes_20905.csv_sorted
[CMD] [2020-05-05 14:10:50] /usr/bin/sort /app/public/converted_csv/lovegrove_media_20906.csv > /app/public/converted_csv/lovegrove_media_20906.csv_sorted
[CMD] [2020-05-05 14:10:50] /usr/bin/sort /app/public/converted_csv/lovegrove_vernaculars_20907.csv > /app/public/converted_csv/lovegrove_vernaculars_20907.csv_sorted
[CMD] [2020-05-05 14:10:50] /usr/bin/sort /app/public/converted_csv/lovegrove_occurrences_20908.csv > /app/public/converted_csv/lovegrove_occurrences_20908.csv_sorted
[CMD] [2020-05-05 14:10:50] /usr/bin/sort /app/public/converted_csv/lovegrove_assocs_20909.csv > /app/public/converted_csv/lovegrove_assocs_20909.csv_sorted
[CMD] [2020-05-05 14:10:50] /usr/bin/sort /app/public/converted_csv/lovegrove_measurements_20910.csv > /app/public/converted_csv/lovegrove_measurements_20910.csv_sorted
[STOP] [2020-05-05 14:10:50] convert_to_csv
[START] [2020-05-05 14:10:50] calculate_delta
[CMD] [2020-05-05 14:10:50] echo "0a" > /app/public/diff/lovegrove_agents_20903.diff
[CMD] [2020-05-05 14:10:51] tail -n +1 /app/public/converted_csv/lovegrove_agents_20903.csv >> /app/public/diff/lovegrove_agents_20903.diff
[CMD] [2020-05-05 14:10:51] echo "." >> /app/public/diff/lovegrove_agents_20903.diff
[CMD] [2020-05-05 14:10:51] echo "0a" > /app/public/diff/lovegrove_refs_20904.diff
[CMD] [2020-05-05 14:10:51] tail -n +1 /app/public/converted_csv/lovegrove_refs_20904.csv >> /app/public/diff/lovegrove_refs_20904.diff
[CMD] [2020-05-05 14:10:51] echo "." >> /app/public/diff/lovegrove_refs_20904.diff
[CMD] [2020-05-05 14:10:51] echo "0a" > /app/public/diff/lovegrove_nodes_20905.diff
[CMD] [2020-05-05 14:10:51] tail -n +1 /app/public/converted_csv/lovegrove_nodes_20905.csv >> /app/public/diff/lovegrove_nodes_20905.diff
[CMD] [2020-05-05 14:10:51] echo "." >> /app/public/diff/lovegrove_nodes_20905.diff
[CMD] [2020-05-05 14:10:51] echo "0a" > /app/public/diff/lovegrove_media_20906.diff
[CMD] [2020-05-05 14:10:51] tail -n +1 /app/public/converted_csv/lovegrove_media_20906.csv >> /app/public/diff/lovegrove_media_20906.diff
[CMD] [2020-05-05 14:10:51] echo "." >> /app/public/diff/lovegrove_media_20906.diff
[CMD] [2020-05-05 14:10:51] echo "0a" > /app/public/diff/lovegrove_vernaculars_20907.diff
[CMD] [2020-05-05 14:10:51] tail -n +1 /app/public/converted_csv/lovegrove_vernaculars_20907.csv >> /app/public/diff/lovegrove_vernaculars_20907.diff
[CMD] [2020-05-05 14:10:51] echo "." >> /app/public/diff/lovegrove_vernaculars_20907.diff
[CMD] [2020-05-05 14:10:51] echo "0a" > /app/public/diff/lovegrove_occurrences_20908.diff
[CMD] [2020-05-05 14:10:51] tail -n +1 /app/public/converted_csv/lovegrove_occurrences_20908.csv >> /app/public/diff/lovegrove_occurrences_20908.diff
[CMD] [2020-05-05 14:10:52] echo "." >> /app/public/diff/lovegrove_occurrences_20908.diff
[CMD] [2020-05-05 14:10:52] echo "0a" > /app/public/diff/lovegrove_assocs_20909.diff
[CMD] [2020-05-05 14:10:52] tail -n +1 /app/public/converted_csv/lovegrove_assocs_20909.csv >> /app/public/diff/lovegrove_assocs_20909.diff
[CMD] [2020-05-05 14:10:52] echo "." >> /app/public/diff/lovegrove_assocs_20909.diff
[CMD] [2020-05-05 14:10:52] echo "0a" > /app/public/diff/lovegrove_measurements_20910.diff
[CMD] [2020-05-05 14:10:52] tail -n +1 /app/public/converted_csv/lovegrove_measurements_20910.csv >> /app/public/diff/lovegrove_measurements_20910.diff
[CMD] [2020-05-05 14:10:52] echo "." >> /app/public/diff/lovegrove_measurements_20910.diff
[STOP] [2020-05-05 14:10:52] calculate_delta
[START] [2020-05-05 14:10:52] parse_diff_and_store
[INFO] [2020-05-05 14:10:52] Loading agents diff file into memory (true lines)...
[INFO] [2020-05-05 14:10:52] Loading refs diff file into memory (true lines)...
[INFO] [2020-05-05 14:10:52] Loading nodes diff file into memory (true lines)...
[INFO] [2020-05-05 14:10:52] Loading media diff file into memory (true lines)...
[INFO] [2020-05-05 14:10:52] Loading vernaculars diff file into memory (true lines)...
[INFO] [2020-05-05 14:10:53] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-05-05 14:10:53] Loading assocs diff file into memory (true lines)...
[INFO] [2020-05-05 14:10:53] Loading measurements diff file into memory (true lines)...
[INFO] [2020-05-05 14:10:58] Storing 6 References
[INFO] [2020-05-05 14:10:58] Processing group of 6 in 1 groups of 1000
[INFO] [2020-05-05 14:10:58] Average Time: 0.0
[INFO] [2020-05-05 14:10:58] Total Time: 1s
[INFO] [2020-05-05 14:10:58] Storing 85 ScientificNames
[INFO] [2020-05-05 14:10:58] Processing group of 85 in 1 groups of 1000
[INFO] [2020-05-05 14:10:58] Average Time: 0.04
[INFO] [2020-05-05 14:10:58] Total Time: 1s
[INFO] [2020-05-05 14:10:58] Storing 85 Nodes
[INFO] [2020-05-05 14:10:58] Processing group of 85 in 1 groups of 1000
[INFO] [2020-05-05 14:10:58] Average Time: 0.04
[INFO] [2020-05-05 14:10:58] Total Time: 1s
[INFO] [2020-05-05 14:10:58] Storing 83 Occurrences
[INFO] [2020-05-05 14:10:58] Processing group of 83 in 1 groups of 1000
[INFO] [2020-05-05 14:10:58] Average Time: 0.02
[INFO] [2020-05-05 14:10:58] Total Time: 1s
[INFO] [2020-05-05 14:10:58] Storing 644 Traits
[INFO] [2020-05-05 14:10:58] Processing group of 644 in 1 groups of 1000
[INFO] [2020-05-05 14:10:58] Average Time: 0.24
[INFO] [2020-05-05 14:10:58] Total Time: 1s
[INFO] [2020-05-05 14:10:58] Storing 586 MetaTraits
[INFO] [2020-05-05 14:10:58] Processing group of 586 in 1 groups of 1000
[INFO] [2020-05-05 14:10:58] Average Time: 0.1
[INFO] [2020-05-05 14:10:58] Total Time: 1s
[INFO] [2020-05-05 14:10:58] Storing 324 TraitsReferences
[INFO] [2020-05-05 14:10:58] Processing group of 324 in 1 groups of 1000
[INFO] [2020-05-05 14:10:59] Average Time: 0.16
[INFO] [2020-05-05 14:10:59] Total Time: 1s
[INFO] [2020-05-05 14:10:59] Storing 332 OccurrenceMetadata
[INFO] [2020-05-05 14:10:59] Processing group of 332 in 1 groups of 1000
[INFO] [2020-05-05 14:10:59] Average Time: 0.06
[INFO] [2020-05-05 14:10:59] Total Time: 1s
[STOP] [2020-05-05 14:10:59] parse_diff_and_store
[START] [2020-05-05 14:10:59] resolve_keys
[INFO] [2020-05-05 14:11:05] Occurrences to nodes (through scientific_names)...
[INFO] [2020-05-05 14:11:05] traits to occurrences...
[INFO] [2020-05-05 14:11:05] traits to nodes (through occurrences)...
[INFO] [2020-05-05 14:11:05] Traits to sex term...
[INFO] [2020-05-05 14:11:05] Traits to lifestage term...
[INFO] [2020-05-05 14:11:05] MetaTraits to traits...
[INFO] [2020-05-05 14:11:05] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-05-05 14:11:05] Assocs to occurrences...
[INFO] [2020-05-05 14:11:05] Assocs to nodes...
[INFO] [2020-05-05 14:11:05] Assoc to sex term...
[INFO] [2020-05-05 14:11:05] Assoc to lifestage term...
[STOP] [2020-05-05 14:11:05] resolve_keys
[START] [2020-05-05 14:11:05] hold_for_later_1
[STOP] [2020-05-05 14:11:05] hold_for_later_1
[START] [2020-05-05 14:11:05] hold_for_later_2
[STOP] [2020-05-05 14:11:05] hold_for_later_2
[START] [2020-05-05 14:11:05] resolve_missing_parents
[STOP] [2020-05-05 14:11:05] resolve_missing_parents
[START] [2020-05-05 14:11:05] rebuild_nodes
[START] [2020-05-05 14:11:05] Flattener#flatten
[START] [2020-05-05 14:11:05] Flattener#study_resource
[START] [2020-05-05 14:11:05] Flattener#build_ancestry
[STOP] [2020-05-05 14:11:05] Flattener#build_ancestry
[INFO] [2020-05-05 14:11:05] 85 ancestry keys
[START] [2020-05-05 14:11:05] build_node_ancestors
[INFO] [2020-05-05 14:11:05] old ancestors deleted.
[STOP] [2020-05-05 14:11:05] build_node_ancestors
[START] [2020-05-05 14:11:05] Flattener#propagate_ancestor_ids
[STOP] [2020-05-05 14:11:05] Flattener#propagate_ancestor_ids
[STOP] [2020-05-05 14:11:05] Flattener#flatten
[STOP] [2020-05-05 14:11:05] rebuild_nodes
[START] [2020-05-05 14:11:05] resolve_missing_media_owners
[STOP] [2020-05-05 14:11:05] resolve_missing_media_owners
[START] [2020-05-05 14:11:05] sanitize_media_verbatims
[STOP] [2020-05-05 14:11:05] sanitize_media_verbatims
[START] [2020-05-05 14:11:05] queue_downloads
[STOP] [2020-05-05 14:11:05] queue_downloads
[START] [2020-05-05 14:11:05] parse_names
[WARN] [2020-05-05 14:11:05] I see 85 names which still need to be parsed.
[STOP] [2020-05-05 14:11:06] parse_names
[START] [2020-05-05 14:11:06] denormalize_canonical_names_to_nodes
[STOP] [2020-05-05 14:11:06] denormalize_canonical_names_to_nodes
[START] [2020-05-05 14:11:06] match_nodes
[START] [2020-05-05 14:11:06] map_all_nodes_to_pages
[STOP] [2020-05-05 14:11:08] map_all_nodes_to_pages
[INFO] [2020-05-05 14:11:08] Unmatched nodes (2 of 85): Crociduridae (#73316993); Scalopidae (#73317045)
[START] [2020-05-05 14:11:08] update_nodes
[STOP] [2020-05-05 14:11:08] update_nodes
[STOP] [2020-05-05 14:11:08] match_nodes
[START] [2020-05-05 14:11:08] reindex_search
[STOP] [2020-05-05 14:11:08] reindex_search
[START] [2020-05-05 14:11:08] normalize_units
[STOP] [2020-05-05 14:11:08] normalize_units
[START] [2020-05-05 14:11:08] calculate_statistics
[STOP] [2020-05-05 14:11:08] calculate_statistics
[START] [2020-05-05 14:11:08] complete_harvest_instance
[START] [2020-05-05 14:11:08] overall_tsv_creation
[INFO] [2020-05-05 14:11:08] Processing group of 85 in 1 batches of 10000
[INFO] [2020-05-05 14:12:02] 158 Traits (unfiltered)...
[INFO] [2020-05-05 14:12:15] 158 Traits (filtered)...
[INFO] [2020-05-05 14:12:15] 0 Associations (filtered)...
[INFO] [2020-05-05 14:12:53] 1584 metadata added.
[INFO] [2020-05-05 14:12:53] 0 metadata added.
[INFO] [2020-05-05 14:12:53] Average Time: 74.64
[INFO] [2020-05-05 14:12:53] Total Time: 1m46s
[STOP] [2020-05-05 14:12:53] overall_tsv_creation
[INFO] [2020-05-05 14:12:53] Done. Check your files:
[INFO] [2020-05-05 14:12:53] (85 lines) /app/public/data/lovegrove/publish_nodes.tsv
[INFO] [2020-05-05 14:12:54] (5 lines) /app/public/data/lovegrove/publish_node_ancestors.tsv
[INFO] [2020-05-05 14:12:54] (85 lines) /app/public/data/lovegrove/publish_scientific_names.tsv
[INFO] [2020-05-05 14:12:54] (159 lines) /app/public/data/lovegrove/publish_traits.tsv
[INFO] [2020-05-05 14:12:54] (1585 lines) /app/public/data/lovegrove/publish_metadata.tsv
[STOP] [2020-05-05 14:12:54] complete_harvest_instance
[START] [2020-05-05 14:12:54] completed
[STOP] [2020-05-05 14:12:54] completed
[STOP] [2020-05-05 14:12:54] logged process, took 133.28
[INFO] [2020-11-05 16:44:00] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-11-05 16:44:04] ## remove_type: ScientificName
[INFO] [2020-11-05 16:44:04] ++ Calling delete_all on 85 instances...
[INFO] [2020-11-05 16:44:04] [16:44:04.193] Removed 85 Scientificnames
[INFO] [2020-11-05 16:44:04] ## remove_type: Vernacular
[INFO] [2020-11-05 16:44:04] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:44:04] [16:44:04.197] Removed 0 Vernaculars
[INFO] [2020-11-05 16:44:04] ## remove_type: Article
[INFO] [2020-11-05 16:44:04] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:44:04] [16:44:04.200] Removed 0 Articles
[INFO] [2020-11-05 16:44:04] ## remove_type: Medium
[INFO] [2020-11-05 16:44:04] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:44:04] [16:44:04.204] Removed 0 Media
[INFO] [2020-11-05 16:44:04] ## remove_type: Trait
[INFO] [2020-11-05 16:44:04] ++ Calling delete_all on 644 instances...
[INFO] [2020-11-05 16:44:04] [16:44:04.267] Removed 644 Traits
[INFO] [2020-11-05 16:44:04] ## remove_type: MetaTrait
[INFO] [2020-11-05 16:44:04] ++ Calling delete_all on 586 instances...
[INFO] [2020-11-05 16:44:04] [16:44:04.327] Removed 586 Metatraits
[INFO] [2020-11-05 16:44:04] ## remove_type: OccurrenceMetadatum
[INFO] [2020-11-05 16:44:04] ++ Calling delete_all on 332 instances...
[INFO] [2020-11-05 16:44:04] [16:44:04.334] Removed 332 Occurrencemetadata
[INFO] [2020-11-05 16:44:04] ## remove_type: Assoc
[INFO] [2020-11-05 16:44:04] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:44:04] [16:44:04.337] Removed 0 Assocs
[INFO] [2020-11-05 16:44:04] ## remove_type: MetaAssoc
[INFO] [2020-11-05 16:44:04] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:44:04] [16:44:04.340] Removed 0 Metaassocs
[INFO] [2020-11-05 16:44:04] ## remove_type: Identifier
[INFO] [2020-11-05 16:44:04] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:44:04] [16:44:04.352] Removed 0 Identifiers
[INFO] [2020-11-05 16:44:04] ## remove_type: Reference
[INFO] [2020-11-05 16:44:04] ++ Calling delete_all on 6 instances...
[INFO] [2020-11-05 16:44:04] [16:44:04.363] Removed 6 References
[INFO] [2020-11-05 16:44:04] Starting batch with ID 73316978...
[INFO] [2020-11-05 16:44:04] Starting batch with ID 73316978...
[INFO] [2020-11-05 16:44:04] Starting batch with ID 73316978...
[INFO] [2020-11-05 16:44:04] Starting batch with ID 73316978...
[INFO] [2020-11-05 16:44:04] Starting batch with ID 73316978...
[INFO] [2020-11-05 16:44:04] Starting batch with ID 73316978...
[INFO] [2020-11-05 16:44:04] Starting batch with ID 73316978...
[INFO] [2020-11-05 16:44:04] Starting batch with ID 73316978...
[INFO] [2020-11-05 16:44:04] Starting batch with ID 73316978...
[INFO] [2020-11-05 16:44:04] Starting batch with ID 73316978...
[INFO] [2020-11-05 16:44:04] Starting batch with ID 73316978...
[INFO] [2020-11-05 16:44:04] ## remove_type: Node
[INFO] [2020-11-05 16:44:04] ++ Calling delete_all on 85 instances...
[INFO] [2020-11-05 16:44:04] [16:44:04.940] Removed 85 Nodes
[START] [2020-11-05 16:44:05] logged process: 9719e9b2443e9b331c213f6700b6e76a1b7c8548

[START] [2020-11-05 16:44:05] Creating resource from OpenData
[START] [2020-11-05 16:44:05] logged process: 9719e9b2443e9b331c213f6700b6e76a1b7c8548

[START] [2020-11-05 16:44:05] Parse meta.xml file and create formats with fields
[WARN] [2020-11-05 16:44:06] (common) IGNORED  (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2020-11-05 16:44:06] (common) IGNORED  (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2020-11-05 16:44:06] (common) IGNORED  (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2020-11-05 16:44:06] (common) IGNORED  (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2020-11-05 16:44:06] (common) IGNORED  (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2020-11-05 16:44:06] (common) IGNORED  (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2020-11-05 16:44:06] (common) IGNORED  (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[WARN] [2020-11-05 16:44:06] (common) IGNORED  (occurrences) field header: Institution Code term: http://rs.tdwg.org/dwc/terms/institutionCode
[WARN] [2020-11-05 16:44:06] (common) IGNORED  (occurrences) field header: Collection Code term: http://rs.tdwg.org/dwc/terms/collectionCode
[WARN] [2020-11-05 16:44:06] (common) IGNORED  (occurrences) field header: Catalog Number term: http://rs.tdwg.org/dwc/terms/catalogNumber
[WARN] [2020-11-05 16:44:06] SKIPPING http://rs.tdwg.org/dwc/terms/Event config (events.txt)...
[STOP] [2020-11-05 16:44:06] Parse meta.xml file and create formats with fields
[STOP] [2020-11-05 16:44:06] Creating resource from OpenData
[START] [2020-11-05 16:44:06] logged process: 9719e9b2443e9b331c213f6700b6e76a1b7c8548

[START] [2020-11-05 16:44:06] create_harvest_instance
[STOP] [2020-11-05 16:44:15] create_harvest_instance
[START] [2020-11-05 16:44:15] fetch_files
[STOP] [2020-11-05 16:44:15] fetch_files
[START] [2020-11-05 16:44:15] validate_each_file
[STOP] [2020-11-05 16:44:16] validate_each_file
[START] [2020-11-05 16:44:16] convert_to_csv
[CMD] [2020-11-05 16:44:16] /usr/bin/sort /app/public/converted_csv/lovegrove_agents_23570.csv > /app/public/converted_csv/lovegrove_agents_23570.csv_sorted
[CMD] [2020-11-05 16:44:16] /usr/bin/sort /app/public/converted_csv/lovegrove_refs_23571.csv > /app/public/converted_csv/lovegrove_refs_23571.csv_sorted
[CMD] [2020-11-05 16:44:16] /usr/bin/sort /app/public/converted_csv/lovegrove_nodes_23572.csv > /app/public/converted_csv/lovegrove_nodes_23572.csv_sorted
[CMD] [2020-11-05 16:44:16] /usr/bin/sort /app/public/converted_csv/lovegrove_media_23573.csv > /app/public/converted_csv/lovegrove_media_23573.csv_sorted
[CMD] [2020-11-05 16:44:16] /usr/bin/sort /app/public/converted_csv/lovegrove_vernaculars_23574.csv > /app/public/converted_csv/lovegrove_vernaculars_23574.csv_sorted
[CMD] [2020-11-05 16:44:16] /usr/bin/sort /app/public/converted_csv/lovegrove_occurrences_23575.csv > /app/public/converted_csv/lovegrove_occurrences_23575.csv_sorted
[CMD] [2020-11-05 16:44:16] /usr/bin/sort /app/public/converted_csv/lovegrove_assocs_23576.csv > /app/public/converted_csv/lovegrove_assocs_23576.csv_sorted
[CMD] [2020-11-05 16:44:16] /usr/bin/sort /app/public/converted_csv/lovegrove_measurements_23577.csv > /app/public/converted_csv/lovegrove_measurements_23577.csv_sorted
[STOP] [2020-11-05 16:44:16] convert_to_csv
[START] [2020-11-05 16:44:16] calculate_delta
[CMD] [2020-11-05 16:44:16] echo "0a" > /app/public/diff/lovegrove_agents_23570.diff
[CMD] [2020-11-05 16:44:16] tail -n +1 /app/public/converted_csv/lovegrove_agents_23570.csv >> /app/public/diff/lovegrove_agents_23570.diff
[CMD] [2020-11-05 16:44:16] echo "." >> /app/public/diff/lovegrove_agents_23570.diff
[CMD] [2020-11-05 16:44:16] echo "0a" > /app/public/diff/lovegrove_refs_23571.diff
[CMD] [2020-11-05 16:44:16] tail -n +1 /app/public/converted_csv/lovegrove_refs_23571.csv >> /app/public/diff/lovegrove_refs_23571.diff
[CMD] [2020-11-05 16:44:16] echo "." >> /app/public/diff/lovegrove_refs_23571.diff
[CMD] [2020-11-05 16:44:16] echo "0a" > /app/public/diff/lovegrove_nodes_23572.diff
[CMD] [2020-11-05 16:44:16] tail -n +1 /app/public/converted_csv/lovegrove_nodes_23572.csv >> /app/public/diff/lovegrove_nodes_23572.diff
[CMD] [2020-11-05 16:44:16] echo "." >> /app/public/diff/lovegrove_nodes_23572.diff
[CMD] [2020-11-05 16:44:16] echo "0a" > /app/public/diff/lovegrove_media_23573.diff
[CMD] [2020-11-05 16:44:16] tail -n +1 /app/public/converted_csv/lovegrove_media_23573.csv >> /app/public/diff/lovegrove_media_23573.diff
[CMD] [2020-11-05 16:44:16] echo "." >> /app/public/diff/lovegrove_media_23573.diff
[CMD] [2020-11-05 16:44:16] echo "0a" > /app/public/diff/lovegrove_vernaculars_23574.diff
[CMD] [2020-11-05 16:44:16] tail -n +1 /app/public/converted_csv/lovegrove_vernaculars_23574.csv >> /app/public/diff/lovegrove_vernaculars_23574.diff
[CMD] [2020-11-05 16:44:16] echo "." >> /app/public/diff/lovegrove_vernaculars_23574.diff
[CMD] [2020-11-05 16:44:16] echo "0a" > /app/public/diff/lovegrove_occurrences_23575.diff
[CMD] [2020-11-05 16:44:16] tail -n +1 /app/public/converted_csv/lovegrove_occurrences_23575.csv >> /app/public/diff/lovegrove_occurrences_23575.diff
[CMD] [2020-11-05 16:44:16] echo "." >> /app/public/diff/lovegrove_occurrences_23575.diff
[CMD] [2020-11-05 16:44:16] echo "0a" > /app/public/diff/lovegrove_assocs_23576.diff
[CMD] [2020-11-05 16:44:16] tail -n +1 /app/public/converted_csv/lovegrove_assocs_23576.csv >> /app/public/diff/lovegrove_assocs_23576.diff
[CMD] [2020-11-05 16:44:16] echo "." >> /app/public/diff/lovegrove_assocs_23576.diff
[CMD] [2020-11-05 16:44:16] echo "0a" > /app/public/diff/lovegrove_measurements_23577.diff
[CMD] [2020-11-05 16:44:16] tail -n +1 /app/public/converted_csv/lovegrove_measurements_23577.csv >> /app/public/diff/lovegrove_measurements_23577.diff
[CMD] [2020-11-05 16:44:16] echo "." >> /app/public/diff/lovegrove_measurements_23577.diff
[STOP] [2020-11-05 16:44:16] calculate_delta
[START] [2020-11-05 16:44:16] parse_diff_and_store
[INFO] [2020-11-05 16:44:16] Loading agents diff file into memory (true lines)...
[INFO] [2020-11-05 16:44:16] Loading refs diff file into memory (true lines)...
[INFO] [2020-11-05 16:44:16] Loading nodes diff file into memory (true lines)...
[INFO] [2020-11-05 16:44:16] Loading media diff file into memory (true lines)...
[INFO] [2020-11-05 16:44:16] Loading vernaculars diff file into memory (true lines)...
[INFO] [2020-11-05 16:44:16] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-11-05 16:44:16] Loading assocs diff file into memory (true lines)...
[INFO] [2020-11-05 16:44:16] Loading measurements diff file into memory (true lines)...
[STOP] [2020-11-05 16:44:17] parse_diff_and_store
[ERR] [2020-11-05 16:44:17] RuntimeError
[ERR] [2020-11-05 16:44:17] Missing Term for URI `http://purl.obolibrary.org/obo/GO_0040011 `, must be added!
[ERR] [2020-11-05 16:44:17] ../models/store/model_builder.rb:635:in `fail_on_bad_uri'
[ERR] [2020-11-05 16:44:17] ../models/store/model_builder.rb:408:in `build_trait'
[ERR] [2020-11-05 16:44:17] ../models/store/model_builder.rb:28:in `build_models'
[ERR] [2020-11-05 16:44:17] ../models/resource_harvester.rb:357:in `block (3 levels) in parse_diff_and_store'
[ERR] [2020-11-05 16:44:17] ../models/csv_parser.rb:111:in `block in diff_as_hashes'
[ERR] [2020-11-05 16:44:17] ../models/csv_parser.rb:28:in `block in line_at_a_time'
[ERR] [2020-11-05 16:44:17] ../models/csv_parser.rb:25:in `line_at_a_time'
[ERR] [2020-11-05 16:44:17] ../models/csv_parser.rb:96:in `diff_as_hashes'
[ERR] [2020-11-05 16:44:17] ../models/resource_harvester.rb:313:in `block (2 levels) in parse_diff_and_store'
[ERR] [2020-11-05 16:44:17] ../models/logged_process.rb:62:in `enter_group'
[ERR] [2020-11-05 16:44:17] ../models/resource_harvester.rb:312:in `block in parse_diff_and_store'
[ERR] [2020-11-05 16:44:17] ../models/resource_harvester.rb:700:in `block in each_diff'
[ERR] [2020-11-05 16:44:17] ../models/resource_harvester.rb:689:in `each_diff'
[ERR] [2020-11-05 16:44:17] ../models/resource_harvester.rb:307:in `parse_diff_and_store'
[ERR] [2020-11-05 16:44:17] ../models/resource_harvester.rb:85:in `block (3 levels) in start'
[ERR] [2020-11-05 16:44:17] ../models/logged_process.rb:19:in `run_step'
[ERR] [2020-11-05 16:44:17] ../models/resource_harvester.rb:85:in `block (2 levels) in start'
[ERR] [2020-11-05 16:44:17] ../models/resource_harvester.rb:74:in `each_key'
[ERR] [2020-11-05 16:44:17] ../models/resource_harvester.rb:74:in `block in start'
[ERR] [2020-11-05 16:44:17] ../models/resource.rb:151:in `lock'
[ERR] [2020-11-05 16:44:17] ../models/resource_harvester.rb:72:in `start'
[ERR] [2020-11-05 16:44:17] ../models/resource.rb:232:in `harvest'
[ERR] [2020-11-05 16:44:17] ../models/resource.rb:208:in `re_download_opendata_and_harvest'
[ERR] [2020-11-05 16:44:17] bin/rails:4:in `require'
[ERR] [2020-11-05 16:44:17] bin/rails:4:in `<main>'
[STOP] [2020-11-05 16:44:17] logged process, took 11.44
[INFO] [2020-11-05 16:45:48] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-11-05 16:45:52] ## remove_type: ScientificName
[INFO] [2020-11-05 16:45:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:45:52] [16:45:52.116] Removed 0 Scientificnames
[INFO] [2020-11-05 16:45:52] ## remove_type: Vernacular
[INFO] [2020-11-05 16:45:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:45:52] [16:45:52.119] Removed 0 Vernaculars
[INFO] [2020-11-05 16:45:52] ## remove_type: Article
[INFO] [2020-11-05 16:45:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:45:52] [16:45:52.122] Removed 0 Articles
[INFO] [2020-11-05 16:45:52] ## remove_type: Medium
[INFO] [2020-11-05 16:45:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:45:52] [16:45:52.126] Removed 0 Media
[INFO] [2020-11-05 16:45:52] ## remove_type: Trait
[INFO] [2020-11-05 16:45:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:45:52] [16:45:52.130] Removed 0 Traits
[INFO] [2020-11-05 16:45:52] ## remove_type: MetaTrait
[INFO] [2020-11-05 16:45:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:45:52] [16:45:52.133] Removed 0 Metatraits
[INFO] [2020-11-05 16:45:52] ## remove_type: OccurrenceMetadatum
[INFO] [2020-11-05 16:45:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:45:52] [16:45:52.136] Removed 0 Occurrencemetadata
[INFO] [2020-11-05 16:45:52] ## remove_type: Assoc
[INFO] [2020-11-05 16:45:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:45:52] [16:45:52.139] Removed 0 Assocs
[INFO] [2020-11-05 16:45:52] ## remove_type: MetaAssoc
[INFO] [2020-11-05 16:45:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:45:52] [16:45:52.141] Removed 0 Metaassocs
[INFO] [2020-11-05 16:45:52] ## remove_type: Identifier
[INFO] [2020-11-05 16:45:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:45:52] [16:45:52.144] Removed 0 Identifiers
[INFO] [2020-11-05 16:45:52] ## remove_type: Reference
[INFO] [2020-11-05 16:45:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:45:52] [16:45:52.147] Removed 0 References
[INFO] [2020-11-05 16:45:52] ## remove_type: Node
[INFO] [2020-11-05 16:45:52] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:45:52] [16:45:52.177] Removed 0 Nodes
[START] [2020-11-05 16:45:52] logged process: 9719e9b2443e9b331c213f6700b6e76a1b7c8548

[START] [2020-11-05 16:45:52] Creating resource from OpenData
[START] [2020-11-05 16:45:53] logged process: 9719e9b2443e9b331c213f6700b6e76a1b7c8548

[START] [2020-11-05 16:45:53] Parse meta.xml file and create formats with fields
[WARN] [2020-11-05 16:45:53] (common) IGNORED  (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2020-11-05 16:45:53] (common) IGNORED  (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2020-11-05 16:45:53] (common) IGNORED  (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2020-11-05 16:45:53] (common) IGNORED  (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2020-11-05 16:45:53] (common) IGNORED  (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2020-11-05 16:45:53] (common) IGNORED  (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2020-11-05 16:45:53] (common) IGNORED  (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[WARN] [2020-11-05 16:45:53] (common) IGNORED  (occurrences) field header: Institution Code term: http://rs.tdwg.org/dwc/terms/institutionCode
[WARN] [2020-11-05 16:45:53] (common) IGNORED  (occurrences) field header: Collection Code term: http://rs.tdwg.org/dwc/terms/collectionCode
[WARN] [2020-11-05 16:45:53] (common) IGNORED  (occurrences) field header: Catalog Number term: http://rs.tdwg.org/dwc/terms/catalogNumber
[WARN] [2020-11-05 16:45:53] SKIPPING http://rs.tdwg.org/dwc/terms/Event config (events.txt)...
[STOP] [2020-11-05 16:45:53] Parse meta.xml file and create formats with fields
[STOP] [2020-11-05 16:45:53] Creating resource from OpenData
[START] [2020-11-05 16:45:53] logged process: 9719e9b2443e9b331c213f6700b6e76a1b7c8548

[START] [2020-11-05 16:45:53] create_harvest_instance
[STOP] [2020-11-05 16:46:03] create_harvest_instance
[START] [2020-11-05 16:46:03] fetch_files
[STOP] [2020-11-05 16:46:03] fetch_files
[START] [2020-11-05 16:46:03] validate_each_file
[STOP] [2020-11-05 16:46:03] validate_each_file
[START] [2020-11-05 16:46:03] convert_to_csv
[CMD] [2020-11-05 16:46:03] /usr/bin/sort /app/public/converted_csv/lovegrove_agents_23586.csv > /app/public/converted_csv/lovegrove_agents_23586.csv_sorted
[CMD] [2020-11-05 16:46:03] /usr/bin/sort /app/public/converted_csv/lovegrove_refs_23587.csv > /app/public/converted_csv/lovegrove_refs_23587.csv_sorted
[CMD] [2020-11-05 16:46:03] /usr/bin/sort /app/public/converted_csv/lovegrove_nodes_23588.csv > /app/public/converted_csv/lovegrove_nodes_23588.csv_sorted
[CMD] [2020-11-05 16:46:03] /usr/bin/sort /app/public/converted_csv/lovegrove_media_23589.csv > /app/public/converted_csv/lovegrove_media_23589.csv_sorted
[CMD] [2020-11-05 16:46:03] /usr/bin/sort /app/public/converted_csv/lovegrove_vernaculars_23590.csv > /app/public/converted_csv/lovegrove_vernaculars_23590.csv_sorted
[CMD] [2020-11-05 16:46:03] /usr/bin/sort /app/public/converted_csv/lovegrove_occurrences_23591.csv > /app/public/converted_csv/lovegrove_occurrences_23591.csv_sorted
[CMD] [2020-11-05 16:46:03] /usr/bin/sort /app/public/converted_csv/lovegrove_assocs_23592.csv > /app/public/converted_csv/lovegrove_assocs_23592.csv_sorted
[CMD] [2020-11-05 16:46:03] /usr/bin/sort /app/public/converted_csv/lovegrove_measurements_23593.csv > /app/public/converted_csv/lovegrove_measurements_23593.csv_sorted
[STOP] [2020-11-05 16:46:03] convert_to_csv
[START] [2020-11-05 16:46:03] calculate_delta
[CMD] [2020-11-05 16:46:03] echo "0a" > /app/public/diff/lovegrove_agents_23586.diff
[CMD] [2020-11-05 16:46:03] tail -n +1 /app/public/converted_csv/lovegrove_agents_23586.csv >> /app/public/diff/lovegrove_agents_23586.diff
[CMD] [2020-11-05 16:46:03] echo "." >> /app/public/diff/lovegrove_agents_23586.diff
[CMD] [2020-11-05 16:46:03] echo "0a" > /app/public/diff/lovegrove_refs_23587.diff
[CMD] [2020-11-05 16:46:03] tail -n +1 /app/public/converted_csv/lovegrove_refs_23587.csv >> /app/public/diff/lovegrove_refs_23587.diff
[CMD] [2020-11-05 16:46:03] echo "." >> /app/public/diff/lovegrove_refs_23587.diff
[CMD] [2020-11-05 16:46:03] echo "0a" > /app/public/diff/lovegrove_nodes_23588.diff
[CMD] [2020-11-05 16:46:03] tail -n +1 /app/public/converted_csv/lovegrove_nodes_23588.csv >> /app/public/diff/lovegrove_nodes_23588.diff
[CMD] [2020-11-05 16:46:03] echo "." >> /app/public/diff/lovegrove_nodes_23588.diff
[CMD] [2020-11-05 16:46:03] echo "0a" > /app/public/diff/lovegrove_media_23589.diff
[CMD] [2020-11-05 16:46:03] tail -n +1 /app/public/converted_csv/lovegrove_media_23589.csv >> /app/public/diff/lovegrove_media_23589.diff
[CMD] [2020-11-05 16:46:03] echo "." >> /app/public/diff/lovegrove_media_23589.diff
[CMD] [2020-11-05 16:46:03] echo "0a" > /app/public/diff/lovegrove_vernaculars_23590.diff
[CMD] [2020-11-05 16:46:03] tail -n +1 /app/public/converted_csv/lovegrove_vernaculars_23590.csv >> /app/public/diff/lovegrove_vernaculars_23590.diff
[CMD] [2020-11-05 16:46:03] echo "." >> /app/public/diff/lovegrove_vernaculars_23590.diff
[CMD] [2020-11-05 16:46:03] echo "0a" > /app/public/diff/lovegrove_occurrences_23591.diff
[CMD] [2020-11-05 16:46:03] tail -n +1 /app/public/converted_csv/lovegrove_occurrences_23591.csv >> /app/public/diff/lovegrove_occurrences_23591.diff
[CMD] [2020-11-05 16:46:03] echo "." >> /app/public/diff/lovegrove_occurrences_23591.diff
[CMD] [2020-11-05 16:46:03] echo "0a" > /app/public/diff/lovegrove_assocs_23592.diff
[CMD] [2020-11-05 16:46:03] tail -n +1 /app/public/converted_csv/lovegrove_assocs_23592.csv >> /app/public/diff/lovegrove_assocs_23592.diff
[CMD] [2020-11-05 16:46:03] echo "." >> /app/public/diff/lovegrove_assocs_23592.diff
[CMD] [2020-11-05 16:46:03] echo "0a" > /app/public/diff/lovegrove_measurements_23593.diff
[CMD] [2020-11-05 16:46:03] tail -n +1 /app/public/converted_csv/lovegrove_measurements_23593.csv >> /app/public/diff/lovegrove_measurements_23593.diff
[CMD] [2020-11-05 16:46:03] echo "." >> /app/public/diff/lovegrove_measurements_23593.diff
[STOP] [2020-11-05 16:46:03] calculate_delta
[START] [2020-11-05 16:46:03] parse_diff_and_store
[INFO] [2020-11-05 16:46:03] Loading agents diff file into memory (true lines)...
[INFO] [2020-11-05 16:46:03] Loading refs diff file into memory (true lines)...
[INFO] [2020-11-05 16:46:03] Loading nodes diff file into memory (true lines)...
[INFO] [2020-11-05 16:46:03] Loading media diff file into memory (true lines)...
[INFO] [2020-11-05 16:46:03] Loading vernaculars diff file into memory (true lines)...
[INFO] [2020-11-05 16:46:03] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-11-05 16:46:03] Loading assocs diff file into memory (true lines)...
[INFO] [2020-11-05 16:46:03] Loading measurements diff file into memory (true lines)...
[INFO] [2020-11-05 16:46:05] Storing 6 References
[INFO] [2020-11-05 16:46:05] Processing group of 6 in 1 groups of 1000
[INFO] [2020-11-05 16:46:05] Average Time: 0.0
[INFO] [2020-11-05 16:46:05] Total Time: 1s
[INFO] [2020-11-05 16:46:05] Storing 85 ScientificNames
[INFO] [2020-11-05 16:46:05] Processing group of 85 in 1 groups of 1000
[INFO] [2020-11-05 16:46:05] Average Time: 0.03
[INFO] [2020-11-05 16:46:05] Total Time: 1s
[INFO] [2020-11-05 16:46:05] Storing 85 Nodes
[INFO] [2020-11-05 16:46:05] Processing group of 85 in 1 groups of 1000
[INFO] [2020-11-05 16:46:05] Average Time: 0.02
[INFO] [2020-11-05 16:46:05] Total Time: 1s
[INFO] [2020-11-05 16:46:05] Storing 83 Occurrences
[INFO] [2020-11-05 16:46:05] Processing group of 83 in 1 groups of 1000
[INFO] [2020-11-05 16:46:05] Average Time: 0.01
[INFO] [2020-11-05 16:46:05] Total Time: 1s
[INFO] [2020-11-05 16:46:05] Storing 644 Traits
[INFO] [2020-11-05 16:46:05] Processing group of 644 in 1 groups of 1000
[INFO] [2020-11-05 16:46:05] Average Time: 0.2
[INFO] [2020-11-05 16:46:05] Total Time: 1s
[INFO] [2020-11-05 16:46:05] Storing 324 TraitsReferences
[INFO] [2020-11-05 16:46:05] Processing group of 324 in 1 groups of 1000
[INFO] [2020-11-05 16:46:05] Average Time: 0.02
[INFO] [2020-11-05 16:46:05] Total Time: 1s
[INFO] [2020-11-05 16:46:05] Storing 316 MetaTraits
[INFO] [2020-11-05 16:46:05] Processing group of 316 in 1 groups of 1000
[INFO] [2020-11-05 16:46:05] Average Time: 0.04
[INFO] [2020-11-05 16:46:05] Total Time: 1s
[INFO] [2020-11-05 16:46:05] Storing 166 OccurrenceMetadata
[INFO] [2020-11-05 16:46:05] Processing group of 166 in 1 groups of 1000
[INFO] [2020-11-05 16:46:05] Average Time: 0.03
[INFO] [2020-11-05 16:46:05] Total Time: 1s
[STOP] [2020-11-05 16:46:05] parse_diff_and_store
[START] [2020-11-05 16:46:05] resolve_keys
[INFO] [2020-11-05 16:46:11] Occurrences to nodes (through scientific_names)...
[INFO] [2020-11-05 16:46:11] traits to occurrences...
[INFO] [2020-11-05 16:46:11] traits to nodes (through occurrences)...
[INFO] [2020-11-05 16:46:11] Traits to sex term...
[INFO] [2020-11-05 16:46:11] Traits to lifestage term...
[INFO] [2020-11-05 16:46:11] MetaTraits to traits...
[INFO] [2020-11-05 16:46:11] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-11-05 16:46:11] Assocs to occurrences...
[INFO] [2020-11-05 16:46:11] Assocs to nodes...
[INFO] [2020-11-05 16:46:11] Assoc to sex term...
[INFO] [2020-11-05 16:46:11] Assoc to lifestage term...
[INFO] [2020-11-05 16:46:11] MetaAssoc to assocs...
[STOP] [2020-11-05 16:46:11] resolve_keys
[START] [2020-11-05 16:46:11] hold_for_later_1
[STOP] [2020-11-05 16:46:11] hold_for_later_1
[START] [2020-11-05 16:46:11] hold_for_later_2
[STOP] [2020-11-05 16:46:11] hold_for_later_2
[START] [2020-11-05 16:46:11] resolve_missing_parents
[STOP] [2020-11-05 16:46:11] resolve_missing_parents
[START] [2020-11-05 16:46:11] rebuild_nodes
[START] [2020-11-05 16:46:11] Flattener#flatten
[START] [2020-11-05 16:46:11] Flattener#study_resource
[START] [2020-11-05 16:46:11] Flattener#build_ancestry
[STOP] [2020-11-05 16:46:11] Flattener#build_ancestry
[INFO] [2020-11-05 16:46:11] 85 ancestry keys
[START] [2020-11-05 16:46:11] build_node_ancestors
[INFO] [2020-11-05 16:46:11] old ancestors deleted.
[STOP] [2020-11-05 16:46:11] build_node_ancestors
[START] [2020-11-05 16:46:11] Flattener#propagate_ancestor_ids
[STOP] [2020-11-05 16:46:11] Flattener#propagate_ancestor_ids
[STOP] [2020-11-05 16:46:11] Flattener#flatten
[STOP] [2020-11-05 16:46:11] rebuild_nodes
[START] [2020-11-05 16:46:11] resolve_missing_media_owners
[STOP] [2020-11-05 16:46:11] resolve_missing_media_owners
[START] [2020-11-05 16:46:11] sanitize_media_verbatims
[STOP] [2020-11-05 16:46:11] sanitize_media_verbatims
[START] [2020-11-05 16:46:11] queue_downloads
[STOP] [2020-11-05 16:46:11] queue_downloads
[START] [2020-11-05 16:46:11] parse_names
[WARN] [2020-11-05 16:46:11] I see 85 names which still need to be parsed.
[STOP] [2020-11-05 16:46:12] parse_names
[START] [2020-11-05 16:46:12] denormalize_canonical_names_to_nodes
[STOP] [2020-11-05 16:46:12] denormalize_canonical_names_to_nodes
[START] [2020-11-05 16:46:12] match_nodes
[START] [2020-11-05 16:46:12] map_all_nodes_to_pages
[STOP] [2020-11-05 16:46:12] map_all_nodes_to_pages
[INFO] [2020-11-05 16:46:12] Unmatched nodes (1 of 85): Crociduridae (#81423629)
[START] [2020-11-05 16:46:12] update_nodes
[STOP] [2020-11-05 16:46:12] update_nodes
[STOP] [2020-11-05 16:46:12] match_nodes
[START] [2020-11-05 16:46:12] reindex_search
[STOP] [2020-11-05 16:46:13] reindex_search
[START] [2020-11-05 16:46:13] normalize_units
[STOP] [2020-11-05 16:46:13] normalize_units
[START] [2020-11-05 16:46:13] calculate_statistics
[STOP] [2020-11-05 16:46:13] calculate_statistics
[START] [2020-11-05 16:46:13] complete_harvest_instance
[START] [2020-11-05 16:46:13] overall_tsv_creation
[INFO] [2020-11-05 16:46:13] Processing group of 85 in 1 batches of 10000
[INFO] [2020-11-05 16:46:54] 158 Traits (unfiltered)...
[INFO] [2020-11-05 16:47:32] 158 Traits (filtered)...
[INFO] [2020-11-05 16:47:32] 0 Associations (filtered)...
[INFO] [2020-11-05 16:47:32] 1110 metadata added.
[INFO] [2020-11-05 16:47:32] 0 metadata added.
[INFO] [2020-11-05 16:47:32] Average Time: 55.14
[INFO] [2020-11-05 16:47:32] Total Time: 1m20s
[STOP] [2020-11-05 16:47:32] overall_tsv_creation
[INFO] [2020-11-05 16:47:32] Done. Check your files:
[INFO] [2020-11-05 16:47:32] (85 lines) /app/public/data/lovegrove/publish_nodes.tsv
[INFO] [2020-11-05 16:47:32] (5 lines) /app/public/data/lovegrove/publish_node_ancestors.tsv
[INFO] [2020-11-05 16:47:32] (85 lines) /app/public/data/lovegrove/publish_scientific_names.tsv
[INFO] [2020-11-05 16:47:32] (159 lines) /app/public/data/lovegrove/publish_traits.tsv
[INFO] [2020-11-05 16:47:32] (1111 lines) /app/public/data/lovegrove/publish_metadata.tsv
[STOP] [2020-11-05 16:47:32] complete_harvest_instance
[START] [2020-11-05 16:47:32] completed
[STOP] [2020-11-05 16:47:32] completed
[STOP] [2020-11-05 16:47:32] logged process, took 99.46
[INFO] [2021-04-07 14:30:37] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-07 14:30:40] ## remove_type: ScientificName
[INFO] [2021-04-07 14:30:40] ++ Calling delete_all on 85 instances...
[INFO] [2021-04-07 14:30:40] [14:30:40.267] Removed 85 Scientificnames
[INFO] [2021-04-07 14:30:40] ## remove_type: Vernacular
[INFO] [2021-04-07 14:30:40] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 14:30:40] [14:30:40.269] Removed 0 Vernaculars
[INFO] [2021-04-07 14:30:40] ## remove_type: Article
[INFO] [2021-04-07 14:30:40] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 14:30:40] [14:30:40.270] Removed 0 Articles
[INFO] [2021-04-07 14:30:40] ## remove_type: Medium
[INFO] [2021-04-07 14:30:40] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 14:30:40] [14:30:40.271] Removed 0 Media
[INFO] [2021-04-07 14:30:40] ## remove_type: Trait
[INFO] [2021-04-07 14:30:40] ++ Calling delete_all on 644 instances...
[INFO] [2021-04-07 14:30:40] [14:30:40.356] Removed 644 Traits
[INFO] [2021-04-07 14:30:40] ## remove_type: MetaTrait
[INFO] [2021-04-07 14:30:40] ++ Calling delete_all on 316 instances...
[INFO] [2021-04-07 14:30:40] [14:30:40.370] Removed 316 Metatraits
[INFO] [2021-04-07 14:30:40] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-07 14:30:40] ++ Calling delete_all on 166 instances...
[INFO] [2021-04-07 14:30:40] [14:30:40.373] Removed 166 Occurrencemetadata
[INFO] [2021-04-07 14:30:40] ## remove_type: Assoc
[INFO] [2021-04-07 14:30:40] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 14:30:40] [14:30:40.375] Removed 0 Assocs
[INFO] [2021-04-07 14:30:40] ## remove_type: MetaAssoc
[INFO] [2021-04-07 14:30:40] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 14:30:40] [14:30:40.376] Removed 0 Metaassocs
[INFO] [2021-04-07 14:30:40] ## remove_type: Identifier
[INFO] [2021-04-07 14:30:40] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 14:30:40] [14:30:40.378] Removed 0 Identifiers
[INFO] [2021-04-07 14:30:40] ## remove_type: Reference
[INFO] [2021-04-07 14:30:40] ++ Calling delete_all on 6 instances...
[INFO] [2021-04-07 14:30:40] [14:30:40.380] Removed 6 References
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:40] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:41] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:41] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:41] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:41] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:41] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:41] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:41] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:41] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:41] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:41] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:41] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:41] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:41] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:41] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:41] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:41] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:41] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:41] Starting batch with ID 81423608...
[INFO] [2021-04-07 14:30:41] ## remove_type: Node
[INFO] [2021-04-07 14:30:41] ++ Calling delete_all on 85 instances...
[INFO] [2021-04-07 14:30:41] [14:30:41.369] Removed 85 Nodes
[START] [2021-04-07 14:30:42] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 14:30:42] Creating resource from OpenData
[START] [2021-04-07 14:30:43] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 14:30:43] Parse meta.xml file and create formats with fields
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (media) field header: ThumbnailURL term: http://eol.org/schema/media/thumbnailURL
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (media) field header: DerivedFrom term: http://rs.tdwg.org/ac/terms/derivedFrom
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (media) field header: Modified term: http://purl.org/dc/terms/modified
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (media) field header: Rating term: http://ns.adobe.com/xap/1.0/Rating
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (media) field header: Audience term: http://purl.org/dc/terms/audience
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (media) field header: Rights term: http://purl.org/dc/terms/rights
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (media) field header: Publisher term: http://purl.org/dc/terms/publisher
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (media) field header: Contributor term: http://purl.org/dc/terms/contributor
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (media) field header: Creator term: http://purl.org/dc/terms/creator
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (media) field header: GenericLocation term: http://purl.org/dc/terms/spatial
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (media) field header: Latitude term: http://www.w3.org/2003/01/geo/wgs84_pos#lat
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (media) field header: Longitude term: http://www.w3.org/2003/01/geo/wgs84_pos#long
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (media) field header: Altitude term: http://www.w3.org/2003/01/geo/wgs84_pos#alt
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (occurrences) field header: Institution Code term: http://rs.tdwg.org/dwc/terms/institutionCode
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (occurrences) field header: Collection Code term: http://rs.tdwg.org/dwc/terms/collectionCode
[WARN] [2021-04-07 14:30:57] (common) IGNORED  (occurrences) field header: Catalog Number term: http://rs.tdwg.org/dwc/terms/catalogNumber
[WARN] [2021-04-07 14:30:57] SKIPPING http://rs.tdwg.org/dwc/terms/Event config (events.txt)...
[STOP] [2021-04-07 14:30:57] Parse meta.xml file and create formats with fields
[STOP] [2021-04-07 14:30:57] Creating resource from OpenData
[START] [2021-04-07 14:30:57] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 14:30:57] create_harvest_instance
[INFO] [2021-04-07 14:30:57] Created harvest instance #3677
[STOP] [2021-04-07 14:30:57] create_harvest_instance
[START] [2021-04-07 14:30:57] fetch_files
[STOP] [2021-04-07 14:30:57] fetch_files
[START] [2021-04-07 14:30:57] validate_each_file
[INFO] [2021-04-07 14:30:57] Looping over 8 formats...
[INFO] [2021-04-07 14:30:57] ...agents (/app/public/data/lovegrove/agents.txt)
[INFO] [2021-04-07 14:30:57] Valid: /app/public/converted_csv/lovegrove_agents_3677.csv (0 lines)
[INFO] [2021-04-07 14:30:57] ...refs (/app/public/data/lovegrove/references.txt)
[INFO] [2021-04-07 14:30:57] Valid: /app/public/converted_csv/lovegrove_refs_3677.csv (6 lines)
[INFO] [2021-04-07 14:30:57] ...nodes (/app/public/data/lovegrove/taxa.txt)
[INFO] [2021-04-07 14:30:57] Valid: /app/public/converted_csv/lovegrove_nodes_3677.csv (83 lines)
[INFO] [2021-04-07 14:30:57] ...media (/app/public/data/lovegrove/media.txt)
[INFO] [2021-04-07 14:30:57] Valid: /app/public/converted_csv/lovegrove_media_3677.csv (0 lines)
[INFO] [2021-04-07 14:30:57] ...vernaculars (/app/public/data/lovegrove/common names.txt)
[INFO] [2021-04-07 14:30:57] Valid: /app/public/converted_csv/lovegrove_vernaculars_3677.csv (0 lines)
[INFO] [2021-04-07 14:30:57] ...occurrences (/app/public/data/lovegrove/occurrences.txt)
[INFO] [2021-04-07 14:30:57] Valid: /app/public/converted_csv/lovegrove_occurrences_3677.csv (83 lines)
[INFO] [2021-04-07 14:30:57] ...assocs (/app/public/data/lovegrove/associations.txt)
[INFO] [2021-04-07 14:30:57] Valid: /app/public/converted_csv/lovegrove_assocs_3677.csv (0 lines)
[INFO] [2021-04-07 14:30:57] ...measurements (/app/public/data/lovegrove/measurementorfact.txt)
[INFO] [2021-04-07 14:30:57] Valid: /app/public/converted_csv/lovegrove_measurements_3677.csv (810 lines)
[STOP] [2021-04-07 14:30:57] validate_each_file
[START] [2021-04-07 14:30:57] convert_to_csv
[INFO] [2021-04-07 14:30:57] Looping over 8 formats...
[INFO] [2021-04-07 14:30:57] ...agents (/app/public/data/lovegrove/agents.txt)
[CMD] [2021-04-07 14:30:57] /usr/bin/sort /app/public/converted_csv/lovegrove_agents_3677.csv > /app/public/converted_csv/lovegrove_agents_3677.csv_sorted
[INFO] [2021-04-07 14:30:58] Converted: /app/public/converted_csv/lovegrove_agents_3677.csv (0 lines)
[INFO] [2021-04-07 14:30:58] ...refs (/app/public/data/lovegrove/references.txt)
[CMD] [2021-04-07 14:30:58] /usr/bin/sort /app/public/converted_csv/lovegrove_refs_3677.csv > /app/public/converted_csv/lovegrove_refs_3677.csv_sorted
[INFO] [2021-04-07 14:30:58] Converted: /app/public/converted_csv/lovegrove_refs_3677.csv (6 lines)
[INFO] [2021-04-07 14:30:58] ...nodes (/app/public/data/lovegrove/taxa.txt)
[CMD] [2021-04-07 14:30:58] /usr/bin/sort /app/public/converted_csv/lovegrove_nodes_3677.csv > /app/public/converted_csv/lovegrove_nodes_3677.csv_sorted
[INFO] [2021-04-07 14:30:59] Converted: /app/public/converted_csv/lovegrove_nodes_3677.csv (83 lines)
[INFO] [2021-04-07 14:30:59] ...media (/app/public/data/lovegrove/media.txt)
[CMD] [2021-04-07 14:30:59] /usr/bin/sort /app/public/converted_csv/lovegrove_media_3677.csv > /app/public/converted_csv/lovegrove_media_3677.csv_sorted
[INFO] [2021-04-07 14:30:59] Converted: /app/public/converted_csv/lovegrove_media_3677.csv (0 lines)
[INFO] [2021-04-07 14:30:59] ...vernaculars (/app/public/data/lovegrove/common names.txt)
[CMD] [2021-04-07 14:30:59] /usr/bin/sort /app/public/converted_csv/lovegrove_vernaculars_3677.csv > /app/public/converted_csv/lovegrove_vernaculars_3677.csv_sorted
[INFO] [2021-04-07 14:31:00] Converted: /app/public/converted_csv/lovegrove_vernaculars_3677.csv (0 lines)
[INFO] [2021-04-07 14:31:00] ...occurrences (/app/public/data/lovegrove/occurrences.txt)
[CMD] [2021-04-07 14:31:00] /usr/bin/sort /app/public/converted_csv/lovegrove_occurrences_3677.csv > /app/public/converted_csv/lovegrove_occurrences_3677.csv_sorted
[INFO] [2021-04-07 14:31:01] Converted: /app/public/converted_csv/lovegrove_occurrences_3677.csv (83 lines)
[INFO] [2021-04-07 14:31:01] ...assocs (/app/public/data/lovegrove/associations.txt)
[CMD] [2021-04-07 14:31:01] /usr/bin/sort /app/public/converted_csv/lovegrove_assocs_3677.csv > /app/public/converted_csv/lovegrove_assocs_3677.csv_sorted
[INFO] [2021-04-07 14:31:01] Converted: /app/public/converted_csv/lovegrove_assocs_3677.csv (0 lines)
[INFO] [2021-04-07 14:31:01] ...measurements (/app/public/data/lovegrove/measurementorfact.txt)
[CMD] [2021-04-07 14:31:01] /usr/bin/sort /app/public/converted_csv/lovegrove_measurements_3677.csv > /app/public/converted_csv/lovegrove_measurements_3677.csv_sorted
[INFO] [2021-04-07 14:31:02] Converted: /app/public/converted_csv/lovegrove_measurements_3677.csv (810 lines)
[STOP] [2021-04-07 14:31:02] convert_to_csv
[START] [2021-04-07 14:31:02] calculate_delta
[INFO] [2021-04-07 14:31:02] Looping over 8 formats...
[INFO] [2021-04-07 14:31:02] ...agents (/app/public/data/lovegrove/agents.txt)
[CMD] [2021-04-07 14:31:02] echo "0a" > /app/public/diff/lovegrove_agents_3677.diff
[CMD] [2021-04-07 14:31:02] tail -n +1 /app/public/converted_csv/lovegrove_agents_3677.csv >> /app/public/diff/lovegrove_agents_3677.diff
[CMD] [2021-04-07 14:31:03] echo "." >> /app/public/diff/lovegrove_agents_3677.diff
[INFO] [2021-04-07 14:31:03] Created diff: /app/public/diff/lovegrove_agents_3677.diff (2 lines)
[INFO] [2021-04-07 14:31:03] ...refs (/app/public/data/lovegrove/references.txt)
[CMD] [2021-04-07 14:31:03] echo "0a" > /app/public/diff/lovegrove_refs_3677.diff
[CMD] [2021-04-07 14:31:04] tail -n +1 /app/public/converted_csv/lovegrove_refs_3677.csv >> /app/public/diff/lovegrove_refs_3677.diff
[CMD] [2021-04-07 14:31:04] echo "." >> /app/public/diff/lovegrove_refs_3677.diff
[INFO] [2021-04-07 14:31:05] Created diff: /app/public/diff/lovegrove_refs_3677.diff (8 lines)
[INFO] [2021-04-07 14:31:05] ...nodes (/app/public/data/lovegrove/taxa.txt)
[CMD] [2021-04-07 14:31:05] echo "0a" > /app/public/diff/lovegrove_nodes_3677.diff
[CMD] [2021-04-07 14:31:05] tail -n +1 /app/public/converted_csv/lovegrove_nodes_3677.csv >> /app/public/diff/lovegrove_nodes_3677.diff
[CMD] [2021-04-07 14:31:06] echo "." >> /app/public/diff/lovegrove_nodes_3677.diff
[INFO] [2021-04-07 14:31:07] Created diff: /app/public/diff/lovegrove_nodes_3677.diff (85 lines)
[INFO] [2021-04-07 14:31:07] ...media (/app/public/data/lovegrove/media.txt)
[CMD] [2021-04-07 14:31:07] echo "0a" > /app/public/diff/lovegrove_media_3677.diff
[CMD] [2021-04-07 14:31:07] tail -n +1 /app/public/converted_csv/lovegrove_media_3677.csv >> /app/public/diff/lovegrove_media_3677.diff
[CMD] [2021-04-07 14:31:08] echo "." >> /app/public/diff/lovegrove_media_3677.diff
[INFO] [2021-04-07 14:31:08] Created diff: /app/public/diff/lovegrove_media_3677.diff (2 lines)
[INFO] [2021-04-07 14:31:08] ...vernaculars (/app/public/data/lovegrove/common names.txt)
[CMD] [2021-04-07 14:31:08] echo "0a" > /app/public/diff/lovegrove_vernaculars_3677.diff
[CMD] [2021-04-07 14:31:09] tail -n +1 /app/public/converted_csv/lovegrove_vernaculars_3677.csv >> /app/public/diff/lovegrove_vernaculars_3677.diff
[CMD] [2021-04-07 14:31:09] echo "." >> /app/public/diff/lovegrove_vernaculars_3677.diff
[INFO] [2021-04-07 14:31:10] Created diff: /app/public/diff/lovegrove_vernaculars_3677.diff (2 lines)
[INFO] [2021-04-07 14:31:10] ...occurrences (/app/public/data/lovegrove/occurrences.txt)
[CMD] [2021-04-07 14:31:10] echo "0a" > /app/public/diff/lovegrove_occurrences_3677.diff
[CMD] [2021-04-07 14:31:10] tail -n +1 /app/public/converted_csv/lovegrove_occurrences_3677.csv >> /app/public/diff/lovegrove_occurrences_3677.diff
[CMD] [2021-04-07 14:31:11] echo "." >> /app/public/diff/lovegrove_occurrences_3677.diff
[INFO] [2021-04-07 14:31:12] Created diff: /app/public/diff/lovegrove_occurrences_3677.diff (85 lines)
[INFO] [2021-04-07 14:31:12] ...assocs (/app/public/data/lovegrove/associations.txt)
[CMD] [2021-04-07 14:31:12] echo "0a" > /app/public/diff/lovegrove_assocs_3677.diff
[CMD] [2021-04-07 14:31:12] tail -n +1 /app/public/converted_csv/lovegrove_assocs_3677.csv >> /app/public/diff/lovegrove_assocs_3677.diff
[CMD] [2021-04-07 14:31:13] echo "." >> /app/public/diff/lovegrove_assocs_3677.diff
[INFO] [2021-04-07 14:31:13] Created diff: /app/public/diff/lovegrove_assocs_3677.diff (2 lines)
[INFO] [2021-04-07 14:31:13] ...measurements (/app/public/data/lovegrove/measurementorfact.txt)
[CMD] [2021-04-07 14:31:13] echo "0a" > /app/public/diff/lovegrove_measurements_3677.diff
[CMD] [2021-04-07 14:31:14] tail -n +1 /app/public/converted_csv/lovegrove_measurements_3677.csv >> /app/public/diff/lovegrove_measurements_3677.diff
[CMD] [2021-04-07 14:31:14] echo "." >> /app/public/diff/lovegrove_measurements_3677.diff
[INFO] [2021-04-07 14:31:15] Created diff: /app/public/diff/lovegrove_measurements_3677.diff (812 lines)
[STOP] [2021-04-07 14:31:15] calculate_delta
[START] [2021-04-07 14:31:15] parse_diff_and_store
[INFO] [2021-04-07 14:31:15] Handling diff: /app/public/diff/lovegrove_agents_3677.diff (2 lines)
[INFO] [2021-04-07 14:31:16] Loading agents diff file into memory (2 /app/public/diff/lovegrove_agents_3677.diff lines)...
[INFO] [2021-04-07 14:31:16] Handling diff: /app/public/diff/lovegrove_refs_3677.diff (8 lines)
[INFO] [2021-04-07 14:31:17] Loading refs diff file into memory (8 /app/public/diff/lovegrove_refs_3677.diff lines)...
[INFO] [2021-04-07 14:31:17] Handling diff: /app/public/diff/lovegrove_nodes_3677.diff (85 lines)
[INFO] [2021-04-07 14:31:18] Loading nodes diff file into memory (85 /app/public/diff/lovegrove_nodes_3677.diff lines)...
[INFO] [2021-04-07 14:31:18] Handling diff: /app/public/diff/lovegrove_media_3677.diff (2 lines)
[INFO] [2021-04-07 14:31:19] Loading media diff file into memory (2 /app/public/diff/lovegrove_media_3677.diff lines)...
[INFO] [2021-04-07 14:31:19] Handling diff: /app/public/diff/lovegrove_vernaculars_3677.diff (2 lines)
[INFO] [2021-04-07 14:31:20] Loading vernaculars diff file into memory (2 /app/public/diff/lovegrove_vernaculars_3677.diff lines)...
[INFO] [2021-04-07 14:31:20] Handling diff: /app/public/diff/lovegrove_occurrences_3677.diff (85 lines)
[INFO] [2021-04-07 14:31:21] Loading occurrences diff file into memory (85 /app/public/diff/lovegrove_occurrences_3677.diff lines)...
[INFO] [2021-04-07 14:31:22] Handling diff: /app/public/diff/lovegrove_assocs_3677.diff (2 lines)
[INFO] [2021-04-07 14:31:22] Loading assocs diff file into memory (2 /app/public/diff/lovegrove_assocs_3677.diff lines)...
[INFO] [2021-04-07 14:31:23] Handling diff: /app/public/diff/lovegrove_measurements_3677.diff (812 lines)
[INFO] [2021-04-07 14:31:23] Loading measurements diff file into memory (812 /app/public/diff/lovegrove_measurements_3677.diff lines)...
[INFO] [2021-04-07 14:31:24] Storing 6 References
[INFO] [2021-04-07 14:31:24] Processing group of 6 in 1 groups of 1000
[INFO] [2021-04-07 14:31:24] Average Time: 0.0
[INFO] [2021-04-07 14:31:24] Total Time: 1s
[INFO] [2021-04-07 14:31:24] Storing 85 ScientificNames
[INFO] [2021-04-07 14:31:24] Processing group of 85 in 1 groups of 1000
[INFO] [2021-04-07 14:31:24] Average Time: 0.03
[INFO] [2021-04-07 14:31:24] Total Time: 1s
[INFO] [2021-04-07 14:31:24] Storing 85 Nodes
[INFO] [2021-04-07 14:31:24] Processing group of 85 in 1 groups of 1000
[INFO] [2021-04-07 14:31:24] Average Time: 0.02
[INFO] [2021-04-07 14:31:24] Total Time: 1s
[INFO] [2021-04-07 14:31:24] Storing 83 Occurrences
[INFO] [2021-04-07 14:31:24] Processing group of 83 in 1 groups of 1000
[INFO] [2021-04-07 14:31:24] Average Time: 0.01
[INFO] [2021-04-07 14:31:24] Total Time: 1s
[INFO] [2021-04-07 14:31:24] Storing 644 Traits
[INFO] [2021-04-07 14:31:24] Processing group of 644 in 1 groups of 1000
[INFO] [2021-04-07 14:31:24] Average Time: 0.2
[INFO] [2021-04-07 14:31:24] Total Time: 1s
[INFO] [2021-04-07 14:31:24] Storing 324 TraitsReferences
[INFO] [2021-04-07 14:31:24] Processing group of 324 in 1 groups of 1000
[INFO] [2021-04-07 14:31:24] Average Time: 0.02
[INFO] [2021-04-07 14:31:24] Total Time: 1s
[INFO] [2021-04-07 14:31:24] Storing 316 MetaTraits
[INFO] [2021-04-07 14:31:24] Processing group of 316 in 1 groups of 1000
[INFO] [2021-04-07 14:31:24] Average Time: 0.04
[INFO] [2021-04-07 14:31:24] Total Time: 1s
[INFO] [2021-04-07 14:31:24] Storing 166 OccurrenceMetadata
[INFO] [2021-04-07 14:31:24] Processing group of 166 in 1 groups of 1000
[INFO] [2021-04-07 14:31:24] Average Time: 0.02
[INFO] [2021-04-07 14:31:24] Total Time: 1s
[STOP] [2021-04-07 14:31:24] parse_diff_and_store
[START] [2021-04-07 14:31:24] resolve_keys
[INFO] [2021-04-07 14:31:31] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-07 14:31:31] traits to occurrences...
[INFO] [2021-04-07 14:31:31] traits to nodes (through occurrences)...
[INFO] [2021-04-07 14:31:31] Traits to sex term...
[INFO] [2021-04-07 14:31:31] Traits to lifestage term...
[INFO] [2021-04-07 14:31:31] MetaTraits to traits...
[INFO] [2021-04-07 14:31:31] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-07 14:31:31] Assocs to occurrences...
[INFO] [2021-04-07 14:31:31] Assocs to nodes...
[INFO] [2021-04-07 14:31:31] Assoc to sex term...
[INFO] [2021-04-07 14:31:31] Assoc to lifestage term...
[INFO] [2021-04-07 14:31:31] MetaAssoc to assocs...
[STOP] [2021-04-07 14:31:31] resolve_keys
[START] [2021-04-07 14:31:31] hold_for_later_1
[STOP] [2021-04-07 14:31:31] hold_for_later_1
[START] [2021-04-07 14:31:31] hold_for_later_2
[STOP] [2021-04-07 14:31:31] hold_for_later_2
[START] [2021-04-07 14:31:31] resolve_missing_parents
[STOP] [2021-04-07 14:31:31] resolve_missing_parents
[START] [2021-04-07 14:31:31] rebuild_nodes
[START] [2021-04-07 14:31:31] Flattener#flatten
[START] [2021-04-07 14:31:31] Flattener#study_resource
[START] [2021-04-07 14:31:31] Flattener#build_ancestry
[STOP] [2021-04-07 14:31:31] Flattener#build_ancestry
[INFO] [2021-04-07 14:31:31] 85 ancestry keys
[START] [2021-04-07 14:31:31] build_node_ancestors
[INFO] [2021-04-07 14:31:31] old ancestors deleted.
[STOP] [2021-04-07 14:31:31] build_node_ancestors
[START] [2021-04-07 14:31:31] Flattener#propagate_ancestor_ids
[STOP] [2021-04-07 14:31:31] Flattener#propagate_ancestor_ids
[STOP] [2021-04-07 14:31:31] Flattener#flatten
[STOP] [2021-04-07 14:31:31] rebuild_nodes
[START] [2021-04-07 14:31:31] resolve_missing_media_owners
[STOP] [2021-04-07 14:31:31] resolve_missing_media_owners
[START] [2021-04-07 14:31:31] sanitize_media_verbatims
[STOP] [2021-04-07 14:31:31] sanitize_media_verbatims
[START] [2021-04-07 14:31:31] queue_downloads
[STOP] [2021-04-07 14:31:31] queue_downloads
[START] [2021-04-07 14:31:31] parse_names
[WARN] [2021-04-07 14:31:31] I see 85 names which still need to be parsed.
[STOP] [2021-04-07 14:31:32] parse_names
[START] [2021-04-07 14:31:32] denormalize_canonical_names_to_nodes
[STOP] [2021-04-07 14:31:32] denormalize_canonical_names_to_nodes
[START] [2021-04-07 14:31:32] match_nodes
[START] [2021-04-07 14:31:32] map_all_nodes_to_pages
[STOP] [2021-04-07 14:31:33] map_all_nodes_to_pages
[INFO] [2021-04-07 14:31:33] Unmatched nodes (1 of 85): Crociduridae (#92629160)
[START] [2021-04-07 14:31:33] update_nodes
[STOP] [2021-04-07 14:31:33] update_nodes
[STOP] [2021-04-07 14:31:33] match_nodes
[START] [2021-04-07 14:31:33] reindex_search
[STOP] [2021-04-07 14:31:33] reindex_search
[START] [2021-04-07 14:31:33] normalize_units
[STOP] [2021-04-07 14:31:33] normalize_units
[START] [2021-04-07 14:31:33] calculate_statistics
[STOP] [2021-04-07 14:31:33] calculate_statistics
[START] [2021-04-07 14:31:33] complete_harvest_instance
[START] [2021-04-07 14:31:33] overall_tsv_creation
[INFO] [2021-04-07 14:31:33] Processing group of 85 in 1 batches of 10000
[INFO] [2021-04-07 14:32:10] 158 Traits (unfiltered)...
[INFO] [2021-04-07 14:32:44] 158 Traits (filtered)...
[INFO] [2021-04-07 14:32:44] 0 Associations (filtered)...
[INFO] [2021-04-07 14:32:44] 794 metadata added.
[INFO] [2021-04-07 14:32:44] 0 metadata added.
[INFO] [2021-04-07 14:33:10] Average Time: 72.52
[INFO] [2021-04-07 14:33:10] Total Time: 1m38s
[STOP] [2021-04-07 14:33:10] overall_tsv_creation
[INFO] [2021-04-07 14:33:10] Done. Check your files:
[INFO] [2021-04-07 14:33:11] (85 lines) /app/public/data/lovegrove/publish_nodes.tsv
[INFO] [2021-04-07 14:33:11] (5 lines) /app/public/data/lovegrove/publish_node_ancestors.tsv
[INFO] [2021-04-07 14:33:12] (85 lines) /app/public/data/lovegrove/publish_scientific_names.tsv
[INFO] [2021-04-07 14:33:13] (159 lines) /app/public/data/lovegrove/publish_traits.tsv
[INFO] [2021-04-07 14:33:13] (795 lines) /app/public/data/lovegrove/publish_metadata.tsv
[STOP] [2021-04-07 14:33:13] complete_harvest_instance
[START] [2021-04-07 14:33:13] completed
[STOP] [2021-04-07 14:33:13] completed
[STOP] [2021-04-07 14:33:13] logged process, took 136.63

Latest Process

07 Apr 14:30 EXITED currently processing 0