Harvest for Alaskan Arthropods Created 06 Apr 13:04

Stage: completed
Fetched: 06 Apr 13:04
Validated: 06 Apr 13:04
Deltas Created 06 Apr 13:04
Units Normalized: 06 Apr 13:08
Ancestry Built: 06 Apr 13:06
Nodes Matched: 06 Apr 13:08
Names Parsed: 06 Apr 13:06
New Models Stored: 06 Apr 13:05
Indexed: 06 Apr 13:08
Completed: 06 Apr 13:13
Time to Harvest: less than a minute

Harvesting Log (most recent first)

# Logfile created on 2020-04-09 08:43:31 -0400 by logger.rb/v1.4.2
[INFO] [2020-04-09 08:43:31] ## HARVEST: type = -harvest
[START] [2020-04-09 08:43:32] logged process
[START] [2020-04-09 08:43:32] create_harvest_instance
[STOP] [2020-04-09 08:43:41] create_harvest_instance
[START] [2020-04-09 08:43:41] fetch_files
[STOP] [2020-04-09 08:43:41] fetch_files
[START] [2020-04-09 08:43:41] validate_each_file
[STOP] [2020-04-09 08:43:43] validate_each_file
[START] [2020-04-09 08:43:43] convert_to_csv
[CMD] [2020-04-09 08:43:43] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_agents_20699.csv > /app/public/converted_csv/Alaskan_Arthropo_agents_20699.csv_sorted
[CMD] [2020-04-09 08:43:43] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_refs_20700.csv > /app/public/converted_csv/Alaskan_Arthropo_refs_20700.csv_sorted
[CMD] [2020-04-09 08:43:43] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_nodes_20701.csv > /app/public/converted_csv/Alaskan_Arthropo_nodes_20701.csv_sorted
[CMD] [2020-04-09 08:43:43] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_media_20702.csv > /app/public/converted_csv/Alaskan_Arthropo_media_20702.csv_sorted
[CMD] [2020-04-09 08:43:43] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_vernaculars_20703.csv > /app/public/converted_csv/Alaskan_Arthropo_vernaculars_20703.csv_sorted
[CMD] [2020-04-09 08:43:43] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_occurrences_20704.csv > /app/public/converted_csv/Alaskan_Arthropo_occurrences_20704.csv_sorted
[CMD] [2020-04-09 08:43:43] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_assocs_20705.csv > /app/public/converted_csv/Alaskan_Arthropo_assocs_20705.csv_sorted
[CMD] [2020-04-09 08:43:43] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_measurements_20706.csv > /app/public/converted_csv/Alaskan_Arthropo_measurements_20706.csv_sorted
[STOP] [2020-04-09 08:43:43] convert_to_csv
[START] [2020-04-09 08:43:43] calculate_delta
[CMD] [2020-04-09 08:43:43] echo "0a" > /app/public/diff/Alaskan_Arthropo_agents_20699.diff
[CMD] [2020-04-09 08:43:43] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_agents_20699.csv >> /app/public/diff/Alaskan_Arthropo_agents_20699.diff
[CMD] [2020-04-09 08:43:43] echo "." >> /app/public/diff/Alaskan_Arthropo_agents_20699.diff
[CMD] [2020-04-09 08:43:43] echo "0a" > /app/public/diff/Alaskan_Arthropo_refs_20700.diff
[CMD] [2020-04-09 08:43:43] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_refs_20700.csv >> /app/public/diff/Alaskan_Arthropo_refs_20700.diff
[CMD] [2020-04-09 08:43:43] echo "." >> /app/public/diff/Alaskan_Arthropo_refs_20700.diff
[CMD] [2020-04-09 08:43:43] echo "0a" > /app/public/diff/Alaskan_Arthropo_nodes_20701.diff
[CMD] [2020-04-09 08:43:43] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_nodes_20701.csv >> /app/public/diff/Alaskan_Arthropo_nodes_20701.diff
[CMD] [2020-04-09 08:43:43] echo "." >> /app/public/diff/Alaskan_Arthropo_nodes_20701.diff
[CMD] [2020-04-09 08:43:43] echo "0a" > /app/public/diff/Alaskan_Arthropo_media_20702.diff
[CMD] [2020-04-09 08:43:43] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_media_20702.csv >> /app/public/diff/Alaskan_Arthropo_media_20702.diff
[CMD] [2020-04-09 08:43:43] echo "." >> /app/public/diff/Alaskan_Arthropo_media_20702.diff
[CMD] [2020-04-09 08:43:43] echo "0a" > /app/public/diff/Alaskan_Arthropo_vernaculars_20703.diff
[CMD] [2020-04-09 08:43:43] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_vernaculars_20703.csv >> /app/public/diff/Alaskan_Arthropo_vernaculars_20703.diff
[CMD] [2020-04-09 08:43:43] echo "." >> /app/public/diff/Alaskan_Arthropo_vernaculars_20703.diff
[CMD] [2020-04-09 08:43:43] echo "0a" > /app/public/diff/Alaskan_Arthropo_occurrences_20704.diff
[CMD] [2020-04-09 08:43:43] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_occurrences_20704.csv >> /app/public/diff/Alaskan_Arthropo_occurrences_20704.diff
[CMD] [2020-04-09 08:43:43] echo "." >> /app/public/diff/Alaskan_Arthropo_occurrences_20704.diff
[CMD] [2020-04-09 08:43:43] echo "0a" > /app/public/diff/Alaskan_Arthropo_assocs_20705.diff
[CMD] [2020-04-09 08:43:43] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_assocs_20705.csv >> /app/public/diff/Alaskan_Arthropo_assocs_20705.diff
[CMD] [2020-04-09 08:43:43] echo "." >> /app/public/diff/Alaskan_Arthropo_assocs_20705.diff
[CMD] [2020-04-09 08:43:43] echo "0a" > /app/public/diff/Alaskan_Arthropo_measurements_20706.diff
[CMD] [2020-04-09 08:43:43] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_measurements_20706.csv >> /app/public/diff/Alaskan_Arthropo_measurements_20706.diff
[CMD] [2020-04-09 08:43:43] echo "." >> /app/public/diff/Alaskan_Arthropo_measurements_20706.diff
[STOP] [2020-04-09 08:43:43] calculate_delta
[START] [2020-04-09 08:43:43] parse_diff_and_store
[INFO] [2020-04-09 08:43:43] Loading agents diff file into memory (true lines)...
[INFO] [2020-04-09 08:43:43] Loading refs diff file into memory (true lines)...
[INFO] [2020-04-09 08:43:43] Loading nodes diff file into memory (true lines)...
[INFO] [2020-04-09 08:43:46] Loading media diff file into memory (true lines)...
[INFO] [2020-04-09 08:43:46] Loading vernaculars diff file into memory (true lines)...
[INFO] [2020-04-09 08:43:46] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-04-09 08:43:47] Loading assocs diff file into memory (true lines)...
[INFO] [2020-04-09 08:43:47] Loading measurements diff file into memory (true lines)...
[INFO] [2020-04-09 08:44:24] Storing 443 References
[INFO] [2020-04-09 08:44:24] Processing group of 443 in 1 groups of 1000
[INFO] [2020-04-09 08:44:25] Average Time: 0.08
[INFO] [2020-04-09 08:44:25] Total Time: 1s
[INFO] [2020-04-09 08:44:25] Storing 11116 ScientificNames
[INFO] [2020-04-09 08:44:25] Processing group of 11116 in 12 groups of 1000
[INFO] [2020-04-09 08:44:28] Average Time: 0.296
[INFO] [2020-04-09 08:44:28] Total Time: 4s
[INFO] [2020-04-09 08:44:28] last 3 / first 3: 0.67
[INFO] [2020-04-09 08:44:28] Std.Dev: 0.08366600265340755; Max: 0.44
[INFO] [2020-04-09 08:44:28] Storing 11116 Nodes
[INFO] [2020-04-09 08:44:28] Processing group of 11116 in 12 groups of 1000
[INFO] [2020-04-09 08:44:32] Average Time: 0.282
[INFO] [2020-04-09 08:44:32] Total Time: 4s
[INFO] [2020-04-09 08:44:32] last 3 / first 3: 0.51
[INFO] [2020-04-09 08:44:32] Std.Dev: 0.09486832980505137; Max: 0.43
[INFO] [2020-04-09 08:44:32] Storing 7682 Occurrences
[INFO] [2020-04-09 08:44:32] Processing group of 7682 in 8 groups of 1000
[INFO] [2020-04-09 08:44:33] Average Time: 0.11
[INFO] [2020-04-09 08:44:33] Total Time: 1s
[INFO] [2020-04-09 08:44:33] last 3 / first 3: 0.73
[INFO] [2020-04-09 08:44:33] Std.Dev: 0.0; Max: 0.14
[INFO] [2020-04-09 08:44:33] Storing 8258 TraitsReferences
[INFO] [2020-04-09 08:44:33] Processing group of 8258 in 9 groups of 1000
[INFO] [2020-04-09 08:44:33] Average Time: 0.072
[INFO] [2020-04-09 08:44:33] Total Time: 1s
[INFO] [2020-04-09 08:44:33] last 3 / first 3: 0.48
[INFO] [2020-04-09 08:44:33] Std.Dev: 0.044721359549995794; Max: 0.17
[INFO] [2020-04-09 08:44:33] Storing 7880 Traits
[INFO] [2020-04-09 08:44:33] Processing group of 7880 in 8 groups of 1000
[INFO] [2020-04-09 08:44:36] Average Time: 0.336
[INFO] [2020-04-09 08:44:36] Total Time: 3s
[INFO] [2020-04-09 08:44:36] last 3 / first 3: 0.67
[INFO] [2020-04-09 08:44:36] Std.Dev: 0.07745966692414834; Max: 0.44
[INFO] [2020-04-09 08:44:36] Storing 14991 MetaTraits
[INFO] [2020-04-09 08:44:36] Processing group of 14991 in 15 groups of 1000
[INFO] [2020-04-09 08:44:37] Average Time: 0.097
[INFO] [2020-04-09 08:44:37] Total Time: 2s
[INFO] [2020-04-09 08:44:37] last 3 / first 3: 0.79
[INFO] [2020-04-09 08:44:37] Std.Dev: 0.0; Max: 0.16
[STOP] [2020-04-09 08:44:37] parse_diff_and_store
[START] [2020-04-09 08:44:37] resolve_keys
[INFO] [2020-04-09 08:44:55] Occurrences to nodes (through scientific_names)...
[INFO] [2020-04-09 08:44:55] traits to occurrences...
[INFO] [2020-04-09 08:44:56] traits to nodes (through occurrences)...
[INFO] [2020-04-09 08:44:56] Traits to sex term...
[INFO] [2020-04-09 08:44:56] Traits to lifestage term...
[INFO] [2020-04-09 08:44:56] MetaTraits to traits...
[INFO] [2020-04-09 08:44:57] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-04-09 08:44:58] Assocs to occurrences...
[INFO] [2020-04-09 08:44:58] Assocs to nodes...
[INFO] [2020-04-09 08:44:58] Assoc to sex term...
[INFO] [2020-04-09 08:44:58] Assoc to lifestage term...
[STOP] [2020-04-09 08:44:58] resolve_keys
[START] [2020-04-09 08:44:58] hold_for_later_1
[STOP] [2020-04-09 08:44:58] hold_for_later_1
[START] [2020-04-09 08:44:58] hold_for_later_2
[STOP] [2020-04-09 08:44:58] hold_for_later_2
[START] [2020-04-09 08:44:58] resolve_missing_parents
[STOP] [2020-04-09 08:44:59] resolve_missing_parents
[START] [2020-04-09 08:44:59] rebuild_nodes
[START] [2020-04-09 08:44:59] Flattener#flatten
[START] [2020-04-09 08:44:59] Flattener#study_resource
[START] [2020-04-09 08:44:59] Flattener#build_ancestry
[STOP] [2020-04-09 08:45:00] Flattener#build_ancestry
[INFO] [2020-04-09 08:45:00] 11116 ancestry keys
[START] [2020-04-09 08:45:00] build_node_ancestors
[INFO] [2020-04-09 08:45:00] old ancestors deleted.
[STOP] [2020-04-09 08:45:02] build_node_ancestors
[START] [2020-04-09 08:45:05] Flattener#propagate_ancestor_ids
[STOP] [2020-04-09 08:45:06] Flattener#propagate_ancestor_ids
[STOP] [2020-04-09 08:45:06] Flattener#flatten
[STOP] [2020-04-09 08:45:06] rebuild_nodes
[START] [2020-04-09 08:45:06] resolve_missing_media_owners
[STOP] [2020-04-09 08:45:06] resolve_missing_media_owners
[START] [2020-04-09 08:45:06] sanitize_media_verbatims
[STOP] [2020-04-09 08:45:06] sanitize_media_verbatims
[START] [2020-04-09 08:45:06] queue_downloads
[STOP] [2020-04-09 08:45:06] queue_downloads
[START] [2020-04-09 08:45:06] parse_names
[WARN] [2020-04-09 08:45:06] I see 11116 names which still need to be parsed.
[WARN] [2020-04-09 08:45:15] I see 54 names which still need to be parsed.
[STOP] [2020-04-09 08:45:16] parse_names
[START] [2020-04-09 08:45:16] denormalize_canonical_names_to_nodes
[STOP] [2020-04-09 08:45:16] denormalize_canonical_names_to_nodes
[START] [2020-04-09 08:45:16] match_nodes
[START] [2020-04-09 08:45:16] map_all_nodes_to_pages
[STOP] [2020-04-09 08:53:25] map_all_nodes_to_pages
[INFO] [2020-04-09 08:53:25] 1063 Unmatched nodes (of 11116)! That's too many to output. First 10: Acronicta lupini (#68219881); Acronicta vulpina (#68219883); Actebia balanitis (#68219909); Alypia langtoni (#68220299); Anarta nigrolunata (#68220448); Anarta trifolii (#68220449); Androloma (#68220500); Androloma maccullochii (#68220501); Apamea amputatrix (#68220645); Apamea cogitata (#68220647)
[START] [2020-04-09 08:53:25] update_nodes
[STOP] [2020-04-09 08:53:29] update_nodes
[STOP] [2020-04-09 08:53:29] match_nodes
[START] [2020-04-09 08:53:29] reindex_search
[STOP] [2020-04-09 08:53:43] reindex_search
[START] [2020-04-09 08:53:43] normalize_units
[STOP] [2020-04-09 08:53:49] normalize_units
[START] [2020-04-09 08:53:49] calculate_statistics
[STOP] [2020-04-09 08:53:50] calculate_statistics
[START] [2020-04-09 08:53:50] complete_harvest_instance
[START] [2020-04-09 08:53:50] overall_tsv_creation
[INFO] [2020-04-09 08:53:50] Processing group of 11116 in 2 batches of 10000
[INFO] [2020-04-09 08:55:13] 7145 Traits (unfiltered)...
[INFO] [2020-04-09 08:55:26] 7145 Traits (filtered)...
[INFO] [2020-04-09 08:55:26] 0 Associations (filtered)...
[INFO] [2020-04-09 08:56:09] 20721 metadata added.
[INFO] [2020-04-09 08:56:09] 0 metadata added.
[INFO] [2020-04-09 08:57:04] 735 Traits (unfiltered)...
[INFO] [2020-04-09 08:57:17] 735 Traits (filtered)...
[INFO] [2020-04-09 08:57:17] 0 Associations (filtered)...
[INFO] [2020-04-09 08:57:54] 1853 metadata added.
[INFO] [2020-04-09 08:57:54] 0 metadata added.
[INFO] [2020-04-09 08:57:54] Average Time: 89.715
[INFO] [2020-04-09 08:57:54] Total Time: 4m5s
[STOP] [2020-04-09 08:57:54] overall_tsv_creation
[INFO] [2020-04-09 08:57:54] Done. Check your files:
[INFO] [2020-04-09 08:57:54] (11063 lines) /app/public/data/Alaskan_Arthropo/publish_nodes.tsv
[INFO] [2020-04-09 08:57:54] (50687 lines) /app/public/data/Alaskan_Arthropo/publish_node_ancestors.tsv
[INFO] [2020-04-09 08:57:54] (11116 lines) /app/public/data/Alaskan_Arthropo/publish_scientific_names.tsv
[INFO] [2020-04-09 08:57:54] (7881 lines) /app/public/data/Alaskan_Arthropo/publish_traits.tsv
[INFO] [2020-04-09 08:57:54] (22575 lines) /app/public/data/Alaskan_Arthropo/publish_metadata.tsv
[STOP] [2020-04-09 08:57:54] complete_harvest_instance
[START] [2020-04-09 08:57:54] completed
[STOP] [2020-04-09 08:57:54] completed
[STOP] [2020-04-09 08:57:54] logged process, took 862.07
[INFO] [2021-04-06 12:01:41] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-06 12:01:43] ## remove_type: ScientificName
[INFO] [2021-04-06 12:01:44] ++ Calling delete_all on 11116 instances...
[INFO] [2021-04-06 12:01:45] [12:01:45.201] Removed 11116 Scientificnames
[INFO] [2021-04-06 12:01:45] ## remove_type: Vernacular
[INFO] [2021-04-06 12:01:45] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:01:45] [12:01:45.203] Removed 0 Vernaculars
[INFO] [2021-04-06 12:01:45] ## remove_type: Article
[INFO] [2021-04-06 12:01:45] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:01:45] [12:01:45.230] Removed 0 Articles
[INFO] [2021-04-06 12:01:45] ## remove_type: Medium
[INFO] [2021-04-06 12:01:45] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:01:45] [12:01:45.231] Removed 0 Media
[INFO] [2021-04-06 12:01:45] ## remove_type: Trait
[INFO] [2021-04-06 12:01:45] ++ Calling delete_all on 7880 instances...
[INFO] [2021-04-06 12:01:46] [12:01:46.714] Removed 7880 Traits
[INFO] [2021-04-06 12:01:46] ## remove_type: MetaTrait
[INFO] [2021-04-06 12:01:46] ++ Calling delete_all on 14991 instances...
[INFO] [2021-04-06 12:01:47] [12:01:47.485] Removed 14991 Metatraits
[INFO] [2021-04-06 12:01:47] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-06 12:01:47] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:01:47] [12:01:47.498] Removed 0 Occurrencemetadata
[INFO] [2021-04-06 12:01:47] ## remove_type: Assoc
[INFO] [2021-04-06 12:01:47] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:01:47] [12:01:47.499] Removed 0 Assocs
[INFO] [2021-04-06 12:01:47] ## remove_type: MetaAssoc
[INFO] [2021-04-06 12:01:47] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:01:47] [12:01:47.501] Removed 0 Metaassocs
[INFO] [2021-04-06 12:01:47] ## remove_type: Identifier
[INFO] [2021-04-06 12:01:47] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:01:47] [12:01:47.548] Removed 0 Identifiers
[INFO] [2021-04-06 12:01:47] ## remove_type: Reference
[INFO] [2021-04-06 12:01:47] ++ Calling delete_all on 443 instances...
[INFO] [2021-04-06 12:01:47] [12:01:47.578] Removed 443 References
[INFO] [2021-04-06 12:01:49] Starting batch with ID 68221764...
[INFO] [2021-04-06 12:01:50] Starting batch with ID 68221772...
[INFO] [2021-04-06 12:01:50] Starting batch with ID 68221772...
[INFO] [2021-04-06 12:01:54] Starting batch with ID 68224408...
[INFO] [2021-04-06 12:01:55] Starting batch with ID 68224408...
[INFO] [2021-04-06 12:01:56] Starting batch with ID 68220405...
[INFO] [2021-04-06 12:01:56] Starting batch with ID 68220405...
[INFO] [2021-04-06 12:01:56] Starting batch with ID 68220405...
[INFO] [2021-04-06 12:01:56] Starting batch with ID 68220405...
[INFO] [2021-04-06 12:01:56] Starting batch with ID 68220405...
[INFO] [2021-04-06 12:01:56] Starting batch with ID 68220405...
[INFO] [2021-04-06 12:01:56] Starting batch with ID 68220405...
[INFO] [2021-04-06 12:01:56] Starting batch with ID 68220405...
[INFO] [2021-04-06 12:01:56] Starting batch with ID 68220405...
[INFO] [2021-04-06 12:01:56] Starting batch with ID 68220405...
[INFO] [2021-04-06 12:01:56] Starting batch with ID 68220405...
[INFO] [2021-04-06 12:01:56] Starting batch with ID 68220405...
[INFO] [2021-04-06 12:01:57] Starting batch with ID 68220405...
[INFO] [2021-04-06 12:01:57] ## remove_type: Node
[INFO] [2021-04-06 12:01:57] ++ Calling delete_all on 11116 instances...
[INFO] [2021-04-06 12:01:57] [12:01:57.724] Removed 11116 Nodes
[START] [2021-04-06 12:02:02] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-06 12:02:02] Creating resource from OpenData
[START] [2021-04-06 12:02:05] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-06 12:02:05] Parse meta.xml file and create formats with fields
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (media) field header: ThumbnailURL term: http://eol.org/schema/media/thumbnailURL
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (media) field header: DerivedFrom term: http://rs.tdwg.org/ac/terms/derivedFrom
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (media) field header: Modified term: http://purl.org/dc/terms/modified
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (media) field header: Rating term: http://ns.adobe.com/xap/1.0/Rating
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (media) field header: Audience term: http://purl.org/dc/terms/audience
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (media) field header: Rights term: http://purl.org/dc/terms/rights
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (media) field header: Publisher term: http://purl.org/dc/terms/publisher
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (media) field header: Contributor term: http://purl.org/dc/terms/contributor
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (media) field header: Creator term: http://purl.org/dc/terms/creator
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (media) field header: GenericLocation term: http://purl.org/dc/terms/spatial
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (media) field header: Latitude term: http://www.w3.org/2003/01/geo/wgs84_pos#lat
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (media) field header: Longitude term: http://www.w3.org/2003/01/geo/wgs84_pos#long
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (media) field header: Altitude term: http://www.w3.org/2003/01/geo/wgs84_pos#alt
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (occurrences) field header: Institution Code term: http://rs.tdwg.org/dwc/terms/institutionCode
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (occurrences) field header: Collection Code term: http://rs.tdwg.org/dwc/terms/collectionCode
[WARN] [2021-04-06 12:02:42] (common) IGNORED  (occurrences) field header: Catalog Number term: http://rs.tdwg.org/dwc/terms/catalogNumber
[WARN] [2021-04-06 12:02:42] SKIPPING http://rs.tdwg.org/dwc/terms/Event config (events.txt)...
[STOP] [2021-04-06 12:02:42] Parse meta.xml file and create formats with fields
[STOP] [2021-04-06 12:02:42] Creating resource from OpenData
[START] [2021-04-06 12:02:43] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-06 12:02:43] create_harvest_instance
[INFO] [2021-04-06 12:02:43] Created harvest instance #3643
[STOP] [2021-04-06 12:02:43] create_harvest_instance
[START] [2021-04-06 12:02:43] fetch_files
[STOP] [2021-04-06 12:02:43] fetch_files
[START] [2021-04-06 12:02:43] validate_each_file
[INFO] [2021-04-06 12:02:43] Looping over 8 formats...
[INFO] [2021-04-06 12:02:43] ...agents (/app/public/data/Alaskan_Arthropo/agents.txt)
[INFO] [2021-04-06 12:02:43] Valid: /app/public/converted_csv/Alaskan_Arthropo_agents_3643.csv (0 lines)
[INFO] [2021-04-06 12:02:43] ...refs (/app/public/data/Alaskan_Arthropo/references.txt)
[INFO] [2021-04-06 12:02:43] Valid: /app/public/converted_csv/Alaskan_Arthropo_refs_3643.csv (443 lines)
[INFO] [2021-04-06 12:02:43] ...nodes (/app/public/data/Alaskan_Arthropo/taxa.txt)
[INFO] [2021-04-06 12:02:44] Valid: /app/public/converted_csv/Alaskan_Arthropo_nodes_3643.csv (7682 lines)
[INFO] [2021-04-06 12:02:44] ...media (/app/public/data/Alaskan_Arthropo/media.txt)
[INFO] [2021-04-06 12:02:44] Valid: /app/public/converted_csv/Alaskan_Arthropo_media_3643.csv (0 lines)
[INFO] [2021-04-06 12:02:44] ...vernaculars (/app/public/data/Alaskan_Arthropo/common names.txt)
[INFO] [2021-04-06 12:02:44] Valid: /app/public/converted_csv/Alaskan_Arthropo_vernaculars_3643.csv (0 lines)
[INFO] [2021-04-06 12:02:44] ...occurrences (/app/public/data/Alaskan_Arthropo/occurrences.txt)
[INFO] [2021-04-06 12:02:45] Valid: /app/public/converted_csv/Alaskan_Arthropo_occurrences_3643.csv (7682 lines)
[INFO] [2021-04-06 12:02:45] ...assocs (/app/public/data/Alaskan_Arthropo/associations.txt)
[INFO] [2021-04-06 12:02:45] Valid: /app/public/converted_csv/Alaskan_Arthropo_assocs_3643.csv (0 lines)
[INFO] [2021-04-06 12:02:45] ...measurements (/app/public/data/Alaskan_Arthropo/measurements or facts.txt)
[STOP] [2021-04-06 12:02:46] validate_each_file
[ERR] [2021-04-06 12:02:46] Exceptions::ColumnEmpty
[ERR] [2021-04-06 12:02:46] Illegal empty value for measurements/Measurement Type on line 7924
[ERR] [2021-04-06 12:02:46] ../models/resource_harvester.rb:196:in `check_header'
[ERR] [2021-04-06 12:02:46] ../models/resource_harvester.rb:183:in `block (2 levels) in validate_csv'
[ERR] [2021-04-06 12:02:46] ../models/resource_harvester.rb:182:in `each'
[ERR] [2021-04-06 12:02:46] ../models/resource_harvester.rb:182:in `block in validate_csv'
[ERR] [2021-04-06 12:02:46] ../models/csv_parser.rb:88:in `block in rows_as_hashes'
[ERR] [2021-04-06 12:02:46] ../models/csv_parser.rb:28:in `block in line_at_a_time'
[ERR] [2021-04-06 12:02:46] ../models/csv_parser.rb:25:in `line_at_a_time'
[ERR] [2021-04-06 12:02:46] ../models/csv_parser.rb:81:in `rows_as_hashes'
[ERR] [2021-04-06 12:02:46] ../models/resource_harvester.rb:178:in `validate_csv'
[ERR] [2021-04-06 12:02:46] ../models/resource_harvester.rb:144:in `block (2 levels) in validate_each_file'
[ERR] [2021-04-06 12:02:46] ../models/resource_harvester.rb:143:in `block in validate_each_file'
[ERR] [2021-04-06 12:02:46] ../models/resource_harvester.rb:681:in `block in each_format'
[ERR] [2021-04-06 12:02:46] ../models/resource_harvester.rb:666:in `each_format'
[ERR] [2021-04-06 12:02:46] ../models/resource_harvester.rb:130:in `validate_each_file'
[ERR] [2021-04-06 12:02:46] ../models/resource_harvester.rb:86:in `block (2 levels) in start'
[ERR] [2021-04-06 12:02:46] ../models/logged_process.rb:34:in `run_step'
[ERR] [2021-04-06 12:02:46] ../models/resource_harvester.rb:86:in `block in start'
[ERR] [2021-04-06 12:02:46] ../models/resource_harvester.rb:75:in `each_key'
[ERR] [2021-04-06 12:02:46] ../models/resource_harvester.rb:75:in `start'
[ERR] [2021-04-06 12:02:46] ../models/resource.rb:261:in `harvest'
[ERR] [2021-04-06 12:02:46] ../models/resource.rb:237:in `re_download_opendata_and_harvest'
[ERR] [2021-04-06 12:02:46] bin/rails:4:in `require'
[ERR] [2021-04-06 12:02:46] bin/rails:4:in `<main>'
[STOP] [2021-04-06 12:02:46] logged process, took 4.08
[INFO] [2021-04-06 12:07:23] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-06 12:07:26] ## remove_type: ScientificName
[INFO] [2021-04-06 12:07:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:07:26] [12:07:26.758] Removed 0 Scientificnames
[INFO] [2021-04-06 12:07:26] ## remove_type: Vernacular
[INFO] [2021-04-06 12:07:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:07:26] [12:07:26.760] Removed 0 Vernaculars
[INFO] [2021-04-06 12:07:26] ## remove_type: Article
[INFO] [2021-04-06 12:07:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:07:26] [12:07:26.761] Removed 0 Articles
[INFO] [2021-04-06 12:07:26] ## remove_type: Medium
[INFO] [2021-04-06 12:07:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:07:26] [12:07:26.763] Removed 0 Media
[INFO] [2021-04-06 12:07:26] ## remove_type: Trait
[INFO] [2021-04-06 12:07:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:07:26] [12:07:26.765] Removed 0 Traits
[INFO] [2021-04-06 12:07:26] ## remove_type: MetaTrait
[INFO] [2021-04-06 12:07:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:07:26] [12:07:26.767] Removed 0 Metatraits
[INFO] [2021-04-06 12:07:26] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-06 12:07:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:07:26] [12:07:26.768] Removed 0 Occurrencemetadata
[INFO] [2021-04-06 12:07:26] ## remove_type: Assoc
[INFO] [2021-04-06 12:07:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:07:26] [12:07:26.771] Removed 0 Assocs
[INFO] [2021-04-06 12:07:26] ## remove_type: MetaAssoc
[INFO] [2021-04-06 12:07:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:07:26] [12:07:26.772] Removed 0 Metaassocs
[INFO] [2021-04-06 12:07:26] ## remove_type: Identifier
[INFO] [2021-04-06 12:07:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:07:26] [12:07:26.774] Removed 0 Identifiers
[INFO] [2021-04-06 12:07:26] ## remove_type: Reference
[INFO] [2021-04-06 12:07:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:07:26] [12:07:26.775] Removed 0 References
[INFO] [2021-04-06 12:07:26] ## remove_type: Node
[INFO] [2021-04-06 12:07:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:07:26] [12:07:26.785] Removed 0 Nodes
[START] [2021-04-06 12:07:28] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-06 12:07:28] Creating resource from OpenData
[START] [2021-04-06 12:07:31] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-06 12:07:31] Parse meta.xml file and create formats with fields
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (media) field header: ThumbnailURL term: http://eol.org/schema/media/thumbnailURL
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (media) field header: DerivedFrom term: http://rs.tdwg.org/ac/terms/derivedFrom
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (media) field header: Modified term: http://purl.org/dc/terms/modified
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (media) field header: Rating term: http://ns.adobe.com/xap/1.0/Rating
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (media) field header: Audience term: http://purl.org/dc/terms/audience
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (media) field header: Rights term: http://purl.org/dc/terms/rights
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (media) field header: Publisher term: http://purl.org/dc/terms/publisher
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (media) field header: Contributor term: http://purl.org/dc/terms/contributor
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (media) field header: Creator term: http://purl.org/dc/terms/creator
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (media) field header: GenericLocation term: http://purl.org/dc/terms/spatial
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (media) field header: Latitude term: http://www.w3.org/2003/01/geo/wgs84_pos#lat
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (media) field header: Longitude term: http://www.w3.org/2003/01/geo/wgs84_pos#long
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (media) field header: Altitude term: http://www.w3.org/2003/01/geo/wgs84_pos#alt
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (occurrences) field header: Institution Code term: http://rs.tdwg.org/dwc/terms/institutionCode
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (occurrences) field header: Collection Code term: http://rs.tdwg.org/dwc/terms/collectionCode
[WARN] [2021-04-06 12:08:04] (common) IGNORED  (occurrences) field header: Catalog Number term: http://rs.tdwg.org/dwc/terms/catalogNumber
[WARN] [2021-04-06 12:08:04] SKIPPING http://rs.tdwg.org/dwc/terms/Event config (events.txt)...
[STOP] [2021-04-06 12:08:04] Parse meta.xml file and create formats with fields
[STOP] [2021-04-06 12:08:04] Creating resource from OpenData
[START] [2021-04-06 12:08:05] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-06 12:08:05] create_harvest_instance
[INFO] [2021-04-06 12:08:05] Created harvest instance #3644
[STOP] [2021-04-06 12:08:05] create_harvest_instance
[START] [2021-04-06 12:08:05] fetch_files
[STOP] [2021-04-06 12:08:05] fetch_files
[START] [2021-04-06 12:08:05] validate_each_file
[INFO] [2021-04-06 12:08:05] Looping over 8 formats...
[INFO] [2021-04-06 12:08:05] ...agents (/app/public/data/Alaskan_Arthropo/agents.txt)
[INFO] [2021-04-06 12:08:05] Valid: /app/public/converted_csv/Alaskan_Arthropo_agents_3644.csv (0 lines)
[INFO] [2021-04-06 12:08:05] ...refs (/app/public/data/Alaskan_Arthropo/references.txt)
[INFO] [2021-04-06 12:08:05] Valid: /app/public/converted_csv/Alaskan_Arthropo_refs_3644.csv (443 lines)
[INFO] [2021-04-06 12:08:05] ...nodes (/app/public/data/Alaskan_Arthropo/taxa.txt)
[INFO] [2021-04-06 12:08:06] Valid: /app/public/converted_csv/Alaskan_Arthropo_nodes_3644.csv (7682 lines)
[INFO] [2021-04-06 12:08:06] ...media (/app/public/data/Alaskan_Arthropo/media.txt)
[INFO] [2021-04-06 12:08:06] Valid: /app/public/converted_csv/Alaskan_Arthropo_media_3644.csv (0 lines)
[INFO] [2021-04-06 12:08:06] ...vernaculars (/app/public/data/Alaskan_Arthropo/common names.txt)
[INFO] [2021-04-06 12:08:06] Valid: /app/public/converted_csv/Alaskan_Arthropo_vernaculars_3644.csv (0 lines)
[INFO] [2021-04-06 12:08:06] ...occurrences (/app/public/data/Alaskan_Arthropo/occurrences.txt)
[INFO] [2021-04-06 12:08:07] Valid: /app/public/converted_csv/Alaskan_Arthropo_occurrences_3644.csv (7682 lines)
[INFO] [2021-04-06 12:08:07] ...assocs (/app/public/data/Alaskan_Arthropo/associations.txt)
[INFO] [2021-04-06 12:08:07] Valid: /app/public/converted_csv/Alaskan_Arthropo_assocs_3644.csv (0 lines)
[INFO] [2021-04-06 12:08:07] ...measurements (/app/public/data/Alaskan_Arthropo/measurements or facts.txt)
[INFO] [2021-04-06 12:08:08] Valid: /app/public/converted_csv/Alaskan_Arthropo_measurements_3644.csv (7923 lines)
[STOP] [2021-04-06 12:08:08] validate_each_file
[START] [2021-04-06 12:08:08] convert_to_csv
[INFO] [2021-04-06 12:08:08] Looping over 8 formats...
[INFO] [2021-04-06 12:08:08] ...agents (/app/public/data/Alaskan_Arthropo/agents.txt)
[CMD] [2021-04-06 12:08:08] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_agents_3644.csv > /app/public/converted_csv/Alaskan_Arthropo_agents_3644.csv_sorted
[INFO] [2021-04-06 12:08:09] Converted: /app/public/converted_csv/Alaskan_Arthropo_agents_3644.csv (0 lines)
[INFO] [2021-04-06 12:08:09] ...refs (/app/public/data/Alaskan_Arthropo/references.txt)
[CMD] [2021-04-06 12:08:09] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_refs_3644.csv > /app/public/converted_csv/Alaskan_Arthropo_refs_3644.csv_sorted
[INFO] [2021-04-06 12:08:11] Converted: /app/public/converted_csv/Alaskan_Arthropo_refs_3644.csv (443 lines)
[INFO] [2021-04-06 12:08:11] ...nodes (/app/public/data/Alaskan_Arthropo/taxa.txt)
[CMD] [2021-04-06 12:08:11] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_nodes_3644.csv > /app/public/converted_csv/Alaskan_Arthropo_nodes_3644.csv_sorted
[INFO] [2021-04-06 12:08:12] Converted: /app/public/converted_csv/Alaskan_Arthropo_nodes_3644.csv (7682 lines)
[INFO] [2021-04-06 12:08:12] ...media (/app/public/data/Alaskan_Arthropo/media.txt)
[CMD] [2021-04-06 12:08:12] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_media_3644.csv > /app/public/converted_csv/Alaskan_Arthropo_media_3644.csv_sorted
[INFO] [2021-04-06 12:08:13] Converted: /app/public/converted_csv/Alaskan_Arthropo_media_3644.csv (0 lines)
[INFO] [2021-04-06 12:08:13] ...vernaculars (/app/public/data/Alaskan_Arthropo/common names.txt)
[CMD] [2021-04-06 12:08:13] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_vernaculars_3644.csv > /app/public/converted_csv/Alaskan_Arthropo_vernaculars_3644.csv_sorted
[INFO] [2021-04-06 12:08:15] Converted: /app/public/converted_csv/Alaskan_Arthropo_vernaculars_3644.csv (0 lines)
[INFO] [2021-04-06 12:08:15] ...occurrences (/app/public/data/Alaskan_Arthropo/occurrences.txt)
[CMD] [2021-04-06 12:08:15] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_occurrences_3644.csv > /app/public/converted_csv/Alaskan_Arthropo_occurrences_3644.csv_sorted
[INFO] [2021-04-06 12:08:16] Converted: /app/public/converted_csv/Alaskan_Arthropo_occurrences_3644.csv (7682 lines)
[INFO] [2021-04-06 12:08:16] ...assocs (/app/public/data/Alaskan_Arthropo/associations.txt)
[CMD] [2021-04-06 12:08:16] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_assocs_3644.csv > /app/public/converted_csv/Alaskan_Arthropo_assocs_3644.csv_sorted
[INFO] [2021-04-06 12:08:17] Converted: /app/public/converted_csv/Alaskan_Arthropo_assocs_3644.csv (0 lines)
[INFO] [2021-04-06 12:08:17] ...measurements (/app/public/data/Alaskan_Arthropo/measurements or facts.txt)
[CMD] [2021-04-06 12:08:17] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_measurements_3644.csv > /app/public/converted_csv/Alaskan_Arthropo_measurements_3644.csv_sorted
[INFO] [2021-04-06 12:08:19] Converted: /app/public/converted_csv/Alaskan_Arthropo_measurements_3644.csv (7923 lines)
[STOP] [2021-04-06 12:08:19] convert_to_csv
[START] [2021-04-06 12:08:19] calculate_delta
[INFO] [2021-04-06 12:08:19] Looping over 8 formats...
[INFO] [2021-04-06 12:08:19] ...agents (/app/public/data/Alaskan_Arthropo/agents.txt)
[CMD] [2021-04-06 12:08:19] echo "0a" > /app/public/diff/Alaskan_Arthropo_agents_3644.diff
[CMD] [2021-04-06 12:08:20] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_agents_3644.csv >> /app/public/diff/Alaskan_Arthropo_agents_3644.diff
[CMD] [2021-04-06 12:08:22] echo "." >> /app/public/diff/Alaskan_Arthropo_agents_3644.diff
[INFO] [2021-04-06 12:08:23] Created diff: /app/public/diff/Alaskan_Arthropo_agents_3644.diff (2 lines)
[INFO] [2021-04-06 12:08:23] ...refs (/app/public/data/Alaskan_Arthropo/references.txt)
[CMD] [2021-04-06 12:08:23] echo "0a" > /app/public/diff/Alaskan_Arthropo_refs_3644.diff
[CMD] [2021-04-06 12:08:24] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_refs_3644.csv >> /app/public/diff/Alaskan_Arthropo_refs_3644.diff
[CMD] [2021-04-06 12:08:25] echo "." >> /app/public/diff/Alaskan_Arthropo_refs_3644.diff
[INFO] [2021-04-06 12:08:27] Created diff: /app/public/diff/Alaskan_Arthropo_refs_3644.diff (445 lines)
[INFO] [2021-04-06 12:08:27] ...nodes (/app/public/data/Alaskan_Arthropo/taxa.txt)
[CMD] [2021-04-06 12:08:27] echo "0a" > /app/public/diff/Alaskan_Arthropo_nodes_3644.diff
[CMD] [2021-04-06 12:08:28] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_nodes_3644.csv >> /app/public/diff/Alaskan_Arthropo_nodes_3644.diff
[CMD] [2021-04-06 12:08:30] echo "." >> /app/public/diff/Alaskan_Arthropo_nodes_3644.diff
[INFO] [2021-04-06 12:08:31] Created diff: /app/public/diff/Alaskan_Arthropo_nodes_3644.diff (7684 lines)
[INFO] [2021-04-06 12:08:31] ...media (/app/public/data/Alaskan_Arthropo/media.txt)
[CMD] [2021-04-06 12:08:31] echo "0a" > /app/public/diff/Alaskan_Arthropo_media_3644.diff
[CMD] [2021-04-06 12:08:32] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_media_3644.csv >> /app/public/diff/Alaskan_Arthropo_media_3644.diff
[CMD] [2021-04-06 12:08:34] echo "." >> /app/public/diff/Alaskan_Arthropo_media_3644.diff
[INFO] [2021-04-06 12:08:35] Created diff: /app/public/diff/Alaskan_Arthropo_media_3644.diff (2 lines)
[INFO] [2021-04-06 12:08:35] ...vernaculars (/app/public/data/Alaskan_Arthropo/common names.txt)
[CMD] [2021-04-06 12:08:35] echo "0a" > /app/public/diff/Alaskan_Arthropo_vernaculars_3644.diff
[CMD] [2021-04-06 12:08:36] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_vernaculars_3644.csv >> /app/public/diff/Alaskan_Arthropo_vernaculars_3644.diff
[CMD] [2021-04-06 12:08:38] echo "." >> /app/public/diff/Alaskan_Arthropo_vernaculars_3644.diff
[INFO] [2021-04-06 12:08:39] Created diff: /app/public/diff/Alaskan_Arthropo_vernaculars_3644.diff (2 lines)
[INFO] [2021-04-06 12:08:39] ...occurrences (/app/public/data/Alaskan_Arthropo/occurrences.txt)
[CMD] [2021-04-06 12:08:39] echo "0a" > /app/public/diff/Alaskan_Arthropo_occurrences_3644.diff
[CMD] [2021-04-06 12:08:40] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_occurrences_3644.csv >> /app/public/diff/Alaskan_Arthropo_occurrences_3644.diff
[CMD] [2021-04-06 12:08:42] echo "." >> /app/public/diff/Alaskan_Arthropo_occurrences_3644.diff
[INFO] [2021-04-06 12:08:43] Created diff: /app/public/diff/Alaskan_Arthropo_occurrences_3644.diff (7684 lines)
[INFO] [2021-04-06 12:08:43] ...assocs (/app/public/data/Alaskan_Arthropo/associations.txt)
[CMD] [2021-04-06 12:08:43] echo "0a" > /app/public/diff/Alaskan_Arthropo_assocs_3644.diff
[CMD] [2021-04-06 12:08:45] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_assocs_3644.csv >> /app/public/diff/Alaskan_Arthropo_assocs_3644.diff
[CMD] [2021-04-06 12:08:46] echo "." >> /app/public/diff/Alaskan_Arthropo_assocs_3644.diff
[INFO] [2021-04-06 12:08:47] Created diff: /app/public/diff/Alaskan_Arthropo_assocs_3644.diff (2 lines)
[INFO] [2021-04-06 12:08:47] ...measurements (/app/public/data/Alaskan_Arthropo/measurements or facts.txt)
[CMD] [2021-04-06 12:08:47] echo "0a" > /app/public/diff/Alaskan_Arthropo_measurements_3644.diff
[CMD] [2021-04-06 12:08:48] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_measurements_3644.csv >> /app/public/diff/Alaskan_Arthropo_measurements_3644.diff
[CMD] [2021-04-06 12:08:50] echo "." >> /app/public/diff/Alaskan_Arthropo_measurements_3644.diff
[INFO] [2021-04-06 12:08:51] Created diff: /app/public/diff/Alaskan_Arthropo_measurements_3644.diff (7925 lines)
[STOP] [2021-04-06 12:08:51] calculate_delta
[START] [2021-04-06 12:08:51] parse_diff_and_store
[INFO] [2021-04-06 12:08:51] Handling diff: /app/public/diff/Alaskan_Arthropo_agents_3644.diff (2 lines)
[INFO] [2021-04-06 12:08:53] Loading agents diff file into memory (2 /app/public/diff/Alaskan_Arthropo_agents_3644.diff lines)...
[INFO] [2021-04-06 12:08:54] Handling diff: /app/public/diff/Alaskan_Arthropo_refs_3644.diff (445 lines)
[INFO] [2021-04-06 12:08:56] Loading refs diff file into memory (445 /app/public/diff/Alaskan_Arthropo_refs_3644.diff lines)...
[INFO] [2021-04-06 12:08:57] Handling diff: /app/public/diff/Alaskan_Arthropo_nodes_3644.diff (7684 lines)
[INFO] [2021-04-06 12:08:59] Loading nodes diff file into memory (7684 /app/public/diff/Alaskan_Arthropo_nodes_3644.diff lines)...
[INFO] [2021-04-06 12:09:03] Handling diff: /app/public/diff/Alaskan_Arthropo_media_3644.diff (2 lines)
[INFO] [2021-04-06 12:09:04] Loading media diff file into memory (2 /app/public/diff/Alaskan_Arthropo_media_3644.diff lines)...
[INFO] [2021-04-06 12:09:06] Handling diff: /app/public/diff/Alaskan_Arthropo_vernaculars_3644.diff (2 lines)
[INFO] [2021-04-06 12:09:07] Loading vernaculars diff file into memory (2 /app/public/diff/Alaskan_Arthropo_vernaculars_3644.diff lines)...
[INFO] [2021-04-06 12:09:09] Handling diff: /app/public/diff/Alaskan_Arthropo_occurrences_3644.diff (7684 lines)
[INFO] [2021-04-06 12:09:10] Loading occurrences diff file into memory (7684 /app/public/diff/Alaskan_Arthropo_occurrences_3644.diff lines)...
[INFO] [2021-04-06 12:09:12] Handling diff: /app/public/diff/Alaskan_Arthropo_assocs_3644.diff (2 lines)
[INFO] [2021-04-06 12:09:14] Loading assocs diff file into memory (2 /app/public/diff/Alaskan_Arthropo_assocs_3644.diff lines)...
[INFO] [2021-04-06 12:09:15] Handling diff: /app/public/diff/Alaskan_Arthropo_measurements_3644.diff (7925 lines)
[INFO] [2021-04-06 12:09:16] Loading measurements diff file into memory (7925 /app/public/diff/Alaskan_Arthropo_measurements_3644.diff lines)...
[STOP] [2021-04-06 12:09:18] parse_diff_and_store
[ERR] [2021-04-06 12:09:18] RuntimeError
[ERR] [2021-04-06 12:09:18] Missing Term for URI `http://eol.org/schema/terms/D_S_Sikes`, must be added!
[ERR] [2021-04-06 12:09:18] ../models/store/model_builder.rb:640:in `fail_on_bad_uri'
[ERR] [2021-04-06 12:09:18] ../models/store/model_builder.rb:628:in `convert_meta_value'
[ERR] [2021-04-06 12:09:18] ../models/store/model_builder.rb:443:in `block in build_trait'
[ERR] [2021-04-06 12:09:18] ../models/store/model_builder.rb:436:in `each'
[ERR] [2021-04-06 12:09:18] ../models/store/model_builder.rb:436:in `build_trait'
[ERR] [2021-04-06 12:09:18] ../models/store/model_builder.rb:28:in `build_models'
[ERR] [2021-04-06 12:09:18] ../models/resource_harvester.rb:352:in `block (3 levels) in parse_diff_and_store'
[ERR] [2021-04-06 12:09:18] ../models/csv_parser.rb:111:in `block in diff_as_hashes'
[ERR] [2021-04-06 12:09:18] ../models/csv_parser.rb:28:in `block in line_at_a_time'
[ERR] [2021-04-06 12:09:18] ../models/csv_parser.rb:25:in `line_at_a_time'
[ERR] [2021-04-06 12:09:18] ../models/csv_parser.rb:96:in `diff_as_hashes'
[ERR] [2021-04-06 12:09:18] ../models/resource_harvester.rb:306:in `block (2 levels) in parse_diff_and_store'
[ERR] [2021-04-06 12:09:18] ../models/logged_process.rb:77:in `enter_group'
[ERR] [2021-04-06 12:09:18] ../models/resource_harvester.rb:305:in `block in parse_diff_and_store'
[ERR] [2021-04-06 12:09:18] ../models/resource_harvester.rb:702:in `block in each_diff'
[ERR] [2021-04-06 12:09:18] ../models/resource_harvester.rb:689:in `each_diff'
[ERR] [2021-04-06 12:09:18] ../models/resource_harvester.rb:300:in `parse_diff_and_store'
[ERR] [2021-04-06 12:09:18] ../models/resource_harvester.rb:86:in `block (2 levels) in start'
[ERR] [2021-04-06 12:09:18] ../models/logged_process.rb:34:in `run_step'
[ERR] [2021-04-06 12:09:18] ../models/resource_harvester.rb:86:in `block in start'
[ERR] [2021-04-06 12:09:18] ../models/resource_harvester.rb:75:in `each_key'
[ERR] [2021-04-06 12:09:18] ../models/resource_harvester.rb:75:in `start'
[ERR] [2021-04-06 12:09:18] ../models/resource.rb:261:in `harvest'
[ERR] [2021-04-06 12:09:18] ../models/resource.rb:237:in `re_download_opendata_and_harvest'
[ERR] [2021-04-06 12:09:18] bin/rails:4:in `require'
[ERR] [2021-04-06 12:09:18] bin/rails:4:in `<main>'
[STOP] [2021-04-06 12:09:18] logged process, took 73.89
[INFO] [2021-04-06 12:46:04] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-06 12:46:06] ## remove_type: ScientificName
[INFO] [2021-04-06 12:46:06] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:46:06] [12:46:06.241] Removed 0 Scientificnames
[INFO] [2021-04-06 12:46:06] ## remove_type: Vernacular
[INFO] [2021-04-06 12:46:06] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:46:06] [12:46:06.243] Removed 0 Vernaculars
[INFO] [2021-04-06 12:46:06] ## remove_type: Article
[INFO] [2021-04-06 12:46:06] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:46:06] [12:46:06.244] Removed 0 Articles
[INFO] [2021-04-06 12:46:06] ## remove_type: Medium
[INFO] [2021-04-06 12:46:06] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:46:06] [12:46:06.246] Removed 0 Media
[INFO] [2021-04-06 12:46:06] ## remove_type: Trait
[INFO] [2021-04-06 12:46:06] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:46:06] [12:46:06.248] Removed 0 Traits
[INFO] [2021-04-06 12:46:06] ## remove_type: MetaTrait
[INFO] [2021-04-06 12:46:06] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:46:06] [12:46:06.249] Removed 0 Metatraits
[INFO] [2021-04-06 12:46:06] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-06 12:46:06] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:46:06] [12:46:06.251] Removed 0 Occurrencemetadata
[INFO] [2021-04-06 12:46:06] ## remove_type: Assoc
[INFO] [2021-04-06 12:46:06] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:46:06] [12:46:06.253] Removed 0 Assocs
[INFO] [2021-04-06 12:46:06] ## remove_type: MetaAssoc
[INFO] [2021-04-06 12:46:06] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:46:06] [12:46:06.255] Removed 0 Metaassocs
[INFO] [2021-04-06 12:46:06] ## remove_type: Identifier
[INFO] [2021-04-06 12:46:06] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:46:06] [12:46:06.256] Removed 0 Identifiers
[INFO] [2021-04-06 12:46:06] ## remove_type: Reference
[INFO] [2021-04-06 12:46:06] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:46:06] [12:46:06.258] Removed 0 References
[INFO] [2021-04-06 12:46:06] ## remove_type: Node
[INFO] [2021-04-06 12:46:06] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 12:46:06] [12:46:06.268] Removed 0 Nodes
[START] [2021-04-06 12:46:07] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-06 12:46:07] Creating resource from OpenData
[START] [2021-04-06 12:46:10] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-06 12:46:10] Parse meta.xml file and create formats with fields
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (media) field header: ThumbnailURL term: http://eol.org/schema/media/thumbnailURL
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (media) field header: DerivedFrom term: http://rs.tdwg.org/ac/terms/derivedFrom
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (media) field header: Modified term: http://purl.org/dc/terms/modified
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (media) field header: Rating term: http://ns.adobe.com/xap/1.0/Rating
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (media) field header: Audience term: http://purl.org/dc/terms/audience
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (media) field header: Rights term: http://purl.org/dc/terms/rights
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (media) field header: Publisher term: http://purl.org/dc/terms/publisher
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (media) field header: Contributor term: http://purl.org/dc/terms/contributor
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (media) field header: Creator term: http://purl.org/dc/terms/creator
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (media) field header: GenericLocation term: http://purl.org/dc/terms/spatial
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (media) field header: Latitude term: http://www.w3.org/2003/01/geo/wgs84_pos#lat
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (media) field header: Longitude term: http://www.w3.org/2003/01/geo/wgs84_pos#long
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (media) field header: Altitude term: http://www.w3.org/2003/01/geo/wgs84_pos#alt
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (occurrences) field header: Institution Code term: http://rs.tdwg.org/dwc/terms/institutionCode
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (occurrences) field header: Collection Code term: http://rs.tdwg.org/dwc/terms/collectionCode
[WARN] [2021-04-06 12:46:42] (common) IGNORED  (occurrences) field header: Catalog Number term: http://rs.tdwg.org/dwc/terms/catalogNumber
[WARN] [2021-04-06 12:46:42] SKIPPING http://rs.tdwg.org/dwc/terms/Event config (events.txt)...
[STOP] [2021-04-06 12:46:42] Parse meta.xml file and create formats with fields
[STOP] [2021-04-06 12:46:42] Creating resource from OpenData
[START] [2021-04-06 12:46:43] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-06 12:46:43] create_harvest_instance
[INFO] [2021-04-06 12:46:43] Created harvest instance #3645
[STOP] [2021-04-06 12:46:43] create_harvest_instance
[START] [2021-04-06 12:46:43] fetch_files
[STOP] [2021-04-06 12:46:43] fetch_files
[START] [2021-04-06 12:46:43] validate_each_file
[INFO] [2021-04-06 12:46:43] Looping over 8 formats...
[INFO] [2021-04-06 12:46:43] ...agents (/app/public/data/Alaskan_Arthropo/agents.txt)
[INFO] [2021-04-06 12:46:43] Valid: /app/public/converted_csv/Alaskan_Arthropo_agents_3645.csv (0 lines)
[INFO] [2021-04-06 12:46:43] ...refs (/app/public/data/Alaskan_Arthropo/references.txt)
[INFO] [2021-04-06 12:46:43] Valid: /app/public/converted_csv/Alaskan_Arthropo_refs_3645.csv (443 lines)
[INFO] [2021-04-06 12:46:43] ...nodes (/app/public/data/Alaskan_Arthropo/taxa.txt)
[INFO] [2021-04-06 12:46:44] Valid: /app/public/converted_csv/Alaskan_Arthropo_nodes_3645.csv (7682 lines)
[INFO] [2021-04-06 12:46:44] ...media (/app/public/data/Alaskan_Arthropo/media.txt)
[INFO] [2021-04-06 12:46:44] Valid: /app/public/converted_csv/Alaskan_Arthropo_media_3645.csv (0 lines)
[INFO] [2021-04-06 12:46:44] ...vernaculars (/app/public/data/Alaskan_Arthropo/common names.txt)
[INFO] [2021-04-06 12:46:44] Valid: /app/public/converted_csv/Alaskan_Arthropo_vernaculars_3645.csv (0 lines)
[INFO] [2021-04-06 12:46:44] ...occurrences (/app/public/data/Alaskan_Arthropo/occurrences.txt)
[INFO] [2021-04-06 12:46:45] Valid: /app/public/converted_csv/Alaskan_Arthropo_occurrences_3645.csv (7682 lines)
[INFO] [2021-04-06 12:46:45] ...assocs (/app/public/data/Alaskan_Arthropo/associations.txt)
[INFO] [2021-04-06 12:46:45] Valid: /app/public/converted_csv/Alaskan_Arthropo_assocs_3645.csv (0 lines)
[INFO] [2021-04-06 12:46:45] ...measurements (/app/public/data/Alaskan_Arthropo/measurements or facts.txt)
[INFO] [2021-04-06 12:46:45] Valid: /app/public/converted_csv/Alaskan_Arthropo_measurements_3645.csv (7923 lines)
[STOP] [2021-04-06 12:46:45] validate_each_file
[START] [2021-04-06 12:46:45] convert_to_csv
[INFO] [2021-04-06 12:46:45] Looping over 8 formats...
[INFO] [2021-04-06 12:46:45] ...agents (/app/public/data/Alaskan_Arthropo/agents.txt)
[CMD] [2021-04-06 12:46:45] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_agents_3645.csv > /app/public/converted_csv/Alaskan_Arthropo_agents_3645.csv_sorted
[INFO] [2021-04-06 12:46:46] Converted: /app/public/converted_csv/Alaskan_Arthropo_agents_3645.csv (0 lines)
[INFO] [2021-04-06 12:46:46] ...refs (/app/public/data/Alaskan_Arthropo/references.txt)
[CMD] [2021-04-06 12:46:46] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_refs_3645.csv > /app/public/converted_csv/Alaskan_Arthropo_refs_3645.csv_sorted
[INFO] [2021-04-06 12:46:48] Converted: /app/public/converted_csv/Alaskan_Arthropo_refs_3645.csv (443 lines)
[INFO] [2021-04-06 12:46:48] ...nodes (/app/public/data/Alaskan_Arthropo/taxa.txt)
[CMD] [2021-04-06 12:46:48] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_nodes_3645.csv > /app/public/converted_csv/Alaskan_Arthropo_nodes_3645.csv_sorted
[INFO] [2021-04-06 12:46:49] Converted: /app/public/converted_csv/Alaskan_Arthropo_nodes_3645.csv (7682 lines)
[INFO] [2021-04-06 12:46:49] ...media (/app/public/data/Alaskan_Arthropo/media.txt)
[CMD] [2021-04-06 12:46:49] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_media_3645.csv > /app/public/converted_csv/Alaskan_Arthropo_media_3645.csv_sorted
[INFO] [2021-04-06 12:46:50] Converted: /app/public/converted_csv/Alaskan_Arthropo_media_3645.csv (0 lines)
[INFO] [2021-04-06 12:46:50] ...vernaculars (/app/public/data/Alaskan_Arthropo/common names.txt)
[CMD] [2021-04-06 12:46:50] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_vernaculars_3645.csv > /app/public/converted_csv/Alaskan_Arthropo_vernaculars_3645.csv_sorted
[INFO] [2021-04-06 12:46:52] Converted: /app/public/converted_csv/Alaskan_Arthropo_vernaculars_3645.csv (0 lines)
[INFO] [2021-04-06 12:46:52] ...occurrences (/app/public/data/Alaskan_Arthropo/occurrences.txt)
[CMD] [2021-04-06 12:46:52] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_occurrences_3645.csv > /app/public/converted_csv/Alaskan_Arthropo_occurrences_3645.csv_sorted
[INFO] [2021-04-06 12:46:53] Converted: /app/public/converted_csv/Alaskan_Arthropo_occurrences_3645.csv (7682 lines)
[INFO] [2021-04-06 12:46:53] ...assocs (/app/public/data/Alaskan_Arthropo/associations.txt)
[CMD] [2021-04-06 12:46:53] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_assocs_3645.csv > /app/public/converted_csv/Alaskan_Arthropo_assocs_3645.csv_sorted
[INFO] [2021-04-06 12:46:54] Converted: /app/public/converted_csv/Alaskan_Arthropo_assocs_3645.csv (0 lines)
[INFO] [2021-04-06 12:46:54] ...measurements (/app/public/data/Alaskan_Arthropo/measurements or facts.txt)
[CMD] [2021-04-06 12:46:54] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_measurements_3645.csv > /app/public/converted_csv/Alaskan_Arthropo_measurements_3645.csv_sorted
[INFO] [2021-04-06 12:46:56] Converted: /app/public/converted_csv/Alaskan_Arthropo_measurements_3645.csv (7923 lines)
[STOP] [2021-04-06 12:46:56] convert_to_csv
[START] [2021-04-06 12:46:56] calculate_delta
[INFO] [2021-04-06 12:46:56] Looping over 8 formats...
[INFO] [2021-04-06 12:46:56] ...agents (/app/public/data/Alaskan_Arthropo/agents.txt)
[CMD] [2021-04-06 12:46:56] echo "0a" > /app/public/diff/Alaskan_Arthropo_agents_3645.diff
[CMD] [2021-04-06 12:46:57] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_agents_3645.csv >> /app/public/diff/Alaskan_Arthropo_agents_3645.diff
[CMD] [2021-04-06 12:46:58] echo "." >> /app/public/diff/Alaskan_Arthropo_agents_3645.diff
[INFO] [2021-04-06 12:47:00] Created diff: /app/public/diff/Alaskan_Arthropo_agents_3645.diff (2 lines)
[INFO] [2021-04-06 12:47:00] ...refs (/app/public/data/Alaskan_Arthropo/references.txt)
[CMD] [2021-04-06 12:47:00] echo "0a" > /app/public/diff/Alaskan_Arthropo_refs_3645.diff
[CMD] [2021-04-06 12:47:01] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_refs_3645.csv >> /app/public/diff/Alaskan_Arthropo_refs_3645.diff
[CMD] [2021-04-06 12:47:02] echo "." >> /app/public/diff/Alaskan_Arthropo_refs_3645.diff
[INFO] [2021-04-06 12:47:03] Created diff: /app/public/diff/Alaskan_Arthropo_refs_3645.diff (445 lines)
[INFO] [2021-04-06 12:47:03] ...nodes (/app/public/data/Alaskan_Arthropo/taxa.txt)
[CMD] [2021-04-06 12:47:03] echo "0a" > /app/public/diff/Alaskan_Arthropo_nodes_3645.diff
[CMD] [2021-04-06 12:47:05] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_nodes_3645.csv >> /app/public/diff/Alaskan_Arthropo_nodes_3645.diff
[CMD] [2021-04-06 12:47:06] echo "." >> /app/public/diff/Alaskan_Arthropo_nodes_3645.diff
[INFO] [2021-04-06 12:47:08] Created diff: /app/public/diff/Alaskan_Arthropo_nodes_3645.diff (7684 lines)
[INFO] [2021-04-06 12:47:08] ...media (/app/public/data/Alaskan_Arthropo/media.txt)
[CMD] [2021-04-06 12:47:08] echo "0a" > /app/public/diff/Alaskan_Arthropo_media_3645.diff
[CMD] [2021-04-06 12:47:09] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_media_3645.csv >> /app/public/diff/Alaskan_Arthropo_media_3645.diff
[CMD] [2021-04-06 12:47:10] echo "." >> /app/public/diff/Alaskan_Arthropo_media_3645.diff
[INFO] [2021-04-06 12:47:12] Created diff: /app/public/diff/Alaskan_Arthropo_media_3645.diff (2 lines)
[INFO] [2021-04-06 12:47:12] ...vernaculars (/app/public/data/Alaskan_Arthropo/common names.txt)
[CMD] [2021-04-06 12:47:12] echo "0a" > /app/public/diff/Alaskan_Arthropo_vernaculars_3645.diff
[CMD] [2021-04-06 12:47:13] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_vernaculars_3645.csv >> /app/public/diff/Alaskan_Arthropo_vernaculars_3645.diff
[CMD] [2021-04-06 12:47:14] echo "." >> /app/public/diff/Alaskan_Arthropo_vernaculars_3645.diff
[INFO] [2021-04-06 12:47:16] Created diff: /app/public/diff/Alaskan_Arthropo_vernaculars_3645.diff (2 lines)
[INFO] [2021-04-06 12:47:16] ...occurrences (/app/public/data/Alaskan_Arthropo/occurrences.txt)
[CMD] [2021-04-06 12:47:16] echo "0a" > /app/public/diff/Alaskan_Arthropo_occurrences_3645.diff
[CMD] [2021-04-06 12:47:17] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_occurrences_3645.csv >> /app/public/diff/Alaskan_Arthropo_occurrences_3645.diff
[CMD] [2021-04-06 12:47:18] echo "." >> /app/public/diff/Alaskan_Arthropo_occurrences_3645.diff
[INFO] [2021-04-06 12:47:19] Created diff: /app/public/diff/Alaskan_Arthropo_occurrences_3645.diff (7684 lines)
[INFO] [2021-04-06 12:47:19] ...assocs (/app/public/data/Alaskan_Arthropo/associations.txt)
[CMD] [2021-04-06 12:47:19] echo "0a" > /app/public/diff/Alaskan_Arthropo_assocs_3645.diff
[CMD] [2021-04-06 12:47:21] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_assocs_3645.csv >> /app/public/diff/Alaskan_Arthropo_assocs_3645.diff
[CMD] [2021-04-06 12:47:22] echo "." >> /app/public/diff/Alaskan_Arthropo_assocs_3645.diff
[INFO] [2021-04-06 12:47:23] Created diff: /app/public/diff/Alaskan_Arthropo_assocs_3645.diff (2 lines)
[INFO] [2021-04-06 12:47:23] ...measurements (/app/public/data/Alaskan_Arthropo/measurements or facts.txt)
[CMD] [2021-04-06 12:47:23] echo "0a" > /app/public/diff/Alaskan_Arthropo_measurements_3645.diff
[CMD] [2021-04-06 12:47:25] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_measurements_3645.csv >> /app/public/diff/Alaskan_Arthropo_measurements_3645.diff
[CMD] [2021-04-06 12:47:26] echo "." >> /app/public/diff/Alaskan_Arthropo_measurements_3645.diff
[INFO] [2021-04-06 12:47:27] Created diff: /app/public/diff/Alaskan_Arthropo_measurements_3645.diff (7925 lines)
[STOP] [2021-04-06 12:47:27] calculate_delta
[START] [2021-04-06 12:47:27] parse_diff_and_store
[INFO] [2021-04-06 12:47:27] Handling diff: /app/public/diff/Alaskan_Arthropo_agents_3645.diff (2 lines)
[INFO] [2021-04-06 12:47:29] Loading agents diff file into memory (2 /app/public/diff/Alaskan_Arthropo_agents_3645.diff lines)...
[INFO] [2021-04-06 12:47:30] Handling diff: /app/public/diff/Alaskan_Arthropo_refs_3645.diff (445 lines)
[INFO] [2021-04-06 12:47:31] Loading refs diff file into memory (445 /app/public/diff/Alaskan_Arthropo_refs_3645.diff lines)...
[INFO] [2021-04-06 12:47:33] Handling diff: /app/public/diff/Alaskan_Arthropo_nodes_3645.diff (7684 lines)
[INFO] [2021-04-06 12:47:34] Loading nodes diff file into memory (7684 /app/public/diff/Alaskan_Arthropo_nodes_3645.diff lines)...
[INFO] [2021-04-06 12:47:38] Handling diff: /app/public/diff/Alaskan_Arthropo_media_3645.diff (2 lines)
[INFO] [2021-04-06 12:47:40] Loading media diff file into memory (2 /app/public/diff/Alaskan_Arthropo_media_3645.diff lines)...
[INFO] [2021-04-06 12:47:41] Handling diff: /app/public/diff/Alaskan_Arthropo_vernaculars_3645.diff (2 lines)
[INFO] [2021-04-06 12:47:43] Loading vernaculars diff file into memory (2 /app/public/diff/Alaskan_Arthropo_vernaculars_3645.diff lines)...
[INFO] [2021-04-06 12:47:44] Handling diff: /app/public/diff/Alaskan_Arthropo_occurrences_3645.diff (7684 lines)
[INFO] [2021-04-06 12:47:45] Loading occurrences diff file into memory (7684 /app/public/diff/Alaskan_Arthropo_occurrences_3645.diff lines)...
[INFO] [2021-04-06 12:47:47] Handling diff: /app/public/diff/Alaskan_Arthropo_assocs_3645.diff (2 lines)
[INFO] [2021-04-06 12:47:48] Loading assocs diff file into memory (2 /app/public/diff/Alaskan_Arthropo_assocs_3645.diff lines)...
[INFO] [2021-04-06 12:47:50] Handling diff: /app/public/diff/Alaskan_Arthropo_measurements_3645.diff (7925 lines)
[INFO] [2021-04-06 12:47:51] Loading measurements diff file into memory (7925 /app/public/diff/Alaskan_Arthropo_measurements_3645.diff lines)...
[STOP] [2021-04-06 12:47:53] parse_diff_and_store
[ERR] [2021-04-06 12:47:53] RuntimeError
[ERR] [2021-04-06 12:47:53] Missing Term for URI `http://eol.org/schema/terms/W_McCambridge`, must be added!
[ERR] [2021-04-06 12:47:53] ../models/store/model_builder.rb:640:in `fail_on_bad_uri'
[ERR] [2021-04-06 12:47:53] ../models/store/model_builder.rb:628:in `convert_meta_value'
[ERR] [2021-04-06 12:47:53] ../models/store/model_builder.rb:443:in `block in build_trait'
[ERR] [2021-04-06 12:47:53] ../models/store/model_builder.rb:436:in `each'
[ERR] [2021-04-06 12:47:53] ../models/store/model_builder.rb:436:in `build_trait'
[ERR] [2021-04-06 12:47:53] ../models/store/model_builder.rb:28:in `build_models'
[ERR] [2021-04-06 12:47:53] ../models/resource_harvester.rb:352:in `block (3 levels) in parse_diff_and_store'
[ERR] [2021-04-06 12:47:53] ../models/csv_parser.rb:111:in `block in diff_as_hashes'
[ERR] [2021-04-06 12:47:53] ../models/csv_parser.rb:28:in `block in line_at_a_time'
[ERR] [2021-04-06 12:47:53] ../models/csv_parser.rb:25:in `line_at_a_time'
[ERR] [2021-04-06 12:47:53] ../models/csv_parser.rb:96:in `diff_as_hashes'
[ERR] [2021-04-06 12:47:53] ../models/resource_harvester.rb:306:in `block (2 levels) in parse_diff_and_store'
[ERR] [2021-04-06 12:47:53] ../models/logged_process.rb:77:in `enter_group'
[ERR] [2021-04-06 12:47:53] ../models/resource_harvester.rb:305:in `block in parse_diff_and_store'
[ERR] [2021-04-06 12:47:53] ../models/resource_harvester.rb:702:in `block in each_diff'
[ERR] [2021-04-06 12:47:53] ../models/resource_harvester.rb:689:in `each_diff'
[ERR] [2021-04-06 12:47:53] ../models/resource_harvester.rb:300:in `parse_diff_and_store'
[ERR] [2021-04-06 12:47:53] ../models/resource_harvester.rb:86:in `block (2 levels) in start'
[ERR] [2021-04-06 12:47:53] ../models/logged_process.rb:34:in `run_step'
[ERR] [2021-04-06 12:47:53] ../models/resource_harvester.rb:86:in `block in start'
[ERR] [2021-04-06 12:47:53] ../models/resource_harvester.rb:75:in `each_key'
[ERR] [2021-04-06 12:47:53] ../models/resource_harvester.rb:75:in `start'
[ERR] [2021-04-06 12:47:53] ../models/resource.rb:261:in `harvest'
[ERR] [2021-04-06 12:47:53] ../models/resource.rb:237:in `re_download_opendata_and_harvest'
[ERR] [2021-04-06 12:47:53] bin/rails:4:in `require'
[ERR] [2021-04-06 12:47:53] bin/rails:4:in `<main>'
[STOP] [2021-04-06 12:47:53] logged process, took 70.92
[INFO] [2021-04-06 13:03:19] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-06 13:03:24] ## remove_type: ScientificName
[INFO] [2021-04-06 13:03:24] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 13:03:24] [13:03:24.330] Removed 0 Scientificnames
[INFO] [2021-04-06 13:03:24] ## remove_type: Vernacular
[INFO] [2021-04-06 13:03:24] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 13:03:24] [13:03:24.332] Removed 0 Vernaculars
[INFO] [2021-04-06 13:03:24] ## remove_type: Article
[INFO] [2021-04-06 13:03:24] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 13:03:24] [13:03:24.334] Removed 0 Articles
[INFO] [2021-04-06 13:03:24] ## remove_type: Medium
[INFO] [2021-04-06 13:03:24] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 13:03:24] [13:03:24.335] Removed 0 Media
[INFO] [2021-04-06 13:03:24] ## remove_type: Trait
[INFO] [2021-04-06 13:03:24] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 13:03:24] [13:03:24.337] Removed 0 Traits
[INFO] [2021-04-06 13:03:24] ## remove_type: MetaTrait
[INFO] [2021-04-06 13:03:24] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 13:03:24] [13:03:24.338] Removed 0 Metatraits
[INFO] [2021-04-06 13:03:24] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-06 13:03:24] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 13:03:24] [13:03:24.339] Removed 0 Occurrencemetadata
[INFO] [2021-04-06 13:03:24] ## remove_type: Assoc
[INFO] [2021-04-06 13:03:24] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 13:03:24] [13:03:24.340] Removed 0 Assocs
[INFO] [2021-04-06 13:03:24] ## remove_type: MetaAssoc
[INFO] [2021-04-06 13:03:24] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 13:03:24] [13:03:24.342] Removed 0 Metaassocs
[INFO] [2021-04-06 13:03:24] ## remove_type: Identifier
[INFO] [2021-04-06 13:03:24] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 13:03:24] [13:03:24.343] Removed 0 Identifiers
[INFO] [2021-04-06 13:03:24] ## remove_type: Reference
[INFO] [2021-04-06 13:03:24] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 13:03:24] [13:03:24.344] Removed 0 References
[INFO] [2021-04-06 13:03:24] ## remove_type: Node
[INFO] [2021-04-06 13:03:24] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-06 13:03:24] [13:03:24.354] Removed 0 Nodes
[START] [2021-04-06 13:03:25] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-06 13:03:25] Creating resource from OpenData
[START] [2021-04-06 13:03:28] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-06 13:03:28] Parse meta.xml file and create formats with fields
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (media) field header: ThumbnailURL term: http://eol.org/schema/media/thumbnailURL
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (media) field header: DerivedFrom term: http://rs.tdwg.org/ac/terms/derivedFrom
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (media) field header: Modified term: http://purl.org/dc/terms/modified
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (media) field header: Rating term: http://ns.adobe.com/xap/1.0/Rating
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (media) field header: Audience term: http://purl.org/dc/terms/audience
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (media) field header: Rights term: http://purl.org/dc/terms/rights
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (media) field header: Publisher term: http://purl.org/dc/terms/publisher
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (media) field header: Contributor term: http://purl.org/dc/terms/contributor
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (media) field header: Creator term: http://purl.org/dc/terms/creator
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (media) field header: GenericLocation term: http://purl.org/dc/terms/spatial
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (media) field header: Latitude term: http://www.w3.org/2003/01/geo/wgs84_pos#lat
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (media) field header: Longitude term: http://www.w3.org/2003/01/geo/wgs84_pos#long
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (media) field header: Altitude term: http://www.w3.org/2003/01/geo/wgs84_pos#alt
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (occurrences) field header: Institution Code term: http://rs.tdwg.org/dwc/terms/institutionCode
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (occurrences) field header: Collection Code term: http://rs.tdwg.org/dwc/terms/collectionCode
[WARN] [2021-04-06 13:03:59] (common) IGNORED  (occurrences) field header: Catalog Number term: http://rs.tdwg.org/dwc/terms/catalogNumber
[WARN] [2021-04-06 13:03:59] SKIPPING http://rs.tdwg.org/dwc/terms/Event config (events.txt)...
[STOP] [2021-04-06 13:03:59] Parse meta.xml file and create formats with fields
[STOP] [2021-04-06 13:03:59] Creating resource from OpenData
[START] [2021-04-06 13:04:00] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-06 13:04:00] create_harvest_instance
[INFO] [2021-04-06 13:04:00] Created harvest instance #3646
[STOP] [2021-04-06 13:04:00] create_harvest_instance
[START] [2021-04-06 13:04:00] fetch_files
[STOP] [2021-04-06 13:04:00] fetch_files
[START] [2021-04-06 13:04:00] validate_each_file
[INFO] [2021-04-06 13:04:00] Looping over 8 formats...
[INFO] [2021-04-06 13:04:00] ...agents (/app/public/data/Alaskan_Arthropo/agents.txt)
[INFO] [2021-04-06 13:04:00] Valid: /app/public/converted_csv/Alaskan_Arthropo_agents_3646.csv (0 lines)
[INFO] [2021-04-06 13:04:00] ...refs (/app/public/data/Alaskan_Arthropo/references.txt)
[INFO] [2021-04-06 13:04:00] Valid: /app/public/converted_csv/Alaskan_Arthropo_refs_3646.csv (443 lines)
[INFO] [2021-04-06 13:04:00] ...nodes (/app/public/data/Alaskan_Arthropo/taxa.txt)
[INFO] [2021-04-06 13:04:01] Valid: /app/public/converted_csv/Alaskan_Arthropo_nodes_3646.csv (7682 lines)
[INFO] [2021-04-06 13:04:01] ...media (/app/public/data/Alaskan_Arthropo/media.txt)
[INFO] [2021-04-06 13:04:01] Valid: /app/public/converted_csv/Alaskan_Arthropo_media_3646.csv (0 lines)
[INFO] [2021-04-06 13:04:01] ...vernaculars (/app/public/data/Alaskan_Arthropo/common names.txt)
[INFO] [2021-04-06 13:04:01] Valid: /app/public/converted_csv/Alaskan_Arthropo_vernaculars_3646.csv (0 lines)
[INFO] [2021-04-06 13:04:01] ...occurrences (/app/public/data/Alaskan_Arthropo/occurrences.txt)
[INFO] [2021-04-06 13:04:02] Valid: /app/public/converted_csv/Alaskan_Arthropo_occurrences_3646.csv (7682 lines)
[INFO] [2021-04-06 13:04:02] ...assocs (/app/public/data/Alaskan_Arthropo/associations.txt)
[INFO] [2021-04-06 13:04:02] Valid: /app/public/converted_csv/Alaskan_Arthropo_assocs_3646.csv (0 lines)
[INFO] [2021-04-06 13:04:02] ...measurements (/app/public/data/Alaskan_Arthropo/measurements or facts.txt)
[INFO] [2021-04-06 13:04:02] Valid: /app/public/converted_csv/Alaskan_Arthropo_measurements_3646.csv (7923 lines)
[STOP] [2021-04-06 13:04:02] validate_each_file
[START] [2021-04-06 13:04:02] convert_to_csv
[INFO] [2021-04-06 13:04:02] Looping over 8 formats...
[INFO] [2021-04-06 13:04:02] ...agents (/app/public/data/Alaskan_Arthropo/agents.txt)
[CMD] [2021-04-06 13:04:02] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_agents_3646.csv > /app/public/converted_csv/Alaskan_Arthropo_agents_3646.csv_sorted
[INFO] [2021-04-06 13:04:04] Converted: /app/public/converted_csv/Alaskan_Arthropo_agents_3646.csv (0 lines)
[INFO] [2021-04-06 13:04:04] ...refs (/app/public/data/Alaskan_Arthropo/references.txt)
[CMD] [2021-04-06 13:04:04] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_refs_3646.csv > /app/public/converted_csv/Alaskan_Arthropo_refs_3646.csv_sorted
[INFO] [2021-04-06 13:04:05] Converted: /app/public/converted_csv/Alaskan_Arthropo_refs_3646.csv (443 lines)
[INFO] [2021-04-06 13:04:05] ...nodes (/app/public/data/Alaskan_Arthropo/taxa.txt)
[CMD] [2021-04-06 13:04:05] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_nodes_3646.csv > /app/public/converted_csv/Alaskan_Arthropo_nodes_3646.csv_sorted
[INFO] [2021-04-06 13:04:06] Converted: /app/public/converted_csv/Alaskan_Arthropo_nodes_3646.csv (7682 lines)
[INFO] [2021-04-06 13:04:06] ...media (/app/public/data/Alaskan_Arthropo/media.txt)
[CMD] [2021-04-06 13:04:06] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_media_3646.csv > /app/public/converted_csv/Alaskan_Arthropo_media_3646.csv_sorted
[INFO] [2021-04-06 13:04:08] Converted: /app/public/converted_csv/Alaskan_Arthropo_media_3646.csv (0 lines)
[INFO] [2021-04-06 13:04:08] ...vernaculars (/app/public/data/Alaskan_Arthropo/common names.txt)
[CMD] [2021-04-06 13:04:08] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_vernaculars_3646.csv > /app/public/converted_csv/Alaskan_Arthropo_vernaculars_3646.csv_sorted
[INFO] [2021-04-06 13:04:09] Converted: /app/public/converted_csv/Alaskan_Arthropo_vernaculars_3646.csv (0 lines)
[INFO] [2021-04-06 13:04:09] ...occurrences (/app/public/data/Alaskan_Arthropo/occurrences.txt)
[CMD] [2021-04-06 13:04:09] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_occurrences_3646.csv > /app/public/converted_csv/Alaskan_Arthropo_occurrences_3646.csv_sorted
[INFO] [2021-04-06 13:04:10] Converted: /app/public/converted_csv/Alaskan_Arthropo_occurrences_3646.csv (7682 lines)
[INFO] [2021-04-06 13:04:10] ...assocs (/app/public/data/Alaskan_Arthropo/associations.txt)
[CMD] [2021-04-06 13:04:10] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_assocs_3646.csv > /app/public/converted_csv/Alaskan_Arthropo_assocs_3646.csv_sorted
[INFO] [2021-04-06 13:04:12] Converted: /app/public/converted_csv/Alaskan_Arthropo_assocs_3646.csv (0 lines)
[INFO] [2021-04-06 13:04:12] ...measurements (/app/public/data/Alaskan_Arthropo/measurements or facts.txt)
[CMD] [2021-04-06 13:04:12] /usr/bin/sort /app/public/converted_csv/Alaskan_Arthropo_measurements_3646.csv > /app/public/converted_csv/Alaskan_Arthropo_measurements_3646.csv_sorted
[INFO] [2021-04-06 13:04:13] Converted: /app/public/converted_csv/Alaskan_Arthropo_measurements_3646.csv (7923 lines)
[STOP] [2021-04-06 13:04:13] convert_to_csv
[START] [2021-04-06 13:04:13] calculate_delta
[INFO] [2021-04-06 13:04:13] Looping over 8 formats...
[INFO] [2021-04-06 13:04:13] ...agents (/app/public/data/Alaskan_Arthropo/agents.txt)
[CMD] [2021-04-06 13:04:13] echo "0a" > /app/public/diff/Alaskan_Arthropo_agents_3646.diff
[CMD] [2021-04-06 13:04:14] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_agents_3646.csv >> /app/public/diff/Alaskan_Arthropo_agents_3646.diff
[CMD] [2021-04-06 13:04:16] echo "." >> /app/public/diff/Alaskan_Arthropo_agents_3646.diff
[INFO] [2021-04-06 13:04:17] Created diff: /app/public/diff/Alaskan_Arthropo_agents_3646.diff (2 lines)
[INFO] [2021-04-06 13:04:17] ...refs (/app/public/data/Alaskan_Arthropo/references.txt)
[CMD] [2021-04-06 13:04:17] echo "0a" > /app/public/diff/Alaskan_Arthropo_refs_3646.diff
[CMD] [2021-04-06 13:04:18] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_refs_3646.csv >> /app/public/diff/Alaskan_Arthropo_refs_3646.diff
[CMD] [2021-04-06 13:04:19] echo "." >> /app/public/diff/Alaskan_Arthropo_refs_3646.diff
[INFO] [2021-04-06 13:04:21] Created diff: /app/public/diff/Alaskan_Arthropo_refs_3646.diff (445 lines)
[INFO] [2021-04-06 13:04:21] ...nodes (/app/public/data/Alaskan_Arthropo/taxa.txt)
[CMD] [2021-04-06 13:04:21] echo "0a" > /app/public/diff/Alaskan_Arthropo_nodes_3646.diff
[CMD] [2021-04-06 13:04:22] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_nodes_3646.csv >> /app/public/diff/Alaskan_Arthropo_nodes_3646.diff
[CMD] [2021-04-06 13:04:23] echo "." >> /app/public/diff/Alaskan_Arthropo_nodes_3646.diff
[INFO] [2021-04-06 13:04:25] Created diff: /app/public/diff/Alaskan_Arthropo_nodes_3646.diff (7684 lines)
[INFO] [2021-04-06 13:04:25] ...media (/app/public/data/Alaskan_Arthropo/media.txt)
[CMD] [2021-04-06 13:04:25] echo "0a" > /app/public/diff/Alaskan_Arthropo_media_3646.diff
[CMD] [2021-04-06 13:04:26] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_media_3646.csv >> /app/public/diff/Alaskan_Arthropo_media_3646.diff
[CMD] [2021-04-06 13:04:27] echo "." >> /app/public/diff/Alaskan_Arthropo_media_3646.diff
[INFO] [2021-04-06 13:04:29] Created diff: /app/public/diff/Alaskan_Arthropo_media_3646.diff (2 lines)
[INFO] [2021-04-06 13:04:29] ...vernaculars (/app/public/data/Alaskan_Arthropo/common names.txt)
[CMD] [2021-04-06 13:04:29] echo "0a" > /app/public/diff/Alaskan_Arthropo_vernaculars_3646.diff
[CMD] [2021-04-06 13:04:30] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_vernaculars_3646.csv >> /app/public/diff/Alaskan_Arthropo_vernaculars_3646.diff
[CMD] [2021-04-06 13:04:31] echo "." >> /app/public/diff/Alaskan_Arthropo_vernaculars_3646.diff
[INFO] [2021-04-06 13:04:33] Created diff: /app/public/diff/Alaskan_Arthropo_vernaculars_3646.diff (2 lines)
[INFO] [2021-04-06 13:04:33] ...occurrences (/app/public/data/Alaskan_Arthropo/occurrences.txt)
[CMD] [2021-04-06 13:04:33] echo "0a" > /app/public/diff/Alaskan_Arthropo_occurrences_3646.diff
[CMD] [2021-04-06 13:04:34] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_occurrences_3646.csv >> /app/public/diff/Alaskan_Arthropo_occurrences_3646.diff
[CMD] [2021-04-06 13:04:35] echo "." >> /app/public/diff/Alaskan_Arthropo_occurrences_3646.diff
[INFO] [2021-04-06 13:04:37] Created diff: /app/public/diff/Alaskan_Arthropo_occurrences_3646.diff (7684 lines)
[INFO] [2021-04-06 13:04:37] ...assocs (/app/public/data/Alaskan_Arthropo/associations.txt)
[CMD] [2021-04-06 13:04:37] echo "0a" > /app/public/diff/Alaskan_Arthropo_assocs_3646.diff
[CMD] [2021-04-06 13:04:38] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_assocs_3646.csv >> /app/public/diff/Alaskan_Arthropo_assocs_3646.diff
[CMD] [2021-04-06 13:04:39] echo "." >> /app/public/diff/Alaskan_Arthropo_assocs_3646.diff
[INFO] [2021-04-06 13:04:40] Created diff: /app/public/diff/Alaskan_Arthropo_assocs_3646.diff (2 lines)
[INFO] [2021-04-06 13:04:40] ...measurements (/app/public/data/Alaskan_Arthropo/measurements or facts.txt)
[CMD] [2021-04-06 13:04:40] echo "0a" > /app/public/diff/Alaskan_Arthropo_measurements_3646.diff
[CMD] [2021-04-06 13:04:42] tail -n +1 /app/public/converted_csv/Alaskan_Arthropo_measurements_3646.csv >> /app/public/diff/Alaskan_Arthropo_measurements_3646.diff
[CMD] [2021-04-06 13:04:43] echo "." >> /app/public/diff/Alaskan_Arthropo_measurements_3646.diff
[INFO] [2021-04-06 13:04:44] Created diff: /app/public/diff/Alaskan_Arthropo_measurements_3646.diff (7925 lines)
[STOP] [2021-04-06 13:04:44] calculate_delta
[START] [2021-04-06 13:04:44] parse_diff_and_store
[INFO] [2021-04-06 13:04:44] Handling diff: /app/public/diff/Alaskan_Arthropo_agents_3646.diff (2 lines)
[INFO] [2021-04-06 13:04:45] Loading agents diff file into memory (2 /app/public/diff/Alaskan_Arthropo_agents_3646.diff lines)...
[INFO] [2021-04-06 13:04:47] Handling diff: /app/public/diff/Alaskan_Arthropo_refs_3646.diff (445 lines)
[INFO] [2021-04-06 13:04:48] Loading refs diff file into memory (445 /app/public/diff/Alaskan_Arthropo_refs_3646.diff lines)...
[INFO] [2021-04-06 13:04:49] Handling diff: /app/public/diff/Alaskan_Arthropo_nodes_3646.diff (7684 lines)
[INFO] [2021-04-06 13:04:51] Loading nodes diff file into memory (7684 /app/public/diff/Alaskan_Arthropo_nodes_3646.diff lines)...
[INFO] [2021-04-06 13:04:55] Handling diff: /app/public/diff/Alaskan_Arthropo_media_3646.diff (2 lines)
[INFO] [2021-04-06 13:04:56] Loading media diff file into memory (2 /app/public/diff/Alaskan_Arthropo_media_3646.diff lines)...
[INFO] [2021-04-06 13:04:58] Handling diff: /app/public/diff/Alaskan_Arthropo_vernaculars_3646.diff (2 lines)
[INFO] [2021-04-06 13:04:59] Loading vernaculars diff file into memory (2 /app/public/diff/Alaskan_Arthropo_vernaculars_3646.diff lines)...
[INFO] [2021-04-06 13:05:00] Handling diff: /app/public/diff/Alaskan_Arthropo_occurrences_3646.diff (7684 lines)
[INFO] [2021-04-06 13:05:02] Loading occurrences diff file into memory (7684 /app/public/diff/Alaskan_Arthropo_occurrences_3646.diff lines)...
[INFO] [2021-04-06 13:05:04] Handling diff: /app/public/diff/Alaskan_Arthropo_assocs_3646.diff (2 lines)
[INFO] [2021-04-06 13:05:05] Loading assocs diff file into memory (2 /app/public/diff/Alaskan_Arthropo_assocs_3646.diff lines)...
[INFO] [2021-04-06 13:05:07] Handling diff: /app/public/diff/Alaskan_Arthropo_measurements_3646.diff (7925 lines)
[INFO] [2021-04-06 13:05:08] Loading measurements diff file into memory (7925 /app/public/diff/Alaskan_Arthropo_measurements_3646.diff lines)...
[INFO] [2021-04-06 13:05:13] Storing 443 References
[INFO] [2021-04-06 13:05:13] Processing group of 443 in 1 groups of 1000
[INFO] [2021-04-06 13:05:13] Average Time: 0.1
[INFO] [2021-04-06 13:05:13] Total Time: 1s
[INFO] [2021-04-06 13:05:13] Storing 11116 ScientificNames
[INFO] [2021-04-06 13:05:13] Processing group of 11116 in 12 groups of 1000
[INFO] [2021-04-06 13:05:16] Average Time: 0.269
[INFO] [2021-04-06 13:05:16] Total Time: 4s
[INFO] [2021-04-06 13:05:16] last 3 / first 3: 0.66
[INFO] [2021-04-06 13:05:16] Std.Dev: 0.08366600265340755; Max: 0.41
[INFO] [2021-04-06 13:05:16] Storing 11116 Nodes
[INFO] [2021-04-06 13:05:16] Processing group of 11116 in 12 groups of 1000
[INFO] [2021-04-06 13:05:19] Average Time: 0.241
[INFO] [2021-04-06 13:05:19] Total Time: 3s
[INFO] [2021-04-06 13:05:19] last 3 / first 3: 0.62
[INFO] [2021-04-06 13:05:19] Std.Dev: 0.07745966692414834; Max: 0.33
[INFO] [2021-04-06 13:05:19] Storing 7682 Occurrences
[INFO] [2021-04-06 13:05:19] Processing group of 7682 in 8 groups of 1000
[INFO] [2021-04-06 13:05:20] Average Time: 0.11
[INFO] [2021-04-06 13:05:20] Total Time: 1s
[INFO] [2021-04-06 13:05:20] last 3 / first 3: 1.27
[INFO] [2021-04-06 13:05:20] Std.Dev: 0.03162277660168379; Max: 0.17
[INFO] [2021-04-06 13:05:20] Storing 8258 TraitsReferences
[INFO] [2021-04-06 13:05:20] Processing group of 8258 in 9 groups of 1000
[INFO] [2021-04-06 13:05:21] Average Time: 0.066
[INFO] [2021-04-06 13:05:21] Total Time: 1s
[INFO] [2021-04-06 13:05:21] last 3 / first 3: 0.52
[INFO] [2021-04-06 13:05:21] Std.Dev: 0.03162277660168379; Max: 0.15
[INFO] [2021-04-06 13:05:21] Storing 7923 Traits
[INFO] [2021-04-06 13:05:21] Processing group of 7923 in 8 groups of 1000
[INFO] [2021-04-06 13:05:23] Average Time: 0.286
[INFO] [2021-04-06 13:05:23] Total Time: 3s
[INFO] [2021-04-06 13:05:23] last 3 / first 3: 1.04
[INFO] [2021-04-06 13:05:23] Std.Dev: 0.044721359549995794; Max: 0.38
[INFO] [2021-04-06 13:05:23] Storing 3783 MetaTraits
[INFO] [2021-04-06 13:05:23] Processing group of 3783 in 4 groups of 1000
[INFO] [2021-04-06 13:05:24] Average Time: 0.1
[INFO] [2021-04-06 13:05:24] Total Time: 1s
[STOP] [2021-04-06 13:05:24] parse_diff_and_store
[START] [2021-04-06 13:05:24] resolve_keys
[INFO] [2021-04-06 13:05:51] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-06 13:05:52] traits to occurrences...
[INFO] [2021-04-06 13:05:52] traits to nodes (through occurrences)...
[INFO] [2021-04-06 13:05:52] Traits to sex term...
[INFO] [2021-04-06 13:05:52] Traits to lifestage term...
[INFO] [2021-04-06 13:05:52] MetaTraits to traits...
[INFO] [2021-04-06 13:05:52] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-06 13:05:53] Assocs to occurrences...
[INFO] [2021-04-06 13:05:53] Assocs to nodes...
[INFO] [2021-04-06 13:05:53] Assoc to sex term...
[INFO] [2021-04-06 13:05:53] Assoc to lifestage term...
[INFO] [2021-04-06 13:05:53] MetaAssoc to assocs...
[STOP] [2021-04-06 13:05:53] resolve_keys
[START] [2021-04-06 13:05:53] hold_for_later_1
[STOP] [2021-04-06 13:05:53] hold_for_later_1
[START] [2021-04-06 13:05:53] hold_for_later_2
[STOP] [2021-04-06 13:05:53] hold_for_later_2
[START] [2021-04-06 13:05:53] resolve_missing_parents
[STOP] [2021-04-06 13:05:53] resolve_missing_parents
[START] [2021-04-06 13:05:53] rebuild_nodes
[START] [2021-04-06 13:05:53] Flattener#flatten
[START] [2021-04-06 13:05:53] Flattener#study_resource
[START] [2021-04-06 13:05:53] Flattener#build_ancestry
[STOP] [2021-04-06 13:05:54] Flattener#build_ancestry
[INFO] [2021-04-06 13:05:54] 11116 ancestry keys
[START] [2021-04-06 13:05:54] build_node_ancestors
[INFO] [2021-04-06 13:05:54] old ancestors deleted.
[STOP] [2021-04-06 13:05:56] build_node_ancestors
[START] [2021-04-06 13:05:59] Flattener#propagate_ancestor_ids
[STOP] [2021-04-06 13:06:00] Flattener#propagate_ancestor_ids
[STOP] [2021-04-06 13:06:00] Flattener#flatten
[STOP] [2021-04-06 13:06:00] rebuild_nodes
[START] [2021-04-06 13:06:00] resolve_missing_media_owners
[STOP] [2021-04-06 13:06:00] resolve_missing_media_owners
[START] [2021-04-06 13:06:00] sanitize_media_verbatims
[STOP] [2021-04-06 13:06:00] sanitize_media_verbatims
[START] [2021-04-06 13:06:00] queue_downloads
[STOP] [2021-04-06 13:06:00] queue_downloads
[START] [2021-04-06 13:06:00] parse_names
[WARN] [2021-04-06 13:06:00] I see 11116 names which still need to be parsed.
[WARN] [2021-04-06 13:06:09] I see 54 names which still need to be parsed.
[STOP] [2021-04-06 13:06:10] parse_names
[START] [2021-04-06 13:06:10] denormalize_canonical_names_to_nodes
[STOP] [2021-04-06 13:06:10] denormalize_canonical_names_to_nodes
[START] [2021-04-06 13:06:10] match_nodes
[START] [2021-04-06 13:06:10] map_all_nodes_to_pages
[STOP] [2021-04-06 13:08:13] map_all_nodes_to_pages
[INFO] [2021-04-06 13:08:13] 1078 Unmatched nodes (of 11116)! That's too many to output. Full list in /app/public/data/Alaskan_Arthropo/unmatched_nodes.txt ; First 10: Acronicta lupini (#92578920); Acronicta vulpina (#92578922); Actebia balanitis (#92578948); Alypia langtoni (#92579338); Anarta nigrolunata (#92579487); Anarta trifolii (#92579488); Androloma (#92579539); Androloma maccullochii (#92579540); Apamea amputatrix (#92579684); Apamea cogitata (#92579686)
[START] [2021-04-06 13:08:13] update_nodes
[STOP] [2021-04-06 13:08:17] update_nodes
[STOP] [2021-04-06 13:08:17] match_nodes
[START] [2021-04-06 13:08:17] reindex_search
[STOP] [2021-04-06 13:08:27] reindex_search
[START] [2021-04-06 13:08:27] normalize_units
[STOP] [2021-04-06 13:08:33] normalize_units
[START] [2021-04-06 13:08:33] calculate_statistics
[STOP] [2021-04-06 13:08:34] calculate_statistics
[START] [2021-04-06 13:08:34] complete_harvest_instance
[START] [2021-04-06 13:08:34] overall_tsv_creation
[INFO] [2021-04-06 13:08:34] Processing group of 11116 in 2 batches of 10000
[INFO] [2021-04-06 13:09:42] 7145 Traits (unfiltered)...
[INFO] [2021-04-06 13:11:00] 7145 Traits (filtered)...
[INFO] [2021-04-06 13:11:00] 0 Associations (filtered)...
[INFO] [2021-04-06 13:11:01] 9414 metadata added.
[INFO] [2021-04-06 13:11:01] 0 metadata added.
[INFO] [2021-04-06 13:12:09] 735 Traits (unfiltered)...
[INFO] [2021-04-06 13:12:44] 735 Traits (filtered)...
[INFO] [2021-04-06 13:12:44] 0 Associations (filtered)...
[INFO] [2021-04-06 13:12:44] 800 metadata added.
[INFO] [2021-04-06 13:12:44] 0 metadata added.
[INFO] [2021-04-06 13:13:10] Average Time: 110.3
[INFO] [2021-04-06 13:13:10] Total Time: 4m36s
[STOP] [2021-04-06 13:13:10] overall_tsv_creation
[INFO] [2021-04-06 13:13:10] Done. Check your files:
[INFO] [2021-04-06 13:13:11] (11063 lines) /app/public/data/Alaskan_Arthropo/publish_nodes.tsv
[INFO] [2021-04-06 13:13:13] (50687 lines) /app/public/data/Alaskan_Arthropo/publish_node_ancestors.tsv
[INFO] [2021-04-06 13:13:14] (11116 lines) /app/public/data/Alaskan_Arthropo/publish_scientific_names.tsv
[INFO] [2021-04-06 13:13:15] (7881 lines) /app/public/data/Alaskan_Arthropo/publish_traits.tsv
[INFO] [2021-04-06 13:13:16] (10215 lines) /app/public/data/Alaskan_Arthropo/publish_metadata.tsv
[STOP] [2021-04-06 13:13:17] complete_harvest_instance
[START] [2021-04-06 13:13:17] completed
[STOP] [2021-04-06 13:13:17] completed
[STOP] [2021-04-06 13:13:17] logged process, took 557.45

Latest Process