Harvest for Scleractinia Lifestyle Created 07 Apr 19:37

Stage: completed
Fetched: 07 Apr 19:37
Validated: 07 Apr 19:37
Deltas Created 07 Apr 19:37
Units Normalized: 07 Apr 19:38
Ancestry Built: 07 Apr 19:37
Nodes Matched: 07 Apr 19:38
Names Parsed: 07 Apr 19:38
New Models Stored: 07 Apr 19:37
Indexed: 07 Apr 19:38
Completed: 07 Apr 19:40
Time to Harvest: less than a minute

Harvesting Log (most recent first)

# Logfile created on 2021-04-07 17:36:00 -0400 by logger.rb/66358
[INFO] [2021-04-07 17:36:00] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-07 18:02:13] ## remove_type: ScientificName
[INFO] [2021-04-07 18:02:13] ++ Calling delete_all on 1430 instances...
[INFO] [2021-04-07 18:02:13] [18:02:13.293] Removed 1430 Scientificnames
[INFO] [2021-04-07 18:02:13] ## remove_type: Vernacular
[INFO] [2021-04-07 18:02:13] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 18:02:13] [18:02:13.295] Removed 0 Vernaculars
[INFO] [2021-04-07 18:02:13] ## remove_type: Article
[INFO] [2021-04-07 18:02:13] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 18:02:13] [18:02:13.296] Removed 0 Articles
[INFO] [2021-04-07 18:02:13] ## remove_type: Medium
[INFO] [2021-04-07 18:02:13] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 18:02:13] [18:02:13.298] Removed 0 Media
[INFO] [2021-04-07 18:02:13] ## remove_type: Trait
[INFO] [2021-04-07 18:02:13] ++ Calling delete_all on 2248 instances...
[INFO] [2021-04-07 18:02:13] [18:02:13.658] Removed 2248 Traits
[INFO] [2021-04-07 18:02:13] ## remove_type: MetaTrait
[INFO] [2021-04-07 18:02:13] ++ Calling delete_all on 4496 instances...
[INFO] [2021-04-07 18:02:13] [18:02:13.877] Removed 4496 Metatraits
[INFO] [2021-04-07 18:02:13] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-07 18:02:13] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 18:02:13] [18:02:13.879] Removed 0 Occurrencemetadata
[INFO] [2021-04-07 18:02:13] ## remove_type: Assoc
[INFO] [2021-04-07 18:02:13] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 18:02:13] [18:02:13.880] Removed 0 Assocs
[INFO] [2021-04-07 18:02:13] ## remove_type: MetaAssoc
[INFO] [2021-04-07 18:02:13] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 18:02:13] [18:02:13.882] Removed 0 Metaassocs
[INFO] [2021-04-07 18:02:13] ## remove_type: Identifier
[INFO] [2021-04-07 18:02:13] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 18:02:13] [18:02:13.883] Removed 0 Identifiers
[INFO] [2021-04-07 18:02:13] ## remove_type: Reference
[INFO] [2021-04-07 18:02:13] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 18:02:13] [18:02:13.885] Removed 0 References
[INFO] [2021-04-07 18:02:13] Starting batch with ID 32913449...
[INFO] [2021-04-07 18:02:14] Starting batch with ID 32913449...
[INFO] [2021-04-07 18:02:14] Starting batch with ID 32913449...
[INFO] [2021-04-07 18:02:14] Starting batch with ID 32913449...
[INFO] [2021-04-07 18:02:14] Starting batch with ID 32913449...
[INFO] [2021-04-07 18:02:14] Starting batch with ID 32913449...
[INFO] [2021-04-07 18:02:15] Starting batch with ID 32913449...
[INFO] [2021-04-07 18:02:15] Starting batch with ID 32913449...
[INFO] [2021-04-07 18:02:15] ## remove_type: Node
[INFO] [2021-04-07 18:02:15] ++ Calling delete_all on 1430 instances...
[INFO] [2021-04-07 18:02:15] [18:02:15.418] Removed 1430 Nodes
[START] [2021-04-07 18:02:16] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 18:02:16] Creating resource from OpenData
[START] [2021-04-07 18:02:17] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 18:02:17] Parse meta.xml file and create formats with fields
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (media) field header: ThumbnailURL term: http://eol.org/schema/media/thumbnailURL
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (media) field header: DerivedFrom term: http://rs.tdwg.org/ac/terms/derivedFrom
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (media) field header: Modified term: http://purl.org/dc/terms/modified
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (media) field header: Rating term: http://ns.adobe.com/xap/1.0/Rating
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (media) field header: Audience term: http://purl.org/dc/terms/audience
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (media) field header: Rights term: http://purl.org/dc/terms/rights
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (media) field header: Publisher term: http://purl.org/dc/terms/publisher
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (media) field header: Contributor term: http://purl.org/dc/terms/contributor
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (media) field header: Creator term: http://purl.org/dc/terms/creator
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (media) field header: GenericLocation term: http://purl.org/dc/terms/spatial
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (media) field header: Latitude term: http://www.w3.org/2003/01/geo/wgs84_pos#lat
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (media) field header: Longitude term: http://www.w3.org/2003/01/geo/wgs84_pos#long
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (media) field header: Altitude term: http://www.w3.org/2003/01/geo/wgs84_pos#alt
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (occurrences) field header: Institution Code term: http://rs.tdwg.org/dwc/terms/institutionCode
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (occurrences) field header: Collection Code term: http://rs.tdwg.org/dwc/terms/collectionCode
[WARN] [2021-04-07 18:02:31] (common) IGNORED  (occurrences) field header: Catalog Number term: http://rs.tdwg.org/dwc/terms/catalogNumber
[WARN] [2021-04-07 18:02:31] SKIPPING http://rs.tdwg.org/dwc/terms/Event config (events.txt)...
[STOP] [2021-04-07 18:02:31] Parse meta.xml file and create formats with fields
[STOP] [2021-04-07 18:02:31] Creating resource from OpenData
[START] [2021-04-07 18:02:31] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 18:02:31] create_harvest_instance
[INFO] [2021-04-07 18:02:31] Created harvest instance #3694
[STOP] [2021-04-07 18:02:31] create_harvest_instance
[START] [2021-04-07 18:02:31] fetch_files
[STOP] [2021-04-07 18:02:31] fetch_files
[START] [2021-04-07 18:02:31] validate_each_file
[INFO] [2021-04-07 18:02:31] Looping over 8 formats...
[INFO] [2021-04-07 18:02:31] ...agents (/app/public/data/scleractinia/agents.txt)
[INFO] [2021-04-07 18:02:31] Valid: /app/public/converted_csv/scleractinia_agents_3694.csv (0 lines)
[INFO] [2021-04-07 18:02:31] ...refs (/app/public/data/scleractinia/references.txt)
[INFO] [2021-04-07 18:02:31] Valid: /app/public/converted_csv/scleractinia_refs_3694.csv (0 lines)
[INFO] [2021-04-07 18:02:31] ...nodes (/app/public/data/scleractinia/taxa.txt)
[INFO] [2021-04-07 18:02:31] Valid: /app/public/converted_csv/scleractinia_nodes_3694.csv (1426 lines)
[INFO] [2021-04-07 18:02:31] ...media (/app/public/data/scleractinia/media.txt)
[INFO] [2021-04-07 18:02:31] Valid: /app/public/converted_csv/scleractinia_media_3694.csv (0 lines)
[INFO] [2021-04-07 18:02:31] ...vernaculars (/app/public/data/scleractinia/common names.txt)
[INFO] [2021-04-07 18:02:31] Valid: /app/public/converted_csv/scleractinia_vernaculars_3694.csv (0 lines)
[INFO] [2021-04-07 18:02:31] ...occurrences (/app/public/data/scleractinia/occurrences.txt)
[INFO] [2021-04-07 18:02:31] Valid: /app/public/converted_csv/scleractinia_occurrences_3694.csv (1426 lines)
[INFO] [2021-04-07 18:02:31] ...assocs (/app/public/data/scleractinia/associations.txt)
[INFO] [2021-04-07 18:02:32] Valid: /app/public/converted_csv/scleractinia_assocs_3694.csv (0 lines)
[INFO] [2021-04-07 18:02:32] ...measurements (/app/public/data/scleractinia/measurements or facts.txt)
[INFO] [2021-04-07 18:02:32] Valid: /app/public/converted_csv/scleractinia_measurements_3694.csv (2248 lines)
[STOP] [2021-04-07 18:02:32] validate_each_file
[START] [2021-04-07 18:02:32] convert_to_csv
[INFO] [2021-04-07 18:02:32] Looping over 8 formats...
[INFO] [2021-04-07 18:02:32] ...agents (/app/public/data/scleractinia/agents.txt)
[CMD] [2021-04-07 18:02:32] /usr/bin/sort /app/public/converted_csv/scleractinia_agents_3694.csv > /app/public/converted_csv/scleractinia_agents_3694.csv_sorted
[INFO] [2021-04-07 18:02:32] Converted: /app/public/converted_csv/scleractinia_agents_3694.csv (0 lines)
[INFO] [2021-04-07 18:02:32] ...refs (/app/public/data/scleractinia/references.txt)
[CMD] [2021-04-07 18:02:32] /usr/bin/sort /app/public/converted_csv/scleractinia_refs_3694.csv > /app/public/converted_csv/scleractinia_refs_3694.csv_sorted
[INFO] [2021-04-07 18:02:33] Converted: /app/public/converted_csv/scleractinia_refs_3694.csv (0 lines)
[INFO] [2021-04-07 18:02:33] ...nodes (/app/public/data/scleractinia/taxa.txt)
[CMD] [2021-04-07 18:02:33] /usr/bin/sort /app/public/converted_csv/scleractinia_nodes_3694.csv > /app/public/converted_csv/scleractinia_nodes_3694.csv_sorted
[INFO] [2021-04-07 18:02:33] Converted: /app/public/converted_csv/scleractinia_nodes_3694.csv (1426 lines)
[INFO] [2021-04-07 18:02:33] ...media (/app/public/data/scleractinia/media.txt)
[CMD] [2021-04-07 18:02:33] /usr/bin/sort /app/public/converted_csv/scleractinia_media_3694.csv > /app/public/converted_csv/scleractinia_media_3694.csv_sorted
[INFO] [2021-04-07 18:02:34] Converted: /app/public/converted_csv/scleractinia_media_3694.csv (0 lines)
[INFO] [2021-04-07 18:02:34] ...vernaculars (/app/public/data/scleractinia/common names.txt)
[CMD] [2021-04-07 18:02:34] /usr/bin/sort /app/public/converted_csv/scleractinia_vernaculars_3694.csv > /app/public/converted_csv/scleractinia_vernaculars_3694.csv_sorted
[INFO] [2021-04-07 18:02:34] Converted: /app/public/converted_csv/scleractinia_vernaculars_3694.csv (0 lines)
[INFO] [2021-04-07 18:02:34] ...occurrences (/app/public/data/scleractinia/occurrences.txt)
[CMD] [2021-04-07 18:02:34] /usr/bin/sort /app/public/converted_csv/scleractinia_occurrences_3694.csv > /app/public/converted_csv/scleractinia_occurrences_3694.csv_sorted
[INFO] [2021-04-07 18:02:35] Converted: /app/public/converted_csv/scleractinia_occurrences_3694.csv (1426 lines)
[INFO] [2021-04-07 18:02:35] ...assocs (/app/public/data/scleractinia/associations.txt)
[CMD] [2021-04-07 18:02:35] /usr/bin/sort /app/public/converted_csv/scleractinia_assocs_3694.csv > /app/public/converted_csv/scleractinia_assocs_3694.csv_sorted
[INFO] [2021-04-07 18:02:35] Converted: /app/public/converted_csv/scleractinia_assocs_3694.csv (0 lines)
[INFO] [2021-04-07 18:02:35] ...measurements (/app/public/data/scleractinia/measurements or facts.txt)
[CMD] [2021-04-07 18:02:35] /usr/bin/sort /app/public/converted_csv/scleractinia_measurements_3694.csv > /app/public/converted_csv/scleractinia_measurements_3694.csv_sorted
[INFO] [2021-04-07 18:02:36] Converted: /app/public/converted_csv/scleractinia_measurements_3694.csv (2248 lines)
[STOP] [2021-04-07 18:02:36] convert_to_csv
[START] [2021-04-07 18:02:36] calculate_delta
[INFO] [2021-04-07 18:02:36] Looping over 8 formats...
[INFO] [2021-04-07 18:02:36] ...agents (/app/public/data/scleractinia/agents.txt)
[CMD] [2021-04-07 18:02:36] echo "0a" > /app/public/diff/scleractinia_agents_3694.diff
[CMD] [2021-04-07 18:02:37] tail -n +1 /app/public/converted_csv/scleractinia_agents_3694.csv >> /app/public/diff/scleractinia_agents_3694.diff
[CMD] [2021-04-07 18:02:37] echo "." >> /app/public/diff/scleractinia_agents_3694.diff
[INFO] [2021-04-07 18:02:38] Created diff: /app/public/diff/scleractinia_agents_3694.diff (2 lines)
[INFO] [2021-04-07 18:02:38] ...refs (/app/public/data/scleractinia/references.txt)
[CMD] [2021-04-07 18:02:38] echo "0a" > /app/public/diff/scleractinia_refs_3694.diff
[CMD] [2021-04-07 18:02:38] tail -n +1 /app/public/converted_csv/scleractinia_refs_3694.csv >> /app/public/diff/scleractinia_refs_3694.diff
[CMD] [2021-04-07 18:02:39] echo "." >> /app/public/diff/scleractinia_refs_3694.diff
[INFO] [2021-04-07 18:02:39] Created diff: /app/public/diff/scleractinia_refs_3694.diff (2 lines)
[INFO] [2021-04-07 18:02:39] ...nodes (/app/public/data/scleractinia/taxa.txt)
[CMD] [2021-04-07 18:02:39] echo "0a" > /app/public/diff/scleractinia_nodes_3694.diff
[CMD] [2021-04-07 18:02:40] tail -n +1 /app/public/converted_csv/scleractinia_nodes_3694.csv >> /app/public/diff/scleractinia_nodes_3694.diff
[CMD] [2021-04-07 18:02:40] echo "." >> /app/public/diff/scleractinia_nodes_3694.diff
[INFO] [2021-04-07 18:02:41] Created diff: /app/public/diff/scleractinia_nodes_3694.diff (1428 lines)
[INFO] [2021-04-07 18:02:41] ...media (/app/public/data/scleractinia/media.txt)
[CMD] [2021-04-07 18:02:41] echo "0a" > /app/public/diff/scleractinia_media_3694.diff
[CMD] [2021-04-07 18:02:41] tail -n +1 /app/public/converted_csv/scleractinia_media_3694.csv >> /app/public/diff/scleractinia_media_3694.diff
[CMD] [2021-04-07 18:02:42] echo "." >> /app/public/diff/scleractinia_media_3694.diff
[INFO] [2021-04-07 18:02:42] Created diff: /app/public/diff/scleractinia_media_3694.diff (2 lines)
[INFO] [2021-04-07 18:02:42] ...vernaculars (/app/public/data/scleractinia/common names.txt)
[CMD] [2021-04-07 18:02:42] echo "0a" > /app/public/diff/scleractinia_vernaculars_3694.diff
[CMD] [2021-04-07 18:02:43] tail -n +1 /app/public/converted_csv/scleractinia_vernaculars_3694.csv >> /app/public/diff/scleractinia_vernaculars_3694.diff
[CMD] [2021-04-07 18:02:44] echo "." >> /app/public/diff/scleractinia_vernaculars_3694.diff
[INFO] [2021-04-07 18:02:44] Created diff: /app/public/diff/scleractinia_vernaculars_3694.diff (2 lines)
[INFO] [2021-04-07 18:02:44] ...occurrences (/app/public/data/scleractinia/occurrences.txt)
[CMD] [2021-04-07 18:02:44] echo "0a" > /app/public/diff/scleractinia_occurrences_3694.diff
[CMD] [2021-04-07 18:02:45] tail -n +1 /app/public/converted_csv/scleractinia_occurrences_3694.csv >> /app/public/diff/scleractinia_occurrences_3694.diff
[CMD] [2021-04-07 18:02:45] echo "." >> /app/public/diff/scleractinia_occurrences_3694.diff
[INFO] [2021-04-07 18:02:46] Created diff: /app/public/diff/scleractinia_occurrences_3694.diff (1428 lines)
[INFO] [2021-04-07 18:02:46] ...assocs (/app/public/data/scleractinia/associations.txt)
[CMD] [2021-04-07 18:02:46] echo "0a" > /app/public/diff/scleractinia_assocs_3694.diff
[CMD] [2021-04-07 18:02:46] tail -n +1 /app/public/converted_csv/scleractinia_assocs_3694.csv >> /app/public/diff/scleractinia_assocs_3694.diff
[CMD] [2021-04-07 18:02:47] echo "." >> /app/public/diff/scleractinia_assocs_3694.diff
[INFO] [2021-04-07 18:02:47] Created diff: /app/public/diff/scleractinia_assocs_3694.diff (2 lines)
[INFO] [2021-04-07 18:02:47] ...measurements (/app/public/data/scleractinia/measurements or facts.txt)
[CMD] [2021-04-07 18:02:47] echo "0a" > /app/public/diff/scleractinia_measurements_3694.diff
[CMD] [2021-04-07 18:02:48] tail -n +1 /app/public/converted_csv/scleractinia_measurements_3694.csv >> /app/public/diff/scleractinia_measurements_3694.diff
[CMD] [2021-04-07 18:02:48] echo "." >> /app/public/diff/scleractinia_measurements_3694.diff
[INFO] [2021-04-07 18:02:49] Created diff: /app/public/diff/scleractinia_measurements_3694.diff (2250 lines)
[STOP] [2021-04-07 18:02:49] calculate_delta
[START] [2021-04-07 18:02:49] parse_diff_and_store
[INFO] [2021-04-07 18:02:49] Handling diff: /app/public/diff/scleractinia_agents_3694.diff (2 lines)
[INFO] [2021-04-07 18:02:50] Loading agents diff file into memory (2 /app/public/diff/scleractinia_agents_3694.diff lines)...
[INFO] [2021-04-07 18:02:50] Handling diff: /app/public/diff/scleractinia_refs_3694.diff (2 lines)
[INFO] [2021-04-07 18:02:51] Loading refs diff file into memory (2 /app/public/diff/scleractinia_refs_3694.diff lines)...
[INFO] [2021-04-07 18:02:51] Handling diff: /app/public/diff/scleractinia_nodes_3694.diff (1428 lines)
[INFO] [2021-04-07 18:02:52] Loading nodes diff file into memory (1428 /app/public/diff/scleractinia_nodes_3694.diff lines)...
[WARN] [2021-04-07 18:02:52] Filtered Scientific Name `Aulocyathus  juvenescens Marenzeller, 1904` to `Aulocyathus juvenescens Marenzeller, 1904`
[WARN] [2021-04-07 18:02:52] Filtered Scientific Name `Cladangia  exusta Lütken, 1873` to `Cladangia exusta Lütken, 1873`
[WARN] [2021-04-07 18:02:52] Filtered Scientific Name `Coenosmilia  inordinata Cairns, 1984` to `Coenosmilia inordinata Cairns, 1984`
[WARN] [2021-04-07 18:02:52] Filtered Scientific Name `Culicia  stellata Dana, 1846` to `Culicia stellata Dana, 1846`
[WARN] [2021-04-07 18:02:52] Filtered Scientific Name `Flabellum  pavoninum Lesson, 1831` to `Flabellum pavoninum Lesson, 1831`
[WARN] [2021-04-07 18:02:52] Filtered Scientific Name `Flabellum curvatum  Moseley, 1881` to `Flabellum curvatum Moseley, 1881`
[WARN] [2021-04-07 18:02:52] Filtered Scientific Name `Heterocyathus antoniae  Reyes, Santodomingo & Cairns, 2009` to `Heterocyathus antoniae Reyes, Santodomingo & Cairns, 2009`
[WARN] [2021-04-07 18:02:52] Filtered Scientific Name `Javania  insignis Duncan, 1876` to `Javania insignis Duncan, 1876`
[WARN] [2021-04-07 18:02:52] Filtered Scientific Name `Labyrinthocyathus  langae Cairns, 1979: 341` to `Labyrinthocyathus langae Cairns, 1979: 341`
[WARN] [2021-04-07 18:02:53] Filtered Scientific Name `Rhizosmilia  gerdae Cairns, 1978` to `Rhizosmilia gerdae Cairns, 1978`
[WARN] [2021-04-07 18:02:53] Filtered Scientific Name `Solenosmilia  variabilis Duncan, 1873` to `Solenosmilia variabilis Duncan, 1873`
[WARN] [2021-04-07 18:02:53] Filtered Scientific Name `Stephanocyathus (Stephanocyathus) isabellae  Reyes, Santodomingo & Cairns, 2009` to `Stephanocyathus (Stephanocyathus) isabellae Reyes, Santodomingo & Cairns, 2009`
[WARN] [2021-04-07 18:02:53] Filtered Scientific Name `Trochocyathus (Trochocyathus)  aithoseptatus Cairns, 1984` to `Trochocyathus (Trochocyathus) aithoseptatus Cairns, 1984`
[WARN] [2021-04-07 18:02:53] Filtered Scientific Name `Trochocyathus (Trochocyathus)  apertus Cairns & Zibrowius, 1997` to `Trochocyathus (Trochocyathus) apertus Cairns & Zibrowius, 1997`
[WARN] [2021-04-07 18:02:53] Filtered Scientific Name `Trochocyathus (Trochocyathus)  caryophylloides Alcock, 1902` to `Trochocyathus (Trochocyathus) caryophylloides Alcock, 1902`
[WARN] [2021-04-07 18:02:53] Filtered Scientific Name `Trochocyathus (Trochocyathus)  cepulla Cairns, 1995` to `Trochocyathus (Trochocyathus) cepulla Cairns, 1995`
[WARN] [2021-04-07 18:02:53] Filtered Scientific Name `Trochocyathus (Trochocyathus)  decamera Cairns, 1994` to `Trochocyathus (Trochocyathus) decamera Cairns, 1994`
[WARN] [2021-04-07 18:02:53] Filtered Scientific Name `Trochocyathus (Trochocyathus)  discus Cairns & Zibrowius, 1997` to `Trochocyathus (Trochocyathus) discus Cairns & Zibrowius, 1997`
[WARN] [2021-04-07 18:02:53] Filtered Scientific Name `Trochocyathus (Trochocyathus)  efateensis Cairns, 1999` to `Trochocyathus (Trochocyathus) efateensis Cairns, 1999`
[WARN] [2021-04-07 18:02:53] Filtered Scientific Name `Trochocyathus (Trochocyathus)  fasciatus Cairns, 1979` to `Trochocyathus (Trochocyathus) fasciatus Cairns, 1979`
[WARN] [2021-04-07 18:02:53] Filtered Scientific Name `Trochocyathus (Trochocyathus)  fossulus Cairns, 1979` to `Trochocyathus (Trochocyathus) fossulus Cairns, 1979`
[WARN] [2021-04-07 18:02:53] Filtered Scientific Name `Trochocyathus (Trochocyathus)  gordoni Cairns, 1995` to `Trochocyathus (Trochocyathus) gordoni Cairns, 1995`
[WARN] [2021-04-07 18:02:53] Filtered Scientific Name `Trochocyathus (Trochocyathus)  japonicus Eguchi, 1968` to `Trochocyathus (Trochocyathus) japonicus Eguchi, 1968`
[WARN] [2021-04-07 18:02:53] Filtered Scientific Name `Trochocyathus (Trochocyathus)  laboreli Cairns, 2000` to `Trochocyathus (Trochocyathus) laboreli Cairns, 2000`
[WARN] [2021-04-07 18:02:53] Filtered Scientific Name `Trochocyathus (Trochocyathus)  maculatus Cairns, 1995` to `Trochocyathus (Trochocyathus) maculatus Cairns, 1995`
[WARN] [2021-04-07 18:02:53] Filtered Scientific Name `Trochocyathus (Trochocyathus)  oahensis Vaughan, 1907` to `Trochocyathus (Trochocyathus) oahensis Vaughan, 1907`
[WARN] [2021-04-07 18:02:53] Filtered Scientific Name `Trochocyathus (Trochocyathus)  patelliformis Cairns, 1999` to `Trochocyathus (Trochocyathus) patelliformis Cairns, 1999`
[WARN] [2021-04-07 18:02:53] Filtered Scientific Name `Trochocyathus (Trochocyathus)  philippinensis Semper, 1872` to `Trochocyathus (Trochocyathus) philippinensis Semper, 1872`
[WARN] [2021-04-07 18:02:53] Filtered Scientific Name `Trochocyathus (Trochocyathus)  rawsonii Pourtalès, 1874` to `Trochocyathus (Trochocyathus) rawsonii Pourtalès, 1874`
[WARN] [2021-04-07 18:02:53] (Reached filtered-name limit; supressing further warnings.)
[INFO] [2021-04-07 18:02:53] Handling diff: /app/public/diff/scleractinia_media_3694.diff (2 lines)
[INFO] [2021-04-07 18:02:53] Loading media diff file into memory (2 /app/public/diff/scleractinia_media_3694.diff lines)...
[INFO] [2021-04-07 18:02:54] Handling diff: /app/public/diff/scleractinia_vernaculars_3694.diff (2 lines)
[INFO] [2021-04-07 18:02:54] Loading vernaculars diff file into memory (2 /app/public/diff/scleractinia_vernaculars_3694.diff lines)...
[INFO] [2021-04-07 18:02:55] Handling diff: /app/public/diff/scleractinia_occurrences_3694.diff (1428 lines)
[INFO] [2021-04-07 18:02:56] Loading occurrences diff file into memory (1428 /app/public/diff/scleractinia_occurrences_3694.diff lines)...
[INFO] [2021-04-07 18:02:56] Handling diff: /app/public/diff/scleractinia_assocs_3694.diff (2 lines)
[INFO] [2021-04-07 18:02:57] Loading assocs diff file into memory (2 /app/public/diff/scleractinia_assocs_3694.diff lines)...
[INFO] [2021-04-07 18:02:57] Handling diff: /app/public/diff/scleractinia_measurements_3694.diff (2250 lines)
[INFO] [2021-04-07 18:02:58] Loading measurements diff file into memory (2250 /app/public/diff/scleractinia_measurements_3694.diff lines)...
[INFO] [2021-04-07 18:02:59] Storing 1430 ScientificNames
[INFO] [2021-04-07 18:02:59] Processing group of 1430 in 2 groups of 1000
[INFO] [2021-04-07 18:03:00] Average Time: 0.225
[INFO] [2021-04-07 18:03:00] Total Time: 1s
[INFO] [2021-04-07 18:03:00] Storing 1430 Nodes
[INFO] [2021-04-07 18:03:00] Processing group of 1430 in 2 groups of 1000
[INFO] [2021-04-07 18:03:00] Average Time: 0.255
[INFO] [2021-04-07 18:03:00] Total Time: 1s
[INFO] [2021-04-07 18:03:00] Storing 1426 Occurrences
[INFO] [2021-04-07 18:03:00] Processing group of 1426 in 2 groups of 1000
[INFO] [2021-04-07 18:03:01] Average Time: 0.08
[INFO] [2021-04-07 18:03:01] Total Time: 1s
[INFO] [2021-04-07 18:03:01] Storing 2248 Traits
[INFO] [2021-04-07 18:03:01] Processing group of 2248 in 3 groups of 1000
[INFO] [2021-04-07 18:03:01] Average Time: 0.203
[INFO] [2021-04-07 18:03:01] Total Time: 1s
[INFO] [2021-04-07 18:03:01] Storing 2248 MetaTraits
[INFO] [2021-04-07 18:03:01] Processing group of 2248 in 3 groups of 1000
[INFO] [2021-04-07 18:03:01] Average Time: 0.083
[INFO] [2021-04-07 18:03:01] Total Time: 1s
[STOP] [2021-04-07 18:03:01] parse_diff_and_store
[START] [2021-04-07 18:03:01] resolve_keys
[INFO] [2021-04-07 18:03:07] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-07 18:03:07] traits to occurrences...
[INFO] [2021-04-07 18:03:07] traits to nodes (through occurrences)...
[INFO] [2021-04-07 18:03:07] Traits to sex term...
[INFO] [2021-04-07 18:03:08] Traits to lifestage term...
[INFO] [2021-04-07 18:03:08] MetaTraits to traits...
[INFO] [2021-04-07 18:03:08] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-07 18:03:08] Assocs to occurrences...
[INFO] [2021-04-07 18:03:08] Assocs to nodes...
[INFO] [2021-04-07 18:03:08] Assoc to sex term...
[INFO] [2021-04-07 18:03:08] Assoc to lifestage term...
[INFO] [2021-04-07 18:03:08] MetaAssoc to assocs...
[STOP] [2021-04-07 18:03:08] resolve_keys
[START] [2021-04-07 18:03:08] hold_for_later_1
[STOP] [2021-04-07 18:03:08] hold_for_later_1
[START] [2021-04-07 18:03:08] hold_for_later_2
[STOP] [2021-04-07 18:03:08] hold_for_later_2
[START] [2021-04-07 18:03:08] resolve_missing_parents
[STOP] [2021-04-07 18:03:08] resolve_missing_parents
[START] [2021-04-07 18:03:08] rebuild_nodes
[START] [2021-04-07 18:03:08] Flattener#flatten
[START] [2021-04-07 18:03:08] Flattener#study_resource
[START] [2021-04-07 18:03:08] Flattener#build_ancestry
[STOP] [2021-04-07 18:03:08] Flattener#build_ancestry
[INFO] [2021-04-07 18:03:08] 1430 ancestry keys
[START] [2021-04-07 18:03:08] build_node_ancestors
[INFO] [2021-04-07 18:03:08] old ancestors deleted.
[STOP] [2021-04-07 18:03:08] build_node_ancestors
[START] [2021-04-07 18:03:08] Flattener#propagate_ancestor_ids
[STOP] [2021-04-07 18:03:08] Flattener#propagate_ancestor_ids
[STOP] [2021-04-07 18:03:08] Flattener#flatten
[STOP] [2021-04-07 18:03:08] rebuild_nodes
[START] [2021-04-07 18:03:08] resolve_missing_media_owners
[STOP] [2021-04-07 18:03:08] resolve_missing_media_owners
[START] [2021-04-07 18:03:08] sanitize_media_verbatims
[STOP] [2021-04-07 18:03:08] sanitize_media_verbatims
[START] [2021-04-07 18:03:08] queue_downloads
[STOP] [2021-04-07 18:03:08] queue_downloads
[START] [2021-04-07 18:03:08] parse_names
[WARN] [2021-04-07 18:03:08] I see 1430 names which still need to be parsed.
[STOP] [2021-04-07 18:03:11] parse_names
[START] [2021-04-07 18:03:11] denormalize_canonical_names_to_nodes
[STOP] [2021-04-07 18:03:11] denormalize_canonical_names_to_nodes
[START] [2021-04-07 18:03:11] match_nodes
[START] [2021-04-07 18:03:11] map_all_nodes_to_pages
[STOP] [2021-04-07 18:03:56] map_all_nodes_to_pages
[INFO] [2021-04-07 18:03:56] 23 Unmatched nodes (of 1430)! That's too many to output. Full list in /app/public/data/scleractinia/unmatched_nodes.txt ; First 10: Acanthastrea rotundaflora (#92667662); Acropora jacquelinae (#92667711); Acropora lutkeni (#92667723); Acropora mirabilis (#92667729); Acropora pruinosa (#92667750); Acropora teres (#92667776); Coscinaraea hazimanensis (#92668022); Deltocyathus agassizii (#92668059); Deltocyathus inusitatus (#92668067); Dendrophyllia boschmai cyathoheloides (#92668084)
[START] [2021-04-07 18:03:56] update_nodes
[STOP] [2021-04-07 18:03:57] update_nodes
[STOP] [2021-04-07 18:03:57] match_nodes
[START] [2021-04-07 18:03:57] reindex_search
[STOP] [2021-04-07 18:03:58] reindex_search
[START] [2021-04-07 18:03:58] normalize_units
[STOP] [2021-04-07 18:03:58] normalize_units
[START] [2021-04-07 18:03:58] calculate_statistics
[STOP] [2021-04-07 18:03:58] calculate_statistics
[START] [2021-04-07 18:03:58] complete_harvest_instance
[START] [2021-04-07 18:03:58] overall_tsv_creation
[INFO] [2021-04-07 18:03:58] Processing group of 1430 in 1 batches of 10000
[INFO] [2021-04-07 18:04:39] 471 Traits (unfiltered)...
[INFO] [2021-04-07 18:05:14] 471 Traits (filtered)...
[INFO] [2021-04-07 18:05:14] 0 Associations (filtered)...
[INFO] [2021-04-07 18:05:14] 471 metadata added.
[INFO] [2021-04-07 18:05:14] 0 metadata added.
[INFO] [2021-04-07 18:05:41] Average Time: 78.05
[INFO] [2021-04-07 18:05:41] Total Time: 1m43s
[STOP] [2021-04-07 18:05:41] overall_tsv_creation
[INFO] [2021-04-07 18:05:41] Done. Check your files:
[INFO] [2021-04-07 18:05:41] (1430 lines) /app/public/data/scleractinia/publish_nodes.tsv
[INFO] [2021-04-07 18:05:42] (5710 lines) /app/public/data/scleractinia/publish_node_ancestors.tsv
[INFO] [2021-04-07 18:05:42] (1430 lines) /app/public/data/scleractinia/publish_scientific_names.tsv
[INFO] [2021-04-07 18:05:43] (472 lines) /app/public/data/scleractinia/publish_traits.tsv
[INFO] [2021-04-07 18:05:44] (472 lines) /app/public/data/scleractinia/publish_metadata.tsv
[STOP] [2021-04-07 18:05:44] complete_harvest_instance
[START] [2021-04-07 18:05:44] completed
[STOP] [2021-04-07 18:05:44] completed
[STOP] [2021-04-07 18:05:44] logged process, took 192.94
[INFO] [2021-04-07 19:18:54] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-07 19:18:56] ## remove_type: ScientificName
[INFO] [2021-04-07 19:18:56] ++ Calling delete_all on 1430 instances...
[INFO] [2021-04-07 19:18:57] [19:18:57.105] Removed 1430 Scientificnames
[INFO] [2021-04-07 19:18:57] ## remove_type: Vernacular
[INFO] [2021-04-07 19:18:57] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 19:18:57] [19:18:57.107] Removed 0 Vernaculars
[INFO] [2021-04-07 19:18:57] ## remove_type: Article
[INFO] [2021-04-07 19:18:57] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 19:18:57] [19:18:57.109] Removed 0 Articles
[INFO] [2021-04-07 19:18:57] ## remove_type: Medium
[INFO] [2021-04-07 19:18:57] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 19:18:57] [19:18:57.110] Removed 0 Media
[INFO] [2021-04-07 19:18:57] ## remove_type: Trait
[INFO] [2021-04-07 19:18:57] ++ Calling delete_all on 2248 instances...
[INFO] [2021-04-07 19:18:57] [19:18:57.501] Removed 2248 Traits
[INFO] [2021-04-07 19:18:57] ## remove_type: MetaTrait
[INFO] [2021-04-07 19:18:57] ++ Calling delete_all on 2248 instances...
[INFO] [2021-04-07 19:18:57] [19:18:57.588] Removed 2248 Metatraits
[INFO] [2021-04-07 19:18:57] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-07 19:18:57] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 19:18:57] [19:18:57.590] Removed 0 Occurrencemetadata
[INFO] [2021-04-07 19:18:57] ## remove_type: Assoc
[INFO] [2021-04-07 19:18:57] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 19:18:57] [19:18:57.591] Removed 0 Assocs
[INFO] [2021-04-07 19:18:57] ## remove_type: MetaAssoc
[INFO] [2021-04-07 19:18:57] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 19:18:57] [19:18:57.592] Removed 0 Metaassocs
[INFO] [2021-04-07 19:18:57] ## remove_type: Identifier
[INFO] [2021-04-07 19:18:57] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 19:18:57] [19:18:57.594] Removed 0 Identifiers
[INFO] [2021-04-07 19:18:57] ## remove_type: Reference
[INFO] [2021-04-07 19:18:57] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 19:18:57] [19:18:57.595] Removed 0 References
[INFO] [2021-04-07 19:18:57] Starting batch with ID 92667650...
[INFO] [2021-04-07 19:18:57] Starting batch with ID 92667650...
[INFO] [2021-04-07 19:18:57] Starting batch with ID 92667650...
[INFO] [2021-04-07 19:18:58] Starting batch with ID 92667650...
[INFO] [2021-04-07 19:18:58] Starting batch with ID 92667650...
[INFO] [2021-04-07 19:18:58] Starting batch with ID 92667650...
[INFO] [2021-04-07 19:18:58] Starting batch with ID 92667650...
[INFO] [2021-04-07 19:18:58] Starting batch with ID 92667650...
[INFO] [2021-04-07 19:18:58] ## remove_type: Node
[INFO] [2021-04-07 19:18:58] ++ Calling delete_all on 1430 instances...
[INFO] [2021-04-07 19:18:58] [19:18:58.770] Removed 1430 Nodes
[START] [2021-04-07 19:18:59] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 19:18:59] Creating resource from OpenData
[START] [2021-04-07 19:19:00] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 19:19:00] Parse meta.xml file and create formats with fields
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (media) field header: ThumbnailURL term: http://eol.org/schema/media/thumbnailURL
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (media) field header: DerivedFrom term: http://rs.tdwg.org/ac/terms/derivedFrom
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (media) field header: Modified term: http://purl.org/dc/terms/modified
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (media) field header: Rating term: http://ns.adobe.com/xap/1.0/Rating
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (media) field header: Audience term: http://purl.org/dc/terms/audience
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (media) field header: Rights term: http://purl.org/dc/terms/rights
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (media) field header: Publisher term: http://purl.org/dc/terms/publisher
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (media) field header: Contributor term: http://purl.org/dc/terms/contributor
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (media) field header: Creator term: http://purl.org/dc/terms/creator
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (media) field header: GenericLocation term: http://purl.org/dc/terms/spatial
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (media) field header: Latitude term: http://www.w3.org/2003/01/geo/wgs84_pos#lat
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (media) field header: Longitude term: http://www.w3.org/2003/01/geo/wgs84_pos#long
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (media) field header: Altitude term: http://www.w3.org/2003/01/geo/wgs84_pos#alt
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (occurrences) field header: Institution Code term: http://rs.tdwg.org/dwc/terms/institutionCode
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (occurrences) field header: Collection Code term: http://rs.tdwg.org/dwc/terms/collectionCode
[WARN] [2021-04-07 19:19:14] (common) IGNORED  (occurrences) field header: Catalog Number term: http://rs.tdwg.org/dwc/terms/catalogNumber
[WARN] [2021-04-07 19:19:14] SKIPPING http://rs.tdwg.org/dwc/terms/Event config (events.txt)...
[STOP] [2021-04-07 19:19:14] Parse meta.xml file and create formats with fields
[STOP] [2021-04-07 19:19:14] Creating resource from OpenData
[START] [2021-04-07 19:19:14] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 19:19:14] create_harvest_instance
[INFO] [2021-04-07 19:19:14] Created harvest instance #3699
[STOP] [2021-04-07 19:19:14] create_harvest_instance
[START] [2021-04-07 19:19:14] fetch_files
[STOP] [2021-04-07 19:19:14] fetch_files
[START] [2021-04-07 19:19:14] validate_each_file
[INFO] [2021-04-07 19:19:14] Looping over 8 formats...
[INFO] [2021-04-07 19:19:15] ...agents (/app/public/data/scleractinia/agents.txt)
[INFO] [2021-04-07 19:19:15] Valid: /app/public/converted_csv/scleractinia_agents_3699.csv (0 lines)
[INFO] [2021-04-07 19:19:15] ...refs (/app/public/data/scleractinia/references.txt)
[INFO] [2021-04-07 19:19:15] Valid: /app/public/converted_csv/scleractinia_refs_3699.csv (0 lines)
[INFO] [2021-04-07 19:19:15] ...nodes (/app/public/data/scleractinia/taxa.txt)
[INFO] [2021-04-07 19:19:15] Valid: /app/public/converted_csv/scleractinia_nodes_3699.csv (1426 lines)
[INFO] [2021-04-07 19:19:15] ...media (/app/public/data/scleractinia/media.txt)
[INFO] [2021-04-07 19:19:15] Valid: /app/public/converted_csv/scleractinia_media_3699.csv (0 lines)
[INFO] [2021-04-07 19:19:15] ...vernaculars (/app/public/data/scleractinia/common names.txt)
[INFO] [2021-04-07 19:19:15] Valid: /app/public/converted_csv/scleractinia_vernaculars_3699.csv (0 lines)
[INFO] [2021-04-07 19:19:15] ...occurrences (/app/public/data/scleractinia/occurrences.txt)
[INFO] [2021-04-07 19:19:15] Valid: /app/public/converted_csv/scleractinia_occurrences_3699.csv (1426 lines)
[INFO] [2021-04-07 19:19:15] ...assocs (/app/public/data/scleractinia/associations.txt)
[INFO] [2021-04-07 19:19:15] Valid: /app/public/converted_csv/scleractinia_assocs_3699.csv (0 lines)
[INFO] [2021-04-07 19:19:15] ...measurements (/app/public/data/scleractinia/measurements or facts.txt)
[INFO] [2021-04-07 19:19:15] Valid: /app/public/converted_csv/scleractinia_measurements_3699.csv (2248 lines)
[STOP] [2021-04-07 19:19:15] validate_each_file
[START] [2021-04-07 19:19:15] convert_to_csv
[INFO] [2021-04-07 19:19:15] Looping over 8 formats...
[INFO] [2021-04-07 19:19:15] ...agents (/app/public/data/scleractinia/agents.txt)
[CMD] [2021-04-07 19:19:15] /usr/bin/sort /app/public/converted_csv/scleractinia_agents_3699.csv > /app/public/converted_csv/scleractinia_agents_3699.csv_sorted
[INFO] [2021-04-07 19:19:16] Converted: /app/public/converted_csv/scleractinia_agents_3699.csv (0 lines)
[INFO] [2021-04-07 19:19:16] ...refs (/app/public/data/scleractinia/references.txt)
[CMD] [2021-04-07 19:19:16] /usr/bin/sort /app/public/converted_csv/scleractinia_refs_3699.csv > /app/public/converted_csv/scleractinia_refs_3699.csv_sorted
[INFO] [2021-04-07 19:19:16] Converted: /app/public/converted_csv/scleractinia_refs_3699.csv (0 lines)
[INFO] [2021-04-07 19:19:16] ...nodes (/app/public/data/scleractinia/taxa.txt)
[CMD] [2021-04-07 19:19:16] /usr/bin/sort /app/public/converted_csv/scleractinia_nodes_3699.csv > /app/public/converted_csv/scleractinia_nodes_3699.csv_sorted
[INFO] [2021-04-07 19:19:17] Converted: /app/public/converted_csv/scleractinia_nodes_3699.csv (1426 lines)
[INFO] [2021-04-07 19:19:17] ...media (/app/public/data/scleractinia/media.txt)
[CMD] [2021-04-07 19:19:17] /usr/bin/sort /app/public/converted_csv/scleractinia_media_3699.csv > /app/public/converted_csv/scleractinia_media_3699.csv_sorted
[INFO] [2021-04-07 19:19:17] Converted: /app/public/converted_csv/scleractinia_media_3699.csv (0 lines)
[INFO] [2021-04-07 19:19:17] ...vernaculars (/app/public/data/scleractinia/common names.txt)
[CMD] [2021-04-07 19:19:17] /usr/bin/sort /app/public/converted_csv/scleractinia_vernaculars_3699.csv > /app/public/converted_csv/scleractinia_vernaculars_3699.csv_sorted
[INFO] [2021-04-07 19:19:18] Converted: /app/public/converted_csv/scleractinia_vernaculars_3699.csv (0 lines)
[INFO] [2021-04-07 19:19:18] ...occurrences (/app/public/data/scleractinia/occurrences.txt)
[CMD] [2021-04-07 19:19:18] /usr/bin/sort /app/public/converted_csv/scleractinia_occurrences_3699.csv > /app/public/converted_csv/scleractinia_occurrences_3699.csv_sorted
[INFO] [2021-04-07 19:19:19] Converted: /app/public/converted_csv/scleractinia_occurrences_3699.csv (1426 lines)
[INFO] [2021-04-07 19:19:19] ...assocs (/app/public/data/scleractinia/associations.txt)
[CMD] [2021-04-07 19:19:19] /usr/bin/sort /app/public/converted_csv/scleractinia_assocs_3699.csv > /app/public/converted_csv/scleractinia_assocs_3699.csv_sorted
[INFO] [2021-04-07 19:19:19] Converted: /app/public/converted_csv/scleractinia_assocs_3699.csv (0 lines)
[INFO] [2021-04-07 19:19:19] ...measurements (/app/public/data/scleractinia/measurements or facts.txt)
[CMD] [2021-04-07 19:19:19] /usr/bin/sort /app/public/converted_csv/scleractinia_measurements_3699.csv > /app/public/converted_csv/scleractinia_measurements_3699.csv_sorted
[INFO] [2021-04-07 19:19:20] Converted: /app/public/converted_csv/scleractinia_measurements_3699.csv (2248 lines)
[STOP] [2021-04-07 19:19:20] convert_to_csv
[START] [2021-04-07 19:19:20] calculate_delta
[INFO] [2021-04-07 19:19:20] Looping over 8 formats...
[INFO] [2021-04-07 19:19:20] ...agents (/app/public/data/scleractinia/agents.txt)
[CMD] [2021-04-07 19:19:20] echo "0a" > /app/public/diff/scleractinia_agents_3699.diff
[CMD] [2021-04-07 19:19:20] tail -n +1 /app/public/converted_csv/scleractinia_agents_3699.csv >> /app/public/diff/scleractinia_agents_3699.diff
[CMD] [2021-04-07 19:19:21] echo "." >> /app/public/diff/scleractinia_agents_3699.diff
[INFO] [2021-04-07 19:19:21] Created diff: /app/public/diff/scleractinia_agents_3699.diff (2 lines)
[INFO] [2021-04-07 19:19:21] ...refs (/app/public/data/scleractinia/references.txt)
[CMD] [2021-04-07 19:19:21] echo "0a" > /app/public/diff/scleractinia_refs_3699.diff
[CMD] [2021-04-07 19:19:22] tail -n +1 /app/public/converted_csv/scleractinia_refs_3699.csv >> /app/public/diff/scleractinia_refs_3699.diff
[CMD] [2021-04-07 19:19:22] echo "." >> /app/public/diff/scleractinia_refs_3699.diff
[INFO] [2021-04-07 19:19:23] Created diff: /app/public/diff/scleractinia_refs_3699.diff (2 lines)
[INFO] [2021-04-07 19:19:23] ...nodes (/app/public/data/scleractinia/taxa.txt)
[CMD] [2021-04-07 19:19:23] echo "0a" > /app/public/diff/scleractinia_nodes_3699.diff
[CMD] [2021-04-07 19:19:23] tail -n +1 /app/public/converted_csv/scleractinia_nodes_3699.csv >> /app/public/diff/scleractinia_nodes_3699.diff
[CMD] [2021-04-07 19:19:24] echo "." >> /app/public/diff/scleractinia_nodes_3699.diff
[INFO] [2021-04-07 19:19:24] Created diff: /app/public/diff/scleractinia_nodes_3699.diff (1428 lines)
[INFO] [2021-04-07 19:19:24] ...media (/app/public/data/scleractinia/media.txt)
[CMD] [2021-04-07 19:19:24] echo "0a" > /app/public/diff/scleractinia_media_3699.diff
[CMD] [2021-04-07 19:19:25] tail -n +1 /app/public/converted_csv/scleractinia_media_3699.csv >> /app/public/diff/scleractinia_media_3699.diff
[CMD] [2021-04-07 19:19:26] echo "." >> /app/public/diff/scleractinia_media_3699.diff
[INFO] [2021-04-07 19:19:26] Created diff: /app/public/diff/scleractinia_media_3699.diff (2 lines)
[INFO] [2021-04-07 19:19:26] ...vernaculars (/app/public/data/scleractinia/common names.txt)
[CMD] [2021-04-07 19:19:26] echo "0a" > /app/public/diff/scleractinia_vernaculars_3699.diff
[CMD] [2021-04-07 19:19:27] tail -n +1 /app/public/converted_csv/scleractinia_vernaculars_3699.csv >> /app/public/diff/scleractinia_vernaculars_3699.diff
[CMD] [2021-04-07 19:19:27] echo "." >> /app/public/diff/scleractinia_vernaculars_3699.diff
[INFO] [2021-04-07 19:19:28] Created diff: /app/public/diff/scleractinia_vernaculars_3699.diff (2 lines)
[INFO] [2021-04-07 19:19:28] ...occurrences (/app/public/data/scleractinia/occurrences.txt)
[CMD] [2021-04-07 19:19:28] echo "0a" > /app/public/diff/scleractinia_occurrences_3699.diff
[CMD] [2021-04-07 19:19:28] tail -n +1 /app/public/converted_csv/scleractinia_occurrences_3699.csv >> /app/public/diff/scleractinia_occurrences_3699.diff
[CMD] [2021-04-07 19:19:29] echo "." >> /app/public/diff/scleractinia_occurrences_3699.diff
[INFO] [2021-04-07 19:19:29] Created diff: /app/public/diff/scleractinia_occurrences_3699.diff (1428 lines)
[INFO] [2021-04-07 19:19:29] ...assocs (/app/public/data/scleractinia/associations.txt)
[CMD] [2021-04-07 19:19:29] echo "0a" > /app/public/diff/scleractinia_assocs_3699.diff
[CMD] [2021-04-07 19:19:30] tail -n +1 /app/public/converted_csv/scleractinia_assocs_3699.csv >> /app/public/diff/scleractinia_assocs_3699.diff
[CMD] [2021-04-07 19:19:31] echo "." >> /app/public/diff/scleractinia_assocs_3699.diff
[INFO] [2021-04-07 19:19:31] Created diff: /app/public/diff/scleractinia_assocs_3699.diff (2 lines)
[INFO] [2021-04-07 19:19:31] ...measurements (/app/public/data/scleractinia/measurements or facts.txt)
[CMD] [2021-04-07 19:19:31] echo "0a" > /app/public/diff/scleractinia_measurements_3699.diff
[CMD] [2021-04-07 19:19:32] tail -n +1 /app/public/converted_csv/scleractinia_measurements_3699.csv >> /app/public/diff/scleractinia_measurements_3699.diff
[CMD] [2021-04-07 19:19:32] echo "." >> /app/public/diff/scleractinia_measurements_3699.diff
[INFO] [2021-04-07 19:19:33] Created diff: /app/public/diff/scleractinia_measurements_3699.diff (2250 lines)
[STOP] [2021-04-07 19:19:33] calculate_delta
[START] [2021-04-07 19:19:33] parse_diff_and_store
[INFO] [2021-04-07 19:19:33] Handling diff: /app/public/diff/scleractinia_agents_3699.diff (2 lines)
[INFO] [2021-04-07 19:19:33] Loading agents diff file into memory (2 /app/public/diff/scleractinia_agents_3699.diff lines)...
[INFO] [2021-04-07 19:19:34] Handling diff: /app/public/diff/scleractinia_refs_3699.diff (2 lines)
[INFO] [2021-04-07 19:19:34] Loading refs diff file into memory (2 /app/public/diff/scleractinia_refs_3699.diff lines)...
[INFO] [2021-04-07 19:19:35] Handling diff: /app/public/diff/scleractinia_nodes_3699.diff (1428 lines)
[INFO] [2021-04-07 19:19:35] Loading nodes diff file into memory (1428 /app/public/diff/scleractinia_nodes_3699.diff lines)...
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Aulocyathus  juvenescens Marenzeller, 1904` to `Aulocyathus juvenescens Marenzeller, 1904`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Cladangia  exusta Lütken, 1873` to `Cladangia exusta Lütken, 1873`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Coenosmilia  inordinata Cairns, 1984` to `Coenosmilia inordinata Cairns, 1984`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Culicia  stellata Dana, 1846` to `Culicia stellata Dana, 1846`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Flabellum  pavoninum Lesson, 1831` to `Flabellum pavoninum Lesson, 1831`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Flabellum curvatum  Moseley, 1881` to `Flabellum curvatum Moseley, 1881`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Heterocyathus antoniae  Reyes, Santodomingo & Cairns, 2009` to `Heterocyathus antoniae Reyes, Santodomingo & Cairns, 2009`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Javania  insignis Duncan, 1876` to `Javania insignis Duncan, 1876`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Labyrinthocyathus  langae Cairns, 1979: 341` to `Labyrinthocyathus langae Cairns, 1979: 341`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Rhizosmilia  gerdae Cairns, 1978` to `Rhizosmilia gerdae Cairns, 1978`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Solenosmilia  variabilis Duncan, 1873` to `Solenosmilia variabilis Duncan, 1873`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Stephanocyathus (Stephanocyathus) isabellae  Reyes, Santodomingo & Cairns, 2009` to `Stephanocyathus (Stephanocyathus) isabellae Reyes, Santodomingo & Cairns, 2009`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Trochocyathus (Trochocyathus)  aithoseptatus Cairns, 1984` to `Trochocyathus (Trochocyathus) aithoseptatus Cairns, 1984`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Trochocyathus (Trochocyathus)  apertus Cairns & Zibrowius, 1997` to `Trochocyathus (Trochocyathus) apertus Cairns & Zibrowius, 1997`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Trochocyathus (Trochocyathus)  caryophylloides Alcock, 1902` to `Trochocyathus (Trochocyathus) caryophylloides Alcock, 1902`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Trochocyathus (Trochocyathus)  cepulla Cairns, 1995` to `Trochocyathus (Trochocyathus) cepulla Cairns, 1995`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Trochocyathus (Trochocyathus)  decamera Cairns, 1994` to `Trochocyathus (Trochocyathus) decamera Cairns, 1994`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Trochocyathus (Trochocyathus)  discus Cairns & Zibrowius, 1997` to `Trochocyathus (Trochocyathus) discus Cairns & Zibrowius, 1997`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Trochocyathus (Trochocyathus)  efateensis Cairns, 1999` to `Trochocyathus (Trochocyathus) efateensis Cairns, 1999`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Trochocyathus (Trochocyathus)  fasciatus Cairns, 1979` to `Trochocyathus (Trochocyathus) fasciatus Cairns, 1979`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Trochocyathus (Trochocyathus)  fossulus Cairns, 1979` to `Trochocyathus (Trochocyathus) fossulus Cairns, 1979`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Trochocyathus (Trochocyathus)  gordoni Cairns, 1995` to `Trochocyathus (Trochocyathus) gordoni Cairns, 1995`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Trochocyathus (Trochocyathus)  japonicus Eguchi, 1968` to `Trochocyathus (Trochocyathus) japonicus Eguchi, 1968`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Trochocyathus (Trochocyathus)  laboreli Cairns, 2000` to `Trochocyathus (Trochocyathus) laboreli Cairns, 2000`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Trochocyathus (Trochocyathus)  maculatus Cairns, 1995` to `Trochocyathus (Trochocyathus) maculatus Cairns, 1995`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Trochocyathus (Trochocyathus)  oahensis Vaughan, 1907` to `Trochocyathus (Trochocyathus) oahensis Vaughan, 1907`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Trochocyathus (Trochocyathus)  patelliformis Cairns, 1999` to `Trochocyathus (Trochocyathus) patelliformis Cairns, 1999`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Trochocyathus (Trochocyathus)  philippinensis Semper, 1872` to `Trochocyathus (Trochocyathus) philippinensis Semper, 1872`
[WARN] [2021-04-07 19:19:36] Filtered Scientific Name `Trochocyathus (Trochocyathus)  rawsonii Pourtalès, 1874` to `Trochocyathus (Trochocyathus) rawsonii Pourtalès, 1874`
[WARN] [2021-04-07 19:19:36] (Reached filtered-name limit; supressing further warnings.)
[INFO] [2021-04-07 19:19:36] Handling diff: /app/public/diff/scleractinia_media_3699.diff (2 lines)
[INFO] [2021-04-07 19:19:37] Loading media diff file into memory (2 /app/public/diff/scleractinia_media_3699.diff lines)...
[INFO] [2021-04-07 19:19:38] Handling diff: /app/public/diff/scleractinia_vernaculars_3699.diff (2 lines)
[INFO] [2021-04-07 19:19:38] Loading vernaculars diff file into memory (2 /app/public/diff/scleractinia_vernaculars_3699.diff lines)...
[INFO] [2021-04-07 19:19:39] Handling diff: /app/public/diff/scleractinia_occurrences_3699.diff (1428 lines)
[INFO] [2021-04-07 19:19:39] Loading occurrences diff file into memory (1428 /app/public/diff/scleractinia_occurrences_3699.diff lines)...
[INFO] [2021-04-07 19:19:40] Handling diff: /app/public/diff/scleractinia_assocs_3699.diff (2 lines)
[INFO] [2021-04-07 19:19:40] Loading assocs diff file into memory (2 /app/public/diff/scleractinia_assocs_3699.diff lines)...
[INFO] [2021-04-07 19:19:41] Handling diff: /app/public/diff/scleractinia_measurements_3699.diff (2250 lines)
[INFO] [2021-04-07 19:19:42] Loading measurements diff file into memory (2250 /app/public/diff/scleractinia_measurements_3699.diff lines)...
[INFO] [2021-04-07 19:19:43] Storing 1430 ScientificNames
[INFO] [2021-04-07 19:19:43] Processing group of 1430 in 2 groups of 1000
[INFO] [2021-04-07 19:19:44] Average Time: 0.2
[INFO] [2021-04-07 19:19:44] Total Time: 1s
[INFO] [2021-04-07 19:19:44] Storing 1430 Nodes
[INFO] [2021-04-07 19:19:44] Processing group of 1430 in 2 groups of 1000
[INFO] [2021-04-07 19:19:44] Average Time: 0.23
[INFO] [2021-04-07 19:19:44] Total Time: 1s
[INFO] [2021-04-07 19:19:44] Storing 1426 Occurrences
[INFO] [2021-04-07 19:19:44] Processing group of 1426 in 2 groups of 1000
[INFO] [2021-04-07 19:19:44] Average Time: 0.075
[INFO] [2021-04-07 19:19:44] Total Time: 1s
[INFO] [2021-04-07 19:19:44] Storing 2248 Traits
[INFO] [2021-04-07 19:19:44] Processing group of 2248 in 3 groups of 1000
[INFO] [2021-04-07 19:19:45] Average Time: 0.263
[INFO] [2021-04-07 19:19:45] Total Time: 1s
[INFO] [2021-04-07 19:19:45] Storing 4496 MetaTraits
[INFO] [2021-04-07 19:19:45] Processing group of 4496 in 5 groups of 1000
[INFO] [2021-04-07 19:19:46] Average Time: 0.098
[INFO] [2021-04-07 19:19:46] Total Time: 1s
[STOP] [2021-04-07 19:19:46] parse_diff_and_store
[START] [2021-04-07 19:19:46] resolve_keys
[INFO] [2021-04-07 19:19:52] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-07 19:19:52] traits to occurrences...
[INFO] [2021-04-07 19:19:52] traits to nodes (through occurrences)...
[INFO] [2021-04-07 19:19:52] Traits to sex term...
[INFO] [2021-04-07 19:19:52] Traits to lifestage term...
[INFO] [2021-04-07 19:19:52] MetaTraits to traits...
[INFO] [2021-04-07 19:19:52] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-07 19:19:52] Assocs to occurrences...
[INFO] [2021-04-07 19:19:52] Assocs to nodes...
[INFO] [2021-04-07 19:19:52] Assoc to sex term...
[INFO] [2021-04-07 19:19:52] Assoc to lifestage term...
[INFO] [2021-04-07 19:19:52] MetaAssoc to assocs...
[STOP] [2021-04-07 19:19:52] resolve_keys
[START] [2021-04-07 19:19:52] hold_for_later_1
[STOP] [2021-04-07 19:19:52] hold_for_later_1
[START] [2021-04-07 19:19:52] hold_for_later_2
[STOP] [2021-04-07 19:19:52] hold_for_later_2
[START] [2021-04-07 19:19:52] resolve_missing_parents
[STOP] [2021-04-07 19:19:52] resolve_missing_parents
[START] [2021-04-07 19:19:52] rebuild_nodes
[START] [2021-04-07 19:19:52] Flattener#flatten
[START] [2021-04-07 19:19:52] Flattener#study_resource
[START] [2021-04-07 19:19:52] Flattener#build_ancestry
[STOP] [2021-04-07 19:19:52] Flattener#build_ancestry
[INFO] [2021-04-07 19:19:52] 1430 ancestry keys
[START] [2021-04-07 19:19:52] build_node_ancestors
[INFO] [2021-04-07 19:19:52] old ancestors deleted.
[STOP] [2021-04-07 19:19:53] build_node_ancestors
[START] [2021-04-07 19:19:53] Flattener#propagate_ancestor_ids
[STOP] [2021-04-07 19:19:53] Flattener#propagate_ancestor_ids
[STOP] [2021-04-07 19:19:53] Flattener#flatten
[STOP] [2021-04-07 19:19:53] rebuild_nodes
[START] [2021-04-07 19:19:53] resolve_missing_media_owners
[STOP] [2021-04-07 19:19:53] resolve_missing_media_owners
[START] [2021-04-07 19:19:53] sanitize_media_verbatims
[STOP] [2021-04-07 19:19:53] sanitize_media_verbatims
[START] [2021-04-07 19:19:53] queue_downloads
[STOP] [2021-04-07 19:19:53] queue_downloads
[START] [2021-04-07 19:19:53] parse_names
[WARN] [2021-04-07 19:19:53] I see 1430 names which still need to be parsed.
[STOP] [2021-04-07 19:19:55] parse_names
[START] [2021-04-07 19:19:55] denormalize_canonical_names_to_nodes
[STOP] [2021-04-07 19:19:55] denormalize_canonical_names_to_nodes
[START] [2021-04-07 19:19:55] match_nodes
[START] [2021-04-07 19:19:55] map_all_nodes_to_pages
[STOP] [2021-04-07 19:20:40] map_all_nodes_to_pages
[INFO] [2021-04-07 19:20:40] 23 Unmatched nodes (of 1430)! That's too many to output. Full list in /app/public/data/scleractinia/unmatched_nodes.txt ; First 10: Acanthastrea rotundaflora (#92679424); Acropora jacquelinae (#92679473); Acropora lutkeni (#92679485); Acropora mirabilis (#92679491); Acropora pruinosa (#92679512); Acropora teres (#92679538); Coscinaraea hazimanensis (#92679784); Deltocyathus agassizii (#92679821); Deltocyathus inusitatus (#92679829); Dendrophyllia boschmai cyathoheloides (#92679846)
[START] [2021-04-07 19:20:40] update_nodes
[STOP] [2021-04-07 19:20:41] update_nodes
[STOP] [2021-04-07 19:20:41] match_nodes
[START] [2021-04-07 19:20:41] reindex_search
[STOP] [2021-04-07 19:20:42] reindex_search
[START] [2021-04-07 19:20:42] normalize_units
[STOP] [2021-04-07 19:20:42] normalize_units
[START] [2021-04-07 19:20:42] calculate_statistics
[STOP] [2021-04-07 19:20:42] calculate_statistics
[START] [2021-04-07 19:20:42] complete_harvest_instance
[START] [2021-04-07 19:20:42] overall_tsv_creation
[INFO] [2021-04-07 19:20:42] Processing group of 1430 in 1 batches of 10000
[INFO] [2021-04-07 19:21:23] 471 Traits (unfiltered)...
[INFO] [2021-04-07 19:21:58] 471 Traits (filtered)...
[INFO] [2021-04-07 19:21:58] 0 Associations (filtered)...
[INFO] [2021-04-07 19:21:58] 471 metadata added.
[INFO] [2021-04-07 19:21:58] 0 metadata added.
[INFO] [2021-04-07 19:22:25] Average Time: 77.77
[INFO] [2021-04-07 19:22:25] Total Time: 1m43s
[STOP] [2021-04-07 19:22:25] overall_tsv_creation
[INFO] [2021-04-07 19:22:25] Done. Check your files:
[INFO] [2021-04-07 19:22:25] (1430 lines) /app/public/data/scleractinia/publish_nodes.tsv
[INFO] [2021-04-07 19:22:26] (5710 lines) /app/public/data/scleractinia/publish_node_ancestors.tsv
[INFO] [2021-04-07 19:22:26] (1430 lines) /app/public/data/scleractinia/publish_scientific_names.tsv
[INFO] [2021-04-07 19:22:27] (472 lines) /app/public/data/scleractinia/publish_traits.tsv
[INFO] [2021-04-07 19:22:27] (472 lines) /app/public/data/scleractinia/publish_metadata.tsv
[STOP] [2021-04-07 19:22:27] complete_harvest_instance
[START] [2021-04-07 19:22:27] completed
[STOP] [2021-04-07 19:22:27] completed
[STOP] [2021-04-07 19:22:27] logged process, took 193.52
[INFO] [2021-04-07 19:36:59] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-07 19:37:04] ## remove_type: ScientificName
[INFO] [2021-04-07 19:37:04] ++ Calling delete_all on 1430 instances...
[INFO] [2021-04-07 19:37:04] [19:37:04.092] Removed 1430 Scientificnames
[INFO] [2021-04-07 19:37:04] ## remove_type: Vernacular
[INFO] [2021-04-07 19:37:04] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 19:37:04] [19:37:04.093] Removed 0 Vernaculars
[INFO] [2021-04-07 19:37:04] ## remove_type: Article
[INFO] [2021-04-07 19:37:04] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 19:37:04] [19:37:04.095] Removed 0 Articles
[INFO] [2021-04-07 19:37:04] ## remove_type: Medium
[INFO] [2021-04-07 19:37:04] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 19:37:04] [19:37:04.096] Removed 0 Media
[INFO] [2021-04-07 19:37:04] ## remove_type: Trait
[INFO] [2021-04-07 19:37:04] ++ Calling delete_all on 2248 instances...
[INFO] [2021-04-07 19:37:04] [19:37:04.153] Removed 2248 Traits
[INFO] [2021-04-07 19:37:04] ## remove_type: MetaTrait
[INFO] [2021-04-07 19:37:04] ++ Calling delete_all on 4496 instances...
[INFO] [2021-04-07 19:37:04] [19:37:04.199] Removed 4496 Metatraits
[INFO] [2021-04-07 19:37:04] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-07 19:37:04] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 19:37:04] [19:37:04.201] Removed 0 Occurrencemetadata
[INFO] [2021-04-07 19:37:04] ## remove_type: Assoc
[INFO] [2021-04-07 19:37:04] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 19:37:04] [19:37:04.202] Removed 0 Assocs
[INFO] [2021-04-07 19:37:04] ## remove_type: MetaAssoc
[INFO] [2021-04-07 19:37:04] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 19:37:04] [19:37:04.204] Removed 0 Metaassocs
[INFO] [2021-04-07 19:37:04] ## remove_type: Identifier
[INFO] [2021-04-07 19:37:04] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 19:37:04] [19:37:04.205] Removed 0 Identifiers
[INFO] [2021-04-07 19:37:04] ## remove_type: Reference
[INFO] [2021-04-07 19:37:04] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-07 19:37:04] [19:37:04.207] Removed 0 References
[INFO] [2021-04-07 19:37:04] Starting batch with ID 92679415...
[INFO] [2021-04-07 19:37:04] Starting batch with ID 92679415...
[INFO] [2021-04-07 19:37:04] Starting batch with ID 92679415...
[INFO] [2021-04-07 19:37:04] Starting batch with ID 92679415...
[INFO] [2021-04-07 19:37:04] Starting batch with ID 92679415...
[INFO] [2021-04-07 19:37:04] Starting batch with ID 92679415...
[INFO] [2021-04-07 19:37:05] ## remove_type: Node
[INFO] [2021-04-07 19:37:05] ++ Calling delete_all on 1430 instances...
[INFO] [2021-04-07 19:37:05] [19:37:05.043] Removed 1430 Nodes
[START] [2021-04-07 19:37:05] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 19:37:05] Creating resource from OpenData
[START] [2021-04-07 19:37:06] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 19:37:06] Parse meta.xml file and create formats with fields
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (media) field header: ThumbnailURL term: http://eol.org/schema/media/thumbnailURL
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (media) field header: DerivedFrom term: http://rs.tdwg.org/ac/terms/derivedFrom
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (media) field header: Modified term: http://purl.org/dc/terms/modified
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (media) field header: Rating term: http://ns.adobe.com/xap/1.0/Rating
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (media) field header: Audience term: http://purl.org/dc/terms/audience
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (media) field header: Rights term: http://purl.org/dc/terms/rights
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (media) field header: Publisher term: http://purl.org/dc/terms/publisher
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (media) field header: Contributor term: http://purl.org/dc/terms/contributor
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (media) field header: Creator term: http://purl.org/dc/terms/creator
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (media) field header: GenericLocation term: http://purl.org/dc/terms/spatial
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (media) field header: Latitude term: http://www.w3.org/2003/01/geo/wgs84_pos#lat
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (media) field header: Longitude term: http://www.w3.org/2003/01/geo/wgs84_pos#long
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (media) field header: Altitude term: http://www.w3.org/2003/01/geo/wgs84_pos#alt
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (occurrences) field header: Institution Code term: http://rs.tdwg.org/dwc/terms/institutionCode
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (occurrences) field header: Collection Code term: http://rs.tdwg.org/dwc/terms/collectionCode
[WARN] [2021-04-07 19:37:19] (common) IGNORED  (occurrences) field header: Catalog Number term: http://rs.tdwg.org/dwc/terms/catalogNumber
[WARN] [2021-04-07 19:37:19] SKIPPING http://rs.tdwg.org/dwc/terms/Event config (events.txt)...
[STOP] [2021-04-07 19:37:19] Parse meta.xml file and create formats with fields
[STOP] [2021-04-07 19:37:19] Creating resource from OpenData
[START] [2021-04-07 19:37:20] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-07 19:37:20] create_harvest_instance
[INFO] [2021-04-07 19:37:20] Created harvest instance #3700
[STOP] [2021-04-07 19:37:20] create_harvest_instance
[START] [2021-04-07 19:37:20] fetch_files
[STOP] [2021-04-07 19:37:20] fetch_files
[START] [2021-04-07 19:37:20] validate_each_file
[INFO] [2021-04-07 19:37:20] Looping over 8 formats...
[INFO] [2021-04-07 19:37:20] ...agents (/app/public/data/scleractinia/agents.txt)
[INFO] [2021-04-07 19:37:20] Valid: /app/public/converted_csv/scleractinia_agents_3700.csv (0 lines)
[INFO] [2021-04-07 19:37:20] ...refs (/app/public/data/scleractinia/references.txt)
[INFO] [2021-04-07 19:37:20] Valid: /app/public/converted_csv/scleractinia_refs_3700.csv (0 lines)
[INFO] [2021-04-07 19:37:20] ...nodes (/app/public/data/scleractinia/taxa.txt)
[INFO] [2021-04-07 19:37:20] Valid: /app/public/converted_csv/scleractinia_nodes_3700.csv (1426 lines)
[INFO] [2021-04-07 19:37:20] ...media (/app/public/data/scleractinia/media.txt)
[INFO] [2021-04-07 19:37:20] Valid: /app/public/converted_csv/scleractinia_media_3700.csv (0 lines)
[INFO] [2021-04-07 19:37:20] ...vernaculars (/app/public/data/scleractinia/common names.txt)
[INFO] [2021-04-07 19:37:20] Valid: /app/public/converted_csv/scleractinia_vernaculars_3700.csv (0 lines)
[INFO] [2021-04-07 19:37:20] ...occurrences (/app/public/data/scleractinia/occurrences.txt)
[INFO] [2021-04-07 19:37:20] Valid: /app/public/converted_csv/scleractinia_occurrences_3700.csv (1426 lines)
[INFO] [2021-04-07 19:37:20] ...assocs (/app/public/data/scleractinia/associations.txt)
[INFO] [2021-04-07 19:37:20] Valid: /app/public/converted_csv/scleractinia_assocs_3700.csv (0 lines)
[INFO] [2021-04-07 19:37:20] ...measurements (/app/public/data/scleractinia/measurements or facts.txt)
[INFO] [2021-04-07 19:37:20] Valid: /app/public/converted_csv/scleractinia_measurements_3700.csv (2248 lines)
[STOP] [2021-04-07 19:37:20] validate_each_file
[START] [2021-04-07 19:37:20] convert_to_csv
[INFO] [2021-04-07 19:37:20] Looping over 8 formats...
[INFO] [2021-04-07 19:37:20] ...agents (/app/public/data/scleractinia/agents.txt)
[CMD] [2021-04-07 19:37:20] /usr/bin/sort /app/public/converted_csv/scleractinia_agents_3700.csv > /app/public/converted_csv/scleractinia_agents_3700.csv_sorted
[INFO] [2021-04-07 19:37:21] Converted: /app/public/converted_csv/scleractinia_agents_3700.csv (0 lines)
[INFO] [2021-04-07 19:37:21] ...refs (/app/public/data/scleractinia/references.txt)
[CMD] [2021-04-07 19:37:21] /usr/bin/sort /app/public/converted_csv/scleractinia_refs_3700.csv > /app/public/converted_csv/scleractinia_refs_3700.csv_sorted
[INFO] [2021-04-07 19:37:22] Converted: /app/public/converted_csv/scleractinia_refs_3700.csv (0 lines)
[INFO] [2021-04-07 19:37:22] ...nodes (/app/public/data/scleractinia/taxa.txt)
[CMD] [2021-04-07 19:37:22] /usr/bin/sort /app/public/converted_csv/scleractinia_nodes_3700.csv > /app/public/converted_csv/scleractinia_nodes_3700.csv_sorted
[INFO] [2021-04-07 19:37:22] Converted: /app/public/converted_csv/scleractinia_nodes_3700.csv (1426 lines)
[INFO] [2021-04-07 19:37:22] ...media (/app/public/data/scleractinia/media.txt)
[CMD] [2021-04-07 19:37:22] /usr/bin/sort /app/public/converted_csv/scleractinia_media_3700.csv > /app/public/converted_csv/scleractinia_media_3700.csv_sorted
[INFO] [2021-04-07 19:37:23] Converted: /app/public/converted_csv/scleractinia_media_3700.csv (0 lines)
[INFO] [2021-04-07 19:37:23] ...vernaculars (/app/public/data/scleractinia/common names.txt)
[CMD] [2021-04-07 19:37:23] /usr/bin/sort /app/public/converted_csv/scleractinia_vernaculars_3700.csv > /app/public/converted_csv/scleractinia_vernaculars_3700.csv_sorted
[INFO] [2021-04-07 19:37:23] Converted: /app/public/converted_csv/scleractinia_vernaculars_3700.csv (0 lines)
[INFO] [2021-04-07 19:37:23] ...occurrences (/app/public/data/scleractinia/occurrences.txt)
[CMD] [2021-04-07 19:37:23] /usr/bin/sort /app/public/converted_csv/scleractinia_occurrences_3700.csv > /app/public/converted_csv/scleractinia_occurrences_3700.csv_sorted
[INFO] [2021-04-07 19:37:24] Converted: /app/public/converted_csv/scleractinia_occurrences_3700.csv (1426 lines)
[INFO] [2021-04-07 19:37:24] ...assocs (/app/public/data/scleractinia/associations.txt)
[CMD] [2021-04-07 19:37:24] /usr/bin/sort /app/public/converted_csv/scleractinia_assocs_3700.csv > /app/public/converted_csv/scleractinia_assocs_3700.csv_sorted
[INFO] [2021-04-07 19:37:24] Converted: /app/public/converted_csv/scleractinia_assocs_3700.csv (0 lines)
[INFO] [2021-04-07 19:37:24] ...measurements (/app/public/data/scleractinia/measurements or facts.txt)
[CMD] [2021-04-07 19:37:24] /usr/bin/sort /app/public/converted_csv/scleractinia_measurements_3700.csv > /app/public/converted_csv/scleractinia_measurements_3700.csv_sorted
[INFO] [2021-04-07 19:37:25] Converted: /app/public/converted_csv/scleractinia_measurements_3700.csv (2248 lines)
[STOP] [2021-04-07 19:37:25] convert_to_csv
[START] [2021-04-07 19:37:25] calculate_delta
[INFO] [2021-04-07 19:37:25] Looping over 8 formats...
[INFO] [2021-04-07 19:37:25] ...agents (/app/public/data/scleractinia/agents.txt)
[CMD] [2021-04-07 19:37:25] echo "0a" > /app/public/diff/scleractinia_agents_3700.diff
[CMD] [2021-04-07 19:37:26] tail -n +1 /app/public/converted_csv/scleractinia_agents_3700.csv >> /app/public/diff/scleractinia_agents_3700.diff
[CMD] [2021-04-07 19:37:26] echo "." >> /app/public/diff/scleractinia_agents_3700.diff
[INFO] [2021-04-07 19:37:27] Created diff: /app/public/diff/scleractinia_agents_3700.diff (2 lines)
[INFO] [2021-04-07 19:37:27] ...refs (/app/public/data/scleractinia/references.txt)
[CMD] [2021-04-07 19:37:27] echo "0a" > /app/public/diff/scleractinia_refs_3700.diff
[CMD] [2021-04-07 19:37:27] tail -n +1 /app/public/converted_csv/scleractinia_refs_3700.csv >> /app/public/diff/scleractinia_refs_3700.diff
[CMD] [2021-04-07 19:37:28] echo "." >> /app/public/diff/scleractinia_refs_3700.diff
[INFO] [2021-04-07 19:37:28] Created diff: /app/public/diff/scleractinia_refs_3700.diff (2 lines)
[INFO] [2021-04-07 19:37:28] ...nodes (/app/public/data/scleractinia/taxa.txt)
[CMD] [2021-04-07 19:37:28] echo "0a" > /app/public/diff/scleractinia_nodes_3700.diff
[CMD] [2021-04-07 19:37:29] tail -n +1 /app/public/converted_csv/scleractinia_nodes_3700.csv >> /app/public/diff/scleractinia_nodes_3700.diff
[CMD] [2021-04-07 19:37:29] echo "." >> /app/public/diff/scleractinia_nodes_3700.diff
[INFO] [2021-04-07 19:37:30] Created diff: /app/public/diff/scleractinia_nodes_3700.diff (1428 lines)
[INFO] [2021-04-07 19:37:30] ...media (/app/public/data/scleractinia/media.txt)
[CMD] [2021-04-07 19:37:30] echo "0a" > /app/public/diff/scleractinia_media_3700.diff
[CMD] [2021-04-07 19:37:31] tail -n +1 /app/public/converted_csv/scleractinia_media_3700.csv >> /app/public/diff/scleractinia_media_3700.diff
[CMD] [2021-04-07 19:37:31] echo "." >> /app/public/diff/scleractinia_media_3700.diff
[INFO] [2021-04-07 19:37:32] Created diff: /app/public/diff/scleractinia_media_3700.diff (2 lines)
[INFO] [2021-04-07 19:37:32] ...vernaculars (/app/public/data/scleractinia/common names.txt)
[CMD] [2021-04-07 19:37:32] echo "0a" > /app/public/diff/scleractinia_vernaculars_3700.diff
[CMD] [2021-04-07 19:37:32] tail -n +1 /app/public/converted_csv/scleractinia_vernaculars_3700.csv >> /app/public/diff/scleractinia_vernaculars_3700.diff
[CMD] [2021-04-07 19:37:33] echo "." >> /app/public/diff/scleractinia_vernaculars_3700.diff
[INFO] [2021-04-07 19:37:33] Created diff: /app/public/diff/scleractinia_vernaculars_3700.diff (2 lines)
[INFO] [2021-04-07 19:37:33] ...occurrences (/app/public/data/scleractinia/occurrences.txt)
[CMD] [2021-04-07 19:37:33] echo "0a" > /app/public/diff/scleractinia_occurrences_3700.diff
[CMD] [2021-04-07 19:37:34] tail -n +1 /app/public/converted_csv/scleractinia_occurrences_3700.csv >> /app/public/diff/scleractinia_occurrences_3700.diff
[CMD] [2021-04-07 19:37:35] echo "." >> /app/public/diff/scleractinia_occurrences_3700.diff
[INFO] [2021-04-07 19:37:35] Created diff: /app/public/diff/scleractinia_occurrences_3700.diff (1428 lines)
[INFO] [2021-04-07 19:37:35] ...assocs (/app/public/data/scleractinia/associations.txt)
[CMD] [2021-04-07 19:37:35] echo "0a" > /app/public/diff/scleractinia_assocs_3700.diff
[CMD] [2021-04-07 19:37:36] tail -n +1 /app/public/converted_csv/scleractinia_assocs_3700.csv >> /app/public/diff/scleractinia_assocs_3700.diff
[CMD] [2021-04-07 19:37:36] echo "." >> /app/public/diff/scleractinia_assocs_3700.diff
[INFO] [2021-04-07 19:37:37] Created diff: /app/public/diff/scleractinia_assocs_3700.diff (2 lines)
[INFO] [2021-04-07 19:37:37] ...measurements (/app/public/data/scleractinia/measurements or facts.txt)
[CMD] [2021-04-07 19:37:37] echo "0a" > /app/public/diff/scleractinia_measurements_3700.diff
[CMD] [2021-04-07 19:37:37] tail -n +1 /app/public/converted_csv/scleractinia_measurements_3700.csv >> /app/public/diff/scleractinia_measurements_3700.diff
[CMD] [2021-04-07 19:37:38] echo "." >> /app/public/diff/scleractinia_measurements_3700.diff
[INFO] [2021-04-07 19:37:38] Created diff: /app/public/diff/scleractinia_measurements_3700.diff (2250 lines)
[STOP] [2021-04-07 19:37:38] calculate_delta
[START] [2021-04-07 19:37:38] parse_diff_and_store
[INFO] [2021-04-07 19:37:38] Handling diff: /app/public/diff/scleractinia_agents_3700.diff (2 lines)
[INFO] [2021-04-07 19:37:39] Loading agents diff file into memory (2 /app/public/diff/scleractinia_agents_3700.diff lines)...
[INFO] [2021-04-07 19:37:40] Handling diff: /app/public/diff/scleractinia_refs_3700.diff (2 lines)
[INFO] [2021-04-07 19:37:40] Loading refs diff file into memory (2 /app/public/diff/scleractinia_refs_3700.diff lines)...
[INFO] [2021-04-07 19:37:41] Handling diff: /app/public/diff/scleractinia_nodes_3700.diff (1428 lines)
[INFO] [2021-04-07 19:37:41] Loading nodes diff file into memory (1428 /app/public/diff/scleractinia_nodes_3700.diff lines)...
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Aulocyathus  juvenescens Marenzeller, 1904` to `Aulocyathus juvenescens Marenzeller, 1904`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Cladangia  exusta Lütken, 1873` to `Cladangia exusta Lütken, 1873`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Coenosmilia  inordinata Cairns, 1984` to `Coenosmilia inordinata Cairns, 1984`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Culicia  stellata Dana, 1846` to `Culicia stellata Dana, 1846`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Flabellum  pavoninum Lesson, 1831` to `Flabellum pavoninum Lesson, 1831`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Flabellum curvatum  Moseley, 1881` to `Flabellum curvatum Moseley, 1881`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Heterocyathus antoniae  Reyes, Santodomingo & Cairns, 2009` to `Heterocyathus antoniae Reyes, Santodomingo & Cairns, 2009`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Javania  insignis Duncan, 1876` to `Javania insignis Duncan, 1876`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Labyrinthocyathus  langae Cairns, 1979: 341` to `Labyrinthocyathus langae Cairns, 1979: 341`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Rhizosmilia  gerdae Cairns, 1978` to `Rhizosmilia gerdae Cairns, 1978`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Solenosmilia  variabilis Duncan, 1873` to `Solenosmilia variabilis Duncan, 1873`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Stephanocyathus (Stephanocyathus) isabellae  Reyes, Santodomingo & Cairns, 2009` to `Stephanocyathus (Stephanocyathus) isabellae Reyes, Santodomingo & Cairns, 2009`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Trochocyathus (Trochocyathus)  aithoseptatus Cairns, 1984` to `Trochocyathus (Trochocyathus) aithoseptatus Cairns, 1984`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Trochocyathus (Trochocyathus)  apertus Cairns & Zibrowius, 1997` to `Trochocyathus (Trochocyathus) apertus Cairns & Zibrowius, 1997`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Trochocyathus (Trochocyathus)  caryophylloides Alcock, 1902` to `Trochocyathus (Trochocyathus) caryophylloides Alcock, 1902`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Trochocyathus (Trochocyathus)  cepulla Cairns, 1995` to `Trochocyathus (Trochocyathus) cepulla Cairns, 1995`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Trochocyathus (Trochocyathus)  decamera Cairns, 1994` to `Trochocyathus (Trochocyathus) decamera Cairns, 1994`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Trochocyathus (Trochocyathus)  discus Cairns & Zibrowius, 1997` to `Trochocyathus (Trochocyathus) discus Cairns & Zibrowius, 1997`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Trochocyathus (Trochocyathus)  efateensis Cairns, 1999` to `Trochocyathus (Trochocyathus) efateensis Cairns, 1999`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Trochocyathus (Trochocyathus)  fasciatus Cairns, 1979` to `Trochocyathus (Trochocyathus) fasciatus Cairns, 1979`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Trochocyathus (Trochocyathus)  fossulus Cairns, 1979` to `Trochocyathus (Trochocyathus) fossulus Cairns, 1979`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Trochocyathus (Trochocyathus)  gordoni Cairns, 1995` to `Trochocyathus (Trochocyathus) gordoni Cairns, 1995`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Trochocyathus (Trochocyathus)  japonicus Eguchi, 1968` to `Trochocyathus (Trochocyathus) japonicus Eguchi, 1968`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Trochocyathus (Trochocyathus)  laboreli Cairns, 2000` to `Trochocyathus (Trochocyathus) laboreli Cairns, 2000`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Trochocyathus (Trochocyathus)  maculatus Cairns, 1995` to `Trochocyathus (Trochocyathus) maculatus Cairns, 1995`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Trochocyathus (Trochocyathus)  oahensis Vaughan, 1907` to `Trochocyathus (Trochocyathus) oahensis Vaughan, 1907`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Trochocyathus (Trochocyathus)  patelliformis Cairns, 1999` to `Trochocyathus (Trochocyathus) patelliformis Cairns, 1999`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Trochocyathus (Trochocyathus)  philippinensis Semper, 1872` to `Trochocyathus (Trochocyathus) philippinensis Semper, 1872`
[WARN] [2021-04-07 19:37:42] Filtered Scientific Name `Trochocyathus (Trochocyathus)  rawsonii Pourtalès, 1874` to `Trochocyathus (Trochocyathus) rawsonii Pourtalès, 1874`
[WARN] [2021-04-07 19:37:42] (Reached filtered-name limit; supressing further warnings.)
[INFO] [2021-04-07 19:37:42] Handling diff: /app/public/diff/scleractinia_media_3700.diff (2 lines)
[INFO] [2021-04-07 19:37:43] Loading media diff file into memory (2 /app/public/diff/scleractinia_media_3700.diff lines)...
[INFO] [2021-04-07 19:37:43] Handling diff: /app/public/diff/scleractinia_vernaculars_3700.diff (2 lines)
[INFO] [2021-04-07 19:37:44] Loading vernaculars diff file into memory (2 /app/public/diff/scleractinia_vernaculars_3700.diff lines)...
[INFO] [2021-04-07 19:37:45] Handling diff: /app/public/diff/scleractinia_occurrences_3700.diff (1428 lines)
[INFO] [2021-04-07 19:37:45] Loading occurrences diff file into memory (1428 /app/public/diff/scleractinia_occurrences_3700.diff lines)...
[INFO] [2021-04-07 19:37:46] Handling diff: /app/public/diff/scleractinia_assocs_3700.diff (2 lines)
[INFO] [2021-04-07 19:37:46] Loading assocs diff file into memory (2 /app/public/diff/scleractinia_assocs_3700.diff lines)...
[INFO] [2021-04-07 19:37:47] Handling diff: /app/public/diff/scleractinia_measurements_3700.diff (2250 lines)
[INFO] [2021-04-07 19:37:47] Loading measurements diff file into memory (2250 /app/public/diff/scleractinia_measurements_3700.diff lines)...
[INFO] [2021-04-07 19:37:49] Storing 1430 ScientificNames
[INFO] [2021-04-07 19:37:49] Processing group of 1430 in 2 groups of 1000
[INFO] [2021-04-07 19:37:50] Average Time: 0.22
[INFO] [2021-04-07 19:37:50] Total Time: 1s
[INFO] [2021-04-07 19:37:50] Storing 1430 Nodes
[INFO] [2021-04-07 19:37:50] Processing group of 1430 in 2 groups of 1000
[INFO] [2021-04-07 19:37:50] Average Time: 0.17
[INFO] [2021-04-07 19:37:50] Total Time: 1s
[INFO] [2021-04-07 19:37:50] Storing 1426 Occurrences
[INFO] [2021-04-07 19:37:50] Processing group of 1426 in 2 groups of 1000
[INFO] [2021-04-07 19:37:50] Average Time: 0.13
[INFO] [2021-04-07 19:37:50] Total Time: 1s
[INFO] [2021-04-07 19:37:50] Storing 2248 Traits
[INFO] [2021-04-07 19:37:50] Processing group of 2248 in 3 groups of 1000
[INFO] [2021-04-07 19:37:51] Average Time: 0.2
[INFO] [2021-04-07 19:37:51] Total Time: 1s
[INFO] [2021-04-07 19:37:51] Storing 4496 MetaTraits
[INFO] [2021-04-07 19:37:51] Processing group of 4496 in 5 groups of 1000
[INFO] [2021-04-07 19:37:51] Average Time: 0.098
[INFO] [2021-04-07 19:37:51] Total Time: 1s
[STOP] [2021-04-07 19:37:51] parse_diff_and_store
[START] [2021-04-07 19:37:51] resolve_keys
[INFO] [2021-04-07 19:37:57] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-07 19:37:57] traits to occurrences...
[INFO] [2021-04-07 19:37:57] traits to nodes (through occurrences)...
[INFO] [2021-04-07 19:37:57] Traits to sex term...
[INFO] [2021-04-07 19:37:57] Traits to lifestage term...
[INFO] [2021-04-07 19:37:57] MetaTraits to traits...
[INFO] [2021-04-07 19:37:58] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-07 19:37:58] Assocs to occurrences...
[INFO] [2021-04-07 19:37:58] Assocs to nodes...
[INFO] [2021-04-07 19:37:58] Assoc to sex term...
[INFO] [2021-04-07 19:37:58] Assoc to lifestage term...
[INFO] [2021-04-07 19:37:58] MetaAssoc to assocs...
[STOP] [2021-04-07 19:37:58] resolve_keys
[START] [2021-04-07 19:37:58] hold_for_later_1
[STOP] [2021-04-07 19:37:58] hold_for_later_1
[START] [2021-04-07 19:37:58] hold_for_later_2
[STOP] [2021-04-07 19:37:58] hold_for_later_2
[START] [2021-04-07 19:37:58] resolve_missing_parents
[STOP] [2021-04-07 19:37:58] resolve_missing_parents
[START] [2021-04-07 19:37:58] rebuild_nodes
[START] [2021-04-07 19:37:58] Flattener#flatten
[START] [2021-04-07 19:37:58] Flattener#study_resource
[START] [2021-04-07 19:37:58] Flattener#build_ancestry
[STOP] [2021-04-07 19:37:58] Flattener#build_ancestry
[INFO] [2021-04-07 19:37:58] 1430 ancestry keys
[START] [2021-04-07 19:37:58] build_node_ancestors
[INFO] [2021-04-07 19:37:58] old ancestors deleted.
[STOP] [2021-04-07 19:37:58] build_node_ancestors
[START] [2021-04-07 19:37:58] Flattener#propagate_ancestor_ids
[STOP] [2021-04-07 19:37:58] Flattener#propagate_ancestor_ids
[STOP] [2021-04-07 19:37:58] Flattener#flatten
[STOP] [2021-04-07 19:37:58] rebuild_nodes
[START] [2021-04-07 19:37:58] resolve_missing_media_owners
[STOP] [2021-04-07 19:37:58] resolve_missing_media_owners
[START] [2021-04-07 19:37:58] sanitize_media_verbatims
[STOP] [2021-04-07 19:37:58] sanitize_media_verbatims
[START] [2021-04-07 19:37:58] queue_downloads
[STOP] [2021-04-07 19:37:58] queue_downloads
[START] [2021-04-07 19:37:58] parse_names
[WARN] [2021-04-07 19:37:58] I see 1430 names which still need to be parsed.
[STOP] [2021-04-07 19:38:01] parse_names
[START] [2021-04-07 19:38:01] denormalize_canonical_names_to_nodes
[STOP] [2021-04-07 19:38:01] denormalize_canonical_names_to_nodes
[START] [2021-04-07 19:38:01] match_nodes
[START] [2021-04-07 19:38:01] map_all_nodes_to_pages
[STOP] [2021-04-07 19:38:47] map_all_nodes_to_pages
[INFO] [2021-04-07 19:38:47] 23 Unmatched nodes (of 1430)! That's too many to output. Full list in /app/public/data/scleractinia/unmatched_nodes.txt ; First 10: Acanthastrea rotundaflora (#92680854); Acropora jacquelinae (#92680903); Acropora lutkeni (#92680915); Acropora mirabilis (#92680921); Acropora pruinosa (#92680942); Acropora teres (#92680968); Coscinaraea hazimanensis (#92681214); Deltocyathus agassizii (#92681251); Deltocyathus inusitatus (#92681259); Dendrophyllia boschmai cyathoheloides (#92681276)
[START] [2021-04-07 19:38:47] update_nodes
[STOP] [2021-04-07 19:38:47] update_nodes
[STOP] [2021-04-07 19:38:47] match_nodes
[START] [2021-04-07 19:38:47] reindex_search
[STOP] [2021-04-07 19:38:48] reindex_search
[START] [2021-04-07 19:38:48] normalize_units
[STOP] [2021-04-07 19:38:48] normalize_units
[START] [2021-04-07 19:38:48] calculate_statistics
[STOP] [2021-04-07 19:38:48] calculate_statistics
[START] [2021-04-07 19:38:48] complete_harvest_instance
[START] [2021-04-07 19:38:48] overall_tsv_creation
[INFO] [2021-04-07 19:38:48] Processing group of 1430 in 1 batches of 10000
[INFO] [2021-04-07 19:39:29] 2248 Traits (unfiltered)...
[INFO] [2021-04-07 19:40:17] 2248 Traits (filtered)...
[INFO] [2021-04-07 19:40:17] 0 Associations (filtered)...
[INFO] [2021-04-07 19:40:17] 2248 metadata added.
[INFO] [2021-04-07 19:40:17] 0 metadata added.
[INFO] [2021-04-07 19:40:45] Average Time: 91.82
[INFO] [2021-04-07 19:40:45] Total Time: 1m57s
[STOP] [2021-04-07 19:40:45] overall_tsv_creation
[INFO] [2021-04-07 19:40:45] Done. Check your files:
[INFO] [2021-04-07 19:40:45] (1430 lines) /app/public/data/scleractinia/publish_nodes.tsv
[INFO] [2021-04-07 19:40:46] (5710 lines) /app/public/data/scleractinia/publish_node_ancestors.tsv
[INFO] [2021-04-07 19:40:46] (1430 lines) /app/public/data/scleractinia/publish_scientific_names.tsv
[INFO] [2021-04-07 19:40:47] (2249 lines) /app/public/data/scleractinia/publish_traits.tsv
[INFO] [2021-04-07 19:40:47] (2249 lines) /app/public/data/scleractinia/publish_metadata.tsv
[STOP] [2021-04-07 19:40:47] complete_harvest_instance
[START] [2021-04-07 19:40:47] completed
[STOP] [2021-04-07 19:40:47] completed
[STOP] [2021-04-07 19:40:47] logged process, took 207.84

Latest Process