Stage:
completed
Fetched:
04 Apr 09:10
Validated:
04 Apr 09:10
Deltas Created
04 Apr 09:10
Units Normalized:
04 Apr 09:11
Ancestry Built:
04 Apr 09:11
Nodes Matched:
04 Apr 09:11
Names Parsed:
04 Apr 09:11
New Models Stored:
04 Apr 09:10
Indexed:
04 Apr 09:11
Completed:
04 Apr 09:12
Time to Harvest:
less than a minute
Harvesting Log
(667 lines)
# Logfile created on 2020-11-05 16:06:07 -0500 by logger.rb/v1.4.2
[INFO] [2020-11-05 16:06:07] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-11-05 16:06:10] ## remove_type: ScientificName
[INFO] [2020-11-05 16:06:10] ++ Calling delete_all on 208 instances...
[INFO] [2020-11-05 16:06:10] [16:06:10.669] Removed 208 Scientificnames
[INFO] [2020-11-05 16:06:10] ## remove_type: Vernacular
[INFO] [2020-11-05 16:06:10] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:06:10] [16:06:10.671] Removed 0 Vernaculars
[INFO] [2020-11-05 16:06:10] ## remove_type: Article
[INFO] [2020-11-05 16:06:10] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:06:10] [16:06:10.674] Removed 0 Articles
[INFO] [2020-11-05 16:06:10] ## remove_type: Medium
[INFO] [2020-11-05 16:06:10] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:06:10] [16:06:10.678] Removed 0 Media
[INFO] [2020-11-05 16:06:10] ## remove_type: Trait
[INFO] [2020-11-05 16:06:10] ++ Calling delete_all on 1165 instances...
[INFO] [2020-11-05 16:06:10] [16:06:10.846] Removed 1165 Traits
[INFO] [2020-11-05 16:06:10] ## remove_type: MetaTrait
[INFO] [2020-11-05 16:06:10] ++ Calling delete_all on 2330 instances...
[INFO] [2020-11-05 16:06:10] [16:06:10.978] Removed 2330 Metatraits
[INFO] [2020-11-05 16:06:10] ## remove_type: OccurrenceMetadatum
[INFO] [2020-11-05 16:06:10] ++ Calling delete_all on 1165 instances...
[INFO] [2020-11-05 16:06:11] [16:06:11.000] Removed 1165 Occurrencemetadata
[INFO] [2020-11-05 16:06:11] ## remove_type: Assoc
[INFO] [2020-11-05 16:06:11] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:06:11] [16:06:11.003] Removed 0 Assocs
[INFO] [2020-11-05 16:06:11] ## remove_type: MetaAssoc
[INFO] [2020-11-05 16:06:11] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:06:11] [16:06:11.005] Removed 0 Metaassocs
[INFO] [2020-11-05 16:06:11] ## remove_type: Identifier
[INFO] [2020-11-05 16:06:11] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:06:11] [16:06:11.008] Removed 0 Identifiers
[INFO] [2020-11-05 16:06:11] ## remove_type: Reference
[INFO] [2020-11-05 16:06:11] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:06:11] [16:06:11.011] Removed 0 References
[INFO] [2020-11-05 16:06:11] Starting batch with ID 41001116...
[INFO] [2020-11-05 16:06:11] Starting batch with ID 41001116...
[INFO] [2020-11-05 16:06:11] Starting batch with ID 41001116...
[INFO] [2020-11-05 16:06:11] Starting batch with ID 41001116...
[INFO] [2020-11-05 16:06:11] Starting batch with ID 41001116...
[INFO] [2020-11-05 16:06:11] Starting batch with ID 41001116...
[INFO] [2020-11-05 16:06:11] Starting batch with ID 41001116...
[INFO] [2020-11-05 16:06:11] Starting batch with ID 41001116...
[INFO] [2020-11-05 16:06:11] Starting batch with ID 41001116...
[INFO] [2020-11-05 16:06:11] Starting batch with ID 41001116...
[INFO] [2020-11-05 16:06:11] Starting batch with ID 41001116...
[INFO] [2020-11-05 16:06:11] ## remove_type: Node
[INFO] [2020-11-05 16:06:11] ++ Calling delete_all on 208 instances...
[INFO] [2020-11-05 16:06:11] [16:06:11.732] Removed 208 Nodes
[START] [2020-11-05 16:06:13] logged process: 9719e9b2443e9b331c213f6700b6e76a1b7c8548
[START] [2020-11-05 16:06:13] Creating resource from OpenData
[START] [2020-11-05 16:06:13] logged process: 9719e9b2443e9b331c213f6700b6e76a1b7c8548
[START] [2020-11-05 16:06:13] Parse meta.xml file and create formats with fields
[WARN] [2020-11-05 16:06:13] (common) IGNORED (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2020-11-05 16:06:13] (common) IGNORED (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2020-11-05 16:06:13] (common) IGNORED (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2020-11-05 16:06:13] (common) IGNORED (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2020-11-05 16:06:13] (common) IGNORED (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2020-11-05 16:06:13] (common) IGNORED (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2020-11-05 16:06:13] (common) IGNORED (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[WARN] [2020-11-05 16:06:13] (common) IGNORED (occurrences) field header: Institution Code term: http://rs.tdwg.org/dwc/terms/institutionCode
[WARN] [2020-11-05 16:06:13] (common) IGNORED (occurrences) field header: Collection Code term: http://rs.tdwg.org/dwc/terms/collectionCode
[WARN] [2020-11-05 16:06:13] (common) IGNORED (occurrences) field header: Catalog Number term: http://rs.tdwg.org/dwc/terms/catalogNumber
[WARN] [2020-11-05 16:06:13] SKIPPING http://rs.tdwg.org/dwc/terms/Event config (events.txt)...
[STOP] [2020-11-05 16:06:13] Parse meta.xml file and create formats with fields
[STOP] [2020-11-05 16:06:13] Creating resource from OpenData
[START] [2020-11-05 16:06:13] logged process: 9719e9b2443e9b331c213f6700b6e76a1b7c8548
[START] [2020-11-05 16:06:13] create_harvest_instance
[STOP] [2020-11-05 16:06:22] create_harvest_instance
[START] [2020-11-05 16:06:22] fetch_files
[STOP] [2020-11-05 16:06:22] fetch_files
[START] [2020-11-05 16:06:22] validate_each_file
[STOP] [2020-11-05 16:06:23] validate_each_file
[ERR] [2020-11-05 16:06:23] Exceptions::ColumnMissing
[ERR] [2020-11-05 16:06:23] MISSING COLUMN: measurements:
[ERR] [2020-11-05 16:06:23] ../models/resource_harvester.rb:154:in `block in check_each_column'
[ERR] [2020-11-05 16:06:23] ../models/resource_harvester.rb:153:in `each_with_index'
[ERR] [2020-11-05 16:06:23] ../models/resource_harvester.rb:153:in `check_each_column'
[ERR] [2020-11-05 16:06:23] ../models/resource_harvester.rb:132:in `block in validate_each_file'
[ERR] [2020-11-05 16:06:23] ../models/resource_harvester.rb:682:in `block in each_format'
[ERR] [2020-11-05 16:06:23] ../models/resource_harvester.rb:667:in `each_format'
[ERR] [2020-11-05 16:06:23] ../models/resource_harvester.rb:129:in `validate_each_file'
[ERR] [2020-11-05 16:06:23] ../models/resource_harvester.rb:85:in `block (3 levels) in start'
[ERR] [2020-11-05 16:06:23] ../models/logged_process.rb:19:in `run_step'
[ERR] [2020-11-05 16:06:23] ../models/resource_harvester.rb:85:in `block (2 levels) in start'
[ERR] [2020-11-05 16:06:23] ../models/resource_harvester.rb:74:in `each_key'
[ERR] [2020-11-05 16:06:23] ../models/resource_harvester.rb:74:in `block in start'
[ERR] [2020-11-05 16:06:23] ../models/resource.rb:151:in `lock'
[ERR] [2020-11-05 16:06:23] ../models/resource_harvester.rb:72:in `start'
[ERR] [2020-11-05 16:06:23] ../models/resource.rb:232:in `harvest'
[ERR] [2020-11-05 16:06:23] ../models/resource.rb:208:in `re_download_opendata_and_harvest'
[ERR] [2020-11-05 16:06:23] bin/rails:4:in `require'
[ERR] [2020-11-05 16:06:23] bin/rails:4:in `<main>'
[STOP] [2020-11-05 16:06:23] logged process, took 9.72
[INFO] [2020-11-05 16:12:05] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-11-05 16:12:07] ## remove_type: ScientificName
[INFO] [2020-11-05 16:12:07] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:12:07] [16:12:07.628] Removed 0 Scientificnames
[INFO] [2020-11-05 16:12:07] ## remove_type: Vernacular
[INFO] [2020-11-05 16:12:07] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:12:07] [16:12:07.631] Removed 0 Vernaculars
[INFO] [2020-11-05 16:12:07] ## remove_type: Article
[INFO] [2020-11-05 16:12:07] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:12:07] [16:12:07.634] Removed 0 Articles
[INFO] [2020-11-05 16:12:07] ## remove_type: Medium
[INFO] [2020-11-05 16:12:07] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:12:07] [16:12:07.638] Removed 0 Media
[INFO] [2020-11-05 16:12:07] ## remove_type: Trait
[INFO] [2020-11-05 16:12:07] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:12:07] [16:12:07.641] Removed 0 Traits
[INFO] [2020-11-05 16:12:07] ## remove_type: MetaTrait
[INFO] [2020-11-05 16:12:07] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:12:07] [16:12:07.643] Removed 0 Metatraits
[INFO] [2020-11-05 16:12:07] ## remove_type: OccurrenceMetadatum
[INFO] [2020-11-05 16:12:07] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:12:07] [16:12:07.646] Removed 0 Occurrencemetadata
[INFO] [2020-11-05 16:12:07] ## remove_type: Assoc
[INFO] [2020-11-05 16:12:07] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:12:07] [16:12:07.650] Removed 0 Assocs
[INFO] [2020-11-05 16:12:07] ## remove_type: MetaAssoc
[INFO] [2020-11-05 16:12:07] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:12:07] [16:12:07.652] Removed 0 Metaassocs
[INFO] [2020-11-05 16:12:07] ## remove_type: Identifier
[INFO] [2020-11-05 16:12:07] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:12:07] [16:12:07.655] Removed 0 Identifiers
[INFO] [2020-11-05 16:12:07] ## remove_type: Reference
[INFO] [2020-11-05 16:12:07] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:12:07] [16:12:07.657] Removed 0 References
[INFO] [2020-11-05 16:12:07] ## remove_type: Node
[INFO] [2020-11-05 16:12:07] ++ Calling delete_all on 0 instances...
[INFO] [2020-11-05 16:12:07] [16:12:07.677] Removed 0 Nodes
[START] [2020-11-05 16:12:08] logged process: 9719e9b2443e9b331c213f6700b6e76a1b7c8548
[START] [2020-11-05 16:12:08] Creating resource from OpenData
[START] [2020-11-05 16:12:08] logged process: 9719e9b2443e9b331c213f6700b6e76a1b7c8548
[START] [2020-11-05 16:12:08] Parse meta.xml file and create formats with fields
[WARN] [2020-11-05 16:12:08] (common) IGNORED (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2020-11-05 16:12:08] (common) IGNORED (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2020-11-05 16:12:08] (common) IGNORED (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2020-11-05 16:12:08] (common) IGNORED (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2020-11-05 16:12:08] (common) IGNORED (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2020-11-05 16:12:08] (common) IGNORED (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2020-11-05 16:12:08] (common) IGNORED (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[WARN] [2020-11-05 16:12:08] (common) IGNORED (occurrences) field header: Institution Code term: http://rs.tdwg.org/dwc/terms/institutionCode
[WARN] [2020-11-05 16:12:08] (common) IGNORED (occurrences) field header: Collection Code term: http://rs.tdwg.org/dwc/terms/collectionCode
[WARN] [2020-11-05 16:12:08] (common) IGNORED (occurrences) field header: Catalog Number term: http://rs.tdwg.org/dwc/terms/catalogNumber
[WARN] [2020-11-05 16:12:08] SKIPPING http://rs.tdwg.org/dwc/terms/Event config (events.txt)...
[STOP] [2020-11-05 16:12:08] Parse meta.xml file and create formats with fields
[STOP] [2020-11-05 16:12:08] Creating resource from OpenData
[START] [2020-11-05 16:12:08] logged process: 9719e9b2443e9b331c213f6700b6e76a1b7c8548
[START] [2020-11-05 16:12:08] create_harvest_instance
[STOP] [2020-11-05 16:12:18] create_harvest_instance
[START] [2020-11-05 16:12:18] fetch_files
[STOP] [2020-11-05 16:12:18] fetch_files
[START] [2020-11-05 16:12:18] validate_each_file
[STOP] [2020-11-05 16:12:18] validate_each_file
[START] [2020-11-05 16:12:18] convert_to_csv
[CMD] [2020-11-05 16:12:18] /usr/bin/sort /app/public/converted_csv/fcsfi_agents_23506.csv > /app/public/converted_csv/fcsfi_agents_23506.csv_sorted
[CMD] [2020-11-05 16:12:18] /usr/bin/sort /app/public/converted_csv/fcsfi_refs_23507.csv > /app/public/converted_csv/fcsfi_refs_23507.csv_sorted
[CMD] [2020-11-05 16:12:18] /usr/bin/sort /app/public/converted_csv/fcsfi_nodes_23508.csv > /app/public/converted_csv/fcsfi_nodes_23508.csv_sorted
[CMD] [2020-11-05 16:12:18] /usr/bin/sort /app/public/converted_csv/fcsfi_media_23509.csv > /app/public/converted_csv/fcsfi_media_23509.csv_sorted
[CMD] [2020-11-05 16:12:18] /usr/bin/sort /app/public/converted_csv/fcsfi_vernaculars_23510.csv > /app/public/converted_csv/fcsfi_vernaculars_23510.csv_sorted
[CMD] [2020-11-05 16:12:18] /usr/bin/sort /app/public/converted_csv/fcsfi_occurrences_23511.csv > /app/public/converted_csv/fcsfi_occurrences_23511.csv_sorted
[CMD] [2020-11-05 16:12:18] /usr/bin/sort /app/public/converted_csv/fcsfi_assocs_23512.csv > /app/public/converted_csv/fcsfi_assocs_23512.csv_sorted
[CMD] [2020-11-05 16:12:18] /usr/bin/sort /app/public/converted_csv/fcsfi_measurements_23513.csv > /app/public/converted_csv/fcsfi_measurements_23513.csv_sorted
[STOP] [2020-11-05 16:12:18] convert_to_csv
[START] [2020-11-05 16:12:18] calculate_delta
[CMD] [2020-11-05 16:12:18] echo "0a" > /app/public/diff/fcsfi_agents_23506.diff
[CMD] [2020-11-05 16:12:18] tail -n +1 /app/public/converted_csv/fcsfi_agents_23506.csv >> /app/public/diff/fcsfi_agents_23506.diff
[CMD] [2020-11-05 16:12:18] echo "." >> /app/public/diff/fcsfi_agents_23506.diff
[CMD] [2020-11-05 16:12:18] echo "0a" > /app/public/diff/fcsfi_refs_23507.diff
[CMD] [2020-11-05 16:12:18] tail -n +1 /app/public/converted_csv/fcsfi_refs_23507.csv >> /app/public/diff/fcsfi_refs_23507.diff
[CMD] [2020-11-05 16:12:18] echo "." >> /app/public/diff/fcsfi_refs_23507.diff
[CMD] [2020-11-05 16:12:18] echo "0a" > /app/public/diff/fcsfi_nodes_23508.diff
[CMD] [2020-11-05 16:12:18] tail -n +1 /app/public/converted_csv/fcsfi_nodes_23508.csv >> /app/public/diff/fcsfi_nodes_23508.diff
[CMD] [2020-11-05 16:12:18] echo "." >> /app/public/diff/fcsfi_nodes_23508.diff
[CMD] [2020-11-05 16:12:18] echo "0a" > /app/public/diff/fcsfi_media_23509.diff
[CMD] [2020-11-05 16:12:18] tail -n +1 /app/public/converted_csv/fcsfi_media_23509.csv >> /app/public/diff/fcsfi_media_23509.diff
[CMD] [2020-11-05 16:12:18] echo "." >> /app/public/diff/fcsfi_media_23509.diff
[CMD] [2020-11-05 16:12:18] echo "0a" > /app/public/diff/fcsfi_vernaculars_23510.diff
[CMD] [2020-11-05 16:12:18] tail -n +1 /app/public/converted_csv/fcsfi_vernaculars_23510.csv >> /app/public/diff/fcsfi_vernaculars_23510.diff
[CMD] [2020-11-05 16:12:18] echo "." >> /app/public/diff/fcsfi_vernaculars_23510.diff
[CMD] [2020-11-05 16:12:18] echo "0a" > /app/public/diff/fcsfi_occurrences_23511.diff
[CMD] [2020-11-05 16:12:18] tail -n +1 /app/public/converted_csv/fcsfi_occurrences_23511.csv >> /app/public/diff/fcsfi_occurrences_23511.diff
[CMD] [2020-11-05 16:12:18] echo "." >> /app/public/diff/fcsfi_occurrences_23511.diff
[CMD] [2020-11-05 16:12:18] echo "0a" > /app/public/diff/fcsfi_assocs_23512.diff
[CMD] [2020-11-05 16:12:18] tail -n +1 /app/public/converted_csv/fcsfi_assocs_23512.csv >> /app/public/diff/fcsfi_assocs_23512.diff
[CMD] [2020-11-05 16:12:18] echo "." >> /app/public/diff/fcsfi_assocs_23512.diff
[CMD] [2020-11-05 16:12:18] echo "0a" > /app/public/diff/fcsfi_measurements_23513.diff
[CMD] [2020-11-05 16:12:18] tail -n +1 /app/public/converted_csv/fcsfi_measurements_23513.csv >> /app/public/diff/fcsfi_measurements_23513.diff
[CMD] [2020-11-05 16:12:18] echo "." >> /app/public/diff/fcsfi_measurements_23513.diff
[STOP] [2020-11-05 16:12:18] calculate_delta
[START] [2020-11-05 16:12:18] parse_diff_and_store
[INFO] [2020-11-05 16:12:18] Loading agents diff file into memory (true lines)...
[INFO] [2020-11-05 16:12:18] Loading refs diff file into memory (true lines)...
[INFO] [2020-11-05 16:12:18] Loading nodes diff file into memory (true lines)...
[WARN] [2020-11-05 16:12:18] Filtered Scientific Name `Anemone narcissiflora var. monantha` to `Anemone narcissiflora var. monantha`
[INFO] [2020-11-05 16:12:19] Loading media diff file into memory (true lines)...
[INFO] [2020-11-05 16:12:19] Loading vernaculars diff file into memory (true lines)...
[INFO] [2020-11-05 16:12:19] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-11-05 16:12:19] Loading assocs diff file into memory (true lines)...
[INFO] [2020-11-05 16:12:19] Loading measurements diff file into memory (true lines)...
[INFO] [2020-11-05 16:12:20] Storing 208 ScientificNames
[INFO] [2020-11-05 16:12:20] Processing group of 208 in 1 groups of 1000
[INFO] [2020-11-05 16:12:20] Average Time: 0.08
[INFO] [2020-11-05 16:12:20] Total Time: 1s
[INFO] [2020-11-05 16:12:20] Storing 208 Nodes
[INFO] [2020-11-05 16:12:20] Processing group of 208 in 1 groups of 1000
[INFO] [2020-11-05 16:12:21] Average Time: 0.07
[INFO] [2020-11-05 16:12:21] Total Time: 1s
[INFO] [2020-11-05 16:12:21] Storing 1165 Occurrences
[INFO] [2020-11-05 16:12:21] Processing group of 1165 in 2 groups of 1000
[INFO] [2020-11-05 16:12:21] Average Time: 0.09
[INFO] [2020-11-05 16:12:21] Total Time: 1s
[INFO] [2020-11-05 16:12:21] Storing 1165 Traits
[INFO] [2020-11-05 16:12:21] Processing group of 1165 in 2 groups of 1000
[INFO] [2020-11-05 16:12:21] Average Time: 0.165
[INFO] [2020-11-05 16:12:21] Total Time: 1s
[INFO] [2020-11-05 16:12:21] Storing 3495 MetaTraits
[INFO] [2020-11-05 16:12:21] Processing group of 3495 in 4 groups of 1000
[INFO] [2020-11-05 16:12:22] Average Time: 0.093
[INFO] [2020-11-05 16:12:22] Total Time: 1s
[STOP] [2020-11-05 16:12:22] parse_diff_and_store
[START] [2020-11-05 16:12:22] resolve_keys
[INFO] [2020-11-05 16:12:28] Occurrences to nodes (through scientific_names)...
[INFO] [2020-11-05 16:12:28] traits to occurrences...
[INFO] [2020-11-05 16:12:28] traits to nodes (through occurrences)...
[INFO] [2020-11-05 16:12:28] Traits to sex term...
[INFO] [2020-11-05 16:12:28] Traits to lifestage term...
[INFO] [2020-11-05 16:12:28] MetaTraits to traits...
[INFO] [2020-11-05 16:12:28] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-11-05 16:12:28] Assocs to occurrences...
[INFO] [2020-11-05 16:12:28] Assocs to nodes...
[INFO] [2020-11-05 16:12:28] Assoc to sex term...
[INFO] [2020-11-05 16:12:28] Assoc to lifestage term...
[INFO] [2020-11-05 16:12:28] MetaAssoc to assocs...
[STOP] [2020-11-05 16:12:28] resolve_keys
[START] [2020-11-05 16:12:28] hold_for_later_1
[STOP] [2020-11-05 16:12:28] hold_for_later_1
[START] [2020-11-05 16:12:28] hold_for_later_2
[STOP] [2020-11-05 16:12:28] hold_for_later_2
[START] [2020-11-05 16:12:28] resolve_missing_parents
[STOP] [2020-11-05 16:12:28] resolve_missing_parents
[START] [2020-11-05 16:12:28] rebuild_nodes
[START] [2020-11-05 16:12:28] Flattener#flatten
[START] [2020-11-05 16:12:28] Flattener#study_resource
[START] [2020-11-05 16:12:28] Flattener#build_ancestry
[STOP] [2020-11-05 16:12:28] Flattener#build_ancestry
[INFO] [2020-11-05 16:12:28] 208 ancestry keys
[START] [2020-11-05 16:12:28] build_node_ancestors
[INFO] [2020-11-05 16:12:28] old ancestors deleted.
[STOP] [2020-11-05 16:12:28] build_node_ancestors
[START] [2020-11-05 16:12:28] Flattener#propagate_ancestor_ids
[STOP] [2020-11-05 16:12:28] Flattener#propagate_ancestor_ids
[STOP] [2020-11-05 16:12:28] Flattener#flatten
[STOP] [2020-11-05 16:12:28] rebuild_nodes
[START] [2020-11-05 16:12:28] resolve_missing_media_owners
[STOP] [2020-11-05 16:12:28] resolve_missing_media_owners
[START] [2020-11-05 16:12:28] sanitize_media_verbatims
[STOP] [2020-11-05 16:12:28] sanitize_media_verbatims
[START] [2020-11-05 16:12:28] queue_downloads
[STOP] [2020-11-05 16:12:28] queue_downloads
[START] [2020-11-05 16:12:28] parse_names
[WARN] [2020-11-05 16:12:28] I see 208 names which still need to be parsed.
[STOP] [2020-11-05 16:12:29] parse_names
[START] [2020-11-05 16:12:29] denormalize_canonical_names_to_nodes
[STOP] [2020-11-05 16:12:29] denormalize_canonical_names_to_nodes
[START] [2020-11-05 16:12:29] match_nodes
[START] [2020-11-05 16:12:29] map_all_nodes_to_pages
[STOP] [2020-11-05 16:12:39] map_all_nodes_to_pages
[INFO] [2020-11-05 16:12:39] Unmatched nodes (5 of 208): plantae (#81416372); Anemone drummondii lithophila (#81416381); Nemophila pulchella pulchella (#81416518); Pholistoma auritum auritum (#81416535); Asteracea (#81416466)
[START] [2020-11-05 16:12:39] update_nodes
[STOP] [2020-11-05 16:12:39] update_nodes
[STOP] [2020-11-05 16:12:39] match_nodes
[START] [2020-11-05 16:12:39] reindex_search
[STOP] [2020-11-05 16:12:39] reindex_search
[START] [2020-11-05 16:12:39] normalize_units
[STOP] [2020-11-05 16:12:39] normalize_units
[START] [2020-11-05 16:12:39] calculate_statistics
[STOP] [2020-11-05 16:12:39] calculate_statistics
[START] [2020-11-05 16:12:39] complete_harvest_instance
[START] [2020-11-05 16:12:39] overall_tsv_creation
[INFO] [2020-11-05 16:12:39] Processing group of 208 in 1 batches of 10000
[INFO] [2020-11-05 16:13:21] 1165 Traits (unfiltered)...
[INFO] [2020-11-05 16:14:03] 1165 Traits (filtered)...
[INFO] [2020-11-05 16:14:03] 0 Associations (filtered)...
[INFO] [2020-11-05 16:14:03] 3495 metadata added.
[INFO] [2020-11-05 16:14:03] 0 metadata added.
[INFO] [2020-11-05 16:14:03] Average Time: 59.5
[INFO] [2020-11-05 16:14:03] Total Time: 1m25s
[STOP] [2020-11-05 16:14:03] overall_tsv_creation
[INFO] [2020-11-05 16:14:03] Done. Check your files:
[INFO] [2020-11-05 16:14:03] (208 lines) /app/public/data/fcsfi/publish_nodes.tsv
[INFO] [2020-11-05 16:14:03] (389 lines) /app/public/data/fcsfi/publish_node_ancestors.tsv
[INFO] [2020-11-05 16:14:03] (208 lines) /app/public/data/fcsfi/publish_scientific_names.tsv
[INFO] [2020-11-05 16:14:03] (1166 lines) /app/public/data/fcsfi/publish_traits.tsv
[INFO] [2020-11-05 16:14:03] (3496 lines) /app/public/data/fcsfi/publish_metadata.tsv
[STOP] [2020-11-05 16:14:03] complete_harvest_instance
[START] [2020-11-05 16:14:03] completed
[STOP] [2020-11-05 16:14:03] completed
[STOP] [2020-11-05 16:14:03] logged process, took 114.97
[INFO] [2021-04-04 09:09:30] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-04 09:09:35] ## remove_type: ScientificName
[INFO] [2021-04-04 09:09:35] ++ Calling delete_all on 208 instances...
[INFO] [2021-04-04 09:09:35] [09:09:35.672] Removed 208 Scientificnames
[INFO] [2021-04-04 09:09:35] ## remove_type: Vernacular
[INFO] [2021-04-04 09:09:35] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-04 09:09:35] [09:09:35.673] Removed 0 Vernaculars
[INFO] [2021-04-04 09:09:35] ## remove_type: Article
[INFO] [2021-04-04 09:09:35] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-04 09:09:35] [09:09:35.675] Removed 0 Articles
[INFO] [2021-04-04 09:09:35] ## remove_type: Medium
[INFO] [2021-04-04 09:09:35] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-04 09:09:35] [09:09:35.676] Removed 0 Media
[INFO] [2021-04-04 09:09:35] ## remove_type: Trait
[INFO] [2021-04-04 09:09:35] ++ Calling delete_all on 1165 instances...
[INFO] [2021-04-04 09:09:35] [09:09:35.907] Removed 1165 Traits
[INFO] [2021-04-04 09:09:35] ## remove_type: MetaTrait
[INFO] [2021-04-04 09:09:35] ++ Calling delete_all on 3495 instances...
[INFO] [2021-04-04 09:09:36] [09:09:36.091] Removed 3495 Metatraits
[INFO] [2021-04-04 09:09:36] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-04 09:09:36] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-04 09:09:36] [09:09:36.137] Removed 0 Occurrencemetadata
[INFO] [2021-04-04 09:09:36] ## remove_type: Assoc
[INFO] [2021-04-04 09:09:36] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-04 09:09:36] [09:09:36.138] Removed 0 Assocs
[INFO] [2021-04-04 09:09:36] ## remove_type: MetaAssoc
[INFO] [2021-04-04 09:09:36] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-04 09:09:36] [09:09:36.140] Removed 0 Metaassocs
[INFO] [2021-04-04 09:09:36] ## remove_type: Identifier
[INFO] [2021-04-04 09:09:36] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-04 09:09:36] [09:09:36.167] Removed 0 Identifiers
[INFO] [2021-04-04 09:09:36] ## remove_type: Reference
[INFO] [2021-04-04 09:09:36] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-04 09:09:36] [09:09:36.168] Removed 0 References
[INFO] [2021-04-04 09:09:36] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:36] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:36] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:36] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:36] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:36] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:36] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:36] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] Starting batch with ID 81416372...
[INFO] [2021-04-04 09:09:37] ## remove_type: Node
[INFO] [2021-04-04 09:09:37] ++ Calling delete_all on 208 instances...
[INFO] [2021-04-04 09:09:37] [09:09:37.921] Removed 208 Nodes
[START] [2021-04-04 09:09:39] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-04 09:09:39] Creating resource from OpenData
[START] [2021-04-04 09:09:41] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-04 09:09:41] Parse meta.xml file and create formats with fields
[WARN] [2021-04-04 09:10:05] (common) IGNORED (media) field header: ThumbnailURL term: http://eol.org/schema/media/thumbnailURL
[WARN] [2021-04-04 09:10:05] (common) IGNORED (media) field header: DerivedFrom term: http://rs.tdwg.org/ac/terms/derivedFrom
[WARN] [2021-04-04 09:10:05] (common) IGNORED (media) field header: CreateDate term: http://ns.adobe.com/xap/1.0/CreateDate
[WARN] [2021-04-04 09:10:05] (common) IGNORED (media) field header: Modified term: http://purl.org/dc/terms/modified
[WARN] [2021-04-04 09:10:05] (common) IGNORED (media) field header: Rating term: http://ns.adobe.com/xap/1.0/Rating
[WARN] [2021-04-04 09:10:05] (common) IGNORED (media) field header: Audience term: http://purl.org/dc/terms/audience
[WARN] [2021-04-04 09:10:05] (common) IGNORED (media) field header: Rights term: http://purl.org/dc/terms/rights
[WARN] [2021-04-04 09:10:05] (common) IGNORED (media) field header: Publisher term: http://purl.org/dc/terms/publisher
[WARN] [2021-04-04 09:10:05] (common) IGNORED (media) field header: Contributor term: http://purl.org/dc/terms/contributor
[WARN] [2021-04-04 09:10:05] (common) IGNORED (media) field header: Creator term: http://purl.org/dc/terms/creator
[WARN] [2021-04-04 09:10:05] (common) IGNORED (media) field header: GenericLocation term: http://purl.org/dc/terms/spatial
[WARN] [2021-04-04 09:10:05] (common) IGNORED (media) field header: Latitude term: http://www.w3.org/2003/01/geo/wgs84_pos#lat
[WARN] [2021-04-04 09:10:05] (common) IGNORED (media) field header: Longitude term: http://www.w3.org/2003/01/geo/wgs84_pos#long
[WARN] [2021-04-04 09:10:05] (common) IGNORED (media) field header: Altitude term: http://www.w3.org/2003/01/geo/wgs84_pos#alt
[WARN] [2021-04-04 09:10:05] (common) IGNORED (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2021-04-04 09:10:05] (common) IGNORED (refs) field header: PublicationType term: http://eol.org/schema/reference/publicationType
[WARN] [2021-04-04 09:10:05] (common) IGNORED (refs) field header: PageStart term: http://purl.org/ontology/bibo/pageStart
[WARN] [2021-04-04 09:10:05] (common) IGNORED (refs) field header: PageEnd term: http://purl.org/ontology/bibo/pageEnd
[WARN] [2021-04-04 09:10:05] (common) IGNORED (refs) field header: Language term: http://purl.org/dc/terms/language
[WARN] [2021-04-04 09:10:05] (common) IGNORED (occurrences) field header: Event ID term: http://rs.tdwg.org/dwc/terms/eventID
[WARN] [2021-04-04 09:10:05] (common) IGNORED (occurrences) field header: Institution Code term: http://rs.tdwg.org/dwc/terms/institutionCode
[WARN] [2021-04-04 09:10:05] (common) IGNORED (occurrences) field header: Collection Code term: http://rs.tdwg.org/dwc/terms/collectionCode
[WARN] [2021-04-04 09:10:05] (common) IGNORED (occurrences) field header: Catalog Number term: http://rs.tdwg.org/dwc/terms/catalogNumber
[WARN] [2021-04-04 09:10:05] SKIPPING http://rs.tdwg.org/dwc/terms/Event config (events.txt)...
[STOP] [2021-04-04 09:10:05] Parse meta.xml file and create formats with fields
[STOP] [2021-04-04 09:10:05] Creating resource from OpenData
[START] [2021-04-04 09:10:06] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-04 09:10:06] create_harvest_instance
[INFO] [2021-04-04 09:10:06] Created harvest instance #3624
[STOP] [2021-04-04 09:10:06] create_harvest_instance
[START] [2021-04-04 09:10:06] fetch_files
[STOP] [2021-04-04 09:10:06] fetch_files
[START] [2021-04-04 09:10:06] validate_each_file
[INFO] [2021-04-04 09:10:06] Looping over 8 formats...
[INFO] [2021-04-04 09:10:06] ...agents (/app/public/data/fcsfi/agents.txt)
[INFO] [2021-04-04 09:10:06] Valid: /app/public/converted_csv/fcsfi_agents_3624.csv (0 lines)
[INFO] [2021-04-04 09:10:06] ...refs (/app/public/data/fcsfi/references.txt)
[INFO] [2021-04-04 09:10:06] Valid: /app/public/converted_csv/fcsfi_refs_3624.csv (0 lines)
[INFO] [2021-04-04 09:10:06] ...nodes (/app/public/data/fcsfi/taxa.txt)
[INFO] [2021-04-04 09:10:07] Valid: /app/public/converted_csv/fcsfi_nodes_3624.csv (182 lines)
[INFO] [2021-04-04 09:10:07] ...media (/app/public/data/fcsfi/media.txt)
[INFO] [2021-04-04 09:10:07] Valid: /app/public/converted_csv/fcsfi_media_3624.csv (0 lines)
[INFO] [2021-04-04 09:10:07] ...vernaculars (/app/public/data/fcsfi/common names.txt)
[INFO] [2021-04-04 09:10:07] Valid: /app/public/converted_csv/fcsfi_vernaculars_3624.csv (0 lines)
[INFO] [2021-04-04 09:10:07] ...occurrences (/app/public/data/fcsfi/occurrences.txt)
[INFO] [2021-04-04 09:10:07] Valid: /app/public/converted_csv/fcsfi_occurrences_3624.csv (1165 lines)
[INFO] [2021-04-04 09:10:07] ...assocs (/app/public/data/fcsfi/associations.txt)
[INFO] [2021-04-04 09:10:07] Valid: /app/public/converted_csv/fcsfi_assocs_3624.csv (0 lines)
[INFO] [2021-04-04 09:10:07] ...measurements (/app/public/data/fcsfi/measurement_or_fact.txt)
[INFO] [2021-04-04 09:10:07] Valid: /app/public/converted_csv/fcsfi_measurements_3624.csv (1165 lines)
[STOP] [2021-04-04 09:10:07] validate_each_file
[START] [2021-04-04 09:10:07] convert_to_csv
[INFO] [2021-04-04 09:10:07] Looping over 8 formats...
[INFO] [2021-04-04 09:10:07] ...agents (/app/public/data/fcsfi/agents.txt)
[CMD] [2021-04-04 09:10:07] /usr/bin/sort /app/public/converted_csv/fcsfi_agents_3624.csv > /app/public/converted_csv/fcsfi_agents_3624.csv_sorted
[INFO] [2021-04-04 09:10:08] Converted: /app/public/converted_csv/fcsfi_agents_3624.csv (0 lines)
[INFO] [2021-04-04 09:10:08] ...refs (/app/public/data/fcsfi/references.txt)
[CMD] [2021-04-04 09:10:08] /usr/bin/sort /app/public/converted_csv/fcsfi_refs_3624.csv > /app/public/converted_csv/fcsfi_refs_3624.csv_sorted
[INFO] [2021-04-04 09:10:09] Converted: /app/public/converted_csv/fcsfi_refs_3624.csv (0 lines)
[INFO] [2021-04-04 09:10:09] ...nodes (/app/public/data/fcsfi/taxa.txt)
[CMD] [2021-04-04 09:10:09] /usr/bin/sort /app/public/converted_csv/fcsfi_nodes_3624.csv > /app/public/converted_csv/fcsfi_nodes_3624.csv_sorted
[INFO] [2021-04-04 09:10:10] Converted: /app/public/converted_csv/fcsfi_nodes_3624.csv (182 lines)
[INFO] [2021-04-04 09:10:10] ...media (/app/public/data/fcsfi/media.txt)
[CMD] [2021-04-04 09:10:10] /usr/bin/sort /app/public/converted_csv/fcsfi_media_3624.csv > /app/public/converted_csv/fcsfi_media_3624.csv_sorted
[INFO] [2021-04-04 09:10:11] Converted: /app/public/converted_csv/fcsfi_media_3624.csv (0 lines)
[INFO] [2021-04-04 09:10:11] ...vernaculars (/app/public/data/fcsfi/common names.txt)
[CMD] [2021-04-04 09:10:11] /usr/bin/sort /app/public/converted_csv/fcsfi_vernaculars_3624.csv > /app/public/converted_csv/fcsfi_vernaculars_3624.csv_sorted
[INFO] [2021-04-04 09:10:12] Converted: /app/public/converted_csv/fcsfi_vernaculars_3624.csv (0 lines)
[INFO] [2021-04-04 09:10:12] ...occurrences (/app/public/data/fcsfi/occurrences.txt)
[CMD] [2021-04-04 09:10:12] /usr/bin/sort /app/public/converted_csv/fcsfi_occurrences_3624.csv > /app/public/converted_csv/fcsfi_occurrences_3624.csv_sorted
[INFO] [2021-04-04 09:10:13] Converted: /app/public/converted_csv/fcsfi_occurrences_3624.csv (1165 lines)
[INFO] [2021-04-04 09:10:13] ...assocs (/app/public/data/fcsfi/associations.txt)
[CMD] [2021-04-04 09:10:13] /usr/bin/sort /app/public/converted_csv/fcsfi_assocs_3624.csv > /app/public/converted_csv/fcsfi_assocs_3624.csv_sorted
[INFO] [2021-04-04 09:10:14] Converted: /app/public/converted_csv/fcsfi_assocs_3624.csv (0 lines)
[INFO] [2021-04-04 09:10:14] ...measurements (/app/public/data/fcsfi/measurement_or_fact.txt)
[CMD] [2021-04-04 09:10:14] /usr/bin/sort /app/public/converted_csv/fcsfi_measurements_3624.csv > /app/public/converted_csv/fcsfi_measurements_3624.csv_sorted
[INFO] [2021-04-04 09:10:15] Converted: /app/public/converted_csv/fcsfi_measurements_3624.csv (1165 lines)
[STOP] [2021-04-04 09:10:15] convert_to_csv
[START] [2021-04-04 09:10:15] calculate_delta
[INFO] [2021-04-04 09:10:15] Looping over 8 formats...
[INFO] [2021-04-04 09:10:15] ...agents (/app/public/data/fcsfi/agents.txt)
[CMD] [2021-04-04 09:10:15] echo "0a" > /app/public/diff/fcsfi_agents_3624.diff
[CMD] [2021-04-04 09:10:16] tail -n +1 /app/public/converted_csv/fcsfi_agents_3624.csv >> /app/public/diff/fcsfi_agents_3624.diff
[CMD] [2021-04-04 09:10:17] echo "." >> /app/public/diff/fcsfi_agents_3624.diff
[INFO] [2021-04-04 09:10:18] Created diff: /app/public/diff/fcsfi_agents_3624.diff (2 lines)
[INFO] [2021-04-04 09:10:18] ...refs (/app/public/data/fcsfi/references.txt)
[CMD] [2021-04-04 09:10:18] echo "0a" > /app/public/diff/fcsfi_refs_3624.diff
[CMD] [2021-04-04 09:10:19] tail -n +1 /app/public/converted_csv/fcsfi_refs_3624.csv >> /app/public/diff/fcsfi_refs_3624.diff
[CMD] [2021-04-04 09:10:20] echo "." >> /app/public/diff/fcsfi_refs_3624.diff
[INFO] [2021-04-04 09:10:21] Created diff: /app/public/diff/fcsfi_refs_3624.diff (2 lines)
[INFO] [2021-04-04 09:10:21] ...nodes (/app/public/data/fcsfi/taxa.txt)
[CMD] [2021-04-04 09:10:21] echo "0a" > /app/public/diff/fcsfi_nodes_3624.diff
[CMD] [2021-04-04 09:10:22] tail -n +1 /app/public/converted_csv/fcsfi_nodes_3624.csv >> /app/public/diff/fcsfi_nodes_3624.diff
[CMD] [2021-04-04 09:10:23] echo "." >> /app/public/diff/fcsfi_nodes_3624.diff
[INFO] [2021-04-04 09:10:24] Created diff: /app/public/diff/fcsfi_nodes_3624.diff (184 lines)
[INFO] [2021-04-04 09:10:24] ...media (/app/public/data/fcsfi/media.txt)
[CMD] [2021-04-04 09:10:24] echo "0a" > /app/public/diff/fcsfi_media_3624.diff
[CMD] [2021-04-04 09:10:25] tail -n +1 /app/public/converted_csv/fcsfi_media_3624.csv >> /app/public/diff/fcsfi_media_3624.diff
[CMD] [2021-04-04 09:10:26] echo "." >> /app/public/diff/fcsfi_media_3624.diff
[INFO] [2021-04-04 09:10:27] Created diff: /app/public/diff/fcsfi_media_3624.diff (2 lines)
[INFO] [2021-04-04 09:10:27] ...vernaculars (/app/public/data/fcsfi/common names.txt)
[CMD] [2021-04-04 09:10:27] echo "0a" > /app/public/diff/fcsfi_vernaculars_3624.diff
[CMD] [2021-04-04 09:10:28] tail -n +1 /app/public/converted_csv/fcsfi_vernaculars_3624.csv >> /app/public/diff/fcsfi_vernaculars_3624.diff
[CMD] [2021-04-04 09:10:29] echo "." >> /app/public/diff/fcsfi_vernaculars_3624.diff
[INFO] [2021-04-04 09:10:30] Created diff: /app/public/diff/fcsfi_vernaculars_3624.diff (2 lines)
[INFO] [2021-04-04 09:10:30] ...occurrences (/app/public/data/fcsfi/occurrences.txt)
[CMD] [2021-04-04 09:10:30] echo "0a" > /app/public/diff/fcsfi_occurrences_3624.diff
[CMD] [2021-04-04 09:10:31] tail -n +1 /app/public/converted_csv/fcsfi_occurrences_3624.csv >> /app/public/diff/fcsfi_occurrences_3624.diff
[CMD] [2021-04-04 09:10:32] echo "." >> /app/public/diff/fcsfi_occurrences_3624.diff
[INFO] [2021-04-04 09:10:33] Created diff: /app/public/diff/fcsfi_occurrences_3624.diff (1167 lines)
[INFO] [2021-04-04 09:10:33] ...assocs (/app/public/data/fcsfi/associations.txt)
[CMD] [2021-04-04 09:10:33] echo "0a" > /app/public/diff/fcsfi_assocs_3624.diff
[CMD] [2021-04-04 09:10:34] tail -n +1 /app/public/converted_csv/fcsfi_assocs_3624.csv >> /app/public/diff/fcsfi_assocs_3624.diff
[CMD] [2021-04-04 09:10:35] echo "." >> /app/public/diff/fcsfi_assocs_3624.diff
[INFO] [2021-04-04 09:10:36] Created diff: /app/public/diff/fcsfi_assocs_3624.diff (2 lines)
[INFO] [2021-04-04 09:10:36] ...measurements (/app/public/data/fcsfi/measurement_or_fact.txt)
[CMD] [2021-04-04 09:10:36] echo "0a" > /app/public/diff/fcsfi_measurements_3624.diff
[CMD] [2021-04-04 09:10:37] tail -n +1 /app/public/converted_csv/fcsfi_measurements_3624.csv >> /app/public/diff/fcsfi_measurements_3624.diff
[CMD] [2021-04-04 09:10:38] echo "." >> /app/public/diff/fcsfi_measurements_3624.diff
[INFO] [2021-04-04 09:10:39] Created diff: /app/public/diff/fcsfi_measurements_3624.diff (1167 lines)
[STOP] [2021-04-04 09:10:39] calculate_delta
[START] [2021-04-04 09:10:39] parse_diff_and_store
[INFO] [2021-04-04 09:10:39] Handling diff: /app/public/diff/fcsfi_agents_3624.diff (2 lines)
[INFO] [2021-04-04 09:10:40] Loading agents diff file into memory (2 /app/public/diff/fcsfi_agents_3624.diff lines)...
[INFO] [2021-04-04 09:10:41] Handling diff: /app/public/diff/fcsfi_refs_3624.diff (2 lines)
[INFO] [2021-04-04 09:10:42] Loading refs diff file into memory (2 /app/public/diff/fcsfi_refs_3624.diff lines)...
[INFO] [2021-04-04 09:10:43] Handling diff: /app/public/diff/fcsfi_nodes_3624.diff (184 lines)
[INFO] [2021-04-04 09:10:44] Loading nodes diff file into memory (184 /app/public/diff/fcsfi_nodes_3624.diff lines)...
[WARN] [2021-04-04 09:10:45] Filtered Scientific Name `Anemone narcissiflora var. monantha` to `Anemone narcissiflora var. monantha`
[INFO] [2021-04-04 09:10:45] Handling diff: /app/public/diff/fcsfi_media_3624.diff (2 lines)
[INFO] [2021-04-04 09:10:46] Loading media diff file into memory (2 /app/public/diff/fcsfi_media_3624.diff lines)...
[INFO] [2021-04-04 09:10:47] Handling diff: /app/public/diff/fcsfi_vernaculars_3624.diff (2 lines)
[INFO] [2021-04-04 09:10:48] Loading vernaculars diff file into memory (2 /app/public/diff/fcsfi_vernaculars_3624.diff lines)...
[INFO] [2021-04-04 09:10:49] Handling diff: /app/public/diff/fcsfi_occurrences_3624.diff (1167 lines)
[INFO] [2021-04-04 09:10:50] Loading occurrences diff file into memory (1167 /app/public/diff/fcsfi_occurrences_3624.diff lines)...
[INFO] [2021-04-04 09:10:51] Handling diff: /app/public/diff/fcsfi_assocs_3624.diff (2 lines)
[INFO] [2021-04-04 09:10:52] Loading assocs diff file into memory (2 /app/public/diff/fcsfi_assocs_3624.diff lines)...
[INFO] [2021-04-04 09:10:53] Handling diff: /app/public/diff/fcsfi_measurements_3624.diff (1167 lines)
[INFO] [2021-04-04 09:10:54] Loading measurements diff file into memory (1167 /app/public/diff/fcsfi_measurements_3624.diff lines)...
[INFO] [2021-04-04 09:10:56] Storing 208 ScientificNames
[INFO] [2021-04-04 09:10:56] Processing group of 208 in 1 groups of 1000
[INFO] [2021-04-04 09:10:56] Average Time: 0.09
[INFO] [2021-04-04 09:10:56] Total Time: 1s
[INFO] [2021-04-04 09:10:56] Storing 208 Nodes
[INFO] [2021-04-04 09:10:56] Processing group of 208 in 1 groups of 1000
[INFO] [2021-04-04 09:10:56] Average Time: 0.06
[INFO] [2021-04-04 09:10:56] Total Time: 1s
[INFO] [2021-04-04 09:10:56] Storing 1165 Occurrences
[INFO] [2021-04-04 09:10:56] Processing group of 1165 in 2 groups of 1000
[INFO] [2021-04-04 09:10:56] Average Time: 0.06
[INFO] [2021-04-04 09:10:56] Total Time: 1s
[INFO] [2021-04-04 09:10:56] Storing 1165 Traits
[INFO] [2021-04-04 09:10:56] Processing group of 1165 in 2 groups of 1000
[INFO] [2021-04-04 09:10:57] Average Time: 0.15
[INFO] [2021-04-04 09:10:57] Total Time: 1s
[INFO] [2021-04-04 09:10:57] Storing 3495 MetaTraits
[INFO] [2021-04-04 09:10:57] Processing group of 3495 in 4 groups of 1000
[INFO] [2021-04-04 09:10:57] Average Time: 0.098
[INFO] [2021-04-04 09:10:57] Total Time: 1s
[STOP] [2021-04-04 09:10:57] parse_diff_and_store
[START] [2021-04-04 09:10:57] resolve_keys
[INFO] [2021-04-04 09:11:04] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-04 09:11:04] traits to occurrences...
[INFO] [2021-04-04 09:11:04] traits to nodes (through occurrences)...
[INFO] [2021-04-04 09:11:04] Traits to sex term...
[INFO] [2021-04-04 09:11:04] Traits to lifestage term...
[INFO] [2021-04-04 09:11:04] MetaTraits to traits...
[INFO] [2021-04-04 09:11:04] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-04 09:11:04] Assocs to occurrences...
[INFO] [2021-04-04 09:11:04] Assocs to nodes...
[INFO] [2021-04-04 09:11:04] Assoc to sex term...
[INFO] [2021-04-04 09:11:04] Assoc to lifestage term...
[INFO] [2021-04-04 09:11:04] MetaAssoc to assocs...
[STOP] [2021-04-04 09:11:04] resolve_keys
[START] [2021-04-04 09:11:04] hold_for_later_1
[STOP] [2021-04-04 09:11:04] hold_for_later_1
[START] [2021-04-04 09:11:04] hold_for_later_2
[STOP] [2021-04-04 09:11:04] hold_for_later_2
[START] [2021-04-04 09:11:04] resolve_missing_parents
[STOP] [2021-04-04 09:11:04] resolve_missing_parents
[START] [2021-04-04 09:11:04] rebuild_nodes
[START] [2021-04-04 09:11:04] Flattener#flatten
[START] [2021-04-04 09:11:04] Flattener#study_resource
[START] [2021-04-04 09:11:04] Flattener#build_ancestry
[STOP] [2021-04-04 09:11:04] Flattener#build_ancestry
[INFO] [2021-04-04 09:11:04] 208 ancestry keys
[START] [2021-04-04 09:11:04] build_node_ancestors
[INFO] [2021-04-04 09:11:04] old ancestors deleted.
[STOP] [2021-04-04 09:11:04] build_node_ancestors
[START] [2021-04-04 09:11:04] Flattener#propagate_ancestor_ids
[STOP] [2021-04-04 09:11:04] Flattener#propagate_ancestor_ids
[STOP] [2021-04-04 09:11:04] Flattener#flatten
[STOP] [2021-04-04 09:11:04] rebuild_nodes
[START] [2021-04-04 09:11:04] resolve_missing_media_owners
[STOP] [2021-04-04 09:11:04] resolve_missing_media_owners
[START] [2021-04-04 09:11:04] sanitize_media_verbatims
[STOP] [2021-04-04 09:11:04] sanitize_media_verbatims
[START] [2021-04-04 09:11:04] queue_downloads
[STOP] [2021-04-04 09:11:04] queue_downloads
[START] [2021-04-04 09:11:04] parse_names
[WARN] [2021-04-04 09:11:04] I see 208 names which still need to be parsed.
[STOP] [2021-04-04 09:11:06] parse_names
[START] [2021-04-04 09:11:06] denormalize_canonical_names_to_nodes
[STOP] [2021-04-04 09:11:06] denormalize_canonical_names_to_nodes
[START] [2021-04-04 09:11:06] match_nodes
[START] [2021-04-04 09:11:06] map_all_nodes_to_pages
[STOP] [2021-04-04 09:11:08] map_all_nodes_to_pages
[INFO] [2021-04-04 09:11:08] Unmatched nodes (6 of 208): plantae (#91654941); Symphyotrichum laeve laeve (#91655130); Anemone drummondii lithophila (#91654950); Nemophila pulchella pulchella (#91655087); Pholistoma auritum auritum (#91655104); Asteracea (#91655035)
[START] [2021-04-04 09:11:08] update_nodes
[STOP] [2021-04-04 09:11:08] update_nodes
[STOP] [2021-04-04 09:11:08] match_nodes
[START] [2021-04-04 09:11:08] reindex_search
[STOP] [2021-04-04 09:11:08] reindex_search
[START] [2021-04-04 09:11:08] normalize_units
[STOP] [2021-04-04 09:11:08] normalize_units
[START] [2021-04-04 09:11:08] calculate_statistics
[STOP] [2021-04-04 09:11:08] calculate_statistics
[START] [2021-04-04 09:11:08] complete_harvest_instance
[START] [2021-04-04 09:11:08] overall_tsv_creation
[INFO] [2021-04-04 09:11:08] Processing group of 208 in 1 batches of 10000
[INFO] [2021-04-04 09:11:47] 1165 Traits (unfiltered)...
[INFO] [2021-04-04 09:12:26] 1165 Traits (filtered)...
[INFO] [2021-04-04 09:12:26] 0 Associations (filtered)...
[INFO] [2021-04-04 09:12:26] 0 metadata added.
[INFO] [2021-04-04 09:12:26] 0 metadata added.
[INFO] [2021-04-04 09:12:53] Average Time: 79.12
[INFO] [2021-04-04 09:12:53] Total Time: 1m45s
[STOP] [2021-04-04 09:12:53] overall_tsv_creation
[INFO] [2021-04-04 09:12:53] Done. Check your files:
[INFO] [2021-04-04 09:12:54] (208 lines) /app/public/data/fcsfi/publish_nodes.tsv
[INFO] [2021-04-04 09:12:55] (389 lines) /app/public/data/fcsfi/publish_node_ancestors.tsv
[INFO] [2021-04-04 09:12:56] (208 lines) /app/public/data/fcsfi/publish_scientific_names.tsv
[INFO] [2021-04-04 09:12:57] (1166 lines) /app/public/data/fcsfi/publish_traits.tsv
[INFO] [2021-04-04 09:12:58] (1 lines) /app/public/data/fcsfi/publish_metadata.tsv
[STOP] [2021-04-04 09:12:58] complete_harvest_instance
[START] [2021-04-04 09:12:58] completed
[STOP] [2021-04-04 09:12:58] completed
[STOP] [2021-04-04 09:12:58] logged process, took 172.47
Latest Process