Harvest for
MADtraits
Created
06 May 05:14
Stage:
resolve_keys
Fetched:
06 May 05:14
Validated:
06 May 05:14
Deltas Created
06 May 05:15
New Models Stored:
06 May 05:19
Failed:
06 May 05:20
Completed:
06 May 05:20
Time to Harvest:
less than a minute
Harvesting Log
(197 lines)
[INFO] [2025-05-06 05:14:34] Created harvest instance #4571
[STOP] [2025-05-06 05:14:34] create_harvest_instance
[START] [2025-05-06 05:14:34] fetch_files
[STOP] [2025-05-06 05:14:34] fetch_files
[START] [2025-05-06 05:14:34] validate_each_file
[INFO] [2025-05-06 05:14:34] Created new folder: /app/public/converted_csv
[INFO] [2025-05-06 05:14:34] Looping over 4 formats...
[INFO] [2025-05-06 05:14:34] ...refs (/app/public/data/natdb_dwca_dw_na/reference.tab)
[INFO] [2025-05-06 05:14:34] Valid: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_refs_31238.csv (11 lines)
[INFO] [2025-05-06 05:14:34] ...nodes (/app/public/data/natdb_dwca_dw_na/taxon.tab)
[INFO] [2025-05-06 05:14:34] Valid: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_nodes_31237.csv (2778 lines)
[INFO] [2025-05-06 05:14:34] ...occurrences (/app/public/data/natdb_dwca_dw_na/occurrence.tab)
[INFO] [2025-05-06 05:14:37] Valid: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_occurrences_31239.csv (96894 lines)
[INFO] [2025-05-06 05:14:37] ...measurements (/app/public/data/natdb_dwca_dw_na/measurement_or_fact_specific.tab)
[INFO] [2025-05-06 05:14:52] Valid: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_measurements_31240.csv (211530 lines)
[STOP] [2025-05-06 05:14:52] validate_each_file
[START] [2025-05-06 05:14:52] convert_to_csv
[INFO] [2025-05-06 05:14:52] Looping over 4 formats...
[INFO] [2025-05-06 05:14:52] ...refs (/app/public/data/natdb_dwca_dw_na/reference.tab)
[CMD] [2025-05-06 05:14:52] /usr/bin/sort /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_refs_31238.csv > /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_refs_31238.csv_sorted
[INFO] [2025-05-06 05:14:52] Converted: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_refs_31238.csv (11 lines)
[INFO] [2025-05-06 05:14:52] ...nodes (/app/public/data/natdb_dwca_dw_na/taxon.tab)
[CMD] [2025-05-06 05:14:52] /usr/bin/sort /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_nodes_31237.csv > /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_nodes_31237.csv_sorted
[INFO] [2025-05-06 05:14:52] Converted: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_nodes_31237.csv (2778 lines)
[INFO] [2025-05-06 05:14:52] ...occurrences (/app/public/data/natdb_dwca_dw_na/occurrence.tab)
[CMD] [2025-05-06 05:14:52] /usr/bin/sort /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_occurrences_31239.csv > /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_occurrences_31239.csv_sorted
[INFO] [2025-05-06 05:14:52] Converted: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_occurrences_31239.csv (96894 lines)
[INFO] [2025-05-06 05:14:52] ...measurements (/app/public/data/natdb_dwca_dw_na/measurement_or_fact_specific.tab)
[CMD] [2025-05-06 05:14:52] /usr/bin/sort /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_measurements_31240.csv > /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_measurements_31240.csv_sorted
[INFO] [2025-05-06 05:14:56] Converted: /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_measurements_31240.csv (211530 lines)
[STOP] [2025-05-06 05:14:56] convert_to_csv
[START] [2025-05-06 05:14:56] calculate_delta
[INFO] [2025-05-06 05:14:56] Created diff dir: /app/public/diff
[INFO] [2025-05-06 05:14:56] Looping over 4 formats...
[INFO] [2025-05-06 05:14:56] ...refs (/app/public/data/natdb_dwca_dw_na/reference.tab)
[CMD] [2025-05-06 05:14:56] echo "0a" > /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_refs_31238.diff
[CMD] [2025-05-06 05:15:02] tail -n +1 /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_refs_31238.csv >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_refs_31238.diff
[CMD] [2025-05-06 05:15:02] echo "." >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_refs_31238.diff
[INFO] [2025-05-06 05:15:02] Created diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_refs_31238.diff (13 lines)
[INFO] [2025-05-06 05:15:02] ...nodes (/app/public/data/natdb_dwca_dw_na/taxon.tab)
[CMD] [2025-05-06 05:15:02] echo "0a" > /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_nodes_31237.diff
[CMD] [2025-05-06 05:15:02] tail -n +1 /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_nodes_31237.csv >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_nodes_31237.diff
[CMD] [2025-05-06 05:15:02] echo "." >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_nodes_31237.diff
[INFO] [2025-05-06 05:15:03] Created diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_nodes_31237.diff (2780 lines)
[INFO] [2025-05-06 05:15:03] ...occurrences (/app/public/data/natdb_dwca_dw_na/occurrence.tab)
[CMD] [2025-05-06 05:15:03] echo "0a" > /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_occurrences_31239.diff
[CMD] [2025-05-06 05:15:03] tail -n +1 /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_occurrences_31239.csv >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_occurrences_31239.diff
[CMD] [2025-05-06 05:15:03] echo "." >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_occurrences_31239.diff
[INFO] [2025-05-06 05:15:03] Created diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_occurrences_31239.diff (96896 lines)
[INFO] [2025-05-06 05:15:03] ...measurements (/app/public/data/natdb_dwca_dw_na/measurement_or_fact_specific.tab)
[CMD] [2025-05-06 05:15:03] echo "0a" > /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_measurements_31240.diff
[CMD] [2025-05-06 05:15:03] tail -n +1 /app/public/data/natdb_dwca_dw_na/converted_csv/natdb_dwca_dw_na_measurements_31240.csv >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_measurements_31240.diff
[CMD] [2025-05-06 05:15:04] echo "." >> /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_measurements_31240.diff
[INFO] [2025-05-06 05:15:04] Created diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_measurements_31240.diff (211532 lines)
[STOP] [2025-05-06 05:15:04] calculate_delta
[START] [2025-05-06 05:15:04] parse_diff_and_store
[INFO] [2025-05-06 05:15:04] Handling diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_refs_31238.diff (13 lines)
[INFO] [2025-05-06 05:15:04] Loading refs diff file into memory (13 lines)...
[INFO] [2025-05-06 05:15:05] Storing 11 References (11/11/13)
[INFO] [2025-05-06 05:15:05] Handling diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_nodes_31237.diff (2780 lines)
[INFO] [2025-05-06 05:15:05] Loading nodes diff file into memory (2780 lines)...
[WARN] [2025-05-06 05:15:05] Filtered Scientific Name `Entelurus aequoreus` to `Entelurus aequoreus`
[WARN] [2025-05-06 05:15:05] Filtered Scientific Name `Mesocestoides corti/variabilis/vogae` to `Mesocestoides cortivariabilisvogae`
[WARN] [2025-05-06 05:15:05] Filtered Scientific Name `Mesocestoides lineatus/litteratus` to `Mesocestoides lineatuslitteratus`
[WARN] [2025-05-06 05:15:05] Filtered Scientific Name `Parkesia motacilla` to `Parkesia motacilla`
[WARN] [2025-05-06 05:15:05] Filtered Scientific Name `Protostrongylus pulmonalis/kamenskyi` to `Protostrongylus pulmonaliskamenskyi`
[WARN] [2025-05-06 05:15:05] Filtered Scientific Name `Protostrongylus stilesi/rushi` to `Protostrongylus stilesirushi`
[WARN] [2025-05-06 05:15:05] Filtered Scientific Name `Taenia ovis/krabbei` to `Taenia oviskrabbei`
[INFO] [2025-05-06 05:15:05] Storing 2814 ScientificNames (5628/2778/2780)
[INFO] [2025-05-06 05:15:08] Storing 2814 Nodes (5628/2778/2780)
[INFO] [2025-05-06 05:15:11] Handling diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_occurrences_31239.diff (96896 lines)
[INFO] [2025-05-06 05:15:12] Loading occurrences diff file into memory (96896 lines)...
[INFO] [2025-05-06 05:15:15] Storing 9999 Occurrences (10120/10000/96896)
[INFO] [2025-05-06 05:15:19] Storing 121 OccurrenceMetadata (10120/10000/96896)
[WARN] [2025-05-06 05:15:20] SKIPPED 125 Occurrence metadata (20245/20000/96896) with resource_pks already be in the database!
[INFO] [2025-05-06 05:15:20] Storing 10000 Occurrences (20245/20000/96896)
[INFO] [2025-05-06 05:15:24] Storing 0 OccurrenceMetadata (20245/20000/96896)
[WARN] [2025-05-06 05:15:24] No models to import, skipping!
[WARN] [2025-05-06 05:15:25] SKIPPED 116 Occurrence metadata (30361/30000/96896) with resource_pks already be in the database!
[INFO] [2025-05-06 05:15:25] Storing 10000 Occurrences (30361/30000/96896)
[INFO] [2025-05-06 05:15:29] Storing 0 OccurrenceMetadata (30361/30000/96896)
[WARN] [2025-05-06 05:15:29] No models to import, skipping!
[WARN] [2025-05-06 05:15:31] SKIPPED 126 Occurrence metadata (40487/40000/96896) with resource_pks already be in the database!
[INFO] [2025-05-06 05:15:31] Storing 10000 Occurrences (40487/40000/96896)
[INFO] [2025-05-06 05:15:35] Storing 0 OccurrenceMetadata (40487/40000/96896)
[WARN] [2025-05-06 05:15:35] No models to import, skipping!
[WARN] [2025-05-06 05:15:36] SKIPPED 105 Occurrence metadata (50592/50000/96896) with resource_pks already be in the database!
[INFO] [2025-05-06 05:15:36] Storing 10000 Occurrences (50592/50000/96896)
[INFO] [2025-05-06 05:15:42] Storing 0 OccurrenceMetadata (50592/50000/96896)
[WARN] [2025-05-06 05:15:42] No models to import, skipping!
[WARN] [2025-05-06 05:15:43] SKIPPED 141 Occurrence metadata (60733/60000/96896) with resource_pks already be in the database!
[INFO] [2025-05-06 05:15:43] Storing 10000 Occurrences (60733/60000/96896)
[INFO] [2025-05-06 05:15:47] Storing 0 OccurrenceMetadata (60733/60000/96896)
[WARN] [2025-05-06 05:15:47] No models to import, skipping!
[WARN] [2025-05-06 05:15:48] SKIPPED 117 Occurrence metadata (70850/70000/96896) with resource_pks already be in the database!
[INFO] [2025-05-06 05:15:48] Storing 10000 Occurrences (70850/70000/96896)
[INFO] [2025-05-06 05:15:52] Storing 0 OccurrenceMetadata (70850/70000/96896)
[WARN] [2025-05-06 05:15:52] No models to import, skipping!
[WARN] [2025-05-06 05:15:54] SKIPPED 117 Occurrence metadata (80967/80000/96896) with resource_pks already be in the database!
[INFO] [2025-05-06 05:15:54] Storing 10000 Occurrences (80967/80000/96896)
[INFO] [2025-05-06 05:15:57] Storing 0 OccurrenceMetadata (80967/80000/96896)
[WARN] [2025-05-06 05:15:57] No models to import, skipping!
[WARN] [2025-05-06 05:15:59] SKIPPED 103 Occurrence metadata (91070/90000/96896) with resource_pks already be in the database!
[INFO] [2025-05-06 05:15:59] Storing 10000 Occurrences (91070/90000/96896)
[INFO] [2025-05-06 05:16:03] Storing 0 OccurrenceMetadata (91070/90000/96896)
[WARN] [2025-05-06 05:16:03] No models to import, skipping!
[WARN] [2025-05-06 05:16:04] SKIPPED 78 Occurrence metadata (98043/96894/96896) with resource_pks already be in the database!
[INFO] [2025-05-06 05:16:04] Storing 6895 Occurrences (98043/96894/96896)
[INFO] [2025-05-06 05:16:07] Storing 0 OccurrenceMetadata (98043/96894/96896)
[WARN] [2025-05-06 05:16:07] No models to import, skipping!
[INFO] [2025-05-06 05:16:07] Handling diff: /app/public/data/natdb_dwca_dw_na/diff/natdb_dwca_dw_na_measurements_31240.diff (211532 lines)
[INFO] [2025-05-06 05:16:07] Loading measurements diff file into memory (211532 lines)...
[INFO] [2025-05-06 05:16:11] Storing 9999 Traits (22188/10000/211532)
[INFO] [2025-05-06 05:16:17] Storing 7620 MetaTraits (22188/10000/211532)
[INFO] [2025-05-06 05:16:19] Storing 4569 TraitsReferences (22188/10000/211532)
[INFO] [2025-05-06 05:16:24] Storing 10000 Traits (44415/20000/211532)
[INFO] [2025-05-06 05:16:28] Storing 4580 TraitsReferences (44415/20000/211532)
[INFO] [2025-05-06 05:16:29] Storing 7647 MetaTraits (44415/20000/211532)
[INFO] [2025-05-06 05:16:34] Storing 4562 TraitsReferences (66628/30000/211532)
[INFO] [2025-05-06 05:16:34] Storing 10000 Traits (66628/30000/211532)
[INFO] [2025-05-06 05:16:40] Storing 7651 MetaTraits (66628/30000/211532)
[INFO] [2025-05-06 05:16:46] Storing 4553 TraitsReferences (88813/40000/211532)
[INFO] [2025-05-06 05:16:46] Storing 10000 Traits (88813/40000/211532)
[INFO] [2025-05-06 05:16:51] Storing 7632 MetaTraits (88813/40000/211532)
[INFO] [2025-05-06 05:16:56] Storing 10000 Traits (110950/50000/211532)
[INFO] [2025-05-06 05:17:00] Storing 4564 TraitsReferences (110950/50000/211532)
[INFO] [2025-05-06 05:17:00] Storing 7573 MetaTraits (110950/50000/211532)
[INFO] [2025-05-06 05:17:05] Storing 4604 TraitsReferences (133185/60000/211532)
[INFO] [2025-05-06 05:17:06] Storing 10000 Traits (133185/60000/211532)
[INFO] [2025-05-06 05:17:11] Storing 7631 MetaTraits (133185/60000/211532)
[INFO] [2025-05-06 05:17:16] Storing 4429 TraitsReferences (155127/70000/211532)
[INFO] [2025-05-06 05:17:16] Storing 10000 Traits (155127/70000/211532)
[INFO] [2025-05-06 05:17:21] Storing 7513 MetaTraits (155127/70000/211532)
[INFO] [2025-05-06 05:17:27] Storing 10000 Traits (177351/80000/211532)
[INFO] [2025-05-06 05:17:33] Storing 4599 TraitsReferences (177351/80000/211532)
[INFO] [2025-05-06 05:17:34] Storing 7625 MetaTraits (177351/80000/211532)
[INFO] [2025-05-06 05:17:40] Storing 10000 Traits (199513/90000/211532)
[INFO] [2025-05-06 05:17:49] Storing 7591 MetaTraits (199513/90000/211532)
[INFO] [2025-05-06 05:17:55] Storing 4571 TraitsReferences (199513/90000/211532)
[INFO] [2025-05-06 05:18:00] Storing 4545 TraitsReferences (221665/100000/211532)
[INFO] [2025-05-06 05:18:02] Storing 10000 Traits (221665/100000/211532)
[INFO] [2025-05-06 05:18:06] Storing 7607 MetaTraits (221665/100000/211532)
[INFO] [2025-05-06 05:18:12] Storing 10000 Traits (243863/110000/211532)
[INFO] [2025-05-06 05:18:15] Storing 7607 MetaTraits (243863/110000/211532)
[INFO] [2025-05-06 05:18:16] Storing 4591 TraitsReferences (243863/110000/211532)
[INFO] [2025-05-06 05:18:21] Storing 4604 TraitsReferences (266115/120000/211532)
[INFO] [2025-05-06 05:18:21] Storing 10000 Traits (266115/120000/211532)
[INFO] [2025-05-06 05:18:25] Storing 7648 MetaTraits (266115/120000/211532)
[INFO] [2025-05-06 05:18:31] Storing 10000 Traits (288425/130000/211532)
[INFO] [2025-05-06 05:18:34] Storing 7691 MetaTraits (288425/130000/211532)
[INFO] [2025-05-06 05:18:35] Storing 4619 TraitsReferences (288425/130000/211532)
[INFO] [2025-05-06 05:18:40] Storing 4565 TraitsReferences (310574/140000/211532)
[INFO] [2025-05-06 05:18:41] Storing 10000 Traits (310574/140000/211532)
[INFO] [2025-05-06 05:18:44] Storing 7584 MetaTraits (310574/140000/211532)
[INFO] [2025-05-06 05:18:50] Storing 4658 TraitsReferences (332941/150000/211532)
[INFO] [2025-05-06 05:18:50] Storing 10000 Traits (332941/150000/211532)
[INFO] [2025-05-06 05:18:54] Storing 7709 MetaTraits (332941/150000/211532)
[INFO] [2025-05-06 05:19:00] Storing 4532 TraitsReferences (355031/160000/211532)
[INFO] [2025-05-06 05:19:00] Storing 10000 Traits (355031/160000/211532)
[INFO] [2025-05-06 05:19:04] Storing 7558 MetaTraits (355031/160000/211532)
[INFO] [2025-05-06 05:19:09] Storing 4584 TraitsReferences (377273/170000/211532)
[INFO] [2025-05-06 05:19:10] Storing 10000 Traits (377273/170000/211532)
[INFO] [2025-05-06 05:19:14] Storing 7658 MetaTraits (377273/170000/211532)
[INFO] [2025-05-06 05:19:20] Storing 10000 Traits (399569/180000/211532)
[INFO] [2025-05-06 05:19:23] Storing 7626 MetaTraits (399569/180000/211532)
[INFO] [2025-05-06 05:19:24] Storing 4670 TraitsReferences (399569/180000/211532)
[INFO] [2025-05-06 05:19:30] Storing 10000 Traits (421705/190000/211532)
[INFO] [2025-05-06 05:19:33] Storing 7583 MetaTraits (421705/190000/211532)
[INFO] [2025-05-06 05:19:34] Storing 4553 TraitsReferences (421705/190000/211532)
[INFO] [2025-05-06 05:19:40] Storing 4621 TraitsReferences (443972/200000/211532)
[INFO] [2025-05-06 05:19:40] Storing 10000 Traits (443972/200000/211532)
[INFO] [2025-05-06 05:19:43] Storing 7646 MetaTraits (443972/200000/211532)
[INFO] [2025-05-06 05:19:49] Storing 10000 Traits (466218/210000/211532)
[INFO] [2025-05-06 05:19:53] Storing 7636 MetaTraits (466218/210000/211532)
[INFO] [2025-05-06 05:19:54] Storing 4610 TraitsReferences (466218/210000/211532)
[INFO] [2025-05-06 05:19:55] Storing 1531 Traits (469615/211530/211532)
[INFO] [2025-05-06 05:19:56] Storing 1155 MetaTraits (469615/211530/211532)
[INFO] [2025-05-06 05:19:56] Storing 711 TraitsReferences (469615/211530/211532)
[STOP] [2025-05-06 05:19:56] parse_diff_and_store
[START] [2025-05-06 05:19:56] resolve_keys
[STOP] [2025-05-06 05:20:02] resolve_keys
[ERR] [2025-05-06 05:20:02] ActiveRecord::StatementInvalid
[ERR] [2025-05-06 05:20:02] Mysql2::Error: The client was disconnected by the server because of inactivity. See wait_timeout and interactive_timeout for configuring this behavior.: SELECT MIN(`vernaculars`.`id`) FROM `vernaculars` WHERE (1=1) AND `vernaculars`.`harvest_id` = 4571
[ERR] [2025-05-06 05:20:02] ./config/initializers/core_extensions.rb:14:in `propagate_id'
[ERR] [2025-05-06 05:20:02] ../models/resource_harvester.rb:647:in `propagate_id'
[ERR] [2025-05-06 05:20:02] ../models/resource_harvester.rb:506:in `resolve_node_keys'
[ERR] [2025-05-06 05:20:02] ../models/resource_harvester.rb:482:in `resolve_keys'
[ERR] [2025-05-06 05:20:02] ../models/resource_harvester.rb:86:in `block (2 levels) in start'
[ERR] [2025-05-06 05:20:02] ../models/logged_process.rb:43:in `run_step'
[ERR] [2025-05-06 05:20:02] ../models/resource_harvester.rb:86:in `block in start'
[ERR] [2025-05-06 05:20:02] ../models/resource_harvester.rb:75:in `each_key'
[ERR] [2025-05-06 05:20:02] ../models/resource_harvester.rb:75:in `start'
[ERR] [2025-05-06 05:20:02] ../models/resource.rb:306:in `harvest'
[ERR] [2025-05-06 05:20:02] ../models/resource.rb:282:in `re_download_opendata_and_harvest'
[ERR] [2025-05-06 05:20:02] bin/rails:4:in `require'
[ERR] [2025-05-06 05:20:02] bin/rails:4:in `<main>'
[STOP] [2025-05-06 05:20:02] logged process, took 328.64
Latest Process