Harvest for Flora do Brasil Created 04 Apr 13:53

Stage: completed
Fetched: 04 Apr 13:53
Validated: 04 Apr 13:54
Deltas Created 04 Apr 13:54
Units Normalized: 04 Apr 14:33
Ancestry Built: 04 Apr 14:14
Nodes Matched: 04 Apr 14:32
Names Parsed: 04 Apr 14:16
New Models Stored: 04 Apr 14:11
Indexed: 04 Apr 14:33
Completed: 04 Apr 15:38
Time to Harvest: 2 minutes

Harvesting Log

(2028 lines) (showing only the last 1000 lines, see /app/public/data/brazilian_flora_/process.log for the full file)
[STOP] [2021-01-27 16:59:52] map_all_nodes_to_pages
[INFO] [2021-01-27 16:59:52] 7946 Unmatched nodes (of 70159)! That's too many to output. Full list in /app/public/data/brazilian_flora_/unmatched_nodes.txt ; First 10: Mariscus (#87487320); Rhynchospora ciliata (#87539287); Eleocharis radicans (#87492267); Scleria setuloso-ciliata (#87515209); Dichromena (#87506683); Tetraria (#87506920); Websteria (#87515212); Androtrichum (#87526333); Egleria (#87526420); Adupla (#87531295)
[START] [2021-01-27 16:59:52] update_nodes
[STOP] [2021-01-27 17:00:25] update_nodes
[STOP] [2021-01-27 17:00:25] match_nodes
[START] [2021-01-27 17:00:25] reindex_search
[STOP] [2021-01-27 17:02:22] reindex_search
[START] [2021-01-27 17:02:22] normalize_units
[STOP] [2021-01-27 17:02:54] normalize_units
[START] [2021-01-27 17:02:54] calculate_statistics
[STOP] [2021-01-27 17:02:55] calculate_statistics
[START] [2021-01-27 17:02:55] complete_harvest_instance
[START] [2021-01-27 17:02:55] overall_tsv_creation
[INFO] [2021-01-27 17:02:55] Processing group of 70159 in 8 batches of 10000
[INFO] [2021-01-27 17:05:27] 46729 Traits (unfiltered)...
[INFO] [2021-01-27 17:09:09] 46729 Traits (filtered)...
[INFO] [2021-01-27 17:09:09] 0 Associations (filtered)...
[INFO] [2021-01-27 17:09:20] 93458 metadata added.
[INFO] [2021-01-27 17:09:20] 0 metadata added.
[INFO] [2021-01-27 17:11:39] 38149 Traits (unfiltered)...
[INFO] [2021-01-27 17:14:53] 38149 Traits (filtered)...
[INFO] [2021-01-27 17:14:53] 0 Associations (filtered)...
[INFO] [2021-01-27 17:15:02] 76298 metadata added.
[INFO] [2021-01-27 17:15:02] 0 metadata added.
[INFO] [2021-01-27 17:17:10] 54930 Traits (unfiltered)...
[INFO] [2021-01-27 17:21:26] 54930 Traits (filtered)...
[INFO] [2021-01-27 17:21:26] 0 Associations (filtered)...
[INFO] [2021-01-27 17:21:44] 109860 metadata added.
[INFO] [2021-01-27 17:21:44] 0 metadata added.
[INFO] [2021-01-27 17:24:00] 91514 Traits (unfiltered)...
[INFO] [2021-01-27 17:30:56] 91514 Traits (filtered)...
[INFO] [2021-01-27 17:30:56] 0 Associations (filtered)...
[INFO] [2021-01-27 17:31:24] 183028 metadata added.
[INFO] [2021-01-27 17:31:24] 0 metadata added.
[INFO] [2021-01-27 17:33:58] 60210 Traits (unfiltered)...
[INFO] [2021-01-27 17:38:47] 60210 Traits (filtered)...
[INFO] [2021-01-27 17:38:47] 0 Associations (filtered)...
[INFO] [2021-01-27 17:39:07] 120420 metadata added.
[INFO] [2021-01-27 17:39:07] 0 metadata added.
[INFO] [2021-01-27 17:41:25] 59533 Traits (unfiltered)...
[INFO] [2021-01-27 17:46:03] 59533 Traits (filtered)...
[INFO] [2021-01-27 17:46:03] 0 Associations (filtered)...
[INFO] [2021-01-27 17:46:22] 119066 metadata added.
[INFO] [2021-01-27 17:46:22] 0 metadata added.
[INFO] [2021-01-27 17:48:38] 58778 Traits (unfiltered)...
[INFO] [2021-01-27 17:53:12] 58778 Traits (filtered)...
[INFO] [2021-01-27 17:53:12] 0 Associations (filtered)...
[INFO] [2021-01-27 17:53:31] 117556 metadata added.
[INFO] [2021-01-27 17:53:31] 0 metadata added.
[INFO] [2021-01-27 17:55:00] 933 Traits (unfiltered)...
[INFO] [2021-01-27 17:55:37] 933 Traits (filtered)...
[INFO] [2021-01-27 17:55:37] 0 Associations (filtered)...
[INFO] [2021-01-27 17:55:37] 1866 metadata added.
[INFO] [2021-01-27 17:55:37] 0 metadata added.
[INFO] [2021-01-27 17:56:09] Average Time: 351.756
[INFO] [2021-01-27 17:56:09] Total Time: 53m15s
[INFO] [2021-01-27 17:56:09] last 3 / first 3: 0.91
[INFO] [2021-01-27 17:56:09] Std.Dev: 133.842470837922; Max: 553.94
[STOP] [2021-01-27 17:56:09] overall_tsv_creation
[INFO] [2021-01-27 17:56:09] Done. Check your files:
[INFO] [2021-01-27 17:56:09] (70136 lines) /app/public/data/brazilian_flora_/publish_nodes.tsv
[INFO] [2021-01-27 17:56:10] (206343 lines) /app/public/data/brazilian_flora_/publish_node_ancestors.tsv
[INFO] [2021-01-27 17:56:10] (116838 lines) /app/public/data/brazilian_flora_/publish_scientific_names.tsv
[INFO] [2021-01-27 17:56:10] (8548 lines) /app/public/data/brazilian_flora_/publish_vernaculars.tsv
[INFO] [2021-01-27 17:56:11] (26526 lines) /app/public/data/brazilian_flora_/publish_references.tsv
[INFO] [2021-01-27 17:56:11] (26526 lines) /app/public/data/brazilian_flora_/publish_referents.tsv
[INFO] [2021-01-27 17:56:12] (410777 lines) /app/public/data/brazilian_flora_/publish_traits.tsv
[INFO] [2021-01-27 17:56:12] (821553 lines) /app/public/data/brazilian_flora_/publish_metadata.tsv
[STOP] [2021-01-27 17:56:13] complete_harvest_instance
[START] [2021-01-27 17:56:13] completed
[STOP] [2021-01-27 17:56:13] completed
[STOP] [2021-01-27 17:56:13] logged process, took 11803.87
[INFO] [2021-01-27 17:56:13] ## remove_type: ScientificName
[INFO] [2021-01-27 17:56:13] ++ Calling delete_all on 133626 instances...
[INFO] [2021-01-27 17:56:15] There was an error, retrying: Lockfile::StolenLockError: Lockfile::StolenLockError: DELETE FROM `scientific_names` WHERE `scientific_names`.`harvest_id` = 3448
[INFO] [2021-01-27 17:56:17] ## remove_type: ScientificName
[INFO] [2021-01-27 17:56:17] ++ Calling delete_all on 0 instances...
[INFO] [2021-01-27 17:56:17] [17:56:17.569] Removed 0 Scientificnames
[INFO] [2021-01-27 17:56:17] ## remove_type: Vernacular
[INFO] [2021-01-27 17:56:17] ++ Calling delete_all on 9124 instances...
[INFO] [2021-01-27 17:56:17] [17:56:17.656] Removed 9124 Vernaculars
[INFO] [2021-01-27 17:56:17] ## remove_type: Article
[INFO] [2021-01-27 17:56:17] ++ Calling delete_all on 0 instances...
[INFO] [2021-01-27 17:56:17] [17:56:17.658] Removed 0 Articles
[INFO] [2021-01-27 17:56:17] ## remove_type: Medium
[INFO] [2021-01-27 17:56:17] ++ Calling delete_all on 0 instances...
[INFO] [2021-01-27 17:56:17] [17:56:17.659] Removed 0 Media
[INFO] [2021-01-27 17:56:17] ## remove_type: Trait
[INFO] [2021-01-27 17:56:17] ++ Batch removal of 413020 instances...
[INFO] [2021-01-27 17:56:17] [17:56:17.735] Batch 0...
[INFO] [2021-01-27 17:56:18] [17:56:18.890] Batch 1...
[INFO] [2021-01-27 17:56:20] [17:56:20.084] Batch 2...
[INFO] [2021-01-27 17:56:21] [17:56:21.386] Batch 3...
[INFO] [2021-01-27 17:56:23] [17:56:23.073] Batch 4...
[INFO] [2021-01-27 17:56:24] [17:56:24.241] Batch 5...
[INFO] [2021-01-27 17:56:25] [17:56:25.514] Batch 6...
[INFO] [2021-01-27 17:56:27] [17:56:27.459] Batch 7...
[INFO] [2021-01-27 17:56:29] [17:56:29.107] Batch 8...
[INFO] [2021-01-27 17:56:30] [17:56:30.043] Batch 9...
[INFO] [2021-01-27 17:56:31] [17:56:31.867] Batch 10...
[INFO] [2021-01-27 17:56:33] [17:56:33.811] Batch 11...
[INFO] [2021-01-27 17:56:35] [17:56:35.218] Batch 12...
[INFO] [2021-01-27 17:56:37] [17:56:37.477] Batch 13...
[INFO] [2021-01-27 17:56:39] [17:56:39.798] Batch 14...
[INFO] [2021-01-27 17:56:41] [17:56:41.191] Batch 15...
[INFO] [2021-01-27 17:56:43] [17:56:43.477] Batch 16...
[INFO] [2021-01-27 17:56:45] [17:56:45.058] Batch 17...
[INFO] [2021-01-27 17:56:47] [17:56:47.469] Batch 18...
[INFO] [2021-01-27 17:56:49] [17:56:49.625] Batch 19...
[INFO] [2021-01-27 17:56:52] [17:56:52.431] Batch 20...
[INFO] [2021-01-27 17:56:57] [17:56:57.129] Batch 21...
[INFO] [2021-01-27 17:56:59] [17:56:59.960] Batch 22...
[INFO] [2021-01-27 17:57:03] [17:57:03.858] Batch 23...
[INFO] [2021-01-27 17:57:07] [17:57:07.038] Batch 24...
[INFO] [2021-01-27 17:57:10] [17:57:10.947] Batch 25...
[INFO] [2021-01-27 17:57:14] [17:57:14.196] Batch 26...
[INFO] [2021-01-27 17:57:16] [17:57:16.983] Batch 27...
[INFO] [2021-01-27 17:57:19] [17:57:19.920] Batch 28...
[INFO] [2021-01-27 17:57:21] [17:57:21.356] Batch 29...
[INFO] [2021-01-27 17:57:23] [17:57:23.121] Batch 30...
[INFO] [2021-01-27 17:57:25] [17:57:25.727] Batch 31...
[INFO] [2021-01-27 17:57:27] [17:57:27.263] Batch 32...
[INFO] [2021-01-27 17:57:28] [17:57:28.548] Batch 33...
[INFO] [2021-01-27 17:57:30] [17:57:30.431] Batch 34...
[INFO] [2021-01-27 17:57:31] [17:57:31.765] Batch 35...
[INFO] [2021-01-27 17:57:32] [17:57:32.939] Batch 36...
[INFO] [2021-01-27 17:57:33] [17:57:33.925] Batch 37...
[INFO] [2021-01-27 17:57:34] [17:57:34.670] Batch 38...
[INFO] [2021-01-27 17:57:35] [17:57:35.554] Batch 39...
[INFO] [2021-01-27 17:57:36] [17:57:36.810] Batch 40...
[INFO] [2021-01-27 17:57:37] [17:57:37.679] Batch 41...
[INFO] [2021-01-27 17:57:38] [17:57:38.800] Removed 413020 Traits
[INFO] [2021-01-27 17:57:38] ## remove_type: MetaTrait
[INFO] [2021-01-27 17:57:38] ++ Batch removal of 826040 instances...
[INFO] [2021-01-27 17:57:38] [17:57:38.935] Batch 0...
[INFO] [2021-01-27 17:57:39] [17:57:39.657] Batch 1...
[INFO] [2021-01-27 17:57:40] [17:57:40.396] Batch 2...
[INFO] [2021-01-27 17:57:41] [17:57:41.189] Batch 3...
[INFO] [2021-01-27 17:57:41] [17:57:41.904] Batch 4...
[INFO] [2021-01-27 17:57:42] [17:57:42.630] Batch 5...
[INFO] [2021-01-27 17:57:43] [17:57:43.349] Batch 6...
[INFO] [2021-01-27 17:57:44] [17:57:44.176] Batch 7...
[INFO] [2021-01-27 17:57:45] [17:57:45.231] Batch 8...
[INFO] [2021-01-27 17:57:45] [17:57:45.955] Batch 9...
[INFO] [2021-01-27 17:57:46] [17:57:46.710] Batch 10...
[INFO] [2021-01-27 17:57:47] [17:57:47.422] Batch 11...
[INFO] [2021-01-27 17:57:48] [17:57:48.134] Batch 12...
[INFO] [2021-01-27 17:57:48] [17:57:48.845] Batch 13...
[INFO] [2021-01-27 17:57:49] [17:57:49.762] Batch 14...
[INFO] [2021-01-27 17:57:50] [17:57:50.658] Batch 15...
[INFO] [2021-01-27 17:57:51] [17:57:51.353] Batch 16...
[INFO] [2021-01-27 17:57:52] [17:57:52.047] Batch 17...
[INFO] [2021-01-27 17:57:52] [17:57:52.742] Batch 18...
[INFO] [2021-01-27 17:57:53] [17:57:53.449] Batch 19...
[INFO] [2021-01-27 17:57:54] [17:57:54.139] Batch 20...
[INFO] [2021-01-27 17:57:54] [17:57:54.836] Batch 21...
[INFO] [2021-01-27 17:57:55] [17:57:55.533] Batch 22...
[INFO] [2021-01-27 17:57:56] [17:57:56.218] Batch 23...
[INFO] [2021-01-27 17:57:56] [17:57:56.909] Batch 24...
[INFO] [2021-01-27 17:57:57] [17:57:57.593] Batch 25...
[INFO] [2021-01-27 17:57:58] [17:57:58.273] Batch 26...
[INFO] [2021-01-27 17:57:58] [17:57:58.953] Batch 27...
[INFO] [2021-01-27 17:57:59] [17:57:59.641] Batch 28...
[INFO] [2021-01-27 17:58:00] [17:58:00.318] Batch 29...
[INFO] [2021-01-27 17:58:00] [17:58:00.991] Batch 30...
[INFO] [2021-01-27 17:58:01] [17:58:01.664] Batch 31...
[INFO] [2021-01-27 17:58:02] [17:58:02.334] Batch 32...
[INFO] [2021-01-27 17:58:03] [17:58:03.003] Batch 33...
[INFO] [2021-01-27 17:58:03] [17:58:03.670] Batch 34...
[INFO] [2021-01-27 17:58:04] [17:58:04.334] Batch 35...
[INFO] [2021-01-27 17:58:04] [17:58:04.997] Batch 36...
[INFO] [2021-01-27 17:58:05] [17:58:05.660] Batch 37...
[INFO] [2021-01-27 17:58:06] [17:58:06.319] Batch 38...
[INFO] [2021-01-27 17:58:06] [17:58:06.978] Batch 39...
[INFO] [2021-01-27 17:58:07] [17:58:07.637] Batch 40...
[INFO] [2021-01-27 17:58:08] [17:58:08.293] Batch 41...
[INFO] [2021-01-27 17:58:08] [17:58:08.952] Batch 42...
[INFO] [2021-01-27 17:58:09] [17:58:09.678] Batch 43...
[INFO] [2021-01-27 17:58:10] [17:58:10.329] Batch 44...
[INFO] [2021-01-27 17:58:10] [17:58:10.984] Batch 45...
[INFO] [2021-01-27 17:58:11] [17:58:11.636] Batch 46...
[INFO] [2021-01-27 17:58:12] [17:58:12.284] Batch 47...
[INFO] [2021-01-27 17:58:12] [17:58:12.930] Batch 48...
[INFO] [2021-01-27 17:58:13] [17:58:13.573] Batch 49...
[INFO] [2021-01-27 17:58:14] [17:58:14.214] Batch 50...
[INFO] [2021-01-27 17:58:14] [17:58:14.857] Batch 51...
[INFO] [2021-01-27 17:58:15] [17:58:15.496] Batch 52...
[INFO] [2021-01-27 17:58:16] [17:58:16.132] Batch 53...
[INFO] [2021-01-27 17:58:16] [17:58:16.766] Batch 54...
[INFO] [2021-01-27 17:58:17] [17:58:17.451] Batch 55...
[INFO] [2021-01-27 17:58:18] [17:58:18.082] Batch 56...
[INFO] [2021-01-27 17:58:18] [17:58:18.711] Batch 57...
[INFO] [2021-01-27 17:58:19] [17:58:19.346] Batch 58...
[INFO] [2021-01-27 17:58:19] [17:58:19.974] Batch 59...
[INFO] [2021-01-27 17:58:20] [17:58:20.599] Batch 60...
[INFO] [2021-01-27 17:58:21] [17:58:21.220] Batch 61...
[INFO] [2021-01-27 17:58:21] [17:58:21.841] Batch 62...
[INFO] [2021-01-27 17:58:22] [17:58:22.460] Batch 63...
[INFO] [2021-01-27 17:58:23] [17:58:23.080] Batch 64...
[INFO] [2021-01-27 17:58:23] [17:58:23.697] Batch 65...
[INFO] [2021-01-27 17:58:24] [17:58:24.318] Batch 66...
[INFO] [2021-01-27 17:58:24] [17:58:24.935] Batch 67...
[INFO] [2021-01-27 17:58:25] [17:58:25.550] Batch 68...
[INFO] [2021-01-27 17:58:26] [17:58:26.160] Batch 69...
[INFO] [2021-01-27 17:58:26] [17:58:26.767] Batch 70...
[INFO] [2021-01-27 17:58:27] [17:58:27.379] Batch 71...
[INFO] [2021-01-27 17:58:27] [17:58:27.983] Batch 72...
[INFO] [2021-01-27 17:58:28] [17:58:28.586] Batch 73...
[INFO] [2021-01-27 17:58:29] [17:58:29.189] Batch 74...
[INFO] [2021-01-27 17:58:29] [17:58:29.791] Batch 75...
[INFO] [2021-01-27 17:58:30] [17:58:30.401] Batch 76...
[INFO] [2021-01-27 17:58:30] [17:58:30.998] Batch 77...
[INFO] [2021-01-27 17:58:31] [17:58:31.593] Batch 78...
[INFO] [2021-01-27 17:58:32] [17:58:32.188] Batch 79...
[INFO] [2021-01-27 17:58:32] [17:58:32.780] Batch 80...
[INFO] [2021-01-27 17:58:33] [17:58:33.384] Batch 81...
[INFO] [2021-01-27 17:58:33] [17:58:33.973] Batch 82...
[INFO] [2021-01-27 17:58:34] [17:58:34.528] Removed 826040 Metatraits
[INFO] [2021-01-27 17:58:34] ## remove_type: OccurrenceMetadatum
[INFO] [2021-01-27 17:58:34] ++ Calling delete_all on 0 instances...
[INFO] [2021-01-27 17:58:34] [17:58:34.628] Removed 0 Occurrencemetadata
[INFO] [2021-01-27 17:58:34] ## remove_type: Assoc
[INFO] [2021-01-27 17:58:34] ++ Calling delete_all on 0 instances...
[INFO] [2021-01-27 17:58:34] [17:58:34.629] Removed 0 Assocs
[INFO] [2021-01-27 17:58:34] ## remove_type: MetaAssoc
[INFO] [2021-01-27 17:58:34] ++ Calling delete_all on 0 instances...
[INFO] [2021-01-27 17:58:34] [17:58:34.639] Removed 0 Metaassocs
[INFO] [2021-01-27 17:58:34] ## remove_type: Identifier
[INFO] [2021-01-27 17:58:34] ++ Calling delete_all on 0 instances...
[INFO] [2021-01-27 17:58:34] [17:58:34.641] Removed 0 Identifiers
[INFO] [2021-01-27 17:58:34] ## remove_type: Reference
[INFO] [2021-01-27 17:58:34] ++ Calling delete_all on 55234 instances...
[INFO] [2021-01-27 17:58:35] [17:58:35.809] Removed 55234 References
[INFO] [2021-01-27 17:58:36] Starting batch with ID 87485617...
[INFO] [2021-01-27 17:58:37] Starting batch with ID 87485617...
[INFO] [2021-01-27 17:58:38] Starting batch with ID 87505408...
[INFO] [2021-01-27 17:58:39] Starting batch with ID 87505408...
[INFO] [2021-01-27 17:58:40] Starting batch with ID 87537890...
[INFO] [2021-01-27 17:58:41] Starting batch with ID 87537890...
[INFO] [2021-01-27 17:58:42] Starting batch with ID 87529898...
[INFO] [2021-01-27 17:58:43] Starting batch with ID 87529898...
[INFO] [2021-01-27 17:58:44] Starting batch with ID 87498724...
[INFO] [2021-01-27 17:58:45] Starting batch with ID 87498724...
[INFO] [2021-01-27 17:58:46] Starting batch with ID 87491458...
[INFO] [2021-01-27 17:58:47] Starting batch with ID 87491458...
[INFO] [2021-01-27 17:58:51] Starting batch with ID 87507576...
[INFO] [2021-01-27 17:58:52] Starting batch with ID 87507576...
[INFO] [2021-01-27 17:58:53] Starting batch with ID 87513786...
[INFO] [2021-01-27 17:58:54] Starting batch with ID 87513786...
[INFO] [2021-01-27 17:58:55] Starting batch with ID 87528904...
[INFO] [2021-01-27 17:58:56] Starting batch with ID 87548976...
[INFO] [2021-01-27 17:58:57] Starting batch with ID 87507086...
[INFO] [2021-01-27 17:58:59] Starting batch with ID 87507086...
[INFO] [2021-01-27 17:59:00] Starting batch with ID 87544653...
[INFO] [2021-01-27 17:59:01] Starting batch with ID 87544653...
[INFO] [2021-01-27 17:59:02] Starting batch with ID 87545611...
[INFO] [2021-01-27 17:59:03] Starting batch with ID 87545611...
[INFO] [2021-01-27 17:59:04] Starting batch with ID 87524869...
[INFO] [2021-01-27 17:59:05] Starting batch with ID 87524869...
[INFO] [2021-01-27 17:59:06] Starting batch with ID 87501028...
[INFO] [2021-01-27 17:59:07] Starting batch with ID 87501028...
[INFO] [2021-01-27 17:59:07] ## remove_type: Node
[INFO] [2021-01-27 17:59:07] ++ Calling delete_all on 70159 instances...
[INFO] [2021-01-27 17:59:09] [17:59:09.481] Removed 70159 Nodes
[START] [2021-01-27 17:59:32] logged process: e38687fde360d9ce2f48dd55c3bf56ebb9d8efa6

[START] [2021-01-27 17:59:32] Creating resource from OpenData
[START] [2021-01-27 17:59:37] logged process: e38687fde360d9ce2f48dd55c3bf56ebb9d8efa6

[START] [2021-01-27 17:59:37] Parse meta.xml file and create formats with fields
[WARN] [2021-01-27 17:59:43] (common) IGNORED  (nodes) field header: scientificNameAuthorship term: http://rs.tdwg.org/dwc/terms/scientificNameAuthorship
[WARN] [2021-01-27 17:59:43] (common) IGNORED  (nodes) field header: modified term: http://purl.org/dc/terms/modified
[STOP] [2021-01-27 17:59:43] Parse meta.xml file and create formats with fields
[STOP] [2021-01-27 17:59:43] Creating resource from OpenData
[START] [2021-01-27 17:59:44] logged process: e38687fde360d9ce2f48dd55c3bf56ebb9d8efa6

[START] [2021-01-27 17:59:44] create_harvest_instance
[STOP] [2021-01-27 17:59:46] create_harvest_instance
[START] [2021-01-27 17:59:46] fetch_files
[STOP] [2021-01-27 17:59:46] fetch_files
[START] [2021-01-27 17:59:46] validate_each_file
[STOP] [2021-01-27 18:00:16] validate_each_file
[START] [2021-01-27 18:00:16] convert_to_csv
[CMD] [2021-01-27 18:00:16] /usr/bin/sort /app/public/converted_csv/brazilian_flora__refs_26183.csv > /app/public/converted_csv/brazilian_flora__refs_26183.csv_sorted
[CMD] [2021-01-27 18:00:17] /usr/bin/sort /app/public/converted_csv/brazilian_flora__nodes_26184.csv > /app/public/converted_csv/brazilian_flora__nodes_26184.csv_sorted
[CMD] [2021-01-27 18:00:17] /usr/bin/sort /app/public/converted_csv/brazilian_flora__vernaculars_26185.csv > /app/public/converted_csv/brazilian_flora__vernaculars_26185.csv_sorted
[CMD] [2021-01-27 18:00:17] /usr/bin/sort /app/public/converted_csv/brazilian_flora__occurrences_26186.csv > /app/public/converted_csv/brazilian_flora__occurrences_26186.csv_sorted
[CMD] [2021-01-27 18:00:18] /usr/bin/sort /app/public/converted_csv/brazilian_flora__measurements_26187.csv > /app/public/converted_csv/brazilian_flora__measurements_26187.csv_sorted
[STOP] [2021-01-27 18:00:19] convert_to_csv
[START] [2021-01-27 18:00:19] calculate_delta
[CMD] [2021-01-27 18:00:19] echo "0a" > /app/public/diff/brazilian_flora__refs_26183.diff
[CMD] [2021-01-27 18:00:19] tail -n +1 /app/public/converted_csv/brazilian_flora__refs_26183.csv >> /app/public/diff/brazilian_flora__refs_26183.diff
[CMD] [2021-01-27 18:00:19] echo "." >> /app/public/diff/brazilian_flora__refs_26183.diff
[CMD] [2021-01-27 18:00:20] echo "0a" > /app/public/diff/brazilian_flora__nodes_26184.diff
[CMD] [2021-01-27 18:00:20] tail -n +1 /app/public/converted_csv/brazilian_flora__nodes_26184.csv >> /app/public/diff/brazilian_flora__nodes_26184.diff
[CMD] [2021-01-27 18:00:20] echo "." >> /app/public/diff/brazilian_flora__nodes_26184.diff
[CMD] [2021-01-27 18:00:21] echo "0a" > /app/public/diff/brazilian_flora__vernaculars_26185.diff
[CMD] [2021-01-27 18:00:21] tail -n +1 /app/public/converted_csv/brazilian_flora__vernaculars_26185.csv >> /app/public/diff/brazilian_flora__vernaculars_26185.diff
[CMD] [2021-01-27 18:00:21] echo "." >> /app/public/diff/brazilian_flora__vernaculars_26185.diff
[CMD] [2021-01-27 18:00:22] echo "0a" > /app/public/diff/brazilian_flora__occurrences_26186.diff
[CMD] [2021-01-27 18:00:22] tail -n +1 /app/public/converted_csv/brazilian_flora__occurrences_26186.csv >> /app/public/diff/brazilian_flora__occurrences_26186.diff
[CMD] [2021-01-27 18:00:22] echo "." >> /app/public/diff/brazilian_flora__occurrences_26186.diff
[CMD] [2021-01-27 18:00:23] echo "0a" > /app/public/diff/brazilian_flora__measurements_26187.diff
[CMD] [2021-01-27 18:00:23] tail -n +1 /app/public/converted_csv/brazilian_flora__measurements_26187.csv >> /app/public/diff/brazilian_flora__measurements_26187.diff
[CMD] [2021-01-27 18:00:24] echo "." >> /app/public/diff/brazilian_flora__measurements_26187.diff
[STOP] [2021-01-27 18:00:24] calculate_delta
[START] [2021-01-27 18:00:24] parse_diff_and_store
[INFO] [2021-01-27 18:00:24] Loading refs diff file into memory (true lines)...
[INFO] [2021-01-27 18:00:30] Loading nodes diff file into memory (true lines)...
[WARN] [2021-01-27 18:00:39] Filtered Scientific Name `Peridinium gatunense var. madagascariensis  Lefèvre` to `Peridinium gatunense var. madagascariensis Lefèvre`
[WARN] [2021-01-27 18:00:45] Filtered Scientific Name `Glomus pachycaule  (C.G. Wu & Z.C. Chen) Sieverd. & Oehl` to `Glomus pachycaule (C.G. Wu & Z.C. Chen) Sieverd. & Oehl`
[WARN] [2021-01-27 18:00:49] Filtered Scientific Name `Poraqueiba  Aubl.` to `Poraqueiba Aubl.`
[WARN] [2021-01-27 18:00:56] Filtered Scientific Name `Leopoldia  Herb.` to `Leopoldia Herb.`
[WARN] [2021-01-27 18:00:59] Filtered Scientific Name `Lejeunea ulicina (Taylor) Gottsche  et al.` to `Lejeunea ulicina (Taylor) Gottsche et al.`
[WARN] [2021-01-27 18:01:00] Filtered Scientific Name `Psilotum triquetrum  Sw.` to `Psilotum triquetrum Sw.`
[WARN] [2021-01-27 18:01:00] Filtered Scientific Name `Perichaena calongei  Lado, D. Wrigley & Estrada` to `Perichaena calongei Lado, D. Wrigley & Estrada`
[WARN] [2021-01-27 18:01:00] Filtered Scientific Name `Oncidium clovesianum  (V.P.Castro & M.M.L.L.Cardoso) J.M.H.Shaw` to `Oncidium clovesianum (V.P.Castro & M.M.L.L.Cardoso) J.M.H.Shaw`
[WARN] [2021-01-27 18:01:00] Filtered Scientific Name `Craspedaria  Link` to `Craspedaria Link`
[WARN] [2021-01-27 18:01:00] Filtered Scientific Name `Neolacis  Wedd.` to `Neolacis Wedd.`
[WARN] [2021-01-27 18:01:00] Filtered Scientific Name `Croton verbenaefolius  Müll.Arg.` to `Croton verbenaefolius Müll.Arg.`
[WARN] [2021-01-27 18:01:00] Filtered Scientific Name `Xylosma ciliatifolium  (Clos) Eichler` to `Xylosma ciliatifolium (Clos) Eichler`
[WARN] [2021-01-27 18:01:00] Filtered Scientific Name `Physalis linkiana  Nees` to `Physalis linkiana Nees`
[WARN] [2021-01-27 18:01:00] Filtered Scientific Name `Guatteria laevigata  Mart.` to `Guatteria laevigata Mart.`
[WARN] [2021-01-27 18:01:00] Filtered Scientific Name `Lejeunea maxonii (A. Evans)  X.-L. He` to `Lejeunea maxonii (A. Evans) X.-L. He`
[WARN] [2021-01-27 18:01:00] Filtered Scientific Name `Coccoloba fastigata  Meisn.` to `Coccoloba fastigata Meisn.`
[WARN] [2021-01-27 18:01:01] Filtered Scientific Name `Colliguaya brasiliensis  Klotzsch ex Baill.` to `Colliguaya brasiliensis Klotzsch ex Baill.`
[WARN] [2021-01-27 18:01:01] Filtered Scientific Name `Polypodium pendulum  Sw.` to `Polypodium pendulum Sw.`
[WARN] [2021-01-27 18:01:01] Filtered Scientific Name `Miconia saldanhaei  Cogn.` to `Miconia saldanhaei Cogn.`
[WARN] [2021-01-27 18:01:01] Filtered Scientific Name `Coccoloba martii  Meisn.` to `Coccoloba martii Meisn.`
[WARN] [2021-01-27 18:01:01] Filtered Scientific Name `Trembleya selloana  Cogn.` to `Trembleya selloana Cogn.`
[WARN] [2021-01-27 18:01:01] Filtered Scientific Name `Croton muscicapa  Müll.Arg.` to `Croton muscicapa Müll.Arg.`
[WARN] [2021-01-27 18:01:01] Filtered Scientific Name `Coccoloba sticticaulis  Wedd.` to `Coccoloba sticticaulis Wedd.`
[WARN] [2021-01-27 18:01:01] Filtered Scientific Name `Coccoloba polystachya  Wedd.` to `Coccoloba polystachya Wedd.`
[WARN] [2021-01-27 18:01:01] Filtered Scientific Name `Xylosma tweedianum  (Clos) Eichler` to `Xylosma tweedianum (Clos) Eichler`
[WARN] [2021-01-27 18:01:01] Filtered Scientific Name `Coccoloba crescentiaefolia  Meisn.` to `Coccoloba crescentiaefolia Meisn.`
[WARN] [2021-01-27 18:01:01] Filtered Scientific Name `Celtis aculeata  Sw.` to `Celtis aculeata Sw.`
[WARN] [2021-01-27 18:01:01] Filtered Scientific Name `Polypodium diversifolium  Sw.` to `Polypodium diversifolium Sw.`
[WARN] [2021-01-27 18:01:01] Filtered Scientific Name `Croton rufo-argenteus  Müll.Arg.` to `Croton rufo-argenteus Müll.Arg.`
[WARN] [2021-01-27 18:01:01] (Reached filtered-name limit; supressing further warnings.)
[INFO] [2021-01-27 18:01:16] Loading vernaculars diff file into memory (true lines)...
[INFO] [2021-01-27 18:01:18] Loading occurrences diff file into memory (true lines)...
[INFO] [2021-01-27 18:02:08] Loading measurements diff file into memory (true lines)...
[INFO] [2021-01-27 18:08:03] Storing 55234 References
[INFO] [2021-01-27 18:08:03] Processing group of 55234 in 56 groups of 1000
[INFO] [2021-01-27 18:08:14] Average Time: 0.192
[INFO] [2021-01-27 18:08:14] Total Time: 11s
[INFO] [2021-01-27 18:08:14] last 3 / first 3: 0.7
[INFO] [2021-01-27 18:08:14] Std.Dev: 0.044721359549995794; Max: 0.36
[INFO] [2021-01-27 18:08:14] Storing 133626 ScientificNames
[INFO] [2021-01-27 18:08:14] Processing group of 133626 in 134 groups of 1000
[INFO] [2021-01-27 18:09:15] Average Time: 0.446
[INFO] [2021-01-27 18:09:15] Total Time: 1m1s
[INFO] [2021-01-27 18:09:15] last 3 / first 3: 0.79
[INFO] [2021-01-27 18:09:15] Std.Dev: 1.0134100848126586; Max: 9.81
[INFO] [2021-01-27 18:09:15] Storing 70159 Nodes
[INFO] [2021-01-27 18:09:15] Processing group of 70159 in 71 groups of 1000
[INFO] [2021-01-27 18:09:54] Average Time: 0.552
[INFO] [2021-01-27 18:09:54] Total Time: 40s
[INFO] [2021-01-27 18:09:54] last 3 / first 3: 0.73
[INFO] [2021-01-27 18:09:54] Std.Dev: 1.6003124694883808; Max: 9.91
[INFO] [2021-01-27 18:09:54] Storing 40881 NodesReferences
[INFO] [2021-01-27 18:09:54] Processing group of 40881 in 41 groups of 1000
[INFO] [2021-01-27 18:09:58] Average Time: 0.08
[INFO] [2021-01-27 18:09:58] Total Time: 4s
[INFO] [2021-01-27 18:09:58] last 3 / first 3: 0.95
[INFO] [2021-01-27 18:09:58] Std.Dev: 0.044721359549995794; Max: 0.27
[INFO] [2021-01-27 18:09:58] Storing 9124 Vernaculars
[INFO] [2021-01-27 18:09:58] Processing group of 9124 in 10 groups of 1000
[INFO] [2021-01-27 18:09:59] Average Time: 0.161
[INFO] [2021-01-27 18:09:59] Total Time: 2s
[INFO] [2021-01-27 18:09:59] last 3 / first 3: 0.71
[INFO] [2021-01-27 18:09:59] Std.Dev: 0.05477225575051661; Max: 0.24
[INFO] [2021-01-27 18:09:59] Storing 404891 Occurrences
[INFO] [2021-01-27 18:09:59] Processing group of 404891 in 405 groups of 1000
[INFO] [2021-01-27 18:11:31] Average Time: 0.222
[INFO] [2021-01-27 18:11:31] Total Time: 1m32s
[INFO] [2021-01-27 18:11:31] last 3 / first 3: 0.97
[INFO] [2021-01-27 18:11:31] Std.Dev: 1.0168579055108928; Max: 10.99
[INFO] [2021-01-27 18:11:31] Storing 413020 Traits
[INFO] [2021-01-27 18:11:31] Processing group of 413020 in 414 groups of 1000
[INFO] [2021-01-27 18:15:26] Average Time: 0.563
[INFO] [2021-01-27 18:15:26] Total Time: 3m56s
[INFO] [2021-01-27 18:15:26] last 3 / first 3: 0.93
[INFO] [2021-01-27 18:15:26] Std.Dev: 1.6337074401495515; Max: 12.47
[INFO] [2021-01-27 18:15:26] Storing 826040 MetaTraits
[INFO] [2021-01-27 18:15:26] Processing group of 826040 in 827 groups of 1000
[INFO] [2021-01-27 18:18:10] Average Time: 0.193
[INFO] [2021-01-27 18:18:10] Total Time: 2m44s
[INFO] [2021-01-27 18:18:10] last 3 / first 3: 0.68
[INFO] [2021-01-27 18:18:10] Std.Dev: 0.8497058314499201; Max: 12.54
[STOP] [2021-01-27 18:18:10] parse_diff_and_store
[START] [2021-01-27 18:18:10] resolve_keys
[INFO] [2021-01-27 18:18:41] Occurrences to nodes (through scientific_names)...
[INFO] [2021-01-27 18:18:54] traits to occurrences...
[INFO] [2021-01-27 18:19:11] traits to nodes (through occurrences)...
[INFO] [2021-01-27 18:19:19] Traits to sex term...
[INFO] [2021-01-27 18:19:24] Traits to lifestage term...
[INFO] [2021-01-27 18:19:29] MetaTraits to traits...
[INFO] [2021-01-27 18:19:50] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-01-27 18:19:50] Assocs to occurrences...
[INFO] [2021-01-27 18:19:50] Assocs to nodes...
[INFO] [2021-01-27 18:19:50] Assoc to sex term...
[INFO] [2021-01-27 18:19:50] Assoc to lifestage term...
[INFO] [2021-01-27 18:19:50] MetaAssoc to assocs...
[STOP] [2021-01-27 18:19:50] resolve_keys
[START] [2021-01-27 18:19:50] hold_for_later_1
[STOP] [2021-01-27 18:19:50] hold_for_later_1
[START] [2021-01-27 18:19:50] hold_for_later_2
[STOP] [2021-01-27 18:19:50] hold_for_later_2
[START] [2021-01-27 18:19:50] resolve_missing_parents
[STOP] [2021-01-27 18:19:52] resolve_missing_parents
[START] [2021-01-27 18:19:52] rebuild_nodes
[START] [2021-01-27 18:19:52] Flattener#flatten
[START] [2021-01-27 18:19:52] Flattener#study_resource
[START] [2021-01-27 18:19:53] Flattener#build_ancestry
[STOP] [2021-01-27 18:20:23] Flattener#build_ancestry
[INFO] [2021-01-27 18:20:23] 70159 ancestry keys
[START] [2021-01-27 18:20:23] build_node_ancestors
[INFO] [2021-01-27 18:20:23] old ancestors deleted.
[STOP] [2021-01-27 18:20:44] build_node_ancestors
[START] [2021-01-27 18:20:44] Flattener#propagate_ancestor_ids
[STOP] [2021-01-27 18:20:49] Flattener#propagate_ancestor_ids
[STOP] [2021-01-27 18:20:49] Flattener#flatten
[STOP] [2021-01-27 18:20:49] rebuild_nodes
[START] [2021-01-27 18:20:49] resolve_missing_media_owners
[STOP] [2021-01-27 18:20:49] resolve_missing_media_owners
[START] [2021-01-27 18:20:49] sanitize_media_verbatims
[STOP] [2021-01-27 18:20:49] sanitize_media_verbatims
[START] [2021-01-27 18:20:49] queue_downloads
[STOP] [2021-01-27 18:20:49] queue_downloads
[START] [2021-01-27 18:20:49] parse_names
[WARN] [2021-01-27 18:20:49] I see 133626 names which still need to be parsed.
[WARN] [2021-01-27 18:22:28] I see 80 names which still need to be parsed.
[WARN] [2021-01-27 18:22:29] I see 11 names which still need to be parsed.
[WARN] [2021-01-27 18:22:31] I see 7 names which still need to be parsed.
[WARN] [2021-01-27 18:22:32] I see 6 names which still need to be parsed.
[WARN] [2021-01-27 18:22:34] I see 5 names which still need to be parsed.
[WARN] [2021-01-27 18:22:35] I see 4 names which still need to be parsed.
[WARN] [2021-01-27 18:22:37] I see 3 names which still need to be parsed.
[WARN] [2021-01-27 18:22:38] I see 2 names which still need to be parsed.
[WARN] [2021-01-27 18:22:40] I see 1 names which still need to be parsed.
[STOP] [2021-01-27 18:22:41] parse_names
[START] [2021-01-27 18:22:41] denormalize_canonical_names_to_nodes
[STOP] [2021-01-27 18:22:43] denormalize_canonical_names_to_nodes
[START] [2021-01-27 18:22:43] match_nodes
[START] [2021-01-27 18:22:43] map_all_nodes_to_pages
[STOP] [2021-01-27 20:44:15] map_all_nodes_to_pages
[INFO] [2021-01-27 20:44:15] 7946 Unmatched nodes (of 70159)! That's too many to output. Full list in /app/public/data/brazilian_flora_/unmatched_nodes.txt ; First 10: Mariscus (#87557479); Rhynchospora ciliata (#87609446); Eleocharis radicans (#87562426); Scleria setuloso-ciliata (#87585368); Dichromena (#87576842); Tetraria (#87577079); Websteria (#87585371); Androtrichum (#87596492); Egleria (#87596579); Adupla (#87601454)
[START] [2021-01-27 20:44:15] update_nodes
[STOP] [2021-01-27 20:44:44] update_nodes
[STOP] [2021-01-27 20:44:44] match_nodes
[START] [2021-01-27 20:44:44] reindex_search
[STOP] [2021-01-27 20:47:15] reindex_search
[START] [2021-01-27 20:47:15] normalize_units
[STOP] [2021-01-27 20:47:46] normalize_units
[START] [2021-01-27 20:47:46] calculate_statistics
[STOP] [2021-01-27 20:47:47] calculate_statistics
[START] [2021-01-27 20:47:47] complete_harvest_instance
[START] [2021-01-27 20:47:47] overall_tsv_creation
[INFO] [2021-01-27 20:47:47] Processing group of 70159 in 8 batches of 10000
[INFO] [2021-01-27 20:50:11] 46729 Traits (unfiltered)...
[INFO] [2021-01-27 20:54:00] 46729 Traits (filtered)...
[INFO] [2021-01-27 20:54:00] 0 Associations (filtered)...
[INFO] [2021-01-27 20:54:14] 93458 metadata added.
[INFO] [2021-01-27 20:54:14] 0 metadata added.
[INFO] [2021-01-27 20:56:30] 38149 Traits (unfiltered)...
[INFO] [2021-01-27 20:59:42] 38149 Traits (filtered)...
[INFO] [2021-01-27 20:59:42] 0 Associations (filtered)...
[INFO] [2021-01-27 20:59:54] 76298 metadata added.
[INFO] [2021-01-27 20:59:54] 0 metadata added.
[INFO] [2021-01-27 21:01:59] 54930 Traits (unfiltered)...
[INFO] [2021-01-27 21:06:19] 54930 Traits (filtered)...
[INFO] [2021-01-27 21:06:19] 0 Associations (filtered)...
[INFO] [2021-01-27 21:06:37] 109860 metadata added.
[INFO] [2021-01-27 21:06:37] 0 metadata added.
[INFO] [2021-01-27 21:08:56] 91514 Traits (unfiltered)...
[INFO] [2021-01-27 21:15:52] 91514 Traits (filtered)...
[INFO] [2021-01-27 21:15:52] 0 Associations (filtered)...
[INFO] [2021-01-27 21:16:27] 183028 metadata added.
[INFO] [2021-01-27 21:16:27] 0 metadata added.
[INFO] [2021-01-27 21:19:00] 60210 Traits (unfiltered)...
[INFO] [2021-01-27 21:23:48] 60210 Traits (filtered)...
[INFO] [2021-01-27 21:23:48] 0 Associations (filtered)...
[INFO] [2021-01-27 21:24:09] 120420 metadata added.
[INFO] [2021-01-27 21:24:09] 0 metadata added.
[INFO] [2021-01-27 21:26:23] 59533 Traits (unfiltered)...
[INFO] [2021-01-27 21:31:02] 59533 Traits (filtered)...
[INFO] [2021-01-27 21:31:02] 0 Associations (filtered)...
[INFO] [2021-01-27 21:31:22] 119066 metadata added.
[INFO] [2021-01-27 21:31:22] 0 metadata added.
[INFO] [2021-01-27 21:33:37] 58778 Traits (unfiltered)...
[INFO] [2021-01-27 21:38:13] 58778 Traits (filtered)...
[INFO] [2021-01-27 21:38:13] 0 Associations (filtered)...
[INFO] [2021-01-27 21:38:32] 117556 metadata added.
[INFO] [2021-01-27 21:38:32] 0 metadata added.
[INFO] [2021-01-27 21:40:00] 933 Traits (unfiltered)...
[INFO] [2021-01-27 21:40:38] 933 Traits (filtered)...
[INFO] [2021-01-27 21:40:38] 0 Associations (filtered)...
[INFO] [2021-01-27 21:40:38] 1866 metadata added.
[INFO] [2021-01-27 21:40:38] 0 metadata added.
[INFO] [2021-01-27 21:41:10] Average Time: 355.796
[INFO] [2021-01-27 21:41:10] Total Time: 53m23s
[INFO] [2021-01-27 21:41:10] last 3 / first 3: 0.89
[INFO] [2021-01-27 21:41:10] Std.Dev: 135.42631575879187; Max: 562.82
[STOP] [2021-01-27 21:41:10] overall_tsv_creation
[INFO] [2021-01-27 21:41:10] Done. Check your files:
[INFO] [2021-01-27 21:41:10] (70136 lines) /app/public/data/brazilian_flora_/publish_nodes.tsv
[INFO] [2021-01-27 21:41:11] (206343 lines) /app/public/data/brazilian_flora_/publish_node_ancestors.tsv
[INFO] [2021-01-27 21:41:11] (116838 lines) /app/public/data/brazilian_flora_/publish_scientific_names.tsv
[INFO] [2021-01-27 21:41:12] (8548 lines) /app/public/data/brazilian_flora_/publish_vernaculars.tsv
[INFO] [2021-01-27 21:41:12] (26526 lines) /app/public/data/brazilian_flora_/publish_references.tsv
[INFO] [2021-01-27 21:41:12] (26526 lines) /app/public/data/brazilian_flora_/publish_referents.tsv
[INFO] [2021-01-27 21:41:13] (410777 lines) /app/public/data/brazilian_flora_/publish_traits.tsv
[INFO] [2021-01-27 21:41:13] (821553 lines) /app/public/data/brazilian_flora_/publish_metadata.tsv
[STOP] [2021-01-27 21:41:13] complete_harvest_instance
[START] [2021-01-27 21:41:13] completed
[STOP] [2021-01-27 21:41:13] completed
[STOP] [2021-01-27 21:41:13] logged process, took 13290.24
[INFO] [2021-04-04 13:48:10] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-04 13:48:13] ## remove_type: ScientificName
[INFO] [2021-04-04 13:48:13] ++ Calling delete_all on 133626 instances...
[INFO] [2021-04-04 13:48:32] [13:48:32.868] Removed 133626 Scientificnames
[INFO] [2021-04-04 13:48:32] ## remove_type: Vernacular
[INFO] [2021-04-04 13:48:32] ++ Calling delete_all on 9124 instances...
[INFO] [2021-04-04 13:48:33] [13:48:33.070] Removed 9124 Vernaculars
[INFO] [2021-04-04 13:48:33] ## remove_type: Article
[INFO] [2021-04-04 13:48:33] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-04 13:48:33] [13:48:33.071] Removed 0 Articles
[INFO] [2021-04-04 13:48:33] ## remove_type: Medium
[INFO] [2021-04-04 13:48:33] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-04 13:48:33] [13:48:33.073] Removed 0 Media
[INFO] [2021-04-04 13:48:33] ## remove_type: Trait
[INFO] [2021-04-04 13:48:33] ++ Batch removal of 413020 instances...
[INFO] [2021-04-04 13:48:33] [13:48:33.143] Batch 0...
[INFO] [2021-04-04 13:48:35] [13:48:35.755] Batch 1...
[INFO] [2021-04-04 13:48:38] [13:48:38.911] Batch 2...
[INFO] [2021-04-04 13:48:41] [13:48:41.829] Batch 3...
[INFO] [2021-04-04 13:48:44] [13:48:44.730] Batch 4...
[INFO] [2021-04-04 13:48:47] [13:48:47.117] Batch 5...
[INFO] [2021-04-04 13:48:49] [13:48:49.624] Batch 6...
[INFO] [2021-04-04 13:48:53] [13:48:53.555] Batch 7...
[INFO] [2021-04-04 13:48:58] [13:48:58.030] Batch 8...
[INFO] [2021-04-04 13:49:03] [13:49:03.080] Batch 9...
[INFO] [2021-04-04 13:49:07] [13:49:07.653] Batch 10...
[INFO] [2021-04-04 13:49:11] [13:49:11.098] Batch 11...
[INFO] [2021-04-04 13:49:13] [13:49:13.768] Batch 12...
[INFO] [2021-04-04 13:49:17] [13:49:17.757] Batch 13...
[INFO] [2021-04-04 13:49:22] [13:49:22.210] Batch 14...
[INFO] [2021-04-04 13:49:27] [13:49:27.041] Batch 15...
[INFO] [2021-04-04 13:49:31] [13:49:31.822] Batch 16...
[INFO] [2021-04-04 13:49:36] [13:49:36.750] Batch 17...
[INFO] [2021-04-04 13:49:39] [13:49:39.873] Batch 18...
[INFO] [2021-04-04 13:49:42] [13:49:42.853] Batch 19...
[INFO] [2021-04-04 13:49:46] [13:49:46.703] Batch 20...
[INFO] [2021-04-04 13:49:49] [13:49:49.731] Batch 21...
[INFO] [2021-04-04 13:49:57] [13:49:57.299] Batch 22...
[INFO] [2021-04-04 13:50:05] [13:50:05.060] Batch 23...
[INFO] [2021-04-04 13:50:14] [13:50:14.572] Batch 24...
[INFO] [2021-04-04 13:50:20] [13:50:20.350] Batch 25...
[INFO] [2021-04-04 13:50:26] [13:50:26.215] Batch 26...
[INFO] [2021-04-04 13:50:28] [13:50:28.307] Batch 27...
[INFO] [2021-04-04 13:50:30] [13:50:30.389] Batch 28...
[INFO] [2021-04-04 13:50:32] [13:50:32.087] Batch 29...
[INFO] [2021-04-04 13:50:34] [13:50:34.279] Batch 30...
[INFO] [2021-04-04 13:50:36] [13:50:36.809] Batch 31...
[INFO] [2021-04-04 13:50:38] [13:50:38.660] Batch 32...
[INFO] [2021-04-04 13:50:40] [13:50:40.840] Batch 33...
[INFO] [2021-04-04 13:50:42] [13:50:42.435] Batch 34...
[INFO] [2021-04-04 13:50:44] [13:50:44.984] Batch 35...
[INFO] [2021-04-04 13:50:46] [13:50:46.651] Batch 36...
[INFO] [2021-04-04 13:50:49] [13:50:49.165] Batch 37...
[INFO] [2021-04-04 13:50:51] [13:50:51.026] Batch 38...
[INFO] [2021-04-04 13:50:52] [13:50:52.782] Batch 39...
[INFO] [2021-04-04 13:50:54] [13:50:54.601] Batch 40...
[INFO] [2021-04-04 13:50:56] [13:50:56.313] Batch 41...
[INFO] [2021-04-04 13:50:57] [13:50:57.392] Removed 413020 Traits
[INFO] [2021-04-04 13:50:57] ## remove_type: MetaTrait
[INFO] [2021-04-04 13:50:57] ++ Batch removal of 826040 instances...
[INFO] [2021-04-04 13:50:57] [13:50:57.906] Batch 0...
[INFO] [2021-04-04 13:50:58] [13:50:58.928] Batch 1...
[INFO] [2021-04-04 13:50:59] [13:50:59.908] Batch 2...
[INFO] [2021-04-04 13:51:00] [13:51:00.982] Batch 3...
[INFO] [2021-04-04 13:51:02] [13:51:02.077] Batch 4...
[INFO] [2021-04-04 13:51:03] [13:51:03.072] Batch 5...
[INFO] [2021-04-04 13:51:04] [13:51:04.076] Batch 6...
[INFO] [2021-04-04 13:51:05] [13:51:05.169] Batch 7...
[INFO] [2021-04-04 13:51:06] [13:51:06.157] Batch 8...
[INFO] [2021-04-04 13:51:07] [13:51:07.264] Batch 9...
[INFO] [2021-04-04 13:51:08] [13:51:08.235] Batch 10...
[INFO] [2021-04-04 13:51:09] [13:51:09.289] Batch 11...
[INFO] [2021-04-04 13:51:10] [13:51:10.372] Batch 12...
[INFO] [2021-04-04 13:51:11] [13:51:11.393] Batch 13...
[INFO] [2021-04-04 13:51:12] [13:51:12.349] Batch 14...
[INFO] [2021-04-04 13:51:13] [13:51:13.303] Batch 15...
[INFO] [2021-04-04 13:51:14] [13:51:14.262] Batch 16...
[INFO] [2021-04-04 13:51:15] [13:51:15.221] Batch 17...
[INFO] [2021-04-04 13:51:16] [13:51:16.189] Batch 18...
[INFO] [2021-04-04 13:51:17] [13:51:17.145] Batch 19...
[INFO] [2021-04-04 13:51:18] [13:51:18.105] Batch 20...
[INFO] [2021-04-04 13:51:19] [13:51:19.035] Batch 21...
[INFO] [2021-04-04 13:51:19] [13:51:19.992] Batch 22...
[INFO] [2021-04-04 13:51:20] [13:51:20.942] Batch 23...
[INFO] [2021-04-04 13:51:21] [13:51:21.929] Batch 24...
[INFO] [2021-04-04 13:51:22] [13:51:22.865] Batch 25...
[INFO] [2021-04-04 13:51:23] [13:51:23.841] Batch 26...
[INFO] [2021-04-04 13:51:24] [13:51:24.810] Batch 27...
[INFO] [2021-04-04 13:51:25] [13:51:25.782] Batch 28...
[INFO] [2021-04-04 13:51:26] [13:51:26.747] Batch 29...
[INFO] [2021-04-04 13:51:27] [13:51:27.686] Batch 30...
[INFO] [2021-04-04 13:51:28] [13:51:28.614] Batch 31...
[INFO] [2021-04-04 13:51:29] [13:51:29.600] Batch 32...
[INFO] [2021-04-04 13:51:30] [13:51:30.519] Batch 33...
[INFO] [2021-04-04 13:51:31] [13:51:31.408] Batch 34...
[INFO] [2021-04-04 13:51:32] [13:51:32.317] Batch 35...
[INFO] [2021-04-04 13:51:33] [13:51:33.259] Batch 36...
[INFO] [2021-04-04 13:51:34] [13:51:34.179] Batch 37...
[INFO] [2021-04-04 13:51:35] [13:51:35.083] Batch 38...
[INFO] [2021-04-04 13:51:36] [13:51:36.038] Batch 39...
[INFO] [2021-04-04 13:51:37] [13:51:37.006] Batch 40...
[INFO] [2021-04-04 13:51:37] [13:51:37.953] Batch 41...
[INFO] [2021-04-04 13:51:38] [13:51:38.893] Batch 42...
[INFO] [2021-04-04 13:51:39] [13:51:39.822] Batch 43...
[INFO] [2021-04-04 13:51:40] [13:51:40.756] Batch 44...
[INFO] [2021-04-04 13:51:41] [13:51:41.675] Batch 45...
[INFO] [2021-04-04 13:51:42] [13:51:42.579] Batch 46...
[INFO] [2021-04-04 13:51:43] [13:51:43.474] Batch 47...
[INFO] [2021-04-04 13:51:44] [13:51:44.430] Batch 48...
[INFO] [2021-04-04 13:51:45] [13:51:45.311] Batch 49...
[INFO] [2021-04-04 13:51:46] [13:51:46.212] Batch 50...
[INFO] [2021-04-04 13:51:47] [13:51:47.114] Batch 51...
[INFO] [2021-04-04 13:51:48] [13:51:48.034] Batch 52...
[INFO] [2021-04-04 13:51:48] [13:51:48.952] Batch 53...
[INFO] [2021-04-04 13:51:49] [13:51:49.918] Batch 54...
[INFO] [2021-04-04 13:51:50] [13:51:50.849] Batch 55...
[INFO] [2021-04-04 13:51:51] [13:51:51.763] Batch 56...
[INFO] [2021-04-04 13:51:52] [13:51:52.662] Batch 57...
[INFO] [2021-04-04 13:51:53] [13:51:53.551] Batch 58...
[INFO] [2021-04-04 13:51:54] [13:51:54.445] Batch 59...
[INFO] [2021-04-04 13:51:55] [13:51:55.321] Batch 60...
[INFO] [2021-04-04 13:51:56] [13:51:56.204] Batch 61...
[INFO] [2021-04-04 13:51:57] [13:51:57.086] Batch 62...
[INFO] [2021-04-04 13:51:57] [13:51:57.981] Batch 63...
[INFO] [2021-04-04 13:51:58] [13:51:58.860] Batch 64...
[INFO] [2021-04-04 13:51:59] [13:51:59.727] Batch 65...
[INFO] [2021-04-04 13:52:00] [13:52:00.588] Batch 66...
[INFO] [2021-04-04 13:52:01] [13:52:01.470] Batch 67...
[INFO] [2021-04-04 13:52:02] [13:52:02.340] Batch 68...
[INFO] [2021-04-04 13:52:03] [13:52:03.223] Batch 69...
[INFO] [2021-04-04 13:52:04] [13:52:04.117] Batch 70...
[INFO] [2021-04-04 13:52:04] [13:52:04.995] Batch 71...
[INFO] [2021-04-04 13:52:05] [13:52:05.896] Batch 72...
[INFO] [2021-04-04 13:52:06] [13:52:06.813] Batch 73...
[INFO] [2021-04-04 13:52:07] [13:52:07.689] Batch 74...
[INFO] [2021-04-04 13:52:08] [13:52:08.540] Batch 75...
[INFO] [2021-04-04 13:52:09] [13:52:09.401] Batch 76...
[INFO] [2021-04-04 13:52:10] [13:52:10.267] Batch 77...
[INFO] [2021-04-04 13:52:11] [13:52:11.118] Batch 78...
[INFO] [2021-04-04 13:52:11] [13:52:11.967] Batch 79...
[INFO] [2021-04-04 13:52:12] [13:52:12.814] Batch 80...
[INFO] [2021-04-04 13:52:13] [13:52:13.649] Batch 81...
[INFO] [2021-04-04 13:52:14] [13:52:14.514] Batch 82...
[INFO] [2021-04-04 13:52:15] [13:52:15.224] Removed 826040 Metatraits
[INFO] [2021-04-04 13:52:15] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-04 13:52:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-04 13:52:15] [13:52:15.226] Removed 0 Occurrencemetadata
[INFO] [2021-04-04 13:52:15] ## remove_type: Assoc
[INFO] [2021-04-04 13:52:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-04 13:52:15] [13:52:15.227] Removed 0 Assocs
[INFO] [2021-04-04 13:52:15] ## remove_type: MetaAssoc
[INFO] [2021-04-04 13:52:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-04 13:52:15] [13:52:15.229] Removed 0 Metaassocs
[INFO] [2021-04-04 13:52:15] ## remove_type: Identifier
[INFO] [2021-04-04 13:52:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-04 13:52:15] [13:52:15.230] Removed 0 Identifiers
[INFO] [2021-04-04 13:52:15] ## remove_type: Reference
[INFO] [2021-04-04 13:52:15] ++ Calling delete_all on 55234 instances...
[INFO] [2021-04-04 13:52:16] [13:52:16.506] Removed 55234 References
[INFO] [2021-04-04 13:52:17] Starting batch with ID 87602058...
[INFO] [2021-04-04 13:52:17] Starting batch with ID 87602058...
[INFO] [2021-04-04 13:52:18] Starting batch with ID 87572511...
[INFO] [2021-04-04 13:52:18] Starting batch with ID 87562057...
[INFO] [2021-04-04 13:52:19] Starting batch with ID 87562057...
[INFO] [2021-04-04 13:52:20] Starting batch with ID 87622348...
[INFO] [2021-04-04 13:52:20] Starting batch with ID 87622348...
[INFO] [2021-04-04 13:52:21] Starting batch with ID 87587671...
[INFO] [2021-04-04 13:52:21] Starting batch with ID 87620995...
[INFO] [2021-04-04 13:52:22] Starting batch with ID 87620995...
[INFO] [2021-04-04 13:52:22] Starting batch with ID 87561756...
[INFO] [2021-04-04 13:52:23] Starting batch with ID 87561756...
[INFO] [2021-04-04 13:52:24] Starting batch with ID 87588642...
[INFO] [2021-04-04 13:52:24] Starting batch with ID 87588642...
[INFO] [2021-04-04 13:52:25] Starting batch with ID 87612219...
[INFO] [2021-04-04 13:52:25] Starting batch with ID 87555760...
[INFO] [2021-04-04 13:52:26] Starting batch with ID 87555760...
[INFO] [2021-04-04 13:52:26] Starting batch with ID 87601700...
[INFO] [2021-04-04 13:52:27] Starting batch with ID 87601700...
[INFO] [2021-04-04 13:52:28] Starting batch with ID 87604805...
[INFO] [2021-04-04 13:52:28] Starting batch with ID 87604805...
[INFO] [2021-04-04 13:52:29] Starting batch with ID 87586651...
[INFO] [2021-04-04 13:52:29] Starting batch with ID 87586651...
[INFO] [2021-04-04 13:52:30] Starting batch with ID 87601533...
[INFO] [2021-04-04 13:52:30] Starting batch with ID 87570560...
[INFO] [2021-04-04 13:52:30] Starting batch with ID 87570560...
[INFO] [2021-04-04 13:52:30] Starting batch with ID 87570560...
[INFO] [2021-04-04 13:52:31] Starting batch with ID 87570560...
[INFO] [2021-04-04 13:52:31] Starting batch with ID 87570560...
[INFO] [2021-04-04 13:52:31] Starting batch with ID 87570560...
[INFO] [2021-04-04 13:52:31] ## remove_type: Node
[INFO] [2021-04-04 13:52:31] ++ Calling delete_all on 70159 instances...
[INFO] [2021-04-04 13:52:35] [13:52:35.998] Removed 70159 Nodes
[START] [2021-04-04 13:53:22] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-04 13:53:22] Creating resource from OpenData
[START] [2021-04-04 13:53:29] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-04 13:53:29] Parse meta.xml file and create formats with fields
[WARN] [2021-04-04 13:53:45] (common) IGNORED  (nodes) field header: scientificNameAuthorship term: http://rs.tdwg.org/dwc/terms/scientificNameAuthorship
[WARN] [2021-04-04 13:53:45] (common) IGNORED  (nodes) field header: modified term: http://purl.org/dc/terms/modified
[STOP] [2021-04-04 13:53:45] Parse meta.xml file and create formats with fields
[STOP] [2021-04-04 13:53:45] Creating resource from OpenData
[START] [2021-04-04 13:53:46] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-04 13:53:46] create_harvest_instance
[INFO] [2021-04-04 13:53:46] Created harvest instance #3638
[STOP] [2021-04-04 13:53:46] create_harvest_instance
[START] [2021-04-04 13:53:46] fetch_files
[STOP] [2021-04-04 13:53:46] fetch_files
[START] [2021-04-04 13:53:46] validate_each_file
[INFO] [2021-04-04 13:53:46] Looping over 5 formats...
[INFO] [2021-04-04 13:53:46] ...refs (/app/public/data/brazilian_flora_/reference.tab)
[INFO] [2021-04-04 13:53:46] Valid: /app/public/converted_csv/brazilian_flora__refs_3638.csv (55234 lines)
[INFO] [2021-04-04 13:53:46] ...nodes (/app/public/data/brazilian_flora_/taxon.tab)
[INFO] [2021-04-04 13:53:57] Valid: /app/public/converted_csv/brazilian_flora__nodes_3638.csv (122658 lines)
[INFO] [2021-04-04 13:53:57] ...vernaculars (/app/public/data/brazilian_flora_/vernacular_name.tab)
[INFO] [2021-04-04 13:53:57] Valid: /app/public/converted_csv/brazilian_flora__vernaculars_3638.csv (9124 lines)
[INFO] [2021-04-04 13:53:57] ...occurrences (/app/public/data/brazilian_flora_/occurrence_specific.tab)
[INFO] [2021-04-04 13:54:02] Valid: /app/public/converted_csv/brazilian_flora__occurrences_3638.csv (404891 lines)
[INFO] [2021-04-04 13:54:02] ...measurements (/app/public/data/brazilian_flora_/measurement_or_fact_specific.tab)
[INFO] [2021-04-04 13:54:18] Valid: /app/public/converted_csv/brazilian_flora__measurements_3638.csv (413020 lines)
[STOP] [2021-04-04 13:54:18] validate_each_file
[START] [2021-04-04 13:54:18] convert_to_csv
[INFO] [2021-04-04 13:54:18] Looping over 5 formats...
[INFO] [2021-04-04 13:54:18] ...refs (/app/public/data/brazilian_flora_/reference.tab)
[CMD] [2021-04-04 13:54:18] /usr/bin/sort /app/public/converted_csv/brazilian_flora__refs_3638.csv > /app/public/converted_csv/brazilian_flora__refs_3638.csv_sorted
[INFO] [2021-04-04 13:54:19] Converted: /app/public/converted_csv/brazilian_flora__refs_3638.csv (55234 lines)
[INFO] [2021-04-04 13:54:19] ...nodes (/app/public/data/brazilian_flora_/taxon.tab)
[CMD] [2021-04-04 13:54:19] /usr/bin/sort /app/public/converted_csv/brazilian_flora__nodes_3638.csv > /app/public/converted_csv/brazilian_flora__nodes_3638.csv_sorted
[INFO] [2021-04-04 13:54:21] Converted: /app/public/converted_csv/brazilian_flora__nodes_3638.csv (122658 lines)
[INFO] [2021-04-04 13:54:21] ...vernaculars (/app/public/data/brazilian_flora_/vernacular_name.tab)
[CMD] [2021-04-04 13:54:21] /usr/bin/sort /app/public/converted_csv/brazilian_flora__vernaculars_3638.csv > /app/public/converted_csv/brazilian_flora__vernaculars_3638.csv_sorted
[INFO] [2021-04-04 13:54:22] Converted: /app/public/converted_csv/brazilian_flora__vernaculars_3638.csv (9124 lines)
[INFO] [2021-04-04 13:54:22] ...occurrences (/app/public/data/brazilian_flora_/occurrence_specific.tab)
[CMD] [2021-04-04 13:54:22] /usr/bin/sort /app/public/converted_csv/brazilian_flora__occurrences_3638.csv > /app/public/converted_csv/brazilian_flora__occurrences_3638.csv_sorted
[INFO] [2021-04-04 13:54:23] Converted: /app/public/converted_csv/brazilian_flora__occurrences_3638.csv (404891 lines)
[INFO] [2021-04-04 13:54:23] ...measurements (/app/public/data/brazilian_flora_/measurement_or_fact_specific.tab)
[CMD] [2021-04-04 13:54:23] /usr/bin/sort /app/public/converted_csv/brazilian_flora__measurements_3638.csv > /app/public/converted_csv/brazilian_flora__measurements_3638.csv_sorted
[INFO] [2021-04-04 13:54:25] Converted: /app/public/converted_csv/brazilian_flora__measurements_3638.csv (413020 lines)
[STOP] [2021-04-04 13:54:25] convert_to_csv
[START] [2021-04-04 13:54:25] calculate_delta
[INFO] [2021-04-04 13:54:25] Looping over 5 formats...
[INFO] [2021-04-04 13:54:25] ...refs (/app/public/data/brazilian_flora_/reference.tab)
[CMD] [2021-04-04 13:54:25] echo "0a" > /app/public/diff/brazilian_flora__refs_3638.diff
[CMD] [2021-04-04 13:54:26] tail -n +1 /app/public/converted_csv/brazilian_flora__refs_3638.csv >> /app/public/diff/brazilian_flora__refs_3638.diff
[CMD] [2021-04-04 13:54:27] echo "." >> /app/public/diff/brazilian_flora__refs_3638.diff
[INFO] [2021-04-04 13:54:28] Created diff: /app/public/diff/brazilian_flora__refs_3638.diff (55236 lines)
[INFO] [2021-04-04 13:54:28] ...nodes (/app/public/data/brazilian_flora_/taxon.tab)
[CMD] [2021-04-04 13:54:28] echo "0a" > /app/public/diff/brazilian_flora__nodes_3638.diff
[CMD] [2021-04-04 13:54:29] tail -n +1 /app/public/converted_csv/brazilian_flora__nodes_3638.csv >> /app/public/diff/brazilian_flora__nodes_3638.diff
[CMD] [2021-04-04 13:54:30] echo "." >> /app/public/diff/brazilian_flora__nodes_3638.diff
[INFO] [2021-04-04 13:54:31] Created diff: /app/public/diff/brazilian_flora__nodes_3638.diff (122660 lines)
[INFO] [2021-04-04 13:54:31] ...vernaculars (/app/public/data/brazilian_flora_/vernacular_name.tab)
[CMD] [2021-04-04 13:54:31] echo "0a" > /app/public/diff/brazilian_flora__vernaculars_3638.diff
[CMD] [2021-04-04 13:54:32] tail -n +1 /app/public/converted_csv/brazilian_flora__vernaculars_3638.csv >> /app/public/diff/brazilian_flora__vernaculars_3638.diff
[CMD] [2021-04-04 13:54:33] echo "." >> /app/public/diff/brazilian_flora__vernaculars_3638.diff
[INFO] [2021-04-04 13:54:34] Created diff: /app/public/diff/brazilian_flora__vernaculars_3638.diff (9126 lines)
[INFO] [2021-04-04 13:54:34] ...occurrences (/app/public/data/brazilian_flora_/occurrence_specific.tab)
[CMD] [2021-04-04 13:54:34] echo "0a" > /app/public/diff/brazilian_flora__occurrences_3638.diff
[CMD] [2021-04-04 13:54:35] tail -n +1 /app/public/converted_csv/brazilian_flora__occurrences_3638.csv >> /app/public/diff/brazilian_flora__occurrences_3638.diff
[CMD] [2021-04-04 13:54:36] echo "." >> /app/public/diff/brazilian_flora__occurrences_3638.diff
[INFO] [2021-04-04 13:54:38] Created diff: /app/public/diff/brazilian_flora__occurrences_3638.diff (404893 lines)
[INFO] [2021-04-04 13:54:38] ...measurements (/app/public/data/brazilian_flora_/measurement_or_fact_specific.tab)
[CMD] [2021-04-04 13:54:38] echo "0a" > /app/public/diff/brazilian_flora__measurements_3638.diff
[CMD] [2021-04-04 13:54:39] tail -n +1 /app/public/converted_csv/brazilian_flora__measurements_3638.csv >> /app/public/diff/brazilian_flora__measurements_3638.diff
[CMD] [2021-04-04 13:54:40] echo "." >> /app/public/diff/brazilian_flora__measurements_3638.diff
[INFO] [2021-04-04 13:54:41] Created diff: /app/public/diff/brazilian_flora__measurements_3638.diff (413022 lines)
[STOP] [2021-04-04 13:54:41] calculate_delta
[START] [2021-04-04 13:54:41] parse_diff_and_store
[INFO] [2021-04-04 13:54:41] Handling diff: /app/public/diff/brazilian_flora__refs_3638.diff (55236 lines)
[INFO] [2021-04-04 13:54:42] Loading refs diff file into memory (55236 /app/public/diff/brazilian_flora__refs_3638.diff lines)...
[INFO] [2021-04-04 13:54:52] Handling diff: /app/public/diff/brazilian_flora__nodes_3638.diff (122660 lines)
[INFO] [2021-04-04 13:54:53] Loading nodes diff file into memory (122660 /app/public/diff/brazilian_flora__nodes_3638.diff lines)...
[WARN] [2021-04-04 13:55:02] Filtered Scientific Name `Peridinium gatunense var. madagascariensis  Lefèvre` to `Peridinium gatunense var. madagascariensis Lefèvre`
[WARN] [2021-04-04 13:55:08] Filtered Scientific Name `Glomus pachycaule  (C.G. Wu & Z.C. Chen) Sieverd. & Oehl` to `Glomus pachycaule (C.G. Wu & Z.C. Chen) Sieverd. & Oehl`
[WARN] [2021-04-04 13:55:13] Filtered Scientific Name `Poraqueiba  Aubl.` to `Poraqueiba Aubl.`
[WARN] [2021-04-04 13:55:20] Filtered Scientific Name `Leopoldia  Herb.` to `Leopoldia Herb.`
[WARN] [2021-04-04 13:55:24] Filtered Scientific Name `Lejeunea ulicina (Taylor) Gottsche  et al.` to `Lejeunea ulicina (Taylor) Gottsche et al.`
[WARN] [2021-04-04 13:55:24] Filtered Scientific Name `Psilotum triquetrum  Sw.` to `Psilotum triquetrum Sw.`
[WARN] [2021-04-04 13:55:24] Filtered Scientific Name `Perichaena calongei  Lado, D. Wrigley & Estrada` to `Perichaena calongei Lado, D. Wrigley & Estrada`
[WARN] [2021-04-04 13:55:24] Filtered Scientific Name `Oncidium clovesianum  (V.P.Castro & M.M.L.L.Cardoso) J.M.H.Shaw` to `Oncidium clovesianum (V.P.Castro & M.M.L.L.Cardoso) J.M.H.Shaw`
[WARN] [2021-04-04 13:55:25] Filtered Scientific Name `Craspedaria  Link` to `Craspedaria Link`
[WARN] [2021-04-04 13:55:25] Filtered Scientific Name `Neolacis  Wedd.` to `Neolacis Wedd.`
[WARN] [2021-04-04 13:55:25] Filtered Scientific Name `Croton verbenaefolius  Müll.Arg.` to `Croton verbenaefolius Müll.Arg.`
[WARN] [2021-04-04 13:55:25] Filtered Scientific Name `Xylosma ciliatifolium  (Clos) Eichler` to `Xylosma ciliatifolium (Clos) Eichler`
[WARN] [2021-04-04 13:55:25] Filtered Scientific Name `Physalis linkiana  Nees` to `Physalis linkiana Nees`
[WARN] [2021-04-04 13:55:25] Filtered Scientific Name `Guatteria laevigata  Mart.` to `Guatteria laevigata Mart.`
[WARN] [2021-04-04 13:55:25] Filtered Scientific Name `Lejeunea maxonii (A. Evans)  X.-L. He` to `Lejeunea maxonii (A. Evans) X.-L. He`
[WARN] [2021-04-04 13:55:25] Filtered Scientific Name `Coccoloba fastigata  Meisn.` to `Coccoloba fastigata Meisn.`
[WARN] [2021-04-04 13:55:25] Filtered Scientific Name `Colliguaya brasiliensis  Klotzsch ex Baill.` to `Colliguaya brasiliensis Klotzsch ex Baill.`
[WARN] [2021-04-04 13:55:25] Filtered Scientific Name `Polypodium pendulum  Sw.` to `Polypodium pendulum Sw.`
[WARN] [2021-04-04 13:55:25] Filtered Scientific Name `Miconia saldanhaei  Cogn.` to `Miconia saldanhaei Cogn.`
[WARN] [2021-04-04 13:55:25] Filtered Scientific Name `Coccoloba martii  Meisn.` to `Coccoloba martii Meisn.`
[WARN] [2021-04-04 13:55:25] Filtered Scientific Name `Trembleya selloana  Cogn.` to `Trembleya selloana Cogn.`
[WARN] [2021-04-04 13:55:25] Filtered Scientific Name `Croton muscicapa  Müll.Arg.` to `Croton muscicapa Müll.Arg.`
[WARN] [2021-04-04 13:55:25] Filtered Scientific Name `Coccoloba sticticaulis  Wedd.` to `Coccoloba sticticaulis Wedd.`
[WARN] [2021-04-04 13:55:25] Filtered Scientific Name `Coccoloba polystachya  Wedd.` to `Coccoloba polystachya Wedd.`
[WARN] [2021-04-04 13:55:25] Filtered Scientific Name `Xylosma tweedianum  (Clos) Eichler` to `Xylosma tweedianum (Clos) Eichler`
[WARN] [2021-04-04 13:55:25] Filtered Scientific Name `Coccoloba crescentiaefolia  Meisn.` to `Coccoloba crescentiaefolia Meisn.`
[WARN] [2021-04-04 13:55:25] Filtered Scientific Name `Celtis aculeata  Sw.` to `Celtis aculeata Sw.`
[WARN] [2021-04-04 13:55:25] Filtered Scientific Name `Polypodium diversifolium  Sw.` to `Polypodium diversifolium Sw.`
[WARN] [2021-04-04 13:55:25] Filtered Scientific Name `Croton rufo-argenteus  Müll.Arg.` to `Croton rufo-argenteus Müll.Arg.`
[WARN] [2021-04-04 13:55:25] (Reached filtered-name limit; supressing further warnings.)
[INFO] [2021-04-04 13:55:39] Handling diff: /app/public/diff/brazilian_flora__vernaculars_3638.diff (9126 lines)
[INFO] [2021-04-04 13:55:40] Loading vernaculars diff file into memory (9126 /app/public/diff/brazilian_flora__vernaculars_3638.diff lines)...
[INFO] [2021-04-04 13:55:43] Handling diff: /app/public/diff/brazilian_flora__occurrences_3638.diff (404893 lines)
[INFO] [2021-04-04 13:55:44] Loading occurrences diff file into memory (404893 /app/public/diff/brazilian_flora__occurrences_3638.diff lines)...
[INFO] [2021-04-04 13:56:36] Handling diff: /app/public/diff/brazilian_flora__measurements_3638.diff (413022 lines)
[INFO] [2021-04-04 13:56:37] Loading measurements diff file into memory (413022 /app/public/diff/brazilian_flora__measurements_3638.diff lines)...
[INFO] [2021-04-04 14:03:03] Storing 55234 References
[INFO] [2021-04-04 14:03:03] Processing group of 55234 in 56 groups of 1000
[INFO] [2021-04-04 14:03:23] Average Time: 0.352
[INFO] [2021-04-04 14:03:23] Total Time: 20s
[INFO] [2021-04-04 14:03:23] last 3 / first 3: 0.82
[INFO] [2021-04-04 14:03:23] Std.Dev: 0.4979959839195493; Max: 2.1
[INFO] [2021-04-04 14:03:23] Storing 133626 ScientificNames
[INFO] [2021-04-04 14:03:23] Processing group of 133626 in 134 groups of 1000
[INFO] [2021-04-04 14:04:13] Average Time: 0.365
[INFO] [2021-04-04 14:04:13] Total Time: 50s
[INFO] [2021-04-04 14:04:13] last 3 / first 3: 0.89
[INFO] [2021-04-04 14:04:13] Std.Dev: 0.8221921916437787; Max: 9.79
[INFO] [2021-04-04 14:04:13] Storing 70159 Nodes
[INFO] [2021-04-04 14:04:13] Processing group of 70159 in 71 groups of 1000
[INFO] [2021-04-04 14:04:33] Average Time: 0.277
[INFO] [2021-04-04 14:04:33] Total Time: 21s
[INFO] [2021-04-04 14:04:33] last 3 / first 3: 0.77
[INFO] [2021-04-04 14:04:33] Std.Dev: 0.07071067811865475; Max: 0.44
[INFO] [2021-04-04 14:04:33] Storing 40881 NodesReferences
[INFO] [2021-04-04 14:04:33] Processing group of 40881 in 41 groups of 1000
[INFO] [2021-04-04 14:04:36] Average Time: 0.069
[INFO] [2021-04-04 14:04:36] Total Time: 4s
[INFO] [2021-04-04 14:04:36] last 3 / first 3: 0.68
[INFO] [2021-04-04 14:04:36] Std.Dev: 0.03162277660168379; Max: 0.25
[INFO] [2021-04-04 14:04:36] Storing 9124 Vernaculars
[INFO] [2021-04-04 14:04:36] Processing group of 9124 in 10 groups of 1000
[INFO] [2021-04-04 14:04:38] Average Time: 0.234
[INFO] [2021-04-04 14:04:38] Total Time: 3s
[INFO] [2021-04-04 14:04:38] last 3 / first 3: 2.36
[INFO] [2021-04-04 14:04:38] Std.Dev: 0.22583179581272428; Max: 0.86
[INFO] [2021-04-04 14:04:38] Storing 404891 Occurrences
[INFO] [2021-04-04 14:04:38] Processing group of 404891 in 405 groups of 1000
[INFO] [2021-04-04 14:06:08] Average Time: 0.215
[INFO] [2021-04-04 14:06:08] Total Time: 1m30s
[INFO] [2021-04-04 14:06:08] last 3 / first 3: 0.33
[INFO] [2021-04-04 14:06:08] Std.Dev: 0.7224956747275377; Max: 10.07
[INFO] [2021-04-04 14:06:08] Storing 413020 Traits
[INFO] [2021-04-04 14:06:08] Processing group of 413020 in 414 groups of 1000
[INFO] [2021-04-04 14:08:49] Average Time: 0.385
[INFO] [2021-04-04 14:08:49] Total Time: 2m42s
[INFO] [2021-04-04 14:08:49] last 3 / first 3: 0.66
[INFO] [2021-04-04 14:08:49] Std.Dev: 0.4878524367060187; Max: 4.17
[INFO] [2021-04-04 14:08:49] Storing 826040 MetaTraits
[INFO] [2021-04-04 14:08:49] Processing group of 826040 in 827 groups of 1000
[INFO] [2021-04-04 14:11:05] Average Time: 0.158
[INFO] [2021-04-04 14:11:05] Total Time: 2m16s
[INFO] [2021-04-04 14:11:05] last 3 / first 3: 0.45
[INFO] [2021-04-04 14:11:05] Std.Dev: 0.24289915602982237; Max: 2.61
[STOP] [2021-04-04 14:11:05] parse_diff_and_store
[START] [2021-04-04 14:11:05] resolve_keys
[INFO] [2021-04-04 14:11:54] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-04 14:12:08] traits to occurrences...
[INFO] [2021-04-04 14:12:24] traits to nodes (through occurrences)...
[INFO] [2021-04-04 14:12:32] Traits to sex term...
[INFO] [2021-04-04 14:12:41] Traits to lifestage term...
[INFO] [2021-04-04 14:12:47] MetaTraits to traits...
[INFO] [2021-04-04 14:13:06] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-04 14:13:07] Assocs to occurrences...
[INFO] [2021-04-04 14:13:07] Assocs to nodes...
[INFO] [2021-04-04 14:13:07] Assoc to sex term...
[INFO] [2021-04-04 14:13:07] Assoc to lifestage term...
[INFO] [2021-04-04 14:13:07] MetaAssoc to assocs...
[STOP] [2021-04-04 14:13:07] resolve_keys
[START] [2021-04-04 14:13:07] hold_for_later_1
[STOP] [2021-04-04 14:13:07] hold_for_later_1
[START] [2021-04-04 14:13:07] hold_for_later_2
[STOP] [2021-04-04 14:13:07] hold_for_later_2
[START] [2021-04-04 14:13:07] resolve_missing_parents
[STOP] [2021-04-04 14:13:09] resolve_missing_parents
[START] [2021-04-04 14:13:09] rebuild_nodes
[START] [2021-04-04 14:13:09] Flattener#flatten
[START] [2021-04-04 14:13:09] Flattener#study_resource
[START] [2021-04-04 14:13:10] Flattener#build_ancestry
[STOP] [2021-04-04 14:14:07] Flattener#build_ancestry
[INFO] [2021-04-04 14:14:07] 70159 ancestry keys
[START] [2021-04-04 14:14:07] build_node_ancestors
[INFO] [2021-04-04 14:14:07] old ancestors deleted.
[STOP] [2021-04-04 14:14:42] build_node_ancestors
[START] [2021-04-04 14:14:42] Flattener#propagate_ancestor_ids
[STOP] [2021-04-04 14:14:47] Flattener#propagate_ancestor_ids
[STOP] [2021-04-04 14:14:47] Flattener#flatten
[STOP] [2021-04-04 14:14:47] rebuild_nodes
[START] [2021-04-04 14:14:47] resolve_missing_media_owners
[STOP] [2021-04-04 14:14:47] resolve_missing_media_owners
[START] [2021-04-04 14:14:47] sanitize_media_verbatims
[STOP] [2021-04-04 14:14:47] sanitize_media_verbatims
[START] [2021-04-04 14:14:47] queue_downloads
[STOP] [2021-04-04 14:14:47] queue_downloads
[START] [2021-04-04 14:14:47] parse_names
[WARN] [2021-04-04 14:14:47] I see 133626 names which still need to be parsed.
[WARN] [2021-04-04 14:16:23] I see 80 names which still need to be parsed.
[WARN] [2021-04-04 14:16:24] I see 11 names which still need to be parsed.
[WARN] [2021-04-04 14:16:26] I see 7 names which still need to be parsed.
[WARN] [2021-04-04 14:16:27] I see 6 names which still need to be parsed.
[WARN] [2021-04-04 14:16:28] I see 5 names which still need to be parsed.
[WARN] [2021-04-04 14:16:30] I see 4 names which still need to be parsed.
[WARN] [2021-04-04 14:16:31] I see 3 names which still need to be parsed.
[WARN] [2021-04-04 14:16:33] I see 2 names which still need to be parsed.
[WARN] [2021-04-04 14:16:34] I see 1 names which still need to be parsed.
[STOP] [2021-04-04 14:16:36] parse_names
[START] [2021-04-04 14:16:36] denormalize_canonical_names_to_nodes
[STOP] [2021-04-04 14:16:37] denormalize_canonical_names_to_nodes
[START] [2021-04-04 14:16:37] match_nodes
[START] [2021-04-04 14:16:38] map_all_nodes_to_pages
[STOP] [2021-04-04 14:31:54] map_all_nodes_to_pages
[INFO] [2021-04-04 14:31:54] 8227 Unmatched nodes (of 70159)! That's too many to output. Full list in /app/public/data/brazilian_flora_/unmatched_nodes.txt ; First 10: Mariscus (#91717369); Mariscus alopecuroides (#91761755); Rhynchospora ciliata (#91769336); Eleocharis radicans (#91722316); Cryptangium (#91769136); Scleria setuloso-ciliata (#91745258); Dichromena (#91736732); Pleurostachys (#91769277); Tetraria (#91736969); Websteria (#91745261)
[START] [2021-04-04 14:31:54] update_nodes
[STOP] [2021-04-04 14:32:25] update_nodes
[STOP] [2021-04-04 14:32:25] match_nodes
[START] [2021-04-04 14:32:25] reindex_search
[STOP] [2021-04-04 14:33:26] reindex_search
[START] [2021-04-04 14:33:26] normalize_units
[STOP] [2021-04-04 14:33:53] normalize_units
[START] [2021-04-04 14:33:53] calculate_statistics
[STOP] [2021-04-04 14:33:53] calculate_statistics
[START] [2021-04-04 14:33:53] complete_harvest_instance
[START] [2021-04-04 14:33:53] overall_tsv_creation
[INFO] [2021-04-04 14:33:53] Processing group of 70159 in 8 batches of 10000
[INFO] [2021-04-04 14:35:20] 46729 Traits (unfiltered)...
[INFO] [2021-04-04 14:40:44] 46729 Traits (filtered)...
[INFO] [2021-04-04 14:40:47] 0 Associations (filtered)...
[INFO] [2021-04-04 14:40:53] 0 metadata added.
[INFO] [2021-04-04 14:40:53] 0 metadata added.
[INFO] [2021-04-04 14:42:46] 38149 Traits (unfiltered)...
[INFO] [2021-04-04 14:47:21] 38149 Traits (filtered)...
[INFO] [2021-04-04 14:47:24] 0 Associations (filtered)...
[INFO] [2021-04-04 14:47:30] 0 metadata added.
[INFO] [2021-04-04 14:47:30] 0 metadata added.
[INFO] [2021-04-04 14:49:25] 54930 Traits (unfiltered)...
[INFO] [2021-04-04 14:55:50] 54930 Traits (filtered)...
[INFO] [2021-04-04 14:55:53] 0 Associations (filtered)...
[INFO] [2021-04-04 14:55:58] 0 metadata added.
[INFO] [2021-04-04 14:55:58] 0 metadata added.
[INFO] [2021-04-04 14:58:05] 91514 Traits (unfiltered)...
[INFO] [2021-04-04 15:08:28] 91514 Traits (filtered)...
[INFO] [2021-04-04 15:08:31] 0 Associations (filtered)...
[INFO] [2021-04-04 15:08:40] 0 metadata added.
[INFO] [2021-04-04 15:08:40] 0 metadata added.
[INFO] [2021-04-04 15:10:50] 60210 Traits (unfiltered)...
[INFO] [2021-04-04 15:17:49] 60210 Traits (filtered)...
[INFO] [2021-04-04 15:17:52] 0 Associations (filtered)...
[INFO] [2021-04-04 15:17:57] 0 metadata added.
[INFO] [2021-04-04 15:17:57] 0 metadata added.
[INFO] [2021-04-04 15:20:07] 59533 Traits (unfiltered)...
[INFO] [2021-04-04 15:27:00] 59533 Traits (filtered)...
[INFO] [2021-04-04 15:27:03] 0 Associations (filtered)...
[INFO] [2021-04-04 15:27:08] 0 metadata added.
[INFO] [2021-04-04 15:27:08] 0 metadata added.
[INFO] [2021-04-04 15:29:14] 58778 Traits (unfiltered)...
[INFO] [2021-04-04 15:36:00] 58778 Traits (filtered)...
[INFO] [2021-04-04 15:36:03] 0 Associations (filtered)...
[INFO] [2021-04-04 15:36:08] 0 metadata added.
[INFO] [2021-04-04 15:36:08] 0 metadata added.
[INFO] [2021-04-04 15:37:28] 933 Traits (unfiltered)...
[INFO] [2021-04-04 15:38:02] 933 Traits (filtered)...
[INFO] [2021-04-04 15:38:02] 0 Associations (filtered)...
[INFO] [2021-04-04 15:38:02] 0 metadata added.
[INFO] [2021-04-04 15:38:02] 0 metadata added.
[INFO] [2021-04-04 15:38:29] Average Time: 450.854
[INFO] [2021-04-04 15:38:29] Total Time: 1h4m36s
[INFO] [2021-04-04 15:38:29] last 3 / first 3: 0.88
[INFO] [2021-04-04 15:38:29] Std.Dev: 186.73454956167058; Max: 737.13
[STOP] [2021-04-04 15:38:29] overall_tsv_creation
[INFO] [2021-04-04 15:38:29] Done. Check your files:
[INFO] [2021-04-04 15:38:30] (70136 lines) /app/public/data/brazilian_flora_/publish_nodes.tsv
[INFO] [2021-04-04 15:38:31] (206343 lines) /app/public/data/brazilian_flora_/publish_node_ancestors.tsv
[INFO] [2021-04-04 15:38:32] (116838 lines) /app/public/data/brazilian_flora_/publish_scientific_names.tsv
[INFO] [2021-04-04 15:38:33] (8548 lines) /app/public/data/brazilian_flora_/publish_vernaculars.tsv
[INFO] [2021-04-04 15:38:33] (26526 lines) /app/public/data/brazilian_flora_/publish_references.tsv
[INFO] [2021-04-04 15:38:34] (26526 lines) /app/public/data/brazilian_flora_/publish_referents.tsv
[INFO] [2021-04-04 15:38:35] (410777 lines) /app/public/data/brazilian_flora_/publish_traits.tsv
[INFO] [2021-04-04 15:38:36] (1 lines) /app/public/data/brazilian_flora_/publish_metadata.tsv
[STOP] [2021-04-04 15:38:37] complete_harvest_instance
[START] [2021-04-04 15:38:37] completed
[STOP] [2021-04-04 15:38:37] completed
[STOP] [2021-04-04 15:38:37] logged process, took 6292.13

Latest Process