Harvest for Wikidata Created 04 May 10:17

Stage: completed
Fetched: 04 May 10:17
Validated: 04 May 10:18
Deltas Created 04 May 10:18
Units Normalized: 04 May 12:37
Ancestry Built: 04 May 10:39
Nodes Matched: 04 May 12:32
Names Parsed: 04 May 10:41
New Models Stored: 04 May 10:27
Indexed: 04 May 12:37
Completed: 04 May 13:05
Time to Harvest: 3 minutes

Harvesting Log (most recent first)

# Logfile created on 2021-03-16 10:26:02 -0400 by logger.rb/v1.4.2
[INFO] [2021-03-16 10:26:02] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-03-16 10:26:05] ## remove_type: ScientificName
[INFO] [2021-03-16 10:26:05] ++ Batch removal of 386028 instances...
[INFO] [2021-03-16 10:26:05] [10:26:05.838] Batch 0...
[INFO] [2021-03-16 10:26:07] [10:26:07.200] Batch 1...
[INFO] [2021-03-16 10:26:08] [10:26:08.518] Batch 2...
[INFO] [2021-03-16 10:26:09] [10:26:09.912] Batch 3...
[INFO] [2021-03-16 10:26:11] [10:26:11.331] Batch 4...
[INFO] [2021-03-16 10:26:21] [10:26:21.876] Batch 5...
[INFO] [2021-03-16 10:26:30] [10:26:30.079] Batch 6...
[INFO] [2021-03-16 10:26:33] [10:26:33.842] Batch 7...
[INFO] [2021-03-16 10:26:35] [10:26:35.402] Batch 8...
[INFO] [2021-03-16 10:26:36] [10:26:36.953] Batch 9...
[INFO] [2021-03-16 10:26:39] [10:26:39.095] Batch 10...
[INFO] [2021-03-16 10:26:41] [10:26:41.894] Batch 11...
[INFO] [2021-03-16 10:26:48] [10:26:48.051] Batch 12...
[INFO] [2021-03-16 10:26:52] [10:26:52.788] Batch 13...
[INFO] [2021-03-16 10:26:56] [10:26:56.076] Batch 14...
[INFO] [2021-03-16 10:26:57] [10:26:57.919] Batch 15...
[INFO] [2021-03-16 10:26:59] [10:26:59.496] Batch 16...
[INFO] [2021-03-16 10:27:01] [10:27:01.725] Batch 17...
[INFO] [2021-03-16 10:27:03] [10:27:03.165] Batch 18...
[INFO] [2021-03-16 10:27:06] [10:27:06.190] Batch 19...
[INFO] [2021-03-16 10:27:07] [10:27:07.972] Batch 20...
[INFO] [2021-03-16 10:27:09] [10:27:09.268] Batch 21...
[INFO] [2021-03-16 10:27:12] [10:27:12.673] Batch 22...
[INFO] [2021-03-16 10:27:14] [10:27:14.475] Batch 23...
[INFO] [2021-03-16 10:27:17] [10:27:17.903] Batch 24...
[INFO] [2021-03-16 10:27:19] [10:27:19.706] Batch 25...
[INFO] [2021-03-16 10:27:24] [10:27:24.218] Batch 26...
[INFO] [2021-03-16 10:27:25] [10:27:25.675] Batch 27...
[INFO] [2021-03-16 10:27:29] [10:27:29.340] Batch 28...
[INFO] [2021-03-16 10:27:31] [10:27:31.203] Batch 29...
[INFO] [2021-03-16 10:27:34] [10:27:34.411] Batch 30...
[INFO] [2021-03-16 10:27:36] [10:27:36.291] Batch 31...
[INFO] [2021-03-16 10:27:37] [10:27:37.852] Batch 32...
[INFO] [2021-03-16 10:27:41] [10:27:41.317] Batch 33...
[INFO] [2021-03-16 10:27:42] [10:27:42.532] Batch 34...
[INFO] [2021-03-16 10:27:44] [10:27:44.526] Batch 35...
[INFO] [2021-03-16 10:27:48] [10:27:48.739] Batch 36...
[INFO] [2021-03-16 10:27:50] [10:27:50.129] Batch 37...
[INFO] [2021-03-16 10:27:54] [10:27:54.983] Batch 38...
[INFO] [2021-03-16 10:27:55] [10:27:55.959] Removed 386028 Scientificnames
[INFO] [2021-03-16 10:27:55] ## remove_type: Vernacular
[INFO] [2021-03-16 10:27:56] ++ Batch removal of 1926000 instances...
[INFO] [2021-03-16 10:27:56] [10:27:56.263] Batch 0...
[INFO] [2021-03-16 10:28:00] [10:28:00.122] Batch 1...
[INFO] [2021-03-16 10:28:01] [10:28:01.580] Batch 2...
[INFO] [2021-03-16 10:28:02] [10:28:02.513] Batch 3...
[INFO] [2021-03-16 10:28:05] [10:28:05.663] Batch 4...
[INFO] [2021-03-16 10:28:06] [10:28:06.686] Batch 5...
[INFO] [2021-03-16 10:28:07] [10:28:07.798] Batch 6...
[INFO] [2021-03-16 10:28:09] [10:28:09.020] Batch 7...
[INFO] [2021-03-16 10:28:09] [10:28:09.967] Batch 8...
[INFO] [2021-03-16 10:28:11] [10:28:11.019] Batch 9...
[INFO] [2021-03-16 10:28:12] [10:28:12.021] Batch 10...
[INFO] [2021-03-16 10:28:13] [10:28:13.067] Batch 11...
[INFO] [2021-03-16 10:28:14] [10:28:14.097] Batch 12...
[INFO] [2021-03-16 10:28:15] [10:28:15.854] Batch 13...
[INFO] [2021-03-16 10:28:17] [10:28:17.066] Batch 14...
[INFO] [2021-03-16 10:28:18] [10:28:18.072] Batch 15...
[INFO] [2021-03-16 10:28:19] [10:28:19.153] Batch 16...
[INFO] [2021-03-16 10:28:20] [10:28:20.217] Batch 17...
[INFO] [2021-03-16 10:28:21] [10:28:21.080] Batch 18...
[INFO] [2021-03-16 10:28:22] [10:28:22.352] Batch 19...
[INFO] [2021-03-16 10:28:23] [10:28:23.400] Batch 20...
[INFO] [2021-03-16 10:28:24] [10:28:24.276] Batch 21...
[INFO] [2021-03-16 10:28:25] [10:28:25.112] Batch 22...
[INFO] [2021-03-16 10:28:25] [10:28:25.949] Batch 23...
[INFO] [2021-03-16 10:28:26] [10:28:26.798] Batch 24...
[INFO] [2021-03-16 10:28:27] [10:28:27.640] Batch 25...
[INFO] [2021-03-16 10:28:28] [10:28:28.476] Batch 26...
[INFO] [2021-03-16 10:28:29] [10:28:29.303] Batch 27...
[INFO] [2021-03-16 10:28:30] [10:28:30.130] Batch 28...
[INFO] [2021-03-16 10:28:30] [10:28:30.962] Batch 29...
[INFO] [2021-03-16 10:28:31] [10:28:31.798] Batch 30...
[INFO] [2021-03-16 10:28:32] [10:28:32.628] Batch 31...
[INFO] [2021-03-16 10:28:33] [10:28:33.447] Batch 32...
[INFO] [2021-03-16 10:28:34] [10:28:34.264] Batch 33...
[INFO] [2021-03-16 10:28:35] [10:28:35.080] Batch 34...
[INFO] [2021-03-16 10:28:35] [10:28:35.895] Batch 35...
[INFO] [2021-03-16 10:28:36] [10:28:36.724] Batch 36...
[INFO] [2021-03-16 10:28:37] [10:28:37.536] Batch 37...
[INFO] [2021-03-16 10:28:38] [10:28:38.417] Batch 38...
[INFO] [2021-03-16 10:28:39] [10:28:39.225] Batch 39...
[INFO] [2021-03-16 10:28:40] [10:28:40.033] Batch 40...
[INFO] [2021-03-16 10:28:40] [10:28:40.842] Batch 41...
[INFO] [2021-03-16 10:28:41] [10:28:41.658] Batch 42...
[INFO] [2021-03-16 10:28:42] [10:28:42.463] Batch 43...
[INFO] [2021-03-16 10:28:43] [10:28:43.265] Batch 44...
[INFO] [2021-03-16 10:28:44] [10:28:44.080] Batch 45...
[INFO] [2021-03-16 10:28:44] [10:28:44.886] Batch 46...
[INFO] [2021-03-16 10:28:45] [10:28:45.698] Batch 47...
[INFO] [2021-03-16 10:28:46] [10:28:46.495] Batch 48...
[INFO] [2021-03-16 10:28:47] [10:28:47.316] Batch 49...
[INFO] [2021-03-16 10:28:48] [10:28:48.109] Batch 50...
[INFO] [2021-03-16 10:28:48] [10:28:48.909] Batch 51...
[INFO] [2021-03-16 10:28:49] [10:28:49.721] Batch 52...
[INFO] [2021-03-16 10:28:50] [10:28:50.511] Batch 53...
[INFO] [2021-03-16 10:28:51] [10:28:51.314] Batch 54...
[INFO] [2021-03-16 10:28:52] [10:28:52.100] Batch 55...
[INFO] [2021-03-16 10:28:52] [10:28:52.893] Batch 56...
[INFO] [2021-03-16 10:28:53] [10:28:53.691] Batch 57...
[INFO] [2021-03-16 10:28:54] [10:28:54.477] Batch 58...
[INFO] [2021-03-16 10:28:55] [10:28:55.255] Batch 59...
[INFO] [2021-03-16 10:28:56] [10:28:56.032] Batch 60...
[INFO] [2021-03-16 10:28:56] [10:28:56.826] Batch 61...
[INFO] [2021-03-16 10:28:57] [10:28:57.607] Batch 62...
[INFO] [2021-03-16 10:28:58] [10:28:58.386] Batch 63...
[INFO] [2021-03-16 10:28:59] [10:28:59.157] Batch 64...
[INFO] [2021-03-16 10:28:59] [10:28:59.929] Batch 65...
[INFO] [2021-03-16 10:29:00] [10:29:00.711] Batch 66...
[INFO] [2021-03-16 10:29:01] [10:29:01.482] Batch 67...
[INFO] [2021-03-16 10:29:02] [10:29:02.248] Batch 68...
[INFO] [2021-03-16 10:29:03] [10:29:03.015] Batch 69...
[INFO] [2021-03-16 10:29:03] [10:29:03.794] Batch 70...
[INFO] [2021-03-16 10:29:04] [10:29:04.555] Batch 71...
[INFO] [2021-03-16 10:29:05] [10:29:05.328] Batch 72...
[INFO] [2021-03-16 10:29:06] [10:29:06.083] Batch 73...
[INFO] [2021-03-16 10:29:06] [10:29:06.848] Batch 74...
[INFO] [2021-03-16 10:29:07] [10:29:07.612] Batch 75...
[INFO] [2021-03-16 10:29:08] [10:29:08.365] Batch 76...
[INFO] [2021-03-16 10:29:09] [10:29:09.115] Batch 77...
[INFO] [2021-03-16 10:29:09] [10:29:09.871] Batch 78...
[INFO] [2021-03-16 10:29:10] [10:29:10.625] Batch 79...
[INFO] [2021-03-16 10:29:11] [10:29:11.375] Batch 80...
[INFO] [2021-03-16 10:29:12] [10:29:12.120] Batch 81...
[INFO] [2021-03-16 10:29:12] [10:29:12.875] Batch 82...
[INFO] [2021-03-16 10:29:13] [10:29:13.623] Batch 83...
[INFO] [2021-03-16 10:29:14] [10:29:14.371] Batch 84...
[INFO] [2021-03-16 10:29:15] [10:29:15.117] Batch 85...
[INFO] [2021-03-16 10:29:15] [10:29:15.865] Batch 86...
[INFO] [2021-03-16 10:29:16] [10:29:16.606] Batch 87...
[INFO] [2021-03-16 10:29:17] [10:29:17.347] Batch 88...
[INFO] [2021-03-16 10:29:18] [10:29:18.080] Batch 89...
[INFO] [2021-03-16 10:29:18] [10:29:18.822] Batch 90...
[INFO] [2021-03-16 10:29:19] [10:29:19.550] Batch 91...
[INFO] [2021-03-16 10:29:20] [10:29:20.293] Batch 92...
[INFO] [2021-03-16 10:29:21] [10:29:21.019] Batch 93...
[INFO] [2021-03-16 10:29:21] [10:29:21.751] Batch 94...
[INFO] [2021-03-16 10:29:22] [10:29:22.475] Batch 95...
[INFO] [2021-03-16 10:29:23] [10:29:23.195] Batch 96...
[INFO] [2021-03-16 10:29:23] [10:29:23.920] Batch 97...
[INFO] [2021-03-16 10:29:24] [10:29:24.647] Batch 98...
[INFO] [2021-03-16 10:29:25] [10:29:25.367] Batch 99...
[INFO] [2021-03-16 10:29:26] [10:29:26.082] Batch 100...
[INFO] [2021-03-16 10:29:26] [10:29:26.805] Batch 101...
[INFO] [2021-03-16 10:29:27] [10:29:27.519] Batch 102...
[INFO] [2021-03-16 10:29:28] [10:29:28.244] Batch 103...
[INFO] [2021-03-16 10:29:28] [10:29:28.953] Batch 104...
[INFO] [2021-03-16 10:29:29] [10:29:29.665] Batch 105...
[INFO] [2021-03-16 10:29:30] [10:29:30.371] Batch 106...
[INFO] [2021-03-16 10:29:31] [10:29:31.074] Batch 107...
[INFO] [2021-03-16 10:29:31] [10:29:31.785] Batch 108...
[INFO] [2021-03-16 10:29:32] [10:29:32.487] Batch 109...
[INFO] [2021-03-16 10:29:33] [10:29:33.186] Batch 110...
[INFO] [2021-03-16 10:29:33] [10:29:33.885] Batch 111...
[INFO] [2021-03-16 10:29:34] [10:29:34.585] Batch 112...
[INFO] [2021-03-16 10:29:35] [10:29:35.287] Batch 113...
[INFO] [2021-03-16 10:29:35] [10:29:35.982] Batch 114...
[INFO] [2021-03-16 10:29:36] [10:29:36.691] Batch 115...
[INFO] [2021-03-16 10:29:37] [10:29:37.387] Batch 116...
[INFO] [2021-03-16 10:29:38] [10:29:38.076] Batch 117...
[INFO] [2021-03-16 10:29:38] [10:29:38.773] Batch 118...
[INFO] [2021-03-16 10:29:39] [10:29:39.464] Batch 119...
[INFO] [2021-03-16 10:29:40] [10:29:40.150] Batch 120...
[INFO] [2021-03-16 10:29:40] [10:29:40.834] Batch 121...
[INFO] [2021-03-16 10:29:41] [10:29:41.516] Batch 122...
[INFO] [2021-03-16 10:29:42] [10:29:42.206] Batch 123...
[INFO] [2021-03-16 10:29:42] [10:29:42.885] Batch 124...
[INFO] [2021-03-16 10:29:43] [10:29:43.562] Batch 125...
[INFO] [2021-03-16 10:29:44] [10:29:44.254] Batch 126...
[INFO] [2021-03-16 10:29:44] [10:29:44.928] Batch 127...
[INFO] [2021-03-16 10:29:45] [10:29:45.605] Batch 128...
[INFO] [2021-03-16 10:29:46] [10:29:46.282] Batch 129...
[INFO] [2021-03-16 10:29:46] [10:29:46.950] Batch 130...
[INFO] [2021-03-16 10:29:47] [10:29:47.623] Batch 131...
[INFO] [2021-03-16 10:29:48] [10:29:48.292] Batch 132...
[INFO] [2021-03-16 10:29:48] [10:29:48.954] Batch 133...
[INFO] [2021-03-16 10:29:49] [10:29:49.623] Batch 134...
[INFO] [2021-03-16 10:29:50] [10:29:50.291] Batch 135...
[INFO] [2021-03-16 10:29:50] [10:29:50.949] Batch 136...
[INFO] [2021-03-16 10:29:51] [10:29:51.612] Batch 137...
[INFO] [2021-03-16 10:29:52] [10:29:52.274] Batch 138...
[INFO] [2021-03-16 10:29:52] [10:29:52.927] Batch 139...
[INFO] [2021-03-16 10:29:53] [10:29:53.583] Batch 140...
[INFO] [2021-03-16 10:29:54] [10:29:54.243] Batch 141...
[INFO] [2021-03-16 10:29:54] [10:29:54.892] Batch 142...
[INFO] [2021-03-16 10:29:55] [10:29:55.539] Batch 143...
[INFO] [2021-03-16 10:29:56] [10:29:56.196] Batch 144...
[INFO] [2021-03-16 10:29:56] [10:29:56.841] Batch 145...
[INFO] [2021-03-16 10:29:57] [10:29:57.484] Batch 146...
[INFO] [2021-03-16 10:29:58] [10:29:58.126] Batch 147...
[INFO] [2021-03-16 10:29:58] [10:29:58.773] Batch 148...
[INFO] [2021-03-16 10:29:59] [10:29:59.413] Batch 149...
[INFO] [2021-03-16 10:30:00] [10:30:00.055] Batch 150...
[INFO] [2021-03-16 10:30:00] [10:30:00.702] Batch 151...
[INFO] [2021-03-16 10:30:01] [10:30:01.343] Batch 152...
[INFO] [2021-03-16 10:30:01] [10:30:01.975] Batch 153...
[INFO] [2021-03-16 10:30:02] [10:30:02.613] Batch 154...
[INFO] [2021-03-16 10:30:03] [10:30:03.258] Batch 155...
[INFO] [2021-03-16 10:30:03] [10:30:03.887] Batch 156...
[INFO] [2021-03-16 10:30:04] [10:30:04.514] Batch 157...
[INFO] [2021-03-16 10:30:05] [10:30:05.247] Batch 158...
[INFO] [2021-03-16 10:30:05] [10:30:05.870] Batch 159...
[INFO] [2021-03-16 10:30:06] [10:30:06.492] Batch 160...
[INFO] [2021-03-16 10:30:07] [10:30:07.111] Batch 161...
[INFO] [2021-03-16 10:30:07] [10:30:07.735] Batch 162...
[INFO] [2021-03-16 10:30:08] [10:30:08.403] Batch 163...
[INFO] [2021-03-16 10:30:09] [10:30:09.019] Batch 164...
[INFO] [2021-03-16 10:30:09] [10:30:09.636] Batch 165...
[INFO] [2021-03-16 10:30:10] [10:30:10.251] Batch 166...
[INFO] [2021-03-16 10:30:10] [10:30:10.861] Batch 167...
[INFO] [2021-03-16 10:30:11] [10:30:11.471] Batch 168...
[INFO] [2021-03-16 10:30:12] [10:30:12.081] Batch 169...
[INFO] [2021-03-16 10:30:12] [10:30:12.703] Batch 170...
[INFO] [2021-03-16 10:30:13] [10:30:13.311] Batch 171...
[INFO] [2021-03-16 10:30:13] [10:30:13.922] Batch 172...
[INFO] [2021-03-16 10:30:14] [10:30:14.529] Batch 173...
[INFO] [2021-03-16 10:30:15] [10:30:15.137] Batch 174...
[INFO] [2021-03-16 10:30:15] [10:30:15.741] Batch 175...
[INFO] [2021-03-16 10:30:16] [10:30:16.345] Batch 176...
[INFO] [2021-03-16 10:30:16] [10:30:16.942] Batch 177...
[INFO] [2021-03-16 10:30:17] [10:30:17.536] Batch 178...
[INFO] [2021-03-16 10:30:18] [10:30:18.134] Batch 179...
[INFO] [2021-03-16 10:30:18] [10:30:18.729] Batch 180...
[INFO] [2021-03-16 10:30:19] [10:30:19.323] Batch 181...
[INFO] [2021-03-16 10:30:19] [10:30:19.912] Batch 182...
[INFO] [2021-03-16 10:30:20] [10:30:20.498] Batch 183...
[INFO] [2021-03-16 10:30:21] [10:30:21.082] Batch 184...
[INFO] [2021-03-16 10:30:21] [10:30:21.668] Batch 185...
[INFO] [2021-03-16 10:30:22] [10:30:22.257] Batch 186...
[INFO] [2021-03-16 10:30:22] [10:30:22.838] Batch 187...
[INFO] [2021-03-16 10:30:23] [10:30:23.417] Batch 188...
[INFO] [2021-03-16 10:30:23] [10:30:23.993] Batch 189...
[INFO] [2021-03-16 10:30:24] [10:30:24.563] Batch 190...
[INFO] [2021-03-16 10:30:25] [10:30:25.153] Batch 191...
[INFO] [2021-03-16 10:30:25] [10:30:25.721] Batch 192...
[INFO] [2021-03-16 10:30:26] [10:30:26.264] Removed 1926000 Vernaculars
[INFO] [2021-03-16 10:30:26] ## remove_type: Article
[INFO] [2021-03-16 10:30:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-03-16 10:30:26] [10:30:26.267] Removed 0 Articles
[INFO] [2021-03-16 10:30:26] ## remove_type: Medium
[INFO] [2021-03-16 10:30:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-03-16 10:30:26] [10:30:26.271] Removed 0 Media
[INFO] [2021-03-16 10:30:26] ## remove_type: Trait
[INFO] [2021-03-16 10:30:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-03-16 10:30:26] [10:30:26.274] Removed 0 Traits
[INFO] [2021-03-16 10:30:26] ## remove_type: MetaTrait
[INFO] [2021-03-16 10:30:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-03-16 10:30:26] [10:30:26.292] Removed 0 Metatraits
[INFO] [2021-03-16 10:30:26] ## remove_type: OccurrenceMetadatum
[INFO] [2021-03-16 10:30:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-03-16 10:30:26] [10:30:26.316] Removed 0 Occurrencemetadata
[INFO] [2021-03-16 10:30:26] ## remove_type: Assoc
[INFO] [2021-03-16 10:30:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-03-16 10:30:26] [10:30:26.319] Removed 0 Assocs
[INFO] [2021-03-16 10:30:26] ## remove_type: MetaAssoc
[INFO] [2021-03-16 10:30:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-03-16 10:30:26] [10:30:26.323] Removed 0 Metaassocs
[INFO] [2021-03-16 10:30:26] ## remove_type: Identifier
[INFO] [2021-03-16 10:30:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-03-16 10:30:26] [10:30:26.339] Removed 0 Identifiers
[INFO] [2021-03-16 10:30:26] ## remove_type: Reference
[INFO] [2021-03-16 10:30:26] ++ Calling delete_all on 0 instances...
[INFO] [2021-03-16 10:30:26] [10:30:26.341] Removed 0 References
[INFO] [2021-03-16 10:30:27] Starting batch with ID 34389631...
[INFO] [2021-03-16 10:30:28] Starting batch with ID 34422201...
[INFO] [2021-03-16 10:30:29] Starting batch with ID 34422201...
[INFO] [2021-03-16 10:30:30] Starting batch with ID 34388086...
[INFO] [2021-03-16 10:30:31] Starting batch with ID 34388086...
[INFO] [2021-03-16 10:30:32] Starting batch with ID 34443812...
[INFO] [2021-03-16 10:30:33] Starting batch with ID 34443812...
[INFO] [2021-03-16 10:30:33] Starting batch with ID 34447139...
[INFO] [2021-03-16 10:30:34] Starting batch with ID 34460129...
[INFO] [2021-03-16 10:30:35] Starting batch with ID 34460129...
[INFO] [2021-03-16 10:30:36] Starting batch with ID 34503219...
[INFO] [2021-03-16 10:30:37] Starting batch with ID 34503219...
[INFO] [2021-03-16 10:30:38] Starting batch with ID 34529286...
[INFO] [2021-03-16 10:30:38] Starting batch with ID 34536019...
[INFO] [2021-03-16 10:30:39] Starting batch with ID 34562735...
[INFO] [2021-03-16 10:30:40] Starting batch with ID 34562735...
[INFO] [2021-03-16 10:30:41] Starting batch with ID 34694512...
[INFO] [2021-03-16 10:30:42] Starting batch with ID 34694512...
[INFO] [2021-03-16 10:30:43] Starting batch with ID 34588329...
[INFO] [2021-03-16 10:30:43] Starting batch with ID 34588329...
[INFO] [2021-03-16 10:30:44] Starting batch with ID 34742968...
[INFO] [2021-03-16 10:30:45] Starting batch with ID 34742968...
[INFO] [2021-03-16 10:30:46] Starting batch with ID 34680399...
[INFO] [2021-03-16 10:30:47] Starting batch with ID 34680399...
[INFO] [2021-03-16 10:30:48] Starting batch with ID 34610362...
[INFO] [2021-03-16 10:30:49] Starting batch with ID 34610362...
[INFO] [2021-03-16 10:30:49] Starting batch with ID 34735367...
[INFO] [2021-03-16 10:30:50] Starting batch with ID 34634013...
[INFO] [2021-03-16 10:30:51] Starting batch with ID 34634013...
[INFO] [2021-03-16 10:30:52] Starting batch with ID 34756456...
[INFO] [2021-03-16 10:30:53] Starting batch with ID 34756456...
[INFO] [2021-03-16 10:30:54] Starting batch with ID 34409494...
[INFO] [2021-03-16 10:30:55] Starting batch with ID 34449865...
[INFO] [2021-03-16 10:30:56] Starting batch with ID 34449865...
[INFO] [2021-03-16 10:30:57] Starting batch with ID 34510767...
[INFO] [2021-03-16 10:30:58] Starting batch with ID 34510792...
[INFO] [2021-03-16 10:30:58] Starting batch with ID 34505166...
[INFO] [2021-03-16 10:30:59] Starting batch with ID 34505219...
[INFO] [2021-03-16 10:31:00] Starting batch with ID 34505219...
[INFO] [2021-03-16 10:31:01] Starting batch with ID 34416953...
[INFO] [2021-03-16 10:31:02] Starting batch with ID 34416953...
[INFO] [2021-03-16 10:31:03] Starting batch with ID 34433308...
[INFO] [2021-03-16 10:31:04] Starting batch with ID 34433308...
[INFO] [2021-03-16 10:31:05] Starting batch with ID 34587668...
[INFO] [2021-03-16 10:31:05] Starting batch with ID 34587668...
[INFO] [2021-03-16 10:31:06] Starting batch with ID 34697088...
[INFO] [2021-03-16 10:31:07] Starting batch with ID 34697088...
[INFO] [2021-03-16 10:31:08] Starting batch with ID 34636680...
[INFO] [2021-03-16 10:31:09] Starting batch with ID 34636680...
[INFO] [2021-03-16 10:31:10] Starting batch with ID 34536200...
[INFO] [2021-03-16 10:31:11] Starting batch with ID 34536200...
[INFO] [2021-03-16 10:31:12] Starting batch with ID 34671887...
[INFO] [2021-03-16 10:31:13] Starting batch with ID 34518802...
[INFO] [2021-03-16 10:31:14] Starting batch with ID 34567333...
[INFO] [2021-03-16 10:31:16] Starting batch with ID 34648246...
[INFO] [2021-03-16 10:31:17] Starting batch with ID 34691479...
[INFO] [2021-03-16 10:31:18] Starting batch with ID 34570764...
[INFO] [2021-03-16 10:31:20] Starting batch with ID 34614563...
[INFO] [2021-03-16 10:31:20] Starting batch with ID 34684281...
[INFO] [2021-03-16 10:31:21] Starting batch with ID 34743453...
[INFO] [2021-03-16 10:31:22] Starting batch with ID 34765625...
[INFO] [2021-03-16 10:31:23] Starting batch with ID 34724386...
[INFO] [2021-03-16 10:31:24] Starting batch with ID 34740788...
[INFO] [2021-03-16 10:31:25] Starting batch with ID 34390928...
[INFO] [2021-03-16 10:31:26] Starting batch with ID 34409553...
[INFO] [2021-03-16 10:31:27] Starting batch with ID 34434798...
[INFO] [2021-03-16 10:31:28] Starting batch with ID 34501358...
[INFO] [2021-03-16 10:31:29] Starting batch with ID 34423434...
[INFO] [2021-03-16 10:31:30] Starting batch with ID 34462865...
[INFO] [2021-03-16 10:31:31] Starting batch with ID 34462865...
[INFO] [2021-03-16 10:31:32] Starting batch with ID 34427935...
[INFO] [2021-03-16 10:31:33] Starting batch with ID 34427935...
[INFO] [2021-03-16 10:31:34] Starting batch with ID 34508322...
[INFO] [2021-03-16 10:31:35] Starting batch with ID 34508322...
[INFO] [2021-03-16 10:31:36] Starting batch with ID 34677101...
[INFO] [2021-03-16 10:31:36] Starting batch with ID 34677101...
[INFO] [2021-03-16 10:31:37] Starting batch with ID 34595383...
[INFO] [2021-03-16 10:31:38] Starting batch with ID 34604394...
[INFO] [2021-03-16 10:31:39] Starting batch with ID 34628731...
[INFO] [2021-03-16 10:31:40] Starting batch with ID 34628731...
[INFO] [2021-03-16 10:31:41] Starting batch with ID 34728303...
[INFO] [2021-03-16 10:31:42] Starting batch with ID 34728303...
[INFO] [2021-03-16 10:31:43] Starting batch with ID 34650450...
[INFO] [2021-03-16 10:31:44] Starting batch with ID 34650450...
[INFO] [2021-03-16 10:31:45] Starting batch with ID 34666625...
[INFO] [2021-03-16 10:31:46] Starting batch with ID 34666625...
[INFO] [2021-03-16 10:31:47] Starting batch with ID 34686668...
[INFO] [2021-03-16 10:31:48] Starting batch with ID 34686668...
[INFO] [2021-03-16 10:31:49] Starting batch with ID 34559612...
[INFO] [2021-03-16 10:31:50] Starting batch with ID 34580629...
[INFO] [2021-03-16 10:31:51] Starting batch with ID 34535094...
[INFO] [2021-03-16 10:31:52] Starting batch with ID 34568867...
[INFO] [2021-03-16 10:31:54] Starting batch with ID 34397518...
[INFO] [2021-03-16 10:31:55] Starting batch with ID 34387006...
[INFO] [2021-03-16 10:31:56] Starting batch with ID 34464109...
[INFO] [2021-03-16 10:31:57] Starting batch with ID 34503582...
[INFO] [2021-03-16 10:31:58] Starting batch with ID 34414063...
[INFO] [2021-03-16 10:31:59] Starting batch with ID 34433958...
[INFO] [2021-03-16 10:32:00] Starting batch with ID 34403215...
[INFO] [2021-03-16 10:32:02] Starting batch with ID 34451972...
[INFO] [2021-03-16 10:32:03] Starting batch with ID 34519402...
[INFO] [2021-03-16 10:32:04] Starting batch with ID 34625573...
[INFO] [2021-03-16 10:32:05] Starting batch with ID 34686237...
[INFO] [2021-03-16 10:32:06] Starting batch with ID 34512596...
[INFO] [2021-03-16 10:32:07] Starting batch with ID 34646109...
[INFO] [2021-03-16 10:32:08] Starting batch with ID 34707659...
[INFO] [2021-03-16 10:32:09] Starting batch with ID 34569860...
[INFO] [2021-03-16 10:32:10] Starting batch with ID 34636250...
[INFO] [2021-03-16 10:32:12] Starting batch with ID 34757296...
[INFO] [2021-03-16 10:32:13] Starting batch with ID 34546730...
[INFO] [2021-03-16 10:32:14] Starting batch with ID 34607357...
[INFO] [2021-03-16 10:32:15] Starting batch with ID 34672904...
[INFO] [2021-03-16 10:32:16] Starting batch with ID 34539750...
[INFO] [2021-03-16 10:32:18] Starting batch with ID 34737799...
[INFO] [2021-03-16 10:32:19] Starting batch with ID 34612007...
[INFO] [2021-03-16 10:32:20] Starting batch with ID 34614890...
[INFO] [2021-03-16 10:32:21] Starting batch with ID 34522187...
[INFO] [2021-03-16 10:32:22] Starting batch with ID 34525958...
[INFO] [2021-03-16 10:32:23] Starting batch with ID 34725692...
[INFO] [2021-03-16 10:32:25] Starting batch with ID 34770548...
[INFO] [2021-03-16 10:32:26] Starting batch with ID 34451354...
[INFO] [2021-03-16 10:32:27] Starting batch with ID 34435962...
[INFO] [2021-03-16 10:32:28] Starting batch with ID 34467912...
[INFO] [2021-03-16 10:32:30] Starting batch with ID 34388464...
[INFO] [2021-03-16 10:32:31] Starting batch with ID 34410652...
[INFO] [2021-03-16 10:32:32] Starting batch with ID 34404471...
[INFO] [2021-03-16 10:32:34] Starting batch with ID 34500609...
[INFO] [2021-03-16 10:32:35] Starting batch with ID 34750984...
[INFO] [2021-03-16 10:32:37] Starting batch with ID 34758519...
[INFO] [2021-03-16 10:32:38] Starting batch with ID 34675207...
[INFO] [2021-03-16 10:32:39] Starting batch with ID 34673509...
[INFO] [2021-03-16 10:32:40] Starting batch with ID 34593670...
[INFO] [2021-03-16 10:32:42] Starting batch with ID 34666693...
[INFO] [2021-03-16 10:32:43] Starting batch with ID 34510689...
[INFO] [2021-03-16 10:32:44] Starting batch with ID 34584364...
[INFO] [2021-03-16 10:32:46] Starting batch with ID 34726374...
[INFO] [2021-03-16 10:32:48] Starting batch with ID 34533577...
[INFO] [2021-03-16 10:32:49] Starting batch with ID 34638096...
[INFO] [2021-03-16 10:32:50] Starting batch with ID 34706277...
[INFO] [2021-03-16 10:32:52] Starting batch with ID 34572392...
[INFO] [2021-03-16 10:32:53] Starting batch with ID 34644139...
[INFO] [2021-03-16 10:32:54] Starting batch with ID 34542283...
[INFO] [2021-03-16 10:32:56] Starting batch with ID 34625160...
[INFO] [2021-03-16 10:32:57] Starting batch with ID 34696190...
[INFO] [2021-03-16 10:32:59] Starting batch with ID 34618760...
[INFO] [2021-03-16 10:32:59] Starting batch with ID 34620219...
[INFO] [2021-03-16 10:33:00] ## remove_type: Node
[INFO] [2021-03-16 10:33:00] ++ Batch removal of 386028 instances...
[INFO] [2021-03-16 10:33:00] [10:33:00.568] Batch 0...
[INFO] [2021-03-16 10:33:01] [10:33:01.477] Batch 1...
[INFO] [2021-03-16 10:33:02] [10:33:02.345] Batch 2...
[INFO] [2021-03-16 10:33:03] [10:33:03.209] Batch 3...
[INFO] [2021-03-16 10:33:04] [10:33:04.024] Batch 4...
[INFO] [2021-03-16 10:33:04] [10:33:04.831] Batch 5...
[INFO] [2021-03-16 10:33:05] [10:33:05.638] Batch 6...
[INFO] [2021-03-16 10:33:06] [10:33:06.453] Batch 7...
[INFO] [2021-03-16 10:33:07] [10:33:07.289] Batch 8...
[INFO] [2021-03-16 10:33:10] [10:33:10.466] Batch 9...
[INFO] [2021-03-16 10:33:11] [10:33:11.479] Batch 10...
[INFO] [2021-03-16 10:33:12] [10:33:12.885] Batch 11...
[INFO] [2021-03-16 10:33:14] [10:33:14.360] Batch 12...
[INFO] [2021-03-16 10:33:15] [10:33:15.387] Batch 13...
[INFO] [2021-03-16 10:33:16] [10:33:16.424] Batch 14...
[INFO] [2021-03-16 10:33:17] [10:33:17.397] Batch 15...
[INFO] [2021-03-16 10:33:18] [10:33:18.426] Batch 16...
[INFO] [2021-03-16 10:33:19] [10:33:19.436] Batch 17...
[INFO] [2021-03-16 10:33:20] [10:33:20.392] Batch 18...
[INFO] [2021-03-16 10:33:21] [10:33:21.340] Batch 19...
[INFO] [2021-03-16 10:33:24] [10:33:24.150] Batch 20...
[INFO] [2021-03-16 10:33:25] [10:33:25.134] Batch 21...
[INFO] [2021-03-16 10:33:26] [10:33:26.111] Batch 22...
[INFO] [2021-03-16 10:33:27] [10:33:27.572] Batch 23...
[INFO] [2021-03-16 10:33:28] [10:33:28.520] Batch 24...
[INFO] [2021-03-16 10:33:29] [10:33:29.247] Batch 25...
[INFO] [2021-03-16 10:33:29] [10:33:29.972] Batch 26...
[INFO] [2021-03-16 10:33:30] [10:33:30.760] Batch 27...
[INFO] [2021-03-16 10:33:32] [10:33:32.578] Batch 28...
[INFO] [2021-03-16 10:33:33] [10:33:33.297] Batch 29...
[INFO] [2021-03-16 10:33:33] [10:33:33.992] Batch 30...
[INFO] [2021-03-16 10:33:35] [10:33:35.860] Batch 31...
[INFO] [2021-03-16 10:33:36] [10:33:36.854] Batch 32...
[INFO] [2021-03-16 10:33:39] [10:33:39.010] Batch 33...
[INFO] [2021-03-16 10:33:39] [10:33:39.694] Batch 34...
[INFO] [2021-03-16 10:33:40] [10:33:40.456] Batch 35...
[INFO] [2021-03-16 10:33:41] [10:33:41.960] Batch 36...
[INFO] [2021-03-16 10:33:42] [10:33:42.896] Batch 37...
[INFO] [2021-03-16 10:33:43] [10:33:43.555] Batch 38...
[INFO] [2021-03-16 10:33:44] [10:33:44.165] Removed 386028 Nodes
[START] [2021-03-16 10:37:00] logged process: cdc14cc7e396e8d099dbdcb2b6584e4ad6934982

[START] [2021-03-16 10:37:00] Creating resource from OpenData
[START] [2021-03-16 10:37:02] logged process: cdc14cc7e396e8d099dbdcb2b6584e4ad6934982

[START] [2021-03-16 10:37:02] Parse meta.xml file and create formats with fields
[WARN] [2021-03-16 10:37:02] (common) IGNORED  (nodes) field header: scientificNameAuthorship term: http://rs.tdwg.org/dwc/terms/scientificNameAuthorship
[STOP] [2021-03-16 10:37:02] Parse meta.xml file and create formats with fields
[STOP] [2021-03-16 10:37:02] Creating resource from OpenData
[START] [2021-03-16 10:37:02] logged process: cdc14cc7e396e8d099dbdcb2b6584e4ad6934982

[START] [2021-03-16 10:37:02] create_harvest_instance
[STOP] [2021-03-16 10:37:03] create_harvest_instance
[START] [2021-03-16 10:37:03] fetch_files
[STOP] [2021-03-16 10:37:03] fetch_files
[START] [2021-03-16 10:37:03] validate_each_file
[STOP] [2021-03-16 10:37:27] validate_each_file
[START] [2021-03-16 10:37:27] convert_to_csv
[CMD] [2021-03-16 10:37:27] /usr/bin/sort /app/public/converted_csv/wikidata_nodes_26964.csv > /app/public/converted_csv/wikidata_nodes_26964.csv_sorted
[CMD] [2021-03-16 10:37:27] /usr/bin/sort /app/public/converted_csv/wikidata_vernaculars_26965.csv > /app/public/converted_csv/wikidata_vernaculars_26965.csv_sorted
[STOP] [2021-03-16 10:37:28] convert_to_csv
[START] [2021-03-16 10:37:28] calculate_delta
[CMD] [2021-03-16 10:37:28] echo "0a" > /app/public/diff/wikidata_nodes_26964.diff
[CMD] [2021-03-16 10:37:28] tail -n +1 /app/public/converted_csv/wikidata_nodes_26964.csv >> /app/public/diff/wikidata_nodes_26964.diff
[CMD] [2021-03-16 10:37:28] echo "." >> /app/public/diff/wikidata_nodes_26964.diff
[CMD] [2021-03-16 10:37:28] echo "0a" > /app/public/diff/wikidata_vernaculars_26965.diff
[CMD] [2021-03-16 10:37:28] tail -n +1 /app/public/converted_csv/wikidata_vernaculars_26965.csv >> /app/public/diff/wikidata_vernaculars_26965.diff
[CMD] [2021-03-16 10:37:28] echo "." >> /app/public/diff/wikidata_vernaculars_26965.diff
[STOP] [2021-03-16 10:37:28] calculate_delta
[START] [2021-03-16 10:37:28] parse_diff_and_store
[INFO] [2021-03-16 10:37:28] Loading nodes diff file into memory (true lines)...
[INFO] [2021-03-16 10:38:24] Loading vernaculars diff file into memory (true lines)...
[INFO] [2021-03-16 10:41:20] Storing 145396 ScientificNames
[INFO] [2021-03-16 10:41:20] Processing group of 145396 in 146 groups of 1000
[INFO] [2021-03-16 10:42:16] Average Time: 0.381
[INFO] [2021-03-16 10:42:16] Total Time: 57s
[INFO] [2021-03-16 10:42:16] last 3 / first 3: 0.77
[INFO] [2021-03-16 10:42:16] Std.Dev: 0.5787918451395113; Max: 5.76
[INFO] [2021-03-16 10:42:16] Storing 145396 Identifiers
[INFO] [2021-03-16 10:42:16] Processing group of 145396 in 146 groups of 1000
[INFO] [2021-03-16 10:42:40] Average Time: 0.158
[INFO] [2021-03-16 10:42:40] Total Time: 24s
[INFO] [2021-03-16 10:42:40] last 3 / first 3: 0.47
[INFO] [2021-03-16 10:42:40] Std.Dev: 0.044721359549995794; Max: 0.35
[INFO] [2021-03-16 10:42:40] Storing 145396 Nodes
[INFO] [2021-03-16 10:42:40] Processing group of 145396 in 146 groups of 1000
[INFO] [2021-03-16 10:43:46] Average Time: 0.446
[INFO] [2021-03-16 10:43:46] Total Time: 1m6s
[INFO] [2021-03-16 10:43:46] last 3 / first 3: 0.78
[INFO] [2021-03-16 10:43:46] Std.Dev: 0.9787747442593725; Max: 6.4
[INFO] [2021-03-16 10:43:46] Storing 880106 Vernaculars
[INFO] [2021-03-16 10:43:46] Processing group of 880106 in 881 groups of 1000
[INFO] [2021-03-16 10:46:58] Average Time: 0.244
[INFO] [2021-03-16 10:46:58] Total Time: 3m13s
[INFO] [2021-03-16 10:46:58] last 3 / first 3: 0.76
[INFO] [2021-03-16 10:46:58] Std.Dev: 0.6496152707564686; Max: 7.88
[STOP] [2021-03-16 10:46:58] parse_diff_and_store
[ERR] [2021-03-16 10:46:58] RuntimeError
[ERR] [2021-03-16 10:46:58] ActiveRecord::StatementInvalid while parsing something around here: [{"id":null,"resource_id":658,"harvest_id":3545,"node_id":null,"language_id":279,"node_resource_pk":"Q1582612","verbatim":"东方豹鲂𫚒","language_code_verbatim":"zh-hans","locality":null,"remarks":null,"source":"https://www.wikidata.org/wiki/Q1582612","is_preferred":true,"removed_by_harvest_id":null,"created_at":null,"updated_at":null},{"id":null,"resource_id":658,"harvest_id":3545,"node_id":null,"language_id":95,"node_resource_pk":"Q3473543","verbatim":"东方野扇花","language_code_verbatim":"zh","locality":null,"remarks":null,"source":"https://www.wikidata.org/wiki/Q3473543","is_preferred":true,"removed_by_harvest_id":null,"created_at":null,"updated_at":null},{"id":null,"resource_id":658,"harvest_id":3545,"node_id":null,"language_id":95,"node_resource_pk":"Q3281704","verbatim":"东方铁线莲","language_code_verbatim":"zh","locality":null,"remarks":null,"source":"https://www.wikidata.org/wiki/Q3281704","is_preferred":true,"removed_by_harvest_id":null,"created_at":null,"updated_at":null}]
[ERR] [2021-03-16 10:46:58] ../models/resource_harvester.rb:412:in `rescue in block (2 levels) in store_new'
[ERR] [2021-03-16 10:46:58] ../models/resource_harvester.rb:403:in `block (2 levels) in store_new'
[ERR] [2021-03-16 10:46:58] ../models/logged_process.rb:42:in `block in in_groups'
[ERR] [2021-03-16 10:46:58] ../models/logged_process.rb:40:in `in_groups'
[ERR] [2021-03-16 10:46:58] ../models/resource_harvester.rb:399:in `block in store_new'
[ERR] [2021-03-16 10:46:58] ../models/resource_harvester.rb:393:in `each'
[ERR] [2021-03-16 10:46:58] ../models/resource_harvester.rb:393:in `store_new'
[ERR] [2021-03-16 10:46:58] ../models/resource_harvester.rb:367:in `parse_diff_and_store'
[ERR] [2021-03-16 10:46:58] ../models/resource_harvester.rb:85:in `block (3 levels) in start'
[ERR] [2021-03-16 10:46:58] ../models/logged_process.rb:27:in `run_step'
[ERR] [2021-03-16 10:46:58] ../models/resource_harvester.rb:85:in `block (2 levels) in start'
[ERR] [2021-03-16 10:46:58] ../models/resource_harvester.rb:74:in `each_key'
[ERR] [2021-03-16 10:46:58] ../models/resource_harvester.rb:74:in `block in start'
[ERR] [2021-03-16 10:46:58] ../models/resource.rb:159:in `lock'
[ERR] [2021-03-16 10:46:58] ../models/resource_harvester.rb:72:in `start'
[ERR] [2021-03-16 10:46:58] ../models/resource.rb:245:in `harvest'
[ERR] [2021-03-16 10:46:58] ../models/resource.rb:221:in `re_download_opendata_and_harvest'
[ERR] [2021-03-16 10:46:58] bin/rails:4:in `require'
[ERR] [2021-03-16 10:46:58] bin/rails:4:in `<main>'
[STOP] [2021-03-16 10:46:58] logged process, took 596.3
[INFO] [2021-03-25 11:17:48] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-03-25 11:17:48] ## remove_type: ScientificName
[INFO] [2021-03-25 11:17:48] ++ Calling delete_all on 145396 instances...
[INFO] [2021-03-25 11:18:09] [11:18:09.516] Removed 145396 Scientificnames
[INFO] [2021-03-25 11:18:09] ## remove_type: Vernacular
[INFO] [2021-03-25 11:18:09] ++ Batch removal of 774000 instances...
[INFO] [2021-03-25 11:18:09] [11:18:09.742] Batch 0...
[INFO] [2021-03-25 11:18:10] [11:18:10.632] Batch 1...
[INFO] [2021-03-25 11:18:11] [11:18:11.584] Batch 2...
[INFO] [2021-03-25 11:18:12] [11:18:12.463] Batch 3...
[INFO] [2021-03-25 11:18:13] [11:18:13.343] Batch 4...
[INFO] [2021-03-25 11:18:14] [11:18:14.350] Batch 5...
[INFO] [2021-03-25 11:18:15] [11:18:15.357] Batch 6...
[INFO] [2021-03-25 11:18:17] [11:18:17.639] Batch 7...
[INFO] [2021-03-25 11:18:18] [11:18:18.362] Batch 8...
[INFO] [2021-03-25 11:18:19] [11:18:19.126] Batch 9...
[INFO] [2021-03-25 11:18:19] [11:18:19.827] Batch 10...
[INFO] [2021-03-25 11:18:20] [11:18:20.516] Batch 11...
[INFO] [2021-03-25 11:18:21] [11:18:21.196] Batch 12...
[INFO] [2021-03-25 11:18:21] [11:18:21.870] Batch 13...
[INFO] [2021-03-25 11:18:22] [11:18:22.550] Batch 14...
[INFO] [2021-03-25 11:18:23] [11:18:23.223] Batch 15...
[INFO] [2021-03-25 11:18:23] [11:18:23.891] Batch 16...
[INFO] [2021-03-25 11:18:24] [11:18:24.564] Batch 17...
[INFO] [2021-03-25 11:18:25] [11:18:25.236] Batch 18...
[INFO] [2021-03-25 11:18:25] [11:18:25.899] Batch 19...
[INFO] [2021-03-25 11:18:26] [11:18:26.560] Batch 20...
[INFO] [2021-03-25 11:18:27] [11:18:27.222] Batch 21...
[INFO] [2021-03-25 11:18:27] [11:18:27.880] Batch 22...
[INFO] [2021-03-25 11:18:28] [11:18:28.540] Batch 23...
[INFO] [2021-03-25 11:18:29] [11:18:29.196] Batch 24...
[INFO] [2021-03-25 11:18:29] [11:18:29.850] Batch 25...
[INFO] [2021-03-25 11:18:30] [11:18:30.512] Batch 26...
[INFO] [2021-03-25 11:18:31] [11:18:31.169] Batch 27...
[INFO] [2021-03-25 11:18:31] [11:18:31.818] Batch 28...
[INFO] [2021-03-25 11:18:32] [11:18:32.467] Batch 29...
[INFO] [2021-03-25 11:18:33] [11:18:33.119] Batch 30...
[INFO] [2021-03-25 11:18:33] [11:18:33.768] Batch 31...
[INFO] [2021-03-25 11:18:34] [11:18:34.423] Batch 32...
[INFO] [2021-03-25 11:18:35] [11:18:35.065] Batch 33...
[INFO] [2021-03-25 11:18:35] [11:18:35.706] Batch 34...
[INFO] [2021-03-25 11:18:36] [11:18:36.348] Batch 35...
[INFO] [2021-03-25 11:18:37] [11:18:37.004] Batch 36...
[INFO] [2021-03-25 11:18:37] [11:18:37.641] Batch 37...
[INFO] [2021-03-25 11:18:38] [11:18:38.278] Batch 38...
[INFO] [2021-03-25 11:18:38] [11:18:38.916] Batch 39...
[INFO] [2021-03-25 11:18:39] [11:18:39.605] Batch 40...
[INFO] [2021-03-25 11:18:40] [11:18:40.238] Batch 41...
[INFO] [2021-03-25 11:18:40] [11:18:40.870] Batch 42...
[INFO] [2021-03-25 11:18:41] [11:18:41.502] Batch 43...
[INFO] [2021-03-25 11:18:42] [11:18:42.133] Batch 44...
[INFO] [2021-03-25 11:18:42] [11:18:42.762] Batch 45...
[INFO] [2021-03-25 11:18:43] [11:18:43.397] Batch 46...
[INFO] [2021-03-25 11:18:44] [11:18:44.025] Batch 47...
[INFO] [2021-03-25 11:18:44] [11:18:44.713] Batch 48...
[INFO] [2021-03-25 11:18:45] [11:18:45.336] Batch 49...
[INFO] [2021-03-25 11:18:45] [11:18:45.958] Batch 50...
[INFO] [2021-03-25 11:18:46] [11:18:46.586] Batch 51...
[INFO] [2021-03-25 11:18:47] [11:18:47.212] Batch 52...
[INFO] [2021-03-25 11:18:47] [11:18:47.829] Batch 53...
[INFO] [2021-03-25 11:18:48] [11:18:48.442] Batch 54...
[INFO] [2021-03-25 11:18:49] [11:18:49.055] Batch 55...
[INFO] [2021-03-25 11:18:49] [11:18:49.665] Batch 56...
[INFO] [2021-03-25 11:18:50] [11:18:50.269] Batch 57...
[INFO] [2021-03-25 11:18:50] [11:18:50.872] Batch 58...
[INFO] [2021-03-25 11:18:51] [11:18:51.479] Batch 59...
[INFO] [2021-03-25 11:18:52] [11:18:52.082] Batch 60...
[INFO] [2021-03-25 11:18:52] [11:18:52.680] Batch 61...
[INFO] [2021-03-25 11:18:53] [11:18:53.278] Batch 62...
[INFO] [2021-03-25 11:18:53] [11:18:53.875] Batch 63...
[INFO] [2021-03-25 11:18:54] [11:18:54.469] Batch 64...
[INFO] [2021-03-25 11:18:55] [11:18:55.070] Batch 65...
[INFO] [2021-03-25 11:18:55] [11:18:55.663] Batch 66...
[INFO] [2021-03-25 11:18:56] [11:18:56.256] Batch 67...
[INFO] [2021-03-25 11:18:56] [11:18:56.843] Batch 68...
[INFO] [2021-03-25 11:18:57] [11:18:57.432] Batch 69...
[INFO] [2021-03-25 11:18:58] [11:18:58.024] Batch 70...
[INFO] [2021-03-25 11:18:58] [11:18:58.611] Batch 71...
[INFO] [2021-03-25 11:18:59] [11:18:59.196] Batch 72...
[INFO] [2021-03-25 11:18:59] [11:18:59.772] Batch 73...
[INFO] [2021-03-25 11:19:00] [11:19:00.346] Batch 74...
[INFO] [2021-03-25 11:19:00] [11:19:00.929] Batch 75...
[INFO] [2021-03-25 11:19:01] [11:19:01.499] Batch 76...
[INFO] [2021-03-25 11:19:02] [11:19:02.071] Batch 77...
[INFO] [2021-03-25 11:19:02] [11:19:02.599] Removed 774000 Vernaculars
[INFO] [2021-03-25 11:19:02] ## remove_type: Article
[INFO] [2021-03-25 11:19:02] ++ Calling delete_all on 0 instances...
[INFO] [2021-03-25 11:19:02] [11:19:02.723] Removed 0 Articles
[INFO] [2021-03-25 11:19:02] ## remove_type: Medium
[INFO] [2021-03-25 11:19:02] ++ Calling delete_all on 0 instances...
[INFO] [2021-03-25 11:19:02] [11:19:02.726] Removed 0 Media
[INFO] [2021-03-25 11:19:02] ## remove_type: Trait
[INFO] [2021-03-25 11:19:02] ++ Calling delete_all on 0 instances...
[INFO] [2021-03-25 11:19:02] [11:19:02.727] Removed 0 Traits
[INFO] [2021-03-25 11:19:02] ## remove_type: MetaTrait
[INFO] [2021-03-25 11:19:02] ++ Calling delete_all on 0 instances...
[INFO] [2021-03-25 11:19:02] [11:19:02.735] Removed 0 Metatraits
[INFO] [2021-03-25 11:19:02] ## remove_type: OccurrenceMetadatum
[INFO] [2021-03-25 11:19:02] ++ Calling delete_all on 0 instances...
[INFO] [2021-03-25 11:19:02] [11:19:02.762] Removed 0 Occurrencemetadata
[INFO] [2021-03-25 11:19:02] ## remove_type: Assoc
[INFO] [2021-03-25 11:19:02] ++ Calling delete_all on 0 instances...
[INFO] [2021-03-25 11:19:02] [11:19:02.764] Removed 0 Assocs
[INFO] [2021-03-25 11:19:02] ## remove_type: MetaAssoc
[INFO] [2021-03-25 11:19:02] ++ Calling delete_all on 0 instances...
[INFO] [2021-03-25 11:19:02] [11:19:02.765] Removed 0 Metaassocs
[INFO] [2021-03-25 11:19:02] ## remove_type: Identifier
[INFO] [2021-03-25 11:19:02] ++ Calling delete_all on 145396 instances...
[INFO] [2021-03-25 11:19:12] [11:19:12.323] Removed 145396 Identifiers
[INFO] [2021-03-25 11:19:12] ## remove_type: Reference
[INFO] [2021-03-25 11:19:12] ++ Calling delete_all on 0 instances...
[INFO] [2021-03-25 11:19:12] [11:19:12.325] Removed 0 References
[INFO] [2021-03-25 11:19:12] ## remove_type: Node
[INFO] [2021-03-25 11:19:12] ++ Calling delete_all on 145396 instances...
[INFO] [2021-03-25 11:19:30] [11:19:30.812] Removed 145396 Nodes
[START] [2021-03-25 11:19:32] logged process: cb7ec98f0f4fc94af4d81126a6c79dbb3658c39a

[START] [2021-03-25 11:19:32] Creating resource from OpenData
[START] [2021-03-25 11:19:37] logged process: cb7ec98f0f4fc94af4d81126a6c79dbb3658c39a

[START] [2021-03-25 11:19:37] Parse meta.xml file and create formats with fields
[WARN] [2021-03-25 11:19:47] (common) IGNORED  (nodes) field header: scientificNameAuthorship term: http://rs.tdwg.org/dwc/terms/scientificNameAuthorship
[STOP] [2021-03-25 11:19:47] Parse meta.xml file and create formats with fields
[STOP] [2021-03-25 11:19:47] Creating resource from OpenData
[START] [2021-03-25 11:19:48] logged process: cb7ec98f0f4fc94af4d81126a6c79dbb3658c39a

[START] [2021-03-25 11:19:48] create_harvest_instance
[STOP] [2021-03-25 11:19:48] create_harvest_instance
[START] [2021-03-25 11:19:48] fetch_files
[STOP] [2021-03-25 11:19:48] fetch_files
[START] [2021-03-25 11:19:48] validate_each_file
[STOP] [2021-03-25 11:19:48] validate_each_file
[START] [2021-03-25 11:19:48] convert_to_csv
[STOP] [2021-03-25 11:19:48] convert_to_csv
[START] [2021-03-25 11:19:48] calculate_delta
[STOP] [2021-03-25 11:19:48] calculate_delta
[START] [2021-03-25 11:19:48] parse_diff_and_store
[STOP] [2021-03-25 11:19:48] parse_diff_and_store
[START] [2021-03-25 11:19:48] resolve_keys
[INFO] [2021-03-25 11:19:54] Occurrences to nodes (through scientific_names)...
[INFO] [2021-03-25 11:19:54] traits to occurrences...
[INFO] [2021-03-25 11:19:54] traits to nodes (through occurrences)...
[INFO] [2021-03-25 11:19:54] Traits to sex term...
[INFO] [2021-03-25 11:19:54] Traits to lifestage term...
[INFO] [2021-03-25 11:19:54] MetaTraits to traits...
[INFO] [2021-03-25 11:19:54] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-03-25 11:19:54] Assocs to occurrences...
[INFO] [2021-03-25 11:19:54] Assocs to nodes...
[INFO] [2021-03-25 11:19:54] Assoc to sex term...
[INFO] [2021-03-25 11:19:54] Assoc to lifestage term...
[INFO] [2021-03-25 11:19:54] MetaAssoc to assocs...
[STOP] [2021-03-25 11:19:54] resolve_keys
[START] [2021-03-25 11:19:54] hold_for_later_1
[STOP] [2021-03-25 11:19:54] hold_for_later_1
[START] [2021-03-25 11:19:54] hold_for_later_2
[STOP] [2021-03-25 11:19:54] hold_for_later_2
[START] [2021-03-25 11:19:54] resolve_missing_parents
[STOP] [2021-03-25 11:19:54] resolve_missing_parents
[START] [2021-03-25 11:19:54] rebuild_nodes
[START] [2021-03-25 11:19:54] Flattener#flatten
[START] [2021-03-25 11:19:54] Flattener#study_resource
[WARN] [2021-03-25 11:19:54] Flattener: no children found, aborting flatten
[STOP] [2021-03-25 11:19:54] Flattener#flatten
[STOP] [2021-03-25 11:19:54] rebuild_nodes
[START] [2021-03-25 11:19:54] resolve_missing_media_owners
[STOP] [2021-03-25 11:19:54] resolve_missing_media_owners
[START] [2021-03-25 11:19:54] sanitize_media_verbatims
[STOP] [2021-03-25 11:19:54] sanitize_media_verbatims
[START] [2021-03-25 11:19:54] queue_downloads
[STOP] [2021-03-25 11:19:54] queue_downloads
[START] [2021-03-25 11:19:54] parse_names
[STOP] [2021-03-25 11:19:54] parse_names
[START] [2021-03-25 11:19:54] denormalize_canonical_names_to_nodes
[STOP] [2021-03-25 11:19:54] denormalize_canonical_names_to_nodes
[START] [2021-03-25 11:19:54] match_nodes
[START] [2021-03-25 11:19:54] map_all_nodes_to_pages
[STOP] [2021-03-25 11:19:54] map_all_nodes_to_pages
[INFO] [2021-03-25 11:19:54] ZERO unmatched nodes (of 0)! Nicely done.
[START] [2021-03-25 11:19:54] update_nodes
[STOP] [2021-03-25 11:19:54] update_nodes
[STOP] [2021-03-25 11:19:54] match_nodes
[START] [2021-03-25 11:19:54] reindex_search
[STOP] [2021-03-25 11:19:54] reindex_search
[START] [2021-03-25 11:19:54] normalize_units
[STOP] [2021-03-25 11:19:54] normalize_units
[START] [2021-03-25 11:19:54] calculate_statistics
[ERR] [2021-03-25 11:19:54] Exception
[ERR] [2021-03-25 11:19:54] ZERO NODES! That is probably... bad.
[STOP] [2021-03-25 11:19:54] calculate_statistics
[STOP] [2021-03-25 11:19:54] logged process, took 7.51
[START] [2021-03-25 11:32:47] logged process: cb7ec98f0f4fc94af4d81126a6c79dbb3658c39a

[START] [2021-03-25 11:32:47] Parse meta.xml file and create formats with fields
[WARN] [2021-03-25 11:32:47] (common) IGNORED  (nodes) field header: scientificNameAuthorship term: http://rs.tdwg.org/dwc/terms/scientificNameAuthorship
[STOP] [2021-03-25 11:32:47] Parse meta.xml file and create formats with fields
[INFO] [2021-03-25 11:32:52] ## HARVEST: type = resume_-harvest
[START] [2021-03-25 11:32:57] logged process: cb7ec98f0f4fc94af4d81126a6c79dbb3658c39a

[INFO] [2021-03-25 11:32:57] Already completed stage create_harvest_instance, skipping...
[INFO] [2021-03-25 11:32:57] Already completed stage fetch_files, skipping...
[INFO] [2021-03-25 11:32:57] Already completed stage validate_each_file, skipping...
[INFO] [2021-03-25 11:32:57] Already completed stage convert_to_csv, skipping...
[INFO] [2021-03-25 11:32:57] Already completed stage calculate_delta, skipping...
[INFO] [2021-03-25 11:32:57] Already completed stage parse_diff_and_store, skipping...
[INFO] [2021-03-25 11:32:57] Already completed stage resolve_keys, skipping...
[INFO] [2021-03-25 11:32:57] Already completed stage hold_for_later_1, skipping...
[INFO] [2021-03-25 11:32:57] Already completed stage hold_for_later_2, skipping...
[INFO] [2021-03-25 11:32:57] Already completed stage resolve_missing_parents, skipping...
[INFO] [2021-03-25 11:32:57] Already completed stage rebuild_nodes, skipping...
[INFO] [2021-03-25 11:32:57] Already completed stage resolve_missing_media_owners, skipping...
[INFO] [2021-03-25 11:32:57] Already completed stage sanitize_media_verbatims, skipping...
[INFO] [2021-03-25 11:32:57] Already completed stage queue_downloads, skipping...
[INFO] [2021-03-25 11:32:57] Already completed stage parse_names, skipping...
[INFO] [2021-03-25 11:32:57] Already completed stage denormalize_canonical_names_to_nodes, skipping...
[INFO] [2021-03-25 11:32:57] Already completed stage match_nodes, skipping...
[INFO] [2021-03-25 11:32:57] Already completed stage reindex_search, skipping...
[INFO] [2021-03-25 11:32:57] Already completed stage normalize_units, skipping...
[START] [2021-03-25 11:32:57] calculate_statistics
[ERR] [2021-03-25 11:32:57] Exception
[ERR] [2021-03-25 11:32:57] ZERO NODES! That is probably... bad.
[STOP] [2021-03-25 11:32:57] calculate_statistics
[STOP] [2021-03-25 11:32:57] logged process, took 1.54
[INFO] [2021-04-01 13:17:57] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-01 13:17:57] ## remove_type: ScientificName
[INFO] [2021-04-01 13:17:57] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-01 13:17:57] [13:17:57.713] Removed 0 Scientificnames
[INFO] [2021-04-01 13:17:57] ## remove_type: Vernacular
[INFO] [2021-04-01 13:17:57] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-01 13:17:57] [13:17:57.715] Removed 0 Vernaculars
[INFO] [2021-04-01 13:17:57] ## remove_type: Article
[INFO] [2021-04-01 13:17:57] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-01 13:17:57] [13:17:57.717] Removed 0 Articles
[INFO] [2021-04-01 13:17:57] ## remove_type: Medium
[INFO] [2021-04-01 13:17:57] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-01 13:17:57] [13:17:57.718] Removed 0 Media
[INFO] [2021-04-01 13:17:57] ## remove_type: Trait
[INFO] [2021-04-01 13:17:57] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-01 13:17:57] [13:17:57.720] Removed 0 Traits
[INFO] [2021-04-01 13:17:57] ## remove_type: MetaTrait
[INFO] [2021-04-01 13:17:57] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-01 13:17:57] [13:17:57.722] Removed 0 Metatraits
[INFO] [2021-04-01 13:17:57] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-01 13:17:57] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-01 13:17:57] [13:17:57.724] Removed 0 Occurrencemetadata
[INFO] [2021-04-01 13:17:57] ## remove_type: Assoc
[INFO] [2021-04-01 13:17:57] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-01 13:17:57] [13:17:57.725] Removed 0 Assocs
[INFO] [2021-04-01 13:17:57] ## remove_type: MetaAssoc
[INFO] [2021-04-01 13:17:57] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-01 13:17:57] [13:17:57.727] Removed 0 Metaassocs
[INFO] [2021-04-01 13:17:57] ## remove_type: Identifier
[INFO] [2021-04-01 13:17:57] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-01 13:17:57] [13:17:57.729] Removed 0 Identifiers
[INFO] [2021-04-01 13:17:57] ## remove_type: Reference
[INFO] [2021-04-01 13:17:57] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-01 13:17:57] [13:17:57.730] Removed 0 References
[INFO] [2021-04-01 13:17:57] ## remove_type: Node
[INFO] [2021-04-01 13:17:57] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-01 13:17:57] [13:17:57.743] Removed 0 Nodes
[START] [2021-04-01 13:17:58] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-01 13:17:58] Creating resource from OpenData
[START] [2021-04-01 13:18:00] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-01 13:18:00] Parse meta.xml file and create formats with fields
[WARN] [2021-04-01 13:18:01] (common) IGNORED  (nodes) field header: scientificNameAuthorship term: http://rs.tdwg.org/dwc/terms/scientificNameAuthorship
[STOP] [2021-04-01 13:18:01] Parse meta.xml file and create formats with fields
[STOP] [2021-04-01 13:18:01] Creating resource from OpenData
[START] [2021-04-01 13:18:02] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-01 13:18:02] create_harvest_instance
[INFO] [2021-04-01 13:18:02] Created harvest instance #3568
[STOP] [2021-04-01 13:18:02] create_harvest_instance
[START] [2021-04-01 13:18:02] fetch_files
[STOP] [2021-04-01 13:18:02] fetch_files
[START] [2021-04-01 13:18:02] validate_each_file
[INFO] [2021-04-01 13:18:02] Looping over 2 formats...
[INFO] [2021-04-01 13:18:02] ...nodes (/app/public/data/wikidata/taxon.tab)
[INFO] [2021-04-01 13:18:06] Valid: /app/public/converted_csv/wikidata_nodes_3568.csv (145396 lines)
[INFO] [2021-04-01 13:18:06] ...vernaculars (/app/public/data/wikidata/vernacular_name.tab)
[INFO] [2021-04-01 13:18:29] Valid: /app/public/converted_csv/wikidata_vernaculars_3568.csv (880106 lines)
[STOP] [2021-04-01 13:18:29] validate_each_file
[START] [2021-04-01 13:18:29] convert_to_csv
[INFO] [2021-04-01 13:18:29] Looping over 2 formats...
[INFO] [2021-04-01 13:18:29] ...nodes (/app/public/data/wikidata/taxon.tab)
[CMD] [2021-04-01 13:18:29] /usr/bin/sort /app/public/converted_csv/wikidata_nodes_3568.csv > /app/public/converted_csv/wikidata_nodes_3568.csv_sorted
[INFO] [2021-04-01 13:18:30] Converted: /app/public/converted_csv/wikidata_nodes_3568.csv (145396 lines)
[INFO] [2021-04-01 13:18:30] ...vernaculars (/app/public/data/wikidata/vernacular_name.tab)
[CMD] [2021-04-01 13:18:30] /usr/bin/sort /app/public/converted_csv/wikidata_vernaculars_3568.csv > /app/public/converted_csv/wikidata_vernaculars_3568.csv_sorted
[INFO] [2021-04-01 13:18:31] Converted: /app/public/converted_csv/wikidata_vernaculars_3568.csv (880106 lines)
[STOP] [2021-04-01 13:18:31] convert_to_csv
[START] [2021-04-01 13:18:31] calculate_delta
[INFO] [2021-04-01 13:18:31] Looping over 2 formats...
[INFO] [2021-04-01 13:18:31] ...nodes (/app/public/data/wikidata/taxon.tab)
[CMD] [2021-04-01 13:18:31] echo "0a" > /app/public/diff/wikidata_nodes_3568.diff
[CMD] [2021-04-01 13:18:31] tail -n +1 /app/public/converted_csv/wikidata_nodes_3568.csv >> /app/public/diff/wikidata_nodes_3568.diff
[CMD] [2021-04-01 13:18:31] echo "." >> /app/public/diff/wikidata_nodes_3568.diff
[INFO] [2021-04-01 13:18:32] Created diff: /app/public/diff/wikidata_nodes_3568.diff (145398 lines)
[INFO] [2021-04-01 13:18:32] ...vernaculars (/app/public/data/wikidata/vernacular_name.tab)
[CMD] [2021-04-01 13:18:32] echo "0a" > /app/public/diff/wikidata_vernaculars_3568.diff
[CMD] [2021-04-01 13:18:32] tail -n +1 /app/public/converted_csv/wikidata_vernaculars_3568.csv >> /app/public/diff/wikidata_vernaculars_3568.diff
[CMD] [2021-04-01 13:18:32] echo "." >> /app/public/diff/wikidata_vernaculars_3568.diff
[INFO] [2021-04-01 13:18:33] Created diff: /app/public/diff/wikidata_vernaculars_3568.diff (880108 lines)
[STOP] [2021-04-01 13:18:33] calculate_delta
[START] [2021-04-01 13:18:33] parse_diff_and_store
[INFO] [2021-04-01 13:18:33] Handling diff: /app/public/diff/wikidata_nodes_3568.diff (145398 lines)
[INFO] [2021-04-01 13:18:33] Loading nodes diff file into memory (145398 /app/public/diff/wikidata_nodes_3568.diff lines)...
[INFO] [2021-04-01 13:19:37] Handling diff: /app/public/diff/wikidata_vernaculars_3568.diff (880108 lines)
[INFO] [2021-04-01 13:19:37] Loading vernaculars diff file into memory (880108 /app/public/diff/wikidata_vernaculars_3568.diff lines)...
[INFO] [2021-04-01 13:22:51] Storing 145396 ScientificNames
[INFO] [2021-04-01 13:22:51] Processing group of 145396 in 146 groups of 1000
[INFO] [2021-04-01 13:23:58] Average Time: 0.443
[INFO] [2021-04-01 13:23:58] Total Time: 1m7s
[INFO] [2021-04-01 13:23:58] last 3 / first 3: 0.22
[INFO] [2021-04-01 13:23:58] Std.Dev: 0.324037034920393; Max: 2.09
[INFO] [2021-04-01 13:23:58] Storing 145396 Identifiers
[INFO] [2021-04-01 13:23:58] Processing group of 145396 in 146 groups of 1000
[INFO] [2021-04-01 13:24:27] Average Time: 0.186
[INFO] [2021-04-01 13:24:27] Total Time: 29s
[INFO] [2021-04-01 13:24:27] last 3 / first 3: 0.67
[INFO] [2021-04-01 13:24:27] Std.Dev: 0.14832396974191325; Max: 0.79
[INFO] [2021-04-01 13:24:27] Storing 145396 Nodes
[INFO] [2021-04-01 13:24:27] Processing group of 145396 in 146 groups of 1000
[INFO] [2021-04-01 13:25:24] Average Time: 0.38
[INFO] [2021-04-01 13:25:24] Total Time: 58s
[INFO] [2021-04-01 13:25:24] last 3 / first 3: 0.93
[INFO] [2021-04-01 13:25:24] Std.Dev: 0.27748873851023215; Max: 2.08
[INFO] [2021-04-01 13:25:24] Storing 880106 Vernaculars
[INFO] [2021-04-01 13:25:24] Processing group of 880106 in 881 groups of 1000
[INFO] [2021-04-01 13:28:21] Average Time: 0.22
[INFO] [2021-04-01 13:28:21] Total Time: 2m57s
[INFO] [2021-04-01 13:28:21] last 3 / first 3: 1.28
[INFO] [2021-04-01 13:28:21] Std.Dev: 0.31937438845342625; Max: 7.58
[STOP] [2021-04-01 13:28:21] parse_diff_and_store
[ERR] [2021-04-01 13:28:21] RuntimeError
[ERR] [2021-04-01 13:28:21] ActiveRecord::StatementInvalid while parsing something around here: [{"id":null,"resource_id":658,"harvest_id":3568,"node_id":null,"language_id":279,"node_resource_pk":"Q1582612","verbatim":"东方豹鲂𫚒","language_code_verbatim":"zh-hans","locality":null,"remarks":null,"source":"https://www.wikidata.org/wiki/Q1582612","is_preferred":true,"removed_by_harvest_id":null,"created_at":null,"updated_at":null},{"id":null,"resource_id":658,"harvest_id":3568,"node_id":null,"language_id":95,"node_resource_pk":"Q3473543","verbatim":"东方野扇花","language_code_verbatim":"zh","locality":null,"remarks":null,"source":"https://www.wikidata.org/wiki/Q3473543","is_preferred":true,"removed_by_harvest_id":null,"created_at":null,"updated_at":null},{"id":null,"resource_id":658,"harvest_id":3568,"node_id":null,"language_id":95,"node_resource_pk":"Q3281704","verbatim":"东方铁线莲","language_code_verbatim":"zh","locality":null,"remarks":null,"source":"https://www.wikidata.org/wiki/Q3281704","is_preferred":true,"removed_by_harvest_id":null,"created_at":null,"updated_at":null}]
[ERR] [2021-04-01 13:28:21] ../models/resource_harvester.rb:405:in `rescue in block (2 levels) in store_new'
[ERR] [2021-04-01 13:28:21] ../models/resource_harvester.rb:396:in `block (2 levels) in store_new'
[ERR] [2021-04-01 13:28:21] ../models/logged_process.rb:49:in `block in in_groups'
[ERR] [2021-04-01 13:28:21] ../models/logged_process.rb:47:in `in_groups'
[ERR] [2021-04-01 13:28:21] ../models/resource_harvester.rb:392:in `block in store_new'
[ERR] [2021-04-01 13:28:21] ../models/resource_harvester.rb:386:in `each'
[ERR] [2021-04-01 13:28:21] ../models/resource_harvester.rb:386:in `store_new'
[ERR] [2021-04-01 13:28:21] ../models/resource_harvester.rb:360:in `parse_diff_and_store'
[ERR] [2021-04-01 13:28:21] ../models/resource_harvester.rb:86:in `block (2 levels) in start'
[ERR] [2021-04-01 13:28:21] ../models/logged_process.rb:34:in `run_step'
[ERR] [2021-04-01 13:28:21] ../models/resource_harvester.rb:86:in `block in start'
[ERR] [2021-04-01 13:28:21] ../models/resource_harvester.rb:75:in `each_key'
[ERR] [2021-04-01 13:28:21] ../models/resource_harvester.rb:75:in `start'
[ERR] [2021-04-01 13:28:21] ../models/resource.rb:261:in `harvest'
[ERR] [2021-04-01 13:28:21] ../models/resource.rb:237:in `re_download_opendata_and_harvest'
[ERR] [2021-04-01 13:28:21] bin/rails:4:in `require'
[ERR] [2021-04-01 13:28:21] bin/rails:4:in `<main>'
[STOP] [2021-04-01 13:28:21] logged process, took 619.7
[INFO] [2021-04-16 08:02:47] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-16 08:02:48] ## remove_type: ScientificName
[INFO] [2021-04-16 08:02:48] ++ Calling delete_all on 145396 instances...
[INFO] [2021-04-16 08:03:09] [08:03:09.171] Removed 145396 Scientificnames
[INFO] [2021-04-16 08:03:09] ## remove_type: Vernacular
[INFO] [2021-04-16 08:03:09] ++ Batch removal of 774000 instances...
[INFO] [2021-04-16 08:03:09] [08:03:09.311] Batch 0...
[INFO] [2021-04-16 08:03:10] [08:03:10.114] Batch 1...
[INFO] [2021-04-16 08:03:11] [08:03:11.066] Batch 2...
[INFO] [2021-04-16 08:03:12] [08:03:12.007] Batch 3...
[INFO] [2021-04-16 08:03:12] [08:03:12.927] Batch 4...
[INFO] [2021-04-16 08:03:13] [08:03:13.726] Batch 5...
[INFO] [2021-04-16 08:03:14] [08:03:14.534] Batch 6...
[INFO] [2021-04-16 08:03:15] [08:03:15.474] Batch 7...
[INFO] [2021-04-16 08:03:16] [08:03:16.353] Batch 8...
[INFO] [2021-04-16 08:03:17] [08:03:17.328] Batch 9...
[INFO] [2021-04-16 08:03:18] [08:03:18.301] Batch 10...
[INFO] [2021-04-16 08:03:19] [08:03:19.131] Batch 11...
[INFO] [2021-04-16 08:03:20] [08:03:20.129] Batch 12...
[INFO] [2021-04-16 08:03:21] [08:03:21.070] Batch 13...
[INFO] [2021-04-16 08:03:22] [08:03:22.027] Batch 14...
[INFO] [2021-04-16 08:03:22] [08:03:22.995] Batch 15...
[INFO] [2021-04-16 08:03:24] [08:03:24.069] Batch 16...
[INFO] [2021-04-16 08:03:25] [08:03:25.093] Batch 17...
[INFO] [2021-04-16 08:03:26] [08:03:26.043] Batch 18...
[INFO] [2021-04-16 08:03:27] [08:03:27.023] Batch 19...
[INFO] [2021-04-16 08:03:28] [08:03:28.072] Batch 20...
[INFO] [2021-04-16 08:03:29] [08:03:29.121] Batch 21...
[INFO] [2021-04-16 08:03:30] [08:03:30.085] Batch 22...
[INFO] [2021-04-16 08:03:31] [08:03:31.194] Batch 23...
[INFO] [2021-04-16 08:03:32] [08:03:32.252] Batch 24...
[INFO] [2021-04-16 08:03:33] [08:03:33.204] Batch 25...
[INFO] [2021-04-16 08:03:34] [08:03:34.169] Batch 26...
[INFO] [2021-04-16 08:03:35] [08:03:35.198] Batch 27...
[INFO] [2021-04-16 08:03:36] [08:03:36.151] Batch 28...
[INFO] [2021-04-16 08:03:37] [08:03:37.252] Batch 29...
[INFO] [2021-04-16 08:03:38] [08:03:38.511] Batch 30...
[INFO] [2021-04-16 08:03:39] [08:03:39.561] Batch 31...
[INFO] [2021-04-16 08:03:40] [08:03:40.515] Batch 32...
[INFO] [2021-04-16 08:03:41] [08:03:41.453] Batch 33...
[INFO] [2021-04-16 08:03:42] [08:03:42.390] Batch 34...
[INFO] [2021-04-16 08:03:43] [08:03:43.533] Batch 35...
[INFO] [2021-04-16 08:03:44] [08:03:44.723] Batch 36...
[INFO] [2021-04-16 08:03:46] [08:03:46.405] Batch 37...
[INFO] [2021-04-16 08:03:47] [08:03:47.639] Batch 38...
[INFO] [2021-04-16 08:03:48] [08:03:48.847] Batch 39...
[INFO] [2021-04-16 08:03:49] [08:03:49.553] Batch 40...
[INFO] [2021-04-16 08:03:50] [08:03:50.208] Batch 41...
[INFO] [2021-04-16 08:03:50] [08:03:50.838] Batch 42...
[INFO] [2021-04-16 08:03:51] [08:03:51.469] Batch 43...
[INFO] [2021-04-16 08:03:52] [08:03:52.093] Batch 44...
[INFO] [2021-04-16 08:03:52] [08:03:52.715] Batch 45...
[INFO] [2021-04-16 08:03:53] [08:03:53.396] Batch 46...
[INFO] [2021-04-16 08:03:54] [08:03:54.057] Batch 47...
[INFO] [2021-04-16 08:03:54] [08:03:54.684] Batch 48...
[INFO] [2021-04-16 08:03:55] [08:03:55.305] Batch 49...
[INFO] [2021-04-16 08:03:55] [08:03:55.931] Batch 50...
[INFO] [2021-04-16 08:03:56] [08:03:56.549] Batch 51...
[INFO] [2021-04-16 08:03:57] [08:03:57.160] Batch 52...
[INFO] [2021-04-16 08:03:57] [08:03:57.770] Batch 53...
[INFO] [2021-04-16 08:03:58] [08:03:58.384] Batch 54...
[INFO] [2021-04-16 08:03:58] [08:03:58.992] Batch 55...
[INFO] [2021-04-16 08:03:59] [08:03:59.600] Batch 56...
[INFO] [2021-04-16 08:04:00] [08:04:00.210] Batch 57...
[INFO] [2021-04-16 08:04:00] [08:04:00.814] Batch 58...
[INFO] [2021-04-16 08:04:01] [08:04:01.417] Batch 59...
[INFO] [2021-04-16 08:04:02] [08:04:02.020] Batch 60...
[INFO] [2021-04-16 08:04:02] [08:04:02.619] Batch 61...
[INFO] [2021-04-16 08:04:03] [08:04:03.217] Batch 62...
[INFO] [2021-04-16 08:04:03] [08:04:03.813] Batch 63...
[INFO] [2021-04-16 08:04:04] [08:04:04.410] Batch 64...
[INFO] [2021-04-16 08:04:05] [08:04:05.006] Batch 65...
[INFO] [2021-04-16 08:04:05] [08:04:05.599] Batch 66...
[INFO] [2021-04-16 08:04:06] [08:04:06.186] Batch 67...
[INFO] [2021-04-16 08:04:06] [08:04:06.769] Batch 68...
[INFO] [2021-04-16 08:04:07] [08:04:07.354] Batch 69...
[INFO] [2021-04-16 08:04:07] [08:04:07.955] Batch 70...
[INFO] [2021-04-16 08:04:08] [08:04:08.532] Batch 71...
[INFO] [2021-04-16 08:04:09] [08:04:09.110] Batch 72...
[INFO] [2021-04-16 08:04:09] [08:04:09.686] Batch 73...
[INFO] [2021-04-16 08:04:10] [08:04:10.261] Batch 74...
[INFO] [2021-04-16 08:04:10] [08:04:10.833] Batch 75...
[INFO] [2021-04-16 08:04:11] [08:04:11.401] Batch 76...
[INFO] [2021-04-16 08:04:11] [08:04:11.973] Batch 77...
[INFO] [2021-04-16 08:04:12] [08:04:12.500] Removed 774000 Vernaculars
[INFO] [2021-04-16 08:04:12] ## remove_type: Article
[INFO] [2021-04-16 08:04:12] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-16 08:04:12] [08:04:12.503] Removed 0 Articles
[INFO] [2021-04-16 08:04:12] ## remove_type: Medium
[INFO] [2021-04-16 08:04:12] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-16 08:04:12] [08:04:12.507] Removed 0 Media
[INFO] [2021-04-16 08:04:12] ## remove_type: Trait
[INFO] [2021-04-16 08:04:12] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-16 08:04:12] [08:04:12.511] Removed 0 Traits
[INFO] [2021-04-16 08:04:12] ## remove_type: MetaTrait
[INFO] [2021-04-16 08:04:12] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-16 08:04:12] [08:04:12.514] Removed 0 Metatraits
[INFO] [2021-04-16 08:04:12] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-16 08:04:12] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-16 08:04:12] [08:04:12.563] Removed 0 Occurrencemetadata
[INFO] [2021-04-16 08:04:12] ## remove_type: Assoc
[INFO] [2021-04-16 08:04:12] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-16 08:04:12] [08:04:12.567] Removed 0 Assocs
[INFO] [2021-04-16 08:04:12] ## remove_type: MetaAssoc
[INFO] [2021-04-16 08:04:12] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-16 08:04:12] [08:04:12.570] Removed 0 Metaassocs
[INFO] [2021-04-16 08:04:12] ## remove_type: Identifier
[INFO] [2021-04-16 08:04:12] ++ Calling delete_all on 145396 instances...
[INFO] [2021-04-16 08:04:22] [08:04:22.675] Removed 145396 Identifiers
[INFO] [2021-04-16 08:04:22] ## remove_type: Reference
[INFO] [2021-04-16 08:04:22] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-16 08:04:22] [08:04:22.677] Removed 0 References
[INFO] [2021-04-16 08:04:23] ## remove_type: Node
[INFO] [2021-04-16 08:04:23] ++ Calling delete_all on 145396 instances...
[INFO] [2021-04-16 08:04:39] [08:04:39.886] Removed 145396 Nodes
[START] [2021-04-16 08:04:40] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-16 08:04:40] Creating resource from OpenData
[START] [2021-04-16 08:04:42] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-16 08:04:42] Parse meta.xml file and create formats with fields
[WARN] [2021-04-16 08:04:42] (common) IGNORED  (nodes) field header: scientificNameAuthorship term: http://rs.tdwg.org/dwc/terms/scientificNameAuthorship
[STOP] [2021-04-16 08:04:42] Parse meta.xml file and create formats with fields
[STOP] [2021-04-16 08:04:42] Creating resource from OpenData
[START] [2021-04-16 08:04:42] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-16 08:04:42] create_harvest_instance
[INFO] [2021-04-16 08:04:42] Created harvest instance #3707
[STOP] [2021-04-16 08:04:42] create_harvest_instance
[START] [2021-04-16 08:04:42] fetch_files
[STOP] [2021-04-16 08:04:42] fetch_files
[START] [2021-04-16 08:04:42] validate_each_file
[INFO] [2021-04-16 08:04:42] Looping over 2 formats...
[INFO] [2021-04-16 08:04:42] ...nodes (/app/public/data/wikidata/taxon.tab)
[INFO] [2021-04-16 08:04:46] Valid: /app/public/converted_csv/wikidata_nodes_3707.csv (147435 lines)
[INFO] [2021-04-16 08:04:46] ...vernaculars (/app/public/data/wikidata/vernacular_name.tab)
[INFO] [2021-04-16 08:05:06] Valid: /app/public/converted_csv/wikidata_vernaculars_3707.csv (893189 lines)
[STOP] [2021-04-16 08:05:06] validate_each_file
[START] [2021-04-16 08:05:06] convert_to_csv
[INFO] [2021-04-16 08:05:06] Looping over 2 formats...
[INFO] [2021-04-16 08:05:06] ...nodes (/app/public/data/wikidata/taxon.tab)
[CMD] [2021-04-16 08:05:06] /usr/bin/sort /app/public/converted_csv/wikidata_nodes_3707.csv > /app/public/converted_csv/wikidata_nodes_3707.csv_sorted
[INFO] [2021-04-16 08:05:06] Converted: /app/public/converted_csv/wikidata_nodes_3707.csv (147435 lines)
[INFO] [2021-04-16 08:05:06] ...vernaculars (/app/public/data/wikidata/vernacular_name.tab)
[CMD] [2021-04-16 08:05:06] /usr/bin/sort /app/public/converted_csv/wikidata_vernaculars_3707.csv > /app/public/converted_csv/wikidata_vernaculars_3707.csv_sorted
[INFO] [2021-04-16 08:05:07] Converted: /app/public/converted_csv/wikidata_vernaculars_3707.csv (893189 lines)
[STOP] [2021-04-16 08:05:07] convert_to_csv
[START] [2021-04-16 08:05:07] calculate_delta
[INFO] [2021-04-16 08:05:07] Looping over 2 formats...
[INFO] [2021-04-16 08:05:07] ...nodes (/app/public/data/wikidata/taxon.tab)
[CMD] [2021-04-16 08:05:07] echo "0a" > /app/public/diff/wikidata_nodes_3707.diff
[CMD] [2021-04-16 08:05:07] tail -n +1 /app/public/converted_csv/wikidata_nodes_3707.csv >> /app/public/diff/wikidata_nodes_3707.diff
[CMD] [2021-04-16 08:05:07] echo "." >> /app/public/diff/wikidata_nodes_3707.diff
[INFO] [2021-04-16 08:05:07] Created diff: /app/public/diff/wikidata_nodes_3707.diff (147437 lines)
[INFO] [2021-04-16 08:05:07] ...vernaculars (/app/public/data/wikidata/vernacular_name.tab)
[CMD] [2021-04-16 08:05:07] echo "0a" > /app/public/diff/wikidata_vernaculars_3707.diff
[CMD] [2021-04-16 08:05:07] tail -n +1 /app/public/converted_csv/wikidata_vernaculars_3707.csv >> /app/public/diff/wikidata_vernaculars_3707.diff
[CMD] [2021-04-16 08:05:08] echo "." >> /app/public/diff/wikidata_vernaculars_3707.diff
[INFO] [2021-04-16 08:05:08] Created diff: /app/public/diff/wikidata_vernaculars_3707.diff (893191 lines)
[STOP] [2021-04-16 08:05:08] calculate_delta
[START] [2021-04-16 08:05:08] parse_diff_and_store
[INFO] [2021-04-16 08:05:08] Handling diff: /app/public/diff/wikidata_nodes_3707.diff (147437 lines)
[INFO] [2021-04-16 08:05:08] Loading nodes diff file into memory (147437 /app/public/diff/wikidata_nodes_3707.diff lines)...
[INFO] [2021-04-16 08:06:11] Handling diff: /app/public/diff/wikidata_vernaculars_3707.diff (893191 lines)
[INFO] [2021-04-16 08:06:11] Loading vernaculars diff file into memory (893191 /app/public/diff/wikidata_vernaculars_3707.diff lines)...
[INFO] [2021-04-16 08:09:13] Storing 147435 ScientificNames
[INFO] [2021-04-16 08:09:13] Processing group of 147435 in 148 groups of 1000
[INFO] [2021-04-16 08:10:02] Average Time: 0.323
[INFO] [2021-04-16 08:10:02] Total Time: 49s
[INFO] [2021-04-16 08:10:02] last 3 / first 3: 0.94
[INFO] [2021-04-16 08:10:02] Std.Dev: 0.20976176963403032; Max: 1.62
[INFO] [2021-04-16 08:10:02] Storing 147435 Identifiers
[INFO] [2021-04-16 08:10:02] Processing group of 147435 in 148 groups of 1000
[INFO] [2021-04-16 08:10:15] Average Time: 0.086
[INFO] [2021-04-16 08:10:15] Total Time: 14s
[INFO] [2021-04-16 08:10:15] last 3 / first 3: 0.79
[INFO] [2021-04-16 08:10:15] Std.Dev: 0.03162277660168379; Max: 0.29
[INFO] [2021-04-16 08:10:15] Storing 147435 Nodes
[INFO] [2021-04-16 08:10:15] Processing group of 147435 in 148 groups of 1000
[INFO] [2021-04-16 08:11:12] Average Time: 0.385
[INFO] [2021-04-16 08:11:12] Total Time: 58s
[INFO] [2021-04-16 08:11:12] last 3 / first 3: 0.64
[INFO] [2021-04-16 08:11:12] Std.Dev: 0.7536577472566709; Max: 6.6
[INFO] [2021-04-16 08:11:12] Storing 893189 Vernaculars
[INFO] [2021-04-16 08:11:12] Processing group of 893189 in 894 groups of 1000
[INFO] [2021-04-16 08:13:39] Average Time: 0.18
[INFO] [2021-04-16 08:13:39] Total Time: 2m27s
[INFO] [2021-04-16 08:13:39] last 3 / first 3: 1.29
[INFO] [2021-04-16 08:13:39] Std.Dev: 0.24289915602982237; Max: 6.92
[STOP] [2021-04-16 08:13:39] parse_diff_and_store
[ERR] [2021-04-16 08:13:39] RuntimeError
[ERR] [2021-04-16 08:13:39] ActiveRecord::StatementInvalid while parsing something around here: [{"id":null,"resource_id":658,"harvest_id":3707,"node_id":null,"language_id":279,"node_resource_pk":"Q1582612","verbatim":"东方豹鲂𫚒","language_code_verbatim":"zh-hans","locality":null,"remarks":null,"source":"https://www.wikidata.org/wiki/Q1582612","is_preferred":true,"removed_by_harvest_id":null,"created_at":null,"updated_at":null},{"id":null,"resource_id":658,"harvest_id":3707,"node_id":null,"language_id":95,"node_resource_pk":"Q3473543","verbatim":"东方野扇花","language_code_verbatim":"zh","locality":null,"remarks":null,"source":"https://www.wikidata.org/wiki/Q3473543","is_preferred":true,"removed_by_harvest_id":null,"created_at":null,"updated_at":null},{"id":null,"resource_id":658,"harvest_id":3707,"node_id":null,"language_id":95,"node_resource_pk":"Q3281704","verbatim":"东方铁线莲","language_code_verbatim":"zh","locality":null,"remarks":null,"source":"https://www.wikidata.org/wiki/Q3281704","is_preferred":true,"removed_by_harvest_id":null,"created_at":null,"updated_at":null}]
[ERR] [2021-04-16 08:13:39] ../models/resource_harvester.rb:405:in `rescue in block (2 levels) in store_new'
[ERR] [2021-04-16 08:13:39] ../models/resource_harvester.rb:396:in `block (2 levels) in store_new'
[ERR] [2021-04-16 08:13:39] ../models/logged_process.rb:49:in `block in in_groups'
[ERR] [2021-04-16 08:13:39] ../models/logged_process.rb:47:in `in_groups'
[ERR] [2021-04-16 08:13:39] ../models/resource_harvester.rb:392:in `block in store_new'
[ERR] [2021-04-16 08:13:39] ../models/resource_harvester.rb:386:in `each'
[ERR] [2021-04-16 08:13:39] ../models/resource_harvester.rb:386:in `store_new'
[ERR] [2021-04-16 08:13:39] ../models/resource_harvester.rb:360:in `parse_diff_and_store'
[ERR] [2021-04-16 08:13:39] ../models/resource_harvester.rb:86:in `block (2 levels) in start'
[ERR] [2021-04-16 08:13:39] ../models/logged_process.rb:34:in `run_step'
[ERR] [2021-04-16 08:13:39] ../models/resource_harvester.rb:86:in `block in start'
[ERR] [2021-04-16 08:13:39] ../models/resource_harvester.rb:75:in `each_key'
[ERR] [2021-04-16 08:13:39] ../models/resource_harvester.rb:75:in `start'
[ERR] [2021-04-16 08:13:39] ../models/resource.rb:261:in `harvest'
[ERR] [2021-04-16 08:13:39] ../models/resource.rb:237:in `re_download_opendata_and_harvest'
[ERR] [2021-04-16 08:13:39] bin/rails:4:in `require'
[ERR] [2021-04-16 08:13:39] bin/rails:4:in `<main>'
[STOP] [2021-04-16 08:13:39] logged process, took 536.87
[INFO] [2021-05-04 10:15:46] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-05-04 10:15:51] ## remove_type: ScientificName
[INFO] [2021-05-04 10:15:51] ++ Calling delete_all on 147435 instances...
[INFO] [2021-05-04 10:16:12] [10:16:12.408] Removed 147435 Scientificnames
[INFO] [2021-05-04 10:16:12] ## remove_type: Vernacular
[INFO] [2021-05-04 10:16:12] ++ Batch removal of 786000 instances...
[INFO] [2021-05-04 10:16:12] [10:16:12.541] Batch 0...
[INFO] [2021-05-04 10:16:13] [10:16:13.322] Batch 1...
[INFO] [2021-05-04 10:16:14] [10:16:14.124] Batch 2...
[INFO] [2021-05-04 10:16:15] [10:16:15.007] Batch 3...
[INFO] [2021-05-04 10:16:16] [10:16:16.025] Batch 4...
[INFO] [2021-05-04 10:16:16] [10:16:16.818] Batch 5...
[INFO] [2021-05-04 10:16:17] [10:16:17.603] Batch 6...
[INFO] [2021-05-04 10:16:18] [10:16:18.397] Batch 7...
[INFO] [2021-05-04 10:16:19] [10:16:19.656] Batch 8...
[INFO] [2021-05-04 10:16:20] [10:16:20.496] Batch 9...
[INFO] [2021-05-04 10:16:21] [10:16:21.310] Batch 10...
[INFO] [2021-05-04 10:16:22] [10:16:22.648] Batch 11...
[INFO] [2021-05-04 10:16:23] [10:16:23.477] Batch 12...
[INFO] [2021-05-04 10:16:24] [10:16:24.312] Batch 13...
[INFO] [2021-05-04 10:16:25] [10:16:25.201] Batch 14...
[INFO] [2021-05-04 10:16:26] [10:16:26.034] Batch 15...
[INFO] [2021-05-04 10:16:26] [10:16:26.901] Batch 16...
[INFO] [2021-05-04 10:16:27] [10:16:27.738] Batch 17...
[INFO] [2021-05-04 10:16:28] [10:16:28.580] Batch 18...
[INFO] [2021-05-04 10:16:29] [10:16:29.422] Batch 19...
[INFO] [2021-05-04 10:16:30] [10:16:30.283] Batch 20...
[INFO] [2021-05-04 10:16:31] [10:16:31.140] Batch 21...
[INFO] [2021-05-04 10:16:32] [10:16:32.061] Batch 22...
[INFO] [2021-05-04 10:16:33] [10:16:33.040] Batch 23...
[INFO] [2021-05-04 10:16:34] [10:16:34.050] Batch 24...
[INFO] [2021-05-04 10:16:35] [10:16:35.022] Batch 25...
[INFO] [2021-05-04 10:16:35] [10:16:35.981] Batch 26...
[INFO] [2021-05-04 10:16:36] [10:16:36.953] Batch 27...
[INFO] [2021-05-04 10:16:37] [10:16:37.937] Batch 28...
[INFO] [2021-05-04 10:16:39] [10:16:39.208] Batch 29...
[INFO] [2021-05-04 10:16:40] [10:16:40.165] Batch 30...
[INFO] [2021-05-04 10:16:41] [10:16:41.252] Batch 31...
[INFO] [2021-05-04 10:16:42] [10:16:42.274] Batch 32...
[INFO] [2021-05-04 10:16:43] [10:16:43.317] Batch 33...
[INFO] [2021-05-04 10:16:44] [10:16:44.523] Batch 34...
[INFO] [2021-05-04 10:16:45] [10:16:45.477] Batch 35...
[INFO] [2021-05-04 10:16:46] [10:16:46.448] Batch 36...
[INFO] [2021-05-04 10:16:47] [10:16:47.378] Batch 37...
[INFO] [2021-05-04 10:16:48] [10:16:48.385] Batch 38...
[INFO] [2021-05-04 10:16:49] [10:16:49.447] Batch 39...
[INFO] [2021-05-04 10:16:50] [10:16:50.481] Batch 40...
[INFO] [2021-05-04 10:16:51] [10:16:51.442] Batch 41...
[INFO] [2021-05-04 10:16:52] [10:16:52.415] Batch 42...
[INFO] [2021-05-04 10:16:53] [10:16:53.402] Batch 43...
[INFO] [2021-05-04 10:16:54] [10:16:54.955] Batch 44...
[INFO] [2021-05-04 10:16:56] [10:16:56.471] Batch 45...
[INFO] [2021-05-04 10:16:58] [10:16:58.047] Batch 46...
[INFO] [2021-05-04 10:16:59] [10:16:59.592] Batch 47...
[INFO] [2021-05-04 10:17:00] [10:17:00.552] Batch 48...
[INFO] [2021-05-04 10:17:01] [10:17:01.421] Batch 49...
[INFO] [2021-05-04 10:17:02] [10:17:02.193] Batch 50...
[INFO] [2021-05-04 10:17:02] [10:17:02.812] Batch 51...
[INFO] [2021-05-04 10:17:03] [10:17:03.458] Batch 52...
[INFO] [2021-05-04 10:17:04] [10:17:04.076] Batch 53...
[INFO] [2021-05-04 10:17:04] [10:17:04.688] Batch 54...
[INFO] [2021-05-04 10:17:05] [10:17:05.300] Batch 55...
[INFO] [2021-05-04 10:17:05] [10:17:05.912] Batch 56...
[INFO] [2021-05-04 10:17:06] [10:17:06.521] Batch 57...
[INFO] [2021-05-04 10:17:07] [10:17:07.125] Batch 58...
[INFO] [2021-05-04 10:17:07] [10:17:07.725] Batch 59...
[INFO] [2021-05-04 10:17:08] [10:17:08.333] Batch 60...
[INFO] [2021-05-04 10:17:08] [10:17:08.928] Batch 61...
[INFO] [2021-05-04 10:17:09] [10:17:09.525] Batch 62...
[INFO] [2021-05-04 10:17:10] [10:17:10.121] Batch 63...
[INFO] [2021-05-04 10:17:10] [10:17:10.714] Batch 64...
[INFO] [2021-05-04 10:17:11] [10:17:11.305] Batch 65...
[INFO] [2021-05-04 10:17:11] [10:17:11.896] Batch 66...
[INFO] [2021-05-04 10:17:12] [10:17:12.482] Batch 67...
[INFO] [2021-05-04 10:17:13] [10:17:13.073] Batch 68...
[INFO] [2021-05-04 10:17:13] [10:17:13.656] Batch 69...
[INFO] [2021-05-04 10:17:14] [10:17:14.243] Batch 70...
[INFO] [2021-05-04 10:17:14] [10:17:14.828] Batch 71...
[INFO] [2021-05-04 10:17:15] [10:17:15.408] Batch 72...
[INFO] [2021-05-04 10:17:16] [10:17:16.003] Batch 73...
[INFO] [2021-05-04 10:17:16] [10:17:16.576] Batch 74...
[INFO] [2021-05-04 10:17:17] [10:17:17.149] Batch 75...
[INFO] [2021-05-04 10:17:17] [10:17:17.718] Batch 76...
[INFO] [2021-05-04 10:17:18] [10:17:18.286] Batch 77...
[INFO] [2021-05-04 10:17:18] [10:17:18.847] Batch 78...
[INFO] [2021-05-04 10:17:19] [10:17:19.385] Removed 786000 Vernaculars
[INFO] [2021-05-04 10:17:19] ## remove_type: Article
[INFO] [2021-05-04 10:17:19] ++ Calling delete_all on 0 instances...
[INFO] [2021-05-04 10:17:19] [10:17:19.387] Removed 0 Articles
[INFO] [2021-05-04 10:17:19] ## remove_type: Medium
[INFO] [2021-05-04 10:17:19] ++ Calling delete_all on 0 instances...
[INFO] [2021-05-04 10:17:19] [10:17:19.388] Removed 0 Media
[INFO] [2021-05-04 10:17:19] ## remove_type: Trait
[INFO] [2021-05-04 10:17:19] ++ Calling delete_all on 0 instances...
[INFO] [2021-05-04 10:17:19] [10:17:19.390] Removed 0 Traits
[INFO] [2021-05-04 10:17:19] ## remove_type: MetaTrait
[INFO] [2021-05-04 10:17:19] ++ Calling delete_all on 0 instances...
[INFO] [2021-05-04 10:17:19] [10:17:19.403] Removed 0 Metatraits
[INFO] [2021-05-04 10:17:19] ## remove_type: OccurrenceMetadatum
[INFO] [2021-05-04 10:17:19] ++ Calling delete_all on 0 instances...
[INFO] [2021-05-04 10:17:19] [10:17:19.423] Removed 0 Occurrencemetadata
[INFO] [2021-05-04 10:17:19] ## remove_type: Assoc
[INFO] [2021-05-04 10:17:19] ++ Calling delete_all on 0 instances...
[INFO] [2021-05-04 10:17:19] [10:17:19.424] Removed 0 Assocs
[INFO] [2021-05-04 10:17:19] ## remove_type: MetaAssoc
[INFO] [2021-05-04 10:17:19] ++ Calling delete_all on 0 instances...
[INFO] [2021-05-04 10:17:19] [10:17:19.425] Removed 0 Metaassocs
[INFO] [2021-05-04 10:17:19] ## remove_type: Identifier
[INFO] [2021-05-04 10:17:19] ++ Calling delete_all on 147435 instances...
[INFO] [2021-05-04 10:17:29] [10:17:29.593] Removed 147435 Identifiers
[INFO] [2021-05-04 10:17:29] ## remove_type: Reference
[INFO] [2021-05-04 10:17:29] ++ Calling delete_all on 0 instances...
[INFO] [2021-05-04 10:17:29] [10:17:29.594] Removed 0 References
[INFO] [2021-05-04 10:17:29] ## remove_type: Node
[INFO] [2021-05-04 10:17:29] ++ Calling delete_all on 147435 instances...
[INFO] [2021-05-04 10:17:54] [10:17:54.699] Removed 147435 Nodes
[START] [2021-05-04 10:17:54] logged process: 72e1f5f017f3ddba5951efc2ecd831d5ae3d10b1 Unmatched nodes file now has a link at the bottom of the resources page and includes the resource_pk.
[START] [2021-05-04 10:17:54] Creating resource from OpenData
[START] [2021-05-04 10:17:56] logged process: 72e1f5f017f3ddba5951efc2ecd831d5ae3d10b1 Unmatched nodes file now has a link at the bottom of the resources page and includes the resource_pk.
[START] [2021-05-04 10:17:56] Parse meta.xml file and create formats with fields
[WARN] [2021-05-04 10:17:57] (common) IGNORED  (nodes) field header: scientificNameAuthorship term: http://rs.tdwg.org/dwc/terms/scientificNameAuthorship
[STOP] [2021-05-04 10:17:57] Parse meta.xml file and create formats with fields
[STOP] [2021-05-04 10:17:57] Creating resource from OpenData
[START] [2021-05-04 10:17:57] logged process: 72e1f5f017f3ddba5951efc2ecd831d5ae3d10b1 Unmatched nodes file now has a link at the bottom of the resources page and includes the resource_pk.
[START] [2021-05-04 10:17:57] create_harvest_instance
[INFO] [2021-05-04 10:17:57] Created harvest instance #3847
[STOP] [2021-05-04 10:17:57] create_harvest_instance
[START] [2021-05-04 10:17:57] fetch_files
[STOP] [2021-05-04 10:17:57] fetch_files
[START] [2021-05-04 10:17:57] validate_each_file
[INFO] [2021-05-04 10:17:57] Looping over 2 formats...
[INFO] [2021-05-04 10:17:57] ...nodes (/app/public/data/wikidata/taxon.tab)
[INFO] [2021-05-04 10:18:01] Valid: /app/public/converted_csv/wikidata_nodes_3847.csv (147435 lines)
[INFO] [2021-05-04 10:18:01] ...vernaculars (/app/public/data/wikidata/vernacular_name.tab)
[INFO] [2021-05-04 10:18:22] Valid: /app/public/converted_csv/wikidata_vernaculars_3847.csv (893189 lines)
[STOP] [2021-05-04 10:18:22] validate_each_file
[START] [2021-05-04 10:18:22] convert_to_csv
[INFO] [2021-05-04 10:18:22] Looping over 2 formats...
[INFO] [2021-05-04 10:18:22] ...nodes (/app/public/data/wikidata/taxon.tab)
[CMD] [2021-05-04 10:18:22] /usr/bin/sort /app/public/converted_csv/wikidata_nodes_3847.csv > /app/public/converted_csv/wikidata_nodes_3847.csv_sorted
[INFO] [2021-05-04 10:18:22] Converted: /app/public/converted_csv/wikidata_nodes_3847.csv (147435 lines)
[INFO] [2021-05-04 10:18:22] ...vernaculars (/app/public/data/wikidata/vernacular_name.tab)
[CMD] [2021-05-04 10:18:22] /usr/bin/sort /app/public/converted_csv/wikidata_vernaculars_3847.csv > /app/public/converted_csv/wikidata_vernaculars_3847.csv_sorted
[INFO] [2021-05-04 10:18:23] Converted: /app/public/converted_csv/wikidata_vernaculars_3847.csv (893189 lines)
[STOP] [2021-05-04 10:18:23] convert_to_csv
[START] [2021-05-04 10:18:23] calculate_delta
[INFO] [2021-05-04 10:18:23] Looping over 2 formats...
[INFO] [2021-05-04 10:18:23] ...nodes (/app/public/data/wikidata/taxon.tab)
[CMD] [2021-05-04 10:18:23] echo "0a" > /app/public/diff/wikidata_nodes_3847.diff
[CMD] [2021-05-04 10:18:23] tail -n +1 /app/public/converted_csv/wikidata_nodes_3847.csv >> /app/public/diff/wikidata_nodes_3847.diff
[CMD] [2021-05-04 10:18:23] echo "." >> /app/public/diff/wikidata_nodes_3847.diff
[INFO] [2021-05-04 10:18:23] Created diff: /app/public/diff/wikidata_nodes_3847.diff (147437 lines)
[INFO] [2021-05-04 10:18:23] ...vernaculars (/app/public/data/wikidata/vernacular_name.tab)
[CMD] [2021-05-04 10:18:23] echo "0a" > /app/public/diff/wikidata_vernaculars_3847.diff
[CMD] [2021-05-04 10:18:23] tail -n +1 /app/public/converted_csv/wikidata_vernaculars_3847.csv >> /app/public/diff/wikidata_vernaculars_3847.diff
[CMD] [2021-05-04 10:18:23] echo "." >> /app/public/diff/wikidata_vernaculars_3847.diff
[INFO] [2021-05-04 10:18:24] Created diff: /app/public/diff/wikidata_vernaculars_3847.diff (893191 lines)
[STOP] [2021-05-04 10:18:24] calculate_delta
[START] [2021-05-04 10:18:24] parse_diff_and_store
[INFO] [2021-05-04 10:18:24] Handling diff: /app/public/diff/wikidata_nodes_3847.diff (147437 lines)
[INFO] [2021-05-04 10:18:24] Loading nodes diff file into memory (147437 /app/public/diff/wikidata_nodes_3847.diff lines)...
[INFO] [2021-05-04 10:19:25] Handling diff: /app/public/diff/wikidata_vernaculars_3847.diff (893191 lines)
[INFO] [2021-05-04 10:19:25] Loading vernaculars diff file into memory (893191 /app/public/diff/wikidata_vernaculars_3847.diff lines)...
[INFO] [2021-05-04 10:22:25] Storing 147435 ScientificNames
[INFO] [2021-05-04 10:22:25] Processing group of 147435 in 148 groups of 1000
[INFO] [2021-05-04 10:23:14] Average Time: 0.324
[INFO] [2021-05-04 10:23:14] Total Time: 49s
[INFO] [2021-05-04 10:23:14] last 3 / first 3: 0.78
[INFO] [2021-05-04 10:23:14] Std.Dev: 0.21213203435596426; Max: 1.76
[INFO] [2021-05-04 10:23:14] Storing 147435 Identifiers
[INFO] [2021-05-04 10:23:14] Processing group of 147435 in 148 groups of 1000
[INFO] [2021-05-04 10:23:27] Average Time: 0.085
[INFO] [2021-05-04 10:23:27] Total Time: 14s
[INFO] [2021-05-04 10:23:27] last 3 / first 3: 0.8
[INFO] [2021-05-04 10:23:27] Std.Dev: 0.03162277660168379; Max: 0.29
[INFO] [2021-05-04 10:23:27] Storing 147435 Nodes
[INFO] [2021-05-04 10:23:27] Processing group of 147435 in 148 groups of 1000
[INFO] [2021-05-04 10:24:25] Average Time: 0.388
[INFO] [2021-05-04 10:24:25] Total Time: 59s
[INFO] [2021-05-04 10:24:25] last 3 / first 3: 1.15
[INFO] [2021-05-04 10:24:25] Std.Dev: 0.7529940238806679; Max: 6.67
[INFO] [2021-05-04 10:24:25] Storing 893189 Vernaculars
[INFO] [2021-05-04 10:24:25] Processing group of 893189 in 894 groups of 1000
[INFO] [2021-05-04 10:27:18] Average Time: 0.187
[INFO] [2021-05-04 10:27:18] Total Time: 2m53s
[INFO] [2021-05-04 10:27:18] last 3 / first 3: 0.73
[INFO] [2021-05-04 10:27:18] Std.Dev: 0.18439088914585774; Max: 4.88
[STOP] [2021-05-04 10:27:18] parse_diff_and_store
[START] [2021-05-04 10:27:18] resolve_keys
[INFO] [2021-05-04 10:29:27] Occurrences to nodes (through scientific_names)...
[INFO] [2021-05-04 10:29:27] traits to occurrences...
[INFO] [2021-05-04 10:29:27] traits to nodes (through occurrences)...
[INFO] [2021-05-04 10:29:27] Traits to sex term...
[INFO] [2021-05-04 10:29:27] Traits to lifestage term...
[INFO] [2021-05-04 10:29:27] MetaTraits to traits...
[INFO] [2021-05-04 10:29:27] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-05-04 10:29:27] Assocs to occurrences...
[INFO] [2021-05-04 10:29:27] Assocs to nodes...
[INFO] [2021-05-04 10:29:27] Assoc to sex term...
[INFO] [2021-05-04 10:29:27] Assoc to lifestage term...
[INFO] [2021-05-04 10:29:27] MetaAssoc to assocs...
[STOP] [2021-05-04 10:29:27] resolve_keys
[START] [2021-05-04 10:29:27] hold_for_later_1
[STOP] [2021-05-04 10:29:27] hold_for_later_1
[START] [2021-05-04 10:29:27] hold_for_later_2
[STOP] [2021-05-04 10:29:27] hold_for_later_2
[START] [2021-05-04 10:29:27] resolve_missing_parents
[STOP] [2021-05-04 10:29:34] resolve_missing_parents
[START] [2021-05-04 10:29:34] rebuild_nodes
[START] [2021-05-04 10:29:34] Flattener#flatten
[START] [2021-05-04 10:29:34] Flattener#study_resource
[START] [2021-05-04 10:29:58] Flattener#build_ancestry
[STOP] [2021-05-04 10:32:23] Flattener#build_ancestry
[INFO] [2021-05-04 10:32:23] 147435 ancestry keys
[START] [2021-05-04 10:32:23] build_node_ancestors
[INFO] [2021-05-04 10:32:23] old ancestors deleted.
[STOP] [2021-05-04 10:38:15] build_node_ancestors
[START] [2021-05-04 10:38:20] Flattener#propagate_ancestor_ids
[STOP] [2021-05-04 10:39:53] Flattener#propagate_ancestor_ids
[STOP] [2021-05-04 10:39:53] Flattener#flatten
[STOP] [2021-05-04 10:39:53] rebuild_nodes
[START] [2021-05-04 10:39:53] resolve_missing_media_owners
[STOP] [2021-05-04 10:39:53] resolve_missing_media_owners
[START] [2021-05-04 10:39:53] sanitize_media_verbatims
[STOP] [2021-05-04 10:39:53] sanitize_media_verbatims
[START] [2021-05-04 10:39:53] queue_downloads
[STOP] [2021-05-04 10:39:53] queue_downloads
[START] [2021-05-04 10:39:53] parse_names
[WARN] [2021-05-04 10:39:53] I see 147435 names which still need to be parsed.
[WARN] [2021-05-04 10:41:38] I see 37 names which still need to be parsed.
[STOP] [2021-05-04 10:41:39] parse_names
[START] [2021-05-04 10:41:39] denormalize_canonical_names_to_nodes
[STOP] [2021-05-04 10:41:42] denormalize_canonical_names_to_nodes
[START] [2021-05-04 10:41:42] match_nodes
[START] [2021-05-04 10:41:42] map_all_nodes_to_pages
[STOP] [2021-05-04 12:32:13] map_all_nodes_to_pages
[INFO] [2021-05-04 12:32:14] 4465 Unmatched nodes (of 147435)! That's too many to output. Full list in /app/public/data/wikidata/unmatched_nodes.txt ; First 10: Canonical: Canadaspidida; Node#93406449; ResourceID: Q21292503; Canonical: Canadaspididae; Node#93413279; ResourceID: Q24192185; Canonical: Canadapsis perfecta; Node#93421487; ResourceID: Q2731469; Canonical: Biota; Node#93412561; ResourceID: Q2382443; Canonical: Acytota; Node#93390748; ResourceID: Q169731; Canonical: Prokaryota; Node#93398722; ResourceID: Q19081; Canonical: Proteoarchaeota; Node#93406390; ResourceID: Q21282292; Canonical: Bacteria; Node#93354051; ResourceID: Q10876; Canonical: Negibacteria; Node#93435733; ResourceID: Q3337759; Canonical: Thermotogae; Node#93355465; ResourceID: Q1146853
[START] [2021-05-04 12:32:14] update_nodes
[STOP] [2021-05-04 12:32:34] update_nodes
[STOP] [2021-05-04 12:32:34] match_nodes
[START] [2021-05-04 12:32:34] reindex_search
[STOP] [2021-05-04 12:37:47] reindex_search
[START] [2021-05-04 12:37:47] normalize_units
[STOP] [2021-05-04 12:37:47] normalize_units
[START] [2021-05-04 12:37:47] calculate_statistics
[STOP] [2021-05-04 12:37:49] calculate_statistics
[START] [2021-05-04 12:37:49] complete_harvest_instance
[START] [2021-05-04 12:37:49] overall_tsv_creation
[INFO] [2021-05-04 12:37:49] Processing group of 147435 in 15 batches of 10000
[2021-05-04 12:38:29] Encountered new language, please assign it to a Locale and give it a name: zh-mo
[2021-05-04 12:38:29] Encountered new language, please assign it to a Locale and give it a name: zh-my
[2021-05-04 12:38:29] Encountered new language, please assign it to a Locale and give it a name: avk
[2021-05-04 12:38:35] Encountered new language, please assign it to a Locale and give it a name: mad
[2021-05-04 12:38:39] Encountered new language, please assign it to a Locale and give it a name: smn
[2021-05-04 12:38:40] Encountered new language, please assign it to a Locale and give it a name: mnw
[2021-05-04 12:38:40] Encountered new language, please assign it to a Locale and give it a name: hyw
[2021-05-04 12:38:52] Encountered new language, please assign it to a Locale and give it a name: mni
[2021-05-04 12:38:53] Encountered new language, please assign it to a Locale and give it a name: gcr
[2021-05-04 12:38:56] Encountered new language, please assign it to a Locale and give it a name: rmf
[2021-05-04 12:38:56] Encountered new language, please assign it to a Locale and give it a name: sms
[2021-05-04 12:39:03] Encountered new language, please assign it to a Locale and give it a name: awa
[2021-05-04 12:39:03] Encountered new language, please assign it to a Locale and give it a name: alt
[2021-05-04 12:39:03] Encountered new language, please assign it to a Locale and give it a name: pag
[2021-05-04 12:39:10] Encountered new language, please assign it to a Locale and give it a name: iii
[2021-05-04 12:40:22] Encountered new language, please assign it to a Locale and give it a name: gan-hans
[2021-05-04 12:40:48] Encountered new language, please assign it to a Locale and give it a name: arq
[2021-05-04 12:40:51] Encountered new language, please assign it to a Locale and give it a name: nia
[2021-05-04 12:41:07] Encountered new language, please assign it to a Locale and give it a name: nqo
[2021-05-04 12:42:44] Encountered new language, please assign it to a Locale and give it a name: sju
[2021-05-04 12:42:45] Encountered new language, please assign it to a Locale and give it a name: tay
[2021-05-04 12:44:17] Encountered new language, please assign it to a Locale and give it a name: skr
[2021-05-04 12:46:27] Encountered new language, please assign it to a Locale and give it a name: qug
[2021-05-04 12:48:19] Encountered new language, please assign it to a Locale and give it a name: dtp
[2021-05-04 12:48:39] Encountered new language, please assign it to a Locale and give it a name: sty
[2021-05-04 12:50:32] Encountered new language, please assign it to a Locale and give it a name: ko-kp
[2021-05-04 12:50:55] Encountered new language, please assign it to a Locale and give it a name: shy-latn
[2021-05-04 12:54:32] Encountered new language, please assign it to a Locale and give it a name: KEA
[2021-05-04 12:56:04] Encountered new language, please assign it to a Locale and give it a name: shi
[2021-05-04 12:56:14] Encountered new language, please assign it to a Locale and give it a name: sje
[2021-05-04 12:57:47] Encountered new language, please assign it to a Locale and give it a name: egl
[INFO] [2021-05-04 13:05:45] Average Time: 65.467
[INFO] [2021-05-04 13:05:45] Total Time: 27m56s
[INFO] [2021-05-04 13:05:45] last 3 / first 3: 1.03
[INFO] [2021-05-04 13:05:45] Std.Dev: 4.657896520963084; Max: 72.31
[STOP] [2021-05-04 13:05:45] overall_tsv_creation
[INFO] [2021-05-04 13:05:45] Done. Check your files:
[INFO] [2021-05-04 13:05:45] (147435 lines) /app/public/data/wikidata/publish_nodes.tsv
[INFO] [2021-05-04 13:05:45] (147435 lines) /app/public/data/wikidata/publish_identifiers.tsv
[INFO] [2021-05-04 13:05:45] (3469271 lines) /app/public/data/wikidata/publish_node_ancestors.tsv
[INFO] [2021-05-04 13:05:45] (147435 lines) /app/public/data/wikidata/publish_scientific_names.tsv
[INFO] [2021-05-04 13:05:46] (893189 lines) /app/public/data/wikidata/publish_vernaculars.tsv
[STOP] [2021-05-04 13:05:46] complete_harvest_instance
[START] [2021-05-04 13:05:46] completed
[STOP] [2021-05-04 13:05:46] completed
[STOP] [2021-05-04 13:05:46] logged process, took 10069.36

Latest Process