Harvest for wikipedia EN Created 21 May 10:51

Stage: completed
Fetched: 21 May 10:51
Validated: 21 May 10:53
Deltas Created 21 May 10:53
Units Normalized: 26 May 17:09
Ancestry Built: 21 May 16:15
Nodes Matched: 26 May 16:43
Names Parsed: 21 May 16:20
New Models Stored: 21 May 13:49
Indexed: 26 May 17:09
Completed: 26 May 18:34
Time to Harvest: 4 minutes

Expected File Format Definitions

Harvesting Log (most recent first)

# Logfile created on 2020-05-21 10:35:19 -0400 by logger.rb/v1.4.2
[INFO] [2020-05-21 10:35:19] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-05-21 10:35:20] ## remove_type: ScientificName
[INFO] [2020-05-21 10:35:20] ++ Batch removal of 388855 instances...
[INFO] [2020-05-21 10:35:20] [10:35:20.648] Batch 0...
[INFO] [2020-05-21 10:35:22] [10:35:22.255] Batch 1...
[INFO] [2020-05-21 10:35:24] [10:35:24.238] Batch 2...
[INFO] [2020-05-21 10:35:26] [10:35:26.047] Batch 3...
[INFO] [2020-05-21 10:35:28] [10:35:28.665] Batch 4...
[INFO] [2020-05-21 10:35:31] [10:35:31.596] Batch 5...
[INFO] [2020-05-21 10:35:35] [10:35:35.467] Batch 6...
[INFO] [2020-05-21 10:35:40] [10:35:40.655] Batch 7...
[INFO] [2020-05-21 10:35:44] [10:35:44.281] Batch 8...
[INFO] [2020-05-21 10:35:50] [10:35:50.090] Batch 9...
[INFO] [2020-05-21 10:35:55] [10:35:55.230] Batch 10...
[INFO] [2020-05-21 10:35:59] [10:35:59.639] Batch 11...
[INFO] [2020-05-21 10:36:04] [10:36:04.327] Batch 12...
[INFO] [2020-05-21 10:36:10] [10:36:10.854] Batch 13...
[INFO] [2020-05-21 10:36:16] [10:36:16.654] Batch 14...
[INFO] [2020-05-21 10:36:22] [10:36:22.372] Batch 15...
[INFO] [2020-05-21 10:36:28] [10:36:28.888] Batch 16...
[INFO] [2020-05-21 10:36:34] [10:36:34.816] Batch 17...
[INFO] [2020-05-21 10:36:41] [10:36:41.213] Batch 18...
[INFO] [2020-05-21 10:36:45] [10:36:45.926] Batch 19...
[INFO] [2020-05-21 10:36:51] [10:36:51.568] Batch 20...
[INFO] [2020-05-21 10:36:57] [10:36:57.287] Batch 21...
[INFO] [2020-05-21 10:37:03] [10:37:03.188] Batch 22...
[INFO] [2020-05-21 10:37:08] [10:37:08.930] Batch 23...
[INFO] [2020-05-21 10:37:13] [10:37:13.777] Batch 24...
[INFO] [2020-05-21 10:37:16] [10:37:16.218] Batch 25...
[INFO] [2020-05-21 10:37:25] [10:37:25.766] Batch 26...
[INFO] [2020-05-21 10:37:31] [10:37:31.740] Batch 27...
[INFO] [2020-05-21 10:37:35] [10:37:35.539] Batch 28...
[INFO] [2020-05-21 10:37:39] [10:37:39.222] Batch 29...
[INFO] [2020-05-21 10:37:43] [10:37:43.465] Batch 30...
[INFO] [2020-05-21 10:37:47] [10:37:47.924] Batch 31...
[INFO] [2020-05-21 10:37:52] [10:37:52.437] Batch 32...
[INFO] [2020-05-21 10:37:56] [10:37:56.009] Batch 33...
[INFO] [2020-05-21 10:38:02] [10:38:02.101] Batch 34...
[INFO] [2020-05-21 10:38:07] [10:38:07.100] Batch 35...
[INFO] [2020-05-21 10:38:10] [10:38:10.722] Batch 36...
[INFO] [2020-05-21 10:38:14] [10:38:14.843] Batch 37...
[INFO] [2020-05-21 10:38:19] [10:38:19.118] Batch 38...
[INFO] [2020-05-21 10:38:22] [10:38:22.666] Removed 388855 Scientificnames
[INFO] [2020-05-21 10:38:22] ## remove_type: Vernacular
[INFO] [2020-05-21 10:38:22] ++ Calling delete_all on 0 instances...
[INFO] [2020-05-21 10:38:22] [10:38:22.669] Removed 0 Vernaculars
[INFO] [2020-05-21 10:38:22] ## remove_type: Article
[INFO] [2020-05-21 10:38:24] ++ Batch removal of 737715 instances...
[INFO] [2020-05-21 10:38:24] [10:38:24.346] Batch 0...
[INFO] [2020-05-21 10:38:31] [10:38:31.206] Batch 1...
[INFO] [2020-05-21 10:38:37] [10:38:37.965] Batch 2...
[INFO] [2020-05-21 10:38:41] [10:38:41.728] Batch 3...
[INFO] [2020-05-21 10:38:46] [10:38:46.645] Batch 4...
[INFO] [2020-05-21 10:38:51] [10:38:51.003] Batch 5...
[INFO] [2020-05-21 10:38:55] [10:38:55.293] Batch 6...
[INFO] [2020-05-21 10:38:59] [10:38:59.027] Batch 7...
[INFO] [2020-05-21 10:39:01] [10:39:01.543] Batch 8...
[INFO] [2020-05-21 10:39:04] [10:39:04.463] Batch 9...
[INFO] [2020-05-21 10:39:07] [10:39:07.213] Batch 10...
[INFO] [2020-05-21 10:39:09] [10:39:09.790] Batch 11...
[INFO] [2020-05-21 10:39:12] [10:39:12.334] Batch 12...
[INFO] [2020-05-21 10:39:14] [10:39:14.899] Batch 13...
[INFO] [2020-05-21 10:39:17] [10:39:17.841] Batch 14...
[INFO] [2020-05-21 10:39:20] [10:39:20.875] Batch 15...
[INFO] [2020-05-21 10:39:23] [10:39:23.568] Batch 16...
[INFO] [2020-05-21 10:39:26] [10:39:26.352] Batch 17...
[INFO] [2020-05-21 10:39:28] [10:39:28.869] Batch 18...
[INFO] [2020-05-21 10:39:31] [10:39:31.348] Batch 19...
[INFO] [2020-05-21 10:39:34] [10:39:34.314] Batch 20...
[INFO] [2020-05-21 10:39:37] [10:39:37.291] Batch 21...
[INFO] [2020-05-21 10:39:39] [10:39:39.693] Batch 22...
[INFO] [2020-05-21 10:39:41] [10:39:41.747] Batch 23...
[INFO] [2020-05-21 10:39:44] [10:39:44.732] Batch 24...
[INFO] [2020-05-21 10:39:46] [10:39:46.538] Batch 25...
[INFO] [2020-05-21 10:39:49] [10:39:49.489] Batch 26...
[INFO] [2020-05-21 10:39:51] [10:39:51.716] Batch 27...
[INFO] [2020-05-21 10:39:54] [10:39:54.337] Batch 28...
[INFO] [2020-05-21 10:39:56] [10:39:56.196] Batch 29...
[INFO] [2020-05-21 10:39:59] [10:39:59.243] Batch 30...
[INFO] [2020-05-21 10:40:01] [10:40:01.180] Batch 31...
[INFO] [2020-05-21 10:40:04] [10:40:04.210] Batch 32...
[INFO] [2020-05-21 10:40:06] [10:40:06.463] Batch 33...
[INFO] [2020-05-21 10:40:08] [10:40:08.834] Batch 34...
[INFO] [2020-05-21 10:40:10] [10:40:10.927] Batch 35...
[INFO] [2020-05-21 10:40:13] [10:40:13.737] Batch 36...
[INFO] [2020-05-21 10:40:16] [10:40:16.004] Batch 37...
[INFO] [2020-05-21 10:40:18] [10:40:18.016] Batch 38...
[INFO] [2020-05-21 10:40:20] [10:40:20.966] Batch 39...
[INFO] [2020-05-21 10:40:22] [10:40:22.982] Batch 40...
[INFO] [2020-05-21 10:40:25] [10:40:25.515] Batch 41...
[INFO] [2020-05-21 10:40:27] [10:40:27.452] Batch 42...
[INFO] [2020-05-21 10:40:29] [10:40:29.957] Batch 43...
[INFO] [2020-05-21 10:40:31] [10:40:31.676] Batch 44...
[INFO] [2020-05-21 10:40:34] [10:40:34.899] Batch 45...
[INFO] [2020-05-21 10:40:36] [10:40:36.718] Batch 46...
[INFO] [2020-05-21 10:40:39] [10:40:39.437] Batch 47...
[INFO] [2020-05-21 10:40:41] [10:40:41.627] Batch 48...
[INFO] [2020-05-21 10:40:44] [10:40:44.160] Batch 49...
[INFO] [2020-05-21 10:40:46] [10:40:46.142] Batch 50...
[INFO] [2020-05-21 10:40:48] [10:40:48.341] Batch 51...
[INFO] [2020-05-21 10:40:50] [10:40:50.784] Batch 52...
[INFO] [2020-05-21 10:40:52] [10:40:52.700] Batch 53...
[INFO] [2020-05-21 10:40:55] [10:40:55.108] Batch 54...
[INFO] [2020-05-21 10:40:56] [10:40:56.565] Batch 55...
[INFO] [2020-05-21 10:40:59] [10:40:59.593] Batch 56...
[INFO] [2020-05-21 10:41:01] [10:41:01.562] Batch 57...
[INFO] [2020-05-21 10:41:03] [10:41:03.806] Batch 58...
[INFO] [2020-05-21 10:41:05] [10:41:05.840] Batch 59...
[INFO] [2020-05-21 10:41:08] [10:41:08.007] Batch 60...
[INFO] [2020-05-21 10:41:10] [10:41:10.459] Batch 61...
[INFO] [2020-05-21 10:41:12] [10:41:12.795] Batch 62...
[INFO] [2020-05-21 10:41:15] [10:41:15.113] Batch 63...
[INFO] [2020-05-21 10:41:16] [10:41:16.911] Batch 64...
[INFO] [2020-05-21 10:41:19] [10:41:19.459] Batch 65...
[INFO] [2020-05-21 10:41:21] [10:41:21.357] Batch 66...
[INFO] [2020-05-21 10:41:23] [10:41:23.722] Batch 67...
[INFO] [2020-05-21 10:41:25] [10:41:25.992] Batch 68...
[INFO] [2020-05-21 10:41:28] [10:41:28.100] Batch 69...
[INFO] [2020-05-21 10:41:30] [10:41:30.386] Batch 70...
[INFO] [2020-05-21 10:41:32] [10:41:32.245] Batch 71...
[INFO] [2020-05-21 10:41:34] [10:41:34.750] Batch 72...
[INFO] [2020-05-21 10:41:36] [10:41:36.377] Batch 73...
[INFO] [2020-05-21 10:41:38] [10:41:38.811] Removed 737715 Articles
[INFO] [2020-05-21 10:41:38] ## remove_type: Medium
[INFO] [2020-05-21 10:41:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-05-21 10:41:38] [10:41:38.816] Removed 0 Media
[INFO] [2020-05-21 10:41:38] ## remove_type: Trait
[INFO] [2020-05-21 10:41:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-05-21 10:41:38] [10:41:38.858] Removed 0 Traits
[INFO] [2020-05-21 10:41:38] ## remove_type: MetaTrait
[INFO] [2020-05-21 10:41:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-05-21 10:41:38] [10:41:38.925] Removed 0 Metatraits
[INFO] [2020-05-21 10:41:38] ## remove_type: OccurrenceMetadatum
[INFO] [2020-05-21 10:41:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-05-21 10:41:38] [10:41:38.964] Removed 0 Occurrencemetadata
[INFO] [2020-05-21 10:41:38] ## remove_type: Assoc
[INFO] [2020-05-21 10:41:38] ++ Calling delete_all on 0 instances...
[INFO] [2020-05-21 10:41:38] [10:41:38.967] Removed 0 Assocs
[INFO] [2020-05-21 10:41:38] ## remove_type: MetaAssoc
[INFO] [2020-05-21 10:41:39] ++ Calling delete_all on 0 instances...
[INFO] [2020-05-21 10:41:39] [10:41:39.067] Removed 0 Metaassocs
[INFO] [2020-05-21 10:41:39] ## remove_type: Identifier
[INFO] [2020-05-21 10:41:39] ++ Calling delete_all on 0 instances...
[INFO] [2020-05-21 10:41:39] [10:41:39.074] Removed 0 Identifiers
[INFO] [2020-05-21 10:41:39] ## remove_type: Reference
[INFO] [2020-05-21 10:41:39] ++ Calling delete_all on 0 instances...
[INFO] [2020-05-21 10:41:39] [10:41:39.076] Removed 0 References
[INFO] [2020-05-21 10:41:39] Starting batch with ID 28848171...
[INFO] [2020-05-21 10:41:40] Starting batch with ID 28897906...
[INFO] [2020-05-21 10:41:41] Starting batch with ID 28929746...
[INFO] [2020-05-21 10:41:42] Starting batch with ID 28764651...
[INFO] [2020-05-21 10:41:43] Starting batch with ID 28764651...
[INFO] [2020-05-21 10:41:44] Starting batch with ID 28737605...
[INFO] [2020-05-21 10:41:45] Starting batch with ID 28737605...
[INFO] [2020-05-21 10:41:45] Starting batch with ID 28960432...
[INFO] [2020-05-21 10:41:46] Starting batch with ID 28962062...
[INFO] [2020-05-21 10:41:47] Starting batch with ID 29032226...
[INFO] [2020-05-21 10:41:48] Starting batch with ID 29032226...
[INFO] [2020-05-21 10:41:49] Starting batch with ID 28996487...
[INFO] [2020-05-21 10:41:50] Starting batch with ID 28996487...
[INFO] [2020-05-21 10:41:50] Starting batch with ID 28983029...
[INFO] [2020-05-21 10:41:51] Starting batch with ID 28983029...
[INFO] [2020-05-21 10:41:52] Starting batch with ID 28679023...
[INFO] [2020-05-21 10:41:53] Starting batch with ID 28679023...
[INFO] [2020-05-21 10:41:54] Starting batch with ID 28814877...
[INFO] [2020-05-21 10:41:55] Starting batch with ID 28814877...
[INFO] [2020-05-21 10:41:56] Starting batch with ID 28802933...
[INFO] [2020-05-21 10:41:57] Starting batch with ID 28817094...
[INFO] [2020-05-21 10:41:57] Starting batch with ID 28817094...
[INFO] [2020-05-21 10:41:58] Starting batch with ID 28887515...
[INFO] [2020-05-21 10:41:59] Starting batch with ID 28887515...
[INFO] [2020-05-21 10:42:00] Starting batch with ID 28786742...
[INFO] [2020-05-21 10:42:01] Starting batch with ID 28906043...
[INFO] [2020-05-21 10:42:02] Starting batch with ID 28906043...
[INFO] [2020-05-21 10:42:03] Starting batch with ID 29061098...
[INFO] [2020-05-21 10:42:04] Starting batch with ID 29061098...
[INFO] [2020-05-21 10:42:04] Starting batch with ID 28859073...
[INFO] [2020-05-21 10:42:05] Starting batch with ID 28859073...
[INFO] [2020-05-21 10:42:06] Starting batch with ID 28828005...
[INFO] [2020-05-21 10:42:07] Starting batch with ID 28918036...
[INFO] [2020-05-21 10:42:08] Starting batch with ID 28918036...
[INFO] [2020-05-21 10:42:09] Starting batch with ID 28925173...
[INFO] [2020-05-21 10:42:10] Starting batch with ID 28925173...
[INFO] [2020-05-21 10:42:11] Starting batch with ID 28967840...
[INFO] [2020-05-21 10:42:12] Starting batch with ID 28731403...
[INFO] [2020-05-21 10:42:14] Starting batch with ID 28687889...
[INFO] [2020-05-21 10:42:14] Starting batch with ID 28683973...
[INFO] [2020-05-21 10:42:15] Starting batch with ID 28724196...
[INFO] [2020-05-21 10:42:16] Starting batch with ID 28747099...
[INFO] [2020-05-21 10:42:17] Starting batch with ID 28747099...
[INFO] [2020-05-21 10:42:18] Starting batch with ID 28840506...
[INFO] [2020-05-21 10:42:19] Starting batch with ID 28840506...
[INFO] [2020-05-21 10:42:20] Starting batch with ID 28811672...
[INFO] [2020-05-21 10:42:21] Starting batch with ID 28811672...
[INFO] [2020-05-21 10:42:22] Starting batch with ID 28900963...
[INFO] [2020-05-21 10:42:23] Starting batch with ID 28900963...
[INFO] [2020-05-21 10:42:24] Starting batch with ID 29053859...
[INFO] [2020-05-21 10:42:25] Starting batch with ID 29053859...
[INFO] [2020-05-21 10:42:26] Starting batch with ID 29060202...
[INFO] [2020-05-21 10:42:27] Starting batch with ID 29064680...
[INFO] [2020-05-21 10:42:28] Starting batch with ID 29042447...
[INFO] [2020-05-21 10:42:29] Starting batch with ID 29042447...
[INFO] [2020-05-21 10:42:30] Starting batch with ID 28731767...
[INFO] [2020-05-21 10:42:31] Starting batch with ID 28731767...
[INFO] [2020-05-21 10:42:32] Starting batch with ID 28689075...
[INFO] [2020-05-21 10:42:33] Starting batch with ID 28689075...
[INFO] [2020-05-21 10:42:34] Starting batch with ID 28817090...
[INFO] [2020-05-21 10:42:35] Starting batch with ID 28817090...
[INFO] [2020-05-21 10:42:36] Starting batch with ID 29059086...
[INFO] [2020-05-21 10:42:37] Starting batch with ID 29059086...
[INFO] [2020-05-21 10:42:38] Starting batch with ID 28963939...
[INFO] [2020-05-21 10:42:39] Starting batch with ID 28963939...
[INFO] [2020-05-21 10:42:40] Starting batch with ID 29024131...
[INFO] [2020-05-21 10:42:42] Starting batch with ID 29024131...
[INFO] [2020-05-21 10:42:42] Starting batch with ID 28869534...
[INFO] [2020-05-21 10:42:44] Starting batch with ID 28869534...
[INFO] [2020-05-21 10:42:45] Starting batch with ID 28960048...
[INFO] [2020-05-21 10:42:46] Starting batch with ID 28960048...
[INFO] [2020-05-21 10:42:47] Starting batch with ID 28936706...
[INFO] [2020-05-21 10:42:48] Starting batch with ID 29000134...
[INFO] [2020-05-21 10:42:49] Starting batch with ID 28841732...
[INFO] [2020-05-21 10:42:50] Starting batch with ID 28877735...
[INFO] [2020-05-21 10:42:52] Starting batch with ID 28900219...
[INFO] [2020-05-21 10:42:53] Starting batch with ID 28912513...
[INFO] [2020-05-21 10:42:54] Starting batch with ID 28715676...
[INFO] [2020-05-21 10:42:55] Starting batch with ID 28715676...
[INFO] [2020-05-21 10:42:56] Starting batch with ID 28808100...
[INFO] [2020-05-21 10:42:57] Starting batch with ID 28800273...
[INFO] [2020-05-21 10:42:59] Starting batch with ID 28795478...
[INFO] [2020-05-21 10:43:00] Starting batch with ID 28761100...
[INFO] [2020-05-21 10:43:02] Starting batch with ID 28780014...
[INFO] [2020-05-21 10:43:03] Starting batch with ID 28772049...
[INFO] [2020-05-21 10:43:04] Starting batch with ID 28808607...
[INFO] [2020-05-21 10:43:05] Starting batch with ID 28798000...
[INFO] [2020-05-21 10:43:07] Starting batch with ID 28864650...
[INFO] [2020-05-21 10:43:08] Starting batch with ID 28864650...
[INFO] [2020-05-21 10:43:09] Starting batch with ID 28834282...
[INFO] [2020-05-21 10:43:10] Starting batch with ID 28834282...
[INFO] [2020-05-21 10:43:11] Starting batch with ID 28694804...
[INFO] [2020-05-21 10:43:12] Starting batch with ID 28694804...
[INFO] [2020-05-21 10:43:13] Starting batch with ID 28700059...
[INFO] [2020-05-21 10:43:14] Starting batch with ID 28700059...
[INFO] [2020-05-21 10:43:15] Starting batch with ID 29048226...
[INFO] [2020-05-21 10:43:16] Starting batch with ID 29048226...
[INFO] [2020-05-21 10:43:16] Starting batch with ID 29065301...
[INFO] [2020-05-21 10:43:18] Starting batch with ID 29065301...
[INFO] [2020-05-21 10:43:19] Starting batch with ID 28978046...
[INFO] [2020-05-21 10:43:20] Starting batch with ID 28978046...
[INFO] [2020-05-21 10:43:21] Starting batch with ID 28973011...
[INFO] [2020-05-21 10:43:22] Starting batch with ID 28968360...
[INFO] [2020-05-21 10:43:23] Starting batch with ID 29010527...
[INFO] [2020-05-21 10:43:24] Starting batch with ID 29021663...
[INFO] [2020-05-21 10:43:25] Starting batch with ID 28900782...
[INFO] [2020-05-21 10:43:26] Starting batch with ID 28959951...
[INFO] [2020-05-21 10:43:28] Starting batch with ID 28885033...
[INFO] [2020-05-21 10:43:29] Starting batch with ID 28914360...
[INFO] [2020-05-21 10:43:30] Starting batch with ID 28990757...
[INFO] [2020-05-21 10:43:31] Starting batch with ID 28688915...
[INFO] [2020-05-21 10:43:32] Starting batch with ID 28725941...
[INFO] [2020-05-21 10:43:33] Starting batch with ID 28798307...
[INFO] [2020-05-21 10:43:34] Starting batch with ID 28815975...
[INFO] [2020-05-21 10:43:36] Starting batch with ID 28827232...
[INFO] [2020-05-21 10:43:37] Starting batch with ID 28836973...
[INFO] [2020-05-21 10:43:38] Starting batch with ID 28698766...
[INFO] [2020-05-21 10:43:39] Starting batch with ID 28717386...
[INFO] [2020-05-21 10:43:40] Starting batch with ID 28757197...
[INFO] [2020-05-21 10:43:41] Starting batch with ID 28765608...
[INFO] [2020-05-21 10:43:42] Starting batch with ID 28733683...
[INFO] [2020-05-21 10:43:44] Starting batch with ID 28845213...
[INFO] [2020-05-21 10:43:45] Starting batch with ID 28872858...
[INFO] [2020-05-21 10:43:46] Starting batch with ID 28907431...
[INFO] [2020-05-21 10:43:47] Starting batch with ID 28891797...
[INFO] [2020-05-21 10:43:48] Starting batch with ID 28890864...
[INFO] [2020-05-21 10:43:49] Starting batch with ID 28969949...
[INFO] [2020-05-21 10:43:51] Starting batch with ID 28962916...
[INFO] [2020-05-21 10:43:52] Starting batch with ID 28859461...
[INFO] [2020-05-21 10:43:53] Starting batch with ID 28955537...
[INFO] [2020-05-21 10:43:54] Starting batch with ID 28952002...
[INFO] [2020-05-21 10:43:56] Starting batch with ID 28898601...
[INFO] [2020-05-21 10:43:57] Starting batch with ID 28947012...
[INFO] [2020-05-21 10:43:58] Starting batch with ID 29032992...
[INFO] [2020-05-21 10:44:00] Starting batch with ID 29047025...
[INFO] [2020-05-21 10:44:01] Starting batch with ID 29047025...
[INFO] [2020-05-21 10:44:02] Starting batch with ID 29013871...
[INFO] [2020-05-21 10:44:03] Starting batch with ID 29013871...
[INFO] [2020-05-21 10:44:03] ## remove_type: Node
[INFO] [2020-05-21 10:44:04] ++ Batch removal of 388855 instances...
[INFO] [2020-05-21 10:44:04] [10:44:04.034] Batch 0...
[INFO] [2020-05-21 10:44:05] [10:44:05.334] Batch 1...
[INFO] [2020-05-21 10:44:07] [10:44:07.124] Batch 2...
[INFO] [2020-05-21 10:44:09] [10:44:09.906] Batch 3...
[INFO] [2020-05-21 10:44:12] [10:44:12.625] Batch 4...
[INFO] [2020-05-21 10:44:16] [10:44:16.407] Batch 5...
[INFO] [2020-05-21 10:44:19] [10:44:19.210] Batch 6...
[INFO] [2020-05-21 10:44:21] [10:44:21.639] Batch 7...
[INFO] [2020-05-21 10:44:24] [10:44:24.098] Batch 8...
[INFO] [2020-05-21 10:44:26] [10:44:26.474] Batch 9...
[INFO] [2020-05-21 10:44:28] [10:44:28.742] Batch 10...
[INFO] [2020-05-21 10:44:29] [10:44:29.993] Batch 11...
[INFO] [2020-05-21 10:44:33] [10:44:33.646] Batch 12...
[INFO] [2020-05-21 10:44:36] [10:44:36.209] Batch 13...
[INFO] [2020-05-21 10:44:39] [10:44:39.035] Batch 14...
[INFO] [2020-05-21 10:44:41] [10:44:41.765] Batch 15...
[INFO] [2020-05-21 10:44:45] [10:44:45.042] Batch 16...
[INFO] [2020-05-21 10:44:48] [10:44:48.202] Batch 17...
[INFO] [2020-05-21 10:44:51] [10:44:51.619] Batch 18...
[INFO] [2020-05-21 10:44:54] [10:44:54.665] Batch 19...
[INFO] [2020-05-21 10:44:57] [10:44:57.856] Batch 20...
[INFO] [2020-05-21 10:45:00] [10:45:00.726] Batch 21...
[INFO] [2020-05-21 10:45:04] [10:45:04.447] Batch 22...
[INFO] [2020-05-21 10:45:08] [10:45:08.157] Batch 23...
[INFO] [2020-05-21 10:45:10] [10:45:10.661] Batch 24...
[INFO] [2020-05-21 10:45:14] [10:45:14.025] Batch 25...
[INFO] [2020-05-21 10:45:17] [10:45:17.550] Batch 26...
[INFO] [2020-05-21 10:45:20] [10:45:20.576] Batch 27...
[INFO] [2020-05-21 10:45:23] [10:45:23.027] Batch 28...
[INFO] [2020-05-21 10:45:25] [10:45:25.241] Batch 29...
[INFO] [2020-05-21 10:45:28] [10:45:28.231] Batch 30...
[INFO] [2020-05-21 10:45:30] [10:45:30.395] Batch 31...
[INFO] [2020-05-21 10:45:33] [10:45:33.055] Batch 32...
[INFO] [2020-05-21 10:45:35] [10:45:35.607] Batch 33...
[INFO] [2020-05-21 10:45:39] [10:45:39.008] Batch 34...
[INFO] [2020-05-21 10:45:42] [10:45:42.052] Batch 35...
[INFO] [2020-05-21 10:45:44] [10:45:44.207] Batch 36...
[INFO] [2020-05-21 10:45:46] [10:45:46.480] Batch 37...
[INFO] [2020-05-21 10:45:49] [10:45:49.218] Batch 38...
[INFO] [2020-05-21 10:45:51] [10:45:51.667] Removed 388855 Nodes
[START] [2020-05-21 10:51:04] logged process
[START] [2020-05-21 10:51:04] Creating resource from OpenData
[START] [2020-05-21 10:51:47] logged process
[START] [2020-05-21 10:51:47] Parse meta.xml file and create formats with fields
[STOP] [2020-05-21 10:51:50] Parse meta.xml file and create formats with fields
[STOP] [2020-05-21 10:51:50] Creating resource from OpenData
[START] [2020-05-21 10:51:50] logged process
[START] [2020-05-21 10:51:51] create_harvest_instance
[STOP] [2020-05-21 10:51:52] create_harvest_instance
[START] [2020-05-21 10:51:52] fetch_files
[STOP] [2020-05-21 10:51:52] fetch_files
[START] [2020-05-21 10:51:52] validate_each_file
[STOP] [2020-05-21 10:53:08] validate_each_file
[START] [2020-05-21 10:53:08] convert_to_csv
[CMD] [2020-05-21 10:53:08] /usr/bin/sort /app/public/converted_csv/wiki_english_nodes_21012.csv > /app/public/converted_csv/wiki_english_nodes_21012.csv_sorted
[CMD] [2020-05-21 10:53:08] /usr/bin/sort /app/public/converted_csv/wiki_english_media_21013.csv > /app/public/converted_csv/wiki_english_media_21013.csv_sorted
[STOP] [2020-05-21 10:53:12] convert_to_csv
[START] [2020-05-21 10:53:12] calculate_delta
[CMD] [2020-05-21 10:53:12] echo "0a" > /app/public/diff/wiki_english_nodes_21012.diff
[CMD] [2020-05-21 10:53:12] tail -n +1 /app/public/converted_csv/wiki_english_nodes_21012.csv >> /app/public/diff/wiki_english_nodes_21012.diff
[CMD] [2020-05-21 10:53:12] echo "." >> /app/public/diff/wiki_english_nodes_21012.diff
[CMD] [2020-05-21 10:53:12] echo "0a" > /app/public/diff/wiki_english_media_21013.diff
[CMD] [2020-05-21 10:53:12] tail -n +1 /app/public/converted_csv/wiki_english_media_21013.csv >> /app/public/diff/wiki_english_media_21013.diff
[CMD] [2020-05-21 10:53:15] echo "." >> /app/public/diff/wiki_english_media_21013.diff
[STOP] [2020-05-21 10:53:15] calculate_delta
[START] [2020-05-21 10:53:15] parse_diff_and_store
[INFO] [2020-05-21 10:53:15] Loading nodes diff file into memory (true lines)...
[WARN] [2020-05-21 10:54:04] Filtered Scientific Name `London1_novel CoV/2012` to `London1_novel CoV2012`
[WARN] [2020-05-21 10:54:10] Filtered Scientific Name `Visna/maedi virus` to `Visnamaedi virus`
[INFO] [2020-05-21 10:56:24] Loading media diff file into memory (true lines)...
[INFO] [2020-05-21 13:22:40] Storing 407051 ScientificNames
[INFO] [2020-05-21 13:22:40] Processing group of 407051 in 408 groups of 1000
[INFO] [2020-05-21 13:26:54] Average Time: 0.617
[INFO] [2020-05-21 13:26:54] Total Time: 4m14s
[INFO] [2020-05-21 13:26:54] last 3 / first 3: 1.04
[INFO] [2020-05-21 13:26:54] Std.Dev: 1.9491023574969069; Max: 15.94
[INFO] [2020-05-21 13:26:54] Storing 407057 Identifiers
[INFO] [2020-05-21 13:26:54] Processing group of 407057 in 408 groups of 1000
[INFO] [2020-05-21 13:28:17] Average Time: 0.199
[INFO] [2020-05-21 13:28:17] Total Time: 1m23s
[INFO] [2020-05-21 13:28:17] last 3 / first 3: 0.54
[INFO] [2020-05-21 13:28:17] Std.Dev: 1.1090536506409416; Max: 16.2
[INFO] [2020-05-21 13:28:17] Storing 407051 Nodes
[INFO] [2020-05-21 13:28:17] Processing group of 407051 in 408 groups of 1000
[INFO] [2020-05-21 13:33:33] Average Time: 0.771
[INFO] [2020-05-21 13:33:33] Total Time: 5m17s
[INFO] [2020-05-21 13:33:33] last 3 / first 3: 18.43
[INFO] [2020-05-21 13:33:33] Std.Dev: 2.6817904466978773; Max: 17.37
[INFO] [2020-05-21 13:33:33] Storing 766346 ArticlesSections
[INFO] [2020-05-21 13:33:33] Processing group of 766346 in 767 groups of 1000
[INFO] [2020-05-21 13:34:54] Average Time: 0.102
[INFO] [2020-05-21 13:34:54] Total Time: 1m21s
[INFO] [2020-05-21 13:34:54] last 3 / first 3: 0.71
[INFO] [2020-05-21 13:34:54] Std.Dev: 0.8826097665446492; Max: 17.6
[INFO] [2020-05-21 13:34:54] Storing 766346 Articles
[INFO] [2020-05-21 13:34:54] Processing group of 766346 in 767 groups of 1000
[INFO] [2020-05-21 13:49:57] Average Time: 1.17
[INFO] [2020-05-21 13:49:57] Total Time: 15m3s
[INFO] [2020-05-21 13:49:57] last 3 / first 3: 0.76
[INFO] [2020-05-21 13:49:57] Std.Dev: 3.2969683043669074; Max: 20.29
[STOP] [2020-05-21 13:49:57] parse_diff_and_store
[START] [2020-05-21 13:49:57] resolve_keys
[INFO] [2020-05-21 14:10:25] Occurrences to nodes (through scientific_names)...
[INFO] [2020-05-21 14:10:26] traits to occurrences...
[INFO] [2020-05-21 14:10:26] traits to nodes (through occurrences)...
[INFO] [2020-05-21 14:10:26] Traits to sex term...
[INFO] [2020-05-21 14:10:26] Traits to lifestage term...
[INFO] [2020-05-21 14:10:26] MetaTraits to traits...
[INFO] [2020-05-21 14:10:26] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-05-21 14:10:26] Assocs to occurrences...
[INFO] [2020-05-21 14:10:26] Assocs to nodes...
[INFO] [2020-05-21 14:10:26] Assoc to sex term...
[INFO] [2020-05-21 14:10:26] Assoc to lifestage term...
[STOP] [2020-05-21 14:10:26] resolve_keys
[START] [2020-05-21 14:10:26] hold_for_later_1
[STOP] [2020-05-21 14:10:26] hold_for_later_1
[START] [2020-05-21 14:10:26] hold_for_later_2
[STOP] [2020-05-21 14:10:26] hold_for_later_2
[START] [2020-05-21 14:10:26] resolve_missing_parents
[STOP] [2020-05-21 14:11:34] resolve_missing_parents
[START] [2020-05-21 14:11:34] rebuild_nodes
[START] [2020-05-21 14:11:34] Flattener#flatten
[START] [2020-05-21 14:11:34] Flattener#study_resource
[START] [2020-05-21 14:11:45] Flattener#build_ancestry
[STOP] [2020-05-21 15:46:07] Flattener#build_ancestry
[INFO] [2020-05-21 15:46:07] 407051 ancestry keys
[START] [2020-05-21 15:46:07] build_node_ancestors
[INFO] [2020-05-21 15:46:07] old ancestors deleted.
[STOP] [2020-05-21 16:05:18] build_node_ancestors
[START] [2020-05-21 16:05:26] Flattener#propagate_ancestor_ids
[STOP] [2020-05-21 16:15:04] Flattener#propagate_ancestor_ids
[STOP] [2020-05-21 16:15:04] Flattener#flatten
[STOP] [2020-05-21 16:15:04] rebuild_nodes
[START] [2020-05-21 16:15:04] resolve_missing_media_owners
[STOP] [2020-05-21 16:15:04] resolve_missing_media_owners
[START] [2020-05-21 16:15:04] sanitize_media_verbatims
[STOP] [2020-05-21 16:15:04] sanitize_media_verbatims
[START] [2020-05-21 16:15:04] queue_downloads
[STOP] [2020-05-21 16:15:04] queue_downloads
[START] [2020-05-21 16:15:04] parse_names
[WARN] [2020-05-21 16:15:05] I see 407051 names which still need to be parsed.
[WARN] [2020-05-21 16:20:00] I see 75 names which still need to be parsed.
[STOP] [2020-05-21 16:20:02] parse_names
[START] [2020-05-21 16:20:02] denormalize_canonical_names_to_nodes
[STOP] [2020-05-21 16:20:08] denormalize_canonical_names_to_nodes
[START] [2020-05-21 16:20:08] match_nodes
[START] [2020-05-21 16:20:09] map_all_nodes_to_pages
[STOP] [2020-05-23 03:56:31] map_all_nodes_to_pages
[INFO] [2020-05-23 03:56:31] 25699 Unmatched nodes (of 407051)! That's too many to output. First 10: Bionta (#78180730); London1_novel CoV2012 (#78296929); Beltanelloides (#78344398); Parakaryon myojinensis (#78344797); Biota (#78351468); Acytota (#78295950); Prokaryota (#78316725); Hadesarchaea (#78348136); Aciduliprofundum (#78424126); Aciduliprofundum boonei (#78359989)
[START] [2020-05-23 03:56:31] update_nodes
[STOP] [2020-05-23 03:57:01] update_nodes
[STOP] [2020-05-23 03:57:01] match_nodes
[ERR] [2020-05-23 03:57:01] Faraday::ConnectionFailed
[ERR] [2020-05-23 03:57:01] Failed to open TCP connection to elasticsearch:9200 (Connection refused - connect(2) for "elasticsearch" port 9200)
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:131:in `match'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:153:in `match_canonical_in_eol'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:285:in `map_unflagged_node'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:250:in `map_node'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:223:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:229:in `block in map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:228:in `map_if_needed'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:190:in `block in map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:189:in `map_nodes'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:184:in `block in map_all_nodes_to_pages'
[ERR] [2020-05-23 03:57:01] ../models/logged_process.rb:62:in `enter_group'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:182:in `map_all_nodes_to_pages'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:169:in `block in start'
[ERR] [2020-05-23 03:57:01] ../models/logged_process.rb:19:in `run_step'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:169:in `start'
[ERR] [2020-05-23 03:57:01] ../models/names_matcher.rb:22:in `for_harvest'
[ERR] [2020-05-23 03:57:01] ../models/resource_harvester.rb:608:in `match_nodes'
[ERR] [2020-05-23 03:57:01] ../models/resource_harvester.rb:86:in `block (3 levels) in start'
[ERR] [2020-05-23 03:57:01] ../models/logged_process.rb:19:in `run_step'
[ERR] [2020-05-23 03:57:01] ../models/resource_harvester.rb:86:in `block (2 levels) in start'
[ERR] [2020-05-23 03:57:01] ../models/resource_harvester.rb:75:in `each_key'
[ERR] [2020-05-23 03:57:01] ../models/resource_harvester.rb:75:in `block in start'
[ERR] [2020-05-23 03:57:01] ../models/resource.rb:151:in `lock'
[ERR] [2020-05-23 03:57:01] ../models/resource_harvester.rb:72:in `start'
[ERR] [2020-05-23 03:57:01] ../models/resource.rb:232:in `harvest'
[ERR] [2020-05-23 03:57:01] ../models/resource.rb:208:in `re_download_opendata_and_harvest'
[ERR] [2020-05-23 03:57:01] bin/rails:4:in `require'
[ERR] [2020-05-23 03:57:01] bin/rails:4:in `<main>'
[STOP] [2020-05-23 03:57:01] logged process, took 147910.45
[INFO] [2020-05-23 09:21:54] ## HARVEST: type = resume_-harvest
[START] [2020-05-23 09:21:54] logged process
[ERR] [2020-05-23 09:21:54][hdls] *****
[ERR] [2020-05-23 09:21:54][hdls] ***** HARVEST ATTEMPT FAILED: This resource is locked; assuming it is already running. Remove lock if not.
[ERR] [2020-05-23 09:21:54][hdls] *****
[INFO] [2020-05-26 13:23:42] ## HARVEST: type = resume_-harvest
[START] [2020-05-26 13:23:45] logged process
[INFO] [2020-05-26 15:01:48] ## HARVEST: type = resume_-harvest
[START] [2020-05-26 15:01:52] logged process
[INFO] [2020-05-26 15:01:52] Already completed stage create_harvest_instance, skipping...
[INFO] [2020-05-26 15:01:52] Already completed stage fetch_files, skipping...
[INFO] [2020-05-26 15:01:52] Already completed stage validate_each_file, skipping...
[INFO] [2020-05-26 15:01:52] Already completed stage convert_to_csv, skipping...
[INFO] [2020-05-26 15:01:52] Already completed stage calculate_delta, skipping...
[INFO] [2020-05-26 15:01:52] Already completed stage parse_diff_and_store, skipping...
[INFO] [2020-05-26 15:01:52] Already completed stage resolve_keys, skipping...
[INFO] [2020-05-26 15:01:52] Already completed stage hold_for_later_1, skipping...
[INFO] [2020-05-26 15:01:52] Already completed stage hold_for_later_2, skipping...
[INFO] [2020-05-26 15:01:52] Already completed stage resolve_missing_parents, skipping...
[INFO] [2020-05-26 15:01:52] Already completed stage rebuild_nodes, skipping...
[INFO] [2020-05-26 15:01:52] Already completed stage resolve_missing_media_owners, skipping...
[INFO] [2020-05-26 15:01:52] Already completed stage sanitize_media_verbatims, skipping...
[INFO] [2020-05-26 15:01:52] Already completed stage queue_downloads, skipping...
[INFO] [2020-05-26 15:01:52] Already completed stage parse_names, skipping...
[INFO] [2020-05-26 15:01:52] Already completed stage denormalize_canonical_names_to_nodes, skipping...
[START] [2020-05-26 15:01:52] match_nodes
[START] [2020-05-26 15:01:53] map_all_nodes_to_pages
[STOP] [2020-05-26 16:43:26] map_all_nodes_to_pages
[INFO] [2020-05-26 16:43:26] 5267 Unmatched nodes (of 407051)! That's too many to output. First 10: Arhopala matsutaroi (#78434040); Arhopala pseudovihara (#78434046); Arhopala sakaguchii (#78434048); Arhopala hayashihisakazui (#78515646); Mimeresia moreelsi (#78513972); Mimeresia moyambina (#78513973); Phothecla (#78302411); Exorbaetta (#78302568); Porthecla (#78302574); Porthecla annette (#78529328)
[START] [2020-05-26 16:43:26] update_nodes
[STOP] [2020-05-26 16:43:49] update_nodes
[STOP] [2020-05-26 16:43:49] match_nodes
[START] [2020-05-26 16:43:49] reindex_search
[STOP] [2020-05-26 17:09:02] reindex_search
[START] [2020-05-26 17:09:02] normalize_units
[STOP] [2020-05-26 17:09:02] normalize_units
[START] [2020-05-26 17:09:02] calculate_statistics
[STOP] [2020-05-26 17:09:04] calculate_statistics
[START] [2020-05-26 17:09:04] complete_harvest_instance
[START] [2020-05-26 17:09:04] overall_tsv_creation
[INFO] [2020-05-26 17:09:05] Processing group of 407051 in 41 batches of 10000
[INFO] [2020-05-26 18:34:55] Average Time: 69.276
[INFO] [2020-05-26 18:34:55] Total Time: 1h25m51s
[INFO] [2020-05-26 18:34:55] last 3 / first 3: 0.91
[INFO] [2020-05-26 18:34:55] Std.Dev: 4.12298435602174; Max: 78.5
[STOP] [2020-05-26 18:34:55] overall_tsv_creation
[INFO] [2020-05-26 18:34:55] Done. Check your files:
[INFO] [2020-05-26 18:34:56] (407051 lines) /app/public/data/wiki_english/publish_nodes.tsv
[INFO] [2020-05-26 18:34:56] (407057 lines) /app/public/data/wiki_english/publish_identifiers.tsv
[INFO] [2020-05-26 18:34:56] (10097417 lines) /app/public/data/wiki_english/publish_node_ancestors.tsv
[INFO] [2020-05-26 18:34:56] (407051 lines) /app/public/data/wiki_english/publish_scientific_names.tsv
[INFO] [2020-05-26 18:34:57] (5485885 lines) /app/public/data/wiki_english/publish_articles.tsv
[INFO] [2020-05-26 18:34:57] (766346 lines) /app/public/data/wiki_english/publish_content_sections.tsv
[STOP] [2020-05-26 18:34:57] complete_harvest_instance
[START] [2020-05-26 18:34:57] completed
[STOP] [2020-05-26 18:34:57] completed
[STOP] [2020-05-26 18:34:57] logged process, took 12785.12

Latest Process