Harvest for World Register of Marine Species Created 07 Feb 14:46

Stage: completed
Fetched: 07 Feb 14:46
Validated: 07 Feb 14:53
Deltas Created 07 Feb 14:53
Units Normalized: 08 Feb 22:17
Ancestry Built: 08 Feb 10:37
Nodes Matched: 08 Feb 21:55
Names Parsed: 08 Feb 10:44
New Models Stored: 08 Feb 08:26
Indexed: 08 Feb 22:08
Completed: 10 Feb 13:33
Time to Harvest: 3 minutes

Expected File Format Definitions

Harvesting Log (most recent first)

# Logfile created on 2020-02-07 14:46:40 -0500 by logger.rb/56815
[START] [2020-02-07 14:46:40] logged process
[START] [2020-02-07 14:46:40] create_harvest_instance
[STOP] [2020-02-07 14:46:42] create_harvest_instance
[START] [2020-02-07 14:46:42] fetch_files
[STOP] [2020-02-07 14:46:42] fetch_files
[START] [2020-02-07 14:46:42] validate_each_file
[STOP] [2020-02-07 14:53:41] validate_each_file
[START] [2020-02-07 14:53:41] convert_to_csv
[CMD] [2020-02-07 14:53:41] /usr/bin/sort /app/public/converted_csv/world_reg_mar_sp_agents_20180.csv > /app/public/converted_csv/world_reg_mar_sp_agents_20180.csv_sorted
[CMD] [2020-02-07 14:53:41] /usr/bin/sort /app/public/converted_csv/world_reg_mar_sp_refs_20181.csv > /app/public/converted_csv/world_reg_mar_sp_refs_20181.csv_sorted
[CMD] [2020-02-07 14:53:41] /usr/bin/sort /app/public/converted_csv/world_reg_mar_sp_nodes_20182.csv > /app/public/converted_csv/world_reg_mar_sp_nodes_20182.csv_sorted
[CMD] [2020-02-07 14:53:42] /usr/bin/sort /app/public/converted_csv/world_reg_mar_sp_media_20183.csv > /app/public/converted_csv/world_reg_mar_sp_media_20183.csv_sorted
[CMD] [2020-02-07 14:53:42] /usr/bin/sort /app/public/converted_csv/world_reg_mar_sp_vernaculars_20184.csv > /app/public/converted_csv/world_reg_mar_sp_vernaculars_20184.csv_sorted
[CMD] [2020-02-07 14:53:42] /usr/bin/sort /app/public/converted_csv/world_reg_mar_sp_occurrences_20185.csv > /app/public/converted_csv/world_reg_mar_sp_occurrences_20185.csv_sorted
[CMD] [2020-02-07 14:53:42] /usr/bin/sort /app/public/converted_csv/world_reg_mar_sp_measurements_20186.csv > /app/public/converted_csv/world_reg_mar_sp_measurements_20186.csv_sorted
[STOP] [2020-02-07 14:53:44] convert_to_csv
[START] [2020-02-07 14:53:44] calculate_delta
[CMD] [2020-02-07 14:53:44] echo "0a" > /app/public/diff/world_reg_mar_sp_agents_20180.diff
[CMD] [2020-02-07 14:53:44] tail -n +1 /app/public/converted_csv/world_reg_mar_sp_agents_20180.csv >> /app/public/diff/world_reg_mar_sp_agents_20180.diff
[CMD] [2020-02-07 14:53:44] echo "." >> /app/public/diff/world_reg_mar_sp_agents_20180.diff
[CMD] [2020-02-07 14:53:44] echo "0a" > /app/public/diff/world_reg_mar_sp_refs_20181.diff
[CMD] [2020-02-07 14:53:44] tail -n +1 /app/public/converted_csv/world_reg_mar_sp_refs_20181.csv >> /app/public/diff/world_reg_mar_sp_refs_20181.diff
[CMD] [2020-02-07 14:53:45] echo "." >> /app/public/diff/world_reg_mar_sp_refs_20181.diff
[CMD] [2020-02-07 14:53:45] echo "0a" > /app/public/diff/world_reg_mar_sp_nodes_20182.diff
[CMD] [2020-02-07 14:53:45] tail -n +1 /app/public/converted_csv/world_reg_mar_sp_nodes_20182.csv >> /app/public/diff/world_reg_mar_sp_nodes_20182.diff
[CMD] [2020-02-07 14:53:45] echo "." >> /app/public/diff/world_reg_mar_sp_nodes_20182.diff
[CMD] [2020-02-07 14:53:45] echo "0a" > /app/public/diff/world_reg_mar_sp_media_20183.diff
[CMD] [2020-02-07 14:53:45] tail -n +1 /app/public/converted_csv/world_reg_mar_sp_media_20183.csv >> /app/public/diff/world_reg_mar_sp_media_20183.diff
[CMD] [2020-02-07 14:53:45] echo "." >> /app/public/diff/world_reg_mar_sp_media_20183.diff
[CMD] [2020-02-07 14:53:45] echo "0a" > /app/public/diff/world_reg_mar_sp_vernaculars_20184.diff
[CMD] [2020-02-07 14:53:45] tail -n +1 /app/public/converted_csv/world_reg_mar_sp_vernaculars_20184.csv >> /app/public/diff/world_reg_mar_sp_vernaculars_20184.diff
[CMD] [2020-02-07 14:53:45] echo "." >> /app/public/diff/world_reg_mar_sp_vernaculars_20184.diff
[CMD] [2020-02-07 14:53:45] echo "0a" > /app/public/diff/world_reg_mar_sp_occurrences_20185.diff
[CMD] [2020-02-07 14:53:45] tail -n +1 /app/public/converted_csv/world_reg_mar_sp_occurrences_20185.csv >> /app/public/diff/world_reg_mar_sp_occurrences_20185.diff
[CMD] [2020-02-07 14:53:45] echo "." >> /app/public/diff/world_reg_mar_sp_occurrences_20185.diff
[CMD] [2020-02-07 14:53:45] echo "0a" > /app/public/diff/world_reg_mar_sp_measurements_20186.diff
[CMD] [2020-02-07 14:53:45] tail -n +1 /app/public/converted_csv/world_reg_mar_sp_measurements_20186.csv >> /app/public/diff/world_reg_mar_sp_measurements_20186.diff
[CMD] [2020-02-07 14:53:46] echo "." >> /app/public/diff/world_reg_mar_sp_measurements_20186.diff
[STOP] [2020-02-07 14:53:46] calculate_delta
[START] [2020-02-07 14:53:46] parse_diff_and_store
[INFO] [2020-02-07 14:53:46] Loading agents diff file into memory (true lines)...
[INFO] [2020-02-07 14:53:47] Loading refs diff file into memory (true lines)...
[INFO] [2020-02-07 14:55:46] Loading nodes diff file into memory (true lines)...
[WARN] [2020-02-07 14:56:06] Filtered Scientific Name `Erinaceusyllis barbarae Langeneck, Musco & Castelli <i>in</i> Langeneck, Musco, Busoni, Conese, Aliani & Castelli, 2018` to `Erinaceusyllis barbarae Langeneck, Musco & Castelli <i>in<i> Langeneck, Musco, Busoni, Conese, Aliani & Castelli, 2018`
[WARN] [2020-02-07 14:56:06] Filtered Scientific Name `Exogone sophiae Langeneck, Musco & Castelli <i>in</i> Langeneck, Musco, Busoni, Conese, Aliani & Castelli, 2018` to `Exogone sophiae Langeneck, Musco & Castelli <i>in<i> Langeneck, Musco, Busoni, Conese, Aliani & Castelli, 2018`
[WARN] [2020-02-07 14:56:06] Filtered Scientific Name `Prosphaerosyllis danovaroi Langeneck, Musco & Castelli <i>in</i> Langeneck, Musco, Busoni, Conese, Aliani & Castelli, 2018` to `Prosphaerosyllis danovaroi Langeneck, Musco & Castelli <i>in<i> Langeneck, Musco, Busoni, Conese, Aliani & Castelli, 2018`
[WARN] [2020-02-07 14:56:16] Filtered Scientific Name `Nemertopsis flavida (McIntosh, 1873/74)` to `Nemertopsis flavida (McIntosh, 187374)`
[WARN] [2020-02-07 14:56:16] Filtered Scientific Name `Uniporus hyalinus Brinkmann, 1914/15` to `Uniporus hyalinus Brinkmann, 191415`
[WARN] [2020-02-07 14:56:30] Filtered Scientific Name `Tachytrypane McIntosh <i>in</i> Jeffreys, 1876` to `Tachytrypane McIntosh <i>in<i> Jeffreys, 1876`
[WARN] [2020-02-07 14:56:30] Filtered Scientific Name `Streblosoma M. Sars <i>in</i> G.O. Sars, 1872` to `Streblosoma M. Sars <i>in<i> G.O. Sars, 1872`
[WARN] [2020-02-07 14:56:31] Filtered Scientific Name `Fauveliopsis jameoaquensis Núñez <i>in</i> Núñez, Ocaña & Brito, 1997` to `Fauveliopsis jameoaquensis Núñez <i>in<i> Núñez, Ocaña & Brito, 1997`
[WARN] [2020-02-07 14:56:32] Filtered Scientific Name `Ammotrypanella arctica (McIntosh <i>in</i> Jeffreys, 1876)` to `Ammotrypanella arctica (McIntosh <i>in<i> Jeffreys, 1876)`
[WARN] [2020-02-07 14:56:32] Filtered Scientific Name `Tachytrypane jeffreysi McIntosh <i>in</i> Jeffreys, 1876` to `Tachytrypane jeffreysi McIntosh <i>in<i> Jeffreys, 1876`
[WARN] [2020-02-07 14:56:49] Filtered Scientific Name `Cirolana centinelensis Maguire <i>in</i> Maguire, Feldmann, Jones, Schweitzer & Casadio, 2018` to `Cirolana centinelensis Maguire <i>in<i> Maguire, Feldmann, Jones, Schweitzer & Casadio, 2018`
[WARN] [2020-02-07 14:56:49] Filtered Scientific Name `Urda zelandica Buckeridge & Johns <i>in</i> Grant-Mackie, Buckeridge & Johns, 1996` to `Urda zelandica Buckeridge & Johns <i>in<i> Grant-Mackie, Buckeridge & Johns, 1996`
[WARN] [2020-02-07 14:56:49] Filtered Scientific Name `Isopoda <i>incertae sedis</i>` to `Isopoda <i>incertae sedis<i>`
[WARN] [2020-02-07 14:56:53] Filtered Scientific Name `Epicaridea <i>incertae sedis</i>` to `Epicaridea <i>incertae sedis<i>`
[WARN] [2020-02-07 14:57:09] Filtered Scientific Name `Stigmatella miranda Yang, Hu & Xia, 1988/Hu, 1988` to `Stigmatella miranda Yang, Hu & Xia, 1988Hu, 1988`
[WARN] [2020-02-07 14:57:10] Filtered Scientific Name `Cellulipora mamillosa (Michelin, 1846) /(?Römer, 1840?)` to `Cellulipora mamillosa (Michelin, 1846) (?Römer, 1840?)`
[WARN] [2020-02-07 14:57:11] Filtered Scientific Name `Fenestella explanata (Römer, 1843/1850)` to `Fenestella explanata (Römer, 18431850)`
[WARN] [2020-02-07 14:57:13] Filtered Scientific Name `Shulgapora postabundans (Schulga-Nesterenko, 1949)/1952` to `Shulgapora postabundans (Schulga-Nesterenko, 1949)1952`
[WARN] [2020-02-07 14:57:13] Filtered Scientific Name `Spinofenestella taxata (Schulga-Nesterenko, 1941)/ 1951` to `Spinofenestella taxata (Schulga-Nesterenko, 1941) 1951`
[WARN] [2020-02-07 14:57:18] Filtered Scientific Name `Syllis monilaris Savigny <i>in</i> Lamarck, 1818` to `Syllis monilaris Savigny <i>in<i> Lamarck, 1818`
[WARN] [2020-02-07 14:58:37] Filtered Scientific Name `Phyllodoce foliosopapillata Hornell <i>in</i> Willey, 1905` to `Phyllodoce foliosopapillata Hornell <i>in<i> Willey, 1905`
[WARN] [2020-02-07 14:58:56] Filtered Scientific Name `Oncaea waldemari Bersano & Boxshall, 1996 ["1994"]` to `Oncaea waldemari Bersano & Boxshall, 1996 [1994]`
[WARN] [2020-02-07 14:59:31] Filtered Scientific Name `Mesalia opalina (A. Adams & Reeve <i>in</i> Reeve, 1849)` to `Mesalia opalina (A. Adams & Reeve <i>in<i> Reeve, 1849)`
[WARN] [2020-02-07 14:59:35] Filtered Scientific Name `Turritella banksii Gray <i>in</i> Reeve, 1849` to `Turritella banksii Gray <i>in<i> Reeve, 1849`
[WARN] [2020-02-07 14:59:35] Filtered Scientific Name `Turritella conspersa A. Adams & Reeve <i>in</i> Reeve, 1849` to `Turritella conspersa A. Adams & Reeve <i>in<i> Reeve, 1849`
[WARN] [2020-02-07 14:59:35] Filtered Scientific Name `Turritella fastigiata A. Adams & Reeve <i>in</i> Reeve, 1849` to `Turritella fastigiata A. Adams & Reeve <i>in<i> Reeve, 1849`
[WARN] [2020-02-07 15:00:02] Filtered Scientific Name `Turritella congelata A. Adams & Reeve <i>in</i> Reeve, 1849` to `Turritella congelata A. Adams & Reeve <i>in<i> Reeve, 1849`
[WARN] [2020-02-07 15:00:33] Filtered Scientific Name `Ceratosoma gracillimum Semper <i>in</i> Bergh, 1876` to `Ceratosoma gracillimum Semper <i>in<i> Bergh, 1876`
[WARN] [2020-02-07 15:00:34] Filtered Scientific Name `Turritella caelata Mörch <i>in</i> Dunker, 1858` to `Turritella caelata Mörch <i>in<i> Dunker, 1858`
[WARN] [2020-02-07 15:01:07] Filtered Scientific Name `Turritella declivis A. Adams & Reeve <i>in</i> Reeve, 1849` to `Turritella declivis A. Adams & Reeve <i>in<i> Reeve, 1849`
[WARN] [2020-02-07 15:01:12] Filtered Scientific Name `Ranella reticularis (Linnaeus, 1758) <i>sensu</i> Deshayes, 1839` to `Ranella reticularis (Linnaeus, 1758) <i>sensu<i> Deshayes, 1839`
[WARN] [2020-02-07 15:01:12] Filtered Scientific Name `Turritella binaestriata Kuroda <i>in</i> Ida, 1952` to `Turritella binaestriata Kuroda <i>in<i> Ida, 1952`
[WARN] [2020-02-07 15:01:30] Filtered Scientific Name `Turritella vittulata A. Adams & Reeve <i>in</i> Reeve, 1849` to `Turritella vittulata A. Adams & Reeve <i>in<i> Reeve, 1849`
[WARN] [2020-02-07 15:01:40] Filtered Scientific Name `Rissoina flammea "Pease" Tryon, 1887` to `Rissoina flammea Pease Tryon, 1887`
[WARN] [2020-02-07 15:02:08] Filtered Scientific Name `Sigambra papagayu Bamber <i>in</i> Muir & Bamber, 2008` to `Sigambra papagayu Bamber <i>in<i> Muir & Bamber, 2008`
[WARN] [2020-02-07 15:02:46] Filtered Scientific Name `Trypanosyllis taboadai Álvarez-Campos, Riesgo & San Martín <i>in</i> Álvarez-Campos, Giribet, San Martín, Rouse & Riesgo, 2017` to `Trypanosyllis taboadai Álvarez-Campos, Riesgo & San Martín <i>in<i> Álvarez-Campos, Giribet, San Martín, Rouse & Riesgo, 2017`
[WARN] [2020-02-07 15:02:46] Filtered Scientific Name `Trypanosyllis californiensis Álvarez-Campos & Rouse <i>in</i> Álvarez-Campos, Giribet, San Martín, Rouse & Riesgo, 2017` to `Trypanosyllis californiensis Álvarez-Campos & Rouse <i>in<i> Álvarez-Campos, Giribet, San Martín, Rouse & Riesgo, 2017`
[WARN] [2020-02-07 15:02:46] Filtered Scientific Name `Trypanosyllis leivai Álvarez-Campos, Riesgo & San Martín <i>in</i> Álvarez-Campos, Giribet, San Martín, Rouse & Riesgo, 2017` to `Trypanosyllis leivai Álvarez-Campos, Riesgo & San Martín <i>in<i> Álvarez-Campos, Giribet, San Martín, Rouse & Riesgo, 2017`
[WARN] [2020-02-07 15:02:46] Filtered Scientific Name `Trypanosyllis luquei Álvarez-Campos & Verdes <i>in</i> Álvarez-Campos, Giribet, San Martín, Rouse & Riesgo, 2017` to `Trypanosyllis luquei Álvarez-Campos & Verdes <i>in<i> Álvarez-Campos, Giribet, San Martín, Rouse & Riesgo, 2017`
[WARN] [2020-02-07 15:02:46] Filtered Scientific Name `Trypanosyllis kalkin Álvarez-Campos & Verdes <i>in</i> Álvarez-Campos, Giribet, San Martín, Rouse & Riesgo, 2017` to `Trypanosyllis kalkin Álvarez-Campos & Verdes <i>in<i> Álvarez-Campos, Giribet, San Martín, Rouse & Riesgo, 2017`
[INFO] [2020-02-07 15:02:49] Loading media diff file into memory (true lines)...
[WARN] [2020-02-07 15:02:49] Could not find a media subtype (format) of "image/png"
[INFO] [2020-02-07 15:07:16] Loading vernaculars diff file into memory (true lines)...
[INFO] [2020-02-07 15:07:55] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-02-07 15:19:28] Loading measurements diff file into memory (true lines)...
[INFO] [2020-02-08 00:51:46] Storing 1639 Attributions
[INFO] [2020-02-08 00:51:46] Processing group of 1639 in 2 groups of 1000
[INFO] [2020-02-08 00:51:46] Average Time: 0.155
[INFO] [2020-02-08 00:51:46] Total Time: 1s
[INFO] [2020-02-08 00:51:46] Storing 639196 References
[INFO] [2020-02-08 00:51:46] Processing group of 639196 in 640 groups of 1000
[INFO] [2020-02-08 01:14:53] Average Time: 2.162
[INFO] [2020-02-08 01:14:53] Total Time: 23m7s
[INFO] [2020-02-08 01:14:53] last 3 / first 3: 0.17
[INFO] [2020-02-08 01:14:53] Std.Dev: 10.993407115175895; Max: 69.51
[INFO] [2020-02-08 01:14:53] Storing 400674 ScientificNames
[INFO] [2020-02-08 01:14:53] Processing group of 400674 in 401 groups of 1000
[INFO] [2020-02-08 01:36:32] Average Time: 3.236
[INFO] [2020-02-08 01:36:32] Total Time: 21m40s
[INFO] [2020-02-08 01:36:32] last 3 / first 3: 0.92
[INFO] [2020-02-08 01:36:32] Std.Dev: 13.415662488300754; Max: 70.74
[INFO] [2020-02-08 01:36:32] Storing 352690 Identifiers
[INFO] [2020-02-08 01:36:32] Processing group of 352690 in 353 groups of 1000
[INFO] [2020-02-08 01:41:56] Average Time: 0.913
[INFO] [2020-02-08 01:41:56] Total Time: 5m24s
[INFO] [2020-02-08 01:41:56] last 3 / first 3: 0.69
[INFO] [2020-02-08 01:41:56] Std.Dev: 7.503599136414471; Max: 71.18
[INFO] [2020-02-08 01:41:56] Storing 352690 NodesReferences
[INFO] [2020-02-08 01:41:56] Processing group of 352690 in 353 groups of 1000
[INFO] [2020-02-08 01:44:53] Average Time: 0.497
[INFO] [2020-02-08 01:44:53] Total Time: 2m57s
[INFO] [2020-02-08 01:44:53] last 3 / first 3: 5.58
[INFO] [2020-02-08 01:44:53] Std.Dev: 5.3583579574343485; Max: 71.41
[INFO] [2020-02-08 01:44:53] Storing 400674 Nodes
[INFO] [2020-02-08 01:44:53] Processing group of 400674 in 401 groups of 1000
[INFO] [2020-02-08 02:06:55] Average Time: 3.294
[INFO] [2020-02-08 02:06:55] Total Time: 22m3s
[INFO] [2020-02-08 02:06:55] last 3 / first 3: 0.8
[INFO] [2020-02-08 02:06:55] Std.Dev: 14.162909305647622; Max: 73.45
[INFO] [2020-02-08 02:06:55] Storing 83030 ContentAttributions
[INFO] [2020-02-08 02:06:55] Processing group of 83030 in 84 groups of 1000
[INFO] [2020-02-08 02:07:05] Average Time: 0.111
[INFO] [2020-02-08 02:07:05] Total Time: 10s
[INFO] [2020-02-08 02:07:05] last 3 / first 3: 0.38
[INFO] [2020-02-08 02:07:05] Std.Dev: 0.16431676725154984; Max: 1.19
[INFO] [2020-02-08 02:07:05] Storing 19860 Media
[INFO] [2020-02-08 02:07:05] Processing group of 19860 in 20 groups of 1000
[INFO] [2020-02-08 02:08:28] Average Time: 4.182
[INFO] [2020-02-08 02:08:28] Total Time: 1m24s
[INFO] [2020-02-08 02:08:28] last 3 / first 3: 30.73
[INFO] [2020-02-08 02:08:28] Std.Dev: 16.295244705127935; Max: 73.4
[INFO] [2020-02-08 02:08:28] Storing 58449 BibliographicCitations
[INFO] [2020-02-08 02:08:28] Processing group of 58449 in 59 groups of 1000
[INFO] [2020-02-08 02:09:57] Average Time: 1.498
[INFO] [2020-02-08 02:09:57] Total Time: 1m29s
[INFO] [2020-02-08 02:09:57] last 3 / first 3: 0.0
[INFO] [2020-02-08 02:09:57] Std.Dev: 9.50263121456368; Max: 73.22
[INFO] [2020-02-08 02:09:57] Storing 57849 ArticlesReferences
[INFO] [2020-02-08 02:09:57] Processing group of 57849 in 58 groups of 1000
[INFO] [2020-02-08 02:10:02] Average Time: 0.09
[INFO] [2020-02-08 02:10:02] Total Time: 6s
[INFO] [2020-02-08 02:10:02] last 3 / first 3: 0.47
[INFO] [2020-02-08 02:10:02] Std.Dev: 0.10954451150103323; Max: 0.78
[INFO] [2020-02-08 02:10:02] Storing 66318 ArticlesSections
[INFO] [2020-02-08 02:10:02] Processing group of 66318 in 67 groups of 1000
[INFO] [2020-02-08 02:10:08] Average Time: 0.083
[INFO] [2020-02-08 02:10:08] Total Time: 6s
[INFO] [2020-02-08 02:10:08] last 3 / first 3: 5.06
[INFO] [2020-02-08 02:10:08] Std.Dev: 0.1341640786499874; Max: 0.78
[INFO] [2020-02-08 02:10:08] Storing 66318 Articles
[INFO] [2020-02-08 02:10:08] Processing group of 66318 in 67 groups of 1000
[INFO] [2020-02-08 02:12:59] Average Time: 2.541
[INFO] [2020-02-08 02:12:59] Total Time: 2m51s
[INFO] [2020-02-08 02:12:59] last 3 / first 3: 0.45
[INFO] [2020-02-08 02:12:59] Std.Dev: 12.562643034011593; Max: 73.63
[INFO] [2020-02-08 02:12:59] Storing 79229 Vernaculars
[INFO] [2020-02-08 02:12:59] Processing group of 79229 in 80 groups of 1000
[INFO] [2020-02-08 02:15:46] Average Time: 2.085
[INFO] [2020-02-08 02:15:46] Total Time: 2m48s
[INFO] [2020-02-08 02:15:46] last 3 / first 3: 0.01
[INFO] [2020-02-08 02:15:46] Std.Dev: 11.61757289626366; Max: 74.62
[INFO] [2020-02-08 02:15:46] Storing 2073724 Occurrences
[INFO] [2020-02-08 02:15:46] Processing group of 2073724 in 2074 groups of 1000
[INFO] [2020-02-08 03:01:32] Average Time: 1.283
[INFO] [2020-02-08 03:01:32] Total Time: 45m47s
[INFO] [2020-02-08 03:01:32] last 3 / first 3: 0.91
[INFO] [2020-02-08 03:01:32] Std.Dev: 9.17065973635485; Max: 78.09
[INFO] [2020-02-08 03:01:32] Storing 442015 TraitsReferences
[INFO] [2020-02-08 03:01:32] Processing group of 442015 in 443 groups of 1000
[INFO] [2020-02-08 03:04:56] Average Time: 0.457
[INFO] [2020-02-08 03:04:56] Total Time: 3m25s
[INFO] [2020-02-08 03:04:56] last 3 / first 3: 0.41
[INFO] [2020-02-08 03:04:56] Std.Dev: 5.253284686745999; Max: 78.55
[INFO] [2020-02-08 03:04:56] Storing 3416758 Traits
[INFO] [2020-02-08 03:04:56] Processing group of 3416758 in 3417 groups of 1000
[INFO] [2020-02-08 05:50:32] Average Time: 2.901
[INFO] [2020-02-08 05:50:32] Total Time: 2h45m37s
[INFO] [2020-02-08 05:50:32] last 3 / first 3: 0.85
[INFO] [2020-02-08 05:50:32] Std.Dev: 14.305103984242827; Max: 91.26
[INFO] [2020-02-08 05:50:32] Storing 6716910 MetaTraits
[INFO] [2020-02-08 05:50:32] Processing group of 6716910 in 6717 groups of 1000
[INFO] [2020-02-08 08:26:26] Average Time: 1.343
[INFO] [2020-02-08 08:26:26] Total Time: 2h35m54s
[INFO] [2020-02-08 08:26:26] last 3 / first 3: 0.16
[INFO] [2020-02-08 08:26:26] Std.Dev: 10.535938496403631; Max: 100.73
[STOP] [2020-02-08 08:26:26] parse_diff_and_store
[START] [2020-02-08 08:26:26] resolve_keys
[INFO] [2020-02-08 08:29:58] Occurrences to nodes (through scientific_names)...
[INFO] [2020-02-08 08:31:54] traits to occurrences...
[INFO] [2020-02-08 08:36:26] traits to nodes (through occurrences)...
[INFO] [2020-02-08 08:37:32] Traits to sex term...
[INFO] [2020-02-08 08:39:02] Traits to lifestage term...
[INFO] [2020-02-08 08:40:32] MetaTraits to traits...
[INFO] [2020-02-08 08:47:50] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-02-08 08:49:44] Assocs to occurrences...
[INFO] [2020-02-08 08:49:44] Assocs to nodes...
[INFO] [2020-02-08 08:49:44] Assoc to sex term...
[INFO] [2020-02-08 08:49:44] Assoc to lifestage term...
[STOP] [2020-02-08 08:49:50] resolve_keys
[START] [2020-02-08 08:49:50] hold_for_later_1
[STOP] [2020-02-08 08:49:50] hold_for_later_1
[START] [2020-02-08 08:49:50] hold_for_later_2
[STOP] [2020-02-08 08:49:50] hold_for_later_2
[START] [2020-02-08 08:49:50] resolve_missing_parents
[STOP] [2020-02-08 08:50:11] resolve_missing_parents
[START] [2020-02-08 08:50:11] rebuild_nodes
[START] [2020-02-08 08:50:11] Flattener#flatten
[START] [2020-02-08 08:50:11] Flattener#study_resource
[START] [2020-02-08 08:50:22] Flattener#build_ancestry
[STOP] [2020-02-08 10:29:54] Flattener#build_ancestry
[INFO] [2020-02-08 10:29:54] 400674 ancestry keys
[START] [2020-02-08 10:29:54] build_node_ancestors
[INFO] [2020-02-08 10:29:54] old ancestors deleted.
[STOP] [2020-02-08 10:35:21] build_node_ancestors
[START] [2020-02-08 10:35:26] Flattener#propagate_ancestor_ids
[STOP] [2020-02-08 10:37:15] Flattener#propagate_ancestor_ids
[STOP] [2020-02-08 10:37:15] Flattener#flatten
[STOP] [2020-02-08 10:37:15] rebuild_nodes
[START] [2020-02-08 10:37:15] resolve_missing_media_owners
[STOP] [2020-02-08 10:37:15] resolve_missing_media_owners
[START] [2020-02-08 10:37:15] sanitize_media_verbatims
[STOP] [2020-02-08 10:37:15] sanitize_media_verbatims
[START] [2020-02-08 10:37:15] queue_downloads
[STOP] [2020-02-08 10:37:15] queue_downloads
[START] [2020-02-08 10:37:15] parse_names
[WARN] [2020-02-08 10:37:16] I see 400674 names which still need to be parsed.
[WARN] [2020-02-08 10:44:01] I see 594 names which still need to be parsed.
[WARN] [2020-02-08 10:44:05] I see 41 names which still need to be parsed.
[WARN] [2020-02-08 10:44:07] I see 33 names which still need to be parsed.
[WARN] [2020-02-08 10:44:10] I see 27 names which still need to be parsed.
[WARN] [2020-02-08 10:44:13] I see 23 names which still need to be parsed.
[WARN] [2020-02-08 10:44:15] I see 19 names which still need to be parsed.
[WARN] [2020-02-08 10:44:18] I see 16 names which still need to be parsed.
[WARN] [2020-02-08 10:44:21] I see 13 names which still need to be parsed.
[WARN] [2020-02-08 10:44:23] I see 10 names which still need to be parsed.
[WARN] [2020-02-08 10:44:26] I see 7 names which still need to be parsed.
[STOP] [2020-02-08 10:44:29] parse_names
[START] [2020-02-08 10:44:29] denormalize_canonical_names_to_nodes
[STOP] [2020-02-08 10:44:34] denormalize_canonical_names_to_nodes
[START] [2020-02-08 10:44:34] match_nodes
[START] [2020-02-08 10:44:34] map_all_nodes_to_pages
[WARN] [2020-02-08 13:42:25] cannot match node with blank canonical: Node#63238263
[WARN] [2020-02-08 15:04:18] cannot match node with blank canonical: Node#63066922
[WARN] [2020-02-08 15:04:18] cannot match node with blank canonical: Node#63066923
[WARN] [2020-02-08 15:04:18] cannot match node with blank canonical: Node#63067076
[STOP] [2020-02-08 21:55:35] map_all_nodes_to_pages
[INFO] [2020-02-08 21:55:35] 35790 Unmatched nodes (of 400674)! That's too many to output. First 10: Biota (#63016394); Myzostoma khanhkhoaensis (#63100547); Contramyzostoma (#63205833); Branchipolynoe eliseae (#63098895); Branchipolynoe halliseyae (#63098896); Branchipolynoe kajsae (#63098897); Branchipolynoe meridae (#63098898); Branchipolynoe tjiasmantoi (#63098899); Harmothoe sarsi sarsi (#63129403); Thormora setosior (#63092764)
[START] [2020-02-08 21:55:35] update_nodes
[STOP] [2020-02-08 21:55:35] update_nodes
[STOP] [2020-02-08 21:55:35] match_nodes
[START] [2020-02-08 21:55:35] reindex_search
[STOP] [2020-02-08 22:08:31] reindex_search
[START] [2020-02-08 22:08:31] normalize_units
[STOP] [2020-02-08 22:17:10] normalize_units
[START] [2020-02-08 22:17:10] calculate_statistics
[STOP] [2020-02-08 22:17:10] calculate_statistics
[START] [2020-02-08 22:17:10] complete_harvest_instance
[START] [2020-02-08 22:17:10] overall_tsv_creation
[INFO] [2020-02-08 22:17:11] Processing group of 400674 in 41 batches of 10000
[INFO] [2020-02-08 22:19:03] 34795 Traits (unfiltered)...
[INFO] [2020-02-08 22:19:16] 34795 Traits (filtered)...
[INFO] [2020-02-08 22:19:16] 0 Associations (filtered)...
[INFO] [2020-02-08 22:20:55] 106442 metadata added.
[INFO] [2020-02-08 22:20:55] 0 metadata added.
[INFO] [2020-02-08 22:22:42] 36165 Traits (unfiltered)...
[INFO] [2020-02-08 22:22:55] 36165 Traits (filtered)...
[INFO] [2020-02-08 22:22:55] 0 Associations (filtered)...
[INFO] [2020-02-08 22:24:45] 96531 metadata added.
[INFO] [2020-02-08 22:24:45] 0 metadata added.
[INFO] [2020-02-08 22:26:33] 51752 Traits (unfiltered)...
[INFO] [2020-02-08 22:26:46] 51752 Traits (filtered)...
[INFO] [2020-02-08 22:26:46] 0 Associations (filtered)...
[INFO] [2020-02-08 22:31:07] 164986 metadata added.
[INFO] [2020-02-08 22:31:07] 0 metadata added.
[INFO] [2020-02-08 22:33:05] 63321 Traits (unfiltered)...
[INFO] [2020-02-08 22:33:18] 63321 Traits (filtered)...
[INFO] [2020-02-08 22:33:18] 0 Associations (filtered)...
[INFO] [2020-02-08 22:37:51] 220867 metadata added.
[INFO] [2020-02-08 22:37:51] 0 metadata added.
[INFO] [2020-02-08 22:39:42] 45981 Traits (unfiltered)...
[INFO] [2020-02-08 22:39:55] 45981 Traits (filtered)...
[INFO] [2020-02-08 22:39:55] 0 Associations (filtered)...
[INFO] [2020-02-08 22:43:44] 169624 metadata added.
[INFO] [2020-02-08 22:43:44] 0 metadata added.
[INFO] [2020-02-08 22:45:49] 53901 Traits (unfiltered)...
[INFO] [2020-02-08 22:46:02] 53901 Traits (filtered)...
[INFO] [2020-02-08 22:46:02] 0 Associations (filtered)...
[INFO] [2020-02-08 22:50:08] 173834 metadata added.
[INFO] [2020-02-08 22:50:08] 0 metadata added.
[INFO] [2020-02-08 22:52:00] 41691 Traits (unfiltered)...
[INFO] [2020-02-08 22:52:13] 41691 Traits (filtered)...
[INFO] [2020-02-08 22:52:13] 0 Associations (filtered)...
[INFO] [2020-02-08 22:55:44] 117966 metadata added.
[INFO] [2020-02-08 22:55:44] 0 metadata added.
[INFO] [2020-02-08 22:57:43] 46409 Traits (unfiltered)...
[INFO] [2020-02-08 22:57:57] 46409 Traits (filtered)...
[INFO] [2020-02-08 22:57:57] 0 Associations (filtered)...
[INFO] [2020-02-08 23:01:43] 141285 metadata added.
[INFO] [2020-02-08 23:01:43] 0 metadata added.
[INFO] [2020-02-08 23:03:37] 24739 Traits (unfiltered)...
[INFO] [2020-02-08 23:03:50] 24739 Traits (filtered)...
[INFO] [2020-02-08 23:03:50] 0 Associations (filtered)...
[INFO] [2020-02-08 23:05:23] 70339 metadata added.
[INFO] [2020-02-08 23:05:23] 0 metadata added.
[INFO] [2020-02-08 23:07:15] 14084 Traits (unfiltered)...
[INFO] [2020-02-08 23:07:28] 14084 Traits (filtered)...
[INFO] [2020-02-08 23:07:29] 0 Associations (filtered)...
[INFO] [2020-02-08 23:08:33] 31716 metadata added.
[INFO] [2020-02-08 23:08:33] 0 metadata added.
[INFO] [2020-02-08 23:10:29] 24831 Traits (unfiltered)...
[INFO] [2020-02-08 23:10:42] 24831 Traits (filtered)...
[INFO] [2020-02-08 23:10:42] 0 Associations (filtered)...
[INFO] [2020-02-08 23:12:19] 86838 metadata added.
[INFO] [2020-02-08 23:12:19] 0 metadata added.
[INFO] [2020-02-08 23:14:20] 51814 Traits (unfiltered)...
[INFO] [2020-02-08 23:14:33] 51814 Traits (filtered)...
[INFO] [2020-02-08 23:14:33] 0 Associations (filtered)...
[INFO] [2020-02-08 23:18:40] 192004 metadata added.
[INFO] [2020-02-08 23:18:40] 0 metadata added.
[INFO] [2020-02-08 23:20:35] 53978 Traits (unfiltered)...
[INFO] [2020-02-08 23:20:48] 53978 Traits (filtered)...
[INFO] [2020-02-08 23:20:48] 0 Associations (filtered)...
[INFO] [2020-02-08 23:24:52] 169012 metadata added.
[INFO] [2020-02-08 23:24:52] 0 metadata added.
[INFO] [2020-02-08 23:26:54] 66245 Traits (unfiltered)...
[INFO] [2020-02-08 23:27:07] 66245 Traits (filtered)...
[INFO] [2020-02-08 23:27:07] 0 Associations (filtered)...
[INFO] [2020-02-08 23:32:03] 256929 metadata added.
[INFO] [2020-02-08 23:32:03] 0 metadata added.
[INFO] [2020-02-08 23:34:07] 47394 Traits (unfiltered)...
[INFO] [2020-02-08 23:34:20] 47394 Traits (filtered)...
[INFO] [2020-02-08 23:34:20] 0 Associations (filtered)...
[INFO] [2020-02-08 23:38:11] 163615 metadata added.
[INFO] [2020-02-08 23:38:11] 0 metadata added.
[INFO] [2020-02-08 23:40:10] 57563 Traits (unfiltered)...
[INFO] [2020-02-08 23:40:23] 57563 Traits (filtered)...
[INFO] [2020-02-08 23:40:23] 0 Associations (filtered)...
[INFO] [2020-02-08 23:44:31] 164137 metadata added.
[INFO] [2020-02-08 23:44:31] 0 metadata added.
[INFO] [2020-02-08 23:46:29] 31712 Traits (unfiltered)...
[INFO] [2020-02-08 23:46:43] 31712 Traits (filtered)...
[INFO] [2020-02-08 23:46:43] 0 Associations (filtered)...
[INFO] [2020-02-08 23:48:29] 97061 metadata added.
[INFO] [2020-02-08 23:48:29] 0 metadata added.
[INFO] [2020-02-08 23:50:21] 12261 Traits (unfiltered)...
[INFO] [2020-02-08 23:50:35] 12261 Traits (filtered)...
[INFO] [2020-02-08 23:50:35] 0 Associations (filtered)...
[INFO] [2020-02-08 23:51:43] 50381 metadata added.
[INFO] [2020-02-08 23:51:43] 0 metadata added.
[INFO] [2020-02-08 23:53:49] 41583 Traits (unfiltered)...
[INFO] [2020-02-08 23:54:02] 41583 Traits (filtered)...
[INFO] [2020-02-08 23:54:02] 0 Associations (filtered)...
[INFO] [2020-02-08 23:57:43] 133065 metadata added.
[INFO] [2020-02-08 23:57:43] 0 metadata added.
[INFO] [2020-02-08 23:59:45] 47611 Traits (unfiltered)...
[INFO] [2020-02-08 23:59:58] 47611 Traits (filtered)...
[INFO] [2020-02-08 23:59:58] 0 Associations (filtered)...
[INFO] [2020-02-09 00:03:59] 167851 metadata added.
[INFO] [2020-02-09 00:03:59] 0 metadata added.
[INFO] [2020-02-09 00:05:51] 78173 Traits (unfiltered)...
[INFO] [2020-02-09 00:06:04] 78173 Traits (filtered)...
[INFO] [2020-02-09 00:06:04] 0 Associations (filtered)...
[INFO] [2020-02-09 00:10:48] 186550 metadata added.
[INFO] [2020-02-09 00:10:48] 0 metadata added.
[INFO] [2020-02-09 00:12:50] 43948 Traits (unfiltered)...
[INFO] [2020-02-09 00:13:04] 43948 Traits (filtered)...
[INFO] [2020-02-09 00:13:04] 0 Associations (filtered)...
[INFO] [2020-02-09 00:16:36] 118217 metadata added.
[INFO] [2020-02-09 00:16:36] 0 metadata added.
[INFO] [2020-02-09 00:18:34] 33746 Traits (unfiltered)...
[INFO] [2020-02-09 00:18:47] 33746 Traits (filtered)...
[INFO] [2020-02-09 00:18:47] 0 Associations (filtered)...
[INFO] [2020-02-09 00:20:42] 98254 metadata added.
[INFO] [2020-02-09 00:20:42] 0 metadata added.
[INFO] [2020-02-09 00:22:31] 44684 Traits (unfiltered)...
[INFO] [2020-02-09 00:22:44] 44684 Traits (filtered)...
[INFO] [2020-02-09 00:22:44] 0 Associations (filtered)...
[INFO] [2020-02-09 00:26:23] 136440 metadata added.
[INFO] [2020-02-09 00:26:23] 0 metadata added.
[INFO] [2020-02-09 00:28:15] 48925 Traits (unfiltered)...
[INFO] [2020-02-09 00:28:28] 48925 Traits (filtered)...
[INFO] [2020-02-09 00:28:28] 0 Associations (filtered)...
[INFO] [2020-02-09 00:32:05] 116789 metadata added.
[INFO] [2020-02-09 00:32:05] 0 metadata added.
[INFO] [2020-02-09 00:33:58] 50866 Traits (unfiltered)...
[INFO] [2020-02-09 00:34:12] 50866 Traits (filtered)...
[INFO] [2020-02-09 00:34:12] 0 Associations (filtered)...
[INFO] [2020-02-09 00:38:02] 132962 metadata added.
[INFO] [2020-02-09 00:38:02] 0 metadata added.
[INFO] [2020-02-09 00:40:02] 35286 Traits (unfiltered)...
[INFO] [2020-02-09 00:40:15] 35286 Traits (filtered)...
[INFO] [2020-02-09 00:40:15] 0 Associations (filtered)...
[INFO] [2020-02-09 00:42:12] 97102 metadata added.
[INFO] [2020-02-09 00:42:12] 0 metadata added.
[INFO] [2020-02-09 00:44:09] 46025 Traits (unfiltered)...
[INFO] [2020-02-09 00:44:22] 46025 Traits (filtered)...
[INFO] [2020-02-09 00:44:23] 0 Associations (filtered)...
[INFO] [2020-02-09 00:47:52] 109356 metadata added.
[INFO] [2020-02-09 00:47:52] 0 metadata added.
[INFO] [2020-02-09 00:49:52] 32420 Traits (unfiltered)...
[INFO] [2020-02-09 00:50:05] 32420 Traits (filtered)...
[INFO] [2020-02-09 00:50:06] 0 Associations (filtered)...
[INFO] [2020-02-09 00:51:52] 79405 metadata added.
[INFO] [2020-02-09 00:51:52] 0 metadata added.
[INFO] [2020-02-09 00:53:48] 30451 Traits (unfiltered)...
[INFO] [2020-02-09 00:54:01] 30451 Traits (filtered)...
[INFO] [2020-02-09 00:54:01] 0 Associations (filtered)...
[INFO] [2020-02-09 00:55:43] 71186 metadata added.
[INFO] [2020-02-09 00:55:43] 0 metadata added.
[INFO] [2020-02-09 00:57:33] 27236 Traits (unfiltered)...
[INFO] [2020-02-09 00:57:47] 27236 Traits (filtered)...
[INFO] [2020-02-09 00:57:47] 0 Associations (filtered)...
[INFO] [2020-02-09 00:59:26] 69660 metadata added.
[INFO] [2020-02-09 00:59:26] 0 metadata added.
[INFO] [2020-02-09 01:01:22] 31435 Traits (unfiltered)...
[INFO] [2020-02-09 01:01:35] 31435 Traits (filtered)...
[INFO] [2020-02-09 01:01:35] 0 Associations (filtered)...
[INFO] [2020-02-09 01:03:22] 79871 metadata added.
[INFO] [2020-02-09 01:03:22] 0 metadata added.
[INFO] [2020-02-09 01:05:14] 34292 Traits (unfiltered)...
[INFO] [2020-02-09 01:05:27] 34292 Traits (filtered)...
[INFO] [2020-02-09 01:05:27] 0 Associations (filtered)...
[INFO] [2020-02-09 01:07:21] 87702 metadata added.
[INFO] [2020-02-09 01:07:21] 0 metadata added.
[INFO] [2020-02-09 01:09:21] 36269 Traits (unfiltered)...
[INFO] [2020-02-09 01:09:34] 36269 Traits (filtered)...
[INFO] [2020-02-09 01:09:34] 0 Associations (filtered)...
[INFO] [2020-02-09 01:11:33] 88445 metadata added.
[INFO] [2020-02-09 01:11:33] 0 metadata added.
[INFO] [2020-02-09 01:13:23] 77305 Traits (unfiltered)...
[INFO] [2020-02-09 01:13:36] 77305 Traits (filtered)...
[INFO] [2020-02-09 01:13:37] 0 Associations (filtered)...
[INFO] [2020-02-09 01:18:26] 172254 metadata added.
[INFO] [2020-02-09 01:18:26] 0 metadata added.
[INFO] [2020-02-09 01:20:22] 37323 Traits (unfiltered)...
[INFO] [2020-02-09 01:20:35] 37323 Traits (filtered)...
[INFO] [2020-02-09 01:20:36] 0 Associations (filtered)...
[INFO] [2020-02-09 01:23:57] 91939 metadata added.
[INFO] [2020-02-09 01:23:57] 0 metadata added.
[INFO] [2020-02-09 01:25:55] 41489 Traits (unfiltered)...
[INFO] [2020-02-09 01:26:08] 41489 Traits (filtered)...
[INFO] [2020-02-09 01:26:08] 0 Associations (filtered)...
[INFO] [2020-02-09 01:29:33] 106100 metadata added.
[INFO] [2020-02-09 01:29:33] 0 metadata added.
[INFO] [2020-02-09 01:31:30] 35955 Traits (unfiltered)...
[INFO] [2020-02-09 01:31:43] 35955 Traits (filtered)...
[INFO] [2020-02-09 01:31:43] 0 Associations (filtered)...
[INFO] [2020-02-09 01:33:35] 72735 metadata added.
[INFO] [2020-02-09 01:33:35] 0 metadata added.
[INFO] [2020-02-09 01:35:30] 36074 Traits (unfiltered)...
[INFO] [2020-02-09 01:35:43] 36074 Traits (filtered)...
[INFO] [2020-02-09 01:35:43] 0 Associations (filtered)...
[INFO] [2020-02-09 01:37:39] 72509 metadata added.
[INFO] [2020-02-09 01:37:39] 0 metadata added.
[INFO] [2020-02-09 01:39:33] 36712 Traits (unfiltered)...
[INFO] [2020-02-09 01:39:47] 36712 Traits (filtered)...
[INFO] [2020-02-09 01:39:50] 0 Associations (filtered)...
[INFO] [2020-02-09 01:41:49] 81538 metadata added.
[INFO] [2020-02-09 01:41:49] 0 metadata added.
[INFO] [2020-02-09 01:42:40] 2419 Traits (unfiltered)...
[INFO] [2020-02-09 01:42:53] 2419 Traits (filtered)...
[INFO] [2020-02-09 01:42:53] 0 Associations (filtered)...
[INFO] [2020-02-09 01:43:34] 6404 metadata added.
[INFO] [2020-02-09 01:43:34] 0 metadata added.
[INFO] [2020-02-09 01:43:34] Average Time: 261.457
[INFO] [2020-02-09 01:43:34] Total Time: 3h26m24s
[INFO] [2020-02-09 01:43:34] last 3 / first 3: 0.69
[INFO] [2020-02-09 01:43:34] Std.Dev: 77.00491542752319; Max: 388.36
[STOP] [2020-02-09 01:43:34] overall_tsv_creation
[INFO] [2020-02-09 01:43:34] Done. Check your files:
[STOP] [2020-02-09 01:43:39] complete_harvest_instance
[ERR] [2020-02-09 01:43:39] RuntimeError
[ERR] [2020-02-09 01:43:39] OUT OF MEMORY. This is NOT a problem for this resource (really, it isn't), but means that you should have someone restart the containers!
[ERR] [2020-02-09 01:43:39] ../models/publisher.rb:76:in `rescue in block in by_resource'
[ERR] [2020-02-09 01:43:39] ../models/publisher.rb:73:in `block in by_resource'
[ERR] [2020-02-09 01:43:39] ../models/publisher.rb:72:in `each_key'
[ERR] [2020-02-09 01:43:39] ../models/publisher.rb:72:in `by_resource'
[ERR] [2020-02-09 01:43:39] ../models/publisher.rb:6:in `by_resource'
[ERR] [2020-02-09 01:43:39] ../models/resource_harvester.rb:646:in `complete_harvest_instance'
[ERR] [2020-02-09 01:43:39] ../models/resource_harvester.rb:86:in `block (3 levels) in start'
[ERR] [2020-02-09 01:43:39] ../models/logged_process.rb:19:in `run_step'
[ERR] [2020-02-09 01:43:39] ../models/resource_harvester.rb:86:in `block (2 levels) in start'
[ERR] [2020-02-09 01:43:39] ../models/resource_harvester.rb:75:in `each_key'
[ERR] [2020-02-09 01:43:39] ../models/resource_harvester.rb:75:in `block in start'
[ERR] [2020-02-09 01:43:39] ../models/resource.rb:139:in `lock'
[ERR] [2020-02-09 01:43:39] ../models/resource_harvester.rb:72:in `start'
[ERR] [2020-02-09 01:43:39] ../models/resource.rb:223:in `harvest'
[ERR] [2020-02-09 01:43:39] ../models/resource.rb:199:in `re_download_opendata_and_harvest'
[STOP] [2020-02-09 01:43:39] logged process, took 125819.09
[START] [2020-02-10 10:15:29] logged process
[INFO] [2020-02-10 10:15:29] Already completed stage create_harvest_instance, skipping...
[INFO] [2020-02-10 10:15:29] Already completed stage fetch_files, skipping...
[INFO] [2020-02-10 10:15:29] Already completed stage validate_each_file, skipping...
[INFO] [2020-02-10 10:15:29] Already completed stage convert_to_csv, skipping...
[INFO] [2020-02-10 10:15:29] Already completed stage calculate_delta, skipping...
[INFO] [2020-02-10 10:15:29] Already completed stage parse_diff_and_store, skipping...
[INFO] [2020-02-10 10:15:29] Already completed stage resolve_keys, skipping...
[INFO] [2020-02-10 10:15:29] Already completed stage hold_for_later_1, skipping...
[INFO] [2020-02-10 10:15:29] Already completed stage hold_for_later_2, skipping...
[INFO] [2020-02-10 10:15:29] Already completed stage resolve_missing_parents, skipping...
[INFO] [2020-02-10 10:15:29] Already completed stage rebuild_nodes, skipping...
[INFO] [2020-02-10 10:15:29] Already completed stage resolve_missing_media_owners, skipping...
[INFO] [2020-02-10 10:15:29] Already completed stage sanitize_media_verbatims, skipping...
[INFO] [2020-02-10 10:15:29] Already completed stage queue_downloads, skipping...
[INFO] [2020-02-10 10:15:29] Already completed stage parse_names, skipping...
[INFO] [2020-02-10 10:15:29] Already completed stage denormalize_canonical_names_to_nodes, skipping...
[INFO] [2020-02-10 10:15:29] Already completed stage match_nodes, skipping...
[INFO] [2020-02-10 10:15:29] Already completed stage reindex_search, skipping...
[INFO] [2020-02-10 10:15:29] Already completed stage normalize_units, skipping...
[INFO] [2020-02-10 10:15:29] Already completed stage calculate_statistics, skipping...
[START] [2020-02-10 10:15:29] complete_harvest_instance
[START] [2020-02-10 10:15:29] overall_tsv_creation
[INFO] [2020-02-10 10:15:30] Processing group of 400674 in 41 batches of 10000
[INFO] [2020-02-10 10:17:10] 34795 Traits (unfiltered)...
[INFO] [2020-02-10 10:17:23] 34795 Traits (filtered)...
[INFO] [2020-02-10 10:17:24] 0 Associations (filtered)...
[INFO] [2020-02-10 10:19:04] 106442 metadata added.
[INFO] [2020-02-10 10:19:04] 0 metadata added.
[INFO] [2020-02-10 10:20:52] 36165 Traits (unfiltered)...
[INFO] [2020-02-10 10:21:05] 36165 Traits (filtered)...
[INFO] [2020-02-10 10:21:05] 0 Associations (filtered)...
[INFO] [2020-02-10 10:22:58] 96531 metadata added.
[INFO] [2020-02-10 10:22:58] 0 metadata added.
[INFO] [2020-02-10 10:24:49] 51752 Traits (unfiltered)...
[INFO] [2020-02-10 10:25:02] 51752 Traits (filtered)...
[INFO] [2020-02-10 10:25:03] 0 Associations (filtered)...
[INFO] [2020-02-10 10:29:07] 164986 metadata added.
[INFO] [2020-02-10 10:29:07] 0 metadata added.
[INFO] [2020-02-10 10:31:07] 63321 Traits (unfiltered)...
[INFO] [2020-02-10 10:31:20] 63321 Traits (filtered)...
[INFO] [2020-02-10 10:31:20] 0 Associations (filtered)...
[INFO] [2020-02-10 10:35:51] 220867 metadata added.
[INFO] [2020-02-10 10:35:51] 0 metadata added.
[INFO] [2020-02-10 10:37:47] 45981 Traits (unfiltered)...
[INFO] [2020-02-10 10:38:00] 45981 Traits (filtered)...
[INFO] [2020-02-10 10:38:00] 0 Associations (filtered)...
[INFO] [2020-02-10 10:41:51] 169624 metadata added.
[INFO] [2020-02-10 10:41:51] 0 metadata added.
[INFO] [2020-02-10 10:43:58] 53901 Traits (unfiltered)...
[INFO] [2020-02-10 10:44:11] 53901 Traits (filtered)...
[INFO] [2020-02-10 10:44:11] 0 Associations (filtered)...
[INFO] [2020-02-10 10:48:21] 173834 metadata added.
[INFO] [2020-02-10 10:48:21] 0 metadata added.
[INFO] [2020-02-10 10:50:16] 41691 Traits (unfiltered)...
[INFO] [2020-02-10 10:50:30] 41691 Traits (filtered)...
[INFO] [2020-02-10 10:50:30] 0 Associations (filtered)...
[INFO] [2020-02-10 10:54:00] 117966 metadata added.
[INFO] [2020-02-10 10:54:00] 0 metadata added.
[INFO] [2020-02-10 10:55:54] 46409 Traits (unfiltered)...
[INFO] [2020-02-10 10:56:07] 46409 Traits (filtered)...
[INFO] [2020-02-10 10:56:07] 0 Associations (filtered)...
[INFO] [2020-02-10 10:59:49] 141285 metadata added.
[INFO] [2020-02-10 10:59:49] 0 metadata added.
[INFO] [2020-02-10 11:01:40] 24739 Traits (unfiltered)...
[INFO] [2020-02-10 11:01:53] 24739 Traits (filtered)...
[INFO] [2020-02-10 11:01:53] 0 Associations (filtered)...
[INFO] [2020-02-10 11:03:21] 70339 metadata added.
[INFO] [2020-02-10 11:03:21] 0 metadata added.
[INFO] [2020-02-10 11:05:09] 14084 Traits (unfiltered)...
[INFO] [2020-02-10 11:05:22] 14084 Traits (filtered)...
[INFO] [2020-02-10 11:05:22] 0 Associations (filtered)...
[INFO] [2020-02-10 11:06:27] 31716 metadata added.
[INFO] [2020-02-10 11:06:27] 0 metadata added.
[INFO] [2020-02-10 11:08:19] 24831 Traits (unfiltered)...
[INFO] [2020-02-10 11:08:32] 24831 Traits (filtered)...
[INFO] [2020-02-10 11:08:32] 0 Associations (filtered)...
[INFO] [2020-02-10 11:10:06] 86838 metadata added.
[INFO] [2020-02-10 11:10:06] 0 metadata added.
[INFO] [2020-02-10 11:12:05] 51814 Traits (unfiltered)...
[INFO] [2020-02-10 11:12:18] 51814 Traits (filtered)...
[INFO] [2020-02-10 11:12:18] 0 Associations (filtered)...
[INFO] [2020-02-10 11:16:16] 192004 metadata added.
[INFO] [2020-02-10 11:16:16] 0 metadata added.
[INFO] [2020-02-10 11:18:07] 53978 Traits (unfiltered)...
[INFO] [2020-02-10 11:18:20] 53978 Traits (filtered)...
[INFO] [2020-02-10 11:18:21] 0 Associations (filtered)...
[INFO] [2020-02-10 11:22:15] 169012 metadata added.
[INFO] [2020-02-10 11:22:15] 0 metadata added.
[INFO] [2020-02-10 11:24:11] 66245 Traits (unfiltered)...
[INFO] [2020-02-10 11:24:24] 66245 Traits (filtered)...
[INFO] [2020-02-10 11:24:25] 0 Associations (filtered)...
[INFO] [2020-02-10 11:28:58] 256929 metadata added.
[INFO] [2020-02-10 11:28:58] 0 metadata added.
[INFO] [2020-02-10 11:31:01] 47394 Traits (unfiltered)...
[INFO] [2020-02-10 11:31:14] 47394 Traits (filtered)...
[INFO] [2020-02-10 11:31:14] 0 Associations (filtered)...
[INFO] [2020-02-10 11:34:57] 163615 metadata added.
[INFO] [2020-02-10 11:34:57] 0 metadata added.
[INFO] [2020-02-10 11:36:49] 57563 Traits (unfiltered)...
[INFO] [2020-02-10 11:37:03] 57563 Traits (filtered)...
[INFO] [2020-02-10 11:37:03] 0 Associations (filtered)...
[INFO] [2020-02-10 11:41:00] 164137 metadata added.
[INFO] [2020-02-10 11:41:00] 0 metadata added.
[INFO] [2020-02-10 11:42:55] 31712 Traits (unfiltered)...
[INFO] [2020-02-10 11:43:08] 31712 Traits (filtered)...
[INFO] [2020-02-10 11:43:08] 0 Associations (filtered)...
[INFO] [2020-02-10 11:44:49] 97061 metadata added.
[INFO] [2020-02-10 11:44:49] 0 metadata added.
[INFO] [2020-02-10 11:46:38] 12261 Traits (unfiltered)...
[INFO] [2020-02-10 11:46:51] 12261 Traits (filtered)...
[INFO] [2020-02-10 11:46:52] 0 Associations (filtered)...
[INFO] [2020-02-10 11:47:54] 50381 metadata added.
[INFO] [2020-02-10 11:47:54] 0 metadata added.
[INFO] [2020-02-10 11:49:51] 41583 Traits (unfiltered)...
[INFO] [2020-02-10 11:50:04] 41583 Traits (filtered)...
[INFO] [2020-02-10 11:50:04] 0 Associations (filtered)...
[INFO] [2020-02-10 11:53:34] 133065 metadata added.
[INFO] [2020-02-10 11:53:34] 0 metadata added.
[INFO] [2020-02-10 11:55:28] 47611 Traits (unfiltered)...
[INFO] [2020-02-10 11:55:41] 47611 Traits (filtered)...
[INFO] [2020-02-10 11:55:41] 0 Associations (filtered)...
[INFO] [2020-02-10 11:59:29] 167851 metadata added.
[INFO] [2020-02-10 11:59:29] 0 metadata added.
[INFO] [2020-02-10 12:01:20] 78173 Traits (unfiltered)...
[INFO] [2020-02-10 12:01:33] 78173 Traits (filtered)...
[INFO] [2020-02-10 12:01:34] 0 Associations (filtered)...
[INFO] [2020-02-10 12:06:00] 186550 metadata added.
[INFO] [2020-02-10 12:06:00] 0 metadata added.
[INFO] [2020-02-10 12:07:53] 43948 Traits (unfiltered)...
[INFO] [2020-02-10 12:08:06] 43948 Traits (filtered)...
[INFO] [2020-02-10 12:08:07] 0 Associations (filtered)...
[INFO] [2020-02-10 12:11:25] 118217 metadata added.
[INFO] [2020-02-10 12:11:25] 0 metadata added.
[INFO] [2020-02-10 12:13:18] 33746 Traits (unfiltered)...
[INFO] [2020-02-10 12:13:31] 33746 Traits (filtered)...
[INFO] [2020-02-10 12:13:31] 0 Associations (filtered)...
[INFO] [2020-02-10 12:15:15] 98254 metadata added.
[INFO] [2020-02-10 12:15:15] 0 metadata added.
[INFO] [2020-02-10 12:17:06] 44684 Traits (unfiltered)...
[INFO] [2020-02-10 12:17:19] 44684 Traits (filtered)...
[INFO] [2020-02-10 12:17:19] 0 Associations (filtered)...
[INFO] [2020-02-10 12:20:43] 136440 metadata added.
[INFO] [2020-02-10 12:20:43] 0 metadata added.
[INFO] [2020-02-10 12:22:31] 48925 Traits (unfiltered)...
[INFO] [2020-02-10 12:22:44] 48925 Traits (filtered)...
[INFO] [2020-02-10 12:22:44] 0 Associations (filtered)...
[INFO] [2020-02-10 12:26:09] 116789 metadata added.
[INFO] [2020-02-10 12:26:09] 0 metadata added.
[INFO] [2020-02-10 12:27:58] 50866 Traits (unfiltered)...
[INFO] [2020-02-10 12:28:11] 50866 Traits (filtered)...
[INFO] [2020-02-10 12:28:11] 0 Associations (filtered)...
[INFO] [2020-02-10 12:31:44] 132962 metadata added.
[INFO] [2020-02-10 12:31:44] 0 metadata added.
[INFO] [2020-02-10 12:33:39] 35286 Traits (unfiltered)...
[INFO] [2020-02-10 12:33:52] 35286 Traits (filtered)...
[INFO] [2020-02-10 12:33:52] 0 Associations (filtered)...
[INFO] [2020-02-10 12:35:41] 97102 metadata added.
[INFO] [2020-02-10 12:35:41] 0 metadata added.
[INFO] [2020-02-10 12:37:28] 46025 Traits (unfiltered)...
[INFO] [2020-02-10 12:37:41] 46025 Traits (filtered)...
[INFO] [2020-02-10 12:37:41] 0 Associations (filtered)...
[INFO] [2020-02-10 12:41:07] 109356 metadata added.
[INFO] [2020-02-10 12:41:07] 0 metadata added.
[INFO] [2020-02-10 12:42:54] 32420 Traits (unfiltered)...
[INFO] [2020-02-10 12:43:07] 32420 Traits (filtered)...
[INFO] [2020-02-10 12:43:07] 0 Associations (filtered)...
[INFO] [2020-02-10 12:44:46] 79405 metadata added.
[INFO] [2020-02-10 12:44:46] 0 metadata added.
[INFO] [2020-02-10 12:46:34] 30451 Traits (unfiltered)...
[INFO] [2020-02-10 12:46:48] 30451 Traits (filtered)...
[INFO] [2020-02-10 12:46:48] 0 Associations (filtered)...
[INFO] [2020-02-10 12:48:22] 71186 metadata added.
[INFO] [2020-02-10 12:48:22] 0 metadata added.
[INFO] [2020-02-10 12:50:12] 27236 Traits (unfiltered)...
[INFO] [2020-02-10 12:50:25] 27236 Traits (filtered)...
[INFO] [2020-02-10 12:50:25] 0 Associations (filtered)...
[INFO] [2020-02-10 12:51:56] 69660 metadata added.
[INFO] [2020-02-10 12:51:56] 0 metadata added.
[INFO] [2020-02-10 12:53:48] 31435 Traits (unfiltered)...
[INFO] [2020-02-10 12:54:01] 31435 Traits (filtered)...
[INFO] [2020-02-10 12:54:01] 0 Associations (filtered)...
[INFO] [2020-02-10 12:55:42] 79871 metadata added.
[INFO] [2020-02-10 12:55:42] 0 metadata added.
[INFO] [2020-02-10 12:57:28] 34292 Traits (unfiltered)...
[INFO] [2020-02-10 12:57:41] 34292 Traits (filtered)...
[INFO] [2020-02-10 12:57:43] 0 Associations (filtered)...
[INFO] [2020-02-10 12:59:27] 87702 metadata added.
[INFO] [2020-02-10 12:59:27] 0 metadata added.
[INFO] [2020-02-10 13:01:21] 36269 Traits (unfiltered)...
[INFO] [2020-02-10 13:01:34] 36269 Traits (filtered)...
[INFO] [2020-02-10 13:01:34] 0 Associations (filtered)...
[INFO] [2020-02-10 13:03:29] 88445 metadata added.
[INFO] [2020-02-10 13:03:29] 0 metadata added.
[INFO] [2020-02-10 13:05:12] 77305 Traits (unfiltered)...
[INFO] [2020-02-10 13:05:25] 77305 Traits (filtered)...
[INFO] [2020-02-10 13:05:25] 0 Associations (filtered)...
[INFO] [2020-02-10 13:09:59] 172254 metadata added.
[INFO] [2020-02-10 13:09:59] 0 metadata added.
[INFO] [2020-02-10 13:11:47] 37323 Traits (unfiltered)...
[INFO] [2020-02-10 13:12:00] 37323 Traits (filtered)...
[INFO] [2020-02-10 13:12:00] 0 Associations (filtered)...
[INFO] [2020-02-10 13:15:08] 91939 metadata added.
[INFO] [2020-02-10 13:15:08] 0 metadata added.
[INFO] [2020-02-10 13:16:58] 41489 Traits (unfiltered)...
[INFO] [2020-02-10 13:17:11] 41489 Traits (filtered)...
[INFO] [2020-02-10 13:17:11] 0 Associations (filtered)...
[INFO] [2020-02-10 13:20:30] 106100 metadata added.
[INFO] [2020-02-10 13:20:30] 0 metadata added.
[INFO] [2020-02-10 13:22:20] 35955 Traits (unfiltered)...
[INFO] [2020-02-10 13:22:33] 35955 Traits (filtered)...
[INFO] [2020-02-10 13:22:33] 0 Associations (filtered)...
[INFO] [2020-02-10 13:24:17] 72735 metadata added.
[INFO] [2020-02-10 13:24:17] 0 metadata added.
[INFO] [2020-02-10 13:26:06] 36074 Traits (unfiltered)...
[INFO] [2020-02-10 13:26:19] 36074 Traits (filtered)...
[INFO] [2020-02-10 13:26:19] 0 Associations (filtered)...
[INFO] [2020-02-10 13:28:07] 72509 metadata added.
[INFO] [2020-02-10 13:28:07] 0 metadata added.
[INFO] [2020-02-10 13:29:55] 36712 Traits (unfiltered)...
[INFO] [2020-02-10 13:30:11] 36712 Traits (filtered)...
[INFO] [2020-02-10 13:30:11] 0 Associations (filtered)...
[INFO] [2020-02-10 13:32:02] 81538 metadata added.
[INFO] [2020-02-10 13:32:02] 0 metadata added.
[INFO] [2020-02-10 13:32:53] 2419 Traits (unfiltered)...
[INFO] [2020-02-10 13:33:06] 2419 Traits (filtered)...
[INFO] [2020-02-10 13:33:06] 0 Associations (filtered)...
[INFO] [2020-02-10 13:33:46] 6404 metadata added.
[INFO] [2020-02-10 13:33:46] 0 metadata added.
[INFO] [2020-02-10 13:33:46] Average Time: 251.775
[INFO] [2020-02-10 13:33:46] Total Time: 3h18m17s
[INFO] [2020-02-10 13:33:46] last 3 / first 3: 0.66
[INFO] [2020-02-10 13:33:46] Std.Dev: 74.87443488935325; Max: 362.26
[STOP] [2020-02-10 13:33:46] overall_tsv_creation
[INFO] [2020-02-10 13:33:46] Done. Check your files:
[INFO] [2020-02-10 13:33:46] (48032 lines) /app/public/data/world_reg_mar_sp/publish_bibliographic_citations.tsv
[INFO] [2020-02-10 13:33:46] (400659 lines) /app/public/data/world_reg_mar_sp/publish_nodes.tsv
[INFO] [2020-02-10 13:33:46] (352690 lines) /app/public/data/world_reg_mar_sp/publish_identifiers.tsv
[INFO] [2020-02-10 13:33:46] (2253032 lines) /app/public/data/world_reg_mar_sp/publish_node_ancestors.tsv
[INFO] [2020-02-10 13:33:46] (400674 lines) /app/public/data/world_reg_mar_sp/publish_scientific_names.tsv
[INFO] [2020-02-10 13:33:46] (15172 lines) /app/public/data/world_reg_mar_sp/publish_media.tsv
[INFO] [2020-02-10 13:33:46] (54870 lines) /app/public/data/world_reg_mar_sp/publish_articles.tsv
[INFO] [2020-02-10 13:33:46] (14739 lines) /app/public/data/world_reg_mar_sp/publish_image_info.tsv
[INFO] [2020-02-10 13:33:46] (67996 lines) /app/public/data/world_reg_mar_sp/publish_vernaculars.tsv
[INFO] [2020-02-10 13:33:46] (355727 lines) /app/public/data/world_reg_mar_sp/publish_references.tsv
[INFO] [2020-02-10 13:33:46] (66929 lines) /app/public/data/world_reg_mar_sp/publish_attributions.tsv
[INFO] [2020-02-10 13:33:46] (54848 lines) /app/public/data/world_reg_mar_sp/publish_content_sections.tsv
[INFO] [2020-02-10 13:33:47] (355727 lines) /app/public/data/world_reg_mar_sp/publish_referents.tsv
[INFO] [2020-02-10 13:33:47] (1688864 lines) /app/public/data/world_reg_mar_sp/publish_traits.tsv
[INFO] [2020-02-10 13:33:47] (4849902 lines) /app/public/data/world_reg_mar_sp/publish_metadata.tsv
[STOP] [2020-02-10 13:33:47] complete_harvest_instance
[START] [2020-02-10 13:33:47] completed
[STOP] [2020-02-10 13:33:47] completed
[STOP] [2020-02-10 13:33:47] logged process, took 11898.45

Latest Process