EOL Resource Harvester

Abbreviation: Globi

OpenData URL: c8392978-16c2-453b-8f0e-668fbf284b61

Minimum Days Between Harvests: 0

Harvesting Status: harvested

Nodes: 496866

URLs for node IDs will direct the client to: "(The ID itself: this assumes the IDs are valid URIs)"

"This resource will NOT be harvested after harvesting. It must be harvested manually."

Trusted: Yes.

Classification Resource: No.

May Contain Duplicate Taxa: Yes.

Can perform trait diffs: No.

Created: 07 Dec 12:33

Last Updated: 23 Mar 01:29

Expected File Format Definitions

refs: reference.tab (18 fields) (1799879 lines)

nodes: taxon.tab (15 fields) (394185 lines)

occurrences: occurrence_specific.tab (29 fields) (6491143 lines)

assocs: association.tab (12 fields) (2980195 lines)

Harvests

18 Mar 21:34: Completed: 23 Mar 01:29 (took 434 minutes)

Processes

22 Mar 18:16 EXITED currently processing 0

18 Mar 21:33 logged process: 320daa964f6816b4527acd69d8cac2e18b1e7bae Resource#remove_media_from_disk → logged process: 320daa964f6816b4527acd69d8cac2e18b1e7bae Resource#remove_media_from_disk → Flattener#study_resource → match_nodes → map_all_nodes_to_pages currently processing 190000 (38.2%) of 496866 (last five times: 0.2s, 0.0s, 0.0s, 0.0s, 0.0s) (average of all times: 0.03s)

18 Jan 13:22 EXITED currently processing 0

08 Jan 11:07 EXITED currently processing 0

05 Jan 11:01 EXITED currently processing 0

04 Jan 13:46 EXITED currently processing 0

29 Dec 05:43 EXITED currently processing 0

20 Nov 00:53 logged process: 32ceb99c367e59b4eb9538b20081c91debe3470f Restore deleted line

30 Oct 10:03 EXITED currently processing 0

10 Feb 11:39 EXITED currently processing 0

10 Feb 10:19 EXITED

10 Feb 09:50 EXITED

09 Feb 15:12 logged process: 6a863f1e73ce2cb1722cb96b0ebbd087a11c4304 Okay → I think this will fix the downloaded images bug. → logged process: 6a863f1e73ce2cb1722cb96b0ebbd087a11c4304 Okay → I think this will fix the downloaded images bug. → parse_diff_and_store currently processing 8100000 (74.2%) of 10913460 (last five times: 44.6s, 46.1s, 44.7s, 46.0s, 43.7s) (average of all times: 1.83s)

03 Feb 09:37 EXITED currently processing 0

02 Feb 17:42 logged process: 6a863f1e73ce2cb1722cb96b0ebbd087a11c4304 Okay → I think this will fix the downloaded images bug. → logged process: 6a863f1e73ce2cb1722cb96b0ebbd087a11c4304 Okay → I think this will fix the downloaded images bug. → parse_diff_and_store currently processing 6700000 (61.3%) of 10922303 (last five times: 38.7s, 39.0s, 39.4s, 39.5s, 39.2s) (average of all times: 1.54s)

02 Dec 14:57 EXITED currently processing 0

28 Nov 13:09 EXITED

22 Nov 16:44 logged process: 8a4ee2952f2a2ead9dbcbd7dbda333ac2885d8af Attempt to change the WebDb reconnect code to parallel ActiveRecord currently processing 32 (72.7%) of 44 (last five times: 327.6s, 570.2s, 734.6s, 363.4s, 914.0s) (average of all times: 109.68s)

22 Nov 16:17 logged process: c52eec20be9cd368822df46a5ba2cf08e1414a4d Ahh → I think I finally found the culprit for timeouts. → overall_tsv_creation currently processing 1 (2.2%) of 44

22 Nov 14:31 logged process: ae126462f928c7f9d93ad0060b136552c7a1f40e Reverting the connection-maintaining code--I think it may be causing a failure → overall_tsv_creation currently processing 1 (2.2%) of 44

22 Nov 14:14 logged process: b30260a8ec2cfe84a46537c1acf5bd84896d93da Adding indexes to various tables on node_id → overall_tsv_creation currently processing 1 (2.2%) of 44

22 Nov 12:50 logged process: e6307e6d67cff1df9677b2a13bb2155868868f3c More DB connection changes → logged process: e6307e6d67cff1df9677b2a13bb2155868868f3c More DB connection changes currently processing 0

21 Nov 18:50 logged process: e6307e6d67cff1df9677b2a13bb2155868868f3c More DB connection changes → overall_tsv_creation currently processing 1 (2.2%) of 44

21 Nov 17:34 logged process: 1e0186d14475d9a4f6675c9d82c17afba9ce1209 Attempting a fix to DB connection errors. currently processing 0

21 Nov 14:15 logged process: b2d936622fcfc708ed5519cedbe841bb3639fd3a Blast! The command wasn't in the dockerfile... → overall_tsv_creation currently processing 1 (2.2%) of 44

21 Nov 13:09 logged process: b2d936622fcfc708ed5519cedbe841bb3639fd3a Blast! The command wasn't in the dockerfile... → complete_harvest_instance → overall_tsv_creation currently processing 1 (2.2%) of 44

16 Nov 13:48 logged process: 4222b75bd0fa71031cc680547909c16c0485c6c1 Another attempt to stop disconnecting from the DB on large trait files → overall_tsv_creation currently processing 1 (2.2%) of 44

14 Nov 13:09 logged process: 4222b75bd0fa71031cc680547909c16c0485c6c1 Another attempt to stop disconnecting from the DB on large trait files → complete_harvest_instance → overall_tsv_creation currently processing 1 (2.2%) of 44

14 Nov 09:28

12 Nov 21:48

08 Nov 08:22 EXITED currently processing 0

03 Nov 14:20 EXITED currently processing 1 (0.0%) of 8767902

03 Nov 12:25 EXITED

26 Jul 18:58 EXITED currently processing 0

25 Jul 09:32 EXITED currently processing 0

22 Jul 18:27 logged process: 8d5d2f1bbf9481a1db742c79c9f2a9622e4ab562 Hopefully fix a timeout error in publishing step → logged process: 8d5d2f1bbf9481a1db742c79c9f2a9622e4ab562 Hopefully fix a timeout error in publishing step → Flattener#study_resource → complete_harvest_instance → overall_tsv_creation currently processing 32 (74.4%) of 43 (last five times: 381.3s, 596.3s, 743.9s, 396.9s, 884.6s) (average of all times: 110.2s)

16 Jul 15:26

15 Jul 09:43 logged process: 8d5d2f1bbf9481a1db742c79c9f2a9622e4ab562 Hopefully fix a timeout error in publishing step currently processing 0

11 Jul 18:13 EXITED currently processing 0

29 Jun 15:01 logged process: 6df429c723fb9a7a30f4470dcad4149c9a92d9ce Closing in on the problem... → parse_diff_and_store currently processing 9800000 (64.9%) of 15097699 (last five times: 51.2s, 51.8s, 51.2s, 50.7s, 51.0s) (average of all times: 2.15s)

29 Jun 13:29 EXITED

29 Jun 12:53 EXITED

29 Jun 12:28 EXITED

29 Jun 09:17 EXITED

29 Jun 08:25 EXITED

29 Jun 07:24 EXITED

29 Jun 07:02 EXITED

29 Jun 06:08 EXITED

29 Jun 05:49 EXITED

28 Jun 22:32 EXITED

30 May 08:03 EXITED currently processing 0

29 May 17:22 EXITED

27 Mar 09:14 logged process: 33be8921669fcab67b70934a7e27b733da6c8749 Not sure how Resource.publish EVER worked! :S currently processing 0

26 Mar 13:58 logged process: 33be8921669fcab67b70934a7e27b733da6c8749 Not sure how Resource.publish EVER worked! :S currently processing 0

26 Mar 13:50 logged process: 56ce2b69a9028bab6f3848416320c2a17cd07071 merge → logged process: 56ce2b69a9028bab6f3848416320c2a17cd07071 merge

26 Mar 11:52 logged process: 56ce2b69a9028bab6f3848416320c2a17cd07071 merge → logged process: 56ce2b69a9028bab6f3848416320c2a17cd07071 merge

25 Mar 11:36 logged process: 56ce2b69a9028bab6f3848416320c2a17cd07071 merge → logged process: 56ce2b69a9028bab6f3848416320c2a17cd07071 merge currently processing 0

25 Mar 09:47 EXITED currently processing 0

24 Mar 10:47 logged process: 56ce2b69a9028bab6f3848416320c2a17cd07071 merge → Flattener#study_resource → complete_harvest_instance → overall_tsv_creation currently processing 32 (76.1%) of 42 (last five times: 649.4s, 665.1s, 811.9s, 560.3s, 771.6s) (average of all times: 130.43s)

23 Mar 13:30 logged process: 56ce2b69a9028bab6f3848416320c2a17cd07071 merge → resolve_keys currently processing 0

23 Mar 13:23 EXITED

22 Mar 13:27 EXITED currently processing 1 (0.0%) of 417020

22 Mar 13:09 EXITED

21 Mar 07:47 EXITED currently processing 1 (0.0%) of 393311

18 Mar 19:41 EXITED

16 Mar 16:15 logged process: b9e5a7623d4f2699e89c51ab6f485ca13bb4fae0 w/e → Flattener#study_resource → match_nodes → map_all_nodes_to_pages currently processing 11000 (2.6%) of 417020 (last five times: 0.0s, 0.0s, 0.0s, 0.0s, 0.0s) (average of all times: 0.0s)

16 Mar 12:15 logged process: b9e5a7623d4f2699e89c51ab6f485ca13bb4fae0 w/e → parse_diff_and_store currently processing 0

12 Mar 16:54 EXITED currently processing 0 (0.0%) of 5552475

12 Mar 12:15 EXITED currently processing 0 (0.0%) of 12345

12 Mar 06:52 EXITED currently processing 0 (0.0%) of 5552475

10 Mar 15:16 logged process: 5874899cfdc96fbba9d010d2f60fda387591c045 merge → parse_diff_and_store currently processing 1458 (26.2%) of 5553 (last five times: 0.3s, 0.3s, 0.4s, 1.5s, 0.3s) (average of all times: 0.26s)

10 Mar 15:04 EXITED

10 Mar 13:07 EXITED

03 Feb 10:54 EXITED currently processing 0

02 Feb 11:07 EXITED currently processing 0

02 Feb 08:01 EXITED currently processing 0

01 Feb 10:58 EXITED currently processing 0

31 Jan 13:49 EXITED currently processing 0

31 Jan 12:23 logged process: db43953df252c3b7927982cd277219a363391313 Add'l logging for Publishing traits TSV → complete_harvest_instance → overall_tsv_creation currently processing 1 (2.2%) of 44

31 Jan 09:52 logged process: db43953df252c3b7927982cd277219a363391313 Add'l logging for Publishing traits TSV → complete_harvest_instance → overall_tsv_creation currently processing 1 (2.2%) of 44

31 Jan 08:50 logged process: db43953df252c3b7927982cd277219a363391313 Add'l logging for Publishing traits TSV → complete_harvest_instance → overall_tsv_creation currently processing 1 (2.2%) of 44

30 Jan 12:29 EXITED currently processing 0

28 Jan 07:42 logged process: a706e5f1d91cb4f74387805c62f447594cc55029 Dupe of commit (muddled by merge) manually added here → fixes timeout → complete_harvest_instance → overall_tsv_creation currently processing 1 (2.2%) of 44

27 Jan 10:24 EXITED currently processing 0

24 Jan 15:57 logged process: b063302f8525e13e1ebc9c519e66d35b87606061 Fix a timeout bug in the logging framework → Flattener#study_resource → complete_harvest_instance → overall_tsv_creation currently processing 33 (75.0%) of 44 (last five times: 615.5s, 570.2s, 754.3s, 489.0s, 692.4s) (average of all times: 115.17s)

24 Jan 14:25 EXITED currently processing 0 (0.0%) of 10229342

24 Jan 13:12 EXITED currently processing 0 (0.0%) of 10229342

24 Jan 11:02 EXITED currently processing 0 (0.0%) of 10229342

18 Jan 15:41 EXITED currently processing 0

18 Jan 13:48 EXITED currently processing 0 (0.0%) of 2760815

11 Jan 12:56 EXITED currently processing 0 (0.0%) of 2760815

22 Oct 22:04 EXITED currently processing 0

22 Oct 20:40 EXITED currently processing 0 (0.0%) of 2760815

13 Oct 11:50 EXITED currently processing 0

13 Oct 11:04 logged process: 777edefd65edc0fcb4c19dd17ae05338e30f3833 Merge pull request #20 from EOL/dependabot/bundler/nokogiri-1.12.5 → logged process: 51c900e63bdc119cae5033e5404208da37b7270e Fixing another rare bug for when you go back to re-parse names on a resource that has already run.

13 Oct 10:42 EXITED

06 Oct 10:35 EXITED currently processing 0

06 Oct 09:04 EXITED currently processing 0 (0.0%) of 2760815

06 Oct 06:48 EXITED currently processing 0 (0.0%) of 2760815

05 Oct 20:43 EXITED currently processing 0 (0.0%) of 2760815

05 Oct 18:04 EXITED currently processing 0 (0.0%) of 2760815

05 Oct 16:02 EXITED currently processing 0 (0.0%) of 2760815

05 Oct 14:32 EXITED currently processing 0 (0.0%) of 2760815

05 Oct 13:12 EXITED currently processing 0 (0.0%) of 2760815

14 Jun 14:27 EXITED currently processing 0

11 May 18:30 EXITED currently processing 0

11 May 12:32 EXITED currently processing 600000 (27.8%) of 2157307 (last five times: 2.8s, 52.2s, 2.8s, 2.8s, 2.8s) (average of all times: 0.91s)

20 Apr 19:03 EXITED currently processing 0

06 Apr 23:16 EXITED currently processing 440000 (19.6%) of 2240909 (last five times: 2.8s, 25.5s, 25.7s, 2.8s, 2.8s) (average of all times: 0.87s)

05 Apr 22:56 EXITED currently processing 440000 (19.6%) of 2240909 (last five times: 18.7s, 2.8s, 2.8s, 2.8s, 15.5s) (average of all times: 0.85s)

23 Mar 14:45 EXITED currently processing 0

23 Mar 09:35 EXITED currently processing 340000 (last five times: 260.9s, 272.7s, 280.5s, 292.9s, 324.9s) (average of all times: 11.92s)

23 Mar 09:09 EXITED currently processing 0

22 Mar 11:30 EXITED

22 Mar 10:41 logged process: cdc14cc7e396e8d099dbdcb2b6584e4ad6934982

01 Mar 18:31 EXITED

16 Feb 15:29 logged process: cdc14cc7e396e8d099dbdcb2b6584e4ad6934982

16 Feb 15:19 logged process: cdc14cc7e396e8d099dbdcb2b6584e4ad6934982 → logged process: cdc14cc7e396e8d099dbdcb2b6584e4ad6934982

12 Feb 14:34 logged process: 16b22834be7ac1492cba86047bb0f5dbfa370977 → logged process: 16b22834be7ac1492cba86047bb0f5dbfa370977

12 Feb 13:36 logged process: 16b22834be7ac1492cba86047bb0f5dbfa370977

29 Dec 13:08 logged process: ca5be136aef877c71c74100a42de34a9e7a07645 → logged process: ca5be136aef877c71c74100a42de34a9e7a07645

29 Dec 12:47 logged process: ca5be136aef877c71c74100a42de34a9e7a07645 → logged process: ca5be136aef877c71c74100a42de34a9e7a07645 → logged process: ca5be136aef877c71c74100a42de34a9e7a07645

29 Dec 12:06 logged process: ca5be136aef877c71c74100a42de34a9e7a07645

25 Nov 14:41 EXITED currently processing 0

25 Nov 12:33 EXITED currently processing 1 (0.0%) of 1909125

25 Nov 11:38 EXITED currently processing 1 (0.0%) of 1909125

25 Nov 10:10 EXITED

25 Nov 10:08 logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

25 Nov 09:45 EXITED

25 Nov 09:44 logged process: 58bbc42b01abb4c1b2698de049792ffb4b63b979

10 Jun 15:23 EXITED currently processing 0

10 Jun 12:57 EXITED currently processing 0

10 Jun 12:15 logged process

09 Jun 15:26 EXITED currently processing 0