Stage:
completed
Fetched:
26 May 14:22
Validated:
26 May 14:25
Deltas Created
26 May 14:25
Units Normalized:
26 May 19:44
Ancestry Built:
26 May 17:11
Nodes Matched:
26 May 17:49
Names Parsed:
26 May 17:14
New Models Stored:
26 May 16:44
Indexed:
26 May 17:51
Completed:
27 May 00:13
Time to Harvest:
10 minutes
Harvesting Log
(372 lines)
[INFO] [2021-05-26 14:22:30] Created harvest instance #3889
[STOP] [2021-05-26 14:22:30] create_harvest_instance
[START] [2021-05-26 14:22:30] fetch_files
[STOP] [2021-05-26 14:22:30] fetch_files
[START] [2021-05-26 14:22:30] validate_each_file
[INFO] [2021-05-26 14:22:30] Looping over 3 formats...
[INFO] [2021-05-26 14:22:30] ...nodes (/app/public/data/obis_environment/taxon.tab)
[INFO] [2021-05-26 14:22:33] Valid: /app/public/converted_csv/obis_environment_nodes_3889.csv (162187 lines)
[INFO] [2021-05-26 14:22:33] ...occurrences (/app/public/data/obis_environment/occurrence_specific.tab)
[INFO] [2021-05-26 14:22:38] Valid: /app/public/converted_csv/obis_environment_occurrences_3889.csv (486561 lines)
[INFO] [2021-05-26 14:22:38] ...measurements (/app/public/data/obis_environment/measurement_or_fact_specific.tab)
[INFO] [2021-05-26 14:25:30] Valid: /app/public/converted_csv/obis_environment_measurements_3889.csv (3826738 lines)
[STOP] [2021-05-26 14:25:30] validate_each_file
[START] [2021-05-26 14:25:30] convert_to_csv
[INFO] [2021-05-26 14:25:30] Looping over 3 formats...
[INFO] [2021-05-26 14:25:30] ...nodes (/app/public/data/obis_environment/taxon.tab)
[CMD] [2021-05-26 14:25:30] /usr/bin/sort /app/public/converted_csv/obis_environment_nodes_3889.csv > /app/public/converted_csv/obis_environment_nodes_3889.csv_sorted
[INFO] [2021-05-26 14:25:30] Converted: /app/public/converted_csv/obis_environment_nodes_3889.csv (162187 lines)
[INFO] [2021-05-26 14:25:30] ...occurrences (/app/public/data/obis_environment/occurrence_specific.tab)
[CMD] [2021-05-26 14:25:30] /usr/bin/sort /app/public/converted_csv/obis_environment_occurrences_3889.csv > /app/public/converted_csv/obis_environment_occurrences_3889.csv_sorted
[INFO] [2021-05-26 14:25:30] Converted: /app/public/converted_csv/obis_environment_occurrences_3889.csv (486561 lines)
[INFO] [2021-05-26 14:25:30] ...measurements (/app/public/data/obis_environment/measurement_or_fact_specific.tab)
[CMD] [2021-05-26 14:25:30] /usr/bin/sort /app/public/converted_csv/obis_environment_measurements_3889.csv > /app/public/converted_csv/obis_environment_measurements_3889.csv_sorted
[INFO] [2021-05-26 14:25:35] Converted: /app/public/converted_csv/obis_environment_measurements_3889.csv (3826738 lines)
[STOP] [2021-05-26 14:25:35] convert_to_csv
[START] [2021-05-26 14:25:35] calculate_delta
[INFO] [2021-05-26 14:25:35] Looping over 3 formats...
[INFO] [2021-05-26 14:25:35] ...nodes (/app/public/data/obis_environment/taxon.tab)
[CMD] [2021-05-26 14:25:35] echo "0a" > /app/public/diff/obis_environment_nodes_3889.diff
[CMD] [2021-05-26 14:25:35] tail -n +1 /app/public/converted_csv/obis_environment_nodes_3889.csv >> /app/public/diff/obis_environment_nodes_3889.diff
[CMD] [2021-05-26 14:25:35] echo "." >> /app/public/diff/obis_environment_nodes_3889.diff
[INFO] [2021-05-26 14:25:35] Created diff: /app/public/diff/obis_environment_nodes_3889.diff (162189 lines)
[INFO] [2021-05-26 14:25:35] ...occurrences (/app/public/data/obis_environment/occurrence_specific.tab)
[CMD] [2021-05-26 14:25:35] echo "0a" > /app/public/diff/obis_environment_occurrences_3889.diff
[CMD] [2021-05-26 14:25:35] tail -n +1 /app/public/converted_csv/obis_environment_occurrences_3889.csv >> /app/public/diff/obis_environment_occurrences_3889.diff
[CMD] [2021-05-26 14:25:36] echo "." >> /app/public/diff/obis_environment_occurrences_3889.diff
[INFO] [2021-05-26 14:25:36] Created diff: /app/public/diff/obis_environment_occurrences_3889.diff (486563 lines)
[INFO] [2021-05-26 14:25:36] ...measurements (/app/public/data/obis_environment/measurement_or_fact_specific.tab)
[CMD] [2021-05-26 14:25:36] echo "0a" > /app/public/diff/obis_environment_measurements_3889.diff
[CMD] [2021-05-26 14:25:36] tail -n +1 /app/public/converted_csv/obis_environment_measurements_3889.csv >> /app/public/diff/obis_environment_measurements_3889.diff
[CMD] [2021-05-26 14:25:37] echo "." >> /app/public/diff/obis_environment_measurements_3889.diff
[INFO] [2021-05-26 14:25:40] Created diff: /app/public/diff/obis_environment_measurements_3889.diff (3826740 lines)
[STOP] [2021-05-26 14:25:40] calculate_delta
[START] [2021-05-26 14:25:40] parse_diff_and_store
[INFO] [2021-05-26 14:25:40] Handling diff: /app/public/diff/obis_environment_nodes_3889.diff (162189 lines)
[INFO] [2021-05-26 14:25:40] Loading nodes diff file into memory (162189 /app/public/diff/obis_environment_nodes_3889.diff lines)...
[WARN] [2021-05-26 14:25:42] Filtered Scientific Name `Campylosira Grunow <I>ex</I> Van Heurck, 1885` to `Campylosira Grunow <I>ex<I> Van Heurck, 1885`
[WARN] [2021-05-26 14:25:42] Filtered Scientific Name `Campylosira cymbelliformis (A. Schmidt) Grunow <I>ex</I> Van Heurck, 1885` to `Campylosira cymbelliformis (A. Schmidt) Grunow <I>ex<I> Van Heurck, 1885`
[WARN] [2021-05-26 14:25:42] Filtered Scientific Name `Chaetoceros subtilis P.T. Cleve, 1896` to `Chaetoceros subtilis P.T. Cleve, 1896`
[WARN] [2021-05-26 14:25:43] Filtered Scientific Name `Dictyosporium pelagicum (Linder) G.C. Hughes <i>ex</i> E.B.G. Jones` to `Dictyosporium pelagicum (Linder) G.C. Hughes <i>ex<i> E.B.G. Jones`
[WARN] [2021-05-26 14:25:46] Filtered Scientific Name `Lyngbya aestuarii (Mertens in Jürgens) Liebman <i>ex</i> Gomont, 1892` to `Lyngbya aestuarii (Mertens in Jürgens) Liebman <i>ex<i> Gomont, 1892`
[WARN] [2021-05-26 14:25:47] Filtered Scientific Name `Nemertopsis flavida (McIntosh, 1873/74)` to `Nemertopsis flavida (McIntosh, 187374)`
[WARN] [2021-05-26 14:25:48] Filtered Scientific Name `Oscillatoria bonnemaisonii Crouan <i>in</i> Desmazières <i>ex</i> Gomont, 1892` to `Oscillatoria bonnemaisonii Crouan <i>in<i> Desmazières <i>ex<i> Gomont, 1892`
[WARN] [2021-05-26 14:25:48] Filtered Scientific Name `Oscillatoria curviceps C.A. Agardh <i>ex</i> Gomont, 1892` to `Oscillatoria curviceps C.A. Agardh <i>ex<i> Gomont, 1892`
[WARN] [2021-05-26 14:25:48] Filtered Scientific Name `Oscillatoria margaritifera Kützing <i>ex</i> Gomont, 1892` to `Oscillatoria margaritifera Kützing <i>ex<i> Gomont, 1892`
[WARN] [2021-05-26 14:25:48] Filtered Scientific Name `Oscillatoria princeps Vaucher <i>ex</i> Gomont, 1892` to `Oscillatoria princeps Vaucher <i>ex<i> Gomont, 1892`
[WARN] [2021-05-26 14:25:48] Filtered Scientific Name `Oscillatoria sancta Kützing <i>ex</i> Gomont, 1892` to `Oscillatoria sancta Kützing <i>ex<i> Gomont, 1892`
[WARN] [2021-05-26 14:25:49] Filtered Scientific Name `Phacus pyrum (Ehrenberg) F. Stein, 1878` to `Phacus pyrum (Ehrenberg) F. Stein, 1878`
[WARN] [2021-05-26 14:25:51] Filtered Scientific Name `Spirulina Turpin <i>ex</i> Gomont, 1893` to `Spirulina Turpin <i>ex<i> Gomont, 1893`
[WARN] [2021-05-26 14:25:51] Filtered Scientific Name `Spirulina meneghiniana Zanardini <i>ex</i> Gomont, 1892` to `Spirulina meneghiniana Zanardini <i>ex<i> Gomont, 1892`
[WARN] [2021-05-26 14:26:00] Filtered Scientific Name `Nosopsyllus fasciatus Bosc d"" Antic` to `Nosopsyllus fasciatus Bosc d Antic`
[WARN] [2021-05-26 14:26:04] Filtered Scientific Name `Arthrospira Stizenberger <i>ex</i> Gomont, 1892` to `Arthrospira Stizenberger <i>ex<i> Gomont, 1892`
[WARN] [2021-05-26 14:26:05] Filtered Scientific Name `Trichodesmium Ehrenberg <i>ex</i> Gomont, 1892` to `Trichodesmium Ehrenberg <i>ex<i> Gomont, 1892`
[WARN] [2021-05-26 14:26:05] Filtered Scientific Name `Ampithoe (Pleonexes) (Bate, 1857)` to `Ampithoe (Pleonexes) (Bate, 1857)`
[WARN] [2021-05-26 14:26:07] Filtered Scientific Name `Enteroctopus magnificus (Villanueva/Sanchez/Compagno, 1992)` to `Enteroctopus magnificus (VillanuevaSanchezCompagno, 1992)`
[WARN] [2021-05-26 14:26:11] Filtered Scientific Name `Phormidium breve (Kützing <i>ex</i> Gomont) Anagnostidis & Komárek, 1988` to `Phormidium breve (Kützing <i>ex<i> Gomont) Anagnostidis & Komárek, 1988`
[WARN] [2021-05-26 14:26:11] Filtered Scientific Name `Phormidium formosum (Bory de Saint-Vincent <i>ex</i> Gomont) Anagnostidis & Komárek, 1988` to `Phormidium formosum (Bory de Saint-Vincent <i>ex<i> Gomont) Anagnostidis & Komárek, 1988`
[WARN] [2021-05-26 14:26:11] Filtered Scientific Name `Freneixicardia victor (Angas, 1872)` to `Freneixicardia victor (Angas, 1872)`
[WARN] [2021-05-26 14:26:11] Filtered Scientific Name `Anabaena circinalis Rabenhorst <i>ex</i> Bornet & Flahault, 1886` to `Anabaena circinalis Rabenhorst <i>ex<i> Bornet & Flahault, 1886`
[WARN] [2021-05-26 14:26:11] Filtered Scientific Name `Phormidium terebriforme (C.A. Agardh <i>ex</i> Gomont) Anagnostidis & Komárek, 1988` to `Phormidium terebriforme (C.A. Agardh <i>ex<i> Gomont) Anagnostidis & Komárek, 1988`
[WARN] [2021-05-26 14:26:11] Filtered Scientific Name `Fulvia (Fulvia) laevigata (Linnaeus, 1758)` to `Fulvia (Fulvia) laevigata (Linnaeus, 1758)`
[WARN] [2021-05-26 14:26:11] Filtered Scientific Name `Fragum scruposum (Deshayes, 1855)` to `Fragum scruposum (Deshayes, 1855)`
[WARN] [2021-05-26 14:26:11] Filtered Scientific Name `Dinematomonas P.C. Silva, 1960` to `Dinematomonas P.C. Silva, 1960`
[WARN] [2021-05-26 14:26:11] Filtered Scientific Name `Dyspanopeus Martin & Abele, 1986` to `Dyspanopeus Martin & Abele, 1986`
[WARN] [2021-05-26 14:26:12] Filtered Scientific Name `Minyichthys Herald & Randall, 1972` to `Minyichthys Herald & Randall, 1972`
[WARN] [2021-05-26 14:26:12] (Reached filtered-name limit; supressing further warnings.)
[INFO] [2021-05-26 14:26:29] Handling diff: /app/public/diff/obis_environment_occurrences_3889.diff (486563 lines)
[INFO] [2021-05-26 14:26:29] Loading occurrences diff file into memory (486563 /app/public/diff/obis_environment_occurrences_3889.diff lines)...
[INFO] [2021-05-26 14:28:01] Handling diff: /app/public/diff/obis_environment_measurements_3889.diff (3826740 lines)
[INFO] [2021-05-26 14:28:01] Loading measurements diff file into memory (3826740 /app/public/diff/obis_environment_measurements_3889.diff lines)...
[INFO] [2021-05-26 15:41:33] Storing 162187 ScientificNames
[INFO] [2021-05-26 15:41:33] Processing group of 162187 in 163 groups of 1000
[INFO] [2021-05-26 15:43:09] Average Time: 0.582
[INFO] [2021-05-26 15:43:09] Total Time: 1m36s
[INFO] [2021-05-26 15:43:09] last 3 / first 3: 0.93
[INFO] [2021-05-26 15:43:09] Std.Dev: 1.5106290080625355; Max: 11.27
[INFO] [2021-05-26 15:43:09] Storing 162187 Nodes
[INFO] [2021-05-26 15:43:09] Processing group of 162187 in 163 groups of 1000
[INFO] [2021-05-26 15:46:48] Average Time: 1.341
[INFO] [2021-05-26 15:46:48] Total Time: 3m40s
[INFO] [2021-05-26 15:46:48] last 3 / first 3: 0.73
[INFO] [2021-05-26 15:46:48] Std.Dev: 6.709396396100025; Max: 44.18
[INFO] [2021-05-26 15:46:48] Storing 486561 Occurrences
[INFO] [2021-05-26 15:46:48] Processing group of 486561 in 487 groups of 1000
[INFO] [2021-05-26 15:50:04] Average Time: 0.397
[INFO] [2021-05-26 15:50:04] Total Time: 3m17s
[INFO] [2021-05-26 15:50:04] last 3 / first 3: 0.47
[INFO] [2021-05-26 15:50:04] Std.Dev: 3.4214032209022074; Max: 44.34
[INFO] [2021-05-26 15:50:04] Storing 3826738 Traits
[INFO] [2021-05-26 15:50:04] Processing group of 3826738 in 3827 groups of 1000
[INFO] [2021-05-26 16:35:31] Average Time: 0.703
[INFO] [2021-05-26 16:35:31] Total Time: 45m27s
[INFO] [2021-05-26 16:35:31] last 3 / first 3: 1.02
[INFO] [2021-05-26 16:35:31] Std.Dev: 2.3993749185985918; Max: 45.37
[INFO] [2021-05-26 16:35:31] Storing 2036151 MetaTraits
[INFO] [2021-05-26 16:35:31] Processing group of 2036151 in 2037 groups of 1000
[INFO] [2021-05-26 16:44:13] Average Time: 0.25
[INFO] [2021-05-26 16:44:13] Total Time: 8m43s
[INFO] [2021-05-26 16:44:13] last 3 / first 3: 2.21
[INFO] [2021-05-26 16:44:13] Std.Dev: 0.9126883367283708; Max: 10.78
[STOP] [2021-05-26 16:44:13] parse_diff_and_store
[START] [2021-05-26 16:44:13] resolve_keys
[INFO] [2021-05-26 16:44:43] Occurrences to nodes (through scientific_names)...
[INFO] [2021-05-26 16:44:55] traits to occurrences...
[INFO] [2021-05-26 16:46:12] traits to nodes (through occurrences)...
[INFO] [2021-05-26 16:46:56] Traits to sex term...
[INFO] [2021-05-26 16:47:24] Traits to lifestage term...
[INFO] [2021-05-26 16:47:52] MetaTraits to traits...
[INFO] [2021-05-26 16:48:43] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-05-26 16:49:53] Assocs to occurrences...
[INFO] [2021-05-26 16:49:53] Assocs to nodes...
[INFO] [2021-05-26 16:49:53] Assoc to sex term...
[INFO] [2021-05-26 16:49:53] Assoc to lifestage term...
[INFO] [2021-05-26 16:49:53] MetaAssoc to assocs...
[STOP] [2021-05-26 16:49:53] resolve_keys
[START] [2021-05-26 16:49:53] hold_for_later_1
[STOP] [2021-05-26 16:49:53] hold_for_later_1
[START] [2021-05-26 16:49:53] hold_for_later_2
[STOP] [2021-05-26 16:49:53] hold_for_later_2
[START] [2021-05-26 16:49:53] resolve_missing_parents
[STOP] [2021-05-26 16:49:53] resolve_missing_parents
[START] [2021-05-26 16:49:53] rebuild_nodes
[START] [2021-05-26 16:49:53] Flattener#flatten
[START] [2021-05-26 16:49:53] Flattener#study_resource
[START] [2021-05-26 16:50:20] Flattener#build_ancestry
[STOP] [2021-05-26 17:11:05] Flattener#build_ancestry
[INFO] [2021-05-26 17:11:05] 162187 ancestry keys
[START] [2021-05-26 17:11:05] build_node_ancestors
[INFO] [2021-05-26 17:11:05] old ancestors deleted.
[STOP] [2021-05-26 17:11:05] build_node_ancestors
[WARN] [2021-05-26 17:11:05] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2021-05-26 17:11:05] Flattener#flatten
[STOP] [2021-05-26 17:11:05] rebuild_nodes
[START] [2021-05-26 17:11:05] resolve_missing_media_owners
[STOP] [2021-05-26 17:11:05] resolve_missing_media_owners
[START] [2021-05-26 17:11:05] sanitize_media_verbatims
[STOP] [2021-05-26 17:11:05] sanitize_media_verbatims
[START] [2021-05-26 17:11:05] queue_downloads
[STOP] [2021-05-26 17:11:05] queue_downloads
[START] [2021-05-26 17:11:05] parse_names
[WARN] [2021-05-26 17:11:05] I see 162187 names which still need to be parsed.
[STOP] [2021-05-26 17:14:12] parse_names
[START] [2021-05-26 17:14:12] denormalize_canonical_names_to_nodes
[STOP] [2021-05-26 17:14:18] denormalize_canonical_names_to_nodes
[START] [2021-05-26 17:14:18] match_nodes
[START] [2021-05-26 17:14:18] map_all_nodes_to_pages
[STOP] [2021-05-26 17:49:06] map_all_nodes_to_pages
[INFO] [2021-05-26 17:49:06] 49883 Unmatched nodes (of 162187)! That's too many to output. Full list in /app/public/data/obis_environment/unmatched_nodes.txt ; First 10: Canonical: Conus labrolineata; Node#94543804; ResourceID: 388; Canonical: Aaptolasma fosteri; Node#94543805; ResourceID: 395358; Canonical: Aaptolasma nolearia; Node#94543806; ResourceID: 395359; Canonical: Abarenicola; Node#94543812; ResourceID: 395372; Canonical: Abarenicola claparedii oceanica; Node#94543814; ResourceID: 395376; Canonical: Abasia; Node#94543816; ResourceID: 395378; Canonical: Abathescalpellum fissum; Node#94543819; ResourceID: 395382; Canonical: Abatus; Node#94543821; ResourceID: 395384; Canonical: Aberranta; Node#94543831; ResourceID: 395404; Canonical: Abietinaria; Node#94543833; ResourceID: 395408
[START] [2021-05-26 17:49:06] update_nodes
[STOP] [2021-05-26 17:49:36] update_nodes
[STOP] [2021-05-26 17:49:36] match_nodes
[START] [2021-05-26 17:49:36] reindex_search
[STOP] [2021-05-26 17:51:11] reindex_search
[START] [2021-05-26 17:51:11] normalize_units
[STOP] [2021-05-26 19:44:37] normalize_units
[START] [2021-05-26 19:44:37] calculate_statistics
[2021-05-26 19:44:38] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[2021-05-26 19:45:18] WARNING: 1902184 trait(s) without source found! Please confirm that this is intentional.
[2021-05-26 19:45:18] traits w/o source (up to 100):
[2021-05-26 19:45:18] (resource_pk: 00000b08f7858169c9739aee47d8f8f1_692, id: 207834004)
[2021-05-26 19:45:18] (resource_pk: 00001c8d41cb3d60c9ecd9189d7c81df_692, id: 207834008)
[2021-05-26 19:45:18] (resource_pk: 00002454bd3b90db2e987a8e7e03bd1a_692, id: 207834013)
[2021-05-26 19:45:18] (resource_pk: 00002cd39b01148c97eb60547bf30e04_692, id: 207834015)
[2021-05-26 19:45:18] (resource_pk: 00002cf33ccface371f744ed5eeb0883_692, id: 207834016)
[2021-05-26 19:45:18] (resource_pk: 00002f6a51cff855229c8710a77770ed_692, id: 207834018)
[2021-05-26 19:45:18] (resource_pk: 00002ff66452d0e19c48c10e9f500750_692, id: 207834020)
[2021-05-26 19:45:18] (resource_pk: 00003ca0909242683c736b5d947fdcdf_692, id: 207834021)
[2021-05-26 19:45:18] (resource_pk: 0000477fb559743b5726b031ac83e630_692, id: 207834023)
[2021-05-26 19:45:18] (resource_pk: 0000511c22523ec91eedd3f4f4bdda48_692, id: 207834025)
[2021-05-26 19:45:18] (resource_pk: 00005a02934e94a1db7972505afc06fb_692, id: 207834027)
[2021-05-26 19:45:18] (resource_pk: 0000622d9b0f9f7878fa09c9b328ed2b_692, id: 207834032)
[2021-05-26 19:45:18] (resource_pk: 000063ae6c7c60a2c79a52a0f9478df1_692, id: 207834033)
[2021-05-26 19:45:18] (resource_pk: 000068b4236b97c710722fe2e3ed700b_692, id: 207834035)
[2021-05-26 19:45:18] (resource_pk: 00006cb5b7876d5048d6e2f9b6ff187f_692, id: 207834036)
[2021-05-26 19:45:18] (resource_pk: 00009a16b2b98b077a4bf9c895809195_692, id: 207834039)
[2021-05-26 19:45:18] (resource_pk: 0000b399435e082629eb2712d7e23b7e_692, id: 207834041)
[2021-05-26 19:45:18] (resource_pk: 0000bb9e6a1b309d8f1c58bd3b6b1a49_692, id: 207834044)
[2021-05-26 19:45:18] (resource_pk: 0000bc62aa50eee7af120cab1575ad88_692, id: 207834045)
[2021-05-26 19:45:18] (resource_pk: 0000c30ae7dc452dd788dd7066c6f1c3_692, id: 207834046)
[2021-05-26 19:45:18] (resource_pk: 0000ca8b2e3fee082f7412dfaef10592_692, id: 207834050)
[2021-05-26 19:45:18] (resource_pk: 0000cc06f6128c722b7fcd7130113806_692, id: 207834051)
[2021-05-26 19:45:18] (resource_pk: 0000d221c377472566a6c3278ce7193f_692, id: 207834052)
[2021-05-26 19:45:18] (resource_pk: 0000d38700de25d19df5f943307fa2f0_692, id: 207834054)
[2021-05-26 19:45:18] (resource_pk: 0000dc543059ef2c0ac43f403e286b6e_692, id: 207834059)
[2021-05-26 19:45:18] (resource_pk: 0000def70263ae22028140527a26abc0_692, id: 207834061)
[2021-05-26 19:45:18] (resource_pk: 0000e1a489360550fcabd234b06cc87d_692, id: 207834063)
[2021-05-26 19:45:18] (resource_pk: 0000e48f8bc997176298266487b67b59_692, id: 207834064)
[2021-05-26 19:45:18] (resource_pk: 0000f23d841c3ae3a66e6c4b970c8206_692, id: 207834068)
[2021-05-26 19:45:18] (resource_pk: 0000f8faf450c41015640ee094094686_692, id: 207834070)
[2021-05-26 19:45:18] (resource_pk: 0000f9bc9854a0f136adf8f03f2f0d3f_692, id: 207834071)
[2021-05-26 19:45:18] (resource_pk: 0000fa35e075fc8cea183ba36e076e4c_692, id: 207834072)
[2021-05-26 19:45:18] (resource_pk: 0001068802608e163237038386a9239f_692, id: 207834075)
[2021-05-26 19:45:18] (resource_pk: 000111785bae97f06d63aed9ade6e5af_692, id: 207834076)
[2021-05-26 19:45:18] (resource_pk: 000114cbb8ef63c9b21b62337e793e90_692, id: 207834077)
[2021-05-26 19:45:18] (resource_pk: 000127bcd08f6b7294c9e621f211ae24_692, id: 207834078)
[2021-05-26 19:45:18] (resource_pk: 00013773a550cfa283cfd4e0f62a9191_692, id: 207834080)
[2021-05-26 19:45:18] (resource_pk: 00013a561a0c201abb9f588a526d679b_692, id: 207834081)
[2021-05-26 19:45:18] (resource_pk: 00013d7a4cd880bcc6e3e3d62e8f71a3_692, id: 207834082)
[2021-05-26 19:45:18] (resource_pk: 00015ee3ee6873fb06e21a8d0b863cbb_692, id: 207834086)
[2021-05-26 19:45:18] (resource_pk: 0001605d7fadeb22ecf54e7140d9dbeb_692, id: 207834087)
[2021-05-26 19:45:18] (resource_pk: 000160f13f5d79088ac76e4aaf8d1738_692, id: 207834088)
[2021-05-26 19:45:18] (resource_pk: 00016f24ce25d637758c39714d744204_692, id: 207834089)
[2021-05-26 19:45:18] (resource_pk: 00017558bd75a37aae0ef0dd56717a9a_692, id: 207834090)
[2021-05-26 19:45:18] (resource_pk: 0001794e57ea1eced042b94538a95505_692, id: 207834091)
[2021-05-26 19:45:18] (resource_pk: 00017b237702b8fc9df0b4b429ecb4b3_692, id: 207834093)
[2021-05-26 19:45:18] (resource_pk: 0001997463aafa76a1f630c0bdc991ba_692, id: 207834095)
[2021-05-26 19:45:18] (resource_pk: 0001a319a47e69a2ca662a4939f8326e_692, id: 207834097)
[2021-05-26 19:45:18] (resource_pk: 0001a6bfb5befe06b84299df67fc3e67_692, id: 207834098)
[2021-05-26 19:45:18] (resource_pk: 0001afc4d1f852d23933aa60a6c96d72_692, id: 207834100)
[2021-05-26 19:45:18] (resource_pk: 0001aff63af1335c6b506c02c66259f9_692, id: 207834101)
[2021-05-26 19:45:18] (resource_pk: 0001cbaf7d40960184258724e5122a03_692, id: 207834106)
[2021-05-26 19:45:18] (resource_pk: 0001d4108de252b506bca655e06ee192_692, id: 207834107)
[2021-05-26 19:45:18] (resource_pk: 0001d68819539a58c0e10ead1094e0ed_692, id: 207834108)
[2021-05-26 19:45:18] (resource_pk: 0001de4e20dc017a9d434541c74bf220_692, id: 207834111)
[2021-05-26 19:45:18] (resource_pk: 0001e367f7b1b69f8e259a5ce9e90deb_692, id: 207834112)
[2021-05-26 19:45:18] (resource_pk: 0001ec5aca3c8c1ee60c29af7d899b6d_692, id: 207834113)
[2021-05-26 19:45:18] (resource_pk: 0001f37dc45754ce6a5677e4e994100c_692, id: 207834114)
[2021-05-26 19:45:18] (resource_pk: 0001f41d08ca4a850d68d73f11ffb7be_692, id: 207834115)
[2021-05-26 19:45:18] (resource_pk: 0002104319835e685a43fdf9c196ccf7_692, id: 207834118)
[2021-05-26 19:45:18] (resource_pk: 000219a558308bd79c4101a06f122650_692, id: 207834121)
[2021-05-26 19:45:18] (resource_pk: 00021a1d58bd3ce99fb9e28db187f056_692, id: 207834122)
[2021-05-26 19:45:18] (resource_pk: 00021b023a1c8535f6000dad66a39016_692, id: 207834123)
[2021-05-26 19:45:18] (resource_pk: 00026306ecabbe2cd3532b0a22014b8f_692, id: 207834133)
[2021-05-26 19:45:18] (resource_pk: 00026ceda3d6bbe3b6db358d4d35b376_692, id: 207834136)
[2021-05-26 19:45:18] (resource_pk: 00026e1ae75c5e3122e8d26eb3a74d4f_692, id: 207834137)
[2021-05-26 19:45:18] (resource_pk: 00026f2f382f8febe12da9d3004ceb0a_692, id: 207834138)
[2021-05-26 19:45:18] (resource_pk: 00029b119032b1bf3db4066f166a30a0_692, id: 207834145)
[2021-05-26 19:45:18] (resource_pk: 00029bf016ba7d839c0bb5f13b830b5e_692, id: 207834146)
[2021-05-26 19:45:18] (resource_pk: 0002a490d61326932880aaca07f688fd_692, id: 207834150)
[2021-05-26 19:45:18] (resource_pk: 0002ad901aabdeb9f8cc2b2ba646d28a_692, id: 207834152)
[2021-05-26 19:45:18] (resource_pk: 0002bec2e619c3f43280912062fb9151_692, id: 207834154)
[2021-05-26 19:45:18] (resource_pk: 0002bedb67892796da1f7f9dc4b0a440_692, id: 207834155)
[2021-05-26 19:45:18] (resource_pk: 0002cac0bc446400769525c4b3ade722_692, id: 207834156)
[2021-05-26 19:45:18] (resource_pk: 0002cdff7165ac13a1743c2b651c686c_692, id: 207834157)
[2021-05-26 19:45:18] (resource_pk: 0002dde94bfee4037453e8ade6cf505a_692, id: 207834160)
[2021-05-26 19:45:18] (resource_pk: 0002ea081fc9c732bacf52c6494952b8_692, id: 207834162)
[2021-05-26 19:45:18] (resource_pk: 0002f6e0877c480fc0e8f1ba3ca79f15_692, id: 207834163)
[2021-05-26 19:45:18] (resource_pk: 00030cb6b73d0f11cbcb90f38621bd5e_692, id: 207834165)
[2021-05-26 19:45:18] (resource_pk: 00030cc807f56f2b1a217f78f67a2bab_692, id: 207834166)
[2021-05-26 19:45:18] (resource_pk: 00030ea0f2bb93f7c2a1502ce294be2d_692, id: 207834167)
[2021-05-26 19:45:18] (resource_pk: 00030f659ee292e69dc886dee27826b8_692, id: 207834168)
[2021-05-26 19:45:18] (resource_pk: 00031a718dca864e5cefd1d9f47660a9_692, id: 207834169)
[2021-05-26 19:45:18] (resource_pk: 00031ab7d67f03f0c5b330bc0287fc4e_692, id: 207834170)
[2021-05-26 19:45:18] (resource_pk: 00031c933279200bacb56e5120e7c400_692, id: 207834171)
[2021-05-26 19:45:18] (resource_pk: 00032e3d31ad0ff1250e77f27508f36d_692, id: 207834173)
[2021-05-26 19:45:18] (resource_pk: 000335d47d2ed12b40adcd6834ffed16_692, id: 207834175)
[2021-05-26 19:45:18] (resource_pk: 00033beefd2cc38f35608745be23b9f5_692, id: 207834176)
[2021-05-26 19:45:18] (resource_pk: 00033e39cfa53365ed0a35b76ff1ac8c_692, id: 207834177)
[2021-05-26 19:45:18] (resource_pk: 00033e66f3a8a3989ac2a748e0ca0fff_692, id: 207834178)
[2021-05-26 19:45:18] (resource_pk: 000343048f9ed3dc5340d26172862c6b_692, id: 207834179)
[2021-05-26 19:45:18] (resource_pk: 000344e12df25a9102ea1748415bd86d_692, id: 207834181)
[2021-05-26 19:45:18] (resource_pk: 0003493c4969ac04f589a0488938a4f5_692, id: 207834182)
[2021-05-26 19:45:18] (resource_pk: 00034db661b5020a9967e43b85a7ec3b_692, id: 207834183)
[2021-05-26 19:45:18] (resource_pk: 0003578929b0cecb79383e53122f9f6b_692, id: 207834185)
[2021-05-26 19:45:18] (resource_pk: 0003620e6c7abee44584f6019522605d_692, id: 207834187)
[2021-05-26 19:45:18] (resource_pk: 0003684be7fadb7ccb650857eff1e745_692, id: 207834189)
[2021-05-26 19:45:18] (resource_pk: 000369e1e9d2c067af9989b322e8fa6d_692, id: 207834190)
[2021-05-26 19:45:18] (resource_pk: 000370cc0705c8e051c5707862077d66_692, id: 207834192)
[2021-05-26 19:45:18] (resource_pk: 000382bd8b13821327e5c441edbf5a78_692, id: 207834196)
[STOP] [2021-05-26 19:45:18] calculate_statistics
[START] [2021-05-26 19:45:18] complete_harvest_instance
[START] [2021-05-26 19:45:18] overall_tsv_creation
[INFO] [2021-05-26 19:45:19] Processing group of 162187 in 17 batches of 10000
[INFO] [2021-05-26 19:47:32] 135772 Traits (unfiltered)...
[INFO] [2021-05-26 20:02:50] 135772 Traits (filtered)...
[INFO] [2021-05-26 20:02:59] 0 Associations (filtered)...
[INFO] [2021-05-26 20:03:29] 122776 metadata added.
[INFO] [2021-05-26 20:03:29] 0 metadata added.
[INFO] [2021-05-26 20:07:43] 135344 Traits (unfiltered)...
[INFO] [2021-05-26 20:23:05] 135344 Traits (filtered)...
[INFO] [2021-05-26 20:23:10] 0 Associations (filtered)...
[INFO] [2021-05-26 20:23:45] 122311 metadata added.
[INFO] [2021-05-26 20:23:45] 0 metadata added.
[INFO] [2021-05-26 20:26:23] 135373 Traits (unfiltered)...
[INFO] [2021-05-26 20:41:45] 135373 Traits (filtered)...
[INFO] [2021-05-26 20:41:48] 0 Associations (filtered)...
[INFO] [2021-05-26 20:42:19] 121265 metadata added.
[INFO] [2021-05-26 20:42:19] 0 metadata added.
[INFO] [2021-05-26 20:44:47] 132802 Traits (unfiltered)...
[INFO] [2021-05-26 20:59:50] 132802 Traits (filtered)...
[INFO] [2021-05-26 20:59:53] 0 Associations (filtered)...
[INFO] [2021-05-26 21:00:23] 120629 metadata added.
[INFO] [2021-05-26 21:00:23] 0 metadata added.
[INFO] [2021-05-26 21:02:52] 126645 Traits (unfiltered)...
[INFO] [2021-05-26 21:17:18] 126645 Traits (filtered)...
[INFO] [2021-05-26 21:17:20] 0 Associations (filtered)...
[INFO] [2021-05-26 21:17:47] 119646 metadata added.
[INFO] [2021-05-26 21:17:47] 0 metadata added.
[INFO] [2021-05-26 21:20:13] 110213 Traits (unfiltered)...
[INFO] [2021-05-26 21:32:52] 110213 Traits (filtered)...
[INFO] [2021-05-26 21:32:55] 0 Associations (filtered)...
[INFO] [2021-05-26 21:33:13] 89298 metadata added.
[INFO] [2021-05-26 21:33:13] 0 metadata added.
[INFO] [2021-05-26 21:35:33] 108319 Traits (unfiltered)...
[INFO] [2021-05-26 21:47:58] 108319 Traits (filtered)...
[INFO] [2021-05-26 21:48:00] 0 Associations (filtered)...
[INFO] [2021-05-26 21:48:22] 91144 metadata added.
[INFO] [2021-05-26 21:48:22] 0 metadata added.
[INFO] [2021-05-26 21:50:37] 108124 Traits (unfiltered)...
[INFO] [2021-05-26 22:03:01] 108124 Traits (filtered)...
[INFO] [2021-05-26 22:03:04] 0 Associations (filtered)...
[INFO] [2021-05-26 22:03:28] 91877 metadata added.
[INFO] [2021-05-26 22:03:28] 0 metadata added.
[INFO] [2021-05-26 22:05:42] 116855 Traits (unfiltered)...
[INFO] [2021-05-26 22:19:04] 116855 Traits (filtered)...
[INFO] [2021-05-26 22:19:06] 0 Associations (filtered)...
[INFO] [2021-05-26 22:19:34] 104222 metadata added.
[INFO] [2021-05-26 22:19:34] 0 metadata added.
[INFO] [2021-05-26 22:21:51] 97542 Traits (unfiltered)...
[INFO] [2021-05-26 22:33:02] 97542 Traits (filtered)...
[INFO] [2021-05-26 22:33:05] 0 Associations (filtered)...
[INFO] [2021-05-26 22:33:29] 81547 metadata added.
[INFO] [2021-05-26 22:33:29] 0 metadata added.
[INFO] [2021-05-26 22:35:38] 101024 Traits (unfiltered)...
[INFO] [2021-05-26 22:47:17] 101024 Traits (filtered)...
[INFO] [2021-05-26 22:47:19] 0 Associations (filtered)...
[INFO] [2021-05-26 22:47:43] 86567 metadata added.
[INFO] [2021-05-26 22:47:43] 0 metadata added.
[INFO] [2021-05-26 22:50:01] 134362 Traits (unfiltered)...
[INFO] [2021-05-26 23:05:29] 134362 Traits (filtered)...
[INFO] [2021-05-26 23:05:31] 0 Associations (filtered)...
[INFO] [2021-05-26 23:06:05] 123815 metadata added.
[INFO] [2021-05-26 23:06:05] 0 metadata added.
[INFO] [2021-05-26 23:08:35] 98217 Traits (unfiltered)...
[INFO] [2021-05-26 23:19:59] 98217 Traits (filtered)...
[INFO] [2021-05-26 23:20:02] 0 Associations (filtered)...
[INFO] [2021-05-26 23:20:25] 71217 metadata added.
[INFO] [2021-05-26 23:20:25] 0 metadata added.
[INFO] [2021-05-26 23:22:32] 108278 Traits (unfiltered)...
[INFO] [2021-05-26 23:34:58] 108278 Traits (filtered)...
[INFO] [2021-05-26 23:35:00] 0 Associations (filtered)...
[INFO] [2021-05-26 23:35:21] 79130 metadata added.
[INFO] [2021-05-26 23:35:21] 0 metadata added.
[INFO] [2021-05-26 23:37:33] 141784 Traits (unfiltered)...
[INFO] [2021-05-26 23:53:45] 141784 Traits (filtered)...
[INFO] [2021-05-26 23:53:48] 0 Associations (filtered)...
[INFO] [2021-05-26 23:54:23] 136192 metadata added.
[INFO] [2021-05-26 23:54:23] 0 metadata added.
[INFO] [2021-05-26 23:56:51] 93974 Traits (unfiltered)...
[INFO] [2021-05-27 00:07:43] 93974 Traits (filtered)...
[INFO] [2021-05-27 00:07:46] 0 Associations (filtered)...
[INFO] [2021-05-27 00:08:03] 80175 metadata added.
[INFO] [2021-05-27 00:08:03] 0 metadata added.
[INFO] [2021-05-27 00:09:48] 17556 Traits (unfiltered)...
[INFO] [2021-05-27 00:12:16] 17556 Traits (filtered)...
[INFO] [2021-05-27 00:12:16] 0 Associations (filtered)...
[INFO] [2021-05-27 00:12:21] 15230 metadata added.
[INFO] [2021-05-27 00:12:21] 0 metadata added.
[INFO] [2021-05-27 00:12:54] Average Time: 902.497
[INFO] [2021-05-27 00:12:54] Total Time: 4h27m36s
[INFO] [2021-05-27 00:12:54] last 3 / first 3: 0.66
[INFO] [2021-05-27 00:12:54] Std.Dev: 212.5079269109743; Max: 1121.17
[STOP] [2021-05-27 00:12:54] overall_tsv_creation
[INFO] [2021-05-27 00:12:54] Done. Check your files:
[INFO] [2021-05-27 00:12:55] (162187 lines) /app/public/data/obis_environment/publish_nodes.tsv
[INFO] [2021-05-27 00:12:57] (162187 lines) /app/public/data/obis_environment/publish_scientific_names.tsv
[INFO] [2021-05-27 00:12:58] (1902185 lines) /app/public/data/obis_environment/publish_traits.tsv
[INFO] [2021-05-27 00:13:00] (1657042 lines) /app/public/data/obis_environment/publish_metadata.tsv
[STOP] [2021-05-27 00:13:00] complete_harvest_instance
[START] [2021-05-27 00:13:00] completed
[STOP] [2021-05-27 00:13:00] completed
[STOP] [2021-05-27 00:13:00] logged process, took 35429.65
Latest Process