Harvest for GGBN data coverage Created 04 Jun 20:23

Harvesting Log

[INFO] [2024-06-04 20:23:48] Created harvest instance #4549
[STOP] [2024-06-04 20:23:48] create_harvest_instance
[START] [2024-06-04 20:23:48] fetch_files
[STOP] [2024-06-04 20:23:48] fetch_files
[START] [2024-06-04 20:23:48] validate_each_file
[INFO] [2024-06-04 20:23:48] Looping over 3 formats...
[INFO] [2024-06-04 20:23:48] ...nodes (/app/public/data/ggbn_data_covera/taxon.tab)
[INFO] [2024-06-04 20:23:50] Valid: /app/public/data/ggbn_data_covera/converted_csv/ggbn_data_covera_nodes_31158.csv (124776 lines)
[INFO] [2024-06-04 20:23:50] ...occurrences (/app/public/data/ggbn_data_covera/occurrence_specific.tab)
[INFO] [2024-06-04 20:23:53] Valid: /app/public/data/ggbn_data_covera/converted_csv/ggbn_data_covera_occurrences_31159.csv (188410 lines)
[INFO] [2024-06-04 20:23:53] ...measurements (/app/public/data/ggbn_data_covera/measurement_or_fact_specific.tab)
[INFO] [2024-06-04 20:23:58] Valid: /app/public/data/ggbn_data_covera/converted_csv/ggbn_data_covera_measurements_31160.csv (188410 lines)
[STOP] [2024-06-04 20:23:58] validate_each_file
[START] [2024-06-04 20:23:58] convert_to_csv
[INFO] [2024-06-04 20:23:58] Looping over 3 formats...
[INFO] [2024-06-04 20:23:58] ...nodes (/app/public/data/ggbn_data_covera/taxon.tab)
[CMD] [2024-06-04 20:23:58] /usr/bin/sort /app/public/data/ggbn_data_covera/converted_csv/ggbn_data_covera_nodes_31158.csv > /app/public/data/ggbn_data_covera/converted_csv/ggbn_data_covera_nodes_31158.csv_sorted
[INFO] [2024-06-04 20:23:58] Converted: /app/public/data/ggbn_data_covera/converted_csv/ggbn_data_covera_nodes_31158.csv (124776 lines)
[INFO] [2024-06-04 20:23:58] ...occurrences (/app/public/data/ggbn_data_covera/occurrence_specific.tab)
[CMD] [2024-06-04 20:23:58] /usr/bin/sort /app/public/data/ggbn_data_covera/converted_csv/ggbn_data_covera_occurrences_31159.csv > /app/public/data/ggbn_data_covera/converted_csv/ggbn_data_covera_occurrences_31159.csv_sorted
[INFO] [2024-06-04 20:23:59] Converted: /app/public/data/ggbn_data_covera/converted_csv/ggbn_data_covera_occurrences_31159.csv (188410 lines)
[INFO] [2024-06-04 20:23:59] ...measurements (/app/public/data/ggbn_data_covera/measurement_or_fact_specific.tab)
[CMD] [2024-06-04 20:23:59] /usr/bin/sort /app/public/data/ggbn_data_covera/converted_csv/ggbn_data_covera_measurements_31160.csv > /app/public/data/ggbn_data_covera/converted_csv/ggbn_data_covera_measurements_31160.csv_sorted
[INFO] [2024-06-04 20:24:00] Converted: /app/public/data/ggbn_data_covera/converted_csv/ggbn_data_covera_measurements_31160.csv (188410 lines)
[STOP] [2024-06-04 20:24:00] convert_to_csv
[START] [2024-06-04 20:24:00] calculate_delta
[INFO] [2024-06-04 20:24:00] Looping over 3 formats...
[INFO] [2024-06-04 20:24:00] ...nodes (/app/public/data/ggbn_data_covera/taxon.tab)
[CMD] [2024-06-04 20:24:00] echo "0a" > /app/public/data/ggbn_data_covera/diff/ggbn_data_covera_nodes_31158.diff
[CMD] [2024-06-04 20:24:00] tail -n +1 /app/public/data/ggbn_data_covera/converted_csv/ggbn_data_covera_nodes_31158.csv >> /app/public/data/ggbn_data_covera/diff/ggbn_data_covera_nodes_31158.diff
[CMD] [2024-06-04 20:24:00] echo "." >> /app/public/data/ggbn_data_covera/diff/ggbn_data_covera_nodes_31158.diff
[INFO] [2024-06-04 20:24:00] Created diff: /app/public/data/ggbn_data_covera/diff/ggbn_data_covera_nodes_31158.diff (124778 lines)
[INFO] [2024-06-04 20:24:00] ...occurrences (/app/public/data/ggbn_data_covera/occurrence_specific.tab)
[CMD] [2024-06-04 20:24:00] echo "0a" > /app/public/data/ggbn_data_covera/diff/ggbn_data_covera_occurrences_31159.diff
[CMD] [2024-06-04 20:24:00] tail -n +1 /app/public/data/ggbn_data_covera/converted_csv/ggbn_data_covera_occurrences_31159.csv >> /app/public/data/ggbn_data_covera/diff/ggbn_data_covera_occurrences_31159.diff
[CMD] [2024-06-04 20:24:01] echo "." >> /app/public/data/ggbn_data_covera/diff/ggbn_data_covera_occurrences_31159.diff
[INFO] [2024-06-04 20:24:01] Created diff: /app/public/data/ggbn_data_covera/diff/ggbn_data_covera_occurrences_31159.diff (188412 lines)
[INFO] [2024-06-04 20:24:01] ...measurements (/app/public/data/ggbn_data_covera/measurement_or_fact_specific.tab)
[CMD] [2024-06-04 20:24:01] echo "0a" > /app/public/data/ggbn_data_covera/diff/ggbn_data_covera_measurements_31160.diff
[CMD] [2024-06-04 20:24:01] tail -n +1 /app/public/data/ggbn_data_covera/converted_csv/ggbn_data_covera_measurements_31160.csv >> /app/public/data/ggbn_data_covera/diff/ggbn_data_covera_measurements_31160.diff
[CMD] [2024-06-04 20:24:01] echo "." >> /app/public/data/ggbn_data_covera/diff/ggbn_data_covera_measurements_31160.diff
[INFO] [2024-06-04 20:24:02] Created diff: /app/public/data/ggbn_data_covera/diff/ggbn_data_covera_measurements_31160.diff (188412 lines)
[STOP] [2024-06-04 20:24:02] calculate_delta
[START] [2024-06-04 20:24:02] parse_diff_and_store
[INFO] [2024-06-04 20:24:02] Handling diff: /app/public/data/ggbn_data_covera/diff/ggbn_data_covera_nodes_31158.diff (124778 lines)
[INFO] [2024-06-04 20:24:02] Loading nodes diff file into memory (124778 lines)...
[INFO] [2024-06-04 20:24:04] Storing 9999 ScientificNames (19998/10000/124778)
[INFO] [2024-06-04 20:24:08] Storing 9999 Nodes (19998/10000/124778)
[INFO] [2024-06-04 20:24:14] Storing 10000 ScientificNames (39998/20000/124778)
[INFO] [2024-06-04 20:24:17] Storing 10000 Nodes (39998/20000/124778)
[INFO] [2024-06-04 20:24:23] Storing 10000 ScientificNames (59998/30000/124778)
[INFO] [2024-06-04 20:24:26] Storing 10000 Nodes (59998/30000/124778)
[INFO] [2024-06-04 20:24:32] Storing 10000 ScientificNames (79998/40000/124778)
[INFO] [2024-06-04 20:24:35] Storing 10000 Nodes (79998/40000/124778)
[INFO] [2024-06-04 20:24:41] Storing 10000 ScientificNames (99998/50000/124778)
[INFO] [2024-06-04 20:24:45] Storing 10000 Nodes (99998/50000/124778)
[INFO] [2024-06-04 20:24:51] Storing 10000 ScientificNames (119998/60000/124778)
[INFO] [2024-06-04 20:24:54] Storing 10000 Nodes (119998/60000/124778)
[INFO] [2024-06-04 20:25:00] Storing 10000 ScientificNames (139998/70000/124778)
[INFO] [2024-06-04 20:25:04] Storing 10000 Nodes (139998/70000/124778)
[INFO] [2024-06-04 20:25:10] Storing 10000 ScientificNames (159998/80000/124778)
[INFO] [2024-06-04 20:25:13] Storing 10000 Nodes (159998/80000/124778)
[INFO] [2024-06-04 20:25:19] Storing 10000 ScientificNames (179998/90000/124778)
[INFO] [2024-06-04 20:25:23] Storing 10000 Nodes (179998/90000/124778)
[INFO] [2024-06-04 20:25:29] Storing 10000 ScientificNames (199998/100000/124778)
[INFO] [2024-06-04 20:25:33] Storing 10000 Nodes (199998/100000/124778)
[INFO] [2024-06-04 20:25:39] Storing 10000 ScientificNames (219998/110000/124778)
[INFO] [2024-06-04 20:25:42] Storing 10000 Nodes (219998/110000/124778)
[INFO] [2024-06-04 20:25:49] Storing 10000 ScientificNames (239998/120000/124778)
[INFO] [2024-06-04 20:25:52] Storing 10000 Nodes (239998/120000/124778)
[INFO] [2024-06-04 20:25:57] Storing 4777 ScientificNames (249552/124776/124778)
[INFO] [2024-06-04 20:25:59] Storing 4777 Nodes (249552/124776/124778)
[INFO] [2024-06-04 20:26:01] Handling diff: /app/public/data/ggbn_data_covera/diff/ggbn_data_covera_occurrences_31159.diff (188412 lines)
[INFO] [2024-06-04 20:26:01] Loading occurrences diff file into memory (188412 lines)...
[INFO] [2024-06-04 20:26:02] Storing 9999 Occurrences (9999/10000/188412)
[INFO] [2024-06-04 20:26:04] Storing 10000 Occurrences (19999/20000/188412)
[INFO] [2024-06-04 20:26:07] Storing 10000 Occurrences (29999/30000/188412)
[INFO] [2024-06-04 20:26:09] Storing 10000 Occurrences (39999/40000/188412)
[INFO] [2024-06-04 20:26:12] Storing 10000 Occurrences (49999/50000/188412)
[INFO] [2024-06-04 20:26:14] Storing 10000 Occurrences (59999/60000/188412)
[INFO] [2024-06-04 20:26:17] Storing 10000 Occurrences (69999/70000/188412)
[INFO] [2024-06-04 20:26:20] Storing 10000 Occurrences (79999/80000/188412)
[INFO] [2024-06-04 20:26:22] Storing 10000 Occurrences (89999/90000/188412)
[INFO] [2024-06-04 20:26:25] Storing 10000 Occurrences (99999/100000/188412)
[INFO] [2024-06-04 20:26:28] Storing 10000 Occurrences (109999/110000/188412)
[INFO] [2024-06-04 20:26:31] Storing 10000 Occurrences (119999/120000/188412)
[INFO] [2024-06-04 20:26:33] Storing 10000 Occurrences (129999/130000/188412)
[INFO] [2024-06-04 20:26:36] Storing 10000 Occurrences (139999/140000/188412)
[INFO] [2024-06-04 20:26:39] Storing 10000 Occurrences (149999/150000/188412)
[INFO] [2024-06-04 20:26:42] Storing 10000 Occurrences (159999/160000/188412)
[INFO] [2024-06-04 20:26:45] Storing 10000 Occurrences (169999/170000/188412)
[INFO] [2024-06-04 20:26:48] Storing 10000 Occurrences (179999/180000/188412)
[INFO] [2024-06-04 20:26:51] Storing 8411 Occurrences (188410/188410/188412)
[INFO] [2024-06-04 20:26:52] Handling diff: /app/public/data/ggbn_data_covera/diff/ggbn_data_covera_measurements_31160.diff (188412 lines)
[INFO] [2024-06-04 20:26:53] Loading measurements diff file into memory (188412 lines)...
[INFO] [2024-06-04 20:26:55] Storing 9999 Traits (9999/10000/188412)
[INFO] [2024-06-04 20:27:01] Storing 10000 Traits (19999/20000/188412)
[INFO] [2024-06-04 20:27:06] Storing 10000 Traits (29999/30000/188412)
[INFO] [2024-06-04 20:27:12] Storing 10000 Traits (39999/40000/188412)
[INFO] [2024-06-04 20:27:17] Storing 10000 Traits (49999/50000/188412)
[INFO] [2024-06-04 20:27:23] Storing 10000 Traits (59999/60000/188412)
[INFO] [2024-06-04 20:27:29] Storing 10000 Traits (69999/70000/188412)
[INFO] [2024-06-04 20:27:35] Storing 10000 Traits (79999/80000/188412)
[INFO] [2024-06-04 20:27:40] Storing 10000 Traits (89999/90000/188412)
[INFO] [2024-06-04 20:27:46] Storing 10000 Traits (99999/100000/188412)
[INFO] [2024-06-04 20:27:52] Storing 10000 Traits (109999/110000/188412)
[INFO] [2024-06-04 20:27:58] Storing 10000 Traits (119999/120000/188412)
[INFO] [2024-06-04 20:28:04] Storing 10000 Traits (129999/130000/188412)
[INFO] [2024-06-04 20:28:11] Storing 10000 Traits (139999/140000/188412)
[INFO] [2024-06-04 20:28:17] Storing 10000 Traits (149999/150000/188412)
[INFO] [2024-06-04 20:28:23] Storing 10000 Traits (159999/160000/188412)
[INFO] [2024-06-04 20:28:29] Storing 10000 Traits (169999/170000/188412)
[INFO] [2024-06-04 20:28:36] Storing 10000 Traits (179999/180000/188412)
[INFO] [2024-06-04 20:28:42] Storing 8411 Traits (188410/188410/188412)
[STOP] [2024-06-04 20:28:45] parse_diff_and_store
[START] [2024-06-04 20:28:45] resolve_keys
[2024-06-04 20:29:15] Resolving downloaded urls (this is not actually downloading them yet)
[INFO] [2024-06-04 20:29:26] Occurrences to nodes (through scientific_names)...
[INFO] [2024-06-04 20:29:36] traits to occurrences...
[INFO] [2024-06-04 20:30:08] traits to nodes (through occurrences)...
[INFO] [2024-06-04 20:30:25] Traits to sex term...
[INFO] [2024-06-04 20:30:28] Traits to lifestage term...
[INFO] [2024-06-04 20:30:31] MetaTraits to traits...
[INFO] [2024-06-04 20:30:31] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2024-06-04 20:30:31] Assocs to occurrences...
[INFO] [2024-06-04 20:30:31] Assocs to nodes...
[INFO] [2024-06-04 20:30:31] Assoc to sex term...
[INFO] [2024-06-04 20:30:31] Assoc to lifestage term...
[INFO] [2024-06-04 20:30:31] MetaAssoc to assocs...
[STOP] [2024-06-04 20:30:31] resolve_keys
[START] [2024-06-04 20:30:31] hold_for_later_1
[STOP] [2024-06-04 20:30:31] hold_for_later_1
[START] [2024-06-04 20:30:31] hold_for_later_2
[STOP] [2024-06-04 20:30:31] hold_for_later_2
[START] [2024-06-04 20:30:31] resolve_missing_parents
[STOP] [2024-06-04 20:30:32] resolve_missing_parents
[START] [2024-06-04 20:30:32] rebuild_nodes
[START] [2024-06-04 20:30:32] Flattener#flatten
[START] [2024-06-04 20:30:32] Flattener#study_resource
[START] [2024-06-04 20:30:32] Flattener#build_ancestry
[STOP] [2024-06-04 20:31:06] Flattener#build_ancestry
[INFO] [2024-06-04 20:31:06] 124776 ancestry keys
[START] [2024-06-04 20:31:06] build_node_ancestors
[INFO] [2024-06-04 20:31:06] old ancestors deleted.
[STOP] [2024-06-04 20:31:06] build_node_ancestors
[WARN] [2024-06-04 20:31:06] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2024-06-04 20:31:06] Flattener#flatten
[STOP] [2024-06-04 20:31:06] rebuild_nodes
[START] [2024-06-04 20:31:06] resolve_missing_media_owners
[STOP] [2024-06-04 20:31:06] resolve_missing_media_owners
[START] [2024-06-04 20:31:06] sanitize_media_verbatims
[STOP] [2024-06-04 20:31:06] sanitize_media_verbatims
[START] [2024-06-04 20:31:06] queue_downloads
[STOP] [2024-06-04 20:31:06] queue_downloads
[START] [2024-06-04 20:31:06] parse_names
[WARN] [2024-06-04 20:31:06] I see 124776 names which still need to be parsed.
[WARN] [2024-06-04 20:31:12] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2024-06-04 20:31:18] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2024-06-04 20:31:26] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2024-06-04 20:31:33] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2024-06-04 20:31:40] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2024-06-04 20:31:47] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2024-06-04 20:31:54] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2024-06-04 20:32:01] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2024-06-04 20:32:09] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2024-06-04 20:32:16] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2024-06-04 20:32:23] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2024-06-04 20:32:30] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2024-06-04 20:32:37] Names to parse: 4776 formatted: 4776 learned: 4776 parsed: 4776
[STOP] [2024-06-04 20:32:41] parse_names
[START] [2024-06-04 20:32:41] denormalize_canonical_names_to_nodes
[STOP] [2024-06-04 20:32:45] denormalize_canonical_names_to_nodes
[START] [2024-06-04 20:32:45] match_nodes
[START] [2024-06-04 20:32:45] map_all_nodes_to_pages
[STOP] [2024-06-04 21:08:48] map_all_nodes_to_pages
[INFO] [2024-06-04 21:08:48] 11148 Unmatched nodes (of 124776)! That's too many to output. Full list in /app/public/data/ggbn_data_covera/unmatched_nodes.txt ; First 10: Canonical: ? bacterium; Node#155526237; ResourceID: ?_bacterium; Canonical: Abablemma brimleyana; Node#155526241; ResourceID: abablemma_brimleyana; Canonical: Abablemma duomaculata; Node#155526242; ResourceID: abablemma_duomaculata; Canonical: Abacopteris; Node#155526250; ResourceID: abacopteris; Canonical: Abacopteris afra; Node#155526251; ResourceID: abacopteris_afra; Canonical: Abagrotis anchocelioides; Node#155526256; ResourceID: abagrotis_anchocelioides; Canonical: Abagrotis belfragei; Node#155526259; ResourceID: abagrotis_belfragei; Canonical: Abagrotis brunneipennis; Node#155526260; ResourceID: abagrotis_brunneipennis; Canonical: Abagrotis cupida; Node#155526261; ResourceID: abagrotis_cupida; Canonical: Abagrotis hermina; Node#155526264; ResourceID: abagrotis_hermina
[START] [2024-06-04 21:08:48] update_nodes
[STOP] [2024-06-04 21:09:03] update_nodes
[STOP] [2024-06-04 21:09:03] match_nodes
[START] [2024-06-04 21:09:03] reindex_search
[STOP] [2024-06-04 21:10:15] reindex_search
[START] [2024-06-04 21:10:15] normalize_units
[STOP] [2024-06-04 21:11:26] normalize_units
[START] [2024-06-04 21:11:26] calculate_statistics
[2024-06-04 21:11:28] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[INFO] [2024-06-04 21:11:52] Duplicate page_id count: 0
[STOP] [2024-06-04 21:11:52] calculate_statistics
[START] [2024-06-04 21:11:52] complete_harvest_instance
[START] [2024-06-04 21:11:52] overall_tsv_creation
[INFO] [2024-06-04 21:11:52] Exporting 124776 nodes as TSV in batches of 10000...
[INFO] [2024-06-04 21:11:52] Processing group of 124776 in 13 batches of 10000
[INFO] [2024-06-04 21:13:45] 15351 Traits (unfiltered) and 0 associations...
[INFO] [2024-06-04 21:13:45] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-06-04 21:13:52] Mapped 15351 traits (0 meta) for 10000 nodes.
[INFO] [2024-06-04 21:13:52] Building Associations map (this can take a while)...
[INFO] [2024-06-04 21:13:58] Done. 0 assocs mapped (0 meta).
[INFO] [2024-06-04 21:13:58] Adding 15351 traits...
[INFO] [2024-06-04 21:13:59] 0 metadata added.
[INFO] [2024-06-04 21:13:59] Adding 0 assocs...
[INFO] [2024-06-04 21:13:59] 0 metadata added.
[INFO] [2024-06-04 21:15:10] Processed 10000/124776 nodes
[INFO] [2024-06-04 21:17:02] 14956 Traits (unfiltered) and 0 associations...
[INFO] [2024-06-04 21:17:02] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-06-04 21:17:10] Mapped 14956 traits (0 meta) for 10000 nodes.
[INFO] [2024-06-04 21:17:10] Building Associations map (this can take a while)...
[INFO] [2024-06-04 21:17:14] Done. 0 assocs mapped (0 meta).
[INFO] [2024-06-04 21:17:14] Adding 14956 traits...
[INFO] [2024-06-04 21:17:15] 0 metadata added.
[INFO] [2024-06-04 21:17:15] Adding 0 assocs...
[INFO] [2024-06-04 21:17:15] 0 metadata added.
[INFO] [2024-06-04 21:18:26] Processed 20000/124776 nodes
[INFO] [2024-06-04 21:20:18] 15210 Traits (unfiltered) and 0 associations...
[INFO] [2024-06-04 21:20:18] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-06-04 21:20:27] Mapped 15210 traits (0 meta) for 10000 nodes.
[INFO] [2024-06-04 21:20:27] Building Associations map (this can take a while)...
[INFO] [2024-06-04 21:20:31] Done. 0 assocs mapped (0 meta).
[INFO] [2024-06-04 21:20:31] Adding 15210 traits...
[INFO] [2024-06-04 21:20:32] 0 metadata added.
[INFO] [2024-06-04 21:20:32] Adding 0 assocs...
[INFO] [2024-06-04 21:20:32] 0 metadata added.
[INFO] [2024-06-04 21:21:42] Processed 30000/124776 nodes
[INFO] [2024-06-04 21:23:35] 15172 Traits (unfiltered) and 0 associations...
[INFO] [2024-06-04 21:23:35] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-06-04 21:23:44] Mapped 15172 traits (0 meta) for 10000 nodes.
[INFO] [2024-06-04 21:23:44] Building Associations map (this can take a while)...
[INFO] [2024-06-04 21:23:49] Done. 0 assocs mapped (0 meta).
[INFO] [2024-06-04 21:23:49] Adding 15172 traits...
[INFO] [2024-06-04 21:23:49] 0 metadata added.
[INFO] [2024-06-04 21:23:49] Adding 0 assocs...
[INFO] [2024-06-04 21:23:49] 0 metadata added.
[INFO] [2024-06-04 21:25:00] Processed 40000/124776 nodes
[INFO] [2024-06-04 21:26:54] 15319 Traits (unfiltered) and 0 associations...
[INFO] [2024-06-04 21:26:54] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-06-04 21:27:02] Mapped 15319 traits (0 meta) for 10000 nodes.
[INFO] [2024-06-04 21:27:02] Building Associations map (this can take a while)...
[INFO] [2024-06-04 21:27:06] Done. 0 assocs mapped (0 meta).
[INFO] [2024-06-04 21:27:06] Adding 15319 traits...
[INFO] [2024-06-04 21:27:07] 0 metadata added.
[INFO] [2024-06-04 21:27:07] Adding 0 assocs...
[INFO] [2024-06-04 21:27:07] 0 metadata added.
[INFO] [2024-06-04 21:28:17] Processed 50000/124776 nodes
[INFO] [2024-06-04 21:30:10] 15132 Traits (unfiltered) and 0 associations...
[INFO] [2024-06-04 21:30:10] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-06-04 21:30:18] Mapped 15132 traits (0 meta) for 10000 nodes.
[INFO] [2024-06-04 21:30:18] Building Associations map (this can take a while)...
[INFO] [2024-06-04 21:30:22] Done. 0 assocs mapped (0 meta).
[INFO] [2024-06-04 21:30:22] Adding 15132 traits...
[INFO] [2024-06-04 21:30:23] 0 metadata added.
[INFO] [2024-06-04 21:30:23] Adding 0 assocs...
[INFO] [2024-06-04 21:30:23] 0 metadata added.
[INFO] [2024-06-04 21:31:33] Processed 60000/124776 nodes
[INFO] [2024-06-04 21:33:25] 15097 Traits (unfiltered) and 0 associations...
[INFO] [2024-06-04 21:33:25] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-06-04 21:33:34] Mapped 15097 traits (0 meta) for 10000 nodes.
[INFO] [2024-06-04 21:33:34] Building Associations map (this can take a while)...
[INFO] [2024-06-04 21:33:38] Done. 0 assocs mapped (0 meta).
[INFO] [2024-06-04 21:33:38] Adding 15097 traits...
[INFO] [2024-06-04 21:33:39] 0 metadata added.
[INFO] [2024-06-04 21:33:39] Adding 0 assocs...
[INFO] [2024-06-04 21:33:39] 0 metadata added.
[INFO] [2024-06-04 21:34:49] Processed 70000/124776 nodes
[INFO] [2024-06-04 21:36:42] 14839 Traits (unfiltered) and 0 associations...
[INFO] [2024-06-04 21:36:42] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-06-04 21:36:51] Mapped 14839 traits (0 meta) for 10000 nodes.
[INFO] [2024-06-04 21:36:51] Building Associations map (this can take a while)...
[INFO] [2024-06-04 21:36:55] Done. 0 assocs mapped (0 meta).
[INFO] [2024-06-04 21:36:55] Adding 14839 traits...
[INFO] [2024-06-04 21:36:56] 0 metadata added.
[INFO] [2024-06-04 21:36:56] Adding 0 assocs...
[INFO] [2024-06-04 21:36:56] 0 metadata added.
[INFO] [2024-06-04 21:38:05] Processed 80000/124776 nodes
[INFO] [2024-06-04 21:39:56] 15524 Traits (unfiltered) and 0 associations...
[INFO] [2024-06-04 21:39:56] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-06-04 21:40:05] Mapped 15524 traits (0 meta) for 10000 nodes.
[INFO] [2024-06-04 21:40:05] Building Associations map (this can take a while)...
[INFO] [2024-06-04 21:40:09] Done. 0 assocs mapped (0 meta).
[INFO] [2024-06-04 21:40:09] Adding 15524 traits...
[INFO] [2024-06-04 21:40:10] 0 metadata added.
[INFO] [2024-06-04 21:40:10] Adding 0 assocs...
[INFO] [2024-06-04 21:40:10] 0 metadata added.
[INFO] [2024-06-04 21:41:20] Processed 90000/124776 nodes
[INFO] [2024-06-04 21:43:12] 14973 Traits (unfiltered) and 0 associations...
[INFO] [2024-06-04 21:43:13] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-06-04 21:43:20] Mapped 14973 traits (0 meta) for 10000 nodes.
[INFO] [2024-06-04 21:43:20] Building Associations map (this can take a while)...
[INFO] [2024-06-04 21:43:24] Done. 0 assocs mapped (0 meta).
[INFO] [2024-06-04 21:43:24] Adding 14973 traits...
[INFO] [2024-06-04 21:43:25] 0 metadata added.
[INFO] [2024-06-04 21:43:25] Adding 0 assocs...
[INFO] [2024-06-04 21:43:25] 0 metadata added.
[INFO] [2024-06-04 21:44:35] Processed 100000/124776 nodes
[INFO] [2024-06-04 21:46:28] 14686 Traits (unfiltered) and 0 associations...
[INFO] [2024-06-04 21:46:28] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-06-04 21:46:37] Mapped 14686 traits (0 meta) for 10000 nodes.
[INFO] [2024-06-04 21:46:37] Building Associations map (this can take a while)...
[INFO] [2024-06-04 21:46:41] Done. 0 assocs mapped (0 meta).
[INFO] [2024-06-04 21:46:41] Adding 14686 traits...
[INFO] [2024-06-04 21:46:42] 0 metadata added.
[INFO] [2024-06-04 21:46:42] Adding 0 assocs...
[INFO] [2024-06-04 21:46:42] 0 metadata added.
[INFO] [2024-06-04 21:47:51] Processed 110000/124776 nodes
[INFO] [2024-06-04 21:49:43] 15110 Traits (unfiltered) and 0 associations...
[INFO] [2024-06-04 21:49:43] Building Traits map for 10000 nodes (this can take a while)...
[INFO] [2024-06-04 21:49:51] Mapped 15110 traits (0 meta) for 10000 nodes.
[INFO] [2024-06-04 21:49:51] Building Associations map (this can take a while)...
[INFO] [2024-06-04 21:49:55] Done. 0 assocs mapped (0 meta).
[INFO] [2024-06-04 21:49:55] Adding 15110 traits...
[INFO] [2024-06-04 21:49:56] 0 metadata added.
[INFO] [2024-06-04 21:49:56] Adding 0 assocs...
[INFO] [2024-06-04 21:49:56] 0 metadata added.
[INFO] [2024-06-04 21:51:07] Processed 120000/124776 nodes
[INFO] [2024-06-04 21:51:16] 7041 Traits (unfiltered) and 0 associations...
[INFO] [2024-06-04 21:51:16] Building Traits map for 4776 nodes (this can take a while)...
[INFO] [2024-06-04 21:51:19] Mapped 7041 traits (0 meta) for 4776 nodes.
[INFO] [2024-06-04 21:51:19] Building Associations map (this can take a while)...
[INFO] [2024-06-04 21:51:22] Done. 0 assocs mapped (0 meta).
[INFO] [2024-06-04 21:51:22] Adding 7041 traits...
[INFO] [2024-06-04 21:51:23] 0 metadata added.
[INFO] [2024-06-04 21:51:23] Adding 0 assocs...
[INFO] [2024-06-04 21:51:23] 0 metadata added.
[INFO] [2024-06-04 21:52:27] Processed 124776/124776 nodes
[INFO] [2024-06-04 21:52:27] Average Time: 95.895
[INFO] [2024-06-04 21:52:27] Total Time: 40m35s
[INFO] [2024-06-04 21:52:27] last 3 / first 3: 0.94
[INFO] [2024-06-04 21:52:27] Std.Dev: 5.317; Max: 99.0
[STOP] [2024-06-04 21:52:27] overall_tsv_creation
[INFO] [2024-06-04 21:52:27] Done. Check your files:
[INFO] [2024-06-04 21:52:27] (124776 lines) /app/public/data/ggbn_data_covera/publish_nodes.tsv
[INFO] [2024-06-04 21:52:27] (124776 lines) /app/public/data/ggbn_data_covera/publish_scientific_names.tsv
[INFO] [2024-06-04 21:52:27] (188411 lines) /app/public/data/ggbn_data_covera/publish_traits.tsv
[INFO] [2024-06-04 21:52:27] (1 lines) /app/public/data/ggbn_data_covera/publish_metadata.tsv
[STOP] [2024-06-04 21:52:28] complete_harvest_instance
[START] [2024-06-04 21:52:28] completed
[STOP] [2024-06-04 21:52:28] completed
[STOP] [2024-06-04 21:52:28] logged process, took 5320.26

Latest Process

04 Jun 20:19 EXITED currently processing 0