Harvest for GBIF national node type records Sweden Created 27 May 20:23

Harvesting Log

[INFO] [2021-05-27 20:23:47] Created harvest instance #3897
[STOP] [2021-05-27 20:23:47] create_harvest_instance
[START] [2021-05-27 20:23:47] fetch_files
[STOP] [2021-05-27 20:23:47] fetch_files
[START] [2021-05-27 20:23:47] validate_each_file
[INFO] [2021-05-27 20:23:47] Looping over 3 formats...
[INFO] [2021-05-27 20:23:47] ...nodes (/app/public/data/gnntrs/taxon.tab)
[INFO] [2021-05-27 20:23:50] Valid: /app/public/converted_csv/gnntrs_nodes_3897.csv (63065 lines)
[INFO] [2021-05-27 20:23:50] ...occurrences (/app/public/data/gnntrs/occurrence.tab)
[INFO] [2021-05-27 20:23:56] Valid: /app/public/converted_csv/gnntrs_occurrences_3897.csv (129133 lines)
[INFO] [2021-05-27 20:23:56] ...measurements (/app/public/data/gnntrs/measurement_or_fact.tab)
[INFO] [2021-05-27 20:24:24] Valid: /app/public/converted_csv/gnntrs_measurements_3897.csv (760740 lines)
[STOP] [2021-05-27 20:24:24] validate_each_file
[START] [2021-05-27 20:24:24] convert_to_csv
[INFO] [2021-05-27 20:24:24] Looping over 3 formats...
[INFO] [2021-05-27 20:24:24] ...nodes (/app/public/data/gnntrs/taxon.tab)
[CMD] [2021-05-27 20:24:24] /usr/bin/sort /app/public/converted_csv/gnntrs_nodes_3897.csv > /app/public/converted_csv/gnntrs_nodes_3897.csv_sorted
[INFO] [2021-05-27 20:24:26] Converted: /app/public/converted_csv/gnntrs_nodes_3897.csv (63065 lines)
[INFO] [2021-05-27 20:24:26] ...occurrences (/app/public/data/gnntrs/occurrence.tab)
[CMD] [2021-05-27 20:24:26] /usr/bin/sort /app/public/converted_csv/gnntrs_occurrences_3897.csv > /app/public/converted_csv/gnntrs_occurrences_3897.csv_sorted
[INFO] [2021-05-27 20:24:28] Converted: /app/public/converted_csv/gnntrs_occurrences_3897.csv (129133 lines)
[INFO] [2021-05-27 20:24:28] ...measurements (/app/public/data/gnntrs/measurement_or_fact.tab)
[CMD] [2021-05-27 20:24:28] /usr/bin/sort /app/public/converted_csv/gnntrs_measurements_3897.csv > /app/public/converted_csv/gnntrs_measurements_3897.csv_sorted
[INFO] [2021-05-27 20:24:31] Converted: /app/public/converted_csv/gnntrs_measurements_3897.csv (760740 lines)
[STOP] [2021-05-27 20:24:31] convert_to_csv
[START] [2021-05-27 20:24:31] calculate_delta
[INFO] [2021-05-27 20:24:31] Looping over 3 formats...
[INFO] [2021-05-27 20:24:31] ...nodes (/app/public/data/gnntrs/taxon.tab)
[CMD] [2021-05-27 20:24:31] echo "0a" > /app/public/diff/gnntrs_nodes_3897.diff
[CMD] [2021-05-27 20:24:32] tail -n +1 /app/public/converted_csv/gnntrs_nodes_3897.csv >> /app/public/diff/gnntrs_nodes_3897.diff
[CMD] [2021-05-27 20:24:33] echo "." >> /app/public/diff/gnntrs_nodes_3897.diff
[INFO] [2021-05-27 20:24:35] Created diff: /app/public/diff/gnntrs_nodes_3897.diff (63067 lines)
[INFO] [2021-05-27 20:24:35] ...occurrences (/app/public/data/gnntrs/occurrence.tab)
[CMD] [2021-05-27 20:24:35] echo "0a" > /app/public/diff/gnntrs_occurrences_3897.diff
[CMD] [2021-05-27 20:24:36] tail -n +1 /app/public/converted_csv/gnntrs_occurrences_3897.csv >> /app/public/diff/gnntrs_occurrences_3897.diff
[CMD] [2021-05-27 20:24:37] echo "." >> /app/public/diff/gnntrs_occurrences_3897.diff
[INFO] [2021-05-27 20:24:39] Created diff: /app/public/diff/gnntrs_occurrences_3897.diff (129135 lines)
[INFO] [2021-05-27 20:24:39] ...measurements (/app/public/data/gnntrs/measurement_or_fact.tab)
[CMD] [2021-05-27 20:24:39] echo "0a" > /app/public/diff/gnntrs_measurements_3897.diff
[CMD] [2021-05-27 20:24:40] tail -n +1 /app/public/converted_csv/gnntrs_measurements_3897.csv >> /app/public/diff/gnntrs_measurements_3897.diff
[CMD] [2021-05-27 20:24:42] echo "." >> /app/public/diff/gnntrs_measurements_3897.diff
[INFO] [2021-05-27 20:24:44] Created diff: /app/public/diff/gnntrs_measurements_3897.diff (760742 lines)
[STOP] [2021-05-27 20:24:44] calculate_delta
[START] [2021-05-27 20:24:44] parse_diff_and_store
[INFO] [2021-05-27 20:24:44] Handling diff: /app/public/diff/gnntrs_nodes_3897.diff (63067 lines)
[INFO] [2021-05-27 20:24:45] Loading nodes diff file into memory (63067 /app/public/diff/gnntrs_nodes_3897.diff lines)...
[WARN] [2021-05-27 20:25:10] Filtered Scientific Name `Tortrix croceana Hübner, 1796/99` to `Tortrix croceana Hübner, 179699`
[INFO] [2021-05-27 20:25:12] Handling diff: /app/public/diff/gnntrs_occurrences_3897.diff (129135 lines)
[INFO] [2021-05-27 20:25:13] Loading occurrences diff file into memory (129135 /app/public/diff/gnntrs_occurrences_3897.diff lines)...
[INFO] [2021-05-27 20:26:32] Handling diff: /app/public/diff/gnntrs_measurements_3897.diff (760742 lines)
[INFO] [2021-05-27 20:26:34] Loading measurements diff file into memory (760742 /app/public/diff/gnntrs_measurements_3897.diff lines)...
[INFO] [2021-05-27 20:32:34] Storing 83589 ScientificNames
[INFO] [2021-05-27 20:32:34] Processing group of 83589 in 84 groups of 1000
[INFO] [2021-05-27 20:33:17] Average Time: 0.512
[INFO] [2021-05-27 20:33:17] Total Time: 44s
[INFO] [2021-05-27 20:33:17] last 3 / first 3: 0.67
[INFO] [2021-05-27 20:33:17] Std.Dev: 0.589915248150105; Max: 2.51
[INFO] [2021-05-27 20:33:17] Storing 83589 Nodes
[INFO] [2021-05-27 20:33:17] Processing group of 83589 in 84 groups of 1000
[INFO] [2021-05-27 20:33:49] Average Time: 0.377
[INFO] [2021-05-27 20:33:49] Total Time: 33s
[INFO] [2021-05-27 20:33:49] last 3 / first 3: 0.7
[INFO] [2021-05-27 20:33:49] Std.Dev: 0.4868264577855234; Max: 2.87
[INFO] [2021-05-27 20:33:49] Storing 129133 Occurrences
[INFO] [2021-05-27 20:33:49] Processing group of 129133 in 130 groups of 1000
[INFO] [2021-05-27 20:34:07] Average Time: 0.132
[INFO] [2021-05-27 20:34:07] Total Time: 18s
[INFO] [2021-05-27 20:34:07] last 3 / first 3: 0.5
[INFO] [2021-05-27 20:34:07] Std.Dev: 0.06324555320336758; Max: 0.41
[INFO] [2021-05-27 20:34:07] Storing 694484 OccurrenceMetadata
[INFO] [2021-05-27 20:34:07] Processing group of 694484 in 695 groups of 1000
[INFO] [2021-05-27 20:35:56] Average Time: 0.153
[INFO] [2021-05-27 20:35:56] Total Time: 1m50s
[INFO] [2021-05-27 20:35:56] last 3 / first 3: 0.83
[INFO] [2021-05-27 20:35:56] Std.Dev: 0.19235384061671346; Max: 1.69
[INFO] [2021-05-27 20:35:56] Storing 502474 Traits
[INFO] [2021-05-27 20:35:56] Processing group of 502474 in 503 groups of 1000
[INFO] [2021-05-27 20:39:20] Average Time: 0.4
[INFO] [2021-05-27 20:39:20] Total Time: 3m25s
[INFO] [2021-05-27 20:39:20] last 3 / first 3: 0.71
[INFO] [2021-05-27 20:39:20] Std.Dev: 0.4289522117905443; Max: 3.79
[INFO] [2021-05-27 20:39:20] Storing 75898 MetaTraits
[INFO] [2021-05-27 20:39:20] Processing group of 75898 in 76 groups of 1000
[INFO] [2021-05-27 20:39:32] Average Time: 0.143
[INFO] [2021-05-27 20:39:32] Total Time: 12s
[INFO] [2021-05-27 20:39:32] last 3 / first 3: 0.91
[INFO] [2021-05-27 20:39:32] Std.Dev: 0.06324555320336758; Max: 0.43
[STOP] [2021-05-27 20:39:32] parse_diff_and_store
[START] [2021-05-27 20:39:32] resolve_keys
[INFO] [2021-05-27 20:40:28] Occurrences to nodes (through scientific_names)...
[INFO] [2021-05-27 20:40:46] traits to occurrences...
[INFO] [2021-05-27 20:40:51] traits to nodes (through occurrences)...
[INFO] [2021-05-27 20:40:55] Traits to sex term...
[INFO] [2021-05-27 20:40:57] Traits to lifestage term...
[INFO] [2021-05-27 20:41:00] MetaTraits to traits...
[INFO] [2021-05-27 20:41:02] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-05-27 20:41:14] Assocs to occurrences...
[INFO] [2021-05-27 20:41:14] Assocs to nodes...
[INFO] [2021-05-27 20:41:14] Assoc to sex term...
[INFO] [2021-05-27 20:41:14] Assoc to lifestage term...
[INFO] [2021-05-27 20:41:14] MetaAssoc to assocs...
[STOP] [2021-05-27 20:41:14] resolve_keys
[START] [2021-05-27 20:41:14] hold_for_later_1
[STOP] [2021-05-27 20:41:14] hold_for_later_1
[START] [2021-05-27 20:41:14] hold_for_later_2
[STOP] [2021-05-27 20:41:14] hold_for_later_2
[START] [2021-05-27 20:41:14] resolve_missing_parents
[STOP] [2021-05-27 20:41:17] resolve_missing_parents
[START] [2021-05-27 20:41:17] rebuild_nodes
[START] [2021-05-27 20:41:17] Flattener#flatten
[START] [2021-05-27 20:41:17] Flattener#study_resource
[START] [2021-05-27 20:41:18] Flattener#build_ancestry
[STOP] [2021-05-27 20:42:33] Flattener#build_ancestry
[INFO] [2021-05-27 20:42:33] 83589 ancestry keys
[START] [2021-05-27 20:42:33] build_node_ancestors
[INFO] [2021-05-27 20:42:33] old ancestors deleted.
[STOP] [2021-05-27 20:43:32] build_node_ancestors
[START] [2021-05-27 20:43:36] Flattener#propagate_ancestor_ids
[STOP] [2021-05-27 20:43:47] Flattener#propagate_ancestor_ids
[STOP] [2021-05-27 20:43:47] Flattener#flatten
[STOP] [2021-05-27 20:43:47] rebuild_nodes
[START] [2021-05-27 20:43:47] resolve_missing_media_owners
[STOP] [2021-05-27 20:43:47] resolve_missing_media_owners
[START] [2021-05-27 20:43:47] sanitize_media_verbatims
[STOP] [2021-05-27 20:43:47] sanitize_media_verbatims
[START] [2021-05-27 20:43:47] queue_downloads
[STOP] [2021-05-27 20:43:47] queue_downloads
[START] [2021-05-27 20:43:47] parse_names
[WARN] [2021-05-27 20:43:48] I see 83589 names which still need to be parsed.
[WARN] [2021-05-27 20:44:51] I see 130 names which still need to be parsed.
[WARN] [2021-05-27 20:44:52] I see 9 names which still need to be parsed.
[STOP] [2021-05-27 20:44:53] parse_names
[START] [2021-05-27 20:44:53] denormalize_canonical_names_to_nodes
[STOP] [2021-05-27 20:44:55] denormalize_canonical_names_to_nodes
[START] [2021-05-27 20:44:55] match_nodes
[START] [2021-05-27 20:44:55] map_all_nodes_to_pages
[STOP] [2021-05-27 21:12:09] map_all_nodes_to_pages
[INFO] [2021-05-27 21:12:09] 6236 Unmatched nodes (of 83589)! That's too many to output. Full list in /app/public/data/gnntrs/unmatched_nodes.txt ; First 10: Canonical: Arethusa; Node#94900978; ResourceID: Arethusa; Canonical: Coelogyne; Node#94913661; ResourceID: Coelogyne; Canonical: Cymbidium; Node#94917260; ResourceID: Cymbidium; Canonical: Cypripedium; Node#94917523; ResourceID: Cypripedium; Canonical: Epidendrum; Node#94922639; ResourceID: Epidendrum; Canonical: Lepanthes; Node#94936440; ResourceID: Lepanthes; Canonical: Neofinetia; Node#94945714; ResourceID: Neofinetia; Canonical: Pleurothallis; Node#94955918; ResourceID: Pleurothallis; Canonical: Polystachya; Node#94956835; ResourceID: Polystachya; Canonical: Agave; Node#94896973; ResourceID: Agave
[START] [2021-05-27 21:12:09] update_nodes
[STOP] [2021-05-27 21:12:46] update_nodes
[STOP] [2021-05-27 21:12:46] match_nodes
[START] [2021-05-27 21:12:46] reindex_search
[STOP] [2021-05-27 21:14:07] reindex_search
[START] [2021-05-27 21:14:07] normalize_units
[STOP] [2021-05-27 21:14:08] normalize_units
[START] [2021-05-27 21:14:08] calculate_statistics
[STOP] [2021-05-27 21:14:12] calculate_statistics
[START] [2021-05-27 21:14:12] complete_harvest_instance
[START] [2021-05-27 21:14:12] overall_tsv_creation
[INFO] [2021-05-27 21:14:12] Processing group of 83589 in 9 batches of 10000
[INFO] [2021-05-27 21:15:42] 13222 Traits (unfiltered)...
[INFO] [2021-05-27 21:17:52] 13222 Traits (filtered)...
[INFO] [2021-05-27 21:17:57] 0 Associations (filtered)...
[INFO] [2021-05-27 21:18:08] 108009 metadata added.
[INFO] [2021-05-27 21:18:08] 0 metadata added.
[INFO] [2021-05-27 21:19:56] 14173 Traits (unfiltered)...
[INFO] [2021-05-27 21:22:15] 14173 Traits (filtered)...
[INFO] [2021-05-27 21:22:18] 0 Associations (filtered)...
[INFO] [2021-05-27 21:22:31] 112747 metadata added.
[INFO] [2021-05-27 21:22:31] 0 metadata added.
[INFO] [2021-05-27 21:24:18] 14845 Traits (unfiltered)...
[INFO] [2021-05-27 21:26:38] 14845 Traits (filtered)...
[INFO] [2021-05-27 21:26:40] 0 Associations (filtered)...
[INFO] [2021-05-27 21:26:55] 121747 metadata added.
[INFO] [2021-05-27 21:26:55] 0 metadata added.
[INFO] [2021-05-27 21:28:40] 16039 Traits (unfiltered)...
[INFO] [2021-05-27 21:31:16] 16039 Traits (filtered)...
[INFO] [2021-05-27 21:31:19] 0 Associations (filtered)...
[INFO] [2021-05-27 21:31:35] 129376 metadata added.
[INFO] [2021-05-27 21:31:35] 0 metadata added.
[INFO] [2021-05-27 21:33:21] 13995 Traits (unfiltered)...
[INFO] [2021-05-27 21:35:36] 13995 Traits (filtered)...
[INFO] [2021-05-27 21:35:39] 0 Associations (filtered)...
[INFO] [2021-05-27 21:35:53] 114069 metadata added.
[INFO] [2021-05-27 21:35:53] 0 metadata added.
[INFO] [2021-05-27 21:37:41] 14445 Traits (unfiltered)...
[INFO] [2021-05-27 21:40:01] 14445 Traits (filtered)...
[INFO] [2021-05-27 21:40:04] 0 Associations (filtered)...
[INFO] [2021-05-27 21:40:19] 119955 metadata added.
[INFO] [2021-05-27 21:40:19] 0 metadata added.
[INFO] [2021-05-27 21:42:07] 17308 Traits (unfiltered)...
[INFO] [2021-05-27 21:44:53] 17308 Traits (filtered)...
[INFO] [2021-05-27 21:44:56] 0 Associations (filtered)...
[INFO] [2021-05-27 21:45:13] 143070 metadata added.
[INFO] [2021-05-27 21:45:13] 0 metadata added.
[INFO] [2021-05-27 21:47:01] 14195 Traits (unfiltered)...
[INFO] [2021-05-27 21:49:23] 14195 Traits (filtered)...
[INFO] [2021-05-27 21:49:25] 0 Associations (filtered)...
[INFO] [2021-05-27 21:49:40] 116902 metadata added.
[INFO] [2021-05-27 21:49:40] 0 metadata added.
[INFO] [2021-05-27 21:51:04] 10911 Traits (unfiltered)...
[INFO] [2021-05-27 21:52:55] 10911 Traits (filtered)...
[INFO] [2021-05-27 21:52:58] 0 Associations (filtered)...
[INFO] [2021-05-27 21:53:08] 90620 metadata added.
[INFO] [2021-05-27 21:53:08] 0 metadata added.
[INFO] [2021-05-27 21:53:39] Average Time: 227.727
[INFO] [2021-05-27 21:53:39] Total Time: 39m28s
[INFO] [2021-05-27 21:53:39] last 3 / first 3: 1.01
[INFO] [2021-05-27 21:53:39] Std.Dev: 21.67992158657406; Max: 260.04
[STOP] [2021-05-27 21:53:39] overall_tsv_creation
[INFO] [2021-05-27 21:53:39] Done. Check your files:
[INFO] [2021-05-27 21:53:41] (82493 lines) /app/public/data/gnntrs/publish_nodes.tsv
[INFO] [2021-05-27 21:53:42] (462103 lines) /app/public/data/gnntrs/publish_node_ancestors.tsv
[INFO] [2021-05-27 21:53:43] (83589 lines) /app/public/data/gnntrs/publish_scientific_names.tsv
[INFO] [2021-05-27 21:53:45] (129134 lines) /app/public/data/gnntrs/publish_traits.tsv
[INFO] [2021-05-27 21:53:46] (1056496 lines) /app/public/data/gnntrs/publish_metadata.tsv
[STOP] [2021-05-27 21:53:46] complete_harvest_instance
[START] [2021-05-27 21:53:46] completed
[STOP] [2021-05-27 21:53:46] completed
[STOP] [2021-05-27 21:53:46] logged process, took 5400.88

Latest Process

27 May 20:23 EXITED currently processing 0