Harvest for Polytraits Created 15 Aug 05:41

Harvesting Log

[INFO] [2023-08-15 05:41:53] Created harvest instance #4387
[STOP] [2023-08-15 05:41:53] create_harvest_instance
[START] [2023-08-15 05:41:53] fetch_files
[STOP] [2023-08-15 05:41:53] fetch_files
[START] [2023-08-15 05:41:53] validate_each_file
[INFO] [2023-08-15 05:41:53] Looping over 4 formats...
[INFO] [2023-08-15 05:41:53] ...refs (/app/public/data/polytraits/reference.tab)
[INFO] [2023-08-15 05:41:53] Valid: /app/public/data/polytraits/converted_csv/polytraits_refs_30491.csv (996 lines)
[INFO] [2023-08-15 05:41:53] ...nodes (/app/public/data/polytraits/taxon.tab)
[INFO] [2023-08-15 05:41:53] Valid: /app/public/data/polytraits/converted_csv/polytraits_nodes_30490.csv (1030 lines)
[INFO] [2023-08-15 05:41:53] ...occurrences (/app/public/data/polytraits/occurrence_specific.tab)
[INFO] [2023-08-15 05:41:54] Valid: /app/public/data/polytraits/converted_csv/polytraits_occurrences_30492.csv (21403 lines)
[INFO] [2023-08-15 05:41:54] ...measurements (/app/public/data/polytraits/measurement_or_fact_specific.tab)
[INFO] [2023-08-15 05:41:55] Valid: /app/public/data/polytraits/converted_csv/polytraits_measurements_30493.csv (21403 lines)
[STOP] [2023-08-15 05:41:55] validate_each_file
[START] [2023-08-15 05:41:55] convert_to_csv
[INFO] [2023-08-15 05:41:55] Looping over 4 formats...
[INFO] [2023-08-15 05:41:55] ...refs (/app/public/data/polytraits/reference.tab)
[CMD] [2023-08-15 05:41:55] /usr/bin/sort /app/public/data/polytraits/converted_csv/polytraits_refs_30491.csv > /app/public/data/polytraits/converted_csv/polytraits_refs_30491.csv_sorted
[INFO] [2023-08-15 05:41:55] Converted: /app/public/data/polytraits/converted_csv/polytraits_refs_30491.csv (996 lines)
[INFO] [2023-08-15 05:41:55] ...nodes (/app/public/data/polytraits/taxon.tab)
[CMD] [2023-08-15 05:41:55] /usr/bin/sort /app/public/data/polytraits/converted_csv/polytraits_nodes_30490.csv > /app/public/data/polytraits/converted_csv/polytraits_nodes_30490.csv_sorted
[INFO] [2023-08-15 05:41:55] Converted: /app/public/data/polytraits/converted_csv/polytraits_nodes_30490.csv (1030 lines)
[INFO] [2023-08-15 05:41:55] ...occurrences (/app/public/data/polytraits/occurrence_specific.tab)
[CMD] [2023-08-15 05:41:55] /usr/bin/sort /app/public/data/polytraits/converted_csv/polytraits_occurrences_30492.csv > /app/public/data/polytraits/converted_csv/polytraits_occurrences_30492.csv_sorted
[INFO] [2023-08-15 05:41:56] Converted: /app/public/data/polytraits/converted_csv/polytraits_occurrences_30492.csv (21403 lines)
[INFO] [2023-08-15 05:41:56] ...measurements (/app/public/data/polytraits/measurement_or_fact_specific.tab)
[CMD] [2023-08-15 05:41:56] /usr/bin/sort /app/public/data/polytraits/converted_csv/polytraits_measurements_30493.csv > /app/public/data/polytraits/converted_csv/polytraits_measurements_30493.csv_sorted
[INFO] [2023-08-15 05:41:56] Converted: /app/public/data/polytraits/converted_csv/polytraits_measurements_30493.csv (21403 lines)
[STOP] [2023-08-15 05:41:56] convert_to_csv
[START] [2023-08-15 05:41:56] calculate_delta
[INFO] [2023-08-15 05:41:56] Looping over 4 formats...
[INFO] [2023-08-15 05:41:56] ...refs (/app/public/data/polytraits/reference.tab)
[CMD] [2023-08-15 05:41:56] echo "0a" > /app/public/data/polytraits/diff/polytraits_refs_30491.diff
[CMD] [2023-08-15 05:41:56] tail -n +1 /app/public/data/polytraits/converted_csv/polytraits_refs_30491.csv >> /app/public/data/polytraits/diff/polytraits_refs_30491.diff
[CMD] [2023-08-15 05:41:56] echo "." >> /app/public/data/polytraits/diff/polytraits_refs_30491.diff
[INFO] [2023-08-15 05:41:56] Created diff: /app/public/data/polytraits/diff/polytraits_refs_30491.diff (998 lines)
[INFO] [2023-08-15 05:41:56] ...nodes (/app/public/data/polytraits/taxon.tab)
[CMD] [2023-08-15 05:41:56] echo "0a" > /app/public/data/polytraits/diff/polytraits_nodes_30490.diff
[CMD] [2023-08-15 05:41:56] tail -n +1 /app/public/data/polytraits/converted_csv/polytraits_nodes_30490.csv >> /app/public/data/polytraits/diff/polytraits_nodes_30490.diff
[CMD] [2023-08-15 05:41:57] echo "." >> /app/public/data/polytraits/diff/polytraits_nodes_30490.diff
[INFO] [2023-08-15 05:41:57] Created diff: /app/public/data/polytraits/diff/polytraits_nodes_30490.diff (1032 lines)
[INFO] [2023-08-15 05:41:57] ...occurrences (/app/public/data/polytraits/occurrence_specific.tab)
[CMD] [2023-08-15 05:41:57] echo "0a" > /app/public/data/polytraits/diff/polytraits_occurrences_30492.diff
[CMD] [2023-08-15 05:41:57] tail -n +1 /app/public/data/polytraits/converted_csv/polytraits_occurrences_30492.csv >> /app/public/data/polytraits/diff/polytraits_occurrences_30492.diff
[CMD] [2023-08-15 05:41:57] echo "." >> /app/public/data/polytraits/diff/polytraits_occurrences_30492.diff
[INFO] [2023-08-15 05:41:57] Created diff: /app/public/data/polytraits/diff/polytraits_occurrences_30492.diff (21405 lines)
[INFO] [2023-08-15 05:41:57] ...measurements (/app/public/data/polytraits/measurement_or_fact_specific.tab)
[CMD] [2023-08-15 05:41:57] echo "0a" > /app/public/data/polytraits/diff/polytraits_measurements_30493.diff
[CMD] [2023-08-15 05:41:57] tail -n +1 /app/public/data/polytraits/converted_csv/polytraits_measurements_30493.csv >> /app/public/data/polytraits/diff/polytraits_measurements_30493.diff
[CMD] [2023-08-15 05:41:57] echo "." >> /app/public/data/polytraits/diff/polytraits_measurements_30493.diff
[INFO] [2023-08-15 05:41:57] Created diff: /app/public/data/polytraits/diff/polytraits_measurements_30493.diff (21405 lines)
[STOP] [2023-08-15 05:41:57] calculate_delta
[START] [2023-08-15 05:41:57] parse_diff_and_store
[INFO] [2023-08-15 05:41:57] Handling diff: /app/public/data/polytraits/diff/polytraits_refs_30491.diff (998 lines)
[INFO] [2023-08-15 05:41:58] Loading refs diff file into memory (998 lines)...
[INFO] [2023-08-15 05:41:58] Storing 996 References (996/996/998)
[INFO] [2023-08-15 05:41:58] Handling diff: /app/public/data/polytraits/diff/polytraits_nodes_30490.diff (1032 lines)
[INFO] [2023-08-15 05:41:58] Loading nodes diff file into memory (1032 lines)...
[INFO] [2023-08-15 05:41:58] Storing 1030 ScientificNames (3090/1030/1032)
[INFO] [2023-08-15 05:41:59] Storing 1030 Identifiers (3090/1030/1032)
[INFO] [2023-08-15 05:41:59] Storing 1030 Nodes (3090/1030/1032)
[INFO] [2023-08-15 05:42:00] Handling diff: /app/public/data/polytraits/diff/polytraits_occurrences_30492.diff (21405 lines)
[INFO] [2023-08-15 05:42:00] Loading occurrences diff file into memory (21405 lines)...
[INFO] [2023-08-15 05:42:01] Storing 9999 Occurrences (10181/10000/21405)
[INFO] [2023-08-15 05:42:10] Storing 182 OccurrenceMetadata (10181/10000/21405)
[WARN] [2023-08-15 05:42:11] SKIPPED 161 Occurrence metadata (20342/20000/21405) with resource_pks already be in the database!
[INFO] [2023-08-15 05:42:11] Storing 10000 Occurrences (20342/20000/21405)
[INFO] [2023-08-15 05:42:13] Storing 0 OccurrenceMetadata (20342/20000/21405)
[WARN] [2023-08-15 05:42:13] No models to import, skipping!
[WARN] [2023-08-15 05:42:14] SKIPPED 25 Occurrence metadata (21771/21403/21405) with resource_pks already be in the database!
[INFO] [2023-08-15 05:42:14] Storing 1404 Occurrences (21771/21403/21405)
[INFO] [2023-08-15 05:42:14] Storing 0 OccurrenceMetadata (21771/21403/21405)
[WARN] [2023-08-15 05:42:14] No models to import, skipping!
[INFO] [2023-08-15 05:42:14] Handling diff: /app/public/data/polytraits/diff/polytraits_measurements_30493.diff (21405 lines)
[INFO] [2023-08-15 05:42:15] Loading measurements diff file into memory (21405 lines)...
[INFO] [2023-08-15 05:42:22] Storing 9999 TraitsReferences (48201/10000/21405)
[INFO] [2023-08-15 05:42:23] Storing 9999 Traits (48201/10000/21405)
[INFO] [2023-08-15 05:42:26] Storing 28203 MetaTraits (48201/10000/21405)
[INFO] [2023-08-15 05:42:37] Storing 10000 TraitsReferences (96414/20000/21405)
[INFO] [2023-08-15 05:42:38] Storing 10000 Traits (96414/20000/21405)
[INFO] [2023-08-15 05:42:41] Storing 28213 MetaTraits (96414/20000/21405)
[INFO] [2023-08-15 05:42:46] Storing 1404 TraitsReferences (103196/21403/21405)
[INFO] [2023-08-15 05:42:47] Storing 1404 Traits (103196/21403/21405)
[INFO] [2023-08-15 05:42:47] Storing 3974 MetaTraits (103196/21403/21405)
[STOP] [2023-08-15 05:42:48] parse_diff_and_store
[START] [2023-08-15 05:42:48] resolve_keys
[2023-08-15 05:42:57] Resolving downloaded urls (this is not actually downloading them yet)
[INFO] [2023-08-15 05:43:05] Occurrences to nodes (through scientific_names)...
[INFO] [2023-08-15 05:43:06] traits to occurrences...
[INFO] [2023-08-15 05:43:10] traits to nodes (through occurrences)...
[INFO] [2023-08-15 05:43:11] Traits to sex term...
[INFO] [2023-08-15 05:43:12] Traits to lifestage term...
[INFO] [2023-08-15 05:43:12] MetaTraits to traits...
[INFO] [2023-08-15 05:43:15] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2023-08-15 05:43:17] Assocs to occurrences...
[INFO] [2023-08-15 05:43:17] Assocs to nodes...
[INFO] [2023-08-15 05:43:17] Assoc to sex term...
[INFO] [2023-08-15 05:43:17] Assoc to lifestage term...
[INFO] [2023-08-15 05:43:17] MetaAssoc to assocs...
[STOP] [2023-08-15 05:43:17] resolve_keys
[START] [2023-08-15 05:43:17] hold_for_later_1
[STOP] [2023-08-15 05:43:17] hold_for_later_1
[START] [2023-08-15 05:43:17] hold_for_later_2
[STOP] [2023-08-15 05:43:17] hold_for_later_2
[START] [2023-08-15 05:43:17] resolve_missing_parents
[STOP] [2023-08-15 05:43:17] resolve_missing_parents
[START] [2023-08-15 05:43:17] rebuild_nodes
[START] [2023-08-15 05:43:17] Flattener#flatten
[START] [2023-08-15 05:43:17] Flattener#study_resource
[START] [2023-08-15 05:43:17] Flattener#build_ancestry
[STOP] [2023-08-15 05:43:17] Flattener#build_ancestry
[INFO] [2023-08-15 05:43:17] 1030 ancestry keys
[START] [2023-08-15 05:43:17] build_node_ancestors
[INFO] [2023-08-15 05:43:17] old ancestors deleted.
[STOP] [2023-08-15 05:43:17] build_node_ancestors
[START] [2023-08-15 05:43:17] Flattener#propagate_ancestor_ids
[STOP] [2023-08-15 05:43:18] Flattener#propagate_ancestor_ids
[STOP] [2023-08-15 05:43:18] Flattener#flatten
[STOP] [2023-08-15 05:43:18] rebuild_nodes
[START] [2023-08-15 05:43:18] resolve_missing_media_owners
[STOP] [2023-08-15 05:43:18] resolve_missing_media_owners
[START] [2023-08-15 05:43:18] sanitize_media_verbatims
[STOP] [2023-08-15 05:43:18] sanitize_media_verbatims
[START] [2023-08-15 05:43:18] queue_downloads
[STOP] [2023-08-15 05:43:18] queue_downloads
[START] [2023-08-15 05:43:18] parse_names
[WARN] [2023-08-15 05:43:18] I see 1030 names which still need to be parsed.
[WARN] [2023-08-15 05:43:18] Names to parse: 1030 formatted: 1030 learned: 1030 parsed: 1030
[STOP] [2023-08-15 05:43:19] parse_names
[START] [2023-08-15 05:43:19] denormalize_canonical_names_to_nodes
[STOP] [2023-08-15 05:43:19] denormalize_canonical_names_to_nodes
[START] [2023-08-15 05:43:19] match_nodes
[START] [2023-08-15 05:43:19] map_all_nodes_to_pages
[STOP] [2023-08-15 05:43:45] map_all_nodes_to_pages
[INFO] [2023-08-15 05:43:45] 55 Unmatched nodes (of 1030)! That's too many to output. Full list in /app/public/data/polytraits/unmatched_nodes.txt ; First 10: Canonical: Polychaeta; Node#136517972; ResourceID: 386; Canonical: Nematonereis; Node#136517900; ResourceID: 314; Canonical: Scoletoma funchalensis; Node#136517433; ResourceID: 1355; Canonical: Scoletoma tetraura; Node#136517804; ResourceID: 1888; Canonical: Alciopidae; Node#136517411; ResourceID: 1333; Canonical: Hesiocaeca; Node#136517348; ResourceID: 1272; Canonical: Kefersteinia cirrata; Node#136517743; ResourceID: 1765; Canonical: Hediste kermadeca; Node#136517646; ResourceID: 1614; Canonical: Nereis fucata; Node#136517622; ResourceID: 1580; Canonical: Ceratonereis costae; Node#136517489; ResourceID: 1415
[START] [2023-08-15 05:43:45] update_nodes
[STOP] [2023-08-15 05:43:45] update_nodes
[STOP] [2023-08-15 05:43:45] match_nodes
[START] [2023-08-15 05:43:45] reindex_search
[STOP] [2023-08-15 05:43:47] reindex_search
[START] [2023-08-15 05:43:47] normalize_units
[STOP] [2023-08-15 05:43:47] normalize_units
[START] [2023-08-15 05:43:47] calculate_statistics
[INFO] [2023-08-15 05:43:51] Duplicate page_id count: 0
[STOP] [2023-08-15 05:43:51] calculate_statistics
[START] [2023-08-15 05:43:51] complete_harvest_instance
[START] [2023-08-15 05:43:51] overall_tsv_creation
[INFO] [2023-08-15 05:43:51] Exporting 1030 nodes as TSV in batches of 10000...
[INFO] [2023-08-15 05:43:51] Processing group of 1030 in 1 batches of 10000
[INFO] [2023-08-15 05:43:53] 21403 Traits (unfiltered) and 0 associations...
[INFO] [2023-08-15 05:43:53] Building Traits map for 1030 nodes (this can take a while)...
[INFO] [2023-08-15 05:44:02] Mapped 21403 traits (60390 meta) for 1030 nodes.
[INFO] [2023-08-15 05:44:02] Building Associations map (this can take a while)...
[INFO] [2023-08-15 05:44:02] Done. 0 assocs mapped (0 meta).
[INFO] [2023-08-15 05:44:02] Adding 21403 traits...
[INFO] [2023-08-15 05:44:08] 42806 metadata added.
[INFO] [2023-08-15 05:44:08] Adding 0 assocs...
[INFO] [2023-08-15 05:44:08] 0 metadata added.
[INFO] [2023-08-15 05:44:57] Processed 1030/1030 nodes
[INFO] [2023-08-15 05:44:57] Average Time: 66.05
[INFO] [2023-08-15 05:44:57] Total Time: 1m7s
[STOP] [2023-08-15 05:44:57] overall_tsv_creation
[INFO] [2023-08-15 05:44:57] Done. Check your files:
[INFO] [2023-08-15 05:44:57] (1030 lines) /app/public/data/polytraits/publish_nodes.tsv
[INFO] [2023-08-15 05:44:57] (1030 lines) /app/public/data/polytraits/publish_identifiers.tsv
[INFO] [2023-08-15 05:44:57] (4791 lines) /app/public/data/polytraits/publish_node_ancestors.tsv
[INFO] [2023-08-15 05:44:58] (1030 lines) /app/public/data/polytraits/publish_scientific_names.tsv
[INFO] [2023-08-15 05:44:58] (21404 lines) /app/public/data/polytraits/publish_traits.tsv
[INFO] [2023-08-15 05:44:58] (42807 lines) /app/public/data/polytraits/publish_metadata.tsv
[STOP] [2023-08-15 05:44:58] complete_harvest_instance
[START] [2023-08-15 05:44:58] completed
[STOP] [2023-08-15 05:44:58] completed
[STOP] [2023-08-15 05:44:58] logged process, took 184.82

Latest Process

15 Aug 05:41 EXITED currently processing 0