Harvest for protisten.de Created 04 Sep 07:04

Harvesting Log

[INFO] [2023-09-04 07:04:47] Created harvest instance #4392
[STOP] [2023-09-04 07:04:47] create_harvest_instance
[START] [2023-09-04 07:04:47] fetch_files
[STOP] [2023-09-04 07:04:47] fetch_files
[START] [2023-09-04 07:04:47] validate_each_file
[INFO] [2023-09-04 07:04:47] Looping over 3 formats...
[INFO] [2023-09-04 07:04:47] ...agents (/app/public/data/ptgdp/agent.tab)
[INFO] [2023-09-04 07:04:47] Valid: /app/public/data/ptgdp/converted_csv/ptgdp_agents_30500.csv (1 lines)
[INFO] [2023-09-04 07:04:47] ...nodes (/app/public/data/ptgdp/taxon.tab)
[INFO] [2023-09-04 07:04:47] Valid: /app/public/data/ptgdp/converted_csv/ptgdp_nodes_30501.csv (1281 lines)
[INFO] [2023-09-04 07:04:47] ...media (/app/public/data/ptgdp/media_resource.tab)
[INFO] [2023-09-04 07:04:47] Valid: /app/public/data/ptgdp/converted_csv/ptgdp_media_30502.csv (2540 lines)
[STOP] [2023-09-04 07:04:47] validate_each_file
[START] [2023-09-04 07:04:47] convert_to_csv
[INFO] [2023-09-04 07:04:47] Looping over 3 formats...
[INFO] [2023-09-04 07:04:47] ...agents (/app/public/data/ptgdp/agent.tab)
[CMD] [2023-09-04 07:04:47] /usr/bin/sort /app/public/data/ptgdp/converted_csv/ptgdp_agents_30500.csv > /app/public/data/ptgdp/converted_csv/ptgdp_agents_30500.csv_sorted
[INFO] [2023-09-04 07:04:48] Converted: /app/public/data/ptgdp/converted_csv/ptgdp_agents_30500.csv (1 lines)
[INFO] [2023-09-04 07:04:48] ...nodes (/app/public/data/ptgdp/taxon.tab)
[CMD] [2023-09-04 07:04:48] /usr/bin/sort /app/public/data/ptgdp/converted_csv/ptgdp_nodes_30501.csv > /app/public/data/ptgdp/converted_csv/ptgdp_nodes_30501.csv_sorted
[INFO] [2023-09-04 07:04:48] Converted: /app/public/data/ptgdp/converted_csv/ptgdp_nodes_30501.csv (1281 lines)
[INFO] [2023-09-04 07:04:48] ...media (/app/public/data/ptgdp/media_resource.tab)
[CMD] [2023-09-04 07:04:48] /usr/bin/sort /app/public/data/ptgdp/converted_csv/ptgdp_media_30502.csv > /app/public/data/ptgdp/converted_csv/ptgdp_media_30502.csv_sorted
[INFO] [2023-09-04 07:04:48] Converted: /app/public/data/ptgdp/converted_csv/ptgdp_media_30502.csv (2540 lines)
[STOP] [2023-09-04 07:04:48] convert_to_csv
[START] [2023-09-04 07:04:48] calculate_delta
[INFO] [2023-09-04 07:04:48] Looping over 3 formats...
[INFO] [2023-09-04 07:04:48] ...agents (/app/public/data/ptgdp/agent.tab)
[CMD] [2023-09-04 07:04:48] echo "0a" > /app/public/data/ptgdp/diff/ptgdp_agents_30500.diff
[CMD] [2023-09-04 07:04:48] tail -n +1 /app/public/data/ptgdp/converted_csv/ptgdp_agents_30500.csv >> /app/public/data/ptgdp/diff/ptgdp_agents_30500.diff
[CMD] [2023-09-04 07:04:48] echo "." >> /app/public/data/ptgdp/diff/ptgdp_agents_30500.diff
[INFO] [2023-09-04 07:04:48] Created diff: /app/public/data/ptgdp/diff/ptgdp_agents_30500.diff (3 lines)
[INFO] [2023-09-04 07:04:48] ...nodes (/app/public/data/ptgdp/taxon.tab)
[CMD] [2023-09-04 07:04:48] echo "0a" > /app/public/data/ptgdp/diff/ptgdp_nodes_30501.diff
[CMD] [2023-09-04 07:04:48] tail -n +1 /app/public/data/ptgdp/converted_csv/ptgdp_nodes_30501.csv >> /app/public/data/ptgdp/diff/ptgdp_nodes_30501.diff
[CMD] [2023-09-04 07:04:48] echo "." >> /app/public/data/ptgdp/diff/ptgdp_nodes_30501.diff
[INFO] [2023-09-04 07:04:48] Created diff: /app/public/data/ptgdp/diff/ptgdp_nodes_30501.diff (1283 lines)
[INFO] [2023-09-04 07:04:48] ...media (/app/public/data/ptgdp/media_resource.tab)
[CMD] [2023-09-04 07:04:48] echo "0a" > /app/public/data/ptgdp/diff/ptgdp_media_30502.diff
[CMD] [2023-09-04 07:04:49] tail -n +1 /app/public/data/ptgdp/converted_csv/ptgdp_media_30502.csv >> /app/public/data/ptgdp/diff/ptgdp_media_30502.diff
[CMD] [2023-09-04 07:04:49] echo "." >> /app/public/data/ptgdp/diff/ptgdp_media_30502.diff
[INFO] [2023-09-04 07:04:49] Created diff: /app/public/data/ptgdp/diff/ptgdp_media_30502.diff (2542 lines)
[STOP] [2023-09-04 07:04:49] calculate_delta
[START] [2023-09-04 07:04:49] parse_diff_and_store
[INFO] [2023-09-04 07:04:49] Handling diff: /app/public/data/ptgdp/diff/ptgdp_agents_30500.diff (3 lines)
[INFO] [2023-09-04 07:04:49] Loading agents diff file into memory (3 lines)...
[INFO] [2023-09-04 07:04:49] Storing 1 Attributions (1/1/3)
[INFO] [2023-09-04 07:04:49] Handling diff: /app/public/data/ptgdp/diff/ptgdp_nodes_30501.diff (1283 lines)
[INFO] [2023-09-04 07:04:49] Loading nodes diff file into memory (1283 lines)...
[WARN] [2023-09-04 07:04:49] Filtered Scientific Name `Bryozoa/ Moostierchen` to `Bryozoa Moostierchen`
[WARN] [2023-09-04 07:04:49] Filtered Scientific Name `Rhodophyta/Rotalge` to `RhodophytaRotalge`
[WARN] [2023-09-04 07:04:49] Filtered Scientific Name `Chlorophyta-Cyste/Chlorophyta cyst` to `Chlorophyta-CysteChlorophyta cyst`
[INFO] [2023-09-04 07:04:49] Storing 1281 ScientificNames (2562/1281/1283)
[INFO] [2023-09-04 07:04:50] Storing 1281 Nodes (2562/1281/1283)
[INFO] [2023-09-04 07:04:50] Handling diff: /app/public/data/ptgdp/diff/ptgdp_media_30502.diff (2542 lines)
[INFO] [2023-09-04 07:04:50] Loading media diff file into memory (2542 lines)...
[INFO] [2023-09-04 07:04:52] Storing 2540 ContentAttributions (5080/2540/2542)
[INFO] [2023-09-04 07:04:53] Storing 2540 Media (5080/2540/2542)
[STOP] [2023-09-04 07:04:54] parse_diff_and_store
[START] [2023-09-04 07:04:54] resolve_keys
[2023-09-04 07:04:56] Resolving downloaded urls (this is not actually downloading them yet)
[INFO] [2023-09-04 07:05:04] Occurrences to nodes (through scientific_names)...
[INFO] [2023-09-04 07:05:04] traits to occurrences...
[INFO] [2023-09-04 07:05:04] traits to nodes (through occurrences)...
[INFO] [2023-09-04 07:05:04] Traits to sex term...
[INFO] [2023-09-04 07:05:04] Traits to lifestage term...
[INFO] [2023-09-04 07:05:04] MetaTraits to traits...
[INFO] [2023-09-04 07:05:04] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2023-09-04 07:05:04] Assocs to occurrences...
[INFO] [2023-09-04 07:05:04] Assocs to nodes...
[INFO] [2023-09-04 07:05:04] Assoc to sex term...
[INFO] [2023-09-04 07:05:04] Assoc to lifestage term...
[INFO] [2023-09-04 07:05:04] MetaAssoc to assocs...
[STOP] [2023-09-04 07:05:04] resolve_keys
[START] [2023-09-04 07:05:04] hold_for_later_1
[STOP] [2023-09-04 07:05:04] hold_for_later_1
[START] [2023-09-04 07:05:04] hold_for_later_2
[STOP] [2023-09-04 07:05:04] hold_for_later_2
[START] [2023-09-04 07:05:04] resolve_missing_parents
[STOP] [2023-09-04 07:05:04] resolve_missing_parents
[START] [2023-09-04 07:05:04] rebuild_nodes
[START] [2023-09-04 07:05:04] Flattener#flatten
[START] [2023-09-04 07:05:04] Flattener#study_resource
[START] [2023-09-04 07:05:04] Flattener#build_ancestry
[STOP] [2023-09-04 07:05:04] Flattener#build_ancestry
[INFO] [2023-09-04 07:05:04] 1281 ancestry keys
[START] [2023-09-04 07:05:04] build_node_ancestors
[INFO] [2023-09-04 07:05:04] old ancestors deleted.
[STOP] [2023-09-04 07:05:04] build_node_ancestors
[START] [2023-09-04 07:05:05] Flattener#propagate_ancestor_ids
[STOP] [2023-09-04 07:05:05] Flattener#propagate_ancestor_ids
[STOP] [2023-09-04 07:05:05] Flattener#flatten
[STOP] [2023-09-04 07:05:05] rebuild_nodes
[START] [2023-09-04 07:05:05] resolve_missing_media_owners
[STOP] [2023-09-04 07:05:05] resolve_missing_media_owners
[START] [2023-09-04 07:05:05] sanitize_media_verbatims
[STOP] [2023-09-04 07:05:05] sanitize_media_verbatims
[START] [2023-09-04 07:05:05] queue_downloads
[STOP] [2023-09-04 07:05:05] queue_downloads
[START] [2023-09-04 07:05:05] parse_names
[WARN] [2023-09-04 07:05:05] I see 1281 names which still need to be parsed.
[INFO] [2023-09-04 07:05:05] 0% of media downloaded
[WARN] [2023-09-04 07:05:05] Names to parse: 1281 formatted: 1281 learned: 1259 parsed: 1281
[INFO] [2023-09-04 07:05:06] 0% of media downloaded
[STOP] [2023-09-04 07:05:07] parse_names
[START] [2023-09-04 07:05:07] denormalize_canonical_names_to_nodes
[STOP] [2023-09-04 07:05:07] denormalize_canonical_names_to_nodes
[START] [2023-09-04 07:05:07] match_nodes
[START] [2023-09-04 07:05:07] map_all_nodes_to_pages
[INFO] [2023-09-04 07:05:09] 0% of media downloaded
[INFO] [2023-09-04 07:05:09] 0% of media downloaded
[INFO] [2023-09-04 07:05:09] 0% of media downloaded
[INFO] [2023-09-04 07:05:09] 0% of media downloaded
[STOP] [2023-09-04 07:06:06] map_all_nodes_to_pages
[INFO] [2023-09-04 07:06:06] 49 Unmatched nodes (of 1281)! That's too many to output. Full list in /app/public/data/ptgdp/unmatched_nodes.txt ; First 10: Canonical: Chroococcus turgidus with chytrid parasite; Node#136523562; ResourceID: 6d3e0229581823e4e4565eef76343072; Canonical: Synechoccales; Node#136524000; ResourceID: cyanobacteria-synechoccales; Canonical: Merismopediacea; Node#136524477; ResourceID: synechoccales-merismopediacea; Canonical: Siderocapsa; Node#136523679; ResourceID: 976a61ae86cac16ed1137606867d2dce; Canonical: Eucaryota; Node#136524135; ResourceID: eucaryota; Canonical: Nassulophorea; Node#136524267; ResourceID: intramacronucleata-nassulophorea; Canonical: Hymenostomatia; Node#136524325; ResourceID: oligohymenophorea-hymenostomatia; Canonical: Parameciina; Node#136524354; ResourceID: peniculida-parameciina; Canonical: Peniculiada; Node#136524328; ResourceID: oligohymenophorea-peniculiada; Canonical: Protostomatea; Node#136524272; ResourceID: intramacronucleata-protostomatea
[START] [2023-09-04 07:06:06] update_nodes
[STOP] [2023-09-04 07:06:07] update_nodes
[STOP] [2023-09-04 07:06:07] match_nodes
[START] [2023-09-04 07:06:07] reindex_search
[STOP] [2023-09-04 07:06:10] reindex_search
[START] [2023-09-04 07:06:10] normalize_units
[STOP] [2023-09-04 07:06:10] normalize_units
[START] [2023-09-04 07:06:10] calculate_statistics
[INFO] [2023-09-04 07:06:10] Duplicate page_id count: 106
[STOP] [2023-09-04 07:06:10] calculate_statistics
[START] [2023-09-04 07:06:10] complete_harvest_instance
[START] [2023-09-04 07:06:10] overall_tsv_creation
[INFO] [2023-09-04 07:06:10] Exporting 1281 nodes as TSV in batches of 10000...
[INFO] [2023-09-04 07:06:10] Processing group of 1281 in 1 batches of 10000
[INFO] [2023-09-04 07:06:25] Processed 1281/1281 nodes
[INFO] [2023-09-04 07:06:25] Average Time: 13.8
[INFO] [2023-09-04 07:06:25] Total Time: 16s
[STOP] [2023-09-04 07:06:25] overall_tsv_creation
[INFO] [2023-09-04 07:06:25] Done. Check your files:
[INFO] [2023-09-04 07:06:25] (1281 lines) /app/public/data/ptgdp/publish_nodes.tsv
[INFO] [2023-09-04 07:06:25] (8603 lines) /app/public/data/ptgdp/publish_node_ancestors.tsv
[INFO] [2023-09-04 07:06:26] (1281 lines) /app/public/data/ptgdp/publish_scientific_names.tsv
[INFO] [2023-09-04 07:06:26] (2540 lines) /app/public/data/ptgdp/publish_media.tsv
[INFO] [2023-09-04 07:06:26] (283 lines) /app/public/data/ptgdp/publish_image_info.tsv
[INFO] [2023-09-04 07:06:26] (2540 lines) /app/public/data/ptgdp/publish_attributions.tsv
[STOP] [2023-09-04 07:06:26] complete_harvest_instance
[START] [2023-09-04 07:06:26] completed
[STOP] [2023-09-04 07:06:26] completed
[STOP] [2023-09-04 07:06:26] logged process, took 99.02
[INFO] [2023-09-04 07:07:38] 80% of media downloaded
[INFO] [2023-09-04 07:08:00] 100% of media downloaded
[INFO] [2023-09-04 07:08:00] 100% of media downloaded
[INFO] [2023-09-04 07:08:00] 100% of media downloaded
[INFO] [2023-09-04 07:08:00] 100% of media downloaded
[ERR] [2023-09-04 07:08:00][hdls] NO additional images were found to download
[ERR] [2023-09-04 07:08:00][hdls] NO additional images were found to download
[ERR] [2023-09-04 07:08:00][hdls] NO additional images were found to download
[ERR] [2023-09-04 07:08:00][hdls] NO additional images were found to download
[ERR] [2023-09-04 07:08:00][hdls] NO additional images were found to download
[ERR] [2023-09-04 07:08:00][hdls] NO additional images were found to download
[ERR] [2023-09-04 07:08:00][hdls] NO additional images were found to download
[ERR] [2023-09-04 07:08:00][hdls] NO additional images were found to download
[ERR] [2023-09-04 07:08:00][hdls] NO additional images were found to download
[ERR] [2023-09-04 07:08:00][hdls] NO additional images were found to download
[ERR] [2023-09-04 07:08:00][hdls] NO additional images were found to download
[ERR] [2023-09-04 07:08:00][hdls] NO additional images were found to download

Latest Process

04 Sep 07:04 EXITED currently processing 0