harvested resources
sign in

Harvest for protisten.de Created 06 Sep 00:45

Stage: completed
Fetched: 06 Sep 00:45
Validated: 06 Sep 00:45
Deltas Created 06 Sep 00:45
Units Normalized: 06 Sep 00:47
Ancestry Built: 06 Sep 00:46
Nodes Matched: 06 Sep 00:47
Names Parsed: 06 Sep 00:46
New Models Stored: 06 Sep 00:45
Indexed: 06 Sep 00:47
Completed: 06 Sep 00:48
Time to Harvest: less than a minute

Harvesting Log

(187 lines) 
[INFO] [2024-09-06 00:45:31] Created harvest instance #4557
[STOP] [2024-09-06 00:45:31] create_harvest_instance
[START] [2024-09-06 00:45:31] fetch_files
[STOP] [2024-09-06 00:45:31] fetch_files
[START] [2024-09-06 00:45:31] validate_each_file
[INFO] [2024-09-06 00:45:31] Looping over 3 formats...
[INFO] [2024-09-06 00:45:31] ...agents (/app/public/data/ptgdp/agent.tab)
[INFO] [2024-09-06 00:45:32] Valid: /app/public/data/ptgdp/converted_csv/ptgdp_agents_31194.csv (1 lines)
[INFO] [2024-09-06 00:45:32] ...nodes (/app/public/data/ptgdp/taxon.tab)
[INFO] [2024-09-06 00:45:32] Valid: /app/public/data/ptgdp/converted_csv/ptgdp_nodes_31195.csv (1317 lines)
[INFO] [2024-09-06 00:45:32] ...media (/app/public/data/ptgdp/media_resource.tab)
[INFO] [2024-09-06 00:45:32] Valid: /app/public/data/ptgdp/converted_csv/ptgdp_media_31196.csv (2672 lines)
[STOP] [2024-09-06 00:45:32] validate_each_file
[START] [2024-09-06 00:45:32] convert_to_csv
[INFO] [2024-09-06 00:45:32] Looping over 3 formats...
[INFO] [2024-09-06 00:45:32] ...agents (/app/public/data/ptgdp/agent.tab)
[CMD] [2024-09-06 00:45:32] /usr/bin/sort /app/public/data/ptgdp/converted_csv/ptgdp_agents_31194.csv > /app/public/data/ptgdp/converted_csv/ptgdp_agents_31194.csv_sorted
[INFO] [2024-09-06 00:45:32] Converted: /app/public/data/ptgdp/converted_csv/ptgdp_agents_31194.csv (1 lines)
[INFO] [2024-09-06 00:45:32] ...nodes (/app/public/data/ptgdp/taxon.tab)
[CMD] [2024-09-06 00:45:32] /usr/bin/sort /app/public/data/ptgdp/converted_csv/ptgdp_nodes_31195.csv > /app/public/data/ptgdp/converted_csv/ptgdp_nodes_31195.csv_sorted
[INFO] [2024-09-06 00:45:32] Converted: /app/public/data/ptgdp/converted_csv/ptgdp_nodes_31195.csv (1317 lines)
[INFO] [2024-09-06 00:45:32] ...media (/app/public/data/ptgdp/media_resource.tab)
[CMD] [2024-09-06 00:45:32] /usr/bin/sort /app/public/data/ptgdp/converted_csv/ptgdp_media_31196.csv > /app/public/data/ptgdp/converted_csv/ptgdp_media_31196.csv_sorted
[INFO] [2024-09-06 00:45:33] Converted: /app/public/data/ptgdp/converted_csv/ptgdp_media_31196.csv (2672 lines)
[STOP] [2024-09-06 00:45:33] convert_to_csv
[START] [2024-09-06 00:45:33] calculate_delta
[INFO] [2024-09-06 00:45:33] Looping over 3 formats...
[INFO] [2024-09-06 00:45:33] ...agents (/app/public/data/ptgdp/agent.tab)
[CMD] [2024-09-06 00:45:34] echo "0a" > /app/public/data/ptgdp/diff/ptgdp_agents_31194.diff
[CMD] [2024-09-06 00:45:34] tail -n +1 /app/public/data/ptgdp/converted_csv/ptgdp_agents_31194.csv >> /app/public/data/ptgdp/diff/ptgdp_agents_31194.diff
[CMD] [2024-09-06 00:45:34] echo "." >> /app/public/data/ptgdp/diff/ptgdp_agents_31194.diff
[INFO] [2024-09-06 00:45:34] Created diff: /app/public/data/ptgdp/diff/ptgdp_agents_31194.diff (3 lines)
[INFO] [2024-09-06 00:45:34] ...nodes (/app/public/data/ptgdp/taxon.tab)
[CMD] [2024-09-06 00:45:34] echo "0a" > /app/public/data/ptgdp/diff/ptgdp_nodes_31195.diff
[CMD] [2024-09-06 00:45:34] tail -n +1 /app/public/data/ptgdp/converted_csv/ptgdp_nodes_31195.csv >> /app/public/data/ptgdp/diff/ptgdp_nodes_31195.diff
[CMD] [2024-09-06 00:45:34] echo "." >> /app/public/data/ptgdp/diff/ptgdp_nodes_31195.diff
[INFO] [2024-09-06 00:45:34] Created diff: /app/public/data/ptgdp/diff/ptgdp_nodes_31195.diff (1319 lines)
[INFO] [2024-09-06 00:45:34] ...media (/app/public/data/ptgdp/media_resource.tab)
[CMD] [2024-09-06 00:45:34] echo "0a" > /app/public/data/ptgdp/diff/ptgdp_media_31196.diff
[CMD] [2024-09-06 00:45:34] tail -n +1 /app/public/data/ptgdp/converted_csv/ptgdp_media_31196.csv >> /app/public/data/ptgdp/diff/ptgdp_media_31196.diff
[CMD] [2024-09-06 00:45:34] echo "." >> /app/public/data/ptgdp/diff/ptgdp_media_31196.diff
[INFO] [2024-09-06 00:45:34] Created diff: /app/public/data/ptgdp/diff/ptgdp_media_31196.diff (2674 lines)
[STOP] [2024-09-06 00:45:34] calculate_delta
[START] [2024-09-06 00:45:34] parse_diff_and_store
[INFO] [2024-09-06 00:45:34] Handling diff: /app/public/data/ptgdp/diff/ptgdp_agents_31194.diff (3 lines)
[INFO] [2024-09-06 00:45:34] Loading agents diff file into memory (3 lines)...
[INFO] [2024-09-06 00:45:34] Storing 1 Attributions (1/1/3)
[INFO] [2024-09-06 00:45:34] Handling diff: /app/public/data/ptgdp/diff/ptgdp_nodes_31195.diff (1319 lines)
[INFO] [2024-09-06 00:45:34] Loading nodes diff file into memory (1319 lines)...
[WARN] [2024-09-06 00:45:34] Filtered Scientific Name `Bryozoa/ Moostierchen` to `Bryozoa Moostierchen`
[WARN] [2024-09-06 00:45:34] Filtered Scientific Name `Chlorophyta-Cyste/Chlorophyta cyst` to `Chlorophyta-CysteChlorophyta cyst`
[INFO] [2024-09-06 00:45:35] Storing 1317 ScientificNames (2634/1317/1319)
[INFO] [2024-09-06 00:45:37] Storing 1317 Nodes (2634/1317/1319)
[INFO] [2024-09-06 00:45:38] Handling diff: /app/public/data/ptgdp/diff/ptgdp_media_31196.diff (2674 lines)
[INFO] [2024-09-06 00:45:38] Loading media diff file into memory (2674 lines)...
[INFO] [2024-09-06 00:45:40] Storing 2672 ContentAttributions (5344/2672/2674)
[INFO] [2024-09-06 00:45:41] Storing 2672 Media (5344/2672/2674)
[STOP] [2024-09-06 00:45:42] parse_diff_and_store
[START] [2024-09-06 00:45:42] resolve_keys
[2024-09-06 00:45:47] Resolving downloaded urls (this is not actually downloading them yet)
[INFO] [2024-09-06 00:46:00] Occurrences to nodes (through scientific_names)...
[INFO] [2024-09-06 00:46:00] traits to occurrences...
[INFO] [2024-09-06 00:46:00] traits to nodes (through occurrences)...
[INFO] [2024-09-06 00:46:00] Traits to sex term...
[INFO] [2024-09-06 00:46:00] Traits to lifestage term...
[INFO] [2024-09-06 00:46:00] MetaTraits to traits...
[INFO] [2024-09-06 00:46:00] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2024-09-06 00:46:00] Assocs to occurrences...
[INFO] [2024-09-06 00:46:00] Assocs to nodes...
[INFO] [2024-09-06 00:46:00] Assoc to sex term...
[INFO] [2024-09-06 00:46:00] Assoc to lifestage term...
[INFO] [2024-09-06 00:46:00] MetaAssoc to assocs...
[STOP] [2024-09-06 00:46:00] resolve_keys
[START] [2024-09-06 00:46:00] hold_for_later_1
[STOP] [2024-09-06 00:46:00] hold_for_later_1
[START] [2024-09-06 00:46:00] hold_for_later_2
[STOP] [2024-09-06 00:46:00] hold_for_later_2
[START] [2024-09-06 00:46:00] resolve_missing_parents
[STOP] [2024-09-06 00:46:00] resolve_missing_parents
[START] [2024-09-06 00:46:00] rebuild_nodes
[START] [2024-09-06 00:46:00] Flattener#flatten
[START] [2024-09-06 00:46:00] Flattener#study_resource
[START] [2024-09-06 00:46:00] Flattener#build_ancestry
[STOP] [2024-09-06 00:46:00] Flattener#build_ancestry
[INFO] [2024-09-06 00:46:00] 1317 ancestry keys
[START] [2024-09-06 00:46:00] build_node_ancestors
[INFO] [2024-09-06 00:46:00] old ancestors deleted.
[STOP] [2024-09-06 00:46:01] build_node_ancestors
[START] [2024-09-06 00:46:01] Flattener#propagate_ancestor_ids
[STOP] [2024-09-06 00:46:02] Flattener#propagate_ancestor_ids
[STOP] [2024-09-06 00:46:02] Flattener#flatten
[STOP] [2024-09-06 00:46:02] rebuild_nodes
[START] [2024-09-06 00:46:02] resolve_missing_media_owners
[STOP] [2024-09-06 00:46:02] resolve_missing_media_owners
[START] [2024-09-06 00:46:02] sanitize_media_verbatims
[STOP] [2024-09-06 00:46:02] sanitize_media_verbatims
[START] [2024-09-06 00:46:02] queue_downloads
[STOP] [2024-09-06 00:46:02] queue_downloads
[START] [2024-09-06 00:46:02] parse_names
[WARN] [2024-09-06 00:46:02] I see 1317 names which still need to be parsed.
[WARN] [2024-09-06 00:46:02] Names to parse: 1317 formatted: 1317 learned: 1295 parsed: 1317
[INFO] [2024-09-06 00:46:05] 0% of media downloaded
[STOP] [2024-09-06 00:46:05] parse_names
[INFO] [2024-09-06 00:46:05] 0% of media downloaded
[INFO] [2024-09-06 00:46:05] 0% of media downloaded
[START] [2024-09-06 00:46:05] denormalize_canonical_names_to_nodes
[STOP] [2024-09-06 00:46:05] denormalize_canonical_names_to_nodes
[START] [2024-09-06 00:46:05] match_nodes
[START] [2024-09-06 00:46:05] map_all_nodes_to_pages
[INFO] [2024-09-06 00:46:08] 0% of media downloaded
[INFO] [2024-09-06 00:46:08] 0% of media downloaded
[INFO] [2024-09-06 00:46:08] 0% of media downloaded
[INFO] [2024-09-06 00:46:08] 0% of media downloaded
[STOP] [2024-09-06 00:47:28] map_all_nodes_to_pages
[INFO] [2024-09-06 00:47:28] 52 Unmatched nodes (of 1317)! That's too many to output. Full list in /app/public/data/ptgdp/unmatched_nodes.txt ; First 10: Canonical: Chroococcus turgidus with chytrid parasite; Node#163261798; ResourceID: 6d3e0229581823e4e4565eef76343072; Canonical: Synechoccales; Node#163262237; ResourceID: cyanobacteria-synechoccales; Canonical: Merismopediacea; Node#163262744; ResourceID: synechoccales-merismopediacea; Canonical: Siderocapsa; Node#163261915; ResourceID: 976a61ae86cac16ed1137606867d2dce; Canonical: Eucaryota; Node#163262379; ResourceID: eucaryota; Canonical: Nassulophorea; Node#163262531; ResourceID: intramacronucleata-nassulophorea; Canonical: Hymenostomatia; Node#163262587; ResourceID: oligohymenophorea-hymenostomatia; Canonical: Parameciina; Node#163262618; ResourceID: peniculida-parameciina; Canonical: Peniculiada; Node#163262590; ResourceID: oligohymenophorea-peniculiada; Canonical: Protostomatea; Node#163262536; ResourceID: intramacronucleata-protostomatea
[START] [2024-09-06 00:47:28] update_nodes
[STOP] [2024-09-06 00:47:29] update_nodes
[STOP] [2024-09-06 00:47:29] match_nodes
[START] [2024-09-06 00:47:29] reindex_search
[STOP] [2024-09-06 00:47:32] reindex_search
[START] [2024-09-06 00:47:32] normalize_units
[STOP] [2024-09-06 00:47:32] normalize_units
[START] [2024-09-06 00:47:32] calculate_statistics
[ERR] [2024-09-06 00:47:44][hdls] download_and_prep FAILED for Medium.find(29569345): 404 Not Found
[ERR] [2024-09-06 00:47:44][hdls] download_and_prep FAILED for Medium.find(29569345): 404 Not Found
[ERR] [2024-09-06 00:47:44][hdls] download_and_prep FAILED for Medium.find(29569345): 404 Not Found
[ERR] [2024-09-06 00:47:44][hdls] download_and_prep FAILED for Medium.find(29569345): 404 Not Found
[ERR] [2024-09-06 00:47:44][hdls] download_and_prep FAILED for Medium.find(29569345): 404 Not Found
[ERR] [2024-09-06 00:47:44][hdls] download_and_prep FAILED for Medium.find(29569345): 404 Not Found
[ERR] [2024-09-06 00:47:44][hdls] download_and_prep FAILED for Medium.find(29569345): 404 Not Found
[ERR] [2024-09-06 00:47:44][hdls] download_and_prep FAILED for Medium.find(29569345): 404 Not Found
[ERR] [2024-09-06 00:47:44][hdls] download_and_prep FAILED for Medium.find(29569345): 404 Not Found
[ERR] [2024-09-06 00:47:44][hdls] download_and_prep FAILED for Medium.find(29569345): 404 Not Found
[ERR] [2024-09-06 00:47:44][hdls] download_and_prep FAILED for Medium.find(29569345): 404 Not Found
[ERR] [2024-09-06 00:47:44][hdls] download_and_prep FAILED for Medium.find(29569345): 404 Not Found
[INFO] [2024-09-06 00:48:05] Duplicate page_id count: 106
[STOP] [2024-09-06 00:48:05] calculate_statistics
[START] [2024-09-06 00:48:05] complete_harvest_instance
[START] [2024-09-06 00:48:05] overall_tsv_creation
[INFO] [2024-09-06 00:48:05] Exporting 1317 nodes as TSV in batches of 10000...
[INFO] [2024-09-06 00:48:05] Processing group of 1317 in 1 batches of 10000
[INFO] [2024-09-06 00:48:21] Processed 1317/1317 nodes
[INFO] [2024-09-06 00:48:21] Average Time: 14.42
[INFO] [2024-09-06 00:48:21] Total Time: 17s
[STOP] [2024-09-06 00:48:21] overall_tsv_creation
[INFO] [2024-09-06 00:48:21] Done. Check your files:
[INFO] [2024-09-06 00:48:21] (1317 lines) /app/public/data/ptgdp/publish_nodes.tsv
[INFO] [2024-09-06 00:48:21] (8927 lines) /app/public/data/ptgdp/publish_node_ancestors.tsv
[INFO] [2024-09-06 00:48:21] (1317 lines) /app/public/data/ptgdp/publish_scientific_names.tsv
[INFO] [2024-09-06 00:48:21] (2672 lines) /app/public/data/ptgdp/publish_media.tsv
[INFO] [2024-09-06 00:48:21] (607 lines) /app/public/data/ptgdp/publish_image_info.tsv
[INFO] [2024-09-06 00:48:21] (2672 lines) /app/public/data/ptgdp/publish_attributions.tsv
[STOP] [2024-09-06 00:48:21] complete_harvest_instance
[START] [2024-09-06 00:48:21] completed
[STOP] [2024-09-06 00:48:21] completed
[STOP] [2024-09-06 00:48:21] logged process, took 170.4
[ERR] [2024-09-06 00:49:45][hdls] download_and_prep FAILED for Medium.find(29570867): 404 Not Found
[ERR] [2024-09-06 00:49:45][hdls] download_and_prep FAILED for Medium.find(29570867): 404 Not Found
[ERR] [2024-09-06 00:49:45][hdls] download_and_prep FAILED for Medium.find(29570867): 404 Not Found
[ERR] [2024-09-06 00:49:45][hdls] download_and_prep FAILED for Medium.find(29570867): 404 Not Found
[ERR] [2024-09-06 00:49:45][hdls] download_and_prep FAILED for Medium.find(29570867): 404 Not Found
[ERR] [2024-09-06 00:49:46][hdls] download_and_prep FAILED for Medium.find(29570867): 404 Not Found
[ERR] [2024-09-06 00:49:46][hdls] download_and_prep FAILED for Medium.find(29570867): 404 Not Found
[ERR] [2024-09-06 00:49:46][hdls] download_and_prep FAILED for Medium.find(29570867): 404 Not Found
[ERR] [2024-09-06 00:49:46][hdls] download_and_prep FAILED for Medium.find(29570867): 404 Not Found
[ERR] [2024-09-06 00:49:46][hdls] download_and_prep FAILED for Medium.find(29570867): 404 Not Found
[ERR] [2024-09-06 00:49:46][hdls] download_and_prep FAILED for Medium.find(29570867): 404 Not Found
[ERR] [2024-09-06 00:49:46][hdls] download_and_prep FAILED for Medium.find(29570867): 404 Not Found
[INFO] [2024-09-06 00:50:35] 100% of media downloaded
[INFO] [2024-09-06 00:50:35] 100% of media downloaded
[INFO] [2024-09-06 00:50:35] 100% of media downloaded
[INFO] [2024-09-06 00:50:35] 100% of media downloaded
[INFO] [2024-09-06 00:50:35] 100% of media downloaded
[INFO] [2024-09-06 00:50:35] 100% of media downloaded
[INFO] [2024-09-06 00:50:35] 100% of media downloaded
[ERR] [2024-09-06 00:50:35][hdls] NO additional images were found to download, NOTE THAT 2 DOWNLOADS FAILED.
[ERR] [2024-09-06 00:50:35][hdls] NO additional images were found to download, NOTE THAT 2 DOWNLOADS FAILED.
[ERR] [2024-09-06 00:50:35][hdls] NO additional images were found to download, NOTE THAT 2 DOWNLOADS FAILED.
[ERR] [2024-09-06 00:50:35][hdls] NO additional images were found to download, NOTE THAT 2 DOWNLOADS FAILED.
[ERR] [2024-09-06 00:50:35][hdls] NO additional images were found to download, NOTE THAT 2 DOWNLOADS FAILED.
[ERR] [2024-09-06 00:50:35][hdls] NO additional images were found to download, NOTE THAT 2 DOWNLOADS FAILED.
[ERR] [2024-09-06 00:50:35][hdls] NO additional images were found to download, NOTE THAT 2 DOWNLOADS FAILED.
[ERR] [2024-09-06 00:50:35][hdls] NO additional images were found to download, NOTE THAT 2 DOWNLOADS FAILED.
[ERR] [2024-09-06 00:50:35][hdls] NO additional images were found to download, NOTE THAT 2 DOWNLOADS FAILED.
[ERR] [2024-09-06 00:50:35][hdls] NO additional images were found to download, NOTE THAT 2 DOWNLOADS FAILED.
[ERR] [2024-09-06 00:50:35][hdls] NO additional images were found to download, NOTE THAT 2 DOWNLOADS FAILED.
[ERR] [2024-09-06 00:50:35][hdls] NO additional images were found to download, NOTE THAT 2 DOWNLOADS FAILED.

Latest Process