Harvest for Lukas and Clutton Brock 2014 Created 13 Oct 12:10

Harvesting Log

[INFO] [2023-10-13 12:10:15] Created harvest instance #4441
[STOP] [2023-10-13 12:10:15] create_harvest_instance
[START] [2023-10-13 12:10:15] fetch_files
[STOP] [2023-10-13 12:10:15] fetch_files
[START] [2023-10-13 12:10:15] validate_each_file
[INFO] [2023-10-13 12:10:15] Looping over 8 formats...
[INFO] [2023-10-13 12:10:15] ...agents (/app/public/data/lukas_clutton_br/agents.txt)
[INFO] [2023-10-13 12:10:15] Valid: /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_agents_30746.csv (0 lines)
[INFO] [2023-10-13 12:10:15] ...refs (/app/public/data/lukas_clutton_br/references.txt)
[INFO] [2023-10-13 12:10:15] Valid: /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_refs_30745.csv (38 lines)
[INFO] [2023-10-13 12:10:15] ...nodes (/app/public/data/lukas_clutton_br/taxa.txt)
[INFO] [2023-10-13 12:10:15] Valid: /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_nodes_30743.csv (60 lines)
[INFO] [2023-10-13 12:10:15] ...media (/app/public/data/lukas_clutton_br/media.txt)
[INFO] [2023-10-13 12:10:16] Valid: /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_media_30742.csv (0 lines)
[INFO] [2023-10-13 12:10:16] ...vernaculars (/app/public/data/lukas_clutton_br/common names.txt)
[INFO] [2023-10-13 12:10:16] Valid: /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_vernaculars_30744.csv (0 lines)
[INFO] [2023-10-13 12:10:16] ...occurrences (/app/public/data/lukas_clutton_br/occurrences.txt)
[INFO] [2023-10-13 12:10:16] Valid: /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_occurrences_30747.csv (118 lines)
[INFO] [2023-10-13 12:10:16] ...assocs (/app/public/data/lukas_clutton_br/associations.txt)
[INFO] [2023-10-13 12:10:16] Valid: /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_assocs_30749.csv (0 lines)
[INFO] [2023-10-13 12:10:16] ...measurements (/app/public/data/lukas_clutton_br/measurements or facts.txt)
[INFO] [2023-10-13 12:10:16] Valid: /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_measurements_30748.csv (744 lines)
[STOP] [2023-10-13 12:10:16] validate_each_file
[START] [2023-10-13 12:10:16] convert_to_csv
[INFO] [2023-10-13 12:10:16] Looping over 8 formats...
[INFO] [2023-10-13 12:10:16] ...agents (/app/public/data/lukas_clutton_br/agents.txt)
[CMD] [2023-10-13 12:10:16] /usr/bin/sort /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_agents_30746.csv > /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_agents_30746.csv_sorted
[INFO] [2023-10-13 12:10:16] Converted: /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_agents_30746.csv (0 lines)
[INFO] [2023-10-13 12:10:16] ...refs (/app/public/data/lukas_clutton_br/references.txt)
[CMD] [2023-10-13 12:10:16] /usr/bin/sort /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_refs_30745.csv > /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_refs_30745.csv_sorted
[INFO] [2023-10-13 12:10:16] Converted: /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_refs_30745.csv (38 lines)
[INFO] [2023-10-13 12:10:16] ...nodes (/app/public/data/lukas_clutton_br/taxa.txt)
[CMD] [2023-10-13 12:10:16] /usr/bin/sort /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_nodes_30743.csv > /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_nodes_30743.csv_sorted
[INFO] [2023-10-13 12:10:16] Converted: /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_nodes_30743.csv (60 lines)
[INFO] [2023-10-13 12:10:16] ...media (/app/public/data/lukas_clutton_br/media.txt)
[CMD] [2023-10-13 12:10:16] /usr/bin/sort /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_media_30742.csv > /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_media_30742.csv_sorted
[INFO] [2023-10-13 12:10:16] Converted: /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_media_30742.csv (0 lines)
[INFO] [2023-10-13 12:10:16] ...vernaculars (/app/public/data/lukas_clutton_br/common names.txt)
[CMD] [2023-10-13 12:10:16] /usr/bin/sort /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_vernaculars_30744.csv > /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_vernaculars_30744.csv_sorted
[INFO] [2023-10-13 12:10:16] Converted: /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_vernaculars_30744.csv (0 lines)
[INFO] [2023-10-13 12:10:16] ...occurrences (/app/public/data/lukas_clutton_br/occurrences.txt)
[CMD] [2023-10-13 12:10:16] /usr/bin/sort /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_occurrences_30747.csv > /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_occurrences_30747.csv_sorted
[INFO] [2023-10-13 12:10:16] Converted: /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_occurrences_30747.csv (118 lines)
[INFO] [2023-10-13 12:10:16] ...assocs (/app/public/data/lukas_clutton_br/associations.txt)
[CMD] [2023-10-13 12:10:16] /usr/bin/sort /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_assocs_30749.csv > /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_assocs_30749.csv_sorted
[INFO] [2023-10-13 12:10:16] Converted: /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_assocs_30749.csv (0 lines)
[INFO] [2023-10-13 12:10:16] ...measurements (/app/public/data/lukas_clutton_br/measurements or facts.txt)
[CMD] [2023-10-13 12:10:16] /usr/bin/sort /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_measurements_30748.csv > /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_measurements_30748.csv_sorted
[INFO] [2023-10-13 12:10:16] Converted: /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_measurements_30748.csv (744 lines)
[STOP] [2023-10-13 12:10:16] convert_to_csv
[START] [2023-10-13 12:10:16] calculate_delta
[INFO] [2023-10-13 12:10:16] Looping over 8 formats...
[INFO] [2023-10-13 12:10:16] ...agents (/app/public/data/lukas_clutton_br/agents.txt)
[CMD] [2023-10-13 12:10:16] echo "0a" > /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_agents_30746.diff
[CMD] [2023-10-13 12:10:16] tail -n +1 /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_agents_30746.csv >> /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_agents_30746.diff
[CMD] [2023-10-13 12:10:16] echo "." >> /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_agents_30746.diff
[INFO] [2023-10-13 12:10:17] Created diff: /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_agents_30746.diff (2 lines)
[INFO] [2023-10-13 12:10:17] ...refs (/app/public/data/lukas_clutton_br/references.txt)
[CMD] [2023-10-13 12:10:17] echo "0a" > /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_refs_30745.diff
[CMD] [2023-10-13 12:10:17] tail -n +1 /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_refs_30745.csv >> /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_refs_30745.diff
[CMD] [2023-10-13 12:10:17] echo "." >> /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_refs_30745.diff
[INFO] [2023-10-13 12:10:17] Created diff: /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_refs_30745.diff (40 lines)
[INFO] [2023-10-13 12:10:17] ...nodes (/app/public/data/lukas_clutton_br/taxa.txt)
[CMD] [2023-10-13 12:10:17] echo "0a" > /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_nodes_30743.diff
[CMD] [2023-10-13 12:10:17] tail -n +1 /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_nodes_30743.csv >> /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_nodes_30743.diff
[CMD] [2023-10-13 12:10:17] echo "." >> /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_nodes_30743.diff
[INFO] [2023-10-13 12:10:17] Created diff: /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_nodes_30743.diff (62 lines)
[INFO] [2023-10-13 12:10:17] ...media (/app/public/data/lukas_clutton_br/media.txt)
[CMD] [2023-10-13 12:10:17] echo "0a" > /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_media_30742.diff
[CMD] [2023-10-13 12:10:17] tail -n +1 /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_media_30742.csv >> /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_media_30742.diff
[CMD] [2023-10-13 12:10:17] echo "." >> /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_media_30742.diff
[INFO] [2023-10-13 12:10:17] Created diff: /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_media_30742.diff (2 lines)
[INFO] [2023-10-13 12:10:17] ...vernaculars (/app/public/data/lukas_clutton_br/common names.txt)
[CMD] [2023-10-13 12:10:17] echo "0a" > /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_vernaculars_30744.diff
[CMD] [2023-10-13 12:10:17] tail -n +1 /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_vernaculars_30744.csv >> /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_vernaculars_30744.diff
[CMD] [2023-10-13 12:10:17] echo "." >> /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_vernaculars_30744.diff
[INFO] [2023-10-13 12:10:17] Created diff: /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_vernaculars_30744.diff (2 lines)
[INFO] [2023-10-13 12:10:17] ...occurrences (/app/public/data/lukas_clutton_br/occurrences.txt)
[CMD] [2023-10-13 12:10:17] echo "0a" > /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_occurrences_30747.diff
[CMD] [2023-10-13 12:10:18] tail -n +1 /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_occurrences_30747.csv >> /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_occurrences_30747.diff
[CMD] [2023-10-13 12:10:18] echo "." >> /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_occurrences_30747.diff
[INFO] [2023-10-13 12:10:18] Created diff: /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_occurrences_30747.diff (120 lines)
[INFO] [2023-10-13 12:10:18] ...assocs (/app/public/data/lukas_clutton_br/associations.txt)
[CMD] [2023-10-13 12:10:18] echo "0a" > /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_assocs_30749.diff
[CMD] [2023-10-13 12:10:18] tail -n +1 /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_assocs_30749.csv >> /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_assocs_30749.diff
[CMD] [2023-10-13 12:10:18] echo "." >> /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_assocs_30749.diff
[INFO] [2023-10-13 12:10:18] Created diff: /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_assocs_30749.diff (2 lines)
[INFO] [2023-10-13 12:10:18] ...measurements (/app/public/data/lukas_clutton_br/measurements or facts.txt)
[CMD] [2023-10-13 12:10:18] echo "0a" > /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_measurements_30748.diff
[CMD] [2023-10-13 12:10:18] tail -n +1 /app/public/data/lukas_clutton_br/converted_csv/lukas_clutton_br_measurements_30748.csv >> /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_measurements_30748.diff
[CMD] [2023-10-13 12:10:18] echo "." >> /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_measurements_30748.diff
[INFO] [2023-10-13 12:10:18] Created diff: /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_measurements_30748.diff (746 lines)
[STOP] [2023-10-13 12:10:18] calculate_delta
[START] [2023-10-13 12:10:18] parse_diff_and_store
[INFO] [2023-10-13 12:10:18] Handling diff: /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_agents_30746.diff (2 lines)
[INFO] [2023-10-13 12:10:18] Loading agents diff file into memory (2 lines)...
[INFO] [2023-10-13 12:10:18] Handling diff: /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_refs_30745.diff (40 lines)
[INFO] [2023-10-13 12:10:18] Loading refs diff file into memory (40 lines)...
[INFO] [2023-10-13 12:10:18] Storing 38 References (38/38/40)
[INFO] [2023-10-13 12:10:18] Handling diff: /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_nodes_30743.diff (62 lines)
[INFO] [2023-10-13 12:10:18] Loading nodes diff file into memory (62 lines)...
[INFO] [2023-10-13 12:10:18] Storing 111 ScientificNames (222/60/62)
[INFO] [2023-10-13 12:10:18] Storing 111 Nodes (222/60/62)
[INFO] [2023-10-13 12:10:19] Handling diff: /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_media_30742.diff (2 lines)
[INFO] [2023-10-13 12:10:19] Loading media diff file into memory (2 lines)...
[INFO] [2023-10-13 12:10:19] Handling diff: /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_vernaculars_30744.diff (2 lines)
[INFO] [2023-10-13 12:10:19] Loading vernaculars diff file into memory (2 lines)...
[INFO] [2023-10-13 12:10:19] Handling diff: /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_occurrences_30747.diff (120 lines)
[INFO] [2023-10-13 12:10:19] Loading occurrences diff file into memory (120 lines)...
[INFO] [2023-10-13 12:10:19] Storing 118 Occurrences (176/118/120)
[INFO] [2023-10-13 12:10:19] Storing 58 OccurrenceMetadata (176/118/120)
[INFO] [2023-10-13 12:10:19] Handling diff: /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_assocs_30749.diff (2 lines)
[INFO] [2023-10-13 12:10:19] Loading assocs diff file into memory (2 lines)...
[INFO] [2023-10-13 12:10:19] Handling diff: /app/public/data/lukas_clutton_br/diff/lukas_clutton_br_measurements_30748.diff (746 lines)
[INFO] [2023-10-13 12:10:19] Loading measurements diff file into memory (746 lines)...
[INFO] [2023-10-13 12:10:20] Storing 744 Traits (2581/744/746)
[INFO] [2023-10-13 12:10:20] Storing 1777 MetaTraits (2581/744/746)
[INFO] [2023-10-13 12:10:20] Storing 60 TraitsReferences (2581/744/746)
[STOP] [2023-10-13 12:10:20] parse_diff_and_store
[START] [2023-10-13 12:10:20] resolve_keys
[2023-10-13 12:10:20] Resolving downloaded urls (this is not actually downloading them yet)
[INFO] [2023-10-13 12:10:28] Occurrences to nodes (through scientific_names)...
[INFO] [2023-10-13 12:10:28] traits to occurrences...
[INFO] [2023-10-13 12:10:28] traits to nodes (through occurrences)...
[INFO] [2023-10-13 12:10:28] Traits to sex term...
[INFO] [2023-10-13 12:10:28] Traits to lifestage term...
[INFO] [2023-10-13 12:10:28] MetaTraits to traits...
[INFO] [2023-10-13 12:10:28] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2023-10-13 12:10:28] Assocs to occurrences...
[INFO] [2023-10-13 12:10:28] Assocs to nodes...
[INFO] [2023-10-13 12:10:28] Assoc to sex term...
[INFO] [2023-10-13 12:10:28] Assoc to lifestage term...
[INFO] [2023-10-13 12:10:28] MetaAssoc to assocs...
[STOP] [2023-10-13 12:10:28] resolve_keys
[START] [2023-10-13 12:10:28] hold_for_later_1
[STOP] [2023-10-13 12:10:28] hold_for_later_1
[START] [2023-10-13 12:10:28] hold_for_later_2
[STOP] [2023-10-13 12:10:28] hold_for_later_2
[START] [2023-10-13 12:10:28] resolve_missing_parents
[STOP] [2023-10-13 12:10:28] resolve_missing_parents
[START] [2023-10-13 12:10:28] rebuild_nodes
[START] [2023-10-13 12:10:28] Flattener#flatten
[START] [2023-10-13 12:10:28] Flattener#study_resource
[START] [2023-10-13 12:10:28] Flattener#build_ancestry
[STOP] [2023-10-13 12:10:28] Flattener#build_ancestry
[INFO] [2023-10-13 12:10:28] 111 ancestry keys
[START] [2023-10-13 12:10:28] build_node_ancestors
[INFO] [2023-10-13 12:10:28] old ancestors deleted.
[STOP] [2023-10-13 12:10:28] build_node_ancestors
[START] [2023-10-13 12:10:28] Flattener#propagate_ancestor_ids
[STOP] [2023-10-13 12:10:28] Flattener#propagate_ancestor_ids
[STOP] [2023-10-13 12:10:28] Flattener#flatten
[STOP] [2023-10-13 12:10:28] rebuild_nodes
[START] [2023-10-13 12:10:28] resolve_missing_media_owners
[STOP] [2023-10-13 12:10:28] resolve_missing_media_owners
[START] [2023-10-13 12:10:28] sanitize_media_verbatims
[STOP] [2023-10-13 12:10:28] sanitize_media_verbatims
[START] [2023-10-13 12:10:28] queue_downloads
[STOP] [2023-10-13 12:10:28] queue_downloads
[START] [2023-10-13 12:10:28] parse_names
[WARN] [2023-10-13 12:10:28] I see 111 names which still need to be parsed.
[WARN] [2023-10-13 12:10:28] Names to parse: 111 formatted: 111 learned: 111 parsed: 111
[STOP] [2023-10-13 12:10:29] parse_names
[START] [2023-10-13 12:10:30] denormalize_canonical_names_to_nodes
[STOP] [2023-10-13 12:10:30] denormalize_canonical_names_to_nodes
[START] [2023-10-13 12:10:30] match_nodes
[START] [2023-10-13 12:10:30] map_all_nodes_to_pages
[STOP] [2023-10-13 12:10:33] map_all_nodes_to_pages
[INFO] [2023-10-13 12:10:33] ZERO unmatched nodes (of 111)! Nicely done.
[START] [2023-10-13 12:10:33] update_nodes
[STOP] [2023-10-13 12:10:33] update_nodes
[STOP] [2023-10-13 12:10:33] match_nodes
[START] [2023-10-13 12:10:33] reindex_search
[STOP] [2023-10-13 12:10:33] reindex_search
[START] [2023-10-13 12:10:33] normalize_units
[STOP] [2023-10-13 12:10:35] normalize_units
[START] [2023-10-13 12:10:35] calculate_statistics
[INFO] [2023-10-13 12:10:37] Duplicate page_id count: 0
[STOP] [2023-10-13 12:10:37] calculate_statistics
[START] [2023-10-13 12:10:37] complete_harvest_instance
[START] [2023-10-13 12:10:37] overall_tsv_creation
[INFO] [2023-10-13 12:10:37] Exporting 111 nodes as TSV in batches of 10000...
[INFO] [2023-10-13 12:10:37] Processing group of 111 in 1 batches of 10000
[INFO] [2023-10-13 12:10:37] 744 Traits (unfiltered) and 0 associations...
[INFO] [2023-10-13 12:10:37] Building Traits map for 111 nodes (this can take a while)...
[INFO] [2023-10-13 12:10:37] Mapped 744 traits (1777 meta) for 111 nodes.
[INFO] [2023-10-13 12:10:37] Building Associations map (this can take a while)...
[INFO] [2023-10-13 12:10:37] Done. 0 assocs mapped (0 meta).
[INFO] [2023-10-13 12:10:37] Adding 744 traits...
[INFO] [2023-10-13 12:10:37] 60 metadata added.
[INFO] [2023-10-13 12:10:37] Adding 0 assocs...
[INFO] [2023-10-13 12:10:37] 0 metadata added.
[INFO] [2023-10-13 12:11:21] Processed 111/111 nodes
[INFO] [2023-10-13 12:11:21] Average Time: 44.65
[INFO] [2023-10-13 12:11:21] Total Time: 45s
[STOP] [2023-10-13 12:11:21] overall_tsv_creation
[INFO] [2023-10-13 12:11:21] Done. Check your files:
[INFO] [2023-10-13 12:11:21] (111 lines) /app/public/data/lukas_clutton_br/publish_nodes.tsv
[INFO] [2023-10-13 12:11:21] (272 lines) /app/public/data/lukas_clutton_br/publish_node_ancestors.tsv
[INFO] [2023-10-13 12:11:22] (111 lines) /app/public/data/lukas_clutton_br/publish_scientific_names.tsv
[INFO] [2023-10-13 12:11:22] (745 lines) /app/public/data/lukas_clutton_br/publish_traits.tsv
[INFO] [2023-10-13 12:11:22] (61 lines) /app/public/data/lukas_clutton_br/publish_metadata.tsv
[STOP] [2023-10-13 12:11:22] complete_harvest_instance
[START] [2023-10-13 12:11:22] completed
[STOP] [2023-10-13 12:11:22] completed
[STOP] [2023-10-13 12:11:22] logged process, took 66.87

Latest Process

13 Oct 12:10 EXITED currently processing 0