Harvest for Alava and Aguirre 2005 Created 03 Aug 16:37

Stage: completed
Fetched: 03 Aug 16:37
Validated: 03 Aug 16:37
Deltas Created 03 Aug 16:37
Units Normalized: 03 Aug 16:38
Ancestry Built: 03 Aug 16:38
Nodes Matched: 03 Aug 16:38
Names Parsed: 03 Aug 16:38
New Models Stored: 03 Aug 16:37
Indexed: 03 Aug 16:38
Completed: 03 Aug 16:40
Time to Harvest: less than a minute
Harvesting Log

(142 lines)
[INFO] [2022-08-03 16:37:53] Created harvest instance #4175
[STOP] [2022-08-03 16:37:53] create_harvest_instance
[START] [2022-08-03 16:37:53] fetch_files
[STOP] [2022-08-03 16:37:53] fetch_files
[START] [2022-08-03 16:37:53] validate_each_file
[INFO] [2022-08-03 16:37:53] Looping over 3 formats...
[INFO] [2022-08-03 16:37:53] ...nodes (/app/public/data/alava_aguirre_al/taxa.txt)
[INFO] [2022-08-03 16:37:53] Valid: /app/public/data/alava_aguirre_al/converted_csv/alava_aguirre_al_nodes_29591.csv (1 lines)
[INFO] [2022-08-03 16:37:53] ...occurrences (/app/public/data/alava_aguirre_al/occurrences.txt)
[INFO] [2022-08-03 16:37:53] Valid: /app/public/data/alava_aguirre_al/converted_csv/alava_aguirre_al_occurrences_29592.csv (1 lines)
[INFO] [2022-08-03 16:37:53] ...measurements (/app/public/data/alava_aguirre_al/measurementOrFact.txt)
[INFO] [2022-08-03 16:37:53] Valid: /app/public/data/alava_aguirre_al/converted_csv/alava_aguirre_al_measurements_29593.csv (2 lines)
[STOP] [2022-08-03 16:37:53] validate_each_file
[START] [2022-08-03 16:37:53] convert_to_csv
[INFO] [2022-08-03 16:37:53] Looping over 3 formats...
[INFO] [2022-08-03 16:37:53] ...nodes (/app/public/data/alava_aguirre_al/taxa.txt)
[CMD] [2022-08-03 16:37:53] /usr/bin/sort /app/public/data/alava_aguirre_al/converted_csv/alava_aguirre_al_nodes_29591.csv > /app/public/data/alava_aguirre_al/converted_csv/alava_aguirre_al_nodes_29591.csv_sorted
[INFO] [2022-08-03 16:37:53] Converted: /app/public/data/alava_aguirre_al/converted_csv/alava_aguirre_al_nodes_29591.csv (1 lines)
[INFO] [2022-08-03 16:37:53] ...occurrences (/app/public/data/alava_aguirre_al/occurrences.txt)
[CMD] [2022-08-03 16:37:53] /usr/bin/sort /app/public/data/alava_aguirre_al/converted_csv/alava_aguirre_al_occurrences_29592.csv > /app/public/data/alava_aguirre_al/converted_csv/alava_aguirre_al_occurrences_29592.csv_sorted
[INFO] [2022-08-03 16:37:53] Converted: /app/public/data/alava_aguirre_al/converted_csv/alava_aguirre_al_occurrences_29592.csv (1 lines)
[INFO] [2022-08-03 16:37:53] ...measurements (/app/public/data/alava_aguirre_al/measurementOrFact.txt)
[CMD] [2022-08-03 16:37:53] /usr/bin/sort /app/public/data/alava_aguirre_al/converted_csv/alava_aguirre_al_measurements_29593.csv > /app/public/data/alava_aguirre_al/converted_csv/alava_aguirre_al_measurements_29593.csv_sorted
[INFO] [2022-08-03 16:37:53] Converted: /app/public/data/alava_aguirre_al/converted_csv/alava_aguirre_al_measurements_29593.csv (2 lines)
[STOP] [2022-08-03 16:37:53] convert_to_csv
[START] [2022-08-03 16:37:53] calculate_delta
[INFO] [2022-08-03 16:37:53] Looping over 3 formats...
[INFO] [2022-08-03 16:37:53] ...nodes (/app/public/data/alava_aguirre_al/taxa.txt)
[CMD] [2022-08-03 16:37:53] echo "0a" > /app/public/data/alava_aguirre_al/diff/alava_aguirre_al_nodes_29591.diff
[CMD] [2022-08-03 16:37:53] tail -n +1 /app/public/data/alava_aguirre_al/converted_csv/alava_aguirre_al_nodes_29591.csv >> /app/public/data/alava_aguirre_al/diff/alava_aguirre_al_nodes_29591.diff
[CMD] [2022-08-03 16:37:53] echo "." >> /app/public/data/alava_aguirre_al/diff/alava_aguirre_al_nodes_29591.diff
[INFO] [2022-08-03 16:37:53] Created diff: /app/public/data/alava_aguirre_al/diff/alava_aguirre_al_nodes_29591.diff (3 lines)
[INFO] [2022-08-03 16:37:53] ...occurrences (/app/public/data/alava_aguirre_al/occurrences.txt)
[CMD] [2022-08-03 16:37:53] echo "0a" > /app/public/data/alava_aguirre_al/diff/alava_aguirre_al_occurrences_29592.diff
[CMD] [2022-08-03 16:37:53] tail -n +1 /app/public/data/alava_aguirre_al/converted_csv/alava_aguirre_al_occurrences_29592.csv >> /app/public/data/alava_aguirre_al/diff/alava_aguirre_al_occurrences_29592.diff
[CMD] [2022-08-03 16:37:53] echo "." >> /app/public/data/alava_aguirre_al/diff/alava_aguirre_al_occurrences_29592.diff
[INFO] [2022-08-03 16:37:53] Created diff: /app/public/data/alava_aguirre_al/diff/alava_aguirre_al_occurrences_29592.diff (3 lines)
[INFO] [2022-08-03 16:37:53] ...measurements (/app/public/data/alava_aguirre_al/measurementOrFact.txt)
[CMD] [2022-08-03 16:37:53] echo "0a" > /app/public/data/alava_aguirre_al/diff/alava_aguirre_al_measurements_29593.diff
[CMD] [2022-08-03 16:37:53] tail -n +1 /app/public/data/alava_aguirre_al/converted_csv/alava_aguirre_al_measurements_29593.csv >> /app/public/data/alava_aguirre_al/diff/alava_aguirre_al_measurements_29593.diff
[CMD] [2022-08-03 16:37:53] echo "." >> /app/public/data/alava_aguirre_al/diff/alava_aguirre_al_measurements_29593.diff
[INFO] [2022-08-03 16:37:53] Created diff: /app/public/data/alava_aguirre_al/diff/alava_aguirre_al_measurements_29593.diff (4 lines)
[STOP] [2022-08-03 16:37:53] calculate_delta
[START] [2022-08-03 16:37:53] parse_diff_and_store
[INFO] [2022-08-03 16:37:53] Handling diff: /app/public/data/alava_aguirre_al/diff/alava_aguirre_al_nodes_29591.diff (3 lines)
[INFO] [2022-08-03 16:37:53] Loading nodes diff file into memory (3 lines)...
[INFO] [2022-08-03 16:37:53] Storing 1 ScientificNames (2/1/3)
[INFO] [2022-08-03 16:37:53] Storing 1 Nodes (2/1/3)
[INFO] [2022-08-03 16:37:53] Handling diff: /app/public/data/alava_aguirre_al/diff/alava_aguirre_al_occurrences_29592.diff (3 lines)
[INFO] [2022-08-03 16:37:53] Loading occurrences diff file into memory (3 lines)...
[INFO] [2022-08-03 16:37:53] Storing 1 Occurrences (1/1/3)
[INFO] [2022-08-03 16:37:54] Handling diff: /app/public/data/alava_aguirre_al/diff/alava_aguirre_al_measurements_29593.diff (4 lines)
[INFO] [2022-08-03 16:37:54] Loading measurements diff file into memory (4 lines)...
[INFO] [2022-08-03 16:37:54] Storing 2 Traits (3/2/4)
[INFO] [2022-08-03 16:37:54] Storing 1 MetaTraits (3/2/4)
[STOP] [2022-08-03 16:37:54] parse_diff_and_store
[START] [2022-08-03 16:37:54] resolve_keys
[2022-08-03 16:37:54] Resolving downloaded urls (this is not actually downloading them yet)
[INFO] [2022-08-03 16:38:01] Occurrences to nodes (through scientific_names)...
[INFO] [2022-08-03 16:38:01] traits to occurrences...
[INFO] [2022-08-03 16:38:01] traits to nodes (through occurrences)...
[INFO] [2022-08-03 16:38:01] Traits to sex term...
[INFO] [2022-08-03 16:38:01] Traits to lifestage term...
[INFO] [2022-08-03 16:38:01] MetaTraits to traits...
[INFO] [2022-08-03 16:38:01] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2022-08-03 16:38:01] Assocs to occurrences...
[INFO] [2022-08-03 16:38:01] Assocs to nodes...
[INFO] [2022-08-03 16:38:01] Assoc to sex term...
[INFO] [2022-08-03 16:38:01] Assoc to lifestage term...
[INFO] [2022-08-03 16:38:01] MetaAssoc to assocs...
[STOP] [2022-08-03 16:38:01] resolve_keys
[START] [2022-08-03 16:38:01] hold_for_later_1
[STOP] [2022-08-03 16:38:01] hold_for_later_1
[START] [2022-08-03 16:38:01] hold_for_later_2
[STOP] [2022-08-03 16:38:01] hold_for_later_2
[START] [2022-08-03 16:38:01] resolve_missing_parents
[STOP] [2022-08-03 16:38:01] resolve_missing_parents
[START] [2022-08-03 16:38:01] rebuild_nodes
[START] [2022-08-03 16:38:01] Flattener#flatten
[START] [2022-08-03 16:38:01] Flattener#study_resource
[START] [2022-08-03 16:38:01] Flattener#build_ancestry
[STOP] [2022-08-03 16:38:01] Flattener#build_ancestry
[INFO] [2022-08-03 16:38:01] 1 ancestry keys
[START] [2022-08-03 16:38:01] build_node_ancestors
[INFO] [2022-08-03 16:38:01] old ancestors deleted.
[STOP] [2022-08-03 16:38:01] build_node_ancestors
[WARN] [2022-08-03 16:38:01] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2022-08-03 16:38:01] Flattener#flatten
[STOP] [2022-08-03 16:38:01] rebuild_nodes
[START] [2022-08-03 16:38:01] resolve_missing_media_owners
[STOP] [2022-08-03 16:38:01] resolve_missing_media_owners
[START] [2022-08-03 16:38:01] sanitize_media_verbatims
[STOP] [2022-08-03 16:38:01] sanitize_media_verbatims
[START] [2022-08-03 16:38:01] queue_downloads
[STOP] [2022-08-03 16:38:01] queue_downloads
[START] [2022-08-03 16:38:01] parse_names
[WARN] [2022-08-03 16:38:01] I see 1 names which still need to be parsed.
[WARN] [2022-08-03 16:38:01] Names to parse: 1 formatted: 1 learned: 1 parsed: 1
[STOP] [2022-08-03 16:38:02] parse_names
[START] [2022-08-03 16:38:02] denormalize_canonical_names_to_nodes
[STOP] [2022-08-03 16:38:02] denormalize_canonical_names_to_nodes
[START] [2022-08-03 16:38:02] match_nodes
[START] [2022-08-03 16:38:02] map_all_nodes_to_pages
[STOP] [2022-08-03 16:38:02] map_all_nodes_to_pages
[INFO] [2022-08-03 16:38:02] ZERO unmatched nodes (of 1)! Nicely done.
[START] [2022-08-03 16:38:02] update_nodes
[STOP] [2022-08-03 16:38:02] update_nodes
[STOP] [2022-08-03 16:38:02] match_nodes
[START] [2022-08-03 16:38:02] reindex_search
[STOP] [2022-08-03 16:38:02] reindex_search
[START] [2022-08-03 16:38:02] normalize_units
[STOP] [2022-08-03 16:38:02] normalize_units
[START] [2022-08-03 16:38:02] calculate_statistics
[2022-08-03 16:38:02] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[INFO] [2022-08-03 16:38:02] Duplicate page_id count: 0
[STOP] [2022-08-03 16:38:02] calculate_statistics
[START] [2022-08-03 16:38:02] complete_harvest_instance
[START] [2022-08-03 16:38:02] overall_tsv_creation
[INFO] [2022-08-03 16:38:02] Exporting 1 nodes as TSV in batches of 10000...
[INFO] [2022-08-03 16:38:02] Processing group of 1 in 1 batches of 10000
[INFO] [2022-08-03 16:38:44] 1 Traits (unfiltered) and 0 associations...
[INFO] [2022-08-03 16:38:44] Building Traits map for 1 nodes (this can take a while)...
[INFO] [2022-08-03 16:39:36] Mapped 1 traits (1 meta) for 1 nodes.
[INFO] [2022-08-03 16:39:36] Building Associations map (this can take a while)...
[INFO] [2022-08-03 16:39:36] Done. 0 assocs mapped (0 meta).
[INFO] [2022-08-03 16:39:36] Adding 1 traits...
[INFO] [2022-08-03 16:39:36] 1 metadata added.
[INFO] [2022-08-03 16:39:36] Adding 0 assocs...
[INFO] [2022-08-03 16:39:36] 0 metadata added.
[INFO] [2022-08-03 16:40:19] Processed 1/1 nodes
[INFO] [2022-08-03 16:40:19] Average Time: 111.28
[INFO] [2022-08-03 16:40:19] Total Time: 2m17s
[STOP] [2022-08-03 16:40:19] overall_tsv_creation
[INFO] [2022-08-03 16:40:19] Done. Check your files:
[INFO] [2022-08-03 16:40:19] (1 lines) /app/public/data/alava_aguirre_al/publish_nodes.tsv
[INFO] [2022-08-03 16:40:19] (1 lines) /app/public/data/alava_aguirre_al/publish_scientific_names.tsv
[INFO] [2022-08-03 16:40:19] (2 lines) /app/public/data/alava_aguirre_al/publish_traits.tsv
[INFO] [2022-08-03 16:40:19] (2 lines) /app/public/data/alava_aguirre_al/publish_metadata.tsv
[STOP] [2022-08-03 16:40:19] complete_harvest_instance
[START] [2022-08-03 16:40:19] completed
[STOP] [2022-08-03 16:40:19] completed
[STOP] [2022-08-03 16:40:19] logged process, took 145.63
Latest Process

03 Aug 16:37 EXITED currently processing 0