Harvest for Yoon et al 2006 Created 03 Aug 16:42

Stage: completed
Fetched: 03 Aug 16:42
Validated: 03 Aug 16:42
Deltas Created 03 Aug 16:42
Units Normalized: 03 Aug 16:42
Ancestry Built: 03 Aug 16:42
Nodes Matched: 03 Aug 16:42
Names Parsed: 03 Aug 16:42
New Models Stored: 03 Aug 16:42
Indexed: 03 Aug 16:42
Completed: 03 Aug 16:45
Time to Harvest: less than a minute
Harvesting Log

(142 lines)
[INFO] [2022-08-03 16:42:50] Created harvest instance #4177
[STOP] [2022-08-03 16:42:50] create_harvest_instance
[START] [2022-08-03 16:42:50] fetch_files
[STOP] [2022-08-03 16:42:50] fetch_files
[START] [2022-08-03 16:42:50] validate_each_file
[INFO] [2022-08-03 16:42:50] Looping over 3 formats...
[INFO] [2022-08-03 16:42:50] ...nodes (/app/public/data/yoon_et_al_yoon_/taxa.txt)
[INFO] [2022-08-03 16:42:50] Valid: /app/public/data/yoon_et_al_yoon_/converted_csv/yoon_et_al_yoon__nodes_29597.csv (9 lines)
[INFO] [2022-08-03 16:42:50] ...occurrences (/app/public/data/yoon_et_al_yoon_/occurrences.txt)
[INFO] [2022-08-03 16:42:50] Valid: /app/public/data/yoon_et_al_yoon_/converted_csv/yoon_et_al_yoon__occurrences_29598.csv (9 lines)
[INFO] [2022-08-03 16:42:50] ...measurements (/app/public/data/yoon_et_al_yoon_/measurementOrFact.txt)
[INFO] [2022-08-03 16:42:50] Valid: /app/public/data/yoon_et_al_yoon_/converted_csv/yoon_et_al_yoon__measurements_29599.csv (18 lines)
[STOP] [2022-08-03 16:42:50] validate_each_file
[START] [2022-08-03 16:42:50] convert_to_csv
[INFO] [2022-08-03 16:42:50] Looping over 3 formats...
[INFO] [2022-08-03 16:42:50] ...nodes (/app/public/data/yoon_et_al_yoon_/taxa.txt)
[CMD] [2022-08-03 16:42:50] /usr/bin/sort /app/public/data/yoon_et_al_yoon_/converted_csv/yoon_et_al_yoon__nodes_29597.csv > /app/public/data/yoon_et_al_yoon_/converted_csv/yoon_et_al_yoon__nodes_29597.csv_sorted
[INFO] [2022-08-03 16:42:50] Converted: /app/public/data/yoon_et_al_yoon_/converted_csv/yoon_et_al_yoon__nodes_29597.csv (9 lines)
[INFO] [2022-08-03 16:42:50] ...occurrences (/app/public/data/yoon_et_al_yoon_/occurrences.txt)
[CMD] [2022-08-03 16:42:50] /usr/bin/sort /app/public/data/yoon_et_al_yoon_/converted_csv/yoon_et_al_yoon__occurrences_29598.csv > /app/public/data/yoon_et_al_yoon_/converted_csv/yoon_et_al_yoon__occurrences_29598.csv_sorted
[INFO] [2022-08-03 16:42:50] Converted: /app/public/data/yoon_et_al_yoon_/converted_csv/yoon_et_al_yoon__occurrences_29598.csv (9 lines)
[INFO] [2022-08-03 16:42:50] ...measurements (/app/public/data/yoon_et_al_yoon_/measurementOrFact.txt)
[CMD] [2022-08-03 16:42:50] /usr/bin/sort /app/public/data/yoon_et_al_yoon_/converted_csv/yoon_et_al_yoon__measurements_29599.csv > /app/public/data/yoon_et_al_yoon_/converted_csv/yoon_et_al_yoon__measurements_29599.csv_sorted
[INFO] [2022-08-03 16:42:50] Converted: /app/public/data/yoon_et_al_yoon_/converted_csv/yoon_et_al_yoon__measurements_29599.csv (18 lines)
[STOP] [2022-08-03 16:42:50] convert_to_csv
[START] [2022-08-03 16:42:50] calculate_delta
[INFO] [2022-08-03 16:42:50] Looping over 3 formats...
[INFO] [2022-08-03 16:42:50] ...nodes (/app/public/data/yoon_et_al_yoon_/taxa.txt)
[CMD] [2022-08-03 16:42:50] echo "0a" > /app/public/data/yoon_et_al_yoon_/diff/yoon_et_al_yoon__nodes_29597.diff
[CMD] [2022-08-03 16:42:51] tail -n +1 /app/public/data/yoon_et_al_yoon_/converted_csv/yoon_et_al_yoon__nodes_29597.csv >> /app/public/data/yoon_et_al_yoon_/diff/yoon_et_al_yoon__nodes_29597.diff
[CMD] [2022-08-03 16:42:51] echo "." >> /app/public/data/yoon_et_al_yoon_/diff/yoon_et_al_yoon__nodes_29597.diff
[INFO] [2022-08-03 16:42:51] Created diff: /app/public/data/yoon_et_al_yoon_/diff/yoon_et_al_yoon__nodes_29597.diff (11 lines)
[INFO] [2022-08-03 16:42:51] ...occurrences (/app/public/data/yoon_et_al_yoon_/occurrences.txt)
[CMD] [2022-08-03 16:42:51] echo "0a" > /app/public/data/yoon_et_al_yoon_/diff/yoon_et_al_yoon__occurrences_29598.diff
[CMD] [2022-08-03 16:42:51] tail -n +1 /app/public/data/yoon_et_al_yoon_/converted_csv/yoon_et_al_yoon__occurrences_29598.csv >> /app/public/data/yoon_et_al_yoon_/diff/yoon_et_al_yoon__occurrences_29598.diff
[CMD] [2022-08-03 16:42:51] echo "." >> /app/public/data/yoon_et_al_yoon_/diff/yoon_et_al_yoon__occurrences_29598.diff
[INFO] [2022-08-03 16:42:51] Created diff: /app/public/data/yoon_et_al_yoon_/diff/yoon_et_al_yoon__occurrences_29598.diff (11 lines)
[INFO] [2022-08-03 16:42:51] ...measurements (/app/public/data/yoon_et_al_yoon_/measurementOrFact.txt)
[CMD] [2022-08-03 16:42:51] echo "0a" > /app/public/data/yoon_et_al_yoon_/diff/yoon_et_al_yoon__measurements_29599.diff
[CMD] [2022-08-03 16:42:51] tail -n +1 /app/public/data/yoon_et_al_yoon_/converted_csv/yoon_et_al_yoon__measurements_29599.csv >> /app/public/data/yoon_et_al_yoon_/diff/yoon_et_al_yoon__measurements_29599.diff
[CMD] [2022-08-03 16:42:51] echo "." >> /app/public/data/yoon_et_al_yoon_/diff/yoon_et_al_yoon__measurements_29599.diff
[INFO] [2022-08-03 16:42:51] Created diff: /app/public/data/yoon_et_al_yoon_/diff/yoon_et_al_yoon__measurements_29599.diff (20 lines)
[STOP] [2022-08-03 16:42:51] calculate_delta
[START] [2022-08-03 16:42:51] parse_diff_and_store
[INFO] [2022-08-03 16:42:51] Handling diff: /app/public/data/yoon_et_al_yoon_/diff/yoon_et_al_yoon__nodes_29597.diff (11 lines)
[INFO] [2022-08-03 16:42:51] Loading nodes diff file into memory (11 lines)...
[INFO] [2022-08-03 16:42:51] Storing 9 ScientificNames (18/9/11)
[INFO] [2022-08-03 16:42:51] Storing 9 Nodes (18/9/11)
[INFO] [2022-08-03 16:42:51] Handling diff: /app/public/data/yoon_et_al_yoon_/diff/yoon_et_al_yoon__occurrences_29598.diff (11 lines)
[INFO] [2022-08-03 16:42:51] Loading occurrences diff file into memory (11 lines)...
[INFO] [2022-08-03 16:42:51] Storing 9 Occurrences (9/9/11)
[INFO] [2022-08-03 16:42:51] Handling diff: /app/public/data/yoon_et_al_yoon_/diff/yoon_et_al_yoon__measurements_29599.diff (20 lines)
[INFO] [2022-08-03 16:42:51] Loading measurements diff file into memory (20 lines)...
[INFO] [2022-08-03 16:42:51] Storing 18 Traits (27/18/20)
[INFO] [2022-08-03 16:42:51] Storing 9 MetaTraits (27/18/20)
[STOP] [2022-08-03 16:42:51] parse_diff_and_store
[START] [2022-08-03 16:42:51] resolve_keys
[2022-08-03 16:42:51] Resolving downloaded urls (this is not actually downloading them yet)
[INFO] [2022-08-03 16:42:58] Occurrences to nodes (through scientific_names)...
[INFO] [2022-08-03 16:42:58] traits to occurrences...
[INFO] [2022-08-03 16:42:58] traits to nodes (through occurrences)...
[INFO] [2022-08-03 16:42:58] Traits to sex term...
[INFO] [2022-08-03 16:42:58] Traits to lifestage term...
[INFO] [2022-08-03 16:42:58] MetaTraits to traits...
[INFO] [2022-08-03 16:42:58] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2022-08-03 16:42:58] Assocs to occurrences...
[INFO] [2022-08-03 16:42:58] Assocs to nodes...
[INFO] [2022-08-03 16:42:58] Assoc to sex term...
[INFO] [2022-08-03 16:42:58] Assoc to lifestage term...
[INFO] [2022-08-03 16:42:58] MetaAssoc to assocs...
[STOP] [2022-08-03 16:42:58] resolve_keys
[START] [2022-08-03 16:42:58] hold_for_later_1
[STOP] [2022-08-03 16:42:58] hold_for_later_1
[START] [2022-08-03 16:42:58] hold_for_later_2
[STOP] [2022-08-03 16:42:58] hold_for_later_2
[START] [2022-08-03 16:42:58] resolve_missing_parents
[STOP] [2022-08-03 16:42:58] resolve_missing_parents
[START] [2022-08-03 16:42:58] rebuild_nodes
[START] [2022-08-03 16:42:58] Flattener#flatten
[START] [2022-08-03 16:42:58] Flattener#study_resource
[START] [2022-08-03 16:42:58] Flattener#build_ancestry
[STOP] [2022-08-03 16:42:58] Flattener#build_ancestry
[INFO] [2022-08-03 16:42:58] 9 ancestry keys
[START] [2022-08-03 16:42:58] build_node_ancestors
[INFO] [2022-08-03 16:42:58] old ancestors deleted.
[STOP] [2022-08-03 16:42:58] build_node_ancestors
[WARN] [2022-08-03 16:42:58] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2022-08-03 16:42:58] Flattener#flatten
[STOP] [2022-08-03 16:42:58] rebuild_nodes
[START] [2022-08-03 16:42:58] resolve_missing_media_owners
[STOP] [2022-08-03 16:42:58] resolve_missing_media_owners
[START] [2022-08-03 16:42:58] sanitize_media_verbatims
[STOP] [2022-08-03 16:42:58] sanitize_media_verbatims
[START] [2022-08-03 16:42:58] queue_downloads
[STOP] [2022-08-03 16:42:58] queue_downloads
[START] [2022-08-03 16:42:58] parse_names
[WARN] [2022-08-03 16:42:58] I see 9 names which still need to be parsed.
[WARN] [2022-08-03 16:42:58] Names to parse: 9 formatted: 9 learned: 9 parsed: 9
[STOP] [2022-08-03 16:42:59] parse_names
[START] [2022-08-03 16:42:59] denormalize_canonical_names_to_nodes
[STOP] [2022-08-03 16:42:59] denormalize_canonical_names_to_nodes
[START] [2022-08-03 16:42:59] match_nodes
[START] [2022-08-03 16:42:59] map_all_nodes_to_pages
[STOP] [2022-08-03 16:42:59] map_all_nodes_to_pages
[INFO] [2022-08-03 16:42:59] ZERO unmatched nodes (of 9)! Nicely done.
[START] [2022-08-03 16:42:59] update_nodes
[STOP] [2022-08-03 16:42:59] update_nodes
[STOP] [2022-08-03 16:42:59] match_nodes
[START] [2022-08-03 16:42:59] reindex_search
[STOP] [2022-08-03 16:42:59] reindex_search
[START] [2022-08-03 16:42:59] normalize_units
[STOP] [2022-08-03 16:42:59] normalize_units
[START] [2022-08-03 16:42:59] calculate_statistics
[2022-08-03 16:42:59] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[INFO] [2022-08-03 16:42:59] Duplicate page_id count: 0
[STOP] [2022-08-03 16:42:59] calculate_statistics
[START] [2022-08-03 16:42:59] complete_harvest_instance
[START] [2022-08-03 16:42:59] overall_tsv_creation
[INFO] [2022-08-03 16:43:00] Exporting 9 nodes as TSV in batches of 10000...
[INFO] [2022-08-03 16:43:00] Processing group of 9 in 1 batches of 10000
[INFO] [2022-08-03 16:43:41] 9 Traits (unfiltered) and 0 associations...
[INFO] [2022-08-03 16:43:41] Building Traits map for 9 nodes (this can take a while)...
[INFO] [2022-08-03 16:44:34] Mapped 9 traits (9 meta) for 9 nodes.
[INFO] [2022-08-03 16:44:34] Building Associations map (this can take a while)...
[INFO] [2022-08-03 16:44:34] Done. 0 assocs mapped (0 meta).
[INFO] [2022-08-03 16:44:34] Adding 9 traits...
[INFO] [2022-08-03 16:44:34] 9 metadata added.
[INFO] [2022-08-03 16:44:34] Adding 0 assocs...
[INFO] [2022-08-03 16:44:34] 0 metadata added.
[INFO] [2022-08-03 16:45:17] Processed 9/9 nodes
[INFO] [2022-08-03 16:45:17] Average Time: 112.24
[INFO] [2022-08-03 16:45:17] Total Time: 2m18s
[STOP] [2022-08-03 16:45:17] overall_tsv_creation
[INFO] [2022-08-03 16:45:17] Done. Check your files:
[INFO] [2022-08-03 16:45:17] (9 lines) /app/public/data/yoon_et_al_yoon_/publish_nodes.tsv
[INFO] [2022-08-03 16:45:17] (9 lines) /app/public/data/yoon_et_al_yoon_/publish_scientific_names.tsv
[INFO] [2022-08-03 16:45:17] (10 lines) /app/public/data/yoon_et_al_yoon_/publish_traits.tsv
[INFO] [2022-08-03 16:45:17] (10 lines) /app/public/data/yoon_et_al_yoon_/publish_metadata.tsv
[STOP] [2022-08-03 16:45:17] complete_harvest_instance
[START] [2022-08-03 16:45:17] completed
[STOP] [2022-08-03 16:45:17] completed
[STOP] [2022-08-03 16:45:17] logged process, took 146.65
Latest Process

03 Aug 16:42 EXITED currently processing 0