Harvest for Natural History Services Created 10 Jul 14:46

Stage: completed
Fetched: 10 Jul 14:46
Validated: 10 Jul 14:46
Deltas Created 10 Jul 14:46
Units Normalized: 10 Jul 14:47
Ancestry Built: 10 Jul 14:47
Nodes Matched: 10 Jul 14:47
Names Parsed: 10 Jul 14:47
New Models Stored: 10 Jul 14:46
Indexed: 10 Jul 14:47
Completed: 10 Jul 14:47
Time to Harvest: less than a minute
Harvesting Log

(119 lines)
# Logfile created on 2019-07-10 14:46:58 -0400 by logger.rb/56815
[START] [2019-07-10 14:46:58] logged process
[START] [2019-07-10 14:46:58] create_harvest_instance
[STOP] [2019-07-10 14:46:58] create_harvest_instance
[START] [2019-07-10 14:46:58] fetch_files
[STOP] [2019-07-10 14:46:58] fetch_files
[START] [2019-07-10 14:46:58] validate_each_file
[STOP] [2019-07-10 14:46:58] validate_each_file
[START] [2019-07-10 14:46:58] convert_to_csv
[CMD] [2019-07-10 14:46:58] /usr/bin/sort /app/public/converted_csv/natural_history__agents_14126.csv > /app/public/converted_csv/natural_history__agents_14126.csv_sorted
[CMD] [2019-07-10 14:46:58] /usr/bin/sort /app/public/converted_csv/natural_history__nodes_14127.csv > /app/public/converted_csv/natural_history__nodes_14127.csv_sorted
[CMD] [2019-07-10 14:46:58] /usr/bin/sort /app/public/converted_csv/natural_history__media_14128.csv > /app/public/converted_csv/natural_history__media_14128.csv_sorted
[STOP] [2019-07-10 14:46:58] convert_to_csv
[START] [2019-07-10 14:46:58] calculate_delta
[CMD] [2019-07-10 14:46:58] echo "0a" > /app/public/diff/natural_history__agents_14126.diff
[CMD] [2019-07-10 14:46:58] tail -n +1 /app/public/converted_csv/natural_history__agents_14126.csv >> /app/public/diff/natural_history__agents_14126.diff
[CMD] [2019-07-10 14:46:58] echo "." >> /app/public/diff/natural_history__agents_14126.diff
[CMD] [2019-07-10 14:46:58] echo "0a" > /app/public/diff/natural_history__nodes_14127.diff
[CMD] [2019-07-10 14:46:58] tail -n +1 /app/public/converted_csv/natural_history__nodes_14127.csv >> /app/public/diff/natural_history__nodes_14127.diff
[CMD] [2019-07-10 14:46:58] echo "." >> /app/public/diff/natural_history__nodes_14127.diff
[CMD] [2019-07-10 14:46:58] echo "0a" > /app/public/diff/natural_history__media_14128.diff
[CMD] [2019-07-10 14:46:59] tail -n +1 /app/public/converted_csv/natural_history__media_14128.csv >> /app/public/diff/natural_history__media_14128.diff
[CMD] [2019-07-10 14:46:59] echo "." >> /app/public/diff/natural_history__media_14128.diff
[STOP] [2019-07-10 14:46:59] calculate_delta
[START] [2019-07-10 14:46:59] parse_diff_and_store
[INFO] [2019-07-10 14:46:59] Loading agents diff file into memory (true lines)...
[INFO] [2019-07-10 14:46:59] Loading nodes diff file into memory (true lines)...
[INFO] [2019-07-10 14:46:59] Loading media diff file into memory (true lines)...
[INFO] [2019-07-10 14:46:59] Storing 2 Attributions
[INFO] [2019-07-10 14:46:59] Processing group of 2 in 1 groups of 1000
[INFO] [2019-07-10 14:46:59] Average Time: 0.0
[INFO] [2019-07-10 14:46:59] Total Time: 1s
[INFO] [2019-07-10 14:46:59] Storing 153 ScientificNames
[INFO] [2019-07-10 14:46:59] Processing group of 153 in 1 groups of 1000
[INFO] [2019-07-10 14:46:59] Average Time: 0.21
[INFO] [2019-07-10 14:46:59] Total Time: 1s
[INFO] [2019-07-10 14:46:59] Storing 153 Nodes
[INFO] [2019-07-10 14:46:59] Processing group of 153 in 1 groups of 1000
[INFO] [2019-07-10 14:46:59] Average Time: 0.11
[INFO] [2019-07-10 14:46:59] Total Time: 1s
[INFO] [2019-07-10 14:46:59] Storing 92 ContentAttributions
[INFO] [2019-07-10 14:46:59] Processing group of 92 in 1 groups of 1000
[INFO] [2019-07-10 14:46:59] Average Time: 0.05
[INFO] [2019-07-10 14:46:59] Total Time: 1s
[INFO] [2019-07-10 14:46:59] Storing 46 Media
[INFO] [2019-07-10 14:46:59] Processing group of 46 in 1 groups of 1000
[INFO] [2019-07-10 14:46:59] Average Time: 0.03
[INFO] [2019-07-10 14:46:59] Total Time: 1s
[STOP] [2019-07-10 14:46:59] parse_diff_and_store
[START] [2019-07-10 14:46:59] resolve_keys
[INFO] [2019-07-10 14:47:03] Occurrences to nodes (through scientific_names)...
[INFO] [2019-07-10 14:47:03] traits to occurrences...
[INFO] [2019-07-10 14:47:03] traits to nodes (through occurrences)...
[INFO] [2019-07-10 14:47:03] Traits to sex term...
[INFO] [2019-07-10 14:47:03] Traits to lifestage term...
[INFO] [2019-07-10 14:47:03] MetaTraits to traits...
[INFO] [2019-07-10 14:47:03] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-07-10 14:47:03] Assocs to occurrences...
[INFO] [2019-07-10 14:47:03] Assocs to nodes...
[INFO] [2019-07-10 14:47:03] Assoc to sex term...
[INFO] [2019-07-10 14:47:03] Assoc to lifestage term...
[STOP] [2019-07-10 14:47:03] resolve_keys
[START] [2019-07-10 14:47:03] hold_for_later_1
[STOP] [2019-07-10 14:47:03] hold_for_later_1
[START] [2019-07-10 14:47:03] hold_for_later_2
[STOP] [2019-07-10 14:47:03] hold_for_later_2
[START] [2019-07-10 14:47:03] resolve_missing_parents
[STOP] [2019-07-10 14:47:03] resolve_missing_parents
[START] [2019-07-10 14:47:03] rebuild_nodes
[START] [2019-07-10 14:47:03] Flattener#flatten
[START] [2019-07-10 14:47:03] Flattener#study_resource
[START] [2019-07-10 14:47:03] Flattener#build_ancestry
[STOP] [2019-07-10 14:47:03] Flattener#build_ancestry
[INFO] [2019-07-10 14:47:03] 153 ancestry keys
[START] [2019-07-10 14:47:03] build_node_ancestors
[INFO] [2019-07-10 14:47:03] old ancestors deleted.
[STOP] [2019-07-10 14:47:03] build_node_ancestors
[WARN] [2019-07-10 14:47:03] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2019-07-10 14:47:03] Flattener#flatten
[STOP] [2019-07-10 14:47:03] rebuild_nodes
[START] [2019-07-10 14:47:03] resolve_missing_media_owners
[STOP] [2019-07-10 14:47:03] resolve_missing_media_owners
[START] [2019-07-10 14:47:03] sanitize_media_verbatims
[STOP] [2019-07-10 14:47:03] sanitize_media_verbatims
[START] [2019-07-10 14:47:03] queue_downloads
[STOP] [2019-07-10 14:47:03] queue_downloads
[START] [2019-07-10 14:47:03] parse_names
[WARN] [2019-07-10 14:47:03] I see 153 names which still need to be parsed.
[STOP] [2019-07-10 14:47:05] parse_names
[START] [2019-07-10 14:47:05] denormalize_canonical_names_to_nodes
[STOP] [2019-07-10 14:47:05] denormalize_canonical_names_to_nodes
[START] [2019-07-10 14:47:05] match_nodes
[START] [2019-07-10 14:47:05] map_all_nodes_to_pages
[STOP] [2019-07-10 14:47:10] map_all_nodes_to_pages
[INFO] [2019-07-10 14:47:10] 23 Unmatched nodes (of 153)! That's too many to output. First 10: Megarhyssa macrurus (#44147777); Entypus unifasciatus (#44147784); Thiodina puepera (#44147794); Sitticus pubescens (#44147801); Wallengrenia egeremet (#44147809); Auplopus architectus (#44147824); Dolicovespula maculata (#44147835); Zebra (#44147836); Dasymutilla nigipes (#44147840); Thorybes bathyllus (#44147848)
[START] [2019-07-10 14:47:10] update_nodes
[STOP] [2019-07-10 14:47:10] update_nodes
[STOP] [2019-07-10 14:47:10] match_nodes
[START] [2019-07-10 14:47:10] reindex_search
[STOP] [2019-07-10 14:47:10] reindex_search
[START] [2019-07-10 14:47:10] normalize_units
[STOP] [2019-07-10 14:47:10] normalize_units
[START] [2019-07-10 14:47:10] calculate_statistics
[STOP] [2019-07-10 14:47:10] calculate_statistics
[START] [2019-07-10 14:47:10] complete_harvest_instance
[START] [2019-07-10 14:47:10] overall_tsv_creation
[INFO] [2019-07-10 14:47:10] Processing group of 153 in 1 batches of 10000
[INFO] [2019-07-10 14:47:43] Average Time: 12.09
[INFO] [2019-07-10 14:47:43] Total Time: 33s
[STOP] [2019-07-10 14:47:43] overall_tsv_creation
[INFO] [2019-07-10 14:47:43] Done. Check your files:
[INFO] [2019-07-10 14:47:43] (153 lines) /app/public/data/natural_history_/publish_nodes.tsv
[INFO] [2019-07-10 14:47:43] (153 lines) /app/public/data/natural_history_/publish_scientific_names.tsv
[INFO] [2019-07-10 14:47:43] (46 lines) /app/public/data/natural_history_/publish_media.tsv
[INFO] [2019-07-10 14:47:43] (92 lines) /app/public/data/natural_history_/publish_attributions.tsv
[STOP] [2019-07-10 14:47:43] complete_harvest_instance
[START] [2019-07-10 14:47:43] completed
[STOP] [2019-07-10 14:47:43] completed
[STOP] [2019-07-10 14:47:43] logged process, took 45.42
Latest Process

10 Jul 14:46 EXITED currently processing 0