# Logfile created on 2019-07-16 12:23:11 -0400 by logger.rb/56815 [START] [2019-07-16 12:23:11] logged process [START] [2019-07-16 12:23:11] create_harvest_instance [STOP] [2019-07-16 12:23:12] create_harvest_instance [START] [2019-07-16 12:23:12] fetch_files [STOP] [2019-07-16 12:23:12] fetch_files [START] [2019-07-16 12:23:12] validate_each_file [STOP] [2019-07-16 12:23:12] validate_each_file [START] [2019-07-16 12:23:12] convert_to_csv [CMD] [2019-07-16 12:23:12] /usr/bin/sort /app/public/converted_csv/osatp_agents_14233.csv > /app/public/converted_csv/osatp_agents_14233.csv_sorted [CMD] [2019-07-16 12:23:13] /usr/bin/sort /app/public/converted_csv/osatp_nodes_14234.csv > /app/public/converted_csv/osatp_nodes_14234.csv_sorted [CMD] [2019-07-16 12:23:15] /usr/bin/sort /app/public/converted_csv/osatp_media_14235.csv > /app/public/converted_csv/osatp_media_14235.csv_sorted [STOP] [2019-07-16 12:23:17] convert_to_csv [START] [2019-07-16 12:23:17] calculate_delta [CMD] [2019-07-16 12:23:17] echo "0a" > /app/public/diff/osatp_agents_14233.diff [CMD] [2019-07-16 12:23:18] tail -n +1 /app/public/converted_csv/osatp_agents_14233.csv >> /app/public/diff/osatp_agents_14233.diff [CMD] [2019-07-16 12:23:20] echo "." >> /app/public/diff/osatp_agents_14233.diff [CMD] [2019-07-16 12:23:22] echo "0a" > /app/public/diff/osatp_nodes_14234.diff [CMD] [2019-07-16 12:23:23] tail -n +1 /app/public/converted_csv/osatp_nodes_14234.csv >> /app/public/diff/osatp_nodes_14234.diff [CMD] [2019-07-16 12:23:25] echo "." >> /app/public/diff/osatp_nodes_14234.diff [CMD] [2019-07-16 12:23:27] echo "0a" > /app/public/diff/osatp_media_14235.diff [CMD] [2019-07-16 12:23:28] tail -n +1 /app/public/converted_csv/osatp_media_14235.csv >> /app/public/diff/osatp_media_14235.diff [CMD] [2019-07-16 12:23:30] echo "." >> /app/public/diff/osatp_media_14235.diff [STOP] [2019-07-16 12:23:32] calculate_delta [START] [2019-07-16 12:23:32] parse_diff_and_store [INFO] [2019-07-16 12:23:33] Loading agents diff file into memory (true lines)... [INFO] [2019-07-16 12:23:35] Loading nodes diff file into memory (true lines)... [INFO] [2019-07-16 12:23:37] Loading media diff file into memory (true lines)... [INFO] [2019-07-16 12:23:37] Storing 3 Attributions [INFO] [2019-07-16 12:23:37] Processing group of 3 in 1 groups of 1000 [INFO] [2019-07-16 12:23:37] Average Time: 0.0 [INFO] [2019-07-16 12:23:37] Total Time: 1s [INFO] [2019-07-16 12:23:37] Storing 230 ScientificNames [INFO] [2019-07-16 12:23:37] Processing group of 230 in 1 groups of 1000 [INFO] [2019-07-16 12:23:37] Average Time: 0.2 [INFO] [2019-07-16 12:23:37] Total Time: 1s [INFO] [2019-07-16 12:23:37] Storing 230 Nodes [INFO] [2019-07-16 12:23:37] Processing group of 230 in 1 groups of 1000 [INFO] [2019-07-16 12:23:37] Average Time: 0.08 [INFO] [2019-07-16 12:23:37] Total Time: 1s [INFO] [2019-07-16 12:23:37] Storing 219 ContentAttributions [INFO] [2019-07-16 12:23:37] Processing group of 219 in 1 groups of 1000 [INFO] [2019-07-16 12:23:37] Average Time: 0.1 [INFO] [2019-07-16 12:23:37] Total Time: 1s [INFO] [2019-07-16 12:23:37] Storing 73 Media [INFO] [2019-07-16 12:23:37] Processing group of 73 in 1 groups of 1000 [INFO] [2019-07-16 12:23:37] Average Time: 0.07 [INFO] [2019-07-16 12:23:37] Total Time: 1s [STOP] [2019-07-16 12:23:37] parse_diff_and_store [START] [2019-07-16 12:23:37] resolve_keys [INFO] [2019-07-16 12:23:41] Occurrences to nodes (through scientific_names)... [INFO] [2019-07-16 12:23:41] traits to occurrences... [INFO] [2019-07-16 12:23:41] traits to nodes (through occurrences)... [INFO] [2019-07-16 12:23:41] Traits to sex term... [INFO] [2019-07-16 12:23:41] Traits to lifestage term... [INFO] [2019-07-16 12:23:41] MetaTraits to traits... [INFO] [2019-07-16 12:23:41] MetaTraits (simple, measurement row refers to parent) to traits... [INFO] [2019-07-16 12:23:41] Assocs to occurrences... [INFO] [2019-07-16 12:23:41] Assocs to nodes... [INFO] [2019-07-16 12:23:41] Assoc to sex term... [INFO] [2019-07-16 12:23:41] Assoc to lifestage term... [STOP] [2019-07-16 12:23:41] resolve_keys [START] [2019-07-16 12:23:41] hold_for_later_1 [STOP] [2019-07-16 12:23:41] hold_for_later_1 [START] [2019-07-16 12:23:41] hold_for_later_2 [STOP] [2019-07-16 12:23:41] hold_for_later_2 [START] [2019-07-16 12:23:41] resolve_missing_parents [STOP] [2019-07-16 12:23:41] resolve_missing_parents [START] [2019-07-16 12:23:41] rebuild_nodes [START] [2019-07-16 12:23:41] Flattener#flatten [START] [2019-07-16 12:23:41] Flattener#study_resource [START] [2019-07-16 12:23:41] Flattener#build_ancestry [STOP] [2019-07-16 12:23:41] Flattener#build_ancestry [INFO] [2019-07-16 12:23:41] 230 ancestry keys [START] [2019-07-16 12:23:41] build_node_ancestors [INFO] [2019-07-16 12:23:41] old ancestors deleted. [STOP] [2019-07-16 12:23:41] build_node_ancestors [START] [2019-07-16 12:23:41] Flattener#propagate_ancestor_ids [STOP] [2019-07-16 12:23:41] Flattener#propagate_ancestor_ids [STOP] [2019-07-16 12:23:41] Flattener#flatten [STOP] [2019-07-16 12:23:41] rebuild_nodes [START] [2019-07-16 12:23:41] resolve_missing_media_owners [STOP] [2019-07-16 12:23:41] resolve_missing_media_owners [START] [2019-07-16 12:23:41] sanitize_media_verbatims [STOP] [2019-07-16 12:23:41] sanitize_media_verbatims [START] [2019-07-16 12:23:41] queue_downloads [STOP] [2019-07-16 12:23:42] queue_downloads [START] [2019-07-16 12:23:42] parse_names [WARN] [2019-07-16 12:23:42] I see 230 names which still need to be parsed. [STOP] [2019-07-16 12:23:43] parse_names [START] [2019-07-16 12:23:43] denormalize_canonical_names_to_nodes [STOP] [2019-07-16 12:23:43] denormalize_canonical_names_to_nodes [START] [2019-07-16 12:23:43] match_nodes [START] [2019-07-16 12:23:43] map_all_nodes_to_pages [STOP] [2019-07-16 12:23:53] map_all_nodes_to_pages [INFO] [2019-07-16 12:23:53] 15 Unmatched nodes (of 230)! That's too many to output. First 10: Pyrrophycophyta (#44581335); Calidris canutus rufa (#44581336); Leptothrix (#44581343); Leptothrix ochracea (#44581344); Bacillus subtilis (#44581349); Leptoptilos crumeniferus (#44581357); Korscheltellus (#44581382); Korscheltellus gracilis (#44581383); Leptoptilos crumeniferus (#44581396); Leptothrix ochracea (#44581402) [START] [2019-07-16 12:23:53] update_nodes [STOP] [2019-07-16 12:23:53] update_nodes [STOP] [2019-07-16 12:23:53] match_nodes [START] [2019-07-16 12:23:53] reindex_search [STOP] [2019-07-16 12:23:53] reindex_search [START] [2019-07-16 12:23:53] normalize_units [STOP] [2019-07-16 12:23:53] normalize_units [START] [2019-07-16 12:23:53] calculate_statistics [STOP] [2019-07-16 12:23:53] calculate_statistics [START] [2019-07-16 12:23:53] complete_harvest_instance [START] [2019-07-16 12:23:53] overall_tsv_creation [INFO] [2019-07-16 12:23:53] Processing group of 230 in 1 batches of 10000 [INFO] [2019-07-16 12:24:27] Average Time: 12.29 [INFO] [2019-07-16 12:24:27] Total Time: 34s [STOP] [2019-07-16 12:24:27] overall_tsv_creation [INFO] [2019-07-16 12:24:27] Done. Check your files: [INFO] [2019-07-16 12:24:28] (230 lines) /app/public/data/osatp/publish_nodes.tsv [INFO] [2019-07-16 12:24:30] (79 lines) /app/public/data/osatp/publish_node_ancestors.tsv [INFO] [2019-07-16 12:24:32] (230 lines) /app/public/data/osatp/publish_scientific_names.tsv [INFO] [2019-07-16 12:24:33] (73 lines) /app/public/data/osatp/publish_media.tsv [INFO] [2019-07-16 12:24:35] (219 lines) /app/public/data/osatp/publish_attributions.tsv [STOP] [2019-07-16 12:24:35] complete_harvest_instance [START] [2019-07-16 12:24:35] completed [STOP] [2019-07-16 12:24:35] completed [STOP] [2019-07-16 12:24:35] logged process, took 83.76 [START] [2019-07-16 12:24:35] logged process [START] [2019-07-16 12:24:35] create_harvest_instance [STOP] [2019-07-16 12:24:35] create_harvest_instance [START] [2019-07-16 12:24:35] fetch_files [STOP] [2019-07-16 12:24:35] fetch_files [ERR] [2019-07-16 12:24:35] RuntimeError [ERR] [2019-07-16 12:24:35] No files have changed! [ERR] [2019-07-16 12:24:35] ../models/resource_harvester.rb:115:in `fetch_files' [ERR] [2019-07-16 12:24:35] ../models/resource_harvester.rb:86:in `block (3 levels) in start' [ERR] [2019-07-16 12:24:35] ../models/logged_process.rb:19:in `run_step' [ERR] [2019-07-16 12:24:35] ../models/resource_harvester.rb:86:in `block (2 levels) in start' [ERR] [2019-07-16 12:24:35] ../models/resource_harvester.rb:75:in `each_key' [ERR] [2019-07-16 12:24:35] ../models/resource_harvester.rb:75:in `block in start' [ERR] [2019-07-16 12:24:35] ../models/resource.rb:134:in `lock' [ERR] [2019-07-16 12:24:35] ../models/resource_harvester.rb:72:in `start' [ERR] [2019-07-16 12:24:35] ../models/resource.rb:218:in `harvest' [STOP] [2019-07-16 12:24:35] logged process, took 0.56