Harvest for INOTAXA archive Created 17 Jan 17:11

Harvesting Log

# Logfile created on 2020-01-17 17:11:07 -0500 by logger.rb/56815
[START] [2020-01-17 17:11:07] logged process
[START] [2020-01-17 17:11:07] create_harvest_instance
[STOP] [2020-01-17 17:11:07] create_harvest_instance
[START] [2020-01-17 17:11:08] fetch_files
[STOP] [2020-01-17 17:11:08] fetch_files
[START] [2020-01-17 17:11:08] validate_each_file
[STOP] [2020-01-17 17:11:08] validate_each_file
[START] [2020-01-17 17:11:08] convert_to_csv
[CMD] [2020-01-17 17:11:08] /usr/bin/sort /app/public/converted_csv/inotaxa_archived_agents_19996.csv > /app/public/converted_csv/inotaxa_archived_agents_19996.csv_sorted
[CMD] [2020-01-17 17:11:08] /usr/bin/sort /app/public/converted_csv/inotaxa_archived_refs_19997.csv > /app/public/converted_csv/inotaxa_archived_refs_19997.csv_sorted
[CMD] [2020-01-17 17:11:09] /usr/bin/sort /app/public/converted_csv/inotaxa_archived_nodes_19998.csv > /app/public/converted_csv/inotaxa_archived_nodes_19998.csv_sorted
[CMD] [2020-01-17 17:11:09] /usr/bin/sort /app/public/converted_csv/inotaxa_archived_media_19999.csv > /app/public/converted_csv/inotaxa_archived_media_19999.csv_sorted
[STOP] [2020-01-17 17:11:09] convert_to_csv
[START] [2020-01-17 17:11:09] calculate_delta
[CMD] [2020-01-17 17:11:09] echo "0a" > /app/public/diff/inotaxa_archived_agents_19996.diff
[CMD] [2020-01-17 17:11:09] tail -n +1 /app/public/converted_csv/inotaxa_archived_agents_19996.csv >> /app/public/diff/inotaxa_archived_agents_19996.diff
[CMD] [2020-01-17 17:11:09] echo "." >> /app/public/diff/inotaxa_archived_agents_19996.diff
[CMD] [2020-01-17 17:11:09] echo "0a" > /app/public/diff/inotaxa_archived_refs_19997.diff
[CMD] [2020-01-17 17:11:10] tail -n +1 /app/public/converted_csv/inotaxa_archived_refs_19997.csv >> /app/public/diff/inotaxa_archived_refs_19997.diff
[CMD] [2020-01-17 17:11:10] echo "." >> /app/public/diff/inotaxa_archived_refs_19997.diff
[CMD] [2020-01-17 17:11:10] echo "0a" > /app/public/diff/inotaxa_archived_nodes_19998.diff
[CMD] [2020-01-17 17:11:10] tail -n +1 /app/public/converted_csv/inotaxa_archived_nodes_19998.csv >> /app/public/diff/inotaxa_archived_nodes_19998.diff
[CMD] [2020-01-17 17:11:10] echo "." >> /app/public/diff/inotaxa_archived_nodes_19998.diff
[CMD] [2020-01-17 17:11:10] echo "0a" > /app/public/diff/inotaxa_archived_media_19999.diff
[CMD] [2020-01-17 17:11:11] tail -n +1 /app/public/converted_csv/inotaxa_archived_media_19999.csv >> /app/public/diff/inotaxa_archived_media_19999.diff
[CMD] [2020-01-17 17:11:11] echo "." >> /app/public/diff/inotaxa_archived_media_19999.diff
[STOP] [2020-01-17 17:11:11] calculate_delta
[START] [2020-01-17 17:11:11] parse_diff_and_store
[INFO] [2020-01-17 17:11:11] Loading agents diff file into memory (true lines)...
[INFO] [2020-01-17 17:11:11] Loading refs diff file into memory (true lines)...
[INFO] [2020-01-17 17:11:11] Loading nodes diff file into memory (true lines)...
[INFO] [2020-01-17 17:11:12] Loading media diff file into memory (true lines)...
[INFO] [2020-01-17 17:11:16] Storing 3 Attributions
[INFO] [2020-01-17 17:11:16] Processing group of 3 in 1 groups of 1000
[INFO] [2020-01-17 17:11:16] Average Time: 0.0
[INFO] [2020-01-17 17:11:16] Total Time: 1s
[INFO] [2020-01-17 17:11:16] Storing 343 References
[INFO] [2020-01-17 17:11:16] Processing group of 343 in 1 groups of 1000
[INFO] [2020-01-17 17:11:16] Average Time: 0.08
[INFO] [2020-01-17 17:11:16] Total Time: 1s
[INFO] [2020-01-17 17:11:16] Storing 874 ScientificNames
[INFO] [2020-01-17 17:11:16] Processing group of 874 in 1 groups of 1000
[INFO] [2020-01-17 17:11:17] Average Time: 0.35
[INFO] [2020-01-17 17:11:17] Total Time: 1s
[INFO] [2020-01-17 17:11:17] Storing 874 Nodes
[INFO] [2020-01-17 17:11:17] Processing group of 874 in 1 groups of 1000
[INFO] [2020-01-17 17:11:17] Average Time: 0.26
[INFO] [2020-01-17 17:11:17] Total Time: 1s
[INFO] [2020-01-17 17:11:17] Storing 871 NodesReferences
[INFO] [2020-01-17 17:11:17] Processing group of 871 in 1 groups of 1000
[INFO] [2020-01-17 17:11:17] Average Time: 0.16
[INFO] [2020-01-17 17:11:17] Total Time: 1s
[INFO] [2020-01-17 17:11:17] Storing 2165 ArticlesReferences
[INFO] [2020-01-17 17:11:17] Processing group of 2165 in 3 groups of 1000
[INFO] [2020-01-17 17:11:17] Average Time: 0.093
[INFO] [2020-01-17 17:11:17] Total Time: 1s
[INFO] [2020-01-17 17:11:17] Storing 2174 ContentAttributions
[INFO] [2020-01-17 17:11:17] Processing group of 2174 in 3 groups of 1000
[INFO] [2020-01-17 17:11:18] Average Time: 0.093
[INFO] [2020-01-17 17:11:18] Total Time: 1s
[INFO] [2020-01-17 17:11:18] Storing 2174 ArticlesSections
[INFO] [2020-01-17 17:11:18] Processing group of 2174 in 3 groups of 1000
[INFO] [2020-01-17 17:11:18] Average Time: 0.037
[INFO] [2020-01-17 17:11:18] Total Time: 1s
[INFO] [2020-01-17 17:11:18] Storing 2174 Articles
[INFO] [2020-01-17 17:11:18] Processing group of 2174 in 3 groups of 1000
[INFO] [2020-01-17 17:11:19] Average Time: 0.27
[INFO] [2020-01-17 17:11:19] Total Time: 1s
[INFO] [2020-01-17 17:11:19] Storing 576 Media
[INFO] [2020-01-17 17:11:19] Processing group of 576 in 1 groups of 1000
[INFO] [2020-01-17 17:11:19] Average Time: 0.3
[INFO] [2020-01-17 17:11:19] Total Time: 1s
[STOP] [2020-01-17 17:11:19] parse_diff_and_store
[START] [2020-01-17 17:11:19] resolve_keys
[INFO] [2020-01-17 17:11:29] Occurrences to nodes (through scientific_names)...
[INFO] [2020-01-17 17:11:29] traits to occurrences...
[INFO] [2020-01-17 17:11:29] traits to nodes (through occurrences)...
[INFO] [2020-01-17 17:11:29] Traits to sex term...
[INFO] [2020-01-17 17:11:29] Traits to lifestage term...
[INFO] [2020-01-17 17:11:29] MetaTraits to traits...
[INFO] [2020-01-17 17:11:29] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-01-17 17:11:29] Assocs to occurrences...
[INFO] [2020-01-17 17:11:29] Assocs to nodes...
[INFO] [2020-01-17 17:11:29] Assoc to sex term...
[INFO] [2020-01-17 17:11:29] Assoc to lifestage term...
[STOP] [2020-01-17 17:11:29] resolve_keys
[START] [2020-01-17 17:11:29] hold_for_later_1
[STOP] [2020-01-17 17:11:29] hold_for_later_1
[START] [2020-01-17 17:11:29] hold_for_later_2
[STOP] [2020-01-17 17:11:29] hold_for_later_2
[START] [2020-01-17 17:11:29] resolve_missing_parents
[STOP] [2020-01-17 17:11:29] resolve_missing_parents
[START] [2020-01-17 17:11:29] rebuild_nodes
[START] [2020-01-17 17:11:29] Flattener#flatten
[START] [2020-01-17 17:11:29] Flattener#study_resource
[START] [2020-01-17 17:11:29] Flattener#build_ancestry
[STOP] [2020-01-17 17:11:29] Flattener#build_ancestry
[INFO] [2020-01-17 17:11:29] 874 ancestry keys
[START] [2020-01-17 17:11:29] build_node_ancestors
[INFO] [2020-01-17 17:11:29] old ancestors deleted.
[STOP] [2020-01-17 17:11:29] build_node_ancestors
[START] [2020-01-17 17:11:30] Flattener#propagate_ancestor_ids
[STOP] [2020-01-17 17:11:30] Flattener#propagate_ancestor_ids
[STOP] [2020-01-17 17:11:30] Flattener#flatten
[STOP] [2020-01-17 17:11:30] rebuild_nodes
[START] [2020-01-17 17:11:30] resolve_missing_media_owners
[STOP] [2020-01-17 17:11:30] resolve_missing_media_owners
[START] [2020-01-17 17:11:30] sanitize_media_verbatims
[STOP] [2020-01-17 17:11:30] sanitize_media_verbatims
[START] [2020-01-17 17:11:30] queue_downloads
[STOP] [2020-01-17 17:11:30] queue_downloads
[START] [2020-01-17 17:11:30] parse_names
[WARN] [2020-01-17 17:11:30] I see 874 names which still need to be parsed.
[STOP] [2020-01-17 17:11:32] parse_names
[START] [2020-01-17 17:11:32] denormalize_canonical_names_to_nodes
[STOP] [2020-01-17 17:11:32] denormalize_canonical_names_to_nodes
[START] [2020-01-17 17:11:32] match_nodes
[START] [2020-01-17 17:11:32] map_all_nodes_to_pages
[ERR] [2020-01-17 17:11:45][hdls] http://www.nhm.ac.uk/hosted-sites/inotaxa/images/img/BCA-coleoptv4p3-pfig-t11f16a.jpg is text/html, which is unsupported. Medium.find(10380657) resource: INOTAXA archive (870), PK: BCA-coleoptv4p3-pfig-t11f16a
[ERR] [2020-01-17 17:11:45][hdls] download_and_prep FAILED for Medium.find(10380657): http://www.nhm.ac.uk/hosted-sites/inotaxa/images/img/BCA-coleoptv4p3-pfig-t11f16a.jpg is text/html, which is unsupported. Medium.find(10380657) resource: INOTAXA archive (870), PK: BCA-coleoptv4p3-pfig-t11f16a
[ERR] [2020-01-17 17:12:48][hdls] http://www.nhm.ac.uk/hosted-sites/inotaxa/images/img/BCA-coleoptv4p3-pfig-t8f18a.jpg is text/html, which is unsupported. Medium.find(10381091) resource: INOTAXA archive (870), PK: BCA-coleoptv4p3-pfig-t8f18a
[ERR] [2020-01-17 17:12:48][hdls] download_and_prep FAILED for Medium.find(10381091): http://www.nhm.ac.uk/hosted-sites/inotaxa/images/img/BCA-coleoptv4p3-pfig-t8f18a.jpg is text/html, which is unsupported. Medium.find(10381091) resource: INOTAXA archive (870), PK: BCA-coleoptv4p3-pfig-t8f18a
[STOP] [2020-01-17 17:13:00] map_all_nodes_to_pages
[INFO] [2020-01-17 17:13:00] 166 Unmatched nodes (of 874)! That's too many to output. First 10: Attelabinae (#62896689); Auletobius optatus (#62896718); Allocoryninae (#62896726); Apioninae (#62896729); Apion (#62896730); Apion division (#62896731); Apion submetallicum (#62896732); Apion spretissimum (#62896733); Apion praeditum (#62896735); Apion dissimilipes (#62896736)
[START] [2020-01-17 17:13:00] update_nodes
[STOP] [2020-01-17 17:13:00] update_nodes
[STOP] [2020-01-17 17:13:00] match_nodes
[START] [2020-01-17 17:13:00] reindex_search
[STOP] [2020-01-17 17:13:01] reindex_search
[START] [2020-01-17 17:13:01] normalize_units
[STOP] [2020-01-17 17:13:01] normalize_units
[START] [2020-01-17 17:13:02] calculate_statistics
[STOP] [2020-01-17 17:13:02] calculate_statistics
[START] [2020-01-17 17:13:02] complete_harvest_instance
[START] [2020-01-17 17:13:02] overall_tsv_creation
[INFO] [2020-01-17 17:13:02] Processing group of 874 in 1 batches of 10000
[INFO] [2020-01-17 17:13:48] Average Time: 17.58
[INFO] [2020-01-17 17:13:48] Total Time: 47s
[STOP] [2020-01-17 17:13:48] overall_tsv_creation
[INFO] [2020-01-17 17:13:48] Done. Check your files:
[INFO] [2020-01-17 17:13:48] (874 lines) /app/public/data/inotaxa_archived/publish_nodes.tsv
[INFO] [2020-01-17 17:13:48] (2616 lines) /app/public/data/inotaxa_archived/publish_node_ancestors.tsv
[INFO] [2020-01-17 17:13:48] (874 lines) /app/public/data/inotaxa_archived/publish_scientific_names.tsv
[INFO] [2020-01-17 17:13:48] (576 lines) /app/public/data/inotaxa_archived/publish_media.tsv
[INFO] [2020-01-17 17:13:49] (2174 lines) /app/public/data/inotaxa_archived/publish_articles.tsv
[INFO] [2020-01-17 17:13:49] (574 lines) /app/public/data/inotaxa_archived/publish_image_info.tsv
[INFO] [2020-01-17 17:13:49] (343 lines) /app/public/data/inotaxa_archived/publish_references.tsv
[INFO] [2020-01-17 17:13:49] (2174 lines) /app/public/data/inotaxa_archived/publish_attributions.tsv
[INFO] [2020-01-17 17:13:49] (2174 lines) /app/public/data/inotaxa_archived/publish_content_sections.tsv
[INFO] [2020-01-17 17:13:49] (343 lines) /app/public/data/inotaxa_archived/publish_referents.tsv
[STOP] [2020-01-17 17:13:49] complete_harvest_instance
[START] [2020-01-17 17:13:49] completed
[STOP] [2020-01-17 17:13:49] completed
[STOP] [2020-01-17 17:13:49] logged process, took 162.78

Latest Process

17 Jan 17:11 EXITED currently processing 0