Harvest for Integrated Taxonomic Information System (ITIS) Taxonomic Hierarchy Created 02 Apr 11:12

Stage: completed
Fetched: 02 Apr 11:12
Validated: 02 Apr 11:13
Deltas Created 02 Apr 11:13
Units Normalized: 03 Apr 09:12
Ancestry Built: 02 Apr 13:00
Nodes Matched: 03 Apr 08:45
Names Parsed: 02 Apr 13:12
New Models Stored: 02 Apr 11:38
Indexed: 03 Apr 09:12
Completed: 03 Apr 10:58
Time to Harvest: 24 minutes

Harvesting Log (most recent first)

# Logfile created on 2020-04-02 11:12:11 -0400 by logger.rb/56815
[START] [2020-04-02 11:12:11] logged process
[START] [2020-04-02 11:12:11] create_harvest_instance
[STOP] [2020-04-02 11:12:11] create_harvest_instance
[START] [2020-04-02 11:12:11] fetch_files
[STOP] [2020-04-02 11:12:11] fetch_files
[START] [2020-04-02 11:12:11] validate_each_file
[STOP] [2020-04-02 11:13:04] validate_each_file
[START] [2020-04-02 11:13:04] convert_to_csv
[CMD] [2020-04-02 11:13:04] /usr/bin/sort /app/public/converted_csv/ihfifid_nodes_20506.csv > /app/public/converted_csv/ihfifid_nodes_20506.csv_sorted
[STOP] [2020-04-02 11:13:04] convert_to_csv
[START] [2020-04-02 11:13:04] calculate_delta
[CMD] [2020-04-02 11:13:04] echo "0a" > /app/public/diff/ihfifid_nodes_20506.diff
[CMD] [2020-04-02 11:13:04] tail -n +1 /app/public/converted_csv/ihfifid_nodes_20506.csv >> /app/public/diff/ihfifid_nodes_20506.diff
[CMD] [2020-04-02 11:13:04] echo "." >> /app/public/diff/ihfifid_nodes_20506.diff
[STOP] [2020-04-02 11:13:04] calculate_delta
[START] [2020-04-02 11:13:04] parse_diff_and_store
[INFO] [2020-04-02 11:13:05] Loading nodes diff file into memory (true lines)...
[WARN] [2020-04-02 11:13:39] Filtered Scientific Name `Holothuria austrinabassa O'Loughlin in O'Loughlin,  Paulay, VandenSpiegel & Samyn, 2007` to `Holothuria austrinabassa O'Loughlin in O'Loughlin, Paulay, VandenSpiegel & Samyn, 2007`
[WARN] [2020-04-02 11:14:00] Filtered Scientific Name `Cephaloziella phyllacantha (C. Massal.  & Carestia) K. Müll.` to `Cephaloziella phyllacantha (C. Massal. & Carestia) K. Müll.`
[WARN] [2020-04-02 11:14:03] Filtered Scientific Name `Pseudokephyrion/part Pascher, 1913` to `Pseudokephyrionpart Pascher, 1913`
[WARN] [2020-04-02 11:14:06] Filtered Scientific Name `Pseudokephyrion/part` to `Pseudokephyrionpart`
[WARN] [2020-04-02 11:14:20] Filtered Scientific Name `Polyxenus Latreille, 1802/1803` to `Polyxenus Latreille, 18021803`
[WARN] [2020-04-02 11:14:31] Filtered Scientific Name `Chrysonema/chrysotilaceae` to `Chrysonemachrysotilaceae`
[WARN] [2020-04-02 11:14:37] Filtered Scientific Name `Catenula/bacillariophyta Mersechkowsky` to `Catenulabacillariophyta Mersechkowsky`
[WARN] [2020-04-02 11:14:57] Filtered Scientific Name `Carex lasiocarpa ssp. americana (Fernald)  D. Löve &  J.-P.Bernard` to `Carex lasiocarpa ssp. americana (Fernald) D. Löve & J.-P.Bernard`
[WARN] [2020-04-02 11:15:21] Filtered Scientific Name `Jacquinia pauciflora B. Ståhl  &  F.S. Axelrod` to `Jacquinia pauciflora B. Ståhl & F.S. Axelrod`
[WARN] [2020-04-02 11:15:23] Filtered Scientific Name `Polydesmus Latreille, 1802/1803` to `Polydesmus Latreille, 18021803`
[WARN] [2020-04-02 11:18:15] Filtered Scientific Name `Oshimella formosana Masam. &  Suzuki` to `Oshimella formosana Masam. & Suzuki`
[WARN] [2020-04-02 11:18:22] Filtered Scientific Name `Eremothera boothii ssp. condensata (Munz)  W.L. Wagner & Hoch` to `Eremothera boothii ssp. condensata (Munz) W.L. Wagner & Hoch`
[WARN] [2020-04-02 11:18:22] Filtered Scientific Name `Eremothera boothii ssp. decorticans (Hook. & Arn.)  W.L. Wagner & Hoch` to `Eremothera boothii ssp. decorticans (Hook. & Arn.) W.L. Wagner & Hoch`
[WARN] [2020-04-02 11:18:23] Filtered Scientific Name `Rosa pisocarpa ssp. ahartii Ertter &  W.H. Lewis` to `Rosa pisocarpa ssp. ahartii Ertter & W.H. Lewis`
[WARN] [2020-04-02 11:19:10] Filtered Scientific Name `Cycas panzhihuaensis L. Zhou  & S.Y. Yang` to `Cycas panzhihuaensis L. Zhou & S.Y. Yang`
[WARN] [2020-04-02 11:19:10] Filtered Scientific Name `Cycas debaoensis Y.C. Zhong  & C.J. Chen` to `Cycas debaoensis Y.C. Zhong & C.J. Chen`
[WARN] [2020-04-02 11:19:10] Filtered Scientific Name `Cycas longiconifera Hung T. Chang, Y.C. Zhong  & Y.Y. Huang` to `Cycas longiconifera Hung T. Chang, Y.C. Zhong & Y.Y. Huang`
[WARN] [2020-04-02 11:19:52] Filtered Scientific Name `Flaviemys Le, Reid, McCord, Naro-Maciel, Raxworthy, Amato and  Georges, 2013` to `Flaviemys Le, Reid, McCord, Naro-Maciel, Raxworthy, Amato and Georges, 2013`
[WARN] [2020-04-02 11:20:10] Filtered Scientific Name `Amynthas acinus Hong &  James, 2009` to `Amynthas acinus Hong & James, 2009`
[WARN] [2020-04-02 11:20:10] Filtered Scientific Name `Amynthas ani Hong &  James, 2009` to `Amynthas ani Hong & James, 2009`
[WARN] [2020-04-02 11:20:10] Filtered Scientific Name `Amynthas baekamensis Hong &  James, 2009` to `Amynthas baekamensis Hong & James, 2009`
[WARN] [2020-04-02 11:20:10] Filtered Scientific Name `Amynthas calculatus Hong &  James, 2009` to `Amynthas calculatus Hong & James, 2009`
[WARN] [2020-04-02 11:20:10] Filtered Scientific Name `Amynthas conferticurtus Hong &  James, 2009` to `Amynthas conferticurtus Hong & James, 2009`
[WARN] [2020-04-02 11:20:10] Filtered Scientific Name `Amynthas cucullatus Hong &  James, 2009` to `Amynthas cucullatus Hong & James, 2009`
[WARN] [2020-04-02 11:20:10] Filtered Scientific Name `Amynthas cuneatus Hong &  James, 2001` to `Amynthas cuneatus Hong & James, 2001`
[WARN] [2020-04-02 11:20:10] Filtered Scientific Name `Amynthas dabudongensis Hong &  James, 2009` to `Amynthas dabudongensis Hong & James, 2009`
[WARN] [2020-04-02 11:20:10] Filtered Scientific Name `Amynthas deogyusanensis Hong &  James, 2001` to `Amynthas deogyusanensis Hong & James, 2001`
[WARN] [2020-04-02 11:20:10] Filtered Scientific Name `Amynthas draconis Hong &  James, 2001` to `Amynthas draconis Hong & James, 2001`
[WARN] [2020-04-02 11:20:10] Filtered Scientific Name `Amynthas eastoni Hong &  James, 2001` to `Amynthas eastoni Hong & James, 2001`
[WARN] [2020-04-02 11:20:10] Filtered Scientific Name `Amynthas laceratus Hong &  James, 2009` to `Amynthas laceratus Hong & James, 2009`
[WARN] [2020-04-02 11:20:10] Filtered Scientific Name `Amynthas pyeongchangensis Hong &  James, 2009` to `Amynthas pyeongchangensis Hong & James, 2009`
[WARN] [2020-04-02 11:20:10] Filtered Scientific Name `Amynthas quinqueconvexus Hong &  James, 2009` to `Amynthas quinqueconvexus Hong & James, 2009`
[WARN] [2020-04-02 11:20:10] Filtered Scientific Name `Amynthas sonjaesiki Hong &  James, 2009` to `Amynthas sonjaesiki Hong & James, 2009`
[WARN] [2020-04-02 11:20:10] Filtered Scientific Name `Amynthas tabulatus Hong &  James, 2009` to `Amynthas tabulatus Hong & James, 2009`
[WARN] [2020-04-02 11:20:10] Filtered Scientific Name `Amynthas yunlongensis (Chen &  Zhifang, 1977)` to `Amynthas yunlongensis (Chen & Zhifang, 1977)`
[WARN] [2020-04-02 11:20:10] Filtered Scientific Name `Andiorrhinus motto Righi &  Araujo, 1999` to `Andiorrhinus motto Righi & Araujo, 1999`
[WARN] [2020-04-02 11:20:11] Filtered Scientific Name `Dichogaster dzwilloi Csuzdi &  Zicsi, 1989` to `Dichogaster dzwilloi Csuzdi & Zicsi, 1989`
[WARN] [2020-04-02 11:20:11] Filtered Scientific Name `Dichogaster graffi Csuzdi &  Zicsi, 1989` to `Dichogaster graffi Csuzdi & Zicsi, 1989`
[WARN] [2020-04-02 11:20:11] Filtered Scientific Name `Dichogaster meyaensis Csuzdi &  Zicsi, 1989` to `Dichogaster meyaensis Csuzdi & Zicsi, 1989`
[WARN] [2020-04-02 11:20:11] Filtered Scientific Name `Dichogaster pafuriensis Reinecke &  Ackerman, 1977` to `Dichogaster pafuriensis Reinecke & Ackerman, 1977`
[WARN] [2020-04-02 11:20:12] Filtered Scientific Name `Millsonia lamtoiana Omodeo &  Vaillaud, 1967` to `Millsonia lamtoiana Omodeo & Vaillaud, 1967`
[WARN] [2020-04-02 11:20:12] Filtered Scientific Name `Pheretima banauensis Hong &  James, 2008` to `Pheretima banauensis Hong & James, 2008`
[WARN] [2020-04-02 11:20:12] Filtered Scientific Name `Pheretima batoensis Hong &  James, 2009` to `Pheretima batoensis Hong & James, 2009`
[WARN] [2020-04-02 11:20:12] Filtered Scientific Name `Pheretima bicolensis Hong &  James, 2009` to `Pheretima bicolensis Hong & James, 2009`
[WARN] [2020-04-02 11:20:12] Filtered Scientific Name `Pheretima buhiensis Hong &  James, 2009` to `Pheretima buhiensis Hong & James, 2009`
[WARN] [2020-04-02 11:20:12] Filtered Scientific Name `Pheretima cabigati Hong &  James, 2008` to `Pheretima cabigati Hong & James, 2008`
[WARN] [2020-04-02 11:20:12] Filtered Scientific Name `Pheretima camarinensis Hong &  James, 2009` to `Pheretima camarinensis Hong & James, 2009`
[WARN] [2020-04-02 11:20:12] Filtered Scientific Name `Pheretima doriae Hong &  James, 2009` to `Pheretima doriae Hong & James, 2009`
[WARN] [2020-04-02 11:20:12] Filtered Scientific Name `Pheretima gorasi Hong &  James, 2009` to `Pheretima gorasi Hong & James, 2009`
[WARN] [2020-04-02 11:20:12] Filtered Scientific Name `Pheretima viracensis Hong &  James, 2009` to `Pheretima viracensis Hong & James, 2009`
[WARN] [2020-04-02 11:20:12] Filtered Scientific Name `Pithemera duhuani Hong &  James, 2008` to `Pithemera duhuani Hong & James, 2008`
[WARN] [2020-04-02 11:20:12] Filtered Scientific Name `Pithemera fragumae Hong &  James, 2008` to `Pithemera fragumae Hong & James, 2008`
[WARN] [2020-04-02 11:20:12] Filtered Scientific Name `Pithemera ifugaoensis Hong &  James, 2008` to `Pithemera ifugaoensis Hong & James, 2008`
[WARN] [2020-04-02 11:20:12] Filtered Scientific Name `Pithemera triangulata Hong &  James, 2008` to `Pithemera triangulata Hong & James, 2008`
[WARN] [2020-04-02 11:20:12] Filtered Scientific Name `Polypheretima bannaworensis Hong &  James, 2008` to `Polypheretima bannaworensis Hong & James, 2008`
[WARN] [2020-04-02 11:20:12] Filtered Scientific Name `Polypheretima fruticosa Hong &  James, 2008` to `Polypheretima fruticosa Hong & James, 2008`
[WARN] [2020-04-02 11:20:12] Filtered Scientific Name `Polypheretima pagudpudensis Hong &  James, 2011` to `Polypheretima pagudpudensis Hong & James, 2011`
[WARN] [2020-04-02 11:20:12] Filtered Scientific Name `Polypheretima perlucidula Hong &  James, 2008` to `Polypheretima perlucidula Hong & James, 2008`
[WARN] [2020-04-02 11:20:12] Filtered Scientific Name `Sparganophilus langi Qiu  & Bouché, 2000` to `Sparganophilus langi Qiu & Bouché, 2000`
[INFO] [2020-04-02 11:20:21] Storing 802601 ScientificNames
[INFO] [2020-04-02 11:20:21] Processing group of 802601 in 803 groups of 1000
[INFO] [2020-04-02 11:30:43] Average Time: 0.77
[INFO] [2020-04-02 11:30:43] Total Time: 10m22s
[INFO] [2020-04-02 11:30:43] last 3 / first 3: 0.66
[INFO] [2020-04-02 11:30:43] Std.Dev: 1.7770762504743571; Max: 11.26
[INFO] [2020-04-02 11:30:43] Storing 586204 Nodes
[INFO] [2020-04-02 11:30:43] Processing group of 586204 in 587 groups of 1000
[INFO] [2020-04-02 11:38:46] Average Time: 0.817
[INFO] [2020-04-02 11:38:46] Total Time: 8m3s
[INFO] [2020-04-02 11:38:46] last 3 / first 3: 1.16
[INFO] [2020-04-02 11:38:46] Std.Dev: 2.368121618498509; Max: 13.84
[STOP] [2020-04-02 11:38:46] parse_diff_and_store
[START] [2020-04-02 11:38:46] resolve_keys
[INFO] [2020-04-02 11:42:11] Occurrences to nodes (through scientific_names)...
[INFO] [2020-04-02 11:42:11] traits to occurrences...
[INFO] [2020-04-02 11:42:11] traits to nodes (through occurrences)...
[INFO] [2020-04-02 11:42:11] Traits to sex term...
[INFO] [2020-04-02 11:42:11] Traits to lifestage term...
[INFO] [2020-04-02 11:42:11] MetaTraits to traits...
[INFO] [2020-04-02 11:42:11] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-04-02 11:42:11] Assocs to occurrences...
[INFO] [2020-04-02 11:42:11] Assocs to nodes...
[INFO] [2020-04-02 11:42:11] Assoc to sex term...
[INFO] [2020-04-02 11:42:11] Assoc to lifestage term...
[STOP] [2020-04-02 11:42:11] resolve_keys
[START] [2020-04-02 11:42:11] hold_for_later_1
[STOP] [2020-04-02 11:42:11] hold_for_later_1
[START] [2020-04-02 11:42:11] hold_for_later_2
[STOP] [2020-04-02 11:42:11] hold_for_later_2
[START] [2020-04-02 11:42:11] resolve_missing_parents
[STOP] [2020-04-02 11:42:23] resolve_missing_parents
[START] [2020-04-02 11:42:23] rebuild_nodes
[START] [2020-04-02 11:42:23] Flattener#flatten
[START] [2020-04-02 11:42:23] Flattener#study_resource
[START] [2020-04-02 11:42:42] Flattener#build_ancestry
[STOP] [2020-04-02 12:34:53] Flattener#build_ancestry
[INFO] [2020-04-02 12:34:53] 586204 ancestry keys
[START] [2020-04-02 12:34:53] build_node_ancestors
[INFO] [2020-04-02 12:34:53] old ancestors deleted.
[STOP] [2020-04-02 12:53:47] build_node_ancestors
[START] [2020-04-02 12:53:51] Flattener#propagate_ancestor_ids
[STOP] [2020-04-02 13:00:59] Flattener#propagate_ancestor_ids
[STOP] [2020-04-02 13:00:59] Flattener#flatten
[STOP] [2020-04-02 13:00:59] rebuild_nodes
[START] [2020-04-02 13:00:59] resolve_missing_media_owners
[STOP] [2020-04-02 13:00:59] resolve_missing_media_owners
[START] [2020-04-02 13:00:59] sanitize_media_verbatims
[STOP] [2020-04-02 13:00:59] sanitize_media_verbatims
[START] [2020-04-02 13:00:59] queue_downloads
[STOP] [2020-04-02 13:00:59] queue_downloads
[START] [2020-04-02 13:00:59] parse_names
[WARN] [2020-04-02 13:01:01] I see 802601 names which still need to be parsed.
[WARN] [2020-04-02 13:11:51] I see 599 names which still need to be parsed.
[WARN] [2020-04-02 13:11:56] I see 51 names which still need to be parsed.
[WARN] [2020-04-02 13:12:01] I see 15 names which still need to be parsed.
[WARN] [2020-04-02 13:12:06] I see 8 names which still need to be parsed.
[WARN] [2020-04-02 13:12:10] I see 4 names which still need to be parsed.
[WARN] [2020-04-02 13:12:15] I see 2 names which still need to be parsed.
[WARN] [2020-04-02 13:12:20] I see 1 names which still need to be parsed.
[STOP] [2020-04-02 13:12:24] parse_names
[START] [2020-04-02 13:12:24] denormalize_canonical_names_to_nodes
[STOP] [2020-04-02 13:12:38] denormalize_canonical_names_to_nodes
[START] [2020-04-02 13:12:38] match_nodes
[START] [2020-04-02 13:12:39] map_all_nodes_to_pages
[STOP] [2020-04-03 08:45:00] map_all_nodes_to_pages
[INFO] [2020-04-03 08:45:00] 17006 Unmatched nodes (of 586204)! That's too many to output. First 10: Biliphyta (#68047921); Cyanidiophytina (#68047926); Cyanidiophyceae (#68047931); Galdieriaceae (#68148875); Bangiophyceae (#68047932); Goniotrichales (#67668443); Goniotrichaceae (#67668454); Goniotrichum elegans (#67668487); Goniotrichum ceramicola (#67668506); Bangia fusco-purpurea (#67669023)
[START] [2020-04-03 08:45:00] update_nodes
[STOP] [2020-04-03 08:45:37] update_nodes
[STOP] [2020-04-03 08:45:37] match_nodes
[START] [2020-04-03 08:45:37] reindex_search
[STOP] [2020-04-03 09:12:54] reindex_search
[START] [2020-04-03 09:12:54] normalize_units
[STOP] [2020-04-03 09:12:54] normalize_units
[START] [2020-04-03 09:12:54] calculate_statistics
[STOP] [2020-04-03 09:12:56] calculate_statistics
[START] [2020-04-03 09:12:56] complete_harvest_instance
[START] [2020-04-03 09:12:56] overall_tsv_creation
[INFO] [2020-04-03 09:12:57] Processing group of 586204 in 59 batches of 10000
[INFO] [2020-04-03 10:58:28] Average Time: 64.602
[INFO] [2020-04-03 10:58:28] Total Time: 1h45m32s
[INFO] [2020-04-03 10:58:28] last 3 / first 3: 0.93
[INFO] [2020-04-03 10:58:28] Std.Dev: 5.352102390649865; Max: 82.48
[STOP] [2020-04-03 10:58:28] overall_tsv_creation
[INFO] [2020-04-03 10:58:28] Done. Check your files:
[INFO] [2020-04-03 10:58:28] (586204 lines) /app/public/data/ihfifid/publish_nodes.tsv
[INFO] [2020-04-03 10:58:28] (7852220 lines) /app/public/data/ihfifid/publish_node_ancestors.tsv
[INFO] [2020-04-03 10:58:28] (802601 lines) /app/public/data/ihfifid/publish_scientific_names.tsv
[STOP] [2020-04-03 10:58:28] complete_harvest_instance
[START] [2020-04-03 10:58:28] completed
[STOP] [2020-04-03 10:58:28] completed
[STOP] [2020-04-03 10:58:28] logged process, took 85577.57

Latest Process