Harvest for Carnivorous Plants Database Created 31 Oct 13:53

Stage: completed
Fetched: 31 Oct 13:53
Validated: 31 Oct 13:53
Deltas Created 31 Oct 13:53
Units Normalized: 31 Oct 13:54
Ancestry Built: 31 Oct 13:53
Nodes Matched: 31 Oct 13:54
Names Parsed: 31 Oct 13:53
New Models Stored: 31 Oct 13:53
Indexed: 31 Oct 13:54
Completed: 31 Oct 13:55
Time to Harvest: less than a minute

Harvesting Log

(143 lines)
# Logfile created on 2019-10-31 13:53:20 -0400 by logger.rb/56815
[START] [2019-10-31 13:53:20] logged process
[START] [2019-10-31 13:53:20] create_harvest_instance
[STOP] [2019-10-31 13:53:21] create_harvest_instance
[START] [2019-10-31 13:53:21] fetch_files
[STOP] [2019-10-31 13:53:21] fetch_files
[START] [2019-10-31 13:53:21] validate_each_file
[STOP] [2019-10-31 13:53:21] validate_each_file
[START] [2019-10-31 13:53:21] convert_to_csv
[CMD] [2019-10-31 13:53:21] /usr/bin/sort /app/public/converted_csv/carnivorous_pl_d_refs_17932.csv > /app/public/converted_csv/carnivorous_pl_d_refs_17932.csv_sorted
[CMD] [2019-10-31 13:53:21] /usr/bin/sort /app/public/converted_csv/carnivorous_pl_d_nodes_17933.csv > /app/public/converted_csv/carnivorous_pl_d_nodes_17933.csv_sorted
[CMD] [2019-10-31 13:53:22] /usr/bin/sort /app/public/converted_csv/carnivorous_pl_d_occurrences_17934.csv > /app/public/converted_csv/carnivorous_pl_d_occurrences_17934.csv_sorted
[CMD] [2019-10-31 13:53:22] /usr/bin/sort /app/public/converted_csv/carnivorous_pl_d_measurements_17935.csv > /app/public/converted_csv/carnivorous_pl_d_measurements_17935.csv_sorted
[STOP] [2019-10-31 13:53:22] convert_to_csv
[START] [2019-10-31 13:53:22] calculate_delta
[CMD] [2019-10-31 13:53:22] echo "0a" > /app/public/diff/carnivorous_pl_d_refs_17932.diff
[CMD] [2019-10-31 13:53:22] tail -n +1 /app/public/converted_csv/carnivorous_pl_d_refs_17932.csv >> /app/public/diff/carnivorous_pl_d_refs_17932.diff
[CMD] [2019-10-31 13:53:22] echo "." >> /app/public/diff/carnivorous_pl_d_refs_17932.diff
[CMD] [2019-10-31 13:53:23] echo "0a" > /app/public/diff/carnivorous_pl_d_nodes_17933.diff
[CMD] [2019-10-31 13:53:23] tail -n +1 /app/public/converted_csv/carnivorous_pl_d_nodes_17933.csv >> /app/public/diff/carnivorous_pl_d_nodes_17933.diff
[CMD] [2019-10-31 13:53:23] echo "." >> /app/public/diff/carnivorous_pl_d_nodes_17933.diff
[CMD] [2019-10-31 13:53:23] echo "0a" > /app/public/diff/carnivorous_pl_d_occurrences_17934.diff
[CMD] [2019-10-31 13:53:24] tail -n +1 /app/public/converted_csv/carnivorous_pl_d_occurrences_17934.csv >> /app/public/diff/carnivorous_pl_d_occurrences_17934.diff
[CMD] [2019-10-31 13:53:24] echo "." >> /app/public/diff/carnivorous_pl_d_occurrences_17934.diff
[CMD] [2019-10-31 13:53:24] echo "0a" > /app/public/diff/carnivorous_pl_d_measurements_17935.diff
[CMD] [2019-10-31 13:53:24] tail -n +1 /app/public/converted_csv/carnivorous_pl_d_measurements_17935.csv >> /app/public/diff/carnivorous_pl_d_measurements_17935.diff
[CMD] [2019-10-31 13:53:24] echo "." >> /app/public/diff/carnivorous_pl_d_measurements_17935.diff
[STOP] [2019-10-31 13:53:25] calculate_delta
[START] [2019-10-31 13:53:25] parse_diff_and_store
[INFO] [2019-10-31 13:53:25] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-31 13:53:25] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-31 13:53:26] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-31 13:53:27] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-31 13:53:33] Storing 663 References
[INFO] [2019-10-31 13:53:33] Processing group of 663 in 1 groups of 1000
[INFO] [2019-10-31 13:53:33] Average Time: 0.13
[INFO] [2019-10-31 13:53:33] Total Time: 1s
[INFO] [2019-10-31 13:53:33] Storing 675 ScientificNames
[INFO] [2019-10-31 13:53:33] Processing group of 675 in 1 groups of 1000
[INFO] [2019-10-31 13:53:33] Average Time: 0.3
[INFO] [2019-10-31 13:53:33] Total Time: 1s
[INFO] [2019-10-31 13:53:33] Storing 675 Nodes
[INFO] [2019-10-31 13:53:33] Processing group of 675 in 1 groups of 1000
[INFO] [2019-10-31 13:53:33] Average Time: 0.22
[INFO] [2019-10-31 13:53:33] Total Time: 1s
[INFO] [2019-10-31 13:53:33] Storing 663 Occurrences
[INFO] [2019-10-31 13:53:33] Processing group of 663 in 1 groups of 1000
[INFO] [2019-10-31 13:53:34] Average Time: 0.07
[INFO] [2019-10-31 13:53:34] Total Time: 1s
[INFO] [2019-10-31 13:53:34] Storing 1259 OccurrenceMetadata
[INFO] [2019-10-31 13:53:34] Processing group of 1259 in 2 groups of 1000
[INFO] [2019-10-31 13:53:34] Average Time: 0.075
[INFO] [2019-10-31 13:53:34] Total Time: 1s
[INFO] [2019-10-31 13:53:34] Storing 1274 TraitsReferences
[INFO] [2019-10-31 13:53:34] Processing group of 1274 in 2 groups of 1000
[INFO] [2019-10-31 13:53:34] Average Time: 0.08
[INFO] [2019-10-31 13:53:34] Total Time: 1s
[INFO] [2019-10-31 13:53:34] Storing 1274 Traits
[INFO] [2019-10-31 13:53:34] Processing group of 1274 in 2 groups of 1000
[INFO] [2019-10-31 13:53:34] Average Time: 0.195
[INFO] [2019-10-31 13:53:34] Total Time: 1s
[INFO] [2019-10-31 13:53:34] Storing 2620 MetaTraits
[INFO] [2019-10-31 13:53:34] Processing group of 2620 in 3 groups of 1000
[INFO] [2019-10-31 13:53:35] Average Time: 0.11
[INFO] [2019-10-31 13:53:35] Total Time: 1s
[STOP] [2019-10-31 13:53:35] parse_diff_and_store
[START] [2019-10-31 13:53:35] resolve_keys
[INFO] [2019-10-31 13:53:39] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-31 13:53:39] traits to occurrences...
[INFO] [2019-10-31 13:53:39] traits to nodes (through occurrences)...
[INFO] [2019-10-31 13:53:39] Traits to sex term...
[INFO] [2019-10-31 13:53:39] Traits to lifestage term...
[INFO] [2019-10-31 13:53:39] MetaTraits to traits...
[INFO] [2019-10-31 13:53:39] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-31 13:53:39] Assocs to occurrences...
[INFO] [2019-10-31 13:53:39] Assocs to nodes...
[INFO] [2019-10-31 13:53:39] Assoc to sex term...
[INFO] [2019-10-31 13:53:39] Assoc to lifestage term...
[STOP] [2019-10-31 13:53:39] resolve_keys
[START] [2019-10-31 13:53:39] hold_for_later_1
[STOP] [2019-10-31 13:53:39] hold_for_later_1
[START] [2019-10-31 13:53:39] hold_for_later_2
[STOP] [2019-10-31 13:53:39] hold_for_later_2
[START] [2019-10-31 13:53:39] resolve_missing_parents
[STOP] [2019-10-31 13:53:39] resolve_missing_parents
[START] [2019-10-31 13:53:39] rebuild_nodes
[START] [2019-10-31 13:53:39] Flattener#flatten
[START] [2019-10-31 13:53:39] Flattener#study_resource
[START] [2019-10-31 13:53:39] Flattener#build_ancestry
[STOP] [2019-10-31 13:53:39] Flattener#build_ancestry
[INFO] [2019-10-31 13:53:39] 675 ancestry keys
[START] [2019-10-31 13:53:39] build_node_ancestors
[INFO] [2019-10-31 13:53:39] old ancestors deleted.
[STOP] [2019-10-31 13:53:39] build_node_ancestors
[START] [2019-10-31 13:53:39] Flattener#propagate_ancestor_ids
[STOP] [2019-10-31 13:53:39] Flattener#propagate_ancestor_ids
[STOP] [2019-10-31 13:53:39] Flattener#flatten
[STOP] [2019-10-31 13:53:39] rebuild_nodes
[START] [2019-10-31 13:53:39] resolve_missing_media_owners
[STOP] [2019-10-31 13:53:39] resolve_missing_media_owners
[START] [2019-10-31 13:53:39] sanitize_media_verbatims
[STOP] [2019-10-31 13:53:39] sanitize_media_verbatims
[START] [2019-10-31 13:53:39] queue_downloads
[STOP] [2019-10-31 13:53:39] queue_downloads
[START] [2019-10-31 13:53:39] parse_names
[WARN] [2019-10-31 13:53:39] I see 675 names which still need to be parsed.
[STOP] [2019-10-31 13:53:41] parse_names
[START] [2019-10-31 13:53:41] denormalize_canonical_names_to_nodes
[STOP] [2019-10-31 13:53:41] denormalize_canonical_names_to_nodes
[START] [2019-10-31 13:53:41] match_nodes
[START] [2019-10-31 13:53:41] map_all_nodes_to_pages
[STOP] [2019-10-31 13:54:10] map_all_nodes_to_pages
[INFO] [2019-10-31 13:54:10] 15 Unmatched nodes (of 675)! That's too many to output. First 10: Heliamphora ionae (#53946873); Heliamphora tatei macdonaldiae (#53946975); Drosera spatulata bakoensis (#53946848); Drosera verrucata (#53946871); Drosera yilgarnensis (#53946878); Drosera aquatica (#53947476); Drosera aurantiaca (#53947481); Nepenthes abalata (#53946889); Nepenthes abgracilis (#53946890); Nepenthes alzapan (#53946894)
[START] [2019-10-31 13:54:10] update_nodes
[STOP] [2019-10-31 13:54:11] update_nodes
[STOP] [2019-10-31 13:54:11] match_nodes
[START] [2019-10-31 13:54:11] reindex_search
[STOP] [2019-10-31 13:54:12] reindex_search
[START] [2019-10-31 13:54:12] normalize_units
[STOP] [2019-10-31 13:54:12] normalize_units
[START] [2019-10-31 13:54:12] calculate_statistics
[STOP] [2019-10-31 13:54:12] calculate_statistics
[START] [2019-10-31 13:54:12] complete_harvest_instance
[START] [2019-10-31 13:54:12] overall_tsv_creation
[INFO] [2019-10-31 13:54:12] Processing group of 675 in 1 batches of 10000
[INFO] [2019-10-31 13:54:58] 1274 Traits (unfiltered)...
[INFO] [2019-10-31 13:55:11] 1274 Traits (filtered)...
[INFO] [2019-10-31 13:55:11] 0 Associations (filtered)...
[INFO] [2019-10-31 13:55:49] 6314 metadata added.
[INFO] [2019-10-31 13:55:49] 0 metadata added.
[INFO] [2019-10-31 13:55:49] Average Time: 75.95
[INFO] [2019-10-31 13:55:49] Total Time: 1m38s
[STOP] [2019-10-31 13:55:49] overall_tsv_creation
[INFO] [2019-10-31 13:55:49] Done. Check your files:
[INFO] [2019-10-31 13:55:50] (675 lines) /app/public/data/carnivorous_pl_d/publish_nodes.tsv
[INFO] [2019-10-31 13:55:50] (1336 lines) /app/public/data/carnivorous_pl_d/publish_node_ancestors.tsv
[INFO] [2019-10-31 13:55:50] (675 lines) /app/public/data/carnivorous_pl_d/publish_scientific_names.tsv
[INFO] [2019-10-31 13:55:50] (1275 lines) /app/public/data/carnivorous_pl_d/publish_traits.tsv
[INFO] [2019-10-31 13:55:50] (6315 lines) /app/public/data/carnivorous_pl_d/publish_metadata.tsv
[STOP] [2019-10-31 13:55:50] complete_harvest_instance
[START] [2019-10-31 13:55:50] completed
[STOP] [2019-10-31 13:55:50] completed
[STOP] [2019-10-31 13:55:50] logged process, took 149.92

Latest Process