Harvest for Bailiwick of Jersey Species List Created 02 Oct 01:09

Stage: completed
Fetched: 02 Oct 01:09
Validated: 02 Oct 01:09
Deltas Created 02 Oct 01:09
Units Normalized: 02 Oct 01:11
Ancestry Built: 02 Oct 01:10
Nodes Matched: 02 Oct 01:11
Names Parsed: 02 Oct 01:10
New Models Stored: 02 Oct 01:10
Indexed: 02 Oct 01:11
Completed: 02 Oct 01:13
Time to Harvest: less than a minute

Expected File Format Definitions

Harvesting Log (most recent first)

# Logfile created on 2019-10-02 01:09:31 -0400 by logger.rb/56815
[START] [2019-10-02 01:09:31] logged process
[START] [2019-10-02 01:09:31] create_harvest_instance
[STOP] [2019-10-02 01:09:31] create_harvest_instance
[START] [2019-10-02 01:09:31] fetch_files
[STOP] [2019-10-02 01:09:31] fetch_files
[START] [2019-10-02 01:09:31] validate_each_file
[STOP] [2019-10-02 01:09:31] validate_each_file
[START] [2019-10-02 01:09:31] convert_to_csv
[CMD] [2019-10-02 01:09:31] /usr/bin/sort /app/public/converted_csv/bailiwick_jersey_refs_14888.csv > /app/public/converted_csv/bailiwick_jersey_refs_14888.csv_sorted
[CMD] [2019-10-02 01:09:33] /usr/bin/sort /app/public/converted_csv/bailiwick_jersey_nodes_14889.csv > /app/public/converted_csv/bailiwick_jersey_nodes_14889.csv_sorted
[CMD] [2019-10-02 01:09:34] /usr/bin/sort /app/public/converted_csv/bailiwick_jersey_occurrences_14890.csv > /app/public/converted_csv/bailiwick_jersey_occurrences_14890.csv_sorted
[CMD] [2019-10-02 01:09:36] /usr/bin/sort /app/public/converted_csv/bailiwick_jersey_measurements_14891.csv > /app/public/converted_csv/bailiwick_jersey_measurements_14891.csv_sorted
[STOP] [2019-10-02 01:09:38] convert_to_csv
[START] [2019-10-02 01:09:38] calculate_delta
[CMD] [2019-10-02 01:09:38] echo "0a" > /app/public/diff/bailiwick_jersey_refs_14888.diff
[CMD] [2019-10-02 01:09:39] tail -n +1 /app/public/converted_csv/bailiwick_jersey_refs_14888.csv >> /app/public/diff/bailiwick_jersey_refs_14888.diff
[CMD] [2019-10-02 01:09:41] echo "." >> /app/public/diff/bailiwick_jersey_refs_14888.diff
[CMD] [2019-10-02 01:09:42] echo "0a" > /app/public/diff/bailiwick_jersey_nodes_14889.diff
[CMD] [2019-10-02 01:09:44] tail -n +1 /app/public/converted_csv/bailiwick_jersey_nodes_14889.csv >> /app/public/diff/bailiwick_jersey_nodes_14889.diff
[CMD] [2019-10-02 01:09:45] echo "." >> /app/public/diff/bailiwick_jersey_nodes_14889.diff
[CMD] [2019-10-02 01:09:46] echo "0a" > /app/public/diff/bailiwick_jersey_occurrences_14890.diff
[CMD] [2019-10-02 01:09:48] tail -n +1 /app/public/converted_csv/bailiwick_jersey_occurrences_14890.csv >> /app/public/diff/bailiwick_jersey_occurrences_14890.diff
[CMD] [2019-10-02 01:09:50] echo "." >> /app/public/diff/bailiwick_jersey_occurrences_14890.diff
[CMD] [2019-10-02 01:09:51] echo "0a" > /app/public/diff/bailiwick_jersey_measurements_14891.diff
[CMD] [2019-10-02 01:09:53] tail -n +1 /app/public/converted_csv/bailiwick_jersey_measurements_14891.csv >> /app/public/diff/bailiwick_jersey_measurements_14891.diff
[CMD] [2019-10-02 01:09:54] echo "." >> /app/public/diff/bailiwick_jersey_measurements_14891.diff
[STOP] [2019-10-02 01:09:56] calculate_delta
[START] [2019-10-02 01:09:56] parse_diff_and_store
[INFO] [2019-10-02 01:09:57] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-02 01:09:59] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-02 01:10:01] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-02 01:10:02] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-02 01:10:05] Storing 2 References
[INFO] [2019-10-02 01:10:05] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-02 01:10:05] Average Time: 0.0
[INFO] [2019-10-02 01:10:05] Total Time: 1s
[INFO] [2019-10-02 01:10:05] Storing 1385 ScientificNames
[INFO] [2019-10-02 01:10:05] Processing group of 1385 in 2 groups of 1000
[INFO] [2019-10-02 01:10:06] Average Time: 0.26
[INFO] [2019-10-02 01:10:06] Total Time: 1s
[INFO] [2019-10-02 01:10:06] Storing 1385 Nodes
[INFO] [2019-10-02 01:10:06] Processing group of 1385 in 2 groups of 1000
[INFO] [2019-10-02 01:10:06] Average Time: 0.22
[INFO] [2019-10-02 01:10:06] Total Time: 1s
[INFO] [2019-10-02 01:10:06] Storing 482 Occurrences
[INFO] [2019-10-02 01:10:06] Processing group of 482 in 1 groups of 1000
[INFO] [2019-10-02 01:10:06] Average Time: 0.06
[INFO] [2019-10-02 01:10:06] Total Time: 1s
[INFO] [2019-10-02 01:10:06] Storing 1160 TraitsReferences
[INFO] [2019-10-02 01:10:06] Processing group of 1160 in 2 groups of 1000
[INFO] [2019-10-02 01:10:06] Average Time: 0.055
[INFO] [2019-10-02 01:10:06] Total Time: 1s
[INFO] [2019-10-02 01:10:06] Storing 1159 Traits
[INFO] [2019-10-02 01:10:06] Processing group of 1159 in 2 groups of 1000
[INFO] [2019-10-02 01:10:07] Average Time: 0.215
[INFO] [2019-10-02 01:10:07] Total Time: 1s
[INFO] [2019-10-02 01:10:07] Storing 1160 MetaTraits
[INFO] [2019-10-02 01:10:07] Processing group of 1160 in 2 groups of 1000
[INFO] [2019-10-02 01:10:07] Average Time: 0.075
[INFO] [2019-10-02 01:10:07] Total Time: 1s
[STOP] [2019-10-02 01:10:07] parse_diff_and_store
[START] [2019-10-02 01:10:07] resolve_keys
[INFO] [2019-10-02 01:10:16] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-02 01:10:16] traits to occurrences...
[INFO] [2019-10-02 01:10:17] traits to nodes (through occurrences)...
[INFO] [2019-10-02 01:10:17] Traits to sex term...
[INFO] [2019-10-02 01:10:17] Traits to lifestage term...
[INFO] [2019-10-02 01:10:18] MetaTraits to traits...
[INFO] [2019-10-02 01:10:18] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-02 01:10:18] Assocs to occurrences...
[INFO] [2019-10-02 01:10:18] Assocs to nodes...
[INFO] [2019-10-02 01:10:18] Assoc to sex term...
[INFO] [2019-10-02 01:10:18] Assoc to lifestage term...
[STOP] [2019-10-02 01:10:18] resolve_keys
[START] [2019-10-02 01:10:18] hold_for_later_1
[STOP] [2019-10-02 01:10:18] hold_for_later_1
[START] [2019-10-02 01:10:18] hold_for_later_2
[STOP] [2019-10-02 01:10:18] hold_for_later_2
[START] [2019-10-02 01:10:18] resolve_missing_parents
[STOP] [2019-10-02 01:10:19] resolve_missing_parents
[START] [2019-10-02 01:10:19] rebuild_nodes
[START] [2019-10-02 01:10:19] Flattener#flatten
[START] [2019-10-02 01:10:19] Flattener#study_resource
[START] [2019-10-02 01:10:19] Flattener#build_ancestry
[STOP] [2019-10-02 01:10:19] Flattener#build_ancestry
[INFO] [2019-10-02 01:10:19] 1385 ancestry keys
[START] [2019-10-02 01:10:19] build_node_ancestors
[INFO] [2019-10-02 01:10:19] old ancestors deleted.
[STOP] [2019-10-02 01:10:20] build_node_ancestors
[START] [2019-10-02 01:10:20] Flattener#propagate_ancestor_ids
[STOP] [2019-10-02 01:10:20] Flattener#propagate_ancestor_ids
[STOP] [2019-10-02 01:10:20] Flattener#flatten
[STOP] [2019-10-02 01:10:20] rebuild_nodes
[START] [2019-10-02 01:10:20] resolve_missing_media_owners
[STOP] [2019-10-02 01:10:20] resolve_missing_media_owners
[START] [2019-10-02 01:10:20] sanitize_media_verbatims
[STOP] [2019-10-02 01:10:20] sanitize_media_verbatims
[START] [2019-10-02 01:10:20] queue_downloads
[STOP] [2019-10-02 01:10:20] queue_downloads
[START] [2019-10-02 01:10:20] parse_names
[WARN] [2019-10-02 01:10:20] I see 1385 names which still need to be parsed.
[STOP] [2019-10-02 01:10:22] parse_names
[START] [2019-10-02 01:10:22] denormalize_canonical_names_to_nodes
[STOP] [2019-10-02 01:10:22] denormalize_canonical_names_to_nodes
[START] [2019-10-02 01:10:22] match_nodes
[START] [2019-10-02 01:10:22] map_all_nodes_to_pages
[STOP] [2019-10-02 01:11:29] map_all_nodes_to_pages
[INFO] [2019-10-02 01:11:29] 66 Unmatched nodes (of 1385)! That's too many to output. First 10: Anas clypeata (#47670198); Anas penelope (#47670348); Anas strepera (#47670486); Anas querquedula (#47670770); Carduelis cannabina (#47671454); Phylloscopus sibillatrix (#47670617); Thalaseus (#47670407); Thalaseus sandvicensis (#47670406); Philomachus pugnax (#47670452); Puffinus griseus (#47671027)
[START] [2019-10-02 01:11:29] update_nodes
[STOP] [2019-10-02 01:11:29] update_nodes
[STOP] [2019-10-02 01:11:29] match_nodes
[START] [2019-10-02 01:11:29] reindex_search
[STOP] [2019-10-02 01:11:32] reindex_search
[START] [2019-10-02 01:11:32] normalize_units
[STOP] [2019-10-02 01:11:32] normalize_units
[START] [2019-10-02 01:11:32] calculate_statistics
[STOP] [2019-10-02 01:11:32] calculate_statistics
[START] [2019-10-02 01:11:32] complete_harvest_instance
[START] [2019-10-02 01:11:32] overall_tsv_creation
[INFO] [2019-10-02 01:11:32] Processing group of 1385 in 1 batches of 10000
[INFO] [2019-10-02 01:12:21] 482 Traits (unfiltered)...
[INFO] [2019-10-02 01:12:34] 482 Traits (filtered)...
[INFO] [2019-10-02 01:12:34] 0 Associations (filtered)...
[INFO] [2019-10-02 01:13:10] 2410 metadata added.
[INFO] [2019-10-02 01:13:10] 0 metadata added.
[INFO] [2019-10-02 01:13:10] Average Time: 75.73
[INFO] [2019-10-02 01:13:10] Total Time: 1m38s
[STOP] [2019-10-02 01:13:10] overall_tsv_creation
[INFO] [2019-10-02 01:13:10] Done. Check your files:
[INFO] [2019-10-02 01:13:11] (1385 lines) /app/public/data/bailiwick_jersey/publish_nodes.tsv
[INFO] [2019-10-02 01:13:13] (2641 lines) /app/public/data/bailiwick_jersey/publish_node_ancestors.tsv
[INFO] [2019-10-02 01:13:14] (1385 lines) /app/public/data/bailiwick_jersey/publish_scientific_names.tsv
[INFO] [2019-10-02 01:13:16] (483 lines) /app/public/data/bailiwick_jersey/publish_traits.tsv
[INFO] [2019-10-02 01:13:18] (2411 lines) /app/public/data/bailiwick_jersey/publish_metadata.tsv
[STOP] [2019-10-02 01:13:18] complete_harvest_instance
[START] [2019-10-02 01:13:18] completed
[STOP] [2019-10-02 01:13:18] completed
[STOP] [2019-10-02 01:13:18] logged process, took 226.92

Latest Process