harvested resources
sign in

Harvest for Gulf of California Species List Created 23 Dec 08:08

Stage: completed
Fetched: 23 Dec 08:08
Validated: 23 Dec 08:08
Deltas Created 23 Dec 08:08
Units Normalized: 23 Dec 08:45
Ancestry Built: 23 Dec 08:10
Nodes Matched: 23 Dec 08:45
Names Parsed: 23 Dec 08:10
New Models Stored: 23 Dec 08:08
Indexed: 23 Dec 08:45
Completed: 23 Dec 08:48
Time to Harvest: 1 minute

Harvesting Log

(149 lines) 
# Logfile created on 2019-12-23 08:08:09 -0500 by logger.rb/56815
[START] [2019-12-23 08:08:09] logged process
[START] [2019-12-23 08:08:09] create_harvest_instance
[STOP] [2019-12-23 08:08:10] create_harvest_instance
[START] [2019-12-23 08:08:10] fetch_files
[STOP] [2019-12-23 08:08:10] fetch_files
[START] [2019-12-23 08:08:10] validate_each_file
[STOP] [2019-12-23 08:08:11] validate_each_file
[START] [2019-12-23 08:08:11] convert_to_csv
[CMD] [2019-12-23 08:08:11] /usr/bin/sort /app/public/converted_csv/gulf_california__refs_19299.csv > /app/public/converted_csv/gulf_california__refs_19299.csv_sorted
[CMD] [2019-12-23 08:08:11] /usr/bin/sort /app/public/converted_csv/gulf_california__nodes_19300.csv > /app/public/converted_csv/gulf_california__nodes_19300.csv_sorted
[CMD] [2019-12-23 08:08:11] /usr/bin/sort /app/public/converted_csv/gulf_california__occurrences_19301.csv > /app/public/converted_csv/gulf_california__occurrences_19301.csv_sorted
[CMD] [2019-12-23 08:08:11] /usr/bin/sort /app/public/converted_csv/gulf_california__measurements_19302.csv > /app/public/converted_csv/gulf_california__measurements_19302.csv_sorted
[STOP] [2019-12-23 08:08:11] convert_to_csv
[START] [2019-12-23 08:08:11] calculate_delta
[CMD] [2019-12-23 08:08:11] echo "0a" > /app/public/diff/gulf_california__refs_19299.diff
[CMD] [2019-12-23 08:08:11] tail -n +1 /app/public/converted_csv/gulf_california__refs_19299.csv >> /app/public/diff/gulf_california__refs_19299.diff
[CMD] [2019-12-23 08:08:11] echo "." >> /app/public/diff/gulf_california__refs_19299.diff
[CMD] [2019-12-23 08:08:11] echo "0a" > /app/public/diff/gulf_california__nodes_19300.diff
[CMD] [2019-12-23 08:08:12] tail -n +1 /app/public/converted_csv/gulf_california__nodes_19300.csv >> /app/public/diff/gulf_california__nodes_19300.diff
[CMD] [2019-12-23 08:08:12] echo "." >> /app/public/diff/gulf_california__nodes_19300.diff
[CMD] [2019-12-23 08:08:12] echo "0a" > /app/public/diff/gulf_california__occurrences_19301.diff
[CMD] [2019-12-23 08:08:12] tail -n +1 /app/public/converted_csv/gulf_california__occurrences_19301.csv >> /app/public/diff/gulf_california__occurrences_19301.diff
[CMD] [2019-12-23 08:08:12] echo "." >> /app/public/diff/gulf_california__occurrences_19301.diff
[CMD] [2019-12-23 08:08:12] echo "0a" > /app/public/diff/gulf_california__measurements_19302.diff
[CMD] [2019-12-23 08:08:12] tail -n +1 /app/public/converted_csv/gulf_california__measurements_19302.csv >> /app/public/diff/gulf_california__measurements_19302.diff
[CMD] [2019-12-23 08:08:12] echo "." >> /app/public/diff/gulf_california__measurements_19302.diff
[STOP] [2019-12-23 08:08:12] calculate_delta
[START] [2019-12-23 08:08:12] parse_diff_and_store
[INFO] [2019-12-23 08:08:12] Loading refs diff file into memory (true lines)...
[INFO] [2019-12-23 08:08:12] Loading nodes diff file into memory (true lines)...
[INFO] [2019-12-23 08:08:15] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-12-23 08:08:16] Loading measurements diff file into memory (true lines)...
[INFO] [2019-12-23 08:08:42] Storing 2 References
[INFO] [2019-12-23 08:08:42] Processing group of 2 in 1 groups of 1000
[INFO] [2019-12-23 08:08:42] Average Time: 0.0
[INFO] [2019-12-23 08:08:42] Total Time: 1s
[INFO] [2019-12-23 08:08:42] Storing 8415 ScientificNames
[INFO] [2019-12-23 08:08:42] Processing group of 8415 in 9 groups of 1000
[INFO] [2019-12-23 08:08:45] Average Time: 0.339
[INFO] [2019-12-23 08:08:45] Total Time: 4s
[INFO] [2019-12-23 08:08:45] last 3 / first 3: 0.72
[INFO] [2019-12-23 08:08:45] Std.Dev: 0.07745966692414834; Max: 0.39
[INFO] [2019-12-23 08:08:45] Storing 8415 Nodes
[INFO] [2019-12-23 08:08:45] Processing group of 8415 in 9 groups of 1000
[INFO] [2019-12-23 08:08:48] Average Time: 0.306
[INFO] [2019-12-23 08:08:48] Total Time: 3s
[INFO] [2019-12-23 08:08:48] last 3 / first 3: 0.87
[INFO] [2019-12-23 08:08:48] Std.Dev: 0.06324555320336758; Max: 0.36
[INFO] [2019-12-23 08:08:48] Storing 4808 Occurrences
[INFO] [2019-12-23 08:08:48] Processing group of 4808 in 5 groups of 1000
[INFO] [2019-12-23 08:08:49] Average Time: 0.114
[INFO] [2019-12-23 08:08:49] Total Time: 1s
[INFO] [2019-12-23 08:08:49] Storing 9616 TraitsReferences
[INFO] [2019-12-23 08:08:49] Processing group of 9616 in 10 groups of 1000
[INFO] [2019-12-23 08:08:50] Average Time: 0.076
[INFO] [2019-12-23 08:08:50] Total Time: 1s
[INFO] [2019-12-23 08:08:50] last 3 / first 3: 0.96
[INFO] [2019-12-23 08:08:50] Std.Dev: 0.0; Max: 0.08
[INFO] [2019-12-23 08:08:50] Storing 9616 Traits
[INFO] [2019-12-23 08:08:50] Processing group of 9616 in 10 groups of 1000
[INFO] [2019-12-23 08:08:53] Average Time: 0.331
[INFO] [2019-12-23 08:08:53] Total Time: 4s
[INFO] [2019-12-23 08:08:53] last 3 / first 3: 0.78
[INFO] [2019-12-23 08:08:53] Std.Dev: 0.044721359549995794; Max: 0.43
[INFO] [2019-12-23 08:08:53] Storing 9611 MetaTraits
[INFO] [2019-12-23 08:08:53] Processing group of 9611 in 10 groups of 1000
[INFO] [2019-12-23 08:08:55] Average Time: 0.173
[INFO] [2019-12-23 08:08:55] Total Time: 2s
[INFO] [2019-12-23 08:08:55] last 3 / first 3: 1.24
[INFO] [2019-12-23 08:08:55] Std.Dev: 0.044721359549995794; Max: 0.24
[STOP] [2019-12-23 08:08:55] parse_diff_and_store
[START] [2019-12-23 08:08:55] resolve_keys
[INFO] [2019-12-23 08:09:29] Occurrences to nodes (through scientific_names)...
[INFO] [2019-12-23 08:09:33] traits to occurrences...
[INFO] [2019-12-23 08:09:35] traits to nodes (through occurrences)...
[INFO] [2019-12-23 08:09:35] Traits to sex term...
[INFO] [2019-12-23 08:09:37] Traits to lifestage term...
[INFO] [2019-12-23 08:09:39] MetaTraits to traits...
[INFO] [2019-12-23 08:09:39] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-12-23 08:09:41] Assocs to occurrences...
[INFO] [2019-12-23 08:09:41] Assocs to nodes...
[INFO] [2019-12-23 08:09:41] Assoc to sex term...
[INFO] [2019-12-23 08:09:41] Assoc to lifestage term...
[STOP] [2019-12-23 08:09:41] resolve_keys
[START] [2019-12-23 08:09:41] hold_for_later_1
[STOP] [2019-12-23 08:09:41] hold_for_later_1
[START] [2019-12-23 08:09:41] hold_for_later_2
[STOP] [2019-12-23 08:09:41] hold_for_later_2
[START] [2019-12-23 08:09:41] resolve_missing_parents
[STOP] [2019-12-23 08:09:58] resolve_missing_parents
[START] [2019-12-23 08:09:58] rebuild_nodes
[START] [2019-12-23 08:09:58] Flattener#flatten
[START] [2019-12-23 08:09:58] Flattener#study_resource
[START] [2019-12-23 08:09:58] Flattener#build_ancestry
[STOP] [2019-12-23 08:09:59] Flattener#build_ancestry
[INFO] [2019-12-23 08:09:59] 8415 ancestry keys
[START] [2019-12-23 08:09:59] build_node_ancestors
[INFO] [2019-12-23 08:09:59] old ancestors deleted.
[STOP] [2019-12-23 08:10:02] build_node_ancestors
[START] [2019-12-23 08:10:06] Flattener#propagate_ancestor_ids
[STOP] [2019-12-23 08:10:07] Flattener#propagate_ancestor_ids
[STOP] [2019-12-23 08:10:07] Flattener#flatten
[STOP] [2019-12-23 08:10:07] rebuild_nodes
[START] [2019-12-23 08:10:07] resolve_missing_media_owners
[STOP] [2019-12-23 08:10:07] resolve_missing_media_owners
[START] [2019-12-23 08:10:07] sanitize_media_verbatims
[STOP] [2019-12-23 08:10:07] sanitize_media_verbatims
[START] [2019-12-23 08:10:07] queue_downloads
[STOP] [2019-12-23 08:10:07] queue_downloads
[START] [2019-12-23 08:10:07] parse_names
[WARN] [2019-12-23 08:10:07] I see 8415 names which still need to be parsed.
[STOP] [2019-12-23 08:10:15] parse_names
[START] [2019-12-23 08:10:15] denormalize_canonical_names_to_nodes
[STOP] [2019-12-23 08:10:15] denormalize_canonical_names_to_nodes
[START] [2019-12-23 08:10:15] match_nodes
[START] [2019-12-23 08:10:15] map_all_nodes_to_pages
[STOP] [2019-12-23 08:45:12] map_all_nodes_to_pages
[INFO] [2019-12-23 08:45:12] 519 Unmatched nodes (of 8415)! That's too many to output. First 10: Larus thayeri (#61916039); Thalasseus maxima (#61910231); Tringa semipalmatus (#61910145); Limnodromus (#61910230); Aphriza (#61912224); Aphriza virgata (#61912223); Anas clypeata (#61910007); Anas americana (#61910141); Anas cyanoptera (#61910217); Anas discors (#61910493)
[START] [2019-12-23 08:45:12] update_nodes
[STOP] [2019-12-23 08:45:16] update_nodes
[STOP] [2019-12-23 08:45:16] match_nodes
[START] [2019-12-23 08:45:16] reindex_search
[STOP] [2019-12-23 08:45:45] reindex_search
[START] [2019-12-23 08:45:45] normalize_units
[STOP] [2019-12-23 08:45:45] normalize_units
[START] [2019-12-23 08:45:45] calculate_statistics
[STOP] [2019-12-23 08:45:45] calculate_statistics
[START] [2019-12-23 08:45:45] complete_harvest_instance
[START] [2019-12-23 08:45:45] overall_tsv_creation
[INFO] [2019-12-23 08:45:45] Processing group of 8415 in 1 batches of 10000
[INFO] [2019-12-23 08:47:11] 4808 Traits (unfiltered)...
[INFO] [2019-12-23 08:47:24] 4808 Traits (filtered)...
[INFO] [2019-12-23 08:47:24] 0 Associations (filtered)...
[INFO] [2019-12-23 08:48:13] 24035 metadata added.
[INFO] [2019-12-23 08:48:13] 0 metadata added.
[INFO] [2019-12-23 08:48:13] Average Time: 118.95
[INFO] [2019-12-23 08:48:13] Total Time: 2m28s
[STOP] [2019-12-23 08:48:13] overall_tsv_creation
[INFO] [2019-12-23 08:48:13] Done. Check your files:
[INFO] [2019-12-23 08:48:13] (8415 lines) /app/public/data/gulf_california_/publish_nodes.tsv
[INFO] [2019-12-23 08:48:13] (43441 lines) /app/public/data/gulf_california_/publish_node_ancestors.tsv
[INFO] [2019-12-23 08:48:13] (8415 lines) /app/public/data/gulf_california_/publish_scientific_names.tsv
[INFO] [2019-12-23 08:48:13] (4809 lines) /app/public/data/gulf_california_/publish_traits.tsv
[INFO] [2019-12-23 08:48:13] (24036 lines) /app/public/data/gulf_california_/publish_metadata.tsv
[STOP] [2019-12-23 08:48:13] complete_harvest_instance
[START] [2019-12-23 08:48:13] completed
[STOP] [2019-12-23 08:48:13] completed
[STOP] [2019-12-23 08:48:13] logged process, took 2403.92

Latest Process