Harvest for Inner Seas off the West Coast of Scotland Species Created 23 Dec 10:39

Stage: completed
Fetched: 23 Dec 10:39
Validated: 23 Dec 10:39
Deltas Created 23 Dec 10:39
Units Normalized: 23 Dec 11:01
Ancestry Built: 23 Dec 10:40
Nodes Matched: 23 Dec 11:01
Names Parsed: 23 Dec 10:40
New Models Stored: 23 Dec 10:39
Indexed: 23 Dec 11:01
Completed: 23 Dec 11:04
Time to Harvest: less than a minute
Harvesting Log

(145 lines)
# Logfile created on 2019-12-23 10:39:05 -0500 by logger.rb/56815
[START] [2019-12-23 10:39:05] logged process
[START] [2019-12-23 10:39:05] create_harvest_instance
[STOP] [2019-12-23 10:39:05] create_harvest_instance
[START] [2019-12-23 10:39:05] fetch_files
[STOP] [2019-12-23 10:39:05] fetch_files
[START] [2019-12-23 10:39:06] validate_each_file
[STOP] [2019-12-23 10:39:06] validate_each_file
[START] [2019-12-23 10:39:06] convert_to_csv
[CMD] [2019-12-23 10:39:06] /usr/bin/sort /app/public/converted_csv/isowcss_refs_19378.csv > /app/public/converted_csv/isowcss_refs_19378.csv_sorted
[CMD] [2019-12-23 10:39:07] /usr/bin/sort /app/public/converted_csv/isowcss_nodes_19379.csv > /app/public/converted_csv/isowcss_nodes_19379.csv_sorted
[CMD] [2019-12-23 10:39:08] /usr/bin/sort /app/public/converted_csv/isowcss_occurrences_19380.csv > /app/public/converted_csv/isowcss_occurrences_19380.csv_sorted
[CMD] [2019-12-23 10:39:08] /usr/bin/sort /app/public/converted_csv/isowcss_measurements_19381.csv > /app/public/converted_csv/isowcss_measurements_19381.csv_sorted
[STOP] [2019-12-23 10:39:09] convert_to_csv
[START] [2019-12-23 10:39:09] calculate_delta
[CMD] [2019-12-23 10:39:09] echo "0a" > /app/public/diff/isowcss_refs_19378.diff
[CMD] [2019-12-23 10:39:10] tail -n +1 /app/public/converted_csv/isowcss_refs_19378.csv >> /app/public/diff/isowcss_refs_19378.diff
[CMD] [2019-12-23 10:39:10] echo "." >> /app/public/diff/isowcss_refs_19378.diff
[CMD] [2019-12-23 10:39:11] echo "0a" > /app/public/diff/isowcss_nodes_19379.diff
[CMD] [2019-12-23 10:39:12] tail -n +1 /app/public/converted_csv/isowcss_nodes_19379.csv >> /app/public/diff/isowcss_nodes_19379.diff
[CMD] [2019-12-23 10:39:12] echo "." >> /app/public/diff/isowcss_nodes_19379.diff
[CMD] [2019-12-23 10:39:13] echo "0a" > /app/public/diff/isowcss_occurrences_19380.diff
[CMD] [2019-12-23 10:39:14] tail -n +1 /app/public/converted_csv/isowcss_occurrences_19380.csv >> /app/public/diff/isowcss_occurrences_19380.diff
[CMD] [2019-12-23 10:39:14] echo "." >> /app/public/diff/isowcss_occurrences_19380.diff
[CMD] [2019-12-23 10:39:15] echo "0a" > /app/public/diff/isowcss_measurements_19381.diff
[CMD] [2019-12-23 10:39:16] tail -n +1 /app/public/converted_csv/isowcss_measurements_19381.csv >> /app/public/diff/isowcss_measurements_19381.diff
[CMD] [2019-12-23 10:39:16] echo "." >> /app/public/diff/isowcss_measurements_19381.diff
[STOP] [2019-12-23 10:39:17] calculate_delta
[START] [2019-12-23 10:39:17] parse_diff_and_store
[INFO] [2019-12-23 10:39:18] Loading refs diff file into memory (true lines)...
[INFO] [2019-12-23 10:39:18] Loading nodes diff file into memory (true lines)...
[INFO] [2019-12-23 10:39:21] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-12-23 10:39:23] Loading measurements diff file into memory (true lines)...
[INFO] [2019-12-23 10:39:40] Storing 2 References
[INFO] [2019-12-23 10:39:40] Processing group of 2 in 1 groups of 1000
[INFO] [2019-12-23 10:39:40] Average Time: 0.0
[INFO] [2019-12-23 10:39:40] Total Time: 1s
[INFO] [2019-12-23 10:39:40] Storing 5970 ScientificNames
[INFO] [2019-12-23 10:39:40] Processing group of 5970 in 6 groups of 1000
[INFO] [2019-12-23 10:39:43] Average Time: 0.41
[INFO] [2019-12-23 10:39:43] Total Time: 3s
[INFO] [2019-12-23 10:39:43] Storing 5970 Nodes
[INFO] [2019-12-23 10:39:43] Processing group of 5970 in 6 groups of 1000
[INFO] [2019-12-23 10:39:45] Average Time: 0.313
[INFO] [2019-12-23 10:39:45] Total Time: 2s
[INFO] [2019-12-23 10:39:45] Storing 3025 Occurrences
[INFO] [2019-12-23 10:39:45] Processing group of 3025 in 4 groups of 1000
[INFO] [2019-12-23 10:39:45] Average Time: 0.088
[INFO] [2019-12-23 10:39:45] Total Time: 1s
[INFO] [2019-12-23 10:39:45] Storing 6050 TraitsReferences
[INFO] [2019-12-23 10:39:45] Processing group of 6050 in 7 groups of 1000
[INFO] [2019-12-23 10:39:46] Average Time: 0.079
[INFO] [2019-12-23 10:39:46] Total Time: 1s
[INFO] [2019-12-23 10:39:46] last 3 / first 3: 0.55
[INFO] [2019-12-23 10:39:46] Std.Dev: 0.044721359549995794; Max: 0.16
[INFO] [2019-12-23 10:39:46] Storing 6050 Traits
[INFO] [2019-12-23 10:39:46] Processing group of 6050 in 7 groups of 1000
[INFO] [2019-12-23 10:39:48] Average Time: 0.296
[INFO] [2019-12-23 10:39:48] Total Time: 3s
[INFO] [2019-12-23 10:39:48] last 3 / first 3: 0.44
[INFO] [2019-12-23 10:39:48] Std.Dev: 0.17888543819998318; Max: 0.62
[INFO] [2019-12-23 10:39:48] Storing 6048 MetaTraits
[INFO] [2019-12-23 10:39:48] Processing group of 6048 in 7 groups of 1000
[INFO] [2019-12-23 10:39:48] Average Time: 0.103
[INFO] [2019-12-23 10:39:48] Total Time: 1s
[INFO] [2019-12-23 10:39:48] last 3 / first 3: 0.56
[INFO] [2019-12-23 10:39:48] Std.Dev: 0.044721359549995794; Max: 0.15
[STOP] [2019-12-23 10:39:48] parse_diff_and_store
[START] [2019-12-23 10:39:48] resolve_keys
[INFO] [2019-12-23 10:40:16] Occurrences to nodes (through scientific_names)...
[INFO] [2019-12-23 10:40:19] traits to occurrences...
[INFO] [2019-12-23 10:40:20] traits to nodes (through occurrences)...
[INFO] [2019-12-23 10:40:20] Traits to sex term...
[INFO] [2019-12-23 10:40:22] Traits to lifestage term...
[INFO] [2019-12-23 10:40:23] MetaTraits to traits...
[INFO] [2019-12-23 10:40:23] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-12-23 10:40:24] Assocs to occurrences...
[INFO] [2019-12-23 10:40:24] Assocs to nodes...
[INFO] [2019-12-23 10:40:24] Assoc to sex term...
[INFO] [2019-12-23 10:40:24] Assoc to lifestage term...
[STOP] [2019-12-23 10:40:24] resolve_keys
[START] [2019-12-23 10:40:24] hold_for_later_1
[STOP] [2019-12-23 10:40:24] hold_for_later_1
[START] [2019-12-23 10:40:24] hold_for_later_2
[STOP] [2019-12-23 10:40:24] hold_for_later_2
[START] [2019-12-23 10:40:24] resolve_missing_parents
[STOP] [2019-12-23 10:40:36] resolve_missing_parents
[START] [2019-12-23 10:40:36] rebuild_nodes
[START] [2019-12-23 10:40:36] Flattener#flatten
[START] [2019-12-23 10:40:36] Flattener#study_resource
[START] [2019-12-23 10:40:36] Flattener#build_ancestry
[STOP] [2019-12-23 10:40:36] Flattener#build_ancestry
[INFO] [2019-12-23 10:40:36] 5970 ancestry keys
[START] [2019-12-23 10:40:36] build_node_ancestors
[INFO] [2019-12-23 10:40:36] old ancestors deleted.
[STOP] [2019-12-23 10:40:38] build_node_ancestors
[START] [2019-12-23 10:40:41] Flattener#propagate_ancestor_ids
[STOP] [2019-12-23 10:40:42] Flattener#propagate_ancestor_ids
[STOP] [2019-12-23 10:40:42] Flattener#flatten
[STOP] [2019-12-23 10:40:42] rebuild_nodes
[START] [2019-12-23 10:40:42] resolve_missing_media_owners
[STOP] [2019-12-23 10:40:42] resolve_missing_media_owners
[START] [2019-12-23 10:40:42] sanitize_media_verbatims
[STOP] [2019-12-23 10:40:42] sanitize_media_verbatims
[START] [2019-12-23 10:40:42] queue_downloads
[STOP] [2019-12-23 10:40:42] queue_downloads
[START] [2019-12-23 10:40:42] parse_names
[WARN] [2019-12-23 10:40:42] I see 5970 names which still need to be parsed.
[STOP] [2019-12-23 10:40:47] parse_names
[START] [2019-12-23 10:40:47] denormalize_canonical_names_to_nodes
[STOP] [2019-12-23 10:40:47] denormalize_canonical_names_to_nodes
[START] [2019-12-23 10:40:47] match_nodes
[START] [2019-12-23 10:40:47] map_all_nodes_to_pages
[STOP] [2019-12-23 11:01:38] map_all_nodes_to_pages
[INFO] [2019-12-23 11:01:38] 316 Unmatched nodes (of 5970)! That's too many to output. First 10: Puffinus griseus (#61967610); Larus melanocephalus (#61971170); Larus ichthyaetus (#61971600); Philomachus (#61968375); Philomachus pugnax (#61968374); Anas penelope (#61967372); Anas clypeata (#61967848); Anas strepera (#61968274); Anas querquedula (#61969570); Anas americana (#61969774)
[START] [2019-12-23 11:01:38] update_nodes
[STOP] [2019-12-23 11:01:41] update_nodes
[STOP] [2019-12-23 11:01:41] match_nodes
[START] [2019-12-23 11:01:41] reindex_search
[STOP] [2019-12-23 11:01:58] reindex_search
[START] [2019-12-23 11:01:58] normalize_units
[STOP] [2019-12-23 11:01:58] normalize_units
[START] [2019-12-23 11:01:58] calculate_statistics
[STOP] [2019-12-23 11:01:58] calculate_statistics
[START] [2019-12-23 11:01:58] complete_harvest_instance
[START] [2019-12-23 11:01:58] overall_tsv_creation
[INFO] [2019-12-23 11:01:58] Processing group of 5970 in 1 batches of 10000
[INFO] [2019-12-23 11:03:19] 3025 Traits (unfiltered)...
[INFO] [2019-12-23 11:03:35] 3025 Traits (filtered)...
[INFO] [2019-12-23 11:03:35] 0 Associations (filtered)...
[INFO] [2019-12-23 11:04:25] 15123 metadata added.
[INFO] [2019-12-23 11:04:25] 0 metadata added.
[INFO] [2019-12-23 11:04:25] Average Time: 116.98
[INFO] [2019-12-23 11:04:25] Total Time: 2m27s
[STOP] [2019-12-23 11:04:25] overall_tsv_creation
[INFO] [2019-12-23 11:04:25] Done. Check your files:
[INFO] [2019-12-23 11:04:25] (5970 lines) /app/public/data/isowcss/publish_nodes.tsv
[INFO] [2019-12-23 11:04:26] (29654 lines) /app/public/data/isowcss/publish_node_ancestors.tsv
[INFO] [2019-12-23 11:04:26] (5970 lines) /app/public/data/isowcss/publish_scientific_names.tsv
[INFO] [2019-12-23 11:04:27] (3026 lines) /app/public/data/isowcss/publish_traits.tsv
[INFO] [2019-12-23 11:04:28] (15124 lines) /app/public/data/isowcss/publish_metadata.tsv
[STOP] [2019-12-23 11:04:28] complete_harvest_instance
[START] [2019-12-23 11:04:28] completed
[STOP] [2019-12-23 11:04:28] completed
[STOP] [2019-12-23 11:04:28] logged process, took 1523.01
Latest Process

23 Dec 10:39 EXITED currently processing 0