Harvest for Gulf of Finland Species List Created 23 Dec 08:08

Stage: completed
Fetched: 23 Dec 08:08
Validated: 23 Dec 08:08
Deltas Created 23 Dec 08:08
Units Normalized: 23 Dec 08:13
Ancestry Built: 23 Dec 08:08
Nodes Matched: 23 Dec 08:13
Names Parsed: 23 Dec 08:08
New Models Stored: 23 Dec 08:08
Indexed: 23 Dec 08:13
Completed: 23 Dec 08:15
Time to Harvest: less than a minute
Harvesting Log

(139 lines)
# Logfile created on 2019-12-23 08:08:14 -0500 by logger.rb/56815
[START] [2019-12-23 08:08:14] logged process
[START] [2019-12-23 08:08:14] create_harvest_instance
[STOP] [2019-12-23 08:08:15] create_harvest_instance
[START] [2019-12-23 08:08:15] fetch_files
[STOP] [2019-12-23 08:08:15] fetch_files
[START] [2019-12-23 08:08:15] validate_each_file
[STOP] [2019-12-23 08:08:15] validate_each_file
[START] [2019-12-23 08:08:15] convert_to_csv
[CMD] [2019-12-23 08:08:15] /usr/bin/sort /app/public/converted_csv/gulf_finland_sp__refs_19306.csv > /app/public/converted_csv/gulf_finland_sp__refs_19306.csv_sorted
[CMD] [2019-12-23 08:08:16] /usr/bin/sort /app/public/converted_csv/gulf_finland_sp__nodes_19307.csv > /app/public/converted_csv/gulf_finland_sp__nodes_19307.csv_sorted
[CMD] [2019-12-23 08:08:16] /usr/bin/sort /app/public/converted_csv/gulf_finland_sp__occurrences_19308.csv > /app/public/converted_csv/gulf_finland_sp__occurrences_19308.csv_sorted
[CMD] [2019-12-23 08:08:17] /usr/bin/sort /app/public/converted_csv/gulf_finland_sp__measurements_19309.csv > /app/public/converted_csv/gulf_finland_sp__measurements_19309.csv_sorted
[STOP] [2019-12-23 08:08:18] convert_to_csv
[START] [2019-12-23 08:08:18] calculate_delta
[CMD] [2019-12-23 08:08:18] echo "0a" > /app/public/diff/gulf_finland_sp__refs_19306.diff
[CMD] [2019-12-23 08:08:18] tail -n +1 /app/public/converted_csv/gulf_finland_sp__refs_19306.csv >> /app/public/diff/gulf_finland_sp__refs_19306.diff
[CMD] [2019-12-23 08:08:19] echo "." >> /app/public/diff/gulf_finland_sp__refs_19306.diff
[CMD] [2019-12-23 08:08:20] echo "0a" > /app/public/diff/gulf_finland_sp__nodes_19307.diff
[CMD] [2019-12-23 08:08:20] tail -n +1 /app/public/converted_csv/gulf_finland_sp__nodes_19307.csv >> /app/public/diff/gulf_finland_sp__nodes_19307.diff
[CMD] [2019-12-23 08:08:21] echo "." >> /app/public/diff/gulf_finland_sp__nodes_19307.diff
[CMD] [2019-12-23 08:08:21] echo "0a" > /app/public/diff/gulf_finland_sp__occurrences_19308.diff
[CMD] [2019-12-23 08:08:22] tail -n +1 /app/public/converted_csv/gulf_finland_sp__occurrences_19308.csv >> /app/public/diff/gulf_finland_sp__occurrences_19308.diff
[CMD] [2019-12-23 08:08:23] echo "." >> /app/public/diff/gulf_finland_sp__occurrences_19308.diff
[CMD] [2019-12-23 08:08:23] echo "0a" > /app/public/diff/gulf_finland_sp__measurements_19309.diff
[CMD] [2019-12-23 08:08:24] tail -n +1 /app/public/converted_csv/gulf_finland_sp__measurements_19309.csv >> /app/public/diff/gulf_finland_sp__measurements_19309.diff
[CMD] [2019-12-23 08:08:25] echo "." >> /app/public/diff/gulf_finland_sp__measurements_19309.diff
[STOP] [2019-12-23 08:08:25] calculate_delta
[START] [2019-12-23 08:08:25] parse_diff_and_store
[INFO] [2019-12-23 08:08:26] Loading refs diff file into memory (true lines)...
[INFO] [2019-12-23 08:08:26] Loading nodes diff file into memory (true lines)...
[INFO] [2019-12-23 08:08:28] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-12-23 08:08:28] Loading measurements diff file into memory (true lines)...
[INFO] [2019-12-23 08:08:31] Storing 2 References
[INFO] [2019-12-23 08:08:31] Processing group of 2 in 1 groups of 1000
[INFO] [2019-12-23 08:08:31] Average Time: 0.0
[INFO] [2019-12-23 08:08:31] Total Time: 1s
[INFO] [2019-12-23 08:08:31] Storing 1156 ScientificNames
[INFO] [2019-12-23 08:08:31] Processing group of 1156 in 2 groups of 1000
[INFO] [2019-12-23 08:08:31] Average Time: 0.27
[INFO] [2019-12-23 08:08:31] Total Time: 1s
[INFO] [2019-12-23 08:08:31] Storing 1156 Nodes
[INFO] [2019-12-23 08:08:31] Processing group of 1156 in 2 groups of 1000
[INFO] [2019-12-23 08:08:32] Average Time: 0.245
[INFO] [2019-12-23 08:08:32] Total Time: 1s
[INFO] [2019-12-23 08:08:32] Storing 459 Occurrences
[INFO] [2019-12-23 08:08:32] Processing group of 459 in 1 groups of 1000
[INFO] [2019-12-23 08:08:32] Average Time: 0.08
[INFO] [2019-12-23 08:08:32] Total Time: 1s
[INFO] [2019-12-23 08:08:32] Storing 918 TraitsReferences
[INFO] [2019-12-23 08:08:32] Processing group of 918 in 1 groups of 1000
[INFO] [2019-12-23 08:08:32] Average Time: 0.16
[INFO] [2019-12-23 08:08:32] Total Time: 1s
[INFO] [2019-12-23 08:08:32] Storing 918 Traits
[INFO] [2019-12-23 08:08:32] Processing group of 918 in 1 groups of 1000
[INFO] [2019-12-23 08:08:32] Average Time: 0.41
[INFO] [2019-12-23 08:08:32] Total Time: 1s
[INFO] [2019-12-23 08:08:32] Storing 916 MetaTraits
[INFO] [2019-12-23 08:08:32] Processing group of 916 in 1 groups of 1000
[INFO] [2019-12-23 08:08:33] Average Time: 0.16
[INFO] [2019-12-23 08:08:33] Total Time: 1s
[STOP] [2019-12-23 08:08:33] parse_diff_and_store
[START] [2019-12-23 08:08:33] resolve_keys
[INFO] [2019-12-23 08:08:40] Occurrences to nodes (through scientific_names)...
[INFO] [2019-12-23 08:08:41] traits to occurrences...
[INFO] [2019-12-23 08:08:41] traits to nodes (through occurrences)...
[INFO] [2019-12-23 08:08:41] Traits to sex term...
[INFO] [2019-12-23 08:08:41] Traits to lifestage term...
[INFO] [2019-12-23 08:08:41] MetaTraits to traits...
[INFO] [2019-12-23 08:08:41] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-12-23 08:08:42] Assocs to occurrences...
[INFO] [2019-12-23 08:08:42] Assocs to nodes...
[INFO] [2019-12-23 08:08:42] Assoc to sex term...
[INFO] [2019-12-23 08:08:42] Assoc to lifestage term...
[STOP] [2019-12-23 08:08:42] resolve_keys
[START] [2019-12-23 08:08:42] hold_for_later_1
[STOP] [2019-12-23 08:08:42] hold_for_later_1
[START] [2019-12-23 08:08:42] hold_for_later_2
[STOP] [2019-12-23 08:08:42] hold_for_later_2
[START] [2019-12-23 08:08:42] resolve_missing_parents
[STOP] [2019-12-23 08:08:43] resolve_missing_parents
[START] [2019-12-23 08:08:43] rebuild_nodes
[START] [2019-12-23 08:08:43] Flattener#flatten
[START] [2019-12-23 08:08:43] Flattener#study_resource
[START] [2019-12-23 08:08:43] Flattener#build_ancestry
[STOP] [2019-12-23 08:08:43] Flattener#build_ancestry
[INFO] [2019-12-23 08:08:43] 1156 ancestry keys
[START] [2019-12-23 08:08:43] build_node_ancestors
[INFO] [2019-12-23 08:08:43] old ancestors deleted.
[STOP] [2019-12-23 08:08:43] build_node_ancestors
[START] [2019-12-23 08:08:43] Flattener#propagate_ancestor_ids
[STOP] [2019-12-23 08:08:44] Flattener#propagate_ancestor_ids
[STOP] [2019-12-23 08:08:44] Flattener#flatten
[STOP] [2019-12-23 08:08:44] rebuild_nodes
[START] [2019-12-23 08:08:44] resolve_missing_media_owners
[STOP] [2019-12-23 08:08:44] resolve_missing_media_owners
[START] [2019-12-23 08:08:44] sanitize_media_verbatims
[STOP] [2019-12-23 08:08:44] sanitize_media_verbatims
[START] [2019-12-23 08:08:44] queue_downloads
[STOP] [2019-12-23 08:08:44] queue_downloads
[START] [2019-12-23 08:08:44] parse_names
[WARN] [2019-12-23 08:08:44] I see 1156 names which still need to be parsed.
[STOP] [2019-12-23 08:08:45] parse_names
[START] [2019-12-23 08:08:46] denormalize_canonical_names_to_nodes
[STOP] [2019-12-23 08:08:46] denormalize_canonical_names_to_nodes
[START] [2019-12-23 08:08:46] match_nodes
[START] [2019-12-23 08:08:46] map_all_nodes_to_pages
[STOP] [2019-12-23 08:13:14] map_all_nodes_to_pages
[INFO] [2019-12-23 08:13:14] 99 Unmatched nodes (of 1156)! That's too many to output. First 10: Cygnus bewickii (#61909698); Anas penelope (#61908628); Anas strepera (#61908652); Anas clypeata (#61908664); Anas querquedula (#61908704); Anas americana (#61909412); Anas cyanoptera (#61909513); Chen (#61908786); Chen caerulescens (#61908785); Larus melanocephalus (#61909123)
[START] [2019-12-23 08:13:14] update_nodes
[STOP] [2019-12-23 08:13:14] update_nodes
[STOP] [2019-12-23 08:13:14] match_nodes
[START] [2019-12-23 08:13:14] reindex_search
[STOP] [2019-12-23 08:13:17] reindex_search
[START] [2019-12-23 08:13:17] normalize_units
[STOP] [2019-12-23 08:13:17] normalize_units
[START] [2019-12-23 08:13:17] calculate_statistics
[STOP] [2019-12-23 08:13:17] calculate_statistics
[START] [2019-12-23 08:13:17] complete_harvest_instance
[START] [2019-12-23 08:13:17] overall_tsv_creation
[INFO] [2019-12-23 08:13:17] Processing group of 1156 in 1 batches of 10000
[INFO] [2019-12-23 08:14:07] 459 Traits (unfiltered)...
[INFO] [2019-12-23 08:14:20] 459 Traits (filtered)...
[INFO] [2019-12-23 08:14:20] 0 Associations (filtered)...
[INFO] [2019-12-23 08:14:58] 2293 metadata added.
[INFO] [2019-12-23 08:14:58] 0 metadata added.
[INFO] [2019-12-23 08:14:58] Average Time: 78.35
[INFO] [2019-12-23 08:14:58] Total Time: 1m42s
[STOP] [2019-12-23 08:14:58] overall_tsv_creation
[INFO] [2019-12-23 08:14:58] Done. Check your files:
[INFO] [2019-12-23 08:14:59] (1156 lines) /app/public/data/gulf_finland_sp_/publish_nodes.tsv
[INFO] [2019-12-23 08:15:00] (5163 lines) /app/public/data/gulf_finland_sp_/publish_node_ancestors.tsv
[INFO] [2019-12-23 08:15:00] (1156 lines) /app/public/data/gulf_finland_sp_/publish_scientific_names.tsv
[INFO] [2019-12-23 08:15:01] (460 lines) /app/public/data/gulf_finland_sp_/publish_traits.tsv
[INFO] [2019-12-23 08:15:02] (2294 lines) /app/public/data/gulf_finland_sp_/publish_metadata.tsv
[STOP] [2019-12-23 08:15:02] complete_harvest_instance
[START] [2019-12-23 08:15:02] completed
[STOP] [2019-12-23 08:15:02] completed
[STOP] [2019-12-23 08:15:02] logged process, took 407.64
Latest Process

23 Dec 08:08 EXITED currently processing 0