Harvest for Gulf of Bothnia Species List Created 23 Dec 03:54

Stage: completed
Fetched: 23 Dec 03:54
Validated: 23 Dec 03:54
Deltas Created 23 Dec 03:55
Units Normalized: 23 Dec 04:00
Ancestry Built: 23 Dec 03:55
Nodes Matched: 23 Dec 04:00
Names Parsed: 23 Dec 03:55
New Models Stored: 23 Dec 03:55
Indexed: 23 Dec 04:00
Completed: 23 Dec 04:01
Time to Harvest: less than a minute
Harvesting Log

(139 lines)
# Logfile created on 2019-12-23 03:54:56 -0500 by logger.rb/56815
[START] [2019-12-23 03:54:56] logged process
[START] [2019-12-23 03:54:56] create_harvest_instance
[STOP] [2019-12-23 03:54:57] create_harvest_instance
[START] [2019-12-23 03:54:57] fetch_files
[STOP] [2019-12-23 03:54:57] fetch_files
[START] [2019-12-23 03:54:57] validate_each_file
[STOP] [2019-12-23 03:54:57] validate_each_file
[START] [2019-12-23 03:54:57] convert_to_csv
[CMD] [2019-12-23 03:54:57] /usr/bin/sort /app/public/converted_csv/gulf_bothnia_sp__refs_19290.csv > /app/public/converted_csv/gulf_bothnia_sp__refs_19290.csv_sorted
[CMD] [2019-12-23 03:54:58] /usr/bin/sort /app/public/converted_csv/gulf_bothnia_sp__nodes_19291.csv > /app/public/converted_csv/gulf_bothnia_sp__nodes_19291.csv_sorted
[CMD] [2019-12-23 03:54:58] /usr/bin/sort /app/public/converted_csv/gulf_bothnia_sp__occurrences_19292.csv > /app/public/converted_csv/gulf_bothnia_sp__occurrences_19292.csv_sorted
[CMD] [2019-12-23 03:54:59] /usr/bin/sort /app/public/converted_csv/gulf_bothnia_sp__measurements_19293.csv > /app/public/converted_csv/gulf_bothnia_sp__measurements_19293.csv_sorted
[STOP] [2019-12-23 03:55:00] convert_to_csv
[START] [2019-12-23 03:55:00] calculate_delta
[CMD] [2019-12-23 03:55:00] echo "0a" > /app/public/diff/gulf_bothnia_sp__refs_19290.diff
[CMD] [2019-12-23 03:55:00] tail -n +1 /app/public/converted_csv/gulf_bothnia_sp__refs_19290.csv >> /app/public/diff/gulf_bothnia_sp__refs_19290.diff
[CMD] [2019-12-23 03:55:01] echo "." >> /app/public/diff/gulf_bothnia_sp__refs_19290.diff
[CMD] [2019-12-23 03:55:02] echo "0a" > /app/public/diff/gulf_bothnia_sp__nodes_19291.diff
[CMD] [2019-12-23 03:55:02] tail -n +1 /app/public/converted_csv/gulf_bothnia_sp__nodes_19291.csv >> /app/public/diff/gulf_bothnia_sp__nodes_19291.diff
[CMD] [2019-12-23 03:55:03] echo "." >> /app/public/diff/gulf_bothnia_sp__nodes_19291.diff
[CMD] [2019-12-23 03:55:04] echo "0a" > /app/public/diff/gulf_bothnia_sp__occurrences_19292.diff
[CMD] [2019-12-23 03:55:04] tail -n +1 /app/public/converted_csv/gulf_bothnia_sp__occurrences_19292.csv >> /app/public/diff/gulf_bothnia_sp__occurrences_19292.diff
[CMD] [2019-12-23 03:55:05] echo "." >> /app/public/diff/gulf_bothnia_sp__occurrences_19292.diff
[CMD] [2019-12-23 03:55:06] echo "0a" > /app/public/diff/gulf_bothnia_sp__measurements_19293.diff
[CMD] [2019-12-23 03:55:06] tail -n +1 /app/public/converted_csv/gulf_bothnia_sp__measurements_19293.csv >> /app/public/diff/gulf_bothnia_sp__measurements_19293.diff
[CMD] [2019-12-23 03:55:07] echo "." >> /app/public/diff/gulf_bothnia_sp__measurements_19293.diff
[STOP] [2019-12-23 03:55:08] calculate_delta
[START] [2019-12-23 03:55:08] parse_diff_and_store
[INFO] [2019-12-23 03:55:08] Loading refs diff file into memory (true lines)...
[INFO] [2019-12-23 03:55:09] Loading nodes diff file into memory (true lines)...
[INFO] [2019-12-23 03:55:10] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-12-23 03:55:11] Loading measurements diff file into memory (true lines)...
[INFO] [2019-12-23 03:55:15] Storing 2 References
[INFO] [2019-12-23 03:55:15] Processing group of 2 in 1 groups of 1000
[INFO] [2019-12-23 03:55:15] Average Time: 0.0
[INFO] [2019-12-23 03:55:15] Total Time: 1s
[INFO] [2019-12-23 03:55:15] Storing 1422 ScientificNames
[INFO] [2019-12-23 03:55:15] Processing group of 1422 in 2 groups of 1000
[INFO] [2019-12-23 03:55:15] Average Time: 0.295
[INFO] [2019-12-23 03:55:15] Total Time: 1s
[INFO] [2019-12-23 03:55:15] Storing 1422 Nodes
[INFO] [2019-12-23 03:55:15] Processing group of 1422 in 2 groups of 1000
[INFO] [2019-12-23 03:55:16] Average Time: 0.245
[INFO] [2019-12-23 03:55:16] Total Time: 1s
[INFO] [2019-12-23 03:55:16] Storing 575 Occurrences
[INFO] [2019-12-23 03:55:16] Processing group of 575 in 1 groups of 1000
[INFO] [2019-12-23 03:55:16] Average Time: 0.08
[INFO] [2019-12-23 03:55:16] Total Time: 1s
[INFO] [2019-12-23 03:55:16] Storing 1150 TraitsReferences
[INFO] [2019-12-23 03:55:16] Processing group of 1150 in 2 groups of 1000
[INFO] [2019-12-23 03:55:16] Average Time: 0.09
[INFO] [2019-12-23 03:55:16] Total Time: 1s
[INFO] [2019-12-23 03:55:16] Storing 1150 Traits
[INFO] [2019-12-23 03:55:16] Processing group of 1150 in 2 groups of 1000
[INFO] [2019-12-23 03:55:16] Average Time: 0.255
[INFO] [2019-12-23 03:55:16] Total Time: 1s
[INFO] [2019-12-23 03:55:16] Storing 1150 MetaTraits
[INFO] [2019-12-23 03:55:16] Processing group of 1150 in 2 groups of 1000
[INFO] [2019-12-23 03:55:17] Average Time: 0.11
[INFO] [2019-12-23 03:55:17] Total Time: 1s
[STOP] [2019-12-23 03:55:17] parse_diff_and_store
[START] [2019-12-23 03:55:17] resolve_keys
[INFO] [2019-12-23 03:55:25] Occurrences to nodes (through scientific_names)...
[INFO] [2019-12-23 03:55:25] traits to occurrences...
[INFO] [2019-12-23 03:55:26] traits to nodes (through occurrences)...
[INFO] [2019-12-23 03:55:26] Traits to sex term...
[INFO] [2019-12-23 03:55:26] Traits to lifestage term...
[INFO] [2019-12-23 03:55:26] MetaTraits to traits...
[INFO] [2019-12-23 03:55:26] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-12-23 03:55:26] Assocs to occurrences...
[INFO] [2019-12-23 03:55:26] Assocs to nodes...
[INFO] [2019-12-23 03:55:26] Assoc to sex term...
[INFO] [2019-12-23 03:55:26] Assoc to lifestage term...
[STOP] [2019-12-23 03:55:26] resolve_keys
[START] [2019-12-23 03:55:26] hold_for_later_1
[STOP] [2019-12-23 03:55:26] hold_for_later_1
[START] [2019-12-23 03:55:26] hold_for_later_2
[STOP] [2019-12-23 03:55:26] hold_for_later_2
[START] [2019-12-23 03:55:26] resolve_missing_parents
[STOP] [2019-12-23 03:55:28] resolve_missing_parents
[START] [2019-12-23 03:55:28] rebuild_nodes
[START] [2019-12-23 03:55:28] Flattener#flatten
[START] [2019-12-23 03:55:28] Flattener#study_resource
[START] [2019-12-23 03:55:28] Flattener#build_ancestry
[STOP] [2019-12-23 03:55:28] Flattener#build_ancestry
[INFO] [2019-12-23 03:55:28] 1422 ancestry keys
[START] [2019-12-23 03:55:28] build_node_ancestors
[INFO] [2019-12-23 03:55:28] old ancestors deleted.
[STOP] [2019-12-23 03:55:28] build_node_ancestors
[START] [2019-12-23 03:55:29] Flattener#propagate_ancestor_ids
[STOP] [2019-12-23 03:55:29] Flattener#propagate_ancestor_ids
[STOP] [2019-12-23 03:55:29] Flattener#flatten
[STOP] [2019-12-23 03:55:29] rebuild_nodes
[START] [2019-12-23 03:55:29] resolve_missing_media_owners
[STOP] [2019-12-23 03:55:29] resolve_missing_media_owners
[START] [2019-12-23 03:55:29] sanitize_media_verbatims
[STOP] [2019-12-23 03:55:29] sanitize_media_verbatims
[START] [2019-12-23 03:55:29] queue_downloads
[STOP] [2019-12-23 03:55:29] queue_downloads
[START] [2019-12-23 03:55:29] parse_names
[WARN] [2019-12-23 03:55:29] I see 1422 names which still need to be parsed.
[STOP] [2019-12-23 03:55:31] parse_names
[START] [2019-12-23 03:55:31] denormalize_canonical_names_to_nodes
[STOP] [2019-12-23 03:55:31] denormalize_canonical_names_to_nodes
[START] [2019-12-23 03:55:31] match_nodes
[START] [2019-12-23 03:55:31] map_all_nodes_to_pages
[STOP] [2019-12-23 04:00:00] map_all_nodes_to_pages
[INFO] [2019-12-23 04:00:00] 97 Unmatched nodes (of 1422)! That's too many to output. First 10: Anas penelope (#61907197); Anas clypeata (#61907246); Anas strepera (#61907274); Anas querquedula (#61907294); Anas americana (#61907586); Anas discors (#61907893); Anas sibilatrix (#61908230); Chen (#61907474); Chen caerulescens (#61907473); Chen rossii (#61907754)
[START] [2019-12-23 04:00:00] update_nodes
[STOP] [2019-12-23 04:00:01] update_nodes
[STOP] [2019-12-23 04:00:01] match_nodes
[START] [2019-12-23 04:00:01] reindex_search
[STOP] [2019-12-23 04:00:04] reindex_search
[START] [2019-12-23 04:00:04] normalize_units
[STOP] [2019-12-23 04:00:04] normalize_units
[START] [2019-12-23 04:00:04] calculate_statistics
[STOP] [2019-12-23 04:00:04] calculate_statistics
[START] [2019-12-23 04:00:04] complete_harvest_instance
[START] [2019-12-23 04:00:04] overall_tsv_creation
[INFO] [2019-12-23 04:00:04] Processing group of 1422 in 1 batches of 10000
[INFO] [2019-12-23 04:00:54] 575 Traits (unfiltered)...
[INFO] [2019-12-23 04:01:07] 575 Traits (filtered)...
[INFO] [2019-12-23 04:01:07] 0 Associations (filtered)...
[INFO] [2019-12-23 04:01:46] 2875 metadata added.
[INFO] [2019-12-23 04:01:46] 0 metadata added.
[INFO] [2019-12-23 04:01:46] Average Time: 79.52
[INFO] [2019-12-23 04:01:46] Total Time: 1m43s
[STOP] [2019-12-23 04:01:46] overall_tsv_creation
[INFO] [2019-12-23 04:01:46] Done. Check your files:
[INFO] [2019-12-23 04:01:47] (1422 lines) /app/public/data/gulf_bothnia_sp_/publish_nodes.tsv
[INFO] [2019-12-23 04:01:48] (6648 lines) /app/public/data/gulf_bothnia_sp_/publish_node_ancestors.tsv
[INFO] [2019-12-23 04:01:48] (1422 lines) /app/public/data/gulf_bothnia_sp_/publish_scientific_names.tsv
[INFO] [2019-12-23 04:01:49] (576 lines) /app/public/data/gulf_bothnia_sp_/publish_traits.tsv
[INFO] [2019-12-23 04:01:50] (2876 lines) /app/public/data/gulf_bothnia_sp_/publish_metadata.tsv
[STOP] [2019-12-23 04:01:50] complete_harvest_instance
[START] [2019-12-23 04:01:50] completed
[STOP] [2019-12-23 04:01:50] completed
[STOP] [2019-12-23 04:01:50] logged process, took 413.39
Latest Process

23 Dec 03:54 EXITED currently processing 0