Harvest for Denmark Species List Created 12 Oct 13:34

Stage: completed
Fetched: 12 Oct 13:34
Validated: 12 Oct 13:34
Deltas Created 12 Oct 13:34
Units Normalized: 12 Oct 14:14
Ancestry Built: 12 Oct 13:41
Nodes Matched: 12 Oct 14:13
Names Parsed: 12 Oct 13:41
New Models Stored: 12 Oct 13:37
Indexed: 12 Oct 14:14
Completed: 12 Oct 14:23
Time to Harvest: 1 minute
Harvesting Log

(161 lines)
# Logfile created on 2019-10-12 13:34:45 -0400 by logger.rb/56815
[START] [2019-10-12 13:34:45] logged process
[START] [2019-10-12 13:34:45] create_harvest_instance
[STOP] [2019-10-12 13:34:46] create_harvest_instance
[START] [2019-10-12 13:34:46] fetch_files
[STOP] [2019-10-12 13:34:46] fetch_files
[START] [2019-10-12 13:34:46] validate_each_file
[STOP] [2019-10-12 13:34:52] validate_each_file
[START] [2019-10-12 13:34:52] convert_to_csv
[CMD] [2019-10-12 13:34:52] /usr/bin/sort /app/public/converted_csv/denmark_sp_list_refs_15587.csv > /app/public/converted_csv/denmark_sp_list_refs_15587.csv_sorted
[CMD] [2019-10-12 13:34:52] /usr/bin/sort /app/public/converted_csv/denmark_sp_list_nodes_15588.csv > /app/public/converted_csv/denmark_sp_list_nodes_15588.csv_sorted
[CMD] [2019-10-12 13:34:52] /usr/bin/sort /app/public/converted_csv/denmark_sp_list_occurrences_15589.csv > /app/public/converted_csv/denmark_sp_list_occurrences_15589.csv_sorted
[CMD] [2019-10-12 13:34:52] /usr/bin/sort /app/public/converted_csv/denmark_sp_list_measurements_15590.csv > /app/public/converted_csv/denmark_sp_list_measurements_15590.csv_sorted
[STOP] [2019-10-12 13:34:52] convert_to_csv
[START] [2019-10-12 13:34:52] calculate_delta
[CMD] [2019-10-12 13:34:52] echo "0a" > /app/public/diff/denmark_sp_list_refs_15587.diff
[CMD] [2019-10-12 13:34:52] tail -n +1 /app/public/converted_csv/denmark_sp_list_refs_15587.csv >> /app/public/diff/denmark_sp_list_refs_15587.diff
[CMD] [2019-10-12 13:34:53] echo "." >> /app/public/diff/denmark_sp_list_refs_15587.diff
[CMD] [2019-10-12 13:34:53] echo "0a" > /app/public/diff/denmark_sp_list_nodes_15588.diff
[CMD] [2019-10-12 13:34:53] tail -n +1 /app/public/converted_csv/denmark_sp_list_nodes_15588.csv >> /app/public/diff/denmark_sp_list_nodes_15588.diff
[CMD] [2019-10-12 13:34:53] echo "." >> /app/public/diff/denmark_sp_list_nodes_15588.diff
[CMD] [2019-10-12 13:34:53] echo "0a" > /app/public/diff/denmark_sp_list_occurrences_15589.diff
[CMD] [2019-10-12 13:34:53] tail -n +1 /app/public/converted_csv/denmark_sp_list_occurrences_15589.csv >> /app/public/diff/denmark_sp_list_occurrences_15589.diff
[CMD] [2019-10-12 13:34:53] echo "." >> /app/public/diff/denmark_sp_list_occurrences_15589.diff
[CMD] [2019-10-12 13:34:53] echo "0a" > /app/public/diff/denmark_sp_list_measurements_15590.diff
[CMD] [2019-10-12 13:34:53] tail -n +1 /app/public/converted_csv/denmark_sp_list_measurements_15590.csv >> /app/public/diff/denmark_sp_list_measurements_15590.diff
[CMD] [2019-10-12 13:34:53] echo "." >> /app/public/diff/denmark_sp_list_measurements_15590.diff
[STOP] [2019-10-12 13:34:53] calculate_delta
[START] [2019-10-12 13:34:53] parse_diff_and_store
[INFO] [2019-10-12 13:34:54] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-12 13:34:54] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-12 13:35:05] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-12 13:35:08] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-12 13:37:02] Storing 2 References
[INFO] [2019-10-12 13:37:02] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-12 13:37:02] Average Time: 0.0
[INFO] [2019-10-12 13:37:02] Total Time: 1s
[INFO] [2019-10-12 13:37:02] Storing 30066 ScientificNames
[INFO] [2019-10-12 13:37:02] Processing group of 30066 in 31 groups of 1000
[INFO] [2019-10-12 13:37:14] Average Time: 0.373
[INFO] [2019-10-12 13:37:14] Total Time: 12s
[INFO] [2019-10-12 13:37:14] last 3 / first 3: 0.57
[INFO] [2019-10-12 13:37:14] Std.Dev: 0.11401754250991379; Max: 0.77
[INFO] [2019-10-12 13:37:14] Storing 30066 Nodes
[INFO] [2019-10-12 13:37:14] Processing group of 30066 in 31 groups of 1000
[INFO] [2019-10-12 13:37:24] Average Time: 0.327
[INFO] [2019-10-12 13:37:24] Total Time: 11s
[INFO] [2019-10-12 13:37:24] last 3 / first 3: 0.67
[INFO] [2019-10-12 13:37:24] Std.Dev: 0.11832159566199232; Max: 0.73
[INFO] [2019-10-12 13:37:24] Storing 19762 Occurrences
[INFO] [2019-10-12 13:37:24] Processing group of 19762 in 20 groups of 1000
[INFO] [2019-10-12 13:37:27] Average Time: 0.107
[INFO] [2019-10-12 13:37:27] Total Time: 3s
[INFO] [2019-10-12 13:37:27] last 3 / first 3: 0.68
[INFO] [2019-10-12 13:37:27] Std.Dev: 0.03162277660168379; Max: 0.2
[INFO] [2019-10-12 13:37:27] Storing 40050 TraitsReferences
[INFO] [2019-10-12 13:37:27] Processing group of 40050 in 41 groups of 1000
[INFO] [2019-10-12 13:37:30] Average Time: 0.082
[INFO] [2019-10-12 13:37:30] Total Time: 4s
[INFO] [2019-10-12 13:37:30] last 3 / first 3: 1.16
[INFO] [2019-10-12 13:37:30] Std.Dev: 0.03162277660168379; Max: 0.18
[INFO] [2019-10-12 13:37:30] Storing 40049 Traits
[INFO] [2019-10-12 13:37:30] Processing group of 40049 in 41 groups of 1000
[INFO] [2019-10-12 13:37:43] Average Time: 0.316
[INFO] [2019-10-12 13:37:43] Total Time: 14s
[INFO] [2019-10-12 13:37:43] last 3 / first 3: 0.35
[INFO] [2019-10-12 13:37:43] Std.Dev: 0.10954451150103323; Max: 0.81
[INFO] [2019-10-12 13:37:43] Storing 40019 MetaTraits
[INFO] [2019-10-12 13:37:43] Processing group of 40019 in 41 groups of 1000
[INFO] [2019-10-12 13:37:48] Average Time: 0.119
[INFO] [2019-10-12 13:37:48] Total Time: 6s
[INFO] [2019-10-12 13:37:48] last 3 / first 3: 0.81
[INFO] [2019-10-12 13:37:48] Std.Dev: 0.03162277660168379; Max: 0.21
[STOP] [2019-10-12 13:37:48] parse_diff_and_store
[START] [2019-10-12 13:37:48] resolve_keys
[INFO] [2019-10-12 13:39:19] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-12 13:39:26] traits to occurrences...
[INFO] [2019-10-12 13:39:31] traits to nodes (through occurrences)...
[INFO] [2019-10-12 13:39:32] Traits to sex term...
[INFO] [2019-10-12 13:39:37] Traits to lifestage term...
[INFO] [2019-10-12 13:39:42] MetaTraits to traits...
[INFO] [2019-10-12 13:39:45] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-12 13:39:50] Assocs to occurrences...
[INFO] [2019-10-12 13:39:50] Assocs to nodes...
[INFO] [2019-10-12 13:39:50] Assoc to sex term...
[INFO] [2019-10-12 13:39:50] Assoc to lifestage term...
[STOP] [2019-10-12 13:39:50] resolve_keys
[START] [2019-10-12 13:39:50] hold_for_later_1
[STOP] [2019-10-12 13:39:50] hold_for_later_1
[START] [2019-10-12 13:39:50] hold_for_later_2
[STOP] [2019-10-12 13:39:50] hold_for_later_2
[START] [2019-10-12 13:39:50] resolve_missing_parents
[STOP] [2019-10-12 13:40:36] resolve_missing_parents
[START] [2019-10-12 13:40:36] rebuild_nodes
[START] [2019-10-12 13:40:36] Flattener#flatten
[START] [2019-10-12 13:40:36] Flattener#study_resource
[START] [2019-10-12 13:40:37] Flattener#build_ancestry
[STOP] [2019-10-12 13:40:41] Flattener#build_ancestry
[INFO] [2019-10-12 13:40:41] 30066 ancestry keys
[START] [2019-10-12 13:40:41] build_node_ancestors
[INFO] [2019-10-12 13:40:41] old ancestors deleted.
[STOP] [2019-10-12 13:40:57] build_node_ancestors
[START] [2019-10-12 13:40:58] Flattener#propagate_ancestor_ids
[STOP] [2019-10-12 13:41:01] Flattener#propagate_ancestor_ids
[STOP] [2019-10-12 13:41:01] Flattener#flatten
[STOP] [2019-10-12 13:41:01] rebuild_nodes
[START] [2019-10-12 13:41:01] resolve_missing_media_owners
[STOP] [2019-10-12 13:41:01] resolve_missing_media_owners
[START] [2019-10-12 13:41:01] sanitize_media_verbatims
[STOP] [2019-10-12 13:41:01] sanitize_media_verbatims
[START] [2019-10-12 13:41:01] queue_downloads
[STOP] [2019-10-12 13:41:01] queue_downloads
[START] [2019-10-12 13:41:01] parse_names
[WARN] [2019-10-12 13:41:01] I see 30066 names which still need to be parsed.
[STOP] [2019-10-12 13:41:25] parse_names
[START] [2019-10-12 13:41:25] denormalize_canonical_names_to_nodes
[STOP] [2019-10-12 13:41:25] denormalize_canonical_names_to_nodes
[START] [2019-10-12 13:41:25] match_nodes
[START] [2019-10-12 13:41:25] map_all_nodes_to_pages
[STOP] [2019-10-12 14:13:26] map_all_nodes_to_pages
[INFO] [2019-10-12 14:13:26] 3009 Unmatched nodes (of 30066)! That's too many to output. First 10: Larus melanocephalus (#49422783); Larus ichthyaetus (#49437709); Thalaseus (#49421343); Thalaseus sandvicensis (#49421342); Thalaseus elegans (#49449858); Chlidonias nigra (#49444118); Onychoprion fuscata (#49447601); Philomachus (#49421185); Philomachus pugnax (#49421184); Limicola (#49422970)
[START] [2019-10-12 14:13:26] update_nodes
[STOP] [2019-10-12 14:13:37] update_nodes
[STOP] [2019-10-12 14:13:37] match_nodes
[START] [2019-10-12 14:13:37] reindex_search
[STOP] [2019-10-12 14:14:42] reindex_search
[START] [2019-10-12 14:14:42] normalize_units
[STOP] [2019-10-12 14:14:42] normalize_units
[START] [2019-10-12 14:14:42] calculate_statistics
[STOP] [2019-10-12 14:14:42] calculate_statistics
[START] [2019-10-12 14:14:42] complete_harvest_instance
[START] [2019-10-12 14:14:42] overall_tsv_creation
[INFO] [2019-10-12 14:14:42] Processing group of 30066 in 4 batches of 10000
[INFO] [2019-10-12 14:16:11] 6172 Traits (unfiltered)...
[INFO] [2019-10-12 14:16:25] 6172 Traits (filtered)...
[INFO] [2019-10-12 14:16:25] 0 Associations (filtered)...
[INFO] [2019-10-12 14:17:16] 30857 metadata added.
[INFO] [2019-10-12 14:17:16] 0 metadata added.
[INFO] [2019-10-12 14:18:49] 6960 Traits (unfiltered)...
[INFO] [2019-10-12 14:19:03] 6960 Traits (filtered)...
[INFO] [2019-10-12 14:19:03] 0 Associations (filtered)...
[INFO] [2019-10-12 14:19:54] 34793 metadata added.
[INFO] [2019-10-12 14:19:54] 0 metadata added.
[INFO] [2019-10-12 14:21:28] 6630 Traits (unfiltered)...
[INFO] [2019-10-12 14:21:42] 6630 Traits (filtered)...
[INFO] [2019-10-12 14:21:42] 0 Associations (filtered)...
[INFO] [2019-10-12 14:22:36] 33129 metadata added.
[INFO] [2019-10-12 14:22:36] 0 metadata added.
[INFO] [2019-10-12 14:23:08] Average Time: 101.555
[INFO] [2019-10-12 14:23:08] Total Time: 8m27s
[STOP] [2019-10-12 14:23:08] overall_tsv_creation
[INFO] [2019-10-12 14:23:08] Done. Check your files:
[INFO] [2019-10-12 14:23:08] (30066 lines) /app/public/data/denmark_sp_list/publish_nodes.tsv
[INFO] [2019-10-12 14:23:09] (112654 lines) /app/public/data/denmark_sp_list/publish_node_ancestors.tsv
[INFO] [2019-10-12 14:23:09] (30066 lines) /app/public/data/denmark_sp_list/publish_scientific_names.tsv
[INFO] [2019-10-12 14:23:09] (19763 lines) /app/public/data/denmark_sp_list/publish_traits.tsv
[INFO] [2019-10-12 14:23:09] (98780 lines) /app/public/data/denmark_sp_list/publish_metadata.tsv
[STOP] [2019-10-12 14:23:09] complete_harvest_instance
[START] [2019-10-12 14:23:09] completed
[STOP] [2019-10-12 14:23:09] completed
[STOP] [2019-10-12 14:23:09] logged process, took 2903.92
Latest Process

12 Oct 13:34 EXITED currently processing 0