Harvest for Faroe Islands Species List Created 12 Oct 20:44

Stage: completed
Fetched: 12 Oct 20:44
Validated: 12 Oct 20:44
Deltas Created 12 Oct 20:44
Units Normalized: 12 Oct 20:47
Ancestry Built: 12 Oct 20:44
Nodes Matched: 12 Oct 20:47
Names Parsed: 12 Oct 20:44
New Models Stored: 12 Oct 20:44
Indexed: 12 Oct 20:47
Completed: 12 Oct 20:49
Time to Harvest: less than a minute
Harvesting Log

(139 lines)
# Logfile created on 2019-10-12 20:44:05 -0400 by logger.rb/56815
[START] [2019-10-12 20:44:05] logged process
[START] [2019-10-12 20:44:05] create_harvest_instance
[STOP] [2019-10-12 20:44:05] create_harvest_instance
[START] [2019-10-12 20:44:05] fetch_files
[STOP] [2019-10-12 20:44:05] fetch_files
[START] [2019-10-12 20:44:05] validate_each_file
[STOP] [2019-10-12 20:44:06] validate_each_file
[START] [2019-10-12 20:44:06] convert_to_csv
[CMD] [2019-10-12 20:44:06] /usr/bin/sort /app/public/converted_csv/faroe_islands_sp_refs_15739.csv > /app/public/converted_csv/faroe_islands_sp_refs_15739.csv_sorted
[CMD] [2019-10-12 20:44:06] /usr/bin/sort /app/public/converted_csv/faroe_islands_sp_nodes_15740.csv > /app/public/converted_csv/faroe_islands_sp_nodes_15740.csv_sorted
[CMD] [2019-10-12 20:44:06] /usr/bin/sort /app/public/converted_csv/faroe_islands_sp_occurrences_15741.csv > /app/public/converted_csv/faroe_islands_sp_occurrences_15741.csv_sorted
[CMD] [2019-10-12 20:44:06] /usr/bin/sort /app/public/converted_csv/faroe_islands_sp_measurements_15742.csv > /app/public/converted_csv/faroe_islands_sp_measurements_15742.csv_sorted
[STOP] [2019-10-12 20:44:06] convert_to_csv
[START] [2019-10-12 20:44:06] calculate_delta
[CMD] [2019-10-12 20:44:06] echo "0a" > /app/public/diff/faroe_islands_sp_refs_15739.diff
[CMD] [2019-10-12 20:44:06] tail -n +1 /app/public/converted_csv/faroe_islands_sp_refs_15739.csv >> /app/public/diff/faroe_islands_sp_refs_15739.diff
[CMD] [2019-10-12 20:44:06] echo "." >> /app/public/diff/faroe_islands_sp_refs_15739.diff
[CMD] [2019-10-12 20:44:06] echo "0a" > /app/public/diff/faroe_islands_sp_nodes_15740.diff
[CMD] [2019-10-12 20:44:06] tail -n +1 /app/public/converted_csv/faroe_islands_sp_nodes_15740.csv >> /app/public/diff/faroe_islands_sp_nodes_15740.diff
[CMD] [2019-10-12 20:44:06] echo "." >> /app/public/diff/faroe_islands_sp_nodes_15740.diff
[CMD] [2019-10-12 20:44:06] echo "0a" > /app/public/diff/faroe_islands_sp_occurrences_15741.diff
[CMD] [2019-10-12 20:44:07] tail -n +1 /app/public/converted_csv/faroe_islands_sp_occurrences_15741.csv >> /app/public/diff/faroe_islands_sp_occurrences_15741.diff
[CMD] [2019-10-12 20:44:07] echo "." >> /app/public/diff/faroe_islands_sp_occurrences_15741.diff
[CMD] [2019-10-12 20:44:07] echo "0a" > /app/public/diff/faroe_islands_sp_measurements_15742.diff
[CMD] [2019-10-12 20:44:07] tail -n +1 /app/public/converted_csv/faroe_islands_sp_measurements_15742.csv >> /app/public/diff/faroe_islands_sp_measurements_15742.diff
[CMD] [2019-10-12 20:44:07] echo "." >> /app/public/diff/faroe_islands_sp_measurements_15742.diff
[STOP] [2019-10-12 20:44:07] calculate_delta
[START] [2019-10-12 20:44:07] parse_diff_and_store
[INFO] [2019-10-12 20:44:07] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-12 20:44:07] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-12 20:44:08] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-12 20:44:09] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-12 20:44:16] Storing 2 References
[INFO] [2019-10-12 20:44:16] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-12 20:44:16] Average Time: 0.0
[INFO] [2019-10-12 20:44:16] Total Time: 1s
[INFO] [2019-10-12 20:44:16] Storing 2637 ScientificNames
[INFO] [2019-10-12 20:44:16] Processing group of 2637 in 3 groups of 1000
[INFO] [2019-10-12 20:44:17] Average Time: 0.357
[INFO] [2019-10-12 20:44:17] Total Time: 2s
[INFO] [2019-10-12 20:44:17] Storing 2637 Nodes
[INFO] [2019-10-12 20:44:17] Processing group of 2637 in 3 groups of 1000
[INFO] [2019-10-12 20:44:18] Average Time: 0.277
[INFO] [2019-10-12 20:44:18] Total Time: 1s
[INFO] [2019-10-12 20:44:18] Storing 964 Occurrences
[INFO] [2019-10-12 20:44:18] Processing group of 964 in 1 groups of 1000
[INFO] [2019-10-12 20:44:18] Average Time: 0.11
[INFO] [2019-10-12 20:44:18] Total Time: 1s
[INFO] [2019-10-12 20:44:18] Storing 2490 TraitsReferences
[INFO] [2019-10-12 20:44:18] Processing group of 2490 in 3 groups of 1000
[INFO] [2019-10-12 20:44:18] Average Time: 0.083
[INFO] [2019-10-12 20:44:18] Total Time: 1s
[INFO] [2019-10-12 20:44:18] Storing 2489 Traits
[INFO] [2019-10-12 20:44:18] Processing group of 2489 in 3 groups of 1000
[INFO] [2019-10-12 20:44:19] Average Time: 0.3
[INFO] [2019-10-12 20:44:19] Total Time: 1s
[INFO] [2019-10-12 20:44:19] Storing 2485 MetaTraits
[INFO] [2019-10-12 20:44:19] Processing group of 2485 in 3 groups of 1000
[INFO] [2019-10-12 20:44:19] Average Time: 0.113
[INFO] [2019-10-12 20:44:19] Total Time: 1s
[STOP] [2019-10-12 20:44:19] parse_diff_and_store
[START] [2019-10-12 20:44:19] resolve_keys
[INFO] [2019-10-12 20:44:32] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-12 20:44:33] traits to occurrences...
[INFO] [2019-10-12 20:44:33] traits to nodes (through occurrences)...
[INFO] [2019-10-12 20:44:33] Traits to sex term...
[INFO] [2019-10-12 20:44:34] Traits to lifestage term...
[INFO] [2019-10-12 20:44:34] MetaTraits to traits...
[INFO] [2019-10-12 20:44:34] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-12 20:44:34] Assocs to occurrences...
[INFO] [2019-10-12 20:44:34] Assocs to nodes...
[INFO] [2019-10-12 20:44:34] Assoc to sex term...
[INFO] [2019-10-12 20:44:34] Assoc to lifestage term...
[STOP] [2019-10-12 20:44:34] resolve_keys
[START] [2019-10-12 20:44:35] hold_for_later_1
[STOP] [2019-10-12 20:44:35] hold_for_later_1
[START] [2019-10-12 20:44:35] hold_for_later_2
[STOP] [2019-10-12 20:44:35] hold_for_later_2
[START] [2019-10-12 20:44:35] resolve_missing_parents
[STOP] [2019-10-12 20:44:38] resolve_missing_parents
[START] [2019-10-12 20:44:39] rebuild_nodes
[START] [2019-10-12 20:44:39] Flattener#flatten
[START] [2019-10-12 20:44:39] Flattener#study_resource
[START] [2019-10-12 20:44:39] Flattener#build_ancestry
[STOP] [2019-10-12 20:44:39] Flattener#build_ancestry
[INFO] [2019-10-12 20:44:39] 2637 ancestry keys
[START] [2019-10-12 20:44:39] build_node_ancestors
[INFO] [2019-10-12 20:44:39] old ancestors deleted.
[STOP] [2019-10-12 20:44:39] build_node_ancestors
[START] [2019-10-12 20:44:39] Flattener#propagate_ancestor_ids
[STOP] [2019-10-12 20:44:40] Flattener#propagate_ancestor_ids
[STOP] [2019-10-12 20:44:40] Flattener#flatten
[STOP] [2019-10-12 20:44:40] rebuild_nodes
[START] [2019-10-12 20:44:40] resolve_missing_media_owners
[STOP] [2019-10-12 20:44:40] resolve_missing_media_owners
[START] [2019-10-12 20:44:40] sanitize_media_verbatims
[STOP] [2019-10-12 20:44:40] sanitize_media_verbatims
[START] [2019-10-12 20:44:40] queue_downloads
[STOP] [2019-10-12 20:44:40] queue_downloads
[START] [2019-10-12 20:44:40] parse_names
[WARN] [2019-10-12 20:44:40] I see 2637 names which still need to be parsed.
[STOP] [2019-10-12 20:44:43] parse_names
[START] [2019-10-12 20:44:43] denormalize_canonical_names_to_nodes
[STOP] [2019-10-12 20:44:43] denormalize_canonical_names_to_nodes
[START] [2019-10-12 20:44:43] match_nodes
[START] [2019-10-12 20:44:43] map_all_nodes_to_pages
[STOP] [2019-10-12 20:47:13] map_all_nodes_to_pages
[INFO] [2019-10-12 20:47:13] 193 Unmatched nodes (of 2637)! That's too many to output. First 10: Devaleraea ramentaceum (#49640721); Ceramium boergesenii (#49639386); Antithamnionella floccosum (#49640859); Scagelothamnion pusillum (#49639488); Callithamniaceae (#49638693); Aglaothamnion scopulorum (#49638898); Anas penelope (#49639585); Laminaria faeroensis (#49638878); Laminaria longicruris (#49640818); Saccharina saccharina (#49638826)
[START] [2019-10-12 20:47:13] update_nodes
[STOP] [2019-10-12 20:47:14] update_nodes
[STOP] [2019-10-12 20:47:14] match_nodes
[START] [2019-10-12 20:47:14] reindex_search
[STOP] [2019-10-12 20:47:20] reindex_search
[START] [2019-10-12 20:47:20] normalize_units
[STOP] [2019-10-12 20:47:20] normalize_units
[START] [2019-10-12 20:47:20] calculate_statistics
[STOP] [2019-10-12 20:47:20] calculate_statistics
[START] [2019-10-12 20:47:20] complete_harvest_instance
[START] [2019-10-12 20:47:20] overall_tsv_creation
[INFO] [2019-10-12 20:47:20] Processing group of 2637 in 1 batches of 10000
[INFO] [2019-10-12 20:48:16] 964 Traits (unfiltered)...
[INFO] [2019-10-12 20:48:30] 964 Traits (filtered)...
[INFO] [2019-10-12 20:48:30] 0 Associations (filtered)...
[INFO] [2019-10-12 20:49:11] 4817 metadata added.
[INFO] [2019-10-12 20:49:11] 0 metadata added.
[INFO] [2019-10-12 20:49:11] Average Time: 88.69
[INFO] [2019-10-12 20:49:11] Total Time: 1m52s
[STOP] [2019-10-12 20:49:11] overall_tsv_creation
[INFO] [2019-10-12 20:49:11] Done. Check your files:
[INFO] [2019-10-12 20:49:11] (2637 lines) /app/public/data/faroe_islands_sp/publish_nodes.tsv
[INFO] [2019-10-12 20:49:11] (5531 lines) /app/public/data/faroe_islands_sp/publish_node_ancestors.tsv
[INFO] [2019-10-12 20:49:11] (2637 lines) /app/public/data/faroe_islands_sp/publish_scientific_names.tsv
[INFO] [2019-10-12 20:49:11] (965 lines) /app/public/data/faroe_islands_sp/publish_traits.tsv
[INFO] [2019-10-12 20:49:11] (4818 lines) /app/public/data/faroe_islands_sp/publish_metadata.tsv
[STOP] [2019-10-12 20:49:12] complete_harvest_instance
[START] [2019-10-12 20:49:12] completed
[STOP] [2019-10-12 20:49:12] completed
[STOP] [2019-10-12 20:49:12] logged process, took 306.92
Latest Process

12 Oct 20:44 EXITED currently processing 0