Harvest for Fiji Species List Created 12 Oct 20:59

Stage: completed
Fetched: 12 Oct 20:59
Validated: 12 Oct 20:59
Deltas Created 12 Oct 20:59
Units Normalized: 12 Oct 21:12
Ancestry Built: 12 Oct 21:01
Nodes Matched: 12 Oct 21:12
Names Parsed: 12 Oct 21:01
New Models Stored: 12 Oct 21:00
Indexed: 12 Oct 21:12
Completed: 12 Oct 21:14
Time to Harvest: less than a minute
Harvesting Log

(149 lines)
# Logfile created on 2019-10-12 20:59:55 -0400 by logger.rb/56815
[START] [2019-10-12 20:59:55] logged process
[START] [2019-10-12 20:59:55] create_harvest_instance
[STOP] [2019-10-12 20:59:56] create_harvest_instance
[START] [2019-10-12 20:59:56] fetch_files
[STOP] [2019-10-12 20:59:56] fetch_files
[START] [2019-10-12 20:59:56] validate_each_file
[STOP] [2019-10-12 20:59:56] validate_each_file
[START] [2019-10-12 20:59:56] convert_to_csv
[CMD] [2019-10-12 20:59:56] /usr/bin/sort /app/public/converted_csv/fiji_sp_list_refs_15755.csv > /app/public/converted_csv/fiji_sp_list_refs_15755.csv_sorted
[CMD] [2019-10-12 20:59:56] /usr/bin/sort /app/public/converted_csv/fiji_sp_list_nodes_15756.csv > /app/public/converted_csv/fiji_sp_list_nodes_15756.csv_sorted
[CMD] [2019-10-12 20:59:57] /usr/bin/sort /app/public/converted_csv/fiji_sp_list_occurrences_15757.csv > /app/public/converted_csv/fiji_sp_list_occurrences_15757.csv_sorted
[CMD] [2019-10-12 20:59:57] /usr/bin/sort /app/public/converted_csv/fiji_sp_list_measurements_15758.csv > /app/public/converted_csv/fiji_sp_list_measurements_15758.csv_sorted
[STOP] [2019-10-12 20:59:57] convert_to_csv
[START] [2019-10-12 20:59:57] calculate_delta
[CMD] [2019-10-12 20:59:57] echo "0a" > /app/public/diff/fiji_sp_list_refs_15755.diff
[CMD] [2019-10-12 20:59:57] tail -n +1 /app/public/converted_csv/fiji_sp_list_refs_15755.csv >> /app/public/diff/fiji_sp_list_refs_15755.diff
[CMD] [2019-10-12 20:59:57] echo "." >> /app/public/diff/fiji_sp_list_refs_15755.diff
[CMD] [2019-10-12 20:59:57] echo "0a" > /app/public/diff/fiji_sp_list_nodes_15756.diff
[CMD] [2019-10-12 20:59:57] tail -n +1 /app/public/converted_csv/fiji_sp_list_nodes_15756.csv >> /app/public/diff/fiji_sp_list_nodes_15756.diff
[CMD] [2019-10-12 20:59:57] echo "." >> /app/public/diff/fiji_sp_list_nodes_15756.diff
[CMD] [2019-10-12 20:59:57] echo "0a" > /app/public/diff/fiji_sp_list_occurrences_15757.diff
[CMD] [2019-10-12 20:59:57] tail -n +1 /app/public/converted_csv/fiji_sp_list_occurrences_15757.csv >> /app/public/diff/fiji_sp_list_occurrences_15757.diff
[CMD] [2019-10-12 20:59:57] echo "." >> /app/public/diff/fiji_sp_list_occurrences_15757.diff
[CMD] [2019-10-12 20:59:58] echo "0a" > /app/public/diff/fiji_sp_list_measurements_15758.diff
[CMD] [2019-10-12 20:59:58] tail -n +1 /app/public/converted_csv/fiji_sp_list_measurements_15758.csv >> /app/public/diff/fiji_sp_list_measurements_15758.diff
[CMD] [2019-10-12 20:59:58] echo "." >> /app/public/diff/fiji_sp_list_measurements_15758.diff
[STOP] [2019-10-12 20:59:58] calculate_delta
[START] [2019-10-12 20:59:58] parse_diff_and_store
[INFO] [2019-10-12 20:59:58] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-12 20:59:58] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-12 21:00:00] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-12 21:00:01] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-12 21:00:22] Storing 2 References
[INFO] [2019-10-12 21:00:22] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-12 21:00:22] Average Time: 0.0
[INFO] [2019-10-12 21:00:22] Total Time: 1s
[INFO] [2019-10-12 21:00:22] Storing 6386 ScientificNames
[INFO] [2019-10-12 21:00:22] Processing group of 6386 in 7 groups of 1000
[INFO] [2019-10-12 21:00:25] Average Time: 0.351
[INFO] [2019-10-12 21:00:25] Total Time: 3s
[INFO] [2019-10-12 21:00:25] last 3 / first 3: 0.71
[INFO] [2019-10-12 21:00:25] Std.Dev: 0.10488088481701516; Max: 0.45
[INFO] [2019-10-12 21:00:25] Storing 6386 Nodes
[INFO] [2019-10-12 21:00:25] Processing group of 6386 in 7 groups of 1000
[INFO] [2019-10-12 21:00:27] Average Time: 0.32
[INFO] [2019-10-12 21:00:27] Total Time: 3s
[INFO] [2019-10-12 21:00:27] last 3 / first 3: 0.61
[INFO] [2019-10-12 21:00:27] Std.Dev: 0.11401754250991379; Max: 0.5
[INFO] [2019-10-12 21:00:27] Storing 3705 Occurrences
[INFO] [2019-10-12 21:00:27] Processing group of 3705 in 4 groups of 1000
[INFO] [2019-10-12 21:00:27] Average Time: 0.1
[INFO] [2019-10-12 21:00:27] Total Time: 1s
[INFO] [2019-10-12 21:00:27] Storing 7410 TraitsReferences
[INFO] [2019-10-12 21:00:27] Processing group of 7410 in 8 groups of 1000
[INFO] [2019-10-12 21:00:28] Average Time: 0.075
[INFO] [2019-10-12 21:00:28] Total Time: 1s
[INFO] [2019-10-12 21:00:28] last 3 / first 3: 0.59
[INFO] [2019-10-12 21:00:28] Std.Dev: 0.03162277660168379; Max: 0.15
[INFO] [2019-10-12 21:00:28] Storing 7410 Traits
[INFO] [2019-10-12 21:00:28] Processing group of 7410 in 8 groups of 1000
[INFO] [2019-10-12 21:00:31] Average Time: 0.321
[INFO] [2019-10-12 21:00:31] Total Time: 3s
[INFO] [2019-10-12 21:00:31] last 3 / first 3: 0.73
[INFO] [2019-10-12 21:00:31] Std.Dev: 0.08366600265340755; Max: 0.44
[INFO] [2019-10-12 21:00:31] Storing 7396 MetaTraits
[INFO] [2019-10-12 21:00:31] Processing group of 7396 in 8 groups of 1000
[INFO] [2019-10-12 21:00:32] Average Time: 0.14
[INFO] [2019-10-12 21:00:32] Total Time: 2s
[INFO] [2019-10-12 21:00:32] last 3 / first 3: 0.91
[INFO] [2019-10-12 21:00:32] Std.Dev: 0.0; Max: 0.19
[STOP] [2019-10-12 21:00:32] parse_diff_and_store
[START] [2019-10-12 21:00:32] resolve_keys
[INFO] [2019-10-12 21:01:01] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-12 21:01:03] traits to occurrences...
[INFO] [2019-10-12 21:01:05] traits to nodes (through occurrences)...
[INFO] [2019-10-12 21:01:05] Traits to sex term...
[INFO] [2019-10-12 21:01:06] Traits to lifestage term...
[INFO] [2019-10-12 21:01:08] MetaTraits to traits...
[INFO] [2019-10-12 21:01:08] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-12 21:01:09] Assocs to occurrences...
[INFO] [2019-10-12 21:01:09] Assocs to nodes...
[INFO] [2019-10-12 21:01:09] Assoc to sex term...
[INFO] [2019-10-12 21:01:09] Assoc to lifestage term...
[STOP] [2019-10-12 21:01:09] resolve_keys
[START] [2019-10-12 21:01:09] hold_for_later_1
[STOP] [2019-10-12 21:01:09] hold_for_later_1
[START] [2019-10-12 21:01:09] hold_for_later_2
[STOP] [2019-10-12 21:01:09] hold_for_later_2
[START] [2019-10-12 21:01:09] resolve_missing_parents
[STOP] [2019-10-12 21:01:25] resolve_missing_parents
[START] [2019-10-12 21:01:25] rebuild_nodes
[START] [2019-10-12 21:01:25] Flattener#flatten
[START] [2019-10-12 21:01:25] Flattener#study_resource
[START] [2019-10-12 21:01:25] Flattener#build_ancestry
[STOP] [2019-10-12 21:01:25] Flattener#build_ancestry
[INFO] [2019-10-12 21:01:25] 6386 ancestry keys
[START] [2019-10-12 21:01:25] build_node_ancestors
[INFO] [2019-10-12 21:01:25] old ancestors deleted.
[STOP] [2019-10-12 21:01:28] build_node_ancestors
[START] [2019-10-12 21:01:30] Flattener#propagate_ancestor_ids
[STOP] [2019-10-12 21:01:31] Flattener#propagate_ancestor_ids
[STOP] [2019-10-12 21:01:31] Flattener#flatten
[STOP] [2019-10-12 21:01:31] rebuild_nodes
[START] [2019-10-12 21:01:31] resolve_missing_media_owners
[STOP] [2019-10-12 21:01:31] resolve_missing_media_owners
[START] [2019-10-12 21:01:31] sanitize_media_verbatims
[STOP] [2019-10-12 21:01:31] sanitize_media_verbatims
[START] [2019-10-12 21:01:31] queue_downloads
[STOP] [2019-10-12 21:01:31] queue_downloads
[START] [2019-10-12 21:01:31] parse_names
[WARN] [2019-10-12 21:01:31] I see 6386 names which still need to be parsed.
[STOP] [2019-10-12 21:01:37] parse_names
[START] [2019-10-12 21:01:37] denormalize_canonical_names_to_nodes
[STOP] [2019-10-12 21:01:37] denormalize_canonical_names_to_nodes
[START] [2019-10-12 21:01:37] match_nodes
[START] [2019-10-12 21:01:37] map_all_nodes_to_pages
[STOP] [2019-10-12 21:12:18] map_all_nodes_to_pages
[INFO] [2019-10-12 21:12:18] 394 Unmatched nodes (of 6386)! That's too many to output. First 10: Pheidole bula (#49647413); Pheidole simplispinosa (#49647425); Pheidole pegasus (#49647469); Pheidole uncagena (#49647505); Strumigenys trada (#49648925); Lordomyrma sukuna (#49647533); Lordomyrma polita (#49647610); Lordomyrma stoneri (#49647706); Lordomyrma vuda (#49647738); Lordomyrma vanua (#49648671)
[START] [2019-10-12 21:12:18] update_nodes
[STOP] [2019-10-12 21:12:20] update_nodes
[STOP] [2019-10-12 21:12:20] match_nodes
[START] [2019-10-12 21:12:20] reindex_search
[STOP] [2019-10-12 21:12:36] reindex_search
[START] [2019-10-12 21:12:36] normalize_units
[STOP] [2019-10-12 21:12:36] normalize_units
[START] [2019-10-12 21:12:36] calculate_statistics
[STOP] [2019-10-12 21:12:36] calculate_statistics
[START] [2019-10-12 21:12:36] complete_harvest_instance
[START] [2019-10-12 21:12:36] overall_tsv_creation
[INFO] [2019-10-12 21:12:36] Processing group of 6386 in 1 batches of 10000
[INFO] [2019-10-12 21:13:50] 3705 Traits (unfiltered)...
[INFO] [2019-10-12 21:14:04] 3705 Traits (filtered)...
[INFO] [2019-10-12 21:14:04] 0 Associations (filtered)...
[INFO] [2019-10-12 21:14:49] 18511 metadata added.
[INFO] [2019-10-12 21:14:49] 0 metadata added.
[INFO] [2019-10-12 21:14:49] Average Time: 107.51
[INFO] [2019-10-12 21:14:49] Total Time: 2m14s
[STOP] [2019-10-12 21:14:49] overall_tsv_creation
[INFO] [2019-10-12 21:14:49] Done. Check your files:
[INFO] [2019-10-12 21:14:49] (6386 lines) /app/public/data/fiji_sp_list/publish_nodes.tsv
[INFO] [2019-10-12 21:14:49] (33504 lines) /app/public/data/fiji_sp_list/publish_node_ancestors.tsv
[INFO] [2019-10-12 21:14:49] (6386 lines) /app/public/data/fiji_sp_list/publish_scientific_names.tsv
[INFO] [2019-10-12 21:14:50] (3706 lines) /app/public/data/fiji_sp_list/publish_traits.tsv
[INFO] [2019-10-12 21:14:50] (18512 lines) /app/public/data/fiji_sp_list/publish_metadata.tsv
[STOP] [2019-10-12 21:14:50] complete_harvest_instance
[START] [2019-10-12 21:14:50] completed
[STOP] [2019-10-12 21:14:50] completed
[STOP] [2019-10-12 21:14:50] logged process, took 894.86
Latest Process

12 Oct 20:59 EXITED currently processing 0