Harvest for Adriatic Sea Species List Created 01 Oct 15:32

Stage: completed
Fetched: 01 Oct 15:32
Validated: 01 Oct 15:32
Deltas Created 01 Oct 15:33
Units Normalized: 01 Oct 15:39
Ancestry Built: 01 Oct 15:34
Nodes Matched: 01 Oct 15:39
Names Parsed: 01 Oct 15:34
New Models Stored: 01 Oct 15:33
Indexed: 01 Oct 15:39
Completed: 01 Oct 15:41
Time to Harvest: less than a minute

Expected File Format Definitions

Harvesting Log (most recent first)

# Logfile created on 2019-10-01 15:32:49 -0400 by logger.rb/56815
[START] [2019-10-01 15:32:49] logged process
[START] [2019-10-01 15:32:49] create_harvest_instance
[STOP] [2019-10-01 15:32:49] create_harvest_instance
[START] [2019-10-01 15:32:49] fetch_files
[STOP] [2019-10-01 15:32:49] fetch_files
[START] [2019-10-01 15:32:49] validate_each_file
[STOP] [2019-10-01 15:32:50] validate_each_file
[START] [2019-10-01 15:32:50] convert_to_csv
[CMD] [2019-10-01 15:32:50] /usr/bin/sort /app/public/converted_csv/adriatic_sea_sp2_refs_14718.csv > /app/public/converted_csv/adriatic_sea_sp2_refs_14718.csv_sorted
[CMD] [2019-10-01 15:32:51] /usr/bin/sort /app/public/converted_csv/adriatic_sea_sp2_nodes_14719.csv > /app/public/converted_csv/adriatic_sea_sp2_nodes_14719.csv_sorted
[CMD] [2019-10-01 15:32:53] /usr/bin/sort /app/public/converted_csv/adriatic_sea_sp2_occurrences_14720.csv > /app/public/converted_csv/adriatic_sea_sp2_occurrences_14720.csv_sorted
[CMD] [2019-10-01 15:32:54] /usr/bin/sort /app/public/converted_csv/adriatic_sea_sp2_measurements_14721.csv > /app/public/converted_csv/adriatic_sea_sp2_measurements_14721.csv_sorted
[STOP] [2019-10-01 15:32:56] convert_to_csv
[START] [2019-10-01 15:32:56] calculate_delta
[CMD] [2019-10-01 15:32:56] echo "0a" > /app/public/diff/adriatic_sea_sp2_refs_14718.diff
[CMD] [2019-10-01 15:32:57] tail -n +1 /app/public/converted_csv/adriatic_sea_sp2_refs_14718.csv >> /app/public/diff/adriatic_sea_sp2_refs_14718.diff
[CMD] [2019-10-01 15:32:59] echo "." >> /app/public/diff/adriatic_sea_sp2_refs_14718.diff
[CMD] [2019-10-01 15:33:00] echo "0a" > /app/public/diff/adriatic_sea_sp2_nodes_14719.diff
[CMD] [2019-10-01 15:33:02] tail -n +1 /app/public/converted_csv/adriatic_sea_sp2_nodes_14719.csv >> /app/public/diff/adriatic_sea_sp2_nodes_14719.diff
[CMD] [2019-10-01 15:33:03] echo "." >> /app/public/diff/adriatic_sea_sp2_nodes_14719.diff
[CMD] [2019-10-01 15:33:05] echo "0a" > /app/public/diff/adriatic_sea_sp2_occurrences_14720.diff
[CMD] [2019-10-01 15:33:06] tail -n +1 /app/public/converted_csv/adriatic_sea_sp2_occurrences_14720.csv >> /app/public/diff/adriatic_sea_sp2_occurrences_14720.diff
[CMD] [2019-10-01 15:33:08] echo "." >> /app/public/diff/adriatic_sea_sp2_occurrences_14720.diff
[CMD] [2019-10-01 15:33:09] echo "0a" > /app/public/diff/adriatic_sea_sp2_measurements_14721.diff
[CMD] [2019-10-01 15:33:11] tail -n +1 /app/public/converted_csv/adriatic_sea_sp2_measurements_14721.csv >> /app/public/diff/adriatic_sea_sp2_measurements_14721.diff
[CMD] [2019-10-01 15:33:13] echo "." >> /app/public/diff/adriatic_sea_sp2_measurements_14721.diff
[STOP] [2019-10-01 15:33:14] calculate_delta
[START] [2019-10-01 15:33:14] parse_diff_and_store
[INFO] [2019-10-01 15:33:16] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-01 15:33:17] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-01 15:33:20] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-01 15:33:22] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-01 15:33:34] Storing 2 References
[INFO] [2019-10-01 15:33:34] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-01 15:33:34] Average Time: 0.0
[INFO] [2019-10-01 15:33:34] Total Time: 1s
[INFO] [2019-10-01 15:33:34] Storing 4442 ScientificNames
[INFO] [2019-10-01 15:33:34] Processing group of 4442 in 5 groups of 1000
[INFO] [2019-10-01 15:33:36] Average Time: 0.362
[INFO] [2019-10-01 15:33:36] Total Time: 2s
[INFO] [2019-10-01 15:33:36] Storing 4442 Nodes
[INFO] [2019-10-01 15:33:36] Processing group of 4442 in 5 groups of 1000
[INFO] [2019-10-01 15:33:37] Average Time: 0.252
[INFO] [2019-10-01 15:33:37] Total Time: 2s
[INFO] [2019-10-01 15:33:37] Storing 2173 Occurrences
[INFO] [2019-10-01 15:33:37] Processing group of 2173 in 3 groups of 1000
[INFO] [2019-10-01 15:33:38] Average Time: 0.077
[INFO] [2019-10-01 15:33:38] Total Time: 1s
[INFO] [2019-10-01 15:33:38] Storing 4346 TraitsReferences
[INFO] [2019-10-01 15:33:38] Processing group of 4346 in 5 groups of 1000
[INFO] [2019-10-01 15:33:38] Average Time: 0.106
[INFO] [2019-10-01 15:33:38] Total Time: 1s
[INFO] [2019-10-01 15:33:38] Storing 4346 Traits
[INFO] [2019-10-01 15:33:38] Processing group of 4346 in 5 groups of 1000
[INFO] [2019-10-01 15:33:40] Average Time: 0.308
[INFO] [2019-10-01 15:33:40] Total Time: 2s
[INFO] [2019-10-01 15:33:40] Storing 4345 MetaTraits
[INFO] [2019-10-01 15:33:40] Processing group of 4345 in 5 groups of 1000
[INFO] [2019-10-01 15:33:40] Average Time: 0.13
[INFO] [2019-10-01 15:33:40] Total Time: 1s
[STOP] [2019-10-01 15:33:40] parse_diff_and_store
[START] [2019-10-01 15:33:40] resolve_keys
[INFO] [2019-10-01 15:34:04] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-01 15:34:05] traits to occurrences...
[INFO] [2019-10-01 15:34:07] traits to nodes (through occurrences)...
[INFO] [2019-10-01 15:34:07] Traits to sex term...
[INFO] [2019-10-01 15:34:08] Traits to lifestage term...
[INFO] [2019-10-01 15:34:08] MetaTraits to traits...
[INFO] [2019-10-01 15:34:09] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-01 15:34:09] Assocs to occurrences...
[INFO] [2019-10-01 15:34:09] Assocs to nodes...
[INFO] [2019-10-01 15:34:09] Assoc to sex term...
[INFO] [2019-10-01 15:34:09] Assoc to lifestage term...
[STOP] [2019-10-01 15:34:09] resolve_keys
[START] [2019-10-01 15:34:09] hold_for_later_1
[STOP] [2019-10-01 15:34:09] hold_for_later_1
[START] [2019-10-01 15:34:09] hold_for_later_2
[STOP] [2019-10-01 15:34:09] hold_for_later_2
[START] [2019-10-01 15:34:09] resolve_missing_parents
[STOP] [2019-10-01 15:34:19] resolve_missing_parents
[START] [2019-10-01 15:34:19] rebuild_nodes
[START] [2019-10-01 15:34:19] Flattener#flatten
[START] [2019-10-01 15:34:19] Flattener#study_resource
[START] [2019-10-01 15:34:19] Flattener#build_ancestry
[STOP] [2019-10-01 15:34:19] Flattener#build_ancestry
[INFO] [2019-10-01 15:34:19] 4442 ancestry keys
[START] [2019-10-01 15:34:19] build_node_ancestors
[INFO] [2019-10-01 15:34:19] old ancestors deleted.
[STOP] [2019-10-01 15:34:20] build_node_ancestors
[START] [2019-10-01 15:34:22] Flattener#propagate_ancestor_ids
[STOP] [2019-10-01 15:34:23] Flattener#propagate_ancestor_ids
[STOP] [2019-10-01 15:34:23] Flattener#flatten
[STOP] [2019-10-01 15:34:23] rebuild_nodes
[START] [2019-10-01 15:34:23] resolve_missing_media_owners
[STOP] [2019-10-01 15:34:23] resolve_missing_media_owners
[START] [2019-10-01 15:34:23] sanitize_media_verbatims
[STOP] [2019-10-01 15:34:23] sanitize_media_verbatims
[START] [2019-10-01 15:34:23] queue_downloads
[STOP] [2019-10-01 15:34:23] queue_downloads
[START] [2019-10-01 15:34:23] parse_names
[WARN] [2019-10-01 15:34:23] I see 4442 names which still need to be parsed.
[STOP] [2019-10-01 15:34:27] parse_names
[START] [2019-10-01 15:34:27] denormalize_canonical_names_to_nodes
[STOP] [2019-10-01 15:34:27] denormalize_canonical_names_to_nodes
[START] [2019-10-01 15:34:27] match_nodes
[START] [2019-10-01 15:34:27] map_all_nodes_to_pages
[STOP] [2019-10-01 15:39:16] map_all_nodes_to_pages
[INFO] [2019-10-01 15:39:16] 294 Unmatched nodes (of 4442)! That's too many to output. First 10: Cassidulina crassa (#47326675); Globocassidulina oblonga (#47327885); Cassidulinoides bradyi (#47326826); Elphidium albiumbilicatum (#47327150); Elphidium poeyanum (#47327737); Bolivina subspinescens (#47326838); Cibicides lobatulus (#47326724); Gyroidinoides umbonatus (#47326716); Cancris auriculus (#47330432); Bigenerina nodosaria (#47326710)
[START] [2019-10-01 15:39:16] update_nodes
[STOP] [2019-10-01 15:39:17] update_nodes
[STOP] [2019-10-01 15:39:17] match_nodes
[START] [2019-10-01 15:39:17] reindex_search
[STOP] [2019-10-01 15:39:27] reindex_search
[START] [2019-10-01 15:39:27] normalize_units
[STOP] [2019-10-01 15:39:27] normalize_units
[START] [2019-10-01 15:39:27] calculate_statistics
[STOP] [2019-10-01 15:39:27] calculate_statistics
[START] [2019-10-01 15:39:27] complete_harvest_instance
[START] [2019-10-01 15:39:27] overall_tsv_creation
[INFO] [2019-10-01 15:39:27] Processing group of 4442 in 1 batches of 10000
[INFO] [2019-10-01 15:40:31] 2173 Traits (unfiltered)...
[INFO] [2019-10-01 15:40:43] 2173 Traits (filtered)...
[INFO] [2019-10-01 15:40:43] 0 Associations (filtered)...
[INFO] [2019-10-01 15:41:22] 10864 metadata added.
[INFO] [2019-10-01 15:41:22] 0 metadata added.
[INFO] [2019-10-01 15:41:22] Average Time: 90.5
[INFO] [2019-10-01 15:41:22] Total Time: 1m56s
[STOP] [2019-10-01 15:41:22] overall_tsv_creation
[INFO] [2019-10-01 15:41:22] Done. Check your files:
[INFO] [2019-10-01 15:41:23] (4442 lines) /app/public/data/adriatic_sea_sp2/publish_nodes.tsv
[INFO] [2019-10-01 15:41:25] (21955 lines) /app/public/data/adriatic_sea_sp2/publish_node_ancestors.tsv
[INFO] [2019-10-01 15:41:26] (4442 lines) /app/public/data/adriatic_sea_sp2/publish_scientific_names.tsv
[INFO] [2019-10-01 15:41:28] (2174 lines) /app/public/data/adriatic_sea_sp2/publish_traits.tsv
[INFO] [2019-10-01 15:41:29] (10865 lines) /app/public/data/adriatic_sea_sp2/publish_metadata.tsv
[STOP] [2019-10-01 15:41:30] complete_harvest_instance
[START] [2019-10-01 15:41:30] completed
[STOP] [2019-10-01 15:41:30] completed
[STOP] [2019-10-01 15:41:30] logged process, took 521.0

Latest Process