Harvest for Maldives Species List Created 14 Oct 04:42

Stage: completed
Fetched: 14 Oct 04:42
Validated: 14 Oct 04:42
Deltas Created 14 Oct 04:42
Units Normalized: 14 Oct 04:48
Ancestry Built: 14 Oct 04:44
Nodes Matched: 14 Oct 04:48
Names Parsed: 14 Oct 04:44
New Models Stored: 14 Oct 04:43
Indexed: 14 Oct 04:48
Completed: 14 Oct 04:50
Time to Harvest: less than a minute
Harvesting Log

(139 lines)
# Logfile created on 2019-10-14 04:42:55 -0400 by logger.rb/56815
[START] [2019-10-14 04:42:55] logged process
[START] [2019-10-14 04:42:55] create_harvest_instance
[STOP] [2019-10-14 04:42:56] create_harvest_instance
[START] [2019-10-14 04:42:56] fetch_files
[STOP] [2019-10-14 04:42:56] fetch_files
[START] [2019-10-14 04:42:56] validate_each_file
[STOP] [2019-10-14 04:42:57] validate_each_file
[START] [2019-10-14 04:42:57] convert_to_csv
[CMD] [2019-10-14 04:42:57] /usr/bin/sort /app/public/converted_csv/maldives_sp_list_refs_16507.csv > /app/public/converted_csv/maldives_sp_list_refs_16507.csv_sorted
[CMD] [2019-10-14 04:42:57] /usr/bin/sort /app/public/converted_csv/maldives_sp_list_nodes_16508.csv > /app/public/converted_csv/maldives_sp_list_nodes_16508.csv_sorted
[CMD] [2019-10-14 04:42:57] /usr/bin/sort /app/public/converted_csv/maldives_sp_list_occurrences_16509.csv > /app/public/converted_csv/maldives_sp_list_occurrences_16509.csv_sorted
[CMD] [2019-10-14 04:42:57] /usr/bin/sort /app/public/converted_csv/maldives_sp_list_measurements_16510.csv > /app/public/converted_csv/maldives_sp_list_measurements_16510.csv_sorted
[STOP] [2019-10-14 04:42:57] convert_to_csv
[START] [2019-10-14 04:42:57] calculate_delta
[CMD] [2019-10-14 04:42:57] echo "0a" > /app/public/diff/maldives_sp_list_refs_16507.diff
[CMD] [2019-10-14 04:42:57] tail -n +1 /app/public/converted_csv/maldives_sp_list_refs_16507.csv >> /app/public/diff/maldives_sp_list_refs_16507.diff
[CMD] [2019-10-14 04:42:58] echo "." >> /app/public/diff/maldives_sp_list_refs_16507.diff
[CMD] [2019-10-14 04:42:58] echo "0a" > /app/public/diff/maldives_sp_list_nodes_16508.diff
[CMD] [2019-10-14 04:42:58] tail -n +1 /app/public/converted_csv/maldives_sp_list_nodes_16508.csv >> /app/public/diff/maldives_sp_list_nodes_16508.diff
[CMD] [2019-10-14 04:42:58] echo "." >> /app/public/diff/maldives_sp_list_nodes_16508.diff
[CMD] [2019-10-14 04:42:58] echo "0a" > /app/public/diff/maldives_sp_list_occurrences_16509.diff
[CMD] [2019-10-14 04:42:58] tail -n +1 /app/public/converted_csv/maldives_sp_list_occurrences_16509.csv >> /app/public/diff/maldives_sp_list_occurrences_16509.diff
[CMD] [2019-10-14 04:42:58] echo "." >> /app/public/diff/maldives_sp_list_occurrences_16509.diff
[CMD] [2019-10-14 04:42:58] echo "0a" > /app/public/diff/maldives_sp_list_measurements_16510.diff
[CMD] [2019-10-14 04:42:58] tail -n +1 /app/public/converted_csv/maldives_sp_list_measurements_16510.csv >> /app/public/diff/maldives_sp_list_measurements_16510.diff
[CMD] [2019-10-14 04:42:58] echo "." >> /app/public/diff/maldives_sp_list_measurements_16510.diff
[STOP] [2019-10-14 04:42:58] calculate_delta
[START] [2019-10-14 04:42:58] parse_diff_and_store
[INFO] [2019-10-14 04:42:59] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-14 04:42:59] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-14 04:43:01] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-14 04:43:01] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-14 04:43:16] Storing 2 References
[INFO] [2019-10-14 04:43:16] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-14 04:43:16] Average Time: 0.0
[INFO] [2019-10-14 04:43:16] Total Time: 1s
[INFO] [2019-10-14 04:43:16] Storing 4646 ScientificNames
[INFO] [2019-10-14 04:43:16] Processing group of 4646 in 5 groups of 1000
[INFO] [2019-10-14 04:43:18] Average Time: 0.326
[INFO] [2019-10-14 04:43:18] Total Time: 2s
[INFO] [2019-10-14 04:43:18] Storing 4646 Nodes
[INFO] [2019-10-14 04:43:18] Processing group of 4646 in 5 groups of 1000
[INFO] [2019-10-14 04:43:21] Average Time: 0.58
[INFO] [2019-10-14 04:43:21] Total Time: 3s
[INFO] [2019-10-14 04:43:21] Storing 2410 Occurrences
[INFO] [2019-10-14 04:43:21] Processing group of 2410 in 3 groups of 1000
[INFO] [2019-10-14 04:43:21] Average Time: 0.103
[INFO] [2019-10-14 04:43:21] Total Time: 1s
[INFO] [2019-10-14 04:43:21] Storing 5140 TraitsReferences
[INFO] [2019-10-14 04:43:21] Processing group of 5140 in 6 groups of 1000
[INFO] [2019-10-14 04:43:21] Average Time: 0.068
[INFO] [2019-10-14 04:43:21] Total Time: 1s
[INFO] [2019-10-14 04:43:21] Storing 5139 Traits
[INFO] [2019-10-14 04:43:21] Processing group of 5139 in 6 groups of 1000
[INFO] [2019-10-14 04:43:23] Average Time: 0.287
[INFO] [2019-10-14 04:43:23] Total Time: 2s
[INFO] [2019-10-14 04:43:23] Storing 5136 MetaTraits
[INFO] [2019-10-14 04:43:23] Processing group of 5136 in 6 groups of 1000
[INFO] [2019-10-14 04:43:24] Average Time: 0.1
[INFO] [2019-10-14 04:43:24] Total Time: 1s
[STOP] [2019-10-14 04:43:24] parse_diff_and_store
[START] [2019-10-14 04:43:24] resolve_keys
[INFO] [2019-10-14 04:43:44] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-14 04:43:46] traits to occurrences...
[INFO] [2019-10-14 04:43:47] traits to nodes (through occurrences)...
[INFO] [2019-10-14 04:43:47] Traits to sex term...
[INFO] [2019-10-14 04:43:48] Traits to lifestage term...
[INFO] [2019-10-14 04:43:49] MetaTraits to traits...
[INFO] [2019-10-14 04:43:50] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-14 04:43:51] Assocs to occurrences...
[INFO] [2019-10-14 04:43:51] Assocs to nodes...
[INFO] [2019-10-14 04:43:51] Assoc to sex term...
[INFO] [2019-10-14 04:43:51] Assoc to lifestage term...
[STOP] [2019-10-14 04:43:51] resolve_keys
[START] [2019-10-14 04:43:51] hold_for_later_1
[STOP] [2019-10-14 04:43:51] hold_for_later_1
[START] [2019-10-14 04:43:51] hold_for_later_2
[STOP] [2019-10-14 04:43:51] hold_for_later_2
[START] [2019-10-14 04:43:51] resolve_missing_parents
[STOP] [2019-10-14 04:43:59] resolve_missing_parents
[START] [2019-10-14 04:43:59] rebuild_nodes
[START] [2019-10-14 04:43:59] Flattener#flatten
[START] [2019-10-14 04:43:59] Flattener#study_resource
[START] [2019-10-14 04:43:59] Flattener#build_ancestry
[STOP] [2019-10-14 04:43:59] Flattener#build_ancestry
[INFO] [2019-10-14 04:44:00] 4646 ancestry keys
[START] [2019-10-14 04:44:00] build_node_ancestors
[INFO] [2019-10-14 04:44:00] old ancestors deleted.
[STOP] [2019-10-14 04:44:00] build_node_ancestors
[START] [2019-10-14 04:44:01] Flattener#propagate_ancestor_ids
[STOP] [2019-10-14 04:44:01] Flattener#propagate_ancestor_ids
[STOP] [2019-10-14 04:44:01] Flattener#flatten
[STOP] [2019-10-14 04:44:01] rebuild_nodes
[START] [2019-10-14 04:44:01] resolve_missing_media_owners
[STOP] [2019-10-14 04:44:01] resolve_missing_media_owners
[START] [2019-10-14 04:44:01] sanitize_media_verbatims
[STOP] [2019-10-14 04:44:01] sanitize_media_verbatims
[START] [2019-10-14 04:44:01] queue_downloads
[STOP] [2019-10-14 04:44:01] queue_downloads
[START] [2019-10-14 04:44:01] parse_names
[WARN] [2019-10-14 04:44:01] I see 4646 names which still need to be parsed.
[STOP] [2019-10-14 04:44:06] parse_names
[START] [2019-10-14 04:44:06] denormalize_canonical_names_to_nodes
[STOP] [2019-10-14 04:44:06] denormalize_canonical_names_to_nodes
[START] [2019-10-14 04:44:06] match_nodes
[START] [2019-10-14 04:44:06] map_all_nodes_to_pages
[STOP] [2019-10-14 04:48:18] map_all_nodes_to_pages
[INFO] [2019-10-14 04:48:18] 395 Unmatched nodes (of 4646)! That's too many to output. First 10: Chromis caeruleus (#50646035); Stegastes luteobrunneus (#50647364); Plectroglyphidodon dicki (#50647485); Cheilodipterus lineatus (#50645972); Nectamia guamensis (#50645838); Nectamia fuscus (#50647088); Ostorhinchus cyanosomus (#50645876); Ostorhinchus robustus (#50647580); Ostorhinchus molluccensis (#50648164); Ostorhinchus robusta (#50648518)
[START] [2019-10-14 04:48:18] update_nodes
[STOP] [2019-10-14 04:48:19] update_nodes
[STOP] [2019-10-14 04:48:19] match_nodes
[START] [2019-10-14 04:48:19] reindex_search
[STOP] [2019-10-14 04:48:30] reindex_search
[START] [2019-10-14 04:48:30] normalize_units
[STOP] [2019-10-14 04:48:30] normalize_units
[START] [2019-10-14 04:48:30] calculate_statistics
[STOP] [2019-10-14 04:48:30] calculate_statistics
[START] [2019-10-14 04:48:30] complete_harvest_instance
[START] [2019-10-14 04:48:30] overall_tsv_creation
[INFO] [2019-10-14 04:48:30] Processing group of 4646 in 1 batches of 10000
[INFO] [2019-10-14 04:49:34] 2410 Traits (unfiltered)...
[INFO] [2019-10-14 04:49:48] 2410 Traits (filtered)...
[INFO] [2019-10-14 04:49:48] 0 Associations (filtered)...
[INFO] [2019-10-14 04:50:30] 12046 metadata added.
[INFO] [2019-10-14 04:50:30] 0 metadata added.
[INFO] [2019-10-14 04:50:30] Average Time: 96.73
[INFO] [2019-10-14 04:50:30] Total Time: 2m1s
[STOP] [2019-10-14 04:50:30] overall_tsv_creation
[INFO] [2019-10-14 04:50:30] Done. Check your files:
[INFO] [2019-10-14 04:50:30] (4646 lines) /app/public/data/maldives_sp_list/publish_nodes.tsv
[INFO] [2019-10-14 04:50:30] (10565 lines) /app/public/data/maldives_sp_list/publish_node_ancestors.tsv
[INFO] [2019-10-14 04:50:30] (4646 lines) /app/public/data/maldives_sp_list/publish_scientific_names.tsv
[INFO] [2019-10-14 04:50:30] (2411 lines) /app/public/data/maldives_sp_list/publish_traits.tsv
[INFO] [2019-10-14 04:50:30] (12047 lines) /app/public/data/maldives_sp_list/publish_metadata.tsv
[STOP] [2019-10-14 04:50:30] complete_harvest_instance
[START] [2019-10-14 04:50:30] completed
[STOP] [2019-10-14 04:50:30] completed
[STOP] [2019-10-14 04:50:30] logged process, took 455.1
Latest Process

14 Oct 04:42 EXITED currently processing 0