Harvest for Gulf of Tomini Species List Created 23 Dec 10:27

Stage: completed
Fetched: 23 Dec 10:27
Validated: 23 Dec 10:27
Deltas Created 23 Dec 10:27
Units Normalized: 23 Dec 10:37
Ancestry Built: 23 Dec 10:27
Nodes Matched: 23 Dec 10:36
Names Parsed: 23 Dec 10:27
New Models Stored: 23 Dec 10:27
Indexed: 23 Dec 10:37
Completed: 23 Dec 10:38
Time to Harvest: less than a minute
Harvesting Log

(139 lines)
# Logfile created on 2019-12-23 10:27:21 -0500 by logger.rb/56815
[START] [2019-12-23 10:27:21] logged process
[START] [2019-12-23 10:27:21] create_harvest_instance
[STOP] [2019-12-23 10:27:21] create_harvest_instance
[START] [2019-12-23 10:27:21] fetch_files
[STOP] [2019-12-23 10:27:22] fetch_files
[START] [2019-12-23 10:27:22] validate_each_file
[STOP] [2019-12-23 10:27:22] validate_each_file
[START] [2019-12-23 10:27:22] convert_to_csv
[CMD] [2019-12-23 10:27:22] /usr/bin/sort /app/public/converted_csv/gulf_tomini_sp_l_refs_19370.csv > /app/public/converted_csv/gulf_tomini_sp_l_refs_19370.csv_sorted
[CMD] [2019-12-23 10:27:22] /usr/bin/sort /app/public/converted_csv/gulf_tomini_sp_l_nodes_19371.csv > /app/public/converted_csv/gulf_tomini_sp_l_nodes_19371.csv_sorted
[CMD] [2019-12-23 10:27:23] /usr/bin/sort /app/public/converted_csv/gulf_tomini_sp_l_occurrences_19372.csv > /app/public/converted_csv/gulf_tomini_sp_l_occurrences_19372.csv_sorted
[CMD] [2019-12-23 10:27:24] /usr/bin/sort /app/public/converted_csv/gulf_tomini_sp_l_measurements_19373.csv > /app/public/converted_csv/gulf_tomini_sp_l_measurements_19373.csv_sorted
[STOP] [2019-12-23 10:27:25] convert_to_csv
[START] [2019-12-23 10:27:25] calculate_delta
[CMD] [2019-12-23 10:27:25] echo "0a" > /app/public/diff/gulf_tomini_sp_l_refs_19370.diff
[CMD] [2019-12-23 10:27:25] tail -n +1 /app/public/converted_csv/gulf_tomini_sp_l_refs_19370.csv >> /app/public/diff/gulf_tomini_sp_l_refs_19370.diff
[CMD] [2019-12-23 10:27:26] echo "." >> /app/public/diff/gulf_tomini_sp_l_refs_19370.diff
[CMD] [2019-12-23 10:27:27] echo "0a" > /app/public/diff/gulf_tomini_sp_l_nodes_19371.diff
[CMD] [2019-12-23 10:27:27] tail -n +1 /app/public/converted_csv/gulf_tomini_sp_l_nodes_19371.csv >> /app/public/diff/gulf_tomini_sp_l_nodes_19371.diff
[CMD] [2019-12-23 10:27:28] echo "." >> /app/public/diff/gulf_tomini_sp_l_nodes_19371.diff
[CMD] [2019-12-23 10:27:29] echo "0a" > /app/public/diff/gulf_tomini_sp_l_occurrences_19372.diff
[CMD] [2019-12-23 10:27:29] tail -n +1 /app/public/converted_csv/gulf_tomini_sp_l_occurrences_19372.csv >> /app/public/diff/gulf_tomini_sp_l_occurrences_19372.diff
[CMD] [2019-12-23 10:27:30] echo "." >> /app/public/diff/gulf_tomini_sp_l_occurrences_19372.diff
[CMD] [2019-12-23 10:27:31] echo "0a" > /app/public/diff/gulf_tomini_sp_l_measurements_19373.diff
[CMD] [2019-12-23 10:27:31] tail -n +1 /app/public/converted_csv/gulf_tomini_sp_l_measurements_19373.csv >> /app/public/diff/gulf_tomini_sp_l_measurements_19373.diff
[CMD] [2019-12-23 10:27:32] echo "." >> /app/public/diff/gulf_tomini_sp_l_measurements_19373.diff
[STOP] [2019-12-23 10:27:33] calculate_delta
[START] [2019-12-23 10:27:33] parse_diff_and_store
[INFO] [2019-12-23 10:27:33] Loading refs diff file into memory (true lines)...
[INFO] [2019-12-23 10:27:34] Loading nodes diff file into memory (true lines)...
[INFO] [2019-12-23 10:27:35] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-12-23 10:27:36] Loading measurements diff file into memory (true lines)...
[INFO] [2019-12-23 10:27:39] Storing 2 References
[INFO] [2019-12-23 10:27:39] Processing group of 2 in 1 groups of 1000
[INFO] [2019-12-23 10:27:39] Average Time: 0.0
[INFO] [2019-12-23 10:27:39] Total Time: 1s
[INFO] [2019-12-23 10:27:39] Storing 1318 ScientificNames
[INFO] [2019-12-23 10:27:39] Processing group of 1318 in 2 groups of 1000
[INFO] [2019-12-23 10:27:40] Average Time: 0.31
[INFO] [2019-12-23 10:27:40] Total Time: 1s
[INFO] [2019-12-23 10:27:40] Storing 1318 Nodes
[INFO] [2019-12-23 10:27:40] Processing group of 1318 in 2 groups of 1000
[INFO] [2019-12-23 10:27:41] Average Time: 0.265
[INFO] [2019-12-23 10:27:41] Total Time: 1s
[INFO] [2019-12-23 10:27:41] Storing 599 Occurrences
[INFO] [2019-12-23 10:27:41] Processing group of 599 in 1 groups of 1000
[INFO] [2019-12-23 10:27:41] Average Time: 0.09
[INFO] [2019-12-23 10:27:41] Total Time: 1s
[INFO] [2019-12-23 10:27:41] Storing 1198 TraitsReferences
[INFO] [2019-12-23 10:27:41] Processing group of 1198 in 2 groups of 1000
[INFO] [2019-12-23 10:27:41] Average Time: 0.09
[INFO] [2019-12-23 10:27:41] Total Time: 1s
[INFO] [2019-12-23 10:27:41] Storing 1198 Traits
[INFO] [2019-12-23 10:27:41] Processing group of 1198 in 2 groups of 1000
[INFO] [2019-12-23 10:27:41] Average Time: 0.265
[INFO] [2019-12-23 10:27:41] Total Time: 1s
[INFO] [2019-12-23 10:27:41] Storing 1198 MetaTraits
[INFO] [2019-12-23 10:27:41] Processing group of 1198 in 2 groups of 1000
[INFO] [2019-12-23 10:27:42] Average Time: 0.105
[INFO] [2019-12-23 10:27:42] Total Time: 1s
[STOP] [2019-12-23 10:27:42] parse_diff_and_store
[START] [2019-12-23 10:27:42] resolve_keys
[INFO] [2019-12-23 10:27:49] Occurrences to nodes (through scientific_names)...
[INFO] [2019-12-23 10:27:49] traits to occurrences...
[INFO] [2019-12-23 10:27:50] traits to nodes (through occurrences)...
[INFO] [2019-12-23 10:27:50] Traits to sex term...
[INFO] [2019-12-23 10:27:50] Traits to lifestage term...
[INFO] [2019-12-23 10:27:50] MetaTraits to traits...
[INFO] [2019-12-23 10:27:50] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-12-23 10:27:50] Assocs to occurrences...
[INFO] [2019-12-23 10:27:50] Assocs to nodes...
[INFO] [2019-12-23 10:27:50] Assoc to sex term...
[INFO] [2019-12-23 10:27:50] Assoc to lifestage term...
[STOP] [2019-12-23 10:27:50] resolve_keys
[START] [2019-12-23 10:27:50] hold_for_later_1
[STOP] [2019-12-23 10:27:50] hold_for_later_1
[START] [2019-12-23 10:27:50] hold_for_later_2
[STOP] [2019-12-23 10:27:50] hold_for_later_2
[START] [2019-12-23 10:27:50] resolve_missing_parents
[STOP] [2019-12-23 10:27:51] resolve_missing_parents
[START] [2019-12-23 10:27:51] rebuild_nodes
[START] [2019-12-23 10:27:51] Flattener#flatten
[START] [2019-12-23 10:27:51] Flattener#study_resource
[START] [2019-12-23 10:27:51] Flattener#build_ancestry
[STOP] [2019-12-23 10:27:51] Flattener#build_ancestry
[INFO] [2019-12-23 10:27:51] 1318 ancestry keys
[START] [2019-12-23 10:27:51] build_node_ancestors
[INFO] [2019-12-23 10:27:51] old ancestors deleted.
[STOP] [2019-12-23 10:27:51] build_node_ancestors
[START] [2019-12-23 10:27:52] Flattener#propagate_ancestor_ids
[STOP] [2019-12-23 10:27:52] Flattener#propagate_ancestor_ids
[STOP] [2019-12-23 10:27:52] Flattener#flatten
[STOP] [2019-12-23 10:27:52] rebuild_nodes
[START] [2019-12-23 10:27:52] resolve_missing_media_owners
[STOP] [2019-12-23 10:27:52] resolve_missing_media_owners
[START] [2019-12-23 10:27:52] sanitize_media_verbatims
[STOP] [2019-12-23 10:27:52] sanitize_media_verbatims
[START] [2019-12-23 10:27:52] queue_downloads
[STOP] [2019-12-23 10:27:52] queue_downloads
[START] [2019-12-23 10:27:52] parse_names
[WARN] [2019-12-23 10:27:52] I see 1318 names which still need to be parsed.
[STOP] [2019-12-23 10:27:54] parse_names
[START] [2019-12-23 10:27:54] denormalize_canonical_names_to_nodes
[STOP] [2019-12-23 10:27:54] denormalize_canonical_names_to_nodes
[START] [2019-12-23 10:27:54] match_nodes
[START] [2019-12-23 10:27:54] map_all_nodes_to_pages
[STOP] [2019-12-23 10:36:54] map_all_nodes_to_pages
[INFO] [2019-12-23 10:36:54] 56 Unmatched nodes (of 1318)! That's too many to output. First 10: Favia valenciennesii (#61966630); Acrhelia (#61965717); Pectiniidae (#61965810); Paraclavarina (#61966385); Euphyllidae (#61966365); Scorpaeniformes (#61965730); Stephanoberyciformes (#61965908); Pleurogona (#61966192); Ostreoida (#61965361); Propeamussidae (#61966617)
[START] [2019-12-23 10:36:54] update_nodes
[STOP] [2019-12-23 10:36:54] update_nodes
[STOP] [2019-12-23 10:36:54] match_nodes
[START] [2019-12-23 10:36:54] reindex_search
[STOP] [2019-12-23 10:37:00] reindex_search
[START] [2019-12-23 10:37:00] normalize_units
[STOP] [2019-12-23 10:37:00] normalize_units
[START] [2019-12-23 10:37:00] calculate_statistics
[STOP] [2019-12-23 10:37:00] calculate_statistics
[START] [2019-12-23 10:37:00] complete_harvest_instance
[START] [2019-12-23 10:37:00] overall_tsv_creation
[INFO] [2019-12-23 10:37:00] Processing group of 1318 in 1 batches of 10000
[INFO] [2019-12-23 10:37:51] 599 Traits (unfiltered)...
[INFO] [2019-12-23 10:38:04] 599 Traits (filtered)...
[INFO] [2019-12-23 10:38:04] 0 Associations (filtered)...
[INFO] [2019-12-23 10:38:43] 2995 metadata added.
[INFO] [2019-12-23 10:38:43] 0 metadata added.
[INFO] [2019-12-23 10:38:43] Average Time: 79.72
[INFO] [2019-12-23 10:38:43] Total Time: 1m43s
[STOP] [2019-12-23 10:38:43] overall_tsv_creation
[INFO] [2019-12-23 10:38:43] Done. Check your files:
[INFO] [2019-12-23 10:38:43] (1318 lines) /app/public/data/gulf_tomini_sp_l/publish_nodes.tsv
[INFO] [2019-12-23 10:38:44] (6426 lines) /app/public/data/gulf_tomini_sp_l/publish_node_ancestors.tsv
[INFO] [2019-12-23 10:38:45] (1318 lines) /app/public/data/gulf_tomini_sp_l/publish_scientific_names.tsv
[INFO] [2019-12-23 10:38:46] (600 lines) /app/public/data/gulf_tomini_sp_l/publish_traits.tsv
[INFO] [2019-12-23 10:38:46] (2996 lines) /app/public/data/gulf_tomini_sp_l/publish_metadata.tsv
[STOP] [2019-12-23 10:38:46] complete_harvest_instance
[START] [2019-12-23 10:38:46] completed
[STOP] [2019-12-23 10:38:46] completed
[STOP] [2019-12-23 10:38:46] logged process, took 685.27
Latest Process

23 Dec 10:27 EXITED currently processing 0