Harvest for Venezuela Species List Created 16 Oct 23:32

Stage: completed
Fetched: 16 Oct 23:32
Validated: 16 Oct 23:32
Deltas Created 16 Oct 23:32
Units Normalized: 16 Oct 23:57
Ancestry Built: 16 Oct 23:40
Nodes Matched: 16 Oct 23:56
Names Parsed: 16 Oct 23:40
New Models Stored: 16 Oct 23:36
Indexed: 16 Oct 23:57
Completed: 17 Oct 00:07
Time to Harvest: 1 minute
Harvesting Log

(166 lines)
# Logfile created on 2019-10-16 23:32:23 -0400 by logger.rb/56815
[START] [2019-10-16 23:32:23] logged process
[START] [2019-10-16 23:32:23] create_harvest_instance
[STOP] [2019-10-16 23:32:23] create_harvest_instance
[START] [2019-10-16 23:32:24] fetch_files
[STOP] [2019-10-16 23:32:24] fetch_files
[START] [2019-10-16 23:32:24] validate_each_file
[STOP] [2019-10-16 23:32:28] validate_each_file
[START] [2019-10-16 23:32:28] convert_to_csv
[CMD] [2019-10-16 23:32:28] /usr/bin/sort /app/public/converted_csv/venezuela_sp_lis_refs_17633.csv > /app/public/converted_csv/venezuela_sp_lis_refs_17633.csv_sorted
[CMD] [2019-10-16 23:32:29] /usr/bin/sort /app/public/converted_csv/venezuela_sp_lis_nodes_17634.csv > /app/public/converted_csv/venezuela_sp_lis_nodes_17634.csv_sorted
[CMD] [2019-10-16 23:32:29] /usr/bin/sort /app/public/converted_csv/venezuela_sp_lis_occurrences_17635.csv > /app/public/converted_csv/venezuela_sp_lis_occurrences_17635.csv_sorted
[CMD] [2019-10-16 23:32:29] /usr/bin/sort /app/public/converted_csv/venezuela_sp_lis_measurements_17636.csv > /app/public/converted_csv/venezuela_sp_lis_measurements_17636.csv_sorted
[STOP] [2019-10-16 23:32:29] convert_to_csv
[START] [2019-10-16 23:32:29] calculate_delta
[CMD] [2019-10-16 23:32:29] echo "0a" > /app/public/diff/venezuela_sp_lis_refs_17633.diff
[CMD] [2019-10-16 23:32:29] tail -n +1 /app/public/converted_csv/venezuela_sp_lis_refs_17633.csv >> /app/public/diff/venezuela_sp_lis_refs_17633.diff
[CMD] [2019-10-16 23:32:30] echo "." >> /app/public/diff/venezuela_sp_lis_refs_17633.diff
[CMD] [2019-10-16 23:32:30] echo "0a" > /app/public/diff/venezuela_sp_lis_nodes_17634.diff
[CMD] [2019-10-16 23:32:30] tail -n +1 /app/public/converted_csv/venezuela_sp_lis_nodes_17634.csv >> /app/public/diff/venezuela_sp_lis_nodes_17634.diff
[CMD] [2019-10-16 23:32:30] echo "." >> /app/public/diff/venezuela_sp_lis_nodes_17634.diff
[CMD] [2019-10-16 23:32:30] echo "0a" > /app/public/diff/venezuela_sp_lis_occurrences_17635.diff
[CMD] [2019-10-16 23:32:30] tail -n +1 /app/public/converted_csv/venezuela_sp_lis_occurrences_17635.csv >> /app/public/diff/venezuela_sp_lis_occurrences_17635.diff
[CMD] [2019-10-16 23:32:31] echo "." >> /app/public/diff/venezuela_sp_lis_occurrences_17635.diff
[CMD] [2019-10-16 23:32:31] echo "0a" > /app/public/diff/venezuela_sp_lis_measurements_17636.diff
[CMD] [2019-10-16 23:32:31] tail -n +1 /app/public/converted_csv/venezuela_sp_lis_measurements_17636.csv >> /app/public/diff/venezuela_sp_lis_measurements_17636.diff
[CMD] [2019-10-16 23:32:31] echo "." >> /app/public/diff/venezuela_sp_lis_measurements_17636.diff
[STOP] [2019-10-16 23:32:31] calculate_delta
[START] [2019-10-16 23:32:31] parse_diff_and_store
[INFO] [2019-10-16 23:32:31] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-16 23:32:31] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-16 23:32:45] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-16 23:32:49] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-16 23:35:26] Storing 2 References
[INFO] [2019-10-16 23:35:26] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-16 23:35:26] Average Time: 0.0
[INFO] [2019-10-16 23:35:26] Total Time: 1s
[INFO] [2019-10-16 23:35:26] Storing 33362 ScientificNames
[INFO] [2019-10-16 23:35:26] Processing group of 33362 in 34 groups of 1000
[INFO] [2019-10-16 23:35:38] Average Time: 0.357
[INFO] [2019-10-16 23:35:38] Total Time: 13s
[INFO] [2019-10-16 23:35:38] last 3 / first 3: 1.41
[INFO] [2019-10-16 23:35:38] Std.Dev: 0.08366600265340755; Max: 0.77
[INFO] [2019-10-16 23:35:38] Storing 33362 Nodes
[INFO] [2019-10-16 23:35:38] Processing group of 33362 in 34 groups of 1000
[INFO] [2019-10-16 23:35:49] Average Time: 0.296
[INFO] [2019-10-16 23:35:49] Total Time: 11s
[INFO] [2019-10-16 23:35:49] last 3 / first 3: 0.52
[INFO] [2019-10-16 23:35:49] Std.Dev: 0.09486832980505137; Max: 0.79
[INFO] [2019-10-16 23:35:49] Storing 25546 Occurrences
[INFO] [2019-10-16 23:35:49] Processing group of 25546 in 26 groups of 1000
[INFO] [2019-10-16 23:35:51] Average Time: 0.096
[INFO] [2019-10-16 23:35:51] Total Time: 3s
[INFO] [2019-10-16 23:35:51] last 3 / first 3: 0.71
[INFO] [2019-10-16 23:35:51] Std.Dev: 0.0; Max: 0.15
[INFO] [2019-10-16 23:35:51] Storing 51140 TraitsReferences
[INFO] [2019-10-16 23:35:51] Processing group of 51140 in 52 groups of 1000
[INFO] [2019-10-16 23:35:55] Average Time: 0.064
[INFO] [2019-10-16 23:35:55] Total Time: 4s
[INFO] [2019-10-16 23:35:55] last 3 / first 3: 0.52
[INFO] [2019-10-16 23:35:55] Std.Dev: 0.0; Max: 0.16
[INFO] [2019-10-16 23:35:55] Storing 51139 Traits
[INFO] [2019-10-16 23:35:55] Processing group of 51139 in 52 groups of 1000
[INFO] [2019-10-16 23:36:11] Average Time: 0.311
[INFO] [2019-10-16 23:36:11] Total Time: 17s
[INFO] [2019-10-16 23:36:11] last 3 / first 3: 0.57
[INFO] [2019-10-16 23:36:11] Std.Dev: 0.1414213562373095; Max: 1.19
[INFO] [2019-10-16 23:36:11] Storing 51090 MetaTraits
[INFO] [2019-10-16 23:36:11] Processing group of 51090 in 52 groups of 1000
[INFO] [2019-10-16 23:36:19] Average Time: 0.141
[INFO] [2019-10-16 23:36:19] Total Time: 8s
[INFO] [2019-10-16 23:36:19] last 3 / first 3: 0.58
[INFO] [2019-10-16 23:36:19] Std.Dev: 0.08366600265340755; Max: 0.57
[STOP] [2019-10-16 23:36:19] parse_diff_and_store
[START] [2019-10-16 23:36:19] resolve_keys
[INFO] [2019-10-16 23:38:04] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-16 23:38:11] traits to occurrences...
[INFO] [2019-10-16 23:38:21] traits to nodes (through occurrences)...
[INFO] [2019-10-16 23:38:22] Traits to sex term...
[INFO] [2019-10-16 23:38:31] Traits to lifestage term...
[INFO] [2019-10-16 23:38:39] MetaTraits to traits...
[INFO] [2019-10-16 23:38:42] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-16 23:38:49] Assocs to occurrences...
[INFO] [2019-10-16 23:38:49] Assocs to nodes...
[INFO] [2019-10-16 23:38:49] Assoc to sex term...
[INFO] [2019-10-16 23:38:49] Assoc to lifestage term...
[STOP] [2019-10-16 23:38:49] resolve_keys
[START] [2019-10-16 23:38:49] hold_for_later_1
[STOP] [2019-10-16 23:38:49] hold_for_later_1
[START] [2019-10-16 23:38:49] hold_for_later_2
[STOP] [2019-10-16 23:38:49] hold_for_later_2
[START] [2019-10-16 23:38:49] resolve_missing_parents
[STOP] [2019-10-16 23:39:45] resolve_missing_parents
[START] [2019-10-16 23:39:45] rebuild_nodes
[START] [2019-10-16 23:39:45] Flattener#flatten
[START] [2019-10-16 23:39:45] Flattener#study_resource
[START] [2019-10-16 23:39:45] Flattener#build_ancestry
[STOP] [2019-10-16 23:39:50] Flattener#build_ancestry
[INFO] [2019-10-16 23:39:50] 33362 ancestry keys
[START] [2019-10-16 23:39:50] build_node_ancestors
[INFO] [2019-10-16 23:39:50] old ancestors deleted.
[STOP] [2019-10-16 23:40:10] build_node_ancestors
[START] [2019-10-16 23:40:14] Flattener#propagate_ancestor_ids
[STOP] [2019-10-16 23:40:18] Flattener#propagate_ancestor_ids
[STOP] [2019-10-16 23:40:18] Flattener#flatten
[STOP] [2019-10-16 23:40:18] rebuild_nodes
[START] [2019-10-16 23:40:18] resolve_missing_media_owners
[STOP] [2019-10-16 23:40:18] resolve_missing_media_owners
[START] [2019-10-16 23:40:18] sanitize_media_verbatims
[STOP] [2019-10-16 23:40:18] sanitize_media_verbatims
[START] [2019-10-16 23:40:18] queue_downloads
[STOP] [2019-10-16 23:40:18] queue_downloads
[START] [2019-10-16 23:40:18] parse_names
[WARN] [2019-10-16 23:40:18] I see 33362 names which still need to be parsed.
[STOP] [2019-10-16 23:40:43] parse_names
[START] [2019-10-16 23:40:43] denormalize_canonical_names_to_nodes
[STOP] [2019-10-16 23:40:43] denormalize_canonical_names_to_nodes
[START] [2019-10-16 23:40:43] match_nodes
[START] [2019-10-16 23:40:43] map_all_nodes_to_pages
[STOP] [2019-10-16 23:56:21] map_all_nodes_to_pages
[INFO] [2019-10-16 23:56:21] 2022 Unmatched nodes (of 33362)! That's too many to output. First 10: Artibeus glauca (#52643989); Platyrrhinus albericoi (#52644689); Dermanura glaucus (#52633115); Dermanura gnomus (#52636569); Dermanura cinereus (#52663007); Hylaeamys capito (#52650844); Hydrochoeris (#52651620); Hydrochoeris hydrochaeris (#52651619); Hydrochaeridae (#52656182); Hydrochaeris (#52656181)
[START] [2019-10-16 23:56:21] update_nodes
[STOP] [2019-10-16 23:56:32] update_nodes
[STOP] [2019-10-16 23:56:32] match_nodes
[START] [2019-10-16 23:56:32] reindex_search
[STOP] [2019-10-16 23:57:37] reindex_search
[START] [2019-10-16 23:57:37] normalize_units
[STOP] [2019-10-16 23:57:38] normalize_units
[START] [2019-10-16 23:57:38] calculate_statistics
[STOP] [2019-10-16 23:57:38] calculate_statistics
[START] [2019-10-16 23:57:38] complete_harvest_instance
[START] [2019-10-16 23:57:38] overall_tsv_creation
[INFO] [2019-10-16 23:57:38] Processing group of 33362 in 4 batches of 10000
[INFO] [2019-10-16 23:59:03] 6702 Traits (unfiltered)...
[INFO] [2019-10-16 23:59:16] 6702 Traits (filtered)...
[INFO] [2019-10-16 23:59:16] 0 Associations (filtered)...
[INFO] [2019-10-17 00:00:06] 33508 metadata added.
[INFO] [2019-10-17 00:00:06] 0 metadata added.
[INFO] [2019-10-17 00:01:35] 7984 Traits (unfiltered)...
[INFO] [2019-10-17 00:01:48] 7984 Traits (filtered)...
[INFO] [2019-10-17 00:01:48] 0 Associations (filtered)...
[INFO] [2019-10-17 00:02:41] 39912 metadata added.
[INFO] [2019-10-17 00:02:41] 0 metadata added.
[INFO] [2019-10-17 00:04:15] 8018 Traits (unfiltered)...
[INFO] [2019-10-17 00:04:28] 8018 Traits (filtered)...
[INFO] [2019-10-17 00:04:28] 0 Associations (filtered)...
[INFO] [2019-10-17 00:05:22] 40067 metadata added.
[INFO] [2019-10-17 00:05:22] 0 metadata added.
[INFO] [2019-10-17 00:06:22] 2842 Traits (unfiltered)...
[INFO] [2019-10-17 00:06:35] 2842 Traits (filtered)...
[INFO] [2019-10-17 00:06:35] 0 Associations (filtered)...
[INFO] [2019-10-17 00:07:16] 14193 metadata added.
[INFO] [2019-10-17 00:07:16] 0 metadata added.
[INFO] [2019-10-17 00:07:17] Average Time: 118.755
[INFO] [2019-10-17 00:07:17] Total Time: 9m39s
[STOP] [2019-10-17 00:07:17] overall_tsv_creation
[INFO] [2019-10-17 00:07:17] Done. Check your files:
[INFO] [2019-10-17 00:07:17] (33362 lines) /app/public/data/venezuela_sp_lis/publish_nodes.tsv
[INFO] [2019-10-17 00:07:17] (155471 lines) /app/public/data/venezuela_sp_lis/publish_node_ancestors.tsv
[INFO] [2019-10-17 00:07:17] (33362 lines) /app/public/data/venezuela_sp_lis/publish_scientific_names.tsv
[INFO] [2019-10-17 00:07:17] (25547 lines) /app/public/data/venezuela_sp_lis/publish_traits.tsv
[INFO] [2019-10-17 00:07:17] (127681 lines) /app/public/data/venezuela_sp_lis/publish_metadata.tsv
[STOP] [2019-10-17 00:07:17] complete_harvest_instance
[START] [2019-10-17 00:07:17] completed
[STOP] [2019-10-17 00:07:17] completed
[STOP] [2019-10-17 00:07:17] logged process, took 2094.27
Latest Process

16 Oct 23:32 EXITED currently processing 0