Harvest for Portugal Species List Created 15 Oct 12:58

Stage: completed
Fetched: 15 Oct 12:58
Validated: 15 Oct 12:58
Deltas Created 15 Oct 12:58
Units Normalized: 15 Oct 13:23
Ancestry Built: 15 Oct 13:03
Nodes Matched: 15 Oct 13:22
Names Parsed: 15 Oct 13:03
New Models Stored: 15 Oct 13:00
Indexed: 15 Oct 13:23
Completed: 15 Oct 13:30
Time to Harvest: 1 minute
Harvesting Log

(161 lines)
# Logfile created on 2019-10-15 12:58:41 -0400 by logger.rb/56815
[START] [2019-10-15 12:58:41] logged process
[START] [2019-10-15 12:58:41] create_harvest_instance
[STOP] [2019-10-15 12:58:41] create_harvest_instance
[START] [2019-10-15 12:58:41] fetch_files
[STOP] [2019-10-15 12:58:41] fetch_files
[START] [2019-10-15 12:58:41] validate_each_file
[STOP] [2019-10-15 12:58:45] validate_each_file
[START] [2019-10-15 12:58:45] convert_to_csv
[CMD] [2019-10-15 12:58:45] /usr/bin/sort /app/public/converted_csv/portugal_sp_list_refs_16941.csv > /app/public/converted_csv/portugal_sp_list_refs_16941.csv_sorted
[CMD] [2019-10-15 12:58:46] /usr/bin/sort /app/public/converted_csv/portugal_sp_list_nodes_16942.csv > /app/public/converted_csv/portugal_sp_list_nodes_16942.csv_sorted
[CMD] [2019-10-15 12:58:46] /usr/bin/sort /app/public/converted_csv/portugal_sp_list_occurrences_16943.csv > /app/public/converted_csv/portugal_sp_list_occurrences_16943.csv_sorted
[CMD] [2019-10-15 12:58:46] /usr/bin/sort /app/public/converted_csv/portugal_sp_list_measurements_16944.csv > /app/public/converted_csv/portugal_sp_list_measurements_16944.csv_sorted
[STOP] [2019-10-15 12:58:47] convert_to_csv
[START] [2019-10-15 12:58:47] calculate_delta
[CMD] [2019-10-15 12:58:47] echo "0a" > /app/public/diff/portugal_sp_list_refs_16941.diff
[CMD] [2019-10-15 12:58:47] tail -n +1 /app/public/converted_csv/portugal_sp_list_refs_16941.csv >> /app/public/diff/portugal_sp_list_refs_16941.diff
[CMD] [2019-10-15 12:58:47] echo "." >> /app/public/diff/portugal_sp_list_refs_16941.diff
[CMD] [2019-10-15 12:58:47] echo "0a" > /app/public/diff/portugal_sp_list_nodes_16942.diff
[CMD] [2019-10-15 12:58:48] tail -n +1 /app/public/converted_csv/portugal_sp_list_nodes_16942.csv >> /app/public/diff/portugal_sp_list_nodes_16942.diff
[CMD] [2019-10-15 12:58:48] echo "." >> /app/public/diff/portugal_sp_list_nodes_16942.diff
[CMD] [2019-10-15 12:58:48] echo "0a" > /app/public/diff/portugal_sp_list_occurrences_16943.diff
[CMD] [2019-10-15 12:58:49] tail -n +1 /app/public/converted_csv/portugal_sp_list_occurrences_16943.csv >> /app/public/diff/portugal_sp_list_occurrences_16943.diff
[CMD] [2019-10-15 12:58:49] echo "." >> /app/public/diff/portugal_sp_list_occurrences_16943.diff
[CMD] [2019-10-15 12:58:49] echo "0a" > /app/public/diff/portugal_sp_list_measurements_16944.diff
[CMD] [2019-10-15 12:58:49] tail -n +1 /app/public/converted_csv/portugal_sp_list_measurements_16944.csv >> /app/public/diff/portugal_sp_list_measurements_16944.diff
[CMD] [2019-10-15 12:58:50] echo "." >> /app/public/diff/portugal_sp_list_measurements_16944.diff
[STOP] [2019-10-15 12:58:50] calculate_delta
[START] [2019-10-15 12:58:50] parse_diff_and_store
[INFO] [2019-10-15 12:58:50] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-15 12:58:51] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-15 12:58:59] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-15 12:59:01] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-15 13:00:17] Storing 2 References
[INFO] [2019-10-15 13:00:17] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-15 13:00:17] Average Time: 0.0
[INFO] [2019-10-15 13:00:17] Total Time: 1s
[INFO] [2019-10-15 13:00:17] Storing 21222 ScientificNames
[INFO] [2019-10-15 13:00:17] Processing group of 21222 in 22 groups of 1000
[INFO] [2019-10-15 13:00:25] Average Time: 0.379
[INFO] [2019-10-15 13:00:25] Total Time: 9s
[INFO] [2019-10-15 13:00:25] last 3 / first 3: 0.66
[INFO] [2019-10-15 13:00:25] Std.Dev: 0.1224744871391589; Max: 0.71
[INFO] [2019-10-15 13:00:25] Storing 21222 Nodes
[INFO] [2019-10-15 13:00:25] Processing group of 21222 in 22 groups of 1000
[INFO] [2019-10-15 13:00:32] Average Time: 0.284
[INFO] [2019-10-15 13:00:32] Total Time: 7s
[INFO] [2019-10-15 13:00:32] last 3 / first 3: 0.82
[INFO] [2019-10-15 13:00:32] Std.Dev: 0.05477225575051661; Max: 0.36
[INFO] [2019-10-15 13:00:32] Storing 13014 Occurrences
[INFO] [2019-10-15 13:00:32] Processing group of 13014 in 14 groups of 1000
[INFO] [2019-10-15 13:00:33] Average Time: 0.097
[INFO] [2019-10-15 13:00:33] Total Time: 2s
[INFO] [2019-10-15 13:00:33] last 3 / first 3: 0.64
[INFO] [2019-10-15 13:00:33] Std.Dev: 0.03162277660168379; Max: 0.13
[INFO] [2019-10-15 13:00:33] Storing 26290 TraitsReferences
[INFO] [2019-10-15 13:00:33] Processing group of 26290 in 27 groups of 1000
[INFO] [2019-10-15 13:00:35] Average Time: 0.073
[INFO] [2019-10-15 13:00:35] Total Time: 3s
[INFO] [2019-10-15 13:00:35] last 3 / first 3: 0.57
[INFO] [2019-10-15 13:00:35] Std.Dev: 0.0; Max: 0.15
[INFO] [2019-10-15 13:00:35] Storing 26289 Traits
[INFO] [2019-10-15 13:00:35] Processing group of 26289 in 27 groups of 1000
[INFO] [2019-10-15 13:00:44] Average Time: 0.327
[INFO] [2019-10-15 13:00:44] Total Time: 9s
[INFO] [2019-10-15 13:00:44] last 3 / first 3: 1.21
[INFO] [2019-10-15 13:00:44] Std.Dev: 0.11401754250991379; Max: 0.66
[INFO] [2019-10-15 13:00:44] Storing 26274 MetaTraits
[INFO] [2019-10-15 13:00:44] Processing group of 26274 in 27 groups of 1000
[INFO] [2019-10-15 13:00:47] Average Time: 0.107
[INFO] [2019-10-15 13:00:47] Total Time: 4s
[INFO] [2019-10-15 13:00:47] last 3 / first 3: 0.73
[INFO] [2019-10-15 13:00:47] Std.Dev: 0.03162277660168379; Max: 0.16
[STOP] [2019-10-15 13:00:47] parse_diff_and_store
[START] [2019-10-15 13:00:47] resolve_keys
[INFO] [2019-10-15 13:01:58] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-15 13:02:03] traits to occurrences...
[INFO] [2019-10-15 13:02:07] traits to nodes (through occurrences)...
[INFO] [2019-10-15 13:02:08] Traits to sex term...
[INFO] [2019-10-15 13:02:12] Traits to lifestage term...
[INFO] [2019-10-15 13:02:16] MetaTraits to traits...
[INFO] [2019-10-15 13:02:18] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-15 13:02:22] Assocs to occurrences...
[INFO] [2019-10-15 13:02:22] Assocs to nodes...
[INFO] [2019-10-15 13:02:22] Assoc to sex term...
[INFO] [2019-10-15 13:02:22] Assoc to lifestage term...
[STOP] [2019-10-15 13:02:22] resolve_keys
[START] [2019-10-15 13:02:22] hold_for_later_1
[STOP] [2019-10-15 13:02:22] hold_for_later_1
[START] [2019-10-15 13:02:22] hold_for_later_2
[STOP] [2019-10-15 13:02:22] hold_for_later_2
[START] [2019-10-15 13:02:22] resolve_missing_parents
[STOP] [2019-10-15 13:03:00] resolve_missing_parents
[START] [2019-10-15 13:03:00] rebuild_nodes
[START] [2019-10-15 13:03:00] Flattener#flatten
[START] [2019-10-15 13:03:00] Flattener#study_resource
[START] [2019-10-15 13:03:00] Flattener#build_ancestry
[STOP] [2019-10-15 13:03:01] Flattener#build_ancestry
[INFO] [2019-10-15 13:03:01] 21222 ancestry keys
[START] [2019-10-15 13:03:01] build_node_ancestors
[INFO] [2019-10-15 13:03:01] old ancestors deleted.
[STOP] [2019-10-15 13:03:06] build_node_ancestors
[START] [2019-10-15 13:03:11] Flattener#propagate_ancestor_ids
[STOP] [2019-10-15 13:03:13] Flattener#propagate_ancestor_ids
[STOP] [2019-10-15 13:03:13] Flattener#flatten
[STOP] [2019-10-15 13:03:13] rebuild_nodes
[START] [2019-10-15 13:03:13] resolve_missing_media_owners
[STOP] [2019-10-15 13:03:13] resolve_missing_media_owners
[START] [2019-10-15 13:03:13] sanitize_media_verbatims
[STOP] [2019-10-15 13:03:13] sanitize_media_verbatims
[START] [2019-10-15 13:03:13] queue_downloads
[STOP] [2019-10-15 13:03:13] queue_downloads
[START] [2019-10-15 13:03:13] parse_names
[WARN] [2019-10-15 13:03:13] I see 21222 names which still need to be parsed.
[STOP] [2019-10-15 13:03:31] parse_names
[START] [2019-10-15 13:03:31] denormalize_canonical_names_to_nodes
[STOP] [2019-10-15 13:03:31] denormalize_canonical_names_to_nodes
[START] [2019-10-15 13:03:31] match_nodes
[START] [2019-10-15 13:03:31] map_all_nodes_to_pages
[STOP] [2019-10-15 13:22:41] map_all_nodes_to_pages
[INFO] [2019-10-15 13:22:41] 1887 Unmatched nodes (of 21222)! That's too many to output. First 10: Larus melanocephalus (#51653106); Larus audouinii (#51654547); Thalaseus (#51649394); Thalaseus sandvicensis (#51649393); Thalaseus bengalensis (#51661611); Thalaseus elegans (#51667644); Onychoprion fuscata (#51661997); Philomachus (#51649702); Philomachus pugnax (#51649701); Limnodromus (#51653283)
[START] [2019-10-15 13:22:41] update_nodes
[STOP] [2019-10-15 13:22:49] update_nodes
[STOP] [2019-10-15 13:22:49] match_nodes
[START] [2019-10-15 13:22:49] reindex_search
[STOP] [2019-10-15 13:23:41] reindex_search
[START] [2019-10-15 13:23:41] normalize_units
[STOP] [2019-10-15 13:23:42] normalize_units
[START] [2019-10-15 13:23:42] calculate_statistics
[STOP] [2019-10-15 13:23:42] calculate_statistics
[START] [2019-10-15 13:23:42] complete_harvest_instance
[START] [2019-10-15 13:23:42] overall_tsv_creation
[INFO] [2019-10-15 13:23:42] Processing group of 21222 in 3 batches of 10000
[INFO] [2019-10-15 13:25:11] 5911 Traits (unfiltered)...
[INFO] [2019-10-15 13:25:24] 5911 Traits (filtered)...
[INFO] [2019-10-15 13:25:25] 0 Associations (filtered)...
[INFO] [2019-10-15 13:26:14] 29553 metadata added.
[INFO] [2019-10-15 13:26:14] 0 metadata added.
[INFO] [2019-10-15 13:27:45] 6382 Traits (unfiltered)...
[INFO] [2019-10-15 13:28:00] 6382 Traits (filtered)...
[INFO] [2019-10-15 13:28:00] 0 Associations (filtered)...
[INFO] [2019-10-15 13:28:52] 31899 metadata added.
[INFO] [2019-10-15 13:28:52] 0 metadata added.
[INFO] [2019-10-15 13:29:43] 721 Traits (unfiltered)...
[INFO] [2019-10-15 13:29:57] 721 Traits (filtered)...
[INFO] [2019-10-15 13:29:57] 0 Associations (filtered)...
[INFO] [2019-10-15 13:30:36] 3604 metadata added.
[INFO] [2019-10-15 13:30:36] 0 metadata added.
[INFO] [2019-10-15 13:30:36] Average Time: 113.077
[INFO] [2019-10-15 13:30:36] Total Time: 6m55s
[STOP] [2019-10-15 13:30:36] overall_tsv_creation
[INFO] [2019-10-15 13:30:36] Done. Check your files:
[INFO] [2019-10-15 13:30:36] (21222 lines) /app/public/data/portugal_sp_list/publish_nodes.tsv
[INFO] [2019-10-15 13:30:36] (71499 lines) /app/public/data/portugal_sp_list/publish_node_ancestors.tsv
[INFO] [2019-10-15 13:30:37] (21222 lines) /app/public/data/portugal_sp_list/publish_scientific_names.tsv
[INFO] [2019-10-15 13:30:37] (13015 lines) /app/public/data/portugal_sp_list/publish_traits.tsv
[INFO] [2019-10-15 13:30:37] (65057 lines) /app/public/data/portugal_sp_list/publish_metadata.tsv
[STOP] [2019-10-15 13:30:37] complete_harvest_instance
[START] [2019-10-15 13:30:37] completed
[STOP] [2019-10-15 13:30:37] completed
[STOP] [2019-10-15 13:30:37] logged process, took 1916.6
Latest Process

15 Oct 12:58 EXITED currently processing 0