Harvest for India Species List Created 13 Oct 11:52

Harvesting Log

# Logfile created on 2019-10-13 11:52:27 -0400 by logger.rb/56815
[START] [2019-10-13 11:52:27] logged process
[START] [2019-10-13 11:52:27] create_harvest_instance
[STOP] [2019-10-13 11:52:27] create_harvest_instance
[START] [2019-10-13 11:52:27] fetch_files
[STOP] [2019-10-13 11:52:27] fetch_files
[START] [2019-10-13 11:52:27] validate_each_file
[STOP] [2019-10-13 11:52:31] validate_each_file
[START] [2019-10-13 11:52:31] convert_to_csv
[CMD] [2019-10-13 11:52:31] /usr/bin/sort /app/public/converted_csv/india_sp_list_refs_16139.csv > /app/public/converted_csv/india_sp_list_refs_16139.csv_sorted
[CMD] [2019-10-13 11:52:31] /usr/bin/sort /app/public/converted_csv/india_sp_list_nodes_16140.csv > /app/public/converted_csv/india_sp_list_nodes_16140.csv_sorted
[CMD] [2019-10-13 11:52:31] /usr/bin/sort /app/public/converted_csv/india_sp_list_occurrences_16141.csv > /app/public/converted_csv/india_sp_list_occurrences_16141.csv_sorted
[CMD] [2019-10-13 11:52:31] /usr/bin/sort /app/public/converted_csv/india_sp_list_measurements_16142.csv > /app/public/converted_csv/india_sp_list_measurements_16142.csv_sorted
[STOP] [2019-10-13 11:52:31] convert_to_csv
[START] [2019-10-13 11:52:31] calculate_delta
[CMD] [2019-10-13 11:52:31] echo "0a" > /app/public/diff/india_sp_list_refs_16139.diff
[CMD] [2019-10-13 11:52:31] tail -n +1 /app/public/converted_csv/india_sp_list_refs_16139.csv >> /app/public/diff/india_sp_list_refs_16139.diff
[CMD] [2019-10-13 11:52:31] echo "." >> /app/public/diff/india_sp_list_refs_16139.diff
[CMD] [2019-10-13 11:52:31] echo "0a" > /app/public/diff/india_sp_list_nodes_16140.diff
[CMD] [2019-10-13 11:52:31] tail -n +1 /app/public/converted_csv/india_sp_list_nodes_16140.csv >> /app/public/diff/india_sp_list_nodes_16140.diff
[CMD] [2019-10-13 11:52:32] echo "." >> /app/public/diff/india_sp_list_nodes_16140.diff
[CMD] [2019-10-13 11:52:32] echo "0a" > /app/public/diff/india_sp_list_occurrences_16141.diff
[CMD] [2019-10-13 11:52:32] tail -n +1 /app/public/converted_csv/india_sp_list_occurrences_16141.csv >> /app/public/diff/india_sp_list_occurrences_16141.diff
[CMD] [2019-10-13 11:52:32] echo "." >> /app/public/diff/india_sp_list_occurrences_16141.diff
[CMD] [2019-10-13 11:52:32] echo "0a" > /app/public/diff/india_sp_list_measurements_16142.diff
[CMD] [2019-10-13 11:52:32] tail -n +1 /app/public/converted_csv/india_sp_list_measurements_16142.csv >> /app/public/diff/india_sp_list_measurements_16142.diff
[CMD] [2019-10-13 11:52:32] echo "." >> /app/public/diff/india_sp_list_measurements_16142.diff
[STOP] [2019-10-13 11:52:32] calculate_delta
[START] [2019-10-13 11:52:32] parse_diff_and_store
[INFO] [2019-10-13 11:52:32] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-13 11:52:32] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-13 11:52:42] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-13 11:52:45] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-13 11:54:30] Storing 2 References
[INFO] [2019-10-13 11:54:30] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-13 11:54:30] Average Time: 0.0
[INFO] [2019-10-13 11:54:30] Total Time: 1s
[INFO] [2019-10-13 11:54:30] Storing 27366 ScientificNames
[INFO] [2019-10-13 11:54:30] Processing group of 27366 in 28 groups of 1000
[INFO] [2019-10-13 11:54:42] Average Time: 0.405
[INFO] [2019-10-13 11:54:42] Total Time: 12s
[INFO] [2019-10-13 11:54:42] last 3 / first 3: 0.71
[INFO] [2019-10-13 11:54:42] Std.Dev: 0.12649110640673517; Max: 0.95
[INFO] [2019-10-13 11:54:42] Storing 27366 Nodes
[INFO] [2019-10-13 11:54:42] Processing group of 27366 in 28 groups of 1000
[INFO] [2019-10-13 11:54:51] Average Time: 0.338
[INFO] [2019-10-13 11:54:51] Total Time: 10s
[INFO] [2019-10-13 11:54:51] last 3 / first 3: 0.92
[INFO] [2019-10-13 11:54:51] Std.Dev: 0.10954451150103323; Max: 0.67
[INFO] [2019-10-13 11:54:51] Storing 17834 Occurrences
[INFO] [2019-10-13 11:54:51] Processing group of 17834 in 18 groups of 1000
[INFO] [2019-10-13 11:54:53] Average Time: 0.109
[INFO] [2019-10-13 11:54:53] Total Time: 3s
[INFO] [2019-10-13 11:54:53] last 3 / first 3: 1.03
[INFO] [2019-10-13 11:54:53] Std.Dev: 0.0; Max: 0.16
[INFO] [2019-10-13 11:54:53] Storing 36240 TraitsReferences
[INFO] [2019-10-13 11:54:53] Processing group of 36240 in 37 groups of 1000
[INFO] [2019-10-13 11:54:56] Average Time: 0.073
[INFO] [2019-10-13 11:54:56] Total Time: 3s
[INFO] [2019-10-13 11:54:56] last 3 / first 3: 0.55
[INFO] [2019-10-13 11:54:56] Std.Dev: 0.0; Max: 0.15
[INFO] [2019-10-13 11:54:56] Storing 36239 Traits
[INFO] [2019-10-13 11:54:56] Processing group of 36239 in 37 groups of 1000
[INFO] [2019-10-13 11:55:09] Average Time: 0.338
[INFO] [2019-10-13 11:55:09] Total Time: 13s
[INFO] [2019-10-13 11:55:09] last 3 / first 3: 0.68
[INFO] [2019-10-13 11:55:09] Std.Dev: 0.11401754250991379; Max: 0.68
[INFO] [2019-10-13 11:55:09] Storing 36200 MetaTraits
[INFO] [2019-10-13 11:55:09] Processing group of 36200 in 37 groups of 1000
[INFO] [2019-10-13 11:55:13] Average Time: 0.115
[INFO] [2019-10-13 11:55:13] Total Time: 5s
[INFO] [2019-10-13 11:55:13] last 3 / first 3: 0.67
[INFO] [2019-10-13 11:55:13] Std.Dev: 0.03162277660168379; Max: 0.17
[STOP] [2019-10-13 11:55:13] parse_diff_and_store
[START] [2019-10-13 11:55:13] resolve_keys
[INFO] [2019-10-13 11:56:30] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-13 11:56:37] traits to occurrences...
[INFO] [2019-10-13 11:56:42] traits to nodes (through occurrences)...
[INFO] [2019-10-13 11:56:43] Traits to sex term...
[INFO] [2019-10-13 11:56:49] Traits to lifestage term...
[INFO] [2019-10-13 11:56:55] MetaTraits to traits...
[INFO] [2019-10-13 11:56:57] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-13 11:57:02] Assocs to occurrences...
[INFO] [2019-10-13 11:57:02] Assocs to nodes...
[INFO] [2019-10-13 11:57:02] Assoc to sex term...
[INFO] [2019-10-13 11:57:02] Assoc to lifestage term...
[STOP] [2019-10-13 11:57:02] resolve_keys
[START] [2019-10-13 11:57:02] hold_for_later_1
[STOP] [2019-10-13 11:57:02] hold_for_later_1
[START] [2019-10-13 11:57:02] hold_for_later_2
[STOP] [2019-10-13 11:57:02] hold_for_later_2
[START] [2019-10-13 11:57:02] resolve_missing_parents
[STOP] [2019-10-13 11:57:53] resolve_missing_parents
[START] [2019-10-13 11:57:53] rebuild_nodes
[START] [2019-10-13 11:57:53] Flattener#flatten
[START] [2019-10-13 11:57:53] Flattener#study_resource
[START] [2019-10-13 11:57:53] Flattener#build_ancestry
[STOP] [2019-10-13 11:57:56] Flattener#build_ancestry
[INFO] [2019-10-13 11:57:56] 27366 ancestry keys
[START] [2019-10-13 11:57:56] build_node_ancestors
[INFO] [2019-10-13 11:57:56] old ancestors deleted.
[STOP] [2019-10-13 11:58:02] build_node_ancestors
[START] [2019-10-13 11:58:07] Flattener#propagate_ancestor_ids
[STOP] [2019-10-13 11:58:08] Flattener#propagate_ancestor_ids
[STOP] [2019-10-13 11:58:08] Flattener#flatten
[STOP] [2019-10-13 11:58:08] rebuild_nodes
[START] [2019-10-13 11:58:08] resolve_missing_media_owners
[STOP] [2019-10-13 11:58:08] resolve_missing_media_owners
[START] [2019-10-13 11:58:08] sanitize_media_verbatims
[STOP] [2019-10-13 11:58:08] sanitize_media_verbatims
[START] [2019-10-13 11:58:08] queue_downloads
[STOP] [2019-10-13 11:58:08] queue_downloads
[START] [2019-10-13 11:58:08] parse_names
[WARN] [2019-10-13 11:58:08] I see 27366 names which still need to be parsed.
[STOP] [2019-10-13 11:58:30] parse_names
[START] [2019-10-13 11:58:30] denormalize_canonical_names_to_nodes
[STOP] [2019-10-13 11:58:30] denormalize_canonical_names_to_nodes
[START] [2019-10-13 11:58:30] match_nodes
[START] [2019-10-13 11:58:30] map_all_nodes_to_pages
[STOP] [2019-10-13 12:24:16] map_all_nodes_to_pages
[INFO] [2019-10-13 12:24:16] 2474 Unmatched nodes (of 27366)! That's too many to output. First 10: Anas clypeata (#50099981); Anas strepera (#50099982); Anas penelope (#50100151); Anas querquedula (#50100263); Anas falcata (#50102994); Anas formosa (#50105514); Sturnia malabaricus (#50103035); Sturnus pagodarum (#50101139); Sturnus malabaricus (#50101651); Saroglossa spiloptera (#50101416)
[START] [2019-10-13 12:24:16] update_nodes
[STOP] [2019-10-13 12:24:26] update_nodes
[STOP] [2019-10-13 12:24:26] match_nodes
[START] [2019-10-13 12:24:26] reindex_search
[STOP] [2019-10-13 12:25:29] reindex_search
[START] [2019-10-13 12:25:29] normalize_units
[STOP] [2019-10-13 12:25:29] normalize_units
[START] [2019-10-13 12:25:29] calculate_statistics
[STOP] [2019-10-13 12:25:29] calculate_statistics
[START] [2019-10-13 12:25:29] complete_harvest_instance
[START] [2019-10-13 12:25:29] overall_tsv_creation
[INFO] [2019-10-13 12:25:29] Processing group of 27366 in 3 batches of 10000
[INFO] [2019-10-13 12:26:58] 6036 Traits (unfiltered)...
[INFO] [2019-10-13 12:27:12] 6036 Traits (filtered)...
[INFO] [2019-10-13 12:27:12] 0 Associations (filtered)...
[INFO] [2019-10-13 12:28:06] 30174 metadata added.
[INFO] [2019-10-13 12:28:06] 0 metadata added.
[INFO] [2019-10-13 12:29:37] 6629 Traits (unfiltered)...
[INFO] [2019-10-13 12:29:51] 6629 Traits (filtered)...
[INFO] [2019-10-13 12:29:51] 0 Associations (filtered)...
[INFO] [2019-10-13 12:30:42] 33128 metadata added.
[INFO] [2019-10-13 12:30:42] 0 metadata added.
[INFO] [2019-10-13 12:32:02] 5169 Traits (unfiltered)...
[INFO] [2019-10-13 12:32:16] 5169 Traits (filtered)...
[INFO] [2019-10-13 12:32:16] 0 Associations (filtered)...
[INFO] [2019-10-13 12:33:04] 25830 metadata added.
[INFO] [2019-10-13 12:33:04] 0 metadata added.
[INFO] [2019-10-13 12:33:04] Average Time: 125.733
[INFO] [2019-10-13 12:33:04] Total Time: 7m35s
[STOP] [2019-10-13 12:33:04] overall_tsv_creation
[INFO] [2019-10-13 12:33:04] Done. Check your files:
[INFO] [2019-10-13 12:33:04] (27366 lines) /app/public/data/india_sp_list/publish_nodes.tsv
[INFO] [2019-10-13 12:33:04] (68749 lines) /app/public/data/india_sp_list/publish_node_ancestors.tsv
[INFO] [2019-10-13 12:33:04] (27366 lines) /app/public/data/india_sp_list/publish_scientific_names.tsv
[INFO] [2019-10-13 12:33:04] (17835 lines) /app/public/data/india_sp_list/publish_traits.tsv
[INFO] [2019-10-13 12:33:04] (89133 lines) /app/public/data/india_sp_list/publish_metadata.tsv
[STOP] [2019-10-13 12:33:05] complete_harvest_instance
[START] [2019-10-13 12:33:05] completed
[STOP] [2019-10-13 12:33:05] completed
[STOP] [2019-10-13 12:33:05] logged process, took 2437.91

Latest Process

13 Oct 11:52 EXITED currently processing 0