Harvest for wikipedia CY Created 19 Nov 16:43

Harvesting Log

# Logfile created on 2019-11-19 16:43:25 -0500 by logger.rb/56815
[START] [2019-11-19 16:43:25] logged process
[START] [2019-11-19 16:43:25] create_harvest_instance
[STOP] [2019-11-19 16:43:26] create_harvest_instance
[START] [2019-11-19 16:43:26] fetch_files
[STOP] [2019-11-19 16:43:26] fetch_files
[START] [2019-11-19 16:43:26] validate_each_file
[STOP] [2019-11-19 16:43:33] validate_each_file
[START] [2019-11-19 16:43:33] convert_to_csv
[CMD] [2019-11-19 16:43:33] /usr/bin/sort /app/public/converted_csv/wiki_cy_welsh_nodes_18714.csv > /app/public/converted_csv/wiki_cy_welsh_nodes_18714.csv_sorted
[CMD] [2019-11-19 16:43:33] /usr/bin/sort /app/public/converted_csv/wiki_cy_welsh_media_18715.csv > /app/public/converted_csv/wiki_cy_welsh_media_18715.csv_sorted
[STOP] [2019-11-19 16:43:33] convert_to_csv
[START] [2019-11-19 16:43:33] calculate_delta
[CMD] [2019-11-19 16:43:33] echo "0a" > /app/public/diff/wiki_cy_welsh_nodes_18714.diff
[CMD] [2019-11-19 16:43:34] tail -n +1 /app/public/converted_csv/wiki_cy_welsh_nodes_18714.csv >> /app/public/diff/wiki_cy_welsh_nodes_18714.diff
[CMD] [2019-11-19 16:43:34] echo "." >> /app/public/diff/wiki_cy_welsh_nodes_18714.diff
[CMD] [2019-11-19 16:43:34] echo "0a" > /app/public/diff/wiki_cy_welsh_media_18715.diff
[CMD] [2019-11-19 16:43:34] tail -n +1 /app/public/converted_csv/wiki_cy_welsh_media_18715.csv >> /app/public/diff/wiki_cy_welsh_media_18715.diff
[CMD] [2019-11-19 16:43:35] echo "." >> /app/public/diff/wiki_cy_welsh_media_18715.diff
[STOP] [2019-11-19 16:43:35] calculate_delta
[START] [2019-11-19 16:43:35] parse_diff_and_store
[INFO] [2019-11-19 16:43:35] Loading nodes diff file into memory (true lines)...
[INFO] [2019-11-19 16:43:48] Loading media diff file into memory (true lines)...
[INFO] [2019-11-19 16:46:25] Storing 23238 ScientificNames
[INFO] [2019-11-19 16:46:25] Processing group of 23238 in 24 groups of 1000
[INFO] [2019-11-19 16:46:33] Average Time: 0.363
[INFO] [2019-11-19 16:46:33] Total Time: 9s
[INFO] [2019-11-19 16:46:33] last 3 / first 3: 0.91
[INFO] [2019-11-19 16:46:33] Std.Dev: 0.05477225575051661; Max: 0.51
[INFO] [2019-11-19 16:46:33] Storing 23239 Identifiers
[INFO] [2019-11-19 16:46:33] Processing group of 23239 in 24 groups of 1000
[INFO] [2019-11-19 16:46:37] Average Time: 0.147
[INFO] [2019-11-19 16:46:37] Total Time: 4s
[INFO] [2019-11-19 16:46:37] last 3 / first 3: 0.51
[INFO] [2019-11-19 16:46:37] Std.Dev: 0.05477225575051661; Max: 0.33
[INFO] [2019-11-19 16:46:37] Storing 23238 Nodes
[INFO] [2019-11-19 16:46:37] Processing group of 23238 in 24 groups of 1000
[INFO] [2019-11-19 16:46:44] Average Time: 0.295
[INFO] [2019-11-19 16:46:44] Total Time: 8s
[INFO] [2019-11-19 16:46:44] last 3 / first 3: 0.75
[INFO] [2019-11-19 16:46:44] Std.Dev: 0.05477225575051661; Max: 0.37
[INFO] [2019-11-19 16:46:44] Storing 34585 ArticlesSections
[INFO] [2019-11-19 16:46:44] Processing group of 34585 in 35 groups of 1000
[INFO] [2019-11-19 16:46:46] Average Time: 0.057
[INFO] [2019-11-19 16:46:46] Total Time: 3s
[INFO] [2019-11-19 16:46:46] last 3 / first 3: 0.83
[INFO] [2019-11-19 16:46:46] Std.Dev: 0.0; Max: 0.1
[INFO] [2019-11-19 16:46:46] Storing 34585 Articles
[INFO] [2019-11-19 16:46:46] Processing group of 34585 in 35 groups of 1000
[INFO] [2019-11-19 16:47:04] Average Time: 0.496
[INFO] [2019-11-19 16:47:04] Total Time: 18s
[INFO] [2019-11-19 16:47:04] last 3 / first 3: 0.73
[INFO] [2019-11-19 16:47:04] Std.Dev: 0.31622776601683794; Max: 2.27
[STOP] [2019-11-19 16:47:04] parse_diff_and_store
[START] [2019-11-19 16:47:04] resolve_keys
[INFO] [2019-11-19 16:48:01] Occurrences to nodes (through scientific_names)...
[INFO] [2019-11-19 16:48:01] traits to occurrences...
[INFO] [2019-11-19 16:48:01] traits to nodes (through occurrences)...
[INFO] [2019-11-19 16:48:01] Traits to sex term...
[INFO] [2019-11-19 16:48:01] Traits to lifestage term...
[INFO] [2019-11-19 16:48:01] MetaTraits to traits...
[INFO] [2019-11-19 16:48:01] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-11-19 16:48:01] Assocs to occurrences...
[INFO] [2019-11-19 16:48:01] Assocs to nodes...
[INFO] [2019-11-19 16:48:01] Assoc to sex term...
[INFO] [2019-11-19 16:48:01] Assoc to lifestage term...
[STOP] [2019-11-19 16:48:01] resolve_keys
[START] [2019-11-19 16:48:01] hold_for_later_1
[STOP] [2019-11-19 16:48:01] hold_for_later_1
[START] [2019-11-19 16:48:01] hold_for_later_2
[STOP] [2019-11-19 16:48:01] hold_for_later_2
[START] [2019-11-19 16:48:01] resolve_missing_parents
[STOP] [2019-11-19 16:48:07] resolve_missing_parents
[START] [2019-11-19 16:48:07] rebuild_nodes
[START] [2019-11-19 16:48:07] Flattener#flatten
[START] [2019-11-19 16:48:07] Flattener#study_resource
[START] [2019-11-19 16:48:07] Flattener#build_ancestry
[STOP] [2019-11-19 16:48:09] Flattener#build_ancestry
[INFO] [2019-11-19 16:48:09] 23238 ancestry keys
[START] [2019-11-19 16:48:09] build_node_ancestors
[INFO] [2019-11-19 16:48:09] old ancestors deleted.
[STOP] [2019-11-19 16:49:15] build_node_ancestors
[START] [2019-11-19 16:49:22] Flattener#propagate_ancestor_ids
[STOP] [2019-11-19 16:49:36] Flattener#propagate_ancestor_ids
[STOP] [2019-11-19 16:49:36] Flattener#flatten
[STOP] [2019-11-19 16:49:36] rebuild_nodes
[START] [2019-11-19 16:49:36] resolve_missing_media_owners
[STOP] [2019-11-19 16:49:36] resolve_missing_media_owners
[START] [2019-11-19 16:49:36] sanitize_media_verbatims
[STOP] [2019-11-19 16:49:36] sanitize_media_verbatims
[START] [2019-11-19 16:49:36] queue_downloads
[STOP] [2019-11-19 16:49:36] queue_downloads
[START] [2019-11-19 16:49:36] parse_names
[WARN] [2019-11-19 16:49:36] I see 23238 names which still need to be parsed.
[WARN] [2019-11-19 16:49:56] I see 5 names which still need to be parsed.
[STOP] [2019-11-19 16:49:57] parse_names
[START] [2019-11-19 16:49:57] denormalize_canonical_names_to_nodes
[STOP] [2019-11-19 16:49:57] denormalize_canonical_names_to_nodes
[START] [2019-11-19 16:49:57] match_nodes
[START] [2019-11-19 16:49:57] map_all_nodes_to_pages
[STOP] [2019-11-19 17:25:17] map_all_nodes_to_pages
[INFO] [2019-11-19 17:25:17] 1503 Unmatched nodes (of 23238)! That's too many to output. First 10: Biota (#55334095); Prokaryota (#55331073); Bacteria (#55323479); Negibacteria (#55338438); Posibacteria (#55338532); Actinobacteria (#55326353); Protista (#55323507); Embryophyta (#55331143); Haplomitriales (#55322447); Blasiidae (#55333563)
[START] [2019-11-19 17:25:17] update_nodes
[STOP] [2019-11-19 17:25:25] update_nodes
[STOP] [2019-11-19 17:25:25] match_nodes
[START] [2019-11-19 17:25:25] reindex_search
[STOP] [2019-11-19 17:26:53] reindex_search
[START] [2019-11-19 17:26:53] normalize_units
[STOP] [2019-11-19 17:26:53] normalize_units
[START] [2019-11-19 17:26:53] calculate_statistics
[STOP] [2019-11-19 17:26:54] calculate_statistics
[START] [2019-11-19 17:26:54] complete_harvest_instance
[START] [2019-11-19 17:26:54] overall_tsv_creation
[INFO] [2019-11-19 17:26:54] Processing group of 23238 in 3 batches of 10000
[INFO] [2019-11-19 17:32:20] Average Time: 70.013
[INFO] [2019-11-19 17:32:20] Total Time: 5m27s
[STOP] [2019-11-19 17:32:20] overall_tsv_creation
[INFO] [2019-11-19 17:32:20] Done. Check your files:
[INFO] [2019-11-19 17:32:20] (23238 lines) /app/public/data/wiki_cy_welsh/publish_nodes.tsv
[INFO] [2019-11-19 17:32:20] (23239 lines) /app/public/data/wiki_cy_welsh/publish_identifiers.tsv
[INFO] [2019-11-19 17:32:21] (498744 lines) /app/public/data/wiki_cy_welsh/publish_node_ancestors.tsv
[INFO] [2019-11-19 17:32:21] (23238 lines) /app/public/data/wiki_cy_welsh/publish_scientific_names.tsv
[INFO] [2019-11-19 17:32:21] (181249 lines) /app/public/data/wiki_cy_welsh/publish_articles.tsv
[INFO] [2019-11-19 17:32:21] (34585 lines) /app/public/data/wiki_cy_welsh/publish_content_sections.tsv
[STOP] [2019-11-19 17:32:22] complete_harvest_instance
[START] [2019-11-19 17:32:22] completed
[STOP] [2019-11-19 17:32:22] completed
[STOP] [2019-11-19 17:32:22] logged process, took 2936.16

Latest Process

19 Nov 16:43 EXITED currently processing 0