Harvest for wikipedia gl Galician Created 04 Aug 10:31

Harvesting Log

# Logfile created on 2020-01-21 16:12:24 -0500 by logger.rb/56815
[START] [2020-01-21 16:12:24] logged process
[START] [2020-01-21 16:12:24] create_harvest_instance
[STOP] [2020-01-21 16:12:24] create_harvest_instance
[START] [2020-01-21 16:12:24] fetch_files
[STOP] [2020-01-21 16:12:24] fetch_files
[START] [2020-01-21 16:12:24] validate_each_file
[STOP] [2020-01-21 16:12:31] validate_each_file
[START] [2020-01-21 16:12:31] convert_to_csv
[CMD] [2020-01-21 16:12:31] /usr/bin/sort /app/public/converted_csv/wiki_gl_galician_nodes_20118.csv > /app/public/converted_csv/wiki_gl_galician_nodes_20118.csv_sorted
[CMD] [2020-01-21 16:12:32] /usr/bin/sort /app/public/converted_csv/wiki_gl_galician_media_20119.csv > /app/public/converted_csv/wiki_gl_galician_media_20119.csv_sorted
[STOP] [2020-01-21 16:12:32] convert_to_csv
[START] [2020-01-21 16:12:32] calculate_delta
[CMD] [2020-01-21 16:12:32] echo "0a" > /app/public/diff/wiki_gl_galician_nodes_20118.diff
[CMD] [2020-01-21 16:12:32] tail -n +1 /app/public/converted_csv/wiki_gl_galician_nodes_20118.csv >> /app/public/diff/wiki_gl_galician_nodes_20118.diff
[CMD] [2020-01-21 16:12:32] echo "." >> /app/public/diff/wiki_gl_galician_nodes_20118.diff
[CMD] [2020-01-21 16:12:32] echo "0a" > /app/public/diff/wiki_gl_galician_media_20119.diff
[CMD] [2020-01-21 16:12:33] tail -n +1 /app/public/converted_csv/wiki_gl_galician_media_20119.csv >> /app/public/diff/wiki_gl_galician_media_20119.diff
[CMD] [2020-01-21 16:12:33] echo "." >> /app/public/diff/wiki_gl_galician_media_20119.diff
[STOP] [2020-01-21 16:12:33] calculate_delta
[START] [2020-01-21 16:12:33] parse_diff_and_store
[INFO] [2020-01-21 16:12:33] Loading nodes diff file into memory (true lines)...
[INFO] [2020-01-21 16:12:38] Loading media diff file into memory (true lines)...
[INFO] [2020-01-21 16:14:45] Storing 8595 ScientificNames
[INFO] [2020-01-21 16:14:45] Processing group of 8595 in 9 groups of 1000
[INFO] [2020-01-21 16:14:49] Average Time: 0.471
[INFO] [2020-01-21 16:14:49] Total Time: 5s
[INFO] [2020-01-21 16:14:49] last 3 / first 3: 0.59
[INFO] [2020-01-21 16:14:49] Std.Dev: 0.1341640786499874; Max: 0.65
[INFO] [2020-01-21 16:14:49] Storing 8595 Identifiers
[INFO] [2020-01-21 16:14:49] Processing group of 8595 in 9 groups of 1000
[INFO] [2020-01-21 16:14:50] Average Time: 0.111
[INFO] [2020-01-21 16:14:50] Total Time: 2s
[INFO] [2020-01-21 16:14:50] last 3 / first 3: 0.94
[INFO] [2020-01-21 16:14:50] Std.Dev: 0.03162277660168379; Max: 0.14
[INFO] [2020-01-21 16:14:50] Storing 8595 Nodes
[INFO] [2020-01-21 16:14:50] Processing group of 8595 in 9 groups of 1000
[INFO] [2020-01-21 16:14:53] Average Time: 0.316
[INFO] [2020-01-21 16:14:53] Total Time: 3s
[INFO] [2020-01-21 16:14:53] last 3 / first 3: 0.83
[INFO] [2020-01-21 16:14:53] Std.Dev: 0.044721359549995794; Max: 0.37
[INFO] [2020-01-21 16:14:53] Storing 10738 ArticlesSections
[INFO] [2020-01-21 16:14:53] Processing group of 10738 in 11 groups of 1000
[INFO] [2020-01-21 16:14:54] Average Time: 0.065
[INFO] [2020-01-21 16:14:54] Total Time: 1s
[INFO] [2020-01-21 16:14:54] last 3 / first 3: 0.86
[INFO] [2020-01-21 16:14:54] Std.Dev: 0.0; Max: 0.08
[INFO] [2020-01-21 16:14:54] Storing 10738 Articles
[INFO] [2020-01-21 16:14:54] Processing group of 10738 in 11 groups of 1000
[INFO] [2020-01-21 16:15:02] Average Time: 0.712
[INFO] [2020-01-21 16:15:02] Total Time: 8s
[INFO] [2020-01-21 16:15:02] last 3 / first 3: 0.76
[INFO] [2020-01-21 16:15:02] Std.Dev: 0.14832396974191325; Max: 1.12
[STOP] [2020-01-21 16:15:02] parse_diff_and_store
[START] [2020-01-21 16:15:02] resolve_keys
[INFO] [2020-01-21 16:15:39] Occurrences to nodes (through scientific_names)...
[INFO] [2020-01-21 16:15:39] traits to occurrences...
[INFO] [2020-01-21 16:15:39] traits to nodes (through occurrences)...
[INFO] [2020-01-21 16:15:39] Traits to sex term...
[INFO] [2020-01-21 16:15:39] Traits to lifestage term...
[INFO] [2020-01-21 16:15:39] MetaTraits to traits...
[INFO] [2020-01-21 16:15:39] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-01-21 16:15:39] Assocs to occurrences...
[INFO] [2020-01-21 16:15:39] Assocs to nodes...
[INFO] [2020-01-21 16:15:39] Assoc to sex term...
[INFO] [2020-01-21 16:15:39] Assoc to lifestage term...
[STOP] [2020-01-21 16:15:39] resolve_keys
[START] [2020-01-21 16:15:39] hold_for_later_1
[STOP] [2020-01-21 16:15:39] hold_for_later_1
[START] [2020-01-21 16:15:39] hold_for_later_2
[STOP] [2020-01-21 16:15:39] hold_for_later_2
[START] [2020-01-21 16:15:39] resolve_missing_parents
[STOP] [2020-01-21 16:15:42] resolve_missing_parents
[START] [2020-01-21 16:15:42] rebuild_nodes
[START] [2020-01-21 16:15:42] Flattener#flatten
[START] [2020-01-21 16:15:42] Flattener#study_resource
[START] [2020-01-21 16:15:42] Flattener#build_ancestry
[STOP] [2020-01-21 16:15:43] Flattener#build_ancestry
[INFO] [2020-01-21 16:15:43] 8593 ancestry keys
[START] [2020-01-21 16:15:43] build_node_ancestors
[INFO] [2020-01-21 16:15:43] old ancestors deleted.
[STOP] [2020-01-21 16:16:04] build_node_ancestors
[START] [2020-01-21 16:16:09] Flattener#propagate_ancestor_ids
[STOP] [2020-01-21 16:16:12] Flattener#propagate_ancestor_ids
[STOP] [2020-01-21 16:16:12] Flattener#flatten
[STOP] [2020-01-21 16:16:12] rebuild_nodes
[START] [2020-01-21 16:16:12] resolve_missing_media_owners
[STOP] [2020-01-21 16:16:12] resolve_missing_media_owners
[START] [2020-01-21 16:16:12] sanitize_media_verbatims
[STOP] [2020-01-21 16:16:12] sanitize_media_verbatims
[START] [2020-01-21 16:16:12] queue_downloads
[STOP] [2020-01-21 16:16:12] queue_downloads
[START] [2020-01-21 16:16:12] parse_names
[WARN] [2020-01-21 16:16:12] I see 8595 names which still need to be parsed.
[WARN] [2020-01-21 16:16:20] I see 25 names which still need to be parsed.
[STOP] [2020-01-21 16:16:21] parse_names
[START] [2020-01-21 16:16:21] denormalize_canonical_names_to_nodes
[STOP] [2020-01-21 16:16:22] denormalize_canonical_names_to_nodes
[START] [2020-01-21 16:16:22] match_nodes
[START] [2020-01-21 16:16:22] map_all_nodes_to_pages
[STOP] [2020-01-21 16:52:25] map_all_nodes_to_pages
[INFO] [2020-01-21 16:52:25] 1217 Unmatched nodes (of 8595)! That's too many to output. First 10: Biota (#62944819); Acytota (#62943216); Prokaryota (#62943728); Aigarchaeota (#62946725); Bacteria (#62940856); Negibacteria (#62946303); Thermotogae (#62940987); Gemmatimonadetes (#62940990); Deferribacteres (#62941250); Borrelia burgdorferi (#62945542)
[START] [2020-01-21 16:52:25] update_nodes
[STOP] [2020-01-21 16:52:28] update_nodes
[STOP] [2020-01-21 16:52:28] match_nodes
[START] [2020-01-21 16:52:28] reindex_search
[STOP] [2020-01-21 16:53:14] reindex_search
[START] [2020-01-21 16:53:14] normalize_units
[STOP] [2020-01-21 16:53:14] normalize_units
[START] [2020-01-21 16:53:14] calculate_statistics
[ERR] [2020-01-21 16:53:15] Exception
[ERR] [2020-01-21 16:53:15] Unmapped page_ids for 2 nodes (IDs: 62944648, 62946428)! That is unacceptable.
[STOP] [2020-01-21 16:53:15] calculate_statistics
[STOP] [2020-01-21 16:53:15] logged process, took 2450.82
[INFO] [2020-07-15 09:45:59] ## HARVEST: type = resume_-harvest
[START] [2020-07-16 08:41:02] logged process
[INFO] [2020-07-16 09:26:43] ## HARVEST: type = resume_-harvest
[START] [2020-07-16 09:26:46] logged process
[INFO] [2020-08-04 10:31:20] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-08-04 10:31:24] ## remove_type: ScientificName
[INFO] [2020-08-04 10:31:24] ++ Calling delete_all on 8595 instances...
[INFO] [2020-08-04 10:31:25] [10:31:25.585] Removed 8595 Scientificnames
[INFO] [2020-08-04 10:31:25] ## remove_type: Vernacular
[INFO] [2020-08-04 10:31:25] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 10:31:25] [10:31:25.589] Removed 0 Vernaculars
[INFO] [2020-08-04 10:31:25] ## remove_type: Article
[INFO] [2020-08-04 10:31:25] ++ Calling delete_all on 10738 instances...
[INFO] [2020-08-04 10:31:32] [10:31:32.809] Removed 10738 Articles
[INFO] [2020-08-04 10:31:32] ## remove_type: Medium
[INFO] [2020-08-04 10:31:32] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 10:31:32] [10:31:32.813] Removed 0 Media
[INFO] [2020-08-04 10:31:32] ## remove_type: Trait
[INFO] [2020-08-04 10:31:32] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 10:31:32] [10:31:32.816] Removed 0 Traits
[INFO] [2020-08-04 10:31:32] ## remove_type: MetaTrait
[INFO] [2020-08-04 10:31:32] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 10:31:32] [10:31:32.881] Removed 0 Metatraits
[INFO] [2020-08-04 10:31:32] ## remove_type: OccurrenceMetadatum
[INFO] [2020-08-04 10:31:32] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 10:31:32] [10:31:32.998] Removed 0 Occurrencemetadata
[INFO] [2020-08-04 10:31:32] ## remove_type: Assoc
[INFO] [2020-08-04 10:31:33] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 10:31:33] [10:31:33.001] Removed 0 Assocs
[INFO] [2020-08-04 10:31:33] ## remove_type: MetaAssoc
[INFO] [2020-08-04 10:31:33] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 10:31:33] [10:31:33.030] Removed 0 Metaassocs
[INFO] [2020-08-04 10:31:33] ## remove_type: Identifier
[INFO] [2020-08-04 10:31:33] ++ Calling delete_all on 8595 instances...
[INFO] [2020-08-04 10:31:33] [10:31:33.783] Removed 8595 Identifiers
[INFO] [2020-08-04 10:31:33] ## remove_type: Reference
[INFO] [2020-08-04 10:31:33] ++ Calling delete_all on 0 instances...
[INFO] [2020-08-04 10:31:33] [10:31:33.786] Removed 0 References
[INFO] [2020-08-04 10:31:34] Starting batch with ID 62940941...
[INFO] [2020-08-04 10:31:36] Starting batch with ID 62946899...
[INFO] [2020-08-04 10:31:37] Starting batch with ID 62949052...
[INFO] [2020-08-04 10:31:38] Starting batch with ID 62949052...
[INFO] [2020-08-04 10:31:38] ## remove_type: Node
[INFO] [2020-08-04 10:31:38] ++ Calling delete_all on 8595 instances...
[INFO] [2020-08-04 10:31:39] [10:31:39.794] Removed 8595 Nodes
[START] [2020-08-04 10:31:57] logged process
[START] [2020-08-04 10:31:57] Creating resource from OpenData
[START] [2020-08-04 10:31:59] logged process
[START] [2020-08-04 10:31:59] Parse meta.xml file and create formats with fields
[STOP] [2020-08-04 10:31:59] Parse meta.xml file and create formats with fields
[STOP] [2020-08-04 10:31:59] Creating resource from OpenData
[START] [2020-08-04 10:31:59] logged process
[START] [2020-08-04 10:31:59] create_harvest_instance
[STOP] [2020-08-04 10:32:00] create_harvest_instance
[START] [2020-08-04 10:32:00] fetch_files
[STOP] [2020-08-04 10:32:00] fetch_files
[START] [2020-08-04 10:32:00] validate_each_file
[STOP] [2020-08-04 10:32:03] validate_each_file
[START] [2020-08-04 10:32:03] convert_to_csv
[CMD] [2020-08-04 10:32:03] /usr/bin/sort /app/public/converted_csv/wiki_gl_galician_nodes_22438.csv > /app/public/converted_csv/wiki_gl_galician_nodes_22438.csv_sorted
[CMD] [2020-08-04 10:32:03] /usr/bin/sort /app/public/converted_csv/wiki_gl_galician_media_22439.csv > /app/public/converted_csv/wiki_gl_galician_media_22439.csv_sorted
[STOP] [2020-08-04 10:32:03] convert_to_csv
[START] [2020-08-04 10:32:03] calculate_delta
[CMD] [2020-08-04 10:32:03] echo "0a" > /app/public/diff/wiki_gl_galician_nodes_22438.diff
[CMD] [2020-08-04 10:32:03] tail -n +1 /app/public/converted_csv/wiki_gl_galician_nodes_22438.csv >> /app/public/diff/wiki_gl_galician_nodes_22438.diff
[CMD] [2020-08-04 10:32:03] echo "." >> /app/public/diff/wiki_gl_galician_nodes_22438.diff
[CMD] [2020-08-04 10:32:03] echo "0a" > /app/public/diff/wiki_gl_galician_media_22439.diff
[CMD] [2020-08-04 10:32:03] tail -n +1 /app/public/converted_csv/wiki_gl_galician_media_22439.csv >> /app/public/diff/wiki_gl_galician_media_22439.diff
[CMD] [2020-08-04 10:32:03] echo "." >> /app/public/diff/wiki_gl_galician_media_22439.diff
[STOP] [2020-08-04 10:32:03] calculate_delta
[START] [2020-08-04 10:32:03] parse_diff_and_store
[INFO] [2020-08-04 10:32:03] Loading nodes diff file into memory (true lines)...
[INFO] [2020-08-04 10:32:07] Loading media diff file into memory (true lines)...
[INFO] [2020-08-04 10:33:41] Storing 8447 ScientificNames
[INFO] [2020-08-04 10:33:41] Processing group of 8447 in 9 groups of 1000
[INFO] [2020-08-04 10:33:44] Average Time: 0.288
[INFO] [2020-08-04 10:33:44] Total Time: 3s
[INFO] [2020-08-04 10:33:44] last 3 / first 3: 0.72
[INFO] [2020-08-04 10:33:44] Std.Dev: 0.06324555320336758; Max: 0.37
[INFO] [2020-08-04 10:33:44] Storing 8447 Identifiers
[INFO] [2020-08-04 10:33:44] Processing group of 8447 in 9 groups of 1000
[INFO] [2020-08-04 10:33:45] Average Time: 0.089
[INFO] [2020-08-04 10:33:45] Total Time: 1s
[INFO] [2020-08-04 10:33:45] last 3 / first 3: 0.62
[INFO] [2020-08-04 10:33:45] Std.Dev: 0.03162277660168379; Max: 0.15
[INFO] [2020-08-04 10:33:45] Storing 8447 Nodes
[INFO] [2020-08-04 10:33:45] Processing group of 8447 in 9 groups of 1000
[INFO] [2020-08-04 10:33:47] Average Time: 0.258
[INFO] [2020-08-04 10:33:47] Total Time: 3s
[INFO] [2020-08-04 10:33:47] last 3 / first 3: 0.78
[INFO] [2020-08-04 10:33:47] Std.Dev: 0.044721359549995794; Max: 0.31
[INFO] [2020-08-04 10:33:47] Storing 10355 ArticlesSections
[INFO] [2020-08-04 10:33:47] Processing group of 10355 in 11 groups of 1000
[INFO] [2020-08-04 10:33:48] Average Time: 0.066
[INFO] [2020-08-04 10:33:48] Total Time: 1s
[INFO] [2020-08-04 10:33:48] last 3 / first 3: 0.95
[INFO] [2020-08-04 10:33:48] Std.Dev: 0.03162277660168379; Max: 0.1
[INFO] [2020-08-04 10:33:48] Storing 10355 Articles
[INFO] [2020-08-04 10:33:48] Processing group of 10355 in 11 groups of 1000
[INFO] [2020-08-04 10:33:57] Average Time: 0.843
[INFO] [2020-08-04 10:33:57] Total Time: 10s
[INFO] [2020-08-04 10:33:57] last 3 / first 3: 0.9
[INFO] [2020-08-04 10:33:57] Std.Dev: 0.232379000772445; Max: 1.22
[STOP] [2020-08-04 10:33:57] parse_diff_and_store
[START] [2020-08-04 10:33:57] resolve_keys
[INFO] [2020-08-04 10:37:28] Occurrences to nodes (through scientific_names)...
[INFO] [2020-08-04 10:37:28] traits to occurrences...
[INFO] [2020-08-04 10:37:28] traits to nodes (through occurrences)...
[INFO] [2020-08-04 10:37:28] Traits to sex term...
[INFO] [2020-08-04 10:37:28] Traits to lifestage term...
[INFO] [2020-08-04 10:37:28] MetaTraits to traits...
[INFO] [2020-08-04 10:37:28] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-08-04 10:37:28] Assocs to occurrences...
[INFO] [2020-08-04 10:37:28] Assocs to nodes...
[INFO] [2020-08-04 10:37:28] Assoc to sex term...
[INFO] [2020-08-04 10:37:28] Assoc to lifestage term...
[STOP] [2020-08-04 10:37:28] resolve_keys
[START] [2020-08-04 10:37:28] hold_for_later_1
[STOP] [2020-08-04 10:37:28] hold_for_later_1
[START] [2020-08-04 10:37:28] hold_for_later_2
[STOP] [2020-08-04 10:37:28] hold_for_later_2
[START] [2020-08-04 10:37:28] resolve_missing_parents
[STOP] [2020-08-04 10:37:31] resolve_missing_parents
[START] [2020-08-04 10:37:31] rebuild_nodes
[START] [2020-08-04 10:37:31] Flattener#flatten
[START] [2020-08-04 10:37:31] Flattener#study_resource
[START] [2020-08-04 10:37:31] Flattener#build_ancestry
[STOP] [2020-08-04 10:37:31] Flattener#build_ancestry
[INFO] [2020-08-04 10:37:31] 8447 ancestry keys
[START] [2020-08-04 10:37:31] build_node_ancestors
[INFO] [2020-08-04 10:37:31] old ancestors deleted.
[STOP] [2020-08-04 10:37:45] build_node_ancestors
[START] [2020-08-04 10:37:50] Flattener#propagate_ancestor_ids
[STOP] [2020-08-04 10:37:54] Flattener#propagate_ancestor_ids
[STOP] [2020-08-04 10:37:54] Flattener#flatten
[STOP] [2020-08-04 10:37:54] rebuild_nodes
[START] [2020-08-04 10:37:54] resolve_missing_media_owners
[STOP] [2020-08-04 10:37:54] resolve_missing_media_owners
[START] [2020-08-04 10:37:54] sanitize_media_verbatims
[STOP] [2020-08-04 10:37:54] sanitize_media_verbatims
[START] [2020-08-04 10:37:54] queue_downloads
[STOP] [2020-08-04 10:37:54] queue_downloads
[START] [2020-08-04 10:37:54] parse_names
[WARN] [2020-08-04 10:37:54] I see 8447 names which still need to be parsed.
[WARN] [2020-08-04 10:38:01] I see 25 names which still need to be parsed.
[STOP] [2020-08-04 10:38:02] parse_names
[START] [2020-08-04 10:38:02] denormalize_canonical_names_to_nodes
[STOP] [2020-08-04 10:38:02] denormalize_canonical_names_to_nodes
[START] [2020-08-04 10:38:02] match_nodes
[START] [2020-08-04 10:38:02] map_all_nodes_to_pages
[STOP] [2020-08-04 11:02:35] map_all_nodes_to_pages
[INFO] [2020-08-04 11:02:35] 1316 Unmatched nodes (of 8447)! That's too many to output. First 10: Biota (#80639087); Acytota (#80637491); Prokaryota (#80637997); Aigarchaeota (#80640966); Bacteria (#80635222); Negibacteria (#80640544); Thermotogae (#80635354); Gemmatimonadetes (#80635357); Deferribacteres (#80635609); Deferribacteres (#80639632)
[START] [2020-08-04 11:02:35] update_nodes
[STOP] [2020-08-04 11:02:38] update_nodes
[STOP] [2020-08-04 11:02:38] match_nodes
[START] [2020-08-04 11:02:38] reindex_search
[STOP] [2020-08-04 11:03:17] reindex_search
[START] [2020-08-04 11:03:17] normalize_units
[STOP] [2020-08-04 11:03:17] normalize_units
[START] [2020-08-04 11:03:17] calculate_statistics
[STOP] [2020-08-04 11:03:18] calculate_statistics
[START] [2020-08-04 11:03:18] complete_harvest_instance
[START] [2020-08-04 11:03:18] overall_tsv_creation
[INFO] [2020-08-04 11:03:18] Processing group of 8447 in 1 batches of 10000
[INFO] [2020-08-04 11:05:20] Average Time: 55.29
[INFO] [2020-08-04 11:05:20] Total Time: 2m2s
[STOP] [2020-08-04 11:05:20] overall_tsv_creation
[INFO] [2020-08-04 11:05:20] Done. Check your files:
[INFO] [2020-08-04 11:05:20] (8447 lines) /app/public/data/wiki_gl_galician/publish_nodes.tsv
[INFO] [2020-08-04 11:05:20] (8447 lines) /app/public/data/wiki_gl_galician/publish_identifiers.tsv
[INFO] [2020-08-04 11:05:20] (171133 lines) /app/public/data/wiki_gl_galician/publish_node_ancestors.tsv
[INFO] [2020-08-04 11:05:20] (8447 lines) /app/public/data/wiki_gl_galician/publish_scientific_names.tsv
[INFO] [2020-08-04 11:05:20] (252899 lines) /app/public/data/wiki_gl_galician/publish_articles.tsv
[INFO] [2020-08-04 11:05:20] (10355 lines) /app/public/data/wiki_gl_galician/publish_content_sections.tsv
[STOP] [2020-08-04 11:05:20] complete_harvest_instance
[START] [2020-08-04 11:05:20] completed
[STOP] [2020-08-04 11:05:20] completed
[STOP] [2020-08-04 11:05:20] logged process, took 2000.99

Latest Process

04 Aug 10:31 EXITED currently processing 0