Harvest for wikipedia FR Created 21 Jul 16:10

Harvesting Log

[INFO] [2022-07-21 16:10:35] Created harvest instance #4147
[STOP] [2022-07-21 16:10:35] create_harvest_instance
[START] [2022-07-21 16:10:35] fetch_files
[STOP] [2022-07-21 16:10:35] fetch_files
[START] [2022-07-21 16:10:35] validate_each_file
[INFO] [2022-07-21 16:10:35] Looping over 2 formats...
[INFO] [2022-07-21 16:10:35] ...nodes (/app/public/data/wiki_fr_tar_gz/taxon.tab)
[INFO] [2022-07-21 16:10:40] Valid: /app/public/data/wiki_fr_tar_gz/converted_csv/wiki_fr_tar_gz_nodes_29408.csv (144956 lines)
[INFO] [2022-07-21 16:10:40] ...media (/app/public/data/wiki_fr_tar_gz/media_resource.tab)
[INFO] [2022-07-21 16:11:21] Valid: /app/public/data/wiki_fr_tar_gz/converted_csv/wiki_fr_tar_gz_media_29407.csv (256606 lines)
[STOP] [2022-07-21 16:11:21] validate_each_file
[START] [2022-07-21 16:11:21] convert_to_csv
[INFO] [2022-07-21 16:11:21] Looping over 2 formats...
[INFO] [2022-07-21 16:11:21] ...nodes (/app/public/data/wiki_fr_tar_gz/taxon.tab)
[CMD] [2022-07-21 16:11:21] /usr/bin/sort /app/public/data/wiki_fr_tar_gz/converted_csv/wiki_fr_tar_gz_nodes_29408.csv > /app/public/data/wiki_fr_tar_gz/converted_csv/wiki_fr_tar_gz_nodes_29408.csv_sorted
[INFO] [2022-07-21 16:11:21] Converted: /app/public/data/wiki_fr_tar_gz/converted_csv/wiki_fr_tar_gz_nodes_29408.csv (144956 lines)
[INFO] [2022-07-21 16:11:21] ...media (/app/public/data/wiki_fr_tar_gz/media_resource.tab)
[CMD] [2022-07-21 16:11:21] /usr/bin/sort /app/public/data/wiki_fr_tar_gz/converted_csv/wiki_fr_tar_gz_media_29407.csv > /app/public/data/wiki_fr_tar_gz/converted_csv/wiki_fr_tar_gz_media_29407.csv_sorted
[INFO] [2022-07-21 16:11:43] Converted: /app/public/data/wiki_fr_tar_gz/converted_csv/wiki_fr_tar_gz_media_29407.csv (256606 lines)
[STOP] [2022-07-21 16:11:43] convert_to_csv
[START] [2022-07-21 16:11:43] calculate_delta
[INFO] [2022-07-21 16:11:43] Looping over 2 formats...
[INFO] [2022-07-21 16:11:43] ...nodes (/app/public/data/wiki_fr_tar_gz/taxon.tab)
[CMD] [2022-07-21 16:11:43] echo "0a" > /app/public/data/wiki_fr_tar_gz/diff/wiki_fr_tar_gz_nodes_29408.diff
[CMD] [2022-07-21 16:11:43] tail -n +1 /app/public/data/wiki_fr_tar_gz/converted_csv/wiki_fr_tar_gz_nodes_29408.csv >> /app/public/data/wiki_fr_tar_gz/diff/wiki_fr_tar_gz_nodes_29408.diff
[CMD] [2022-07-21 16:11:43] echo "." >> /app/public/data/wiki_fr_tar_gz/diff/wiki_fr_tar_gz_nodes_29408.diff
[INFO] [2022-07-21 16:11:43] Created diff: /app/public/data/wiki_fr_tar_gz/diff/wiki_fr_tar_gz_nodes_29408.diff (144958 lines)
[INFO] [2022-07-21 16:11:43] ...media (/app/public/data/wiki_fr_tar_gz/media_resource.tab)
[CMD] [2022-07-21 16:11:43] echo "0a" > /app/public/data/wiki_fr_tar_gz/diff/wiki_fr_tar_gz_media_29407.diff
[CMD] [2022-07-21 16:11:44] tail -n +1 /app/public/data/wiki_fr_tar_gz/converted_csv/wiki_fr_tar_gz_media_29407.csv >> /app/public/data/wiki_fr_tar_gz/diff/wiki_fr_tar_gz_media_29407.diff
[CMD] [2022-07-21 16:11:54] echo "." >> /app/public/data/wiki_fr_tar_gz/diff/wiki_fr_tar_gz_media_29407.diff
[INFO] [2022-07-21 16:11:57] Created diff: /app/public/data/wiki_fr_tar_gz/diff/wiki_fr_tar_gz_media_29407.diff (256608 lines)
[STOP] [2022-07-21 16:11:57] calculate_delta
[START] [2022-07-21 16:11:57] parse_diff_and_store
[INFO] [2022-07-21 16:11:57] Handling diff: /app/public/data/wiki_fr_tar_gz/diff/wiki_fr_tar_gz_nodes_29408.diff (144958 lines)
[INFO] [2022-07-21 16:11:57] Loading nodes diff file into memory (144958 lines)...
[INFO] [2022-07-21 16:12:01] Storing 9999 ScientificNames (29997/10000/144958)
[INFO] [2022-07-21 16:12:04] Storing 9999 Identifiers (29997/10000/144958)
[INFO] [2022-07-21 16:12:05] Storing 9999 Nodes (29997/10000/144958)
[WARN] [2022-07-21 16:12:10] Filtered Scientific Name `Cuon alpinus fumosus/javanicus` to `Cuon alpinus fumosusjavanicus`
[INFO] [2022-07-21 16:12:11] Storing 10000 ScientificNames (59998/20000/144958)
[INFO] [2022-07-21 16:12:14] Storing 10001 Identifiers (59998/20000/144958)
[INFO] [2022-07-21 16:12:15] Storing 10000 Nodes (59998/20000/144958)
[INFO] [2022-07-21 16:12:22] Storing 10000 ScientificNames (89998/30000/144958)
[INFO] [2022-07-21 16:12:25] Storing 10000 Identifiers (89998/30000/144958)
[INFO] [2022-07-21 16:12:26] Storing 10000 Nodes (89998/30000/144958)
[INFO] [2022-07-21 16:12:32] Storing 10000 ScientificNames (119998/40000/144958)
[INFO] [2022-07-21 16:12:35] Storing 10000 Identifiers (119998/40000/144958)
[INFO] [2022-07-21 16:12:36] Storing 10000 Nodes (119998/40000/144958)
[INFO] [2022-07-21 16:12:43] Storing 10000 ScientificNames (149998/50000/144958)
[INFO] [2022-07-21 16:12:46] Storing 10000 Identifiers (149998/50000/144958)
[INFO] [2022-07-21 16:12:47] Storing 10000 Nodes (149998/50000/144958)
[INFO] [2022-07-21 16:12:53] Storing 10000 ScientificNames (179998/60000/144958)
[INFO] [2022-07-21 16:12:56] Storing 10000 Identifiers (179998/60000/144958)
[INFO] [2022-07-21 16:12:57] Storing 10000 Nodes (179998/60000/144958)
[WARN] [2022-07-21 16:13:03] Filtered Scientific Name `Cyanobacteria/Melainabacteria group` to `CyanobacteriaMelainabacteria group`
[INFO] [2022-07-21 16:13:04] Storing 10000 ScientificNames (209999/70000/144958)
[INFO] [2022-07-21 16:13:07] Storing 10001 Identifiers (209999/70000/144958)
[INFO] [2022-07-21 16:13:08] Storing 10000 Nodes (209999/70000/144958)
[INFO] [2022-07-21 16:13:14] Storing 10000 ScientificNames (239999/80000/144958)
[INFO] [2022-07-21 16:13:18] Storing 10000 Identifiers (239999/80000/144958)
[INFO] [2022-07-21 16:13:19] Storing 10000 Nodes (239999/80000/144958)
[INFO] [2022-07-21 16:13:25] Storing 10000 ScientificNames (269999/90000/144958)
[INFO] [2022-07-21 16:13:29] Storing 10000 Identifiers (269999/90000/144958)
[INFO] [2022-07-21 16:13:30] Storing 10000 Nodes (269999/90000/144958)
[INFO] [2022-07-21 16:13:37] Storing 10000 ScientificNames (299999/100000/144958)
[INFO] [2022-07-21 16:13:40] Storing 10000 Identifiers (299999/100000/144958)
[INFO] [2022-07-21 16:13:41] Storing 10000 Nodes (299999/100000/144958)
[INFO] [2022-07-21 16:13:48] Storing 10000 ScientificNames (329999/110000/144958)
[INFO] [2022-07-21 16:13:51] Storing 10000 Identifiers (329999/110000/144958)
[INFO] [2022-07-21 16:13:52] Storing 10000 Nodes (329999/110000/144958)
[INFO] [2022-07-21 16:13:59] Storing 10000 ScientificNames (359999/120000/144958)
[INFO] [2022-07-21 16:14:02] Storing 10000 Identifiers (359999/120000/144958)
[INFO] [2022-07-21 16:14:03] Storing 10000 Nodes (359999/120000/144958)
[INFO] [2022-07-21 16:14:10] Storing 10000 ScientificNames (389999/130000/144958)
[INFO] [2022-07-21 16:14:13] Storing 10000 Identifiers (389999/130000/144958)
[INFO] [2022-07-21 16:14:14] Storing 10000 Nodes (389999/130000/144958)
[INFO] [2022-07-21 16:14:21] Storing 10000 ScientificNames (420000/140000/144958)
[INFO] [2022-07-21 16:14:24] Storing 10001 Identifiers (420000/140000/144958)
[INFO] [2022-07-21 16:14:25] Storing 10000 Nodes (420000/140000/144958)
[WARN] [2022-07-21 16:14:28] Filtered Scientific Name `Coronavirus Neoromicia/PML-PHE1/RSA/2011` to `Coronavirus NeoromiciaPML-PHE1RSA2011`
[INFO] [2022-07-21 16:14:30] Storing 4957 ScientificNames (434872/144956/144958)
[INFO] [2022-07-21 16:14:32] Storing 4958 Identifiers (434872/144956/144958)
[INFO] [2022-07-21 16:14:32] Storing 4957 Nodes (434872/144956/144958)
[INFO] [2022-07-21 16:14:36] Handling diff: /app/public/data/wiki_fr_tar_gz/diff/wiki_fr_tar_gz_media_29407.diff (256608 lines)
[INFO] [2022-07-21 16:14:36] Loading media diff file into memory (256608 lines)...
[INFO] [2022-07-21 16:15:18] Storing 9999 ArticlesSections (19998/10000/256608)
[INFO] [2022-07-21 16:15:18] Storing 9999 Articles (19998/10000/256608)
[INFO] [2022-07-21 16:16:06] Storing 10000 ArticlesSections (39998/20000/256608)
[INFO] [2022-07-21 16:16:07] Storing 10000 Articles (39998/20000/256608)
[INFO] [2022-07-21 16:16:56] Storing 10000 ArticlesSections (59998/30000/256608)
[INFO] [2022-07-21 16:16:57] Storing 10000 Articles (59998/30000/256608)
[INFO] [2022-07-21 16:17:46] Storing 10000 ArticlesSections (79998/40000/256608)
[INFO] [2022-07-21 16:17:47] Storing 10000 Articles (79998/40000/256608)
[INFO] [2022-07-21 16:18:33] Storing 10000 ArticlesSections (99998/50000/256608)
[INFO] [2022-07-21 16:18:33] Storing 10000 Articles (99998/50000/256608)
[INFO] [2022-07-21 16:19:20] Storing 10000 ArticlesSections (119998/60000/256608)
[INFO] [2022-07-21 16:19:21] Storing 10000 Articles (119998/60000/256608)
[INFO] [2022-07-21 16:20:09] Storing 10000 ArticlesSections (139998/70000/256608)
[INFO] [2022-07-21 16:20:09] Storing 10000 Articles (139998/70000/256608)
[INFO] [2022-07-21 16:20:58] Storing 10000 ArticlesSections (159998/80000/256608)
[INFO] [2022-07-21 16:20:59] Storing 10000 Articles (159998/80000/256608)
[INFO] [2022-07-21 16:21:47] Storing 10000 ArticlesSections (179998/90000/256608)
[INFO] [2022-07-21 16:21:48] Storing 10000 Articles (179998/90000/256608)
[INFO] [2022-07-21 16:22:34] Storing 10000 ArticlesSections (199998/100000/256608)
[INFO] [2022-07-21 16:22:35] Storing 10000 Articles (199998/100000/256608)
[INFO] [2022-07-21 16:23:24] Storing 10000 ArticlesSections (219998/110000/256608)
[INFO] [2022-07-21 16:23:24] Storing 10000 Articles (219998/110000/256608)
[INFO] [2022-07-21 16:24:12] Storing 10000 ArticlesSections (239998/120000/256608)
[INFO] [2022-07-21 16:24:13] Storing 10000 Articles (239998/120000/256608)
[INFO] [2022-07-21 16:25:02] Storing 10000 ArticlesSections (259998/130000/256608)
[INFO] [2022-07-21 16:25:03] Storing 10000 Articles (259998/130000/256608)
[INFO] [2022-07-21 16:25:53] Storing 10000 ArticlesSections (279998/140000/256608)
[INFO] [2022-07-21 16:25:54] Storing 10000 Articles (279998/140000/256608)
[INFO] [2022-07-21 16:26:43] Storing 10000 ArticlesSections (299998/150000/256608)
[INFO] [2022-07-21 16:26:44] Storing 10000 Articles (299998/150000/256608)
[INFO] [2022-07-21 16:27:32] Storing 10000 ArticlesSections (319998/160000/256608)
[INFO] [2022-07-21 16:27:33] Storing 10000 Articles (319998/160000/256608)
[INFO] [2022-07-21 16:28:20] Storing 10000 ArticlesSections (339998/170000/256608)
[INFO] [2022-07-21 16:28:20] Storing 10000 Articles (339998/170000/256608)
[INFO] [2022-07-21 16:29:10] Storing 10000 ArticlesSections (359998/180000/256608)
[INFO] [2022-07-21 16:29:10] Storing 10000 Articles (359998/180000/256608)
[INFO] [2022-07-21 16:29:59] Storing 10000 ArticlesSections (379998/190000/256608)
[INFO] [2022-07-21 16:30:00] Storing 10000 Articles (379998/190000/256608)
[INFO] [2022-07-21 16:30:48] Storing 10000 ArticlesSections (399998/200000/256608)
[INFO] [2022-07-21 16:30:49] Storing 10000 Articles (399998/200000/256608)
[INFO] [2022-07-21 16:31:38] Storing 10000 ArticlesSections (419998/210000/256608)
[INFO] [2022-07-21 16:31:39] Storing 10000 Articles (419998/210000/256608)
[INFO] [2022-07-21 16:32:28] Storing 10000 ArticlesSections (439998/220000/256608)
[INFO] [2022-07-21 16:32:29] Storing 10000 Articles (439998/220000/256608)
[INFO] [2022-07-21 16:33:17] Storing 10000 ArticlesSections (459998/230000/256608)
[INFO] [2022-07-21 16:33:17] Storing 10000 Articles (459998/230000/256608)
[INFO] [2022-07-21 16:34:07] Storing 10000 ArticlesSections (479998/240000/256608)
[INFO] [2022-07-21 16:34:07] Storing 10000 Articles (479998/240000/256608)
[INFO] [2022-07-21 16:34:55] Storing 10000 ArticlesSections (499998/250000/256608)
[INFO] [2022-07-21 16:34:56] Storing 10000 Articles (499998/250000/256608)
[INFO] [2022-07-21 16:35:30] Storing 6607 ArticlesSections (513212/256606/256608)
[INFO] [2022-07-21 16:35:30] Storing 6607 Articles (513212/256606/256608)
[STOP] [2022-07-21 16:35:34] parse_diff_and_store
[START] [2022-07-21 16:35:34] resolve_keys
[2022-07-21 16:36:21] Resolving downloaded urls (this is not actually downloading them yet)
[INFO] [2022-07-21 16:56:30] Occurrences to nodes (through scientific_names)...
[INFO] [2022-07-21 16:56:30] traits to occurrences...
[INFO] [2022-07-21 16:56:30] traits to nodes (through occurrences)...
[INFO] [2022-07-21 16:56:30] Traits to sex term...
[INFO] [2022-07-21 16:56:30] Traits to lifestage term...
[INFO] [2022-07-21 16:56:30] MetaTraits to traits...
[INFO] [2022-07-21 16:56:30] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2022-07-21 16:56:30] Assocs to occurrences...
[INFO] [2022-07-21 16:56:30] Assocs to nodes...
[INFO] [2022-07-21 16:56:30] Assoc to sex term...
[INFO] [2022-07-21 16:56:30] Assoc to lifestage term...
[INFO] [2022-07-21 16:56:30] MetaAssoc to assocs...
[STOP] [2022-07-21 16:56:30] resolve_keys
[START] [2022-07-21 16:56:30] hold_for_later_1
[STOP] [2022-07-21 16:56:30] hold_for_later_1
[START] [2022-07-21 16:56:30] hold_for_later_2
[STOP] [2022-07-21 16:56:30] hold_for_later_2
[START] [2022-07-21 16:56:31] resolve_missing_parents
[STOP] [2022-07-21 16:56:38] resolve_missing_parents
[START] [2022-07-21 16:56:38] rebuild_nodes
[START] [2022-07-21 16:56:38] Flattener#flatten
[START] [2022-07-21 16:56:38] Flattener#study_resource
[START] [2022-07-21 16:57:58] Flattener#build_ancestry
[STOP] [2022-07-21 16:58:52] Flattener#build_ancestry
[INFO] [2022-07-21 16:58:52] 144956 ancestry keys
[START] [2022-07-21 16:58:52] build_node_ancestors
[INFO] [2022-07-21 16:58:52] old ancestors deleted.
[STOP] [2022-07-21 17:04:23] build_node_ancestors
[START] [2022-07-21 17:04:28] Flattener#propagate_ancestor_ids
[STOP] [2022-07-21 17:05:56] Flattener#propagate_ancestor_ids
[STOP] [2022-07-21 17:05:56] Flattener#flatten
[STOP] [2022-07-21 17:05:56] rebuild_nodes
[START] [2022-07-21 17:05:56] resolve_missing_media_owners
[STOP] [2022-07-21 17:05:56] resolve_missing_media_owners
[START] [2022-07-21 17:05:56] sanitize_media_verbatims
[STOP] [2022-07-21 17:05:56] sanitize_media_verbatims
[START] [2022-07-21 17:05:56] queue_downloads
[STOP] [2022-07-21 17:05:56] queue_downloads
[START] [2022-07-21 17:05:56] parse_names
[WARN] [2022-07-21 17:05:57] I see 144956 names which still need to be parsed.
[WARN] [2022-07-21 17:05:58] Names to parse: 10000 formatted: 10000 learned: 9999 parsed: 10000
[WARN] [2022-07-21 17:06:04] Names to parse: 10000 formatted: 10000 learned: 9997 parsed: 10000
[WARN] [2022-07-21 17:06:11] Names to parse: 10000 formatted: 10000 learned: 9998 parsed: 10000
[WARN] [2022-07-21 17:06:18] Names to parse: 10000 formatted: 10000 learned: 9999 parsed: 10000
[WARN] [2022-07-21 17:06:26] Names to parse: 10000 formatted: 10000 learned: 9998 parsed: 10000
[WARN] [2022-07-21 17:06:34] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2022-07-21 17:06:42] Names to parse: 10000 formatted: 10000 learned: 9997 parsed: 10000
[WARN] [2022-07-21 17:06:49] Names to parse: 10000 formatted: 10000 learned: 10000 parsed: 10000
[WARN] [2022-07-21 17:06:57] Names to parse: 10000 formatted: 10000 learned: 9998 parsed: 10000
[WARN] [2022-07-21 17:07:04] Names to parse: 10000 formatted: 10000 learned: 9999 parsed: 10000
[WARN] [2022-07-21 17:07:11] Names to parse: 10000 formatted: 10000 learned: 9999 parsed: 10000
[WARN] [2022-07-21 17:07:18] Names to parse: 10000 formatted: 10000 learned: 9998 parsed: 10000
[WARN] [2022-07-21 17:07:25] Names to parse: 10000 formatted: 10000 learned: 9998 parsed: 10000
[WARN] [2022-07-21 17:07:32] Names to parse: 10000 formatted: 10000 learned: 9999 parsed: 10000
[WARN] [2022-07-21 17:07:39] Names to parse: 4956 formatted: 4956 learned: 4953 parsed: 4956
[STOP] [2022-07-21 17:07:43] parse_names
[START] [2022-07-21 17:07:43] denormalize_canonical_names_to_nodes
[STOP] [2022-07-21 17:07:45] denormalize_canonical_names_to_nodes
[START] [2022-07-21 17:07:45] match_nodes
[START] [2022-07-21 17:07:46] map_all_nodes_to_pages
[STOP] [2022-07-21 17:44:17] map_all_nodes_to_pages
[INFO] [2022-07-21 17:44:17] 9626 Unmatched nodes (of 144956)! That's too many to output. Full list in /app/public/data/wiki_fr_tar_gz/unmatched_nodes.txt ; First 10: Canonical: Parainoceramya dubia; Node#118004005; ResourceID: Q106227010; Canonical: Allobuthus pescei; Node#118004173; ResourceID: Q106323841; Canonical: Anthracochaerilus palustris; Node#118004201; ResourceID: Q106345824; Canonical: Phoxiscorpio peachi; Node#118004202; ResourceID: Q106345867; Canonical: Trachyscorpio squarrosus; Node#118004221; ResourceID: Q106364057; Canonical: Gallioscorpio voltzi; Node#118004222; ResourceID: Q106364460; Canonical: Petaloscorpio bureaui; Node#118004223; ResourceID: Q106365102; Canonical: Feistmantelia ornata; Node#118004235; ResourceID: Q106370870; Canonical: Loboarchaeoctonus squamosus; Node#118004239; ResourceID: Q106372039; Canonical: Microlabis sternbergii; Node#118004242; ResourceID: Q106373582
[START] [2022-07-21 17:44:17] update_nodes
[STOP] [2022-07-21 17:44:39] update_nodes
[STOP] [2022-07-21 17:44:39] match_nodes
[START] [2022-07-21 17:44:39] reindex_search
[STOP] [2022-07-21 17:49:43] reindex_search
[START] [2022-07-21 17:49:43] normalize_units
[STOP] [2022-07-21 17:49:43] normalize_units
[START] [2022-07-21 17:49:43] calculate_statistics
[INFO] [2022-07-21 17:49:53] Duplicate page_id count: 0
[STOP] [2022-07-21 17:49:53] calculate_statistics
[START] [2022-07-21 17:49:53] complete_harvest_instance
[START] [2022-07-21 17:49:53] overall_tsv_creation
[INFO] [2022-07-21 17:49:53] Processing group of 144956 in 15 batches of 10000
[INFO] [2022-07-21 18:16:55] Average Time: 52.191
[INFO] [2022-07-21 18:16:55] Total Time: 27m2s
[INFO] [2022-07-21 18:16:55] last 3 / first 3: 0.87
[INFO] [2022-07-21 18:16:55] Std.Dev: 5.886; Max: 63.44
[STOP] [2022-07-21 18:16:55] overall_tsv_creation
[INFO] [2022-07-21 18:16:55] Done. Check your files:
[INFO] [2022-07-21 18:16:55] (144956 lines) /app/public/data/wiki_fr_tar_gz/publish_nodes.tsv
[INFO] [2022-07-21 18:16:55] (144960 lines) /app/public/data/wiki_fr_tar_gz/publish_identifiers.tsv
[INFO] [2022-07-21 18:16:55] (3273237 lines) /app/public/data/wiki_fr_tar_gz/publish_node_ancestors.tsv
[INFO] [2022-07-21 18:16:56] (144956 lines) /app/public/data/wiki_fr_tar_gz/publish_scientific_names.tsv
[INFO] [2022-07-21 18:16:56] (3219416 lines) /app/public/data/wiki_fr_tar_gz/publish_articles.tsv
[INFO] [2022-07-21 18:16:56] (256606 lines) /app/public/data/wiki_fr_tar_gz/publish_content_sections.tsv
[STOP] [2022-07-21 18:16:56] complete_harvest_instance
[START] [2022-07-21 18:16:56] completed
[STOP] [2022-07-21 18:16:56] completed
[STOP] [2022-07-21 18:16:56] logged process, took 7581.16

Latest Process

21 Jul 16:10 EXITED currently processing 0