Harvest for wikipedia EU Created 06 Nov 19:17

Stage: completed
Fetched: 06 Nov 19:17
Validated: 06 Nov 19:17
Deltas Created 06 Nov 19:18
Units Normalized: 06 Nov 23:10
Ancestry Built: 06 Nov 19:39
Nodes Matched: 06 Nov 23:03
Names Parsed: 06 Nov 19:40
New Models Stored: 06 Nov 19:29
Indexed: 06 Nov 23:10
Completed: 06 Nov 23:33
Time to Harvest: 4 minutes
Harvesting Log

(130 lines)
# Logfile created on 2019-11-06 19:17:38 -0500 by logger.rb/56815
[START] [2019-11-06 19:17:38] logged process
[START] [2019-11-06 19:17:38] create_harvest_instance
[STOP] [2019-11-06 19:17:39] create_harvest_instance
[START] [2019-11-06 19:17:39] fetch_files
[STOP] [2019-11-06 19:17:39] fetch_files
[START] [2019-11-06 19:17:39] validate_each_file
[STOP] [2019-11-06 19:17:56] validate_each_file
[START] [2019-11-06 19:17:56] convert_to_csv
[CMD] [2019-11-06 19:17:56] /usr/bin/sort /app/public/converted_csv/wiki_eu_basque_nodes_18206.csv > /app/public/converted_csv/wiki_eu_basque_nodes_18206.csv_sorted
[CMD] [2019-11-06 19:17:57] /usr/bin/sort /app/public/converted_csv/wiki_eu_basque_media_18207.csv > /app/public/converted_csv/wiki_eu_basque_media_18207.csv_sorted
[STOP] [2019-11-06 19:17:58] convert_to_csv
[START] [2019-11-06 19:17:58] calculate_delta
[CMD] [2019-11-06 19:17:58] echo "0a" > /app/public/diff/wiki_eu_basque_nodes_18206.diff
[CMD] [2019-11-06 19:17:58] tail -n +1 /app/public/converted_csv/wiki_eu_basque_nodes_18206.csv >> /app/public/diff/wiki_eu_basque_nodes_18206.diff
[CMD] [2019-11-06 19:17:58] echo "." >> /app/public/diff/wiki_eu_basque_nodes_18206.diff
[CMD] [2019-11-06 19:17:59] echo "0a" > /app/public/diff/wiki_eu_basque_media_18207.diff
[CMD] [2019-11-06 19:17:59] tail -n +1 /app/public/converted_csv/wiki_eu_basque_media_18207.csv >> /app/public/diff/wiki_eu_basque_media_18207.diff
[CMD] [2019-11-06 19:18:00] echo "." >> /app/public/diff/wiki_eu_basque_media_18207.diff
[STOP] [2019-11-06 19:18:00] calculate_delta
[START] [2019-11-06 19:18:00] parse_diff_and_store
[INFO] [2019-11-06 19:18:01] Loading nodes diff file into memory (true lines)...
[INFO] [2019-11-06 19:18:43] Loading media diff file into memory (true lines)...
[INFO] [2019-11-06 19:26:39] Storing 82575 ScientificNames
[INFO] [2019-11-06 19:26:39] Processing group of 82575 in 83 groups of 1000
[INFO] [2019-11-06 19:27:19] Average Time: 0.469
[INFO] [2019-11-06 19:27:19] Total Time: 40s
[INFO] [2019-11-06 19:27:19] last 3 / first 3: 0.25
[INFO] [2019-11-06 19:27:19] Std.Dev: 0.5495452665613635; Max: 3.44
[INFO] [2019-11-06 19:27:19] Storing 82575 Identifiers
[INFO] [2019-11-06 19:27:19] Processing group of 82575 in 83 groups of 1000
[INFO] [2019-11-06 19:27:34] Average Time: 0.18
[INFO] [2019-11-06 19:27:34] Total Time: 16s
[INFO] [2019-11-06 19:27:34] last 3 / first 3: 0.47
[INFO] [2019-11-06 19:27:34] Std.Dev: 0.4538722287164087; Max: 3.41
[INFO] [2019-11-06 19:27:34] Storing 82575 Nodes
[INFO] [2019-11-06 19:27:34] Processing group of 82575 in 83 groups of 1000
[INFO] [2019-11-06 19:28:04] Average Time: 0.366
[INFO] [2019-11-06 19:28:04] Total Time: 31s
[INFO] [2019-11-06 19:28:04] last 3 / first 3: 0.95
[INFO] [2019-11-06 19:28:04] Std.Dev: 0.23664319132398465; Max: 2.31
[INFO] [2019-11-06 19:28:04] Storing 142743 ArticlesSections
[INFO] [2019-11-06 19:28:04] Processing group of 142743 in 143 groups of 1000
[INFO] [2019-11-06 19:28:21] Average Time: 0.109
[INFO] [2019-11-06 19:28:21] Total Time: 17s
[INFO] [2019-11-06 19:28:21] last 3 / first 3: 0.94
[INFO] [2019-11-06 19:28:21] Std.Dev: 0.44944410108488464; Max: 3.87
[INFO] [2019-11-06 19:28:21] Storing 142743 Articles
[INFO] [2019-11-06 19:28:21] Processing group of 142743 in 143 groups of 1000
[INFO] [2019-11-06 19:29:49] Average Time: 0.616
[INFO] [2019-11-06 19:29:49] Total Time: 1m29s
[INFO] [2019-11-06 19:29:49] last 3 / first 3: 0.62
[INFO] [2019-11-06 19:29:49] Std.Dev: 0.8160882305241266; Max: 4.6
[STOP] [2019-11-06 19:29:49] parse_diff_and_store
[START] [2019-11-06 19:29:49] resolve_keys
[INFO] [2019-11-06 19:32:50] Occurrences to nodes (through scientific_names)...
[INFO] [2019-11-06 19:32:50] traits to occurrences...
[INFO] [2019-11-06 19:32:50] traits to nodes (through occurrences)...
[INFO] [2019-11-06 19:32:50] Traits to sex term...
[INFO] [2019-11-06 19:32:50] Traits to lifestage term...
[INFO] [2019-11-06 19:32:50] MetaTraits to traits...
[INFO] [2019-11-06 19:32:50] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-11-06 19:32:50] Assocs to occurrences...
[INFO] [2019-11-06 19:32:50] Assocs to nodes...
[INFO] [2019-11-06 19:32:50] Assoc to sex term...
[INFO] [2019-11-06 19:32:50] Assoc to lifestage term...
[STOP] [2019-11-06 19:32:51] resolve_keys
[START] [2019-11-06 19:32:51] hold_for_later_1
[STOP] [2019-11-06 19:32:51] hold_for_later_1
[START] [2019-11-06 19:32:51] hold_for_later_2
[STOP] [2019-11-06 19:32:51] hold_for_later_2
[START] [2019-11-06 19:32:51] resolve_missing_parents
[STOP] [2019-11-06 19:33:06] resolve_missing_parents
[START] [2019-11-06 19:33:06] rebuild_nodes
[START] [2019-11-06 19:33:06] Flattener#flatten
[START] [2019-11-06 19:33:06] Flattener#study_resource
[START] [2019-11-06 19:33:06] Flattener#build_ancestry
[STOP] [2019-11-06 19:33:26] Flattener#build_ancestry
[INFO] [2019-11-06 19:33:26] 82575 ancestry keys
[START] [2019-11-06 19:33:26] build_node_ancestors
[INFO] [2019-11-06 19:33:26] old ancestors deleted.
[STOP] [2019-11-06 19:38:09] build_node_ancestors
[START] [2019-11-06 19:38:12] Flattener#propagate_ancestor_ids
[STOP] [2019-11-06 19:39:52] Flattener#propagate_ancestor_ids
[STOP] [2019-11-06 19:39:52] Flattener#flatten
[STOP] [2019-11-06 19:39:52] rebuild_nodes
[START] [2019-11-06 19:39:52] resolve_missing_media_owners
[STOP] [2019-11-06 19:39:52] resolve_missing_media_owners
[START] [2019-11-06 19:39:52] sanitize_media_verbatims
[STOP] [2019-11-06 19:39:52] sanitize_media_verbatims
[START] [2019-11-06 19:39:52] queue_downloads
[STOP] [2019-11-06 19:39:52] queue_downloads
[START] [2019-11-06 19:39:52] parse_names
[WARN] [2019-11-06 19:39:52] I see 82575 names which still need to be parsed.
[WARN] [2019-11-06 19:40:55] I see 12 names which still need to be parsed.
[STOP] [2019-11-06 19:40:56] parse_names
[START] [2019-11-06 19:40:56] denormalize_canonical_names_to_nodes
[STOP] [2019-11-06 19:40:57] denormalize_canonical_names_to_nodes
[START] [2019-11-06 19:40:57] match_nodes
[START] [2019-11-06 19:40:57] map_all_nodes_to_pages
[STOP] [2019-11-06 23:03:11] map_all_nodes_to_pages
[INFO] [2019-11-06 23:03:11] 2216 Unmatched nodes (of 82575)! That's too many to output. First 10: Biota (#54380364); Bacteria (#54351666); Negibacteria (#54400783); Escherichia coli (#54383272); Posibacteria (#54401409); Actinobacteria (#54356892); Actinobacteridae (#54349988); Micrococcus (#54369866); Lactobacillus casei (#54359060); Prokaryota (#54368255)
[START] [2019-11-06 23:03:11] update_nodes
[STOP] [2019-11-06 23:03:43] update_nodes
[STOP] [2019-11-06 23:03:43] match_nodes
[START] [2019-11-06 23:03:43] reindex_search
[STOP] [2019-11-06 23:10:48] reindex_search
[START] [2019-11-06 23:10:48] normalize_units
[STOP] [2019-11-06 23:10:48] normalize_units
[START] [2019-11-06 23:10:48] calculate_statistics
[STOP] [2019-11-06 23:10:49] calculate_statistics
[START] [2019-11-06 23:10:49] complete_harvest_instance
[START] [2019-11-06 23:10:49] overall_tsv_creation
[INFO] [2019-11-06 23:10:49] Processing group of 82575 in 9 batches of 10000
[INFO] [2019-11-06 23:33:00] Average Time: 93.102
[INFO] [2019-11-06 23:33:00] Total Time: 22m12s
[INFO] [2019-11-06 23:33:00] last 3 / first 3: 0.89
[INFO] [2019-11-06 23:33:00] Std.Dev: 24.431946299875495; Max: 110.31
[STOP] [2019-11-06 23:33:00] overall_tsv_creation
[INFO] [2019-11-06 23:33:00] Done. Check your files:
[INFO] [2019-11-06 23:33:01] (82575 lines) /app/public/data/wiki_eu_basque/publish_nodes.tsv
[INFO] [2019-11-06 23:33:01] (82575 lines) /app/public/data/wiki_eu_basque/publish_identifiers.tsv
[INFO] [2019-11-06 23:33:02] (1935301 lines) /app/public/data/wiki_eu_basque/publish_node_ancestors.tsv
[INFO] [2019-11-06 23:33:02] (82575 lines) /app/public/data/wiki_eu_basque/publish_scientific_names.tsv
[INFO] [2019-11-06 23:33:02] (758809 lines) /app/public/data/wiki_eu_basque/publish_articles.tsv
[INFO] [2019-11-06 23:33:03] (142743 lines) /app/public/data/wiki_eu_basque/publish_content_sections.tsv
[STOP] [2019-11-06 23:33:03] complete_harvest_instance
[START] [2019-11-06 23:33:03] completed
[STOP] [2019-11-06 23:33:03] completed
[STOP] [2019-11-06 23:33:03] logged process, took 15325.04
Latest Process

06 Nov 19:17 EXITED currently processing 0