Harvest for wikipedia DA Created 30 Jan 13:53

Stage: completed
Fetched: 30 Jan 13:53
Validated: 30 Jan 13:53
Deltas Created 30 Jan 13:53
Units Normalized: 30 Jan 14:28
Ancestry Built: 30 Jan 14:03
Nodes Matched: 30 Jan 14:28
Names Parsed: 30 Jan 14:03
New Models Stored: 30 Jan 13:54
Indexed: 30 Jan 14:28
Completed: 30 Jan 14:31
Time to Harvest: 1 minute

Harvesting Log

(130 lines)
# Logfile created on 2020-01-30 13:53:15 -0500 by logger.rb/56815
[START] [2020-01-30 13:53:15] logged process
[START] [2020-01-30 13:53:15] create_harvest_instance
[STOP] [2020-01-30 13:53:15] create_harvest_instance
[START] [2020-01-30 13:53:15] fetch_files
[STOP] [2020-01-30 13:53:15] fetch_files
[START] [2020-01-30 13:53:15] validate_each_file
[STOP] [2020-01-30 13:53:18] validate_each_file
[START] [2020-01-30 13:53:18] convert_to_csv
[CMD] [2020-01-30 13:53:18] /usr/bin/sort /app/public/converted_csv/wiki_da_danish_nodes_20156.csv > /app/public/converted_csv/wiki_da_danish_nodes_20156.csv_sorted
[CMD] [2020-01-30 13:53:18] /usr/bin/sort /app/public/converted_csv/wiki_da_danish_media_20157.csv > /app/public/converted_csv/wiki_da_danish_media_20157.csv_sorted
[STOP] [2020-01-30 13:53:19] convert_to_csv
[START] [2020-01-30 13:53:19] calculate_delta
[CMD] [2020-01-30 13:53:19] echo "0a" > /app/public/diff/wiki_da_danish_nodes_20156.diff
[CMD] [2020-01-30 13:53:20] tail -n +1 /app/public/converted_csv/wiki_da_danish_nodes_20156.csv >> /app/public/diff/wiki_da_danish_nodes_20156.diff
[CMD] [2020-01-30 13:53:21] echo "." >> /app/public/diff/wiki_da_danish_nodes_20156.diff
[CMD] [2020-01-30 13:53:21] echo "0a" > /app/public/diff/wiki_da_danish_media_20157.diff
[CMD] [2020-01-30 13:53:22] tail -n +1 /app/public/converted_csv/wiki_da_danish_media_20157.csv >> /app/public/diff/wiki_da_danish_media_20157.diff
[CMD] [2020-01-30 13:53:23] echo "." >> /app/public/diff/wiki_da_danish_media_20157.diff
[STOP] [2020-01-30 13:53:24] calculate_delta
[START] [2020-01-30 13:53:24] parse_diff_and_store
[INFO] [2020-01-30 13:53:24] Loading nodes diff file into memory (true lines)...
[WARN] [2020-01-30 13:53:27] Filtered Scientific Name `Ambystoma  hakié jas jas` to `Ambystoma hakié jas jas`
[WARN] [2020-01-30 13:53:28] Filtered Scientific Name `Tyto capensis           תנשמת עשב אפריקאית` to `Tyto capensis תנשמת עשב אפריקאית`
[INFO] [2020-01-30 13:53:30] Loading media diff file into memory (true lines)...
[INFO] [2020-01-30 13:54:17] Storing 9441 ScientificNames
[INFO] [2020-01-30 13:54:17] Processing group of 9441 in 10 groups of 1000
[INFO] [2020-01-30 13:54:20] Average Time: 0.376
[INFO] [2020-01-30 13:54:20] Total Time: 4s
[INFO] [2020-01-30 13:54:20] last 3 / first 3: 0.59
[INFO] [2020-01-30 13:54:20] Std.Dev: 0.10488088481701516; Max: 0.57
[INFO] [2020-01-30 13:54:20] Storing 9441 Identifiers
[INFO] [2020-01-30 13:54:20] Processing group of 9441 in 10 groups of 1000
[INFO] [2020-01-30 13:54:22] Average Time: 0.153
[INFO] [2020-01-30 13:54:22] Total Time: 2s
[INFO] [2020-01-30 13:54:22] last 3 / first 3: 0.54
[INFO] [2020-01-30 13:54:22] Std.Dev: 0.06324555320336758; Max: 0.31
[INFO] [2020-01-30 13:54:22] Storing 9441 Nodes
[INFO] [2020-01-30 13:54:22] Processing group of 9441 in 10 groups of 1000
[INFO] [2020-01-30 13:54:25] Average Time: 0.335
[INFO] [2020-01-30 13:54:25] Total Time: 4s
[INFO] [2020-01-30 13:54:25] last 3 / first 3: 0.52
[INFO] [2020-01-30 13:54:25] Std.Dev: 0.13784048752090222; Max: 0.68
[INFO] [2020-01-30 13:54:25] Storing 11293 ArticlesSections
[INFO] [2020-01-30 13:54:25] Processing group of 11293 in 12 groups of 1000
[INFO] [2020-01-30 13:54:26] Average Time: 0.06
[INFO] [2020-01-30 13:54:26] Total Time: 1s
[INFO] [2020-01-30 13:54:26] last 3 / first 3: 0.65
[INFO] [2020-01-30 13:54:26] Std.Dev: 0.0; Max: 0.1
[INFO] [2020-01-30 13:54:26] Storing 11293 Articles
[INFO] [2020-01-30 13:54:26] Processing group of 11293 in 12 groups of 1000
[INFO] [2020-01-30 13:54:32] Average Time: 0.518
[INFO] [2020-01-30 13:54:32] Total Time: 7s
[INFO] [2020-01-30 13:54:32] last 3 / first 3: 0.6
[INFO] [2020-01-30 13:54:32] Std.Dev: 0.1414213562373095; Max: 0.79
[STOP] [2020-01-30 13:54:32] parse_diff_and_store
[START] [2020-01-30 13:54:32] resolve_keys
[INFO] [2020-01-30 14:02:39] Occurrences to nodes (through scientific_names)...
[INFO] [2020-01-30 14:02:39] traits to occurrences...
[INFO] [2020-01-30 14:02:39] traits to nodes (through occurrences)...
[INFO] [2020-01-30 14:02:39] Traits to sex term...
[INFO] [2020-01-30 14:02:39] Traits to lifestage term...
[INFO] [2020-01-30 14:02:39] MetaTraits to traits...
[INFO] [2020-01-30 14:02:39] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-01-30 14:02:39] Assocs to occurrences...
[INFO] [2020-01-30 14:02:39] Assocs to nodes...
[INFO] [2020-01-30 14:02:39] Assoc to sex term...
[INFO] [2020-01-30 14:02:39] Assoc to lifestage term...
[STOP] [2020-01-30 14:02:39] resolve_keys
[START] [2020-01-30 14:02:39] hold_for_later_1
[STOP] [2020-01-30 14:02:39] hold_for_later_1
[START] [2020-01-30 14:02:39] hold_for_later_2
[STOP] [2020-01-30 14:02:39] hold_for_later_2
[START] [2020-01-30 14:02:39] resolve_missing_parents
[STOP] [2020-01-30 14:02:42] resolve_missing_parents
[START] [2020-01-30 14:02:42] rebuild_nodes
[START] [2020-01-30 14:02:42] Flattener#flatten
[START] [2020-01-30 14:02:42] Flattener#study_resource
[START] [2020-01-30 14:02:42] Flattener#build_ancestry
[STOP] [2020-01-30 14:02:43] Flattener#build_ancestry
[INFO] [2020-01-30 14:02:43] 9441 ancestry keys
[START] [2020-01-30 14:02:43] build_node_ancestors
[INFO] [2020-01-30 14:02:43] old ancestors deleted.
[STOP] [2020-01-30 14:03:03] build_node_ancestors
[START] [2020-01-30 14:03:08] Flattener#propagate_ancestor_ids
[STOP] [2020-01-30 14:03:12] Flattener#propagate_ancestor_ids
[STOP] [2020-01-30 14:03:12] Flattener#flatten
[STOP] [2020-01-30 14:03:12] rebuild_nodes
[START] [2020-01-30 14:03:12] resolve_missing_media_owners
[STOP] [2020-01-30 14:03:12] resolve_missing_media_owners
[START] [2020-01-30 14:03:12] sanitize_media_verbatims
[STOP] [2020-01-30 14:03:12] sanitize_media_verbatims
[START] [2020-01-30 14:03:12] queue_downloads
[STOP] [2020-01-30 14:03:12] queue_downloads
[START] [2020-01-30 14:03:12] parse_names
[WARN] [2020-01-30 14:03:12] I see 9441 names which still need to be parsed.
[WARN] [2020-01-30 14:03:21] I see 17 names which still need to be parsed.
[STOP] [2020-01-30 14:03:23] parse_names
[START] [2020-01-30 14:03:23] denormalize_canonical_names_to_nodes
[STOP] [2020-01-30 14:03:23] denormalize_canonical_names_to_nodes
[START] [2020-01-30 14:03:23] match_nodes
[START] [2020-01-30 14:03:23] map_all_nodes_to_pages
[STOP] [2020-01-30 14:28:16] map_all_nodes_to_pages
[INFO] [2020-01-30 14:28:16] 1125 Unmatched nodes (of 9441)! That's too many to output. First 10: Biota (#62998298); Acytota (#62996778); Prokaryota (#62997277); Bacteria (#62993544); Negibacteria (#62999720); Escherichia coli (#62998525); Posibacteria (#62999786); Actinobacteria (#62994160); Actinobacteria (#62998692); Actinobacteridae (#62993304)
[START] [2020-01-30 14:28:16] update_nodes
[STOP] [2020-01-30 14:28:19] update_nodes
[STOP] [2020-01-30 14:28:19] match_nodes
[START] [2020-01-30 14:28:19] reindex_search
[STOP] [2020-01-30 14:28:59] reindex_search
[START] [2020-01-30 14:28:59] normalize_units
[STOP] [2020-01-30 14:28:59] normalize_units
[START] [2020-01-30 14:28:59] calculate_statistics
[STOP] [2020-01-30 14:28:59] calculate_statistics
[START] [2020-01-30 14:28:59] complete_harvest_instance
[START] [2020-01-30 14:28:59] overall_tsv_creation
[INFO] [2020-01-30 14:28:59] Processing group of 9441 in 1 batches of 10000
[INFO] [2020-01-30 14:30:57] Average Time: 67.82
[INFO] [2020-01-30 14:30:57] Total Time: 1m59s
[STOP] [2020-01-30 14:30:57] overall_tsv_creation
[INFO] [2020-01-30 14:30:57] Done. Check your files:
[INFO] [2020-01-30 14:30:58] (9441 lines) /app/public/data/wiki_da_danish/publish_nodes.tsv
[INFO] [2020-01-30 14:30:59] (9441 lines) /app/public/data/wiki_da_danish/publish_identifiers.tsv
[INFO] [2020-01-30 14:31:00] (171312 lines) /app/public/data/wiki_da_danish/publish_node_ancestors.tsv
[INFO] [2020-01-30 14:31:00] (9441 lines) /app/public/data/wiki_da_danish/publish_scientific_names.tsv
[INFO] [2020-01-30 14:31:01] (92187 lines) /app/public/data/wiki_da_danish/publish_articles.tsv
[INFO] [2020-01-30 14:31:02] (11293 lines) /app/public/data/wiki_da_danish/publish_content_sections.tsv
[STOP] [2020-01-30 14:31:02] complete_harvest_instance
[START] [2020-01-30 14:31:02] completed
[STOP] [2020-01-30 14:31:02] completed
[STOP] [2020-01-30 14:31:02] logged process, took 2267.06

Latest Process