Harvest for wikipedia SK Created 24 Feb 18:18

Harvesting Log

# Logfile created on 2020-02-24 18:18:07 -0500 by logger.rb/56815
[START] [2020-02-24 18:18:07] logged process
[START] [2020-02-24 18:18:07] create_harvest_instance
[STOP] [2020-02-24 18:18:07] create_harvest_instance
[START] [2020-02-24 18:18:07] fetch_files
[STOP] [2020-02-24 18:18:07] fetch_files
[START] [2020-02-24 18:18:07] validate_each_file
[STOP] [2020-02-24 18:18:08] validate_each_file
[START] [2020-02-24 18:18:08] convert_to_csv
[CMD] [2020-02-24 18:18:08] /usr/bin/sort /app/public/converted_csv/wiki_sk_slovak_nodes_20259.csv > /app/public/converted_csv/wiki_sk_slovak_nodes_20259.csv_sorted
[CMD] [2020-02-24 18:18:08] /usr/bin/sort /app/public/converted_csv/wiki_sk_slovak_media_20260.csv > /app/public/converted_csv/wiki_sk_slovak_media_20260.csv_sorted
[STOP] [2020-02-24 18:18:08] convert_to_csv
[START] [2020-02-24 18:18:08] calculate_delta
[CMD] [2020-02-24 18:18:08] echo "0a" > /app/public/diff/wiki_sk_slovak_nodes_20259.diff
[CMD] [2020-02-24 18:18:08] tail -n +1 /app/public/converted_csv/wiki_sk_slovak_nodes_20259.csv >> /app/public/diff/wiki_sk_slovak_nodes_20259.diff
[CMD] [2020-02-24 18:18:08] echo "." >> /app/public/diff/wiki_sk_slovak_nodes_20259.diff
[CMD] [2020-02-24 18:18:08] echo "0a" > /app/public/diff/wiki_sk_slovak_media_20260.diff
[CMD] [2020-02-24 18:18:08] tail -n +1 /app/public/converted_csv/wiki_sk_slovak_media_20260.csv >> /app/public/diff/wiki_sk_slovak_media_20260.diff
[CMD] [2020-02-24 18:18:08] echo "." >> /app/public/diff/wiki_sk_slovak_media_20260.diff
[STOP] [2020-02-24 18:18:08] calculate_delta
[START] [2020-02-24 18:18:09] parse_diff_and_store
[INFO] [2020-02-24 18:18:09] Loading nodes diff file into memory (true lines)...
[WARN] [2020-02-24 18:18:10] Filtered Scientific Name `Ambystoma  hakié jas jas` to `Ambystoma hakié jas jas`
[INFO] [2020-02-24 18:18:11] Loading media diff file into memory (true lines)...
[INFO] [2020-02-24 18:18:40] Storing 5537 ScientificNames
[INFO] [2020-02-24 18:18:40] Processing group of 5537 in 6 groups of 1000
[INFO] [2020-02-24 18:18:42] Average Time: 0.368
[INFO] [2020-02-24 18:18:42] Total Time: 3s
[INFO] [2020-02-24 18:18:42] Storing 5537 Identifiers
[INFO] [2020-02-24 18:18:42] Processing group of 5537 in 6 groups of 1000
[INFO] [2020-02-24 18:18:43] Average Time: 0.09
[INFO] [2020-02-24 18:18:43] Total Time: 1s
[INFO] [2020-02-24 18:18:43] Storing 5537 Nodes
[INFO] [2020-02-24 18:18:43] Processing group of 5537 in 6 groups of 1000
[INFO] [2020-02-24 18:18:44] Average Time: 0.263
[INFO] [2020-02-24 18:18:44] Total Time: 2s
[INFO] [2020-02-24 18:18:44] Storing 5664 ArticlesSections
[INFO] [2020-02-24 18:18:44] Processing group of 5664 in 6 groups of 1000
[INFO] [2020-02-24 18:18:45] Average Time: 0.047
[INFO] [2020-02-24 18:18:45] Total Time: 1s
[INFO] [2020-02-24 18:18:45] Storing 5664 Articles
[INFO] [2020-02-24 18:18:45] Processing group of 5664 in 6 groups of 1000
[INFO] [2020-02-24 18:18:48] Average Time: 0.517
[INFO] [2020-02-24 18:18:48] Total Time: 4s
[STOP] [2020-02-24 18:18:48] parse_diff_and_store
[START] [2020-02-24 18:18:48] resolve_keys
[INFO] [2020-02-24 18:19:19] Occurrences to nodes (through scientific_names)...
[INFO] [2020-02-24 18:19:19] traits to occurrences...
[INFO] [2020-02-24 18:19:19] traits to nodes (through occurrences)...
[INFO] [2020-02-24 18:19:19] Traits to sex term...
[INFO] [2020-02-24 18:19:19] Traits to lifestage term...
[INFO] [2020-02-24 18:19:19] MetaTraits to traits...
[INFO] [2020-02-24 18:19:19] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-02-24 18:19:19] Assocs to occurrences...
[INFO] [2020-02-24 18:19:19] Assocs to nodes...
[INFO] [2020-02-24 18:19:19] Assoc to sex term...
[INFO] [2020-02-24 18:19:19] Assoc to lifestage term...
[STOP] [2020-02-24 18:19:19] resolve_keys
[START] [2020-02-24 18:19:19] hold_for_later_1
[STOP] [2020-02-24 18:19:19] hold_for_later_1
[START] [2020-02-24 18:19:19] hold_for_later_2
[STOP] [2020-02-24 18:19:19] hold_for_later_2
[START] [2020-02-24 18:19:19] resolve_missing_parents
[STOP] [2020-02-24 18:19:22] resolve_missing_parents
[START] [2020-02-24 18:19:22] rebuild_nodes
[START] [2020-02-24 18:19:22] Flattener#flatten
[START] [2020-02-24 18:19:22] Flattener#study_resource
[START] [2020-02-24 18:19:22] Flattener#build_ancestry
[STOP] [2020-02-24 18:19:22] Flattener#build_ancestry
[INFO] [2020-02-24 18:19:22] 5537 ancestry keys
[START] [2020-02-24 18:19:22] build_node_ancestors
[INFO] [2020-02-24 18:19:22] old ancestors deleted.
[STOP] [2020-02-24 18:19:36] build_node_ancestors
[START] [2020-02-24 18:19:37] Flattener#propagate_ancestor_ids
[STOP] [2020-02-24 18:19:40] Flattener#propagate_ancestor_ids
[STOP] [2020-02-24 18:19:40] Flattener#flatten
[STOP] [2020-02-24 18:19:40] rebuild_nodes
[START] [2020-02-24 18:19:40] resolve_missing_media_owners
[STOP] [2020-02-24 18:19:40] resolve_missing_media_owners
[START] [2020-02-24 18:19:40] sanitize_media_verbatims
[STOP] [2020-02-24 18:19:40] sanitize_media_verbatims
[START] [2020-02-24 18:19:40] queue_downloads
[STOP] [2020-02-24 18:19:40] queue_downloads
[START] [2020-02-24 18:19:40] parse_names
[WARN] [2020-02-24 18:19:40] I see 5537 names which still need to be parsed.
[WARN] [2020-02-24 18:19:44] I see 16 names which still need to be parsed.
[STOP] [2020-02-24 18:19:45] parse_names
[START] [2020-02-24 18:19:45] denormalize_canonical_names_to_nodes
[STOP] [2020-02-24 18:19:45] denormalize_canonical_names_to_nodes
[START] [2020-02-24 18:19:45] match_nodes
[START] [2020-02-24 18:19:45] map_all_nodes_to_pages
[STOP] [2020-02-24 18:34:06] map_all_nodes_to_pages
[INFO] [2020-02-24 18:34:06] 924 Unmatched nodes (of 5537)! That's too many to output. First 10: Biota (#63454424); Prokaryota (#63453720); Bacteria (#63451755); Negibacteria (#63455392); Escherichia coli (#63454592); Posibacteria (#63455435); Actinobacteria (#63452088); Actinobacteria (#63454739); Actinobacteridae (#63451628); Sarcomastigophora (#63454342)
[START] [2020-02-24 18:34:06] update_nodes
[STOP] [2020-02-24 18:34:08] update_nodes
[STOP] [2020-02-24 18:34:08] match_nodes
[START] [2020-02-24 18:34:08] reindex_search
[STOP] [2020-02-24 18:34:28] reindex_search
[START] [2020-02-24 18:34:28] normalize_units
[STOP] [2020-02-24 18:34:28] normalize_units
[START] [2020-02-24 18:34:28] calculate_statistics
[STOP] [2020-02-24 18:34:28] calculate_statistics
[START] [2020-02-24 18:34:28] complete_harvest_instance
[START] [2020-02-24 18:34:28] overall_tsv_creation
[INFO] [2020-02-24 18:34:28] Processing group of 5537 in 1 batches of 10000
[INFO] [2020-02-24 18:35:41] Average Time: 39.07
[INFO] [2020-02-24 18:35:41] Total Time: 1m13s
[STOP] [2020-02-24 18:35:41] overall_tsv_creation
[INFO] [2020-02-24 18:35:41] Done. Check your files:
[INFO] [2020-02-24 18:35:41] (5537 lines) /app/public/data/wiki_sk_slovak/publish_nodes.tsv
[INFO] [2020-02-24 18:35:41] (5537 lines) /app/public/data/wiki_sk_slovak/publish_identifiers.tsv
[INFO] [2020-02-24 18:35:41] (107680 lines) /app/public/data/wiki_sk_slovak/publish_node_ancestors.tsv
[INFO] [2020-02-24 18:35:41] (5537 lines) /app/public/data/wiki_sk_slovak/publish_scientific_names.tsv
[INFO] [2020-02-24 18:35:41] (89308 lines) /app/public/data/wiki_sk_slovak/publish_articles.tsv
[INFO] [2020-02-24 18:35:41] (5664 lines) /app/public/data/wiki_sk_slovak/publish_content_sections.tsv
[STOP] [2020-02-24 18:35:41] complete_harvest_instance
[START] [2020-02-24 18:35:41] completed
[STOP] [2020-02-24 18:35:41] completed
[STOP] [2020-02-24 18:35:41] logged process, took 1054.39

Latest Process

24 Feb 18:18 EXITED currently processing 0