Harvest for wikipedia HSB Created 27 Feb 17:41

Stage: completed
Fetched: 27 Feb 17:41
Validated: 27 Feb 17:41
Deltas Created 27 Feb 17:41
Units Normalized: 27 Feb 17:49
Ancestry Built: 27 Feb 17:43
Nodes Matched: 27 Feb 17:49
Names Parsed: 27 Feb 17:43
New Models Stored: 27 Feb 17:42
Indexed: 27 Feb 17:49
Completed: 27 Feb 17:51
Time to Harvest: less than a minute
Harvesting Log

(122 lines)
# Logfile created on 2020-02-27 17:41:41 -0500 by logger.rb/56815
[START] [2020-02-27 17:41:41] logged process
[START] [2020-02-27 17:41:41] create_harvest_instance
[STOP] [2020-02-27 17:41:42] create_harvest_instance
[START] [2020-02-27 17:41:42] fetch_files
[STOP] [2020-02-27 17:41:42] fetch_files
[START] [2020-02-27 17:41:42] validate_each_file
[STOP] [2020-02-27 17:41:43] validate_each_file
[START] [2020-02-27 17:41:43] convert_to_csv
[CMD] [2020-02-27 17:41:43] /usr/bin/sort /app/public/converted_csv/wiki_hsb_upper_s_nodes_20275.csv > /app/public/converted_csv/wiki_hsb_upper_s_nodes_20275.csv_sorted
[CMD] [2020-02-27 17:41:43] /usr/bin/sort /app/public/converted_csv/wiki_hsb_upper_s_media_20276.csv > /app/public/converted_csv/wiki_hsb_upper_s_media_20276.csv_sorted
[STOP] [2020-02-27 17:41:43] convert_to_csv
[START] [2020-02-27 17:41:43] calculate_delta
[CMD] [2020-02-27 17:41:43] echo "0a" > /app/public/diff/wiki_hsb_upper_s_nodes_20275.diff
[CMD] [2020-02-27 17:41:43] tail -n +1 /app/public/converted_csv/wiki_hsb_upper_s_nodes_20275.csv >> /app/public/diff/wiki_hsb_upper_s_nodes_20275.diff
[CMD] [2020-02-27 17:41:43] echo "." >> /app/public/diff/wiki_hsb_upper_s_nodes_20275.diff
[CMD] [2020-02-27 17:41:44] echo "0a" > /app/public/diff/wiki_hsb_upper_s_media_20276.diff
[CMD] [2020-02-27 17:41:44] tail -n +1 /app/public/converted_csv/wiki_hsb_upper_s_media_20276.csv >> /app/public/diff/wiki_hsb_upper_s_media_20276.diff
[CMD] [2020-02-27 17:41:44] echo "." >> /app/public/diff/wiki_hsb_upper_s_media_20276.diff
[STOP] [2020-02-27 17:41:44] calculate_delta
[START] [2020-02-27 17:41:44] parse_diff_and_store
[INFO] [2020-02-27 17:41:44] Loading nodes diff file into memory (true lines)...
[INFO] [2020-02-27 17:41:46] Loading media diff file into memory (true lines)...
[INFO] [2020-02-27 17:42:10] Storing 4357 ScientificNames
[INFO] [2020-02-27 17:42:10] Processing group of 4357 in 5 groups of 1000
[INFO] [2020-02-27 17:42:12] Average Time: 0.348
[INFO] [2020-02-27 17:42:12] Total Time: 2s
[INFO] [2020-02-27 17:42:12] Storing 4357 Identifiers
[INFO] [2020-02-27 17:42:12] Processing group of 4357 in 5 groups of 1000
[INFO] [2020-02-27 17:42:13] Average Time: 0.128
[INFO] [2020-02-27 17:42:13] Total Time: 1s
[INFO] [2020-02-27 17:42:13] Storing 4357 Nodes
[INFO] [2020-02-27 17:42:13] Processing group of 4357 in 5 groups of 1000
[INFO] [2020-02-27 17:42:14] Average Time: 0.28
[INFO] [2020-02-27 17:42:14] Total Time: 2s
[INFO] [2020-02-27 17:42:14] Storing 6512 ArticlesSections
[INFO] [2020-02-27 17:42:14] Processing group of 6512 in 7 groups of 1000
[INFO] [2020-02-27 17:42:14] Average Time: 0.051
[INFO] [2020-02-27 17:42:14] Total Time: 1s
[INFO] [2020-02-27 17:42:14] last 3 / first 3: 0.94
[INFO] [2020-02-27 17:42:14] Std.Dev: 0.0; Max: 0.06
[INFO] [2020-02-27 17:42:14] Storing 6512 Articles
[INFO] [2020-02-27 17:42:14] Processing group of 6512 in 7 groups of 1000
[INFO] [2020-02-27 17:42:17] Average Time: 0.436
[INFO] [2020-02-27 17:42:17] Total Time: 4s
[INFO] [2020-02-27 17:42:17] last 3 / first 3: 0.69
[INFO] [2020-02-27 17:42:17] Std.Dev: 0.10954451150103323; Max: 0.55
[STOP] [2020-02-27 17:42:17] parse_diff_and_store
[START] [2020-02-27 17:42:17] resolve_keys
[INFO] [2020-02-27 17:43:30] Occurrences to nodes (through scientific_names)...
[INFO] [2020-02-27 17:43:30] traits to occurrences...
[INFO] [2020-02-27 17:43:30] traits to nodes (through occurrences)...
[INFO] [2020-02-27 17:43:30] Traits to sex term...
[INFO] [2020-02-27 17:43:30] Traits to lifestage term...
[INFO] [2020-02-27 17:43:30] MetaTraits to traits...
[INFO] [2020-02-27 17:43:30] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-02-27 17:43:30] Assocs to occurrences...
[INFO] [2020-02-27 17:43:30] Assocs to nodes...
[INFO] [2020-02-27 17:43:30] Assoc to sex term...
[INFO] [2020-02-27 17:43:30] Assoc to lifestage term...
[STOP] [2020-02-27 17:43:30] resolve_keys
[START] [2020-02-27 17:43:30] hold_for_later_1
[STOP] [2020-02-27 17:43:30] hold_for_later_1
[START] [2020-02-27 17:43:30] hold_for_later_2
[STOP] [2020-02-27 17:43:30] hold_for_later_2
[START] [2020-02-27 17:43:30] resolve_missing_parents
[STOP] [2020-02-27 17:43:31] resolve_missing_parents
[START] [2020-02-27 17:43:31] rebuild_nodes
[START] [2020-02-27 17:43:31] Flattener#flatten
[START] [2020-02-27 17:43:31] Flattener#study_resource
[START] [2020-02-27 17:43:31] Flattener#build_ancestry
[STOP] [2020-02-27 17:43:31] Flattener#build_ancestry
[INFO] [2020-02-27 17:43:31] 4357 ancestry keys
[START] [2020-02-27 17:43:31] build_node_ancestors
[INFO] [2020-02-27 17:43:31] old ancestors deleted.
[STOP] [2020-02-27 17:43:36] build_node_ancestors
[START] [2020-02-27 17:43:41] Flattener#propagate_ancestor_ids
[STOP] [2020-02-27 17:43:42] Flattener#propagate_ancestor_ids
[STOP] [2020-02-27 17:43:42] Flattener#flatten
[STOP] [2020-02-27 17:43:42] rebuild_nodes
[START] [2020-02-27 17:43:42] resolve_missing_media_owners
[STOP] [2020-02-27 17:43:42] resolve_missing_media_owners
[START] [2020-02-27 17:43:42] sanitize_media_verbatims
[STOP] [2020-02-27 17:43:42] sanitize_media_verbatims
[START] [2020-02-27 17:43:42] queue_downloads
[STOP] [2020-02-27 17:43:42] queue_downloads
[START] [2020-02-27 17:43:42] parse_names
[WARN] [2020-02-27 17:43:42] I see 4357 names which still need to be parsed.
[WARN] [2020-02-27 17:43:46] I see 1 names which still need to be parsed.
[STOP] [2020-02-27 17:43:48] parse_names
[START] [2020-02-27 17:43:48] denormalize_canonical_names_to_nodes
[STOP] [2020-02-27 17:43:48] denormalize_canonical_names_to_nodes
[START] [2020-02-27 17:43:48] match_nodes
[START] [2020-02-27 17:43:48] map_all_nodes_to_pages
[STOP] [2020-02-27 17:49:29] map_all_nodes_to_pages
[INFO] [2020-02-27 17:49:29] 606 Unmatched nodes (of 4357)! That's too many to output. First 10: Biota (#64310221); Prokaryota (#64310015); Bacteria (#64306739); Negibacteria (#64310498); Escherichia coli (#64310273); Embryophyta (#64310022); Spermatophytina (#64310288); Pinophyta (#64306896); Laricoideae (#64306774); Haploxylon (#64307338)
[START] [2020-02-27 17:49:29] update_nodes
[STOP] [2020-02-27 17:49:30] update_nodes
[STOP] [2020-02-27 17:49:30] match_nodes
[START] [2020-02-27 17:49:30] reindex_search
[STOP] [2020-02-27 17:49:43] reindex_search
[START] [2020-02-27 17:49:43] normalize_units
[STOP] [2020-02-27 17:49:43] normalize_units
[START] [2020-02-27 17:49:43] calculate_statistics
[STOP] [2020-02-27 17:49:43] calculate_statistics
[START] [2020-02-27 17:49:43] complete_harvest_instance
[START] [2020-02-27 17:49:43] overall_tsv_creation
[INFO] [2020-02-27 17:49:43] Processing group of 4357 in 1 batches of 10000
[INFO] [2020-02-27 17:51:30] Average Time: 54.91
[INFO] [2020-02-27 17:51:30] Total Time: 1m47s
[STOP] [2020-02-27 17:51:30] overall_tsv_creation
[INFO] [2020-02-27 17:51:30] Done. Check your files:
[INFO] [2020-02-27 17:51:30] (4357 lines) /app/public/data/wiki_hsb_upper_s/publish_nodes.tsv
[INFO] [2020-02-27 17:51:30] (4357 lines) /app/public/data/wiki_hsb_upper_s/publish_identifiers.tsv
[INFO] [2020-02-27 17:51:30] (61273 lines) /app/public/data/wiki_hsb_upper_s/publish_node_ancestors.tsv
[INFO] [2020-02-27 17:51:31] (4357 lines) /app/public/data/wiki_hsb_upper_s/publish_scientific_names.tsv
[INFO] [2020-02-27 17:51:31] (90053 lines) /app/public/data/wiki_hsb_upper_s/publish_articles.tsv
[INFO] [2020-02-27 17:51:31] (6512 lines) /app/public/data/wiki_hsb_upper_s/publish_content_sections.tsv
[STOP] [2020-02-27 17:51:31] complete_harvest_instance
[START] [2020-02-27 17:51:31] completed
[STOP] [2020-02-27 17:51:31] completed
[STOP] [2020-02-27 17:51:31] logged process, took 589.7
Latest Process

27 Feb 17:41 EXITED currently processing 0