Harvest for wikipedia PMS Created 27 Feb 17:51

Stage: completed
Fetched: 27 Feb 17:51
Validated: 27 Feb 17:51
Deltas Created 27 Feb 17:51
Units Normalized: 27 Feb 18:02
Ancestry Built: 27 Feb 17:53
Nodes Matched: 27 Feb 18:01
Names Parsed: 27 Feb 17:53
New Models Stored: 27 Feb 17:52
Indexed: 27 Feb 18:02
Completed: 27 Feb 18:03
Time to Harvest: less than a minute

Harvesting Log

(118 lines)
# Logfile created on 2020-02-27 17:51:31 -0500 by logger.rb/56815
[START] [2020-02-27 17:51:31] logged process
[START] [2020-02-27 17:51:31] create_harvest_instance
[STOP] [2020-02-27 17:51:31] create_harvest_instance
[START] [2020-02-27 17:51:31] fetch_files
[STOP] [2020-02-27 17:51:31] fetch_files
[START] [2020-02-27 17:51:31] validate_each_file
[STOP] [2020-02-27 17:51:33] validate_each_file
[START] [2020-02-27 17:51:33] convert_to_csv
[CMD] [2020-02-27 17:51:33] /usr/bin/sort /app/public/converted_csv/wiki_pms_piedmon_nodes_20277.csv > /app/public/converted_csv/wiki_pms_piedmon_nodes_20277.csv_sorted
[CMD] [2020-02-27 17:51:33] /usr/bin/sort /app/public/converted_csv/wiki_pms_piedmon_media_20278.csv > /app/public/converted_csv/wiki_pms_piedmon_media_20278.csv_sorted
[STOP] [2020-02-27 17:51:33] convert_to_csv
[START] [2020-02-27 17:51:33] calculate_delta
[CMD] [2020-02-27 17:51:33] echo "0a" > /app/public/diff/wiki_pms_piedmon_nodes_20277.diff
[CMD] [2020-02-27 17:51:33] tail -n +1 /app/public/converted_csv/wiki_pms_piedmon_nodes_20277.csv >> /app/public/diff/wiki_pms_piedmon_nodes_20277.diff
[CMD] [2020-02-27 17:51:33] echo "." >> /app/public/diff/wiki_pms_piedmon_nodes_20277.diff
[CMD] [2020-02-27 17:51:33] echo "0a" > /app/public/diff/wiki_pms_piedmon_media_20278.diff
[CMD] [2020-02-27 17:51:33] tail -n +1 /app/public/converted_csv/wiki_pms_piedmon_media_20278.csv >> /app/public/diff/wiki_pms_piedmon_media_20278.diff
[CMD] [2020-02-27 17:51:34] echo "." >> /app/public/diff/wiki_pms_piedmon_media_20278.diff
[STOP] [2020-02-27 17:51:34] calculate_delta
[START] [2020-02-27 17:51:34] parse_diff_and_store
[INFO] [2020-02-27 17:51:34] Loading nodes diff file into memory (true lines)...
[INFO] [2020-02-27 17:51:36] Loading media diff file into memory (true lines)...
[INFO] [2020-02-27 17:52:13] Storing 5039 ScientificNames
[INFO] [2020-02-27 17:52:13] Processing group of 5039 in 6 groups of 1000
[INFO] [2020-02-27 17:52:15] Average Time: 0.353
[INFO] [2020-02-27 17:52:15] Total Time: 3s
[INFO] [2020-02-27 17:52:15] Storing 5039 Identifiers
[INFO] [2020-02-27 17:52:15] Processing group of 5039 in 6 groups of 1000
[INFO] [2020-02-27 17:52:16] Average Time: 0.093
[INFO] [2020-02-27 17:52:16] Total Time: 1s
[INFO] [2020-02-27 17:52:16] Storing 5039 Nodes
[INFO] [2020-02-27 17:52:16] Processing group of 5039 in 6 groups of 1000
[INFO] [2020-02-27 17:52:17] Average Time: 0.26
[INFO] [2020-02-27 17:52:17] Total Time: 2s
[INFO] [2020-02-27 17:52:17] Storing 5563 ArticlesSections
[INFO] [2020-02-27 17:52:17] Processing group of 5563 in 6 groups of 1000
[INFO] [2020-02-27 17:52:18] Average Time: 0.063
[INFO] [2020-02-27 17:52:18] Total Time: 1s
[INFO] [2020-02-27 17:52:18] Storing 5563 Articles
[INFO] [2020-02-27 17:52:18] Processing group of 5563 in 6 groups of 1000
[INFO] [2020-02-27 17:52:21] Average Time: 0.485
[INFO] [2020-02-27 17:52:21] Total Time: 3s
[STOP] [2020-02-27 17:52:21] parse_diff_and_store
[START] [2020-02-27 17:52:21] resolve_keys
[INFO] [2020-02-27 17:52:52] Occurrences to nodes (through scientific_names)...
[INFO] [2020-02-27 17:52:52] traits to occurrences...
[INFO] [2020-02-27 17:52:52] traits to nodes (through occurrences)...
[INFO] [2020-02-27 17:52:52] Traits to sex term...
[INFO] [2020-02-27 17:52:52] Traits to lifestage term...
[INFO] [2020-02-27 17:52:52] MetaTraits to traits...
[INFO] [2020-02-27 17:52:52] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-02-27 17:52:52] Assocs to occurrences...
[INFO] [2020-02-27 17:52:52] Assocs to nodes...
[INFO] [2020-02-27 17:52:52] Assoc to sex term...
[INFO] [2020-02-27 17:52:52] Assoc to lifestage term...
[STOP] [2020-02-27 17:52:52] resolve_keys
[START] [2020-02-27 17:52:52] hold_for_later_1
[STOP] [2020-02-27 17:52:52] hold_for_later_1
[START] [2020-02-27 17:52:52] hold_for_later_2
[STOP] [2020-02-27 17:52:52] hold_for_later_2
[START] [2020-02-27 17:52:52] resolve_missing_parents
[STOP] [2020-02-27 17:52:54] resolve_missing_parents
[START] [2020-02-27 17:52:54] rebuild_nodes
[START] [2020-02-27 17:52:54] Flattener#flatten
[START] [2020-02-27 17:52:54] Flattener#study_resource
[START] [2020-02-27 17:52:54] Flattener#build_ancestry
[STOP] [2020-02-27 17:52:54] Flattener#build_ancestry
[INFO] [2020-02-27 17:52:54] 5039 ancestry keys
[START] [2020-02-27 17:52:54] build_node_ancestors
[INFO] [2020-02-27 17:52:54] old ancestors deleted.
[STOP] [2020-02-27 17:53:00] build_node_ancestors
[START] [2020-02-27 17:53:06] Flattener#propagate_ancestor_ids
[STOP] [2020-02-27 17:53:08] Flattener#propagate_ancestor_ids
[STOP] [2020-02-27 17:53:08] Flattener#flatten
[STOP] [2020-02-27 17:53:08] rebuild_nodes
[START] [2020-02-27 17:53:08] resolve_missing_media_owners
[STOP] [2020-02-27 17:53:08] resolve_missing_media_owners
[START] [2020-02-27 17:53:08] sanitize_media_verbatims
[STOP] [2020-02-27 17:53:08] sanitize_media_verbatims
[START] [2020-02-27 17:53:08] queue_downloads
[STOP] [2020-02-27 17:53:08] queue_downloads
[START] [2020-02-27 17:53:08] parse_names
[WARN] [2020-02-27 17:53:08] I see 5039 names which still need to be parsed.
[WARN] [2020-02-27 17:53:13] I see 3 names which still need to be parsed.
[STOP] [2020-02-27 17:53:14] parse_names
[START] [2020-02-27 17:53:14] denormalize_canonical_names_to_nodes
[STOP] [2020-02-27 17:53:14] denormalize_canonical_names_to_nodes
[START] [2020-02-27 17:53:14] match_nodes
[START] [2020-02-27 17:53:14] map_all_nodes_to_pages
[STOP] [2020-02-27 18:01:52] map_all_nodes_to_pages
[INFO] [2020-02-27 18:01:52] 478 Unmatched nodes (of 5039)! That's too many to output. First 10: Biota (#64313898); Amorphea (#64314988); Physarida (#64312038); Embryophyta (#64313421); Spermatophytina (#64314099); Pinophyta (#64312313); Laricoideae (#64311949); Haploxylon (#64312691); Cembra (#64312692); Pseudostrobus (#64312699)
[START] [2020-02-27 18:01:52] update_nodes
[STOP] [2020-02-27 18:01:54] update_nodes
[STOP] [2020-02-27 18:01:54] match_nodes
[START] [2020-02-27 18:01:54] reindex_search
[STOP] [2020-02-27 18:02:11] reindex_search
[START] [2020-02-27 18:02:11] normalize_units
[STOP] [2020-02-27 18:02:11] normalize_units
[START] [2020-02-27 18:02:11] calculate_statistics
[STOP] [2020-02-27 18:02:11] calculate_statistics
[START] [2020-02-27 18:02:11] complete_harvest_instance
[START] [2020-02-27 18:02:11] overall_tsv_creation
[INFO] [2020-02-27 18:02:11] Processing group of 5039 in 1 batches of 10000
[INFO] [2020-02-27 18:03:25] Average Time: 39.27
[INFO] [2020-02-27 18:03:25] Total Time: 1m14s
[STOP] [2020-02-27 18:03:25] overall_tsv_creation
[INFO] [2020-02-27 18:03:25] Done. Check your files:
[INFO] [2020-02-27 18:03:25] (5039 lines) /app/public/data/wiki_pms_piedmon/publish_nodes.tsv
[INFO] [2020-02-27 18:03:25] (5039 lines) /app/public/data/wiki_pms_piedmon/publish_identifiers.tsv
[INFO] [2020-02-27 18:03:25] (78302 lines) /app/public/data/wiki_pms_piedmon/publish_node_ancestors.tsv
[INFO] [2020-02-27 18:03:25] (5039 lines) /app/public/data/wiki_pms_piedmon/publish_scientific_names.tsv
[INFO] [2020-02-27 18:03:25] (26349 lines) /app/public/data/wiki_pms_piedmon/publish_articles.tsv
[INFO] [2020-02-27 18:03:25] (5563 lines) /app/public/data/wiki_pms_piedmon/publish_content_sections.tsv
[STOP] [2020-02-27 18:03:25] complete_harvest_instance
[START] [2020-02-27 18:03:25] completed
[STOP] [2020-02-27 18:03:25] completed
[STOP] [2020-02-27 18:03:25] logged process, took 714.44

Latest Process