Harvest for BioPedia Created 09 Apr 11:36

Stage: completed
Fetched: 09 Apr 11:36
Validated: 09 Apr 11:36
Deltas Created 09 Apr 11:36
Units Normalized: 09 Apr 11:44
Ancestry Built: 09 Apr 11:37
Nodes Matched: 09 Apr 11:44
Names Parsed: 09 Apr 11:37
New Models Stored: 09 Apr 11:36
Indexed: 09 Apr 11:44
Completed: 09 Apr 11:45
Time to Harvest: less than a minute
Harvesting Log

(129 lines)
# Logfile created on 2020-04-09 11:36:34 -0400 by logger.rb/v1.4.2
[INFO] [2020-04-09 11:36:34] ## HARVEST: type = -harvest
[START] [2020-04-09 11:36:39] logged process
[START] [2020-04-09 11:36:39] create_harvest_instance
[STOP] [2020-04-09 11:36:40] create_harvest_instance
[START] [2020-04-09 11:36:40] fetch_files
[STOP] [2020-04-09 11:36:40] fetch_files
[START] [2020-04-09 11:36:40] validate_each_file
[STOP] [2020-04-09 11:36:40] validate_each_file
[START] [2020-04-09 11:36:40] convert_to_csv
[CMD] [2020-04-09 11:36:40] /usr/bin/sort /app/public/converted_csv/bio_pedia_agents_20715.csv > /app/public/converted_csv/bio_pedia_agents_20715.csv_sorted
[CMD] [2020-04-09 11:36:40] /usr/bin/sort /app/public/converted_csv/bio_pedia_nodes_20716.csv > /app/public/converted_csv/bio_pedia_nodes_20716.csv_sorted
[CMD] [2020-04-09 11:36:41] /usr/bin/sort /app/public/converted_csv/bio_pedia_media_20717.csv > /app/public/converted_csv/bio_pedia_media_20717.csv_sorted
[STOP] [2020-04-09 11:36:41] convert_to_csv
[START] [2020-04-09 11:36:41] calculate_delta
[CMD] [2020-04-09 11:36:41] echo "0a" > /app/public/diff/bio_pedia_agents_20715.diff
[CMD] [2020-04-09 11:36:41] tail -n +1 /app/public/converted_csv/bio_pedia_agents_20715.csv >> /app/public/diff/bio_pedia_agents_20715.diff
[CMD] [2020-04-09 11:36:41] echo "." >> /app/public/diff/bio_pedia_agents_20715.diff
[CMD] [2020-04-09 11:36:41] echo "0a" > /app/public/diff/bio_pedia_nodes_20716.diff
[CMD] [2020-04-09 11:36:41] tail -n +1 /app/public/converted_csv/bio_pedia_nodes_20716.csv >> /app/public/diff/bio_pedia_nodes_20716.diff
[CMD] [2020-04-09 11:36:41] echo "." >> /app/public/diff/bio_pedia_nodes_20716.diff
[CMD] [2020-04-09 11:36:41] echo "0a" > /app/public/diff/bio_pedia_media_20717.diff
[CMD] [2020-04-09 11:36:41] tail -n +1 /app/public/converted_csv/bio_pedia_media_20717.csv >> /app/public/diff/bio_pedia_media_20717.diff
[CMD] [2020-04-09 11:36:41] echo "." >> /app/public/diff/bio_pedia_media_20717.diff
[STOP] [2020-04-09 11:36:41] calculate_delta
[START] [2020-04-09 11:36:41] parse_diff_and_store
[INFO] [2020-04-09 11:36:41] Loading agents diff file into memory (true lines)...
[INFO] [2020-04-09 11:36:42] Loading nodes diff file into memory (true lines)...
[WARN] [2020-04-09 11:36:42] Filtered Scientific Name `Acanthocorbis camarensis Hara <i>et al.</i> 1996` to `Acanthocorbis camarensis Hara <i>et al.<i> 1996`
[INFO] [2020-04-09 11:36:43] Loading media diff file into memory (true lines)...
[INFO] [2020-04-09 11:36:48] Storing 25 Attributions
[INFO] [2020-04-09 11:36:48] Processing group of 25 in 1 groups of 1000
[INFO] [2020-04-09 11:36:48] Average Time: 0.01
[INFO] [2020-04-09 11:36:48] Total Time: 1s
[INFO] [2020-04-09 11:36:48] Storing 4849 ScientificNames
[INFO] [2020-04-09 11:36:48] Processing group of 4849 in 5 groups of 1000
[INFO] [2020-04-09 11:36:49] Average Time: 0.292
[INFO] [2020-04-09 11:36:49] Total Time: 2s
[INFO] [2020-04-09 11:36:49] Storing 4849 Nodes
[INFO] [2020-04-09 11:36:49] Processing group of 4849 in 5 groups of 1000
[INFO] [2020-04-09 11:36:50] Average Time: 0.26
[INFO] [2020-04-09 11:36:50] Total Time: 2s
[INFO] [2020-04-09 11:36:50] Storing 10072 ContentAttributions
[INFO] [2020-04-09 11:36:50] Processing group of 10072 in 11 groups of 1000
[INFO] [2020-04-09 11:36:51] Average Time: 0.072
[INFO] [2020-04-09 11:36:51] Total Time: 1s
[INFO] [2020-04-09 11:36:51] last 3 / first 3: 0.68
[INFO] [2020-04-09 11:36:51] Std.Dev: 0.03162277660168379; Max: 0.11
[INFO] [2020-04-09 11:36:51] Storing 5037 ArticlesSections
[INFO] [2020-04-09 11:36:51] Processing group of 5037 in 6 groups of 1000
[INFO] [2020-04-09 11:36:52] Average Time: 0.045
[INFO] [2020-04-09 11:36:52] Total Time: 1s
[INFO] [2020-04-09 11:36:52] Storing 5037 Articles
[INFO] [2020-04-09 11:36:52] Processing group of 5037 in 6 groups of 1000
[INFO] [2020-04-09 11:36:53] Average Time: 0.235
[INFO] [2020-04-09 11:36:53] Total Time: 2s
[STOP] [2020-04-09 11:36:53] parse_diff_and_store
[START] [2020-04-09 11:36:53] resolve_keys
[INFO] [2020-04-09 11:37:07] Occurrences to nodes (through scientific_names)...
[INFO] [2020-04-09 11:37:07] traits to occurrences...
[INFO] [2020-04-09 11:37:07] traits to nodes (through occurrences)...
[INFO] [2020-04-09 11:37:07] Traits to sex term...
[INFO] [2020-04-09 11:37:07] Traits to lifestage term...
[INFO] [2020-04-09 11:37:07] MetaTraits to traits...
[INFO] [2020-04-09 11:37:07] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-04-09 11:37:07] Assocs to occurrences...
[INFO] [2020-04-09 11:37:07] Assocs to nodes...
[INFO] [2020-04-09 11:37:07] Assoc to sex term...
[INFO] [2020-04-09 11:37:07] Assoc to lifestage term...
[STOP] [2020-04-09 11:37:07] resolve_keys
[START] [2020-04-09 11:37:07] hold_for_later_1
[STOP] [2020-04-09 11:37:08] hold_for_later_1
[START] [2020-04-09 11:37:08] hold_for_later_2
[STOP] [2020-04-09 11:37:08] hold_for_later_2
[START] [2020-04-09 11:37:08] resolve_missing_parents
[STOP] [2020-04-09 11:37:08] resolve_missing_parents
[START] [2020-04-09 11:37:08] rebuild_nodes
[START] [2020-04-09 11:37:08] Flattener#flatten
[START] [2020-04-09 11:37:08] Flattener#study_resource
[START] [2020-04-09 11:37:08] Flattener#build_ancestry
[STOP] [2020-04-09 11:37:08] Flattener#build_ancestry
[INFO] [2020-04-09 11:37:08] 4849 ancestry keys
[START] [2020-04-09 11:37:08] build_node_ancestors
[INFO] [2020-04-09 11:37:08] old ancestors deleted.
[STOP] [2020-04-09 11:37:08] build_node_ancestors
[WARN] [2020-04-09 11:37:08] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2020-04-09 11:37:08] Flattener#flatten
[STOP] [2020-04-09 11:37:08] rebuild_nodes
[START] [2020-04-09 11:37:08] resolve_missing_media_owners
[STOP] [2020-04-09 11:37:08] resolve_missing_media_owners
[START] [2020-04-09 11:37:08] sanitize_media_verbatims
[STOP] [2020-04-09 11:37:08] sanitize_media_verbatims
[START] [2020-04-09 11:37:08] queue_downloads
[STOP] [2020-04-09 11:37:08] queue_downloads
[START] [2020-04-09 11:37:08] parse_names
[WARN] [2020-04-09 11:37:08] I see 4849 names which still need to be parsed.
[STOP] [2020-04-09 11:37:12] parse_names
[START] [2020-04-09 11:37:12] denormalize_canonical_names_to_nodes
[STOP] [2020-04-09 11:37:12] denormalize_canonical_names_to_nodes
[START] [2020-04-09 11:37:12] match_nodes
[START] [2020-04-09 11:37:12] map_all_nodes_to_pages
[STOP] [2020-04-09 11:44:35] map_all_nodes_to_pages
[INFO] [2020-04-09 11:44:35] 1928 Unmatched nodes (of 4849)! That's too many to output. First 10: Plasmodium (#68260853); Ceratium hirundinella (#68260855); Stylodinium (#68260858); Retortamonas (#68260859); Phryganella acropodia (#68260861); Torodinium (#68260864); Diachea leucopodia (#68260869); Epiplocylis exigua (#68260873); Pirsonia formosa (#68260875); Petalomonas raiula (#68260876)
[START] [2020-04-09 11:44:35] update_nodes
[STOP] [2020-04-09 11:44:37] update_nodes
[STOP] [2020-04-09 11:44:37] match_nodes
[START] [2020-04-09 11:44:37] reindex_search
[STOP] [2020-04-09 11:44:41] reindex_search
[START] [2020-04-09 11:44:41] normalize_units
[STOP] [2020-04-09 11:44:41] normalize_units
[START] [2020-04-09 11:44:41] calculate_statistics
[2020-04-09 11:44:42] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2020-04-09 11:44:42] calculate_statistics
[START] [2020-04-09 11:44:42] complete_harvest_instance
[START] [2020-04-09 11:44:42] overall_tsv_creation
[INFO] [2020-04-09 11:44:42] Processing group of 4849 in 1 batches of 10000
[INFO] [2020-04-09 11:45:43] Average Time: 24.79
[INFO] [2020-04-09 11:45:43] Total Time: 1m1s
[STOP] [2020-04-09 11:45:43] overall_tsv_creation
[INFO] [2020-04-09 11:45:43] Done. Check your files:
[INFO] [2020-04-09 11:45:43] (4849 lines) /app/public/data/bio_pedia/publish_nodes.tsv
[INFO] [2020-04-09 11:45:43] (4849 lines) /app/public/data/bio_pedia/publish_scientific_names.tsv
[INFO] [2020-04-09 11:45:43] (5037 lines) /app/public/data/bio_pedia/publish_articles.tsv
[INFO] [2020-04-09 11:45:43] (10072 lines) /app/public/data/bio_pedia/publish_attributions.tsv
[INFO] [2020-04-09 11:45:43] (5037 lines) /app/public/data/bio_pedia/publish_content_sections.tsv
[STOP] [2020-04-09 11:45:43] complete_harvest_instance
[START] [2020-04-09 11:45:43] completed
[STOP] [2020-04-09 11:45:43] completed
[STOP] [2020-04-09 11:45:43] logged process, took 544.79
Latest Process

09 Apr 11:36 EXITED currently processing 0