Harvest for Egypt Species List Created 12 Oct 16:30

Stage: completed
Fetched: 12 Oct 16:30
Validated: 12 Oct 16:30
Deltas Created 12 Oct 16:30
Units Normalized: 12 Oct 16:46
Ancestry Built: 12 Oct 16:32
Nodes Matched: 12 Oct 16:46
Names Parsed: 12 Oct 16:32
New Models Stored: 12 Oct 16:31
Indexed: 12 Oct 16:46
Completed: 12 Oct 16:49
Time to Harvest: less than a minute
Harvesting Log

(149 lines)
# Logfile created on 2019-10-12 16:30:10 -0400 by logger.rb/56815
[START] [2019-10-12 16:30:10] logged process
[START] [2019-10-12 16:30:10] create_harvest_instance
[STOP] [2019-10-12 16:30:11] create_harvest_instance
[START] [2019-10-12 16:30:11] fetch_files
[STOP] [2019-10-12 16:30:11] fetch_files
[START] [2019-10-12 16:30:11] validate_each_file
[STOP] [2019-10-12 16:30:12] validate_each_file
[START] [2019-10-12 16:30:12] convert_to_csv
[CMD] [2019-10-12 16:30:12] /usr/bin/sort /app/public/converted_csv/Egypt_Species_Li_refs_15643.csv > /app/public/converted_csv/Egypt_Species_Li_refs_15643.csv_sorted
[CMD] [2019-10-12 16:30:12] /usr/bin/sort /app/public/converted_csv/Egypt_Species_Li_nodes_15644.csv > /app/public/converted_csv/Egypt_Species_Li_nodes_15644.csv_sorted
[CMD] [2019-10-12 16:30:13] /usr/bin/sort /app/public/converted_csv/Egypt_Species_Li_occurrences_15645.csv > /app/public/converted_csv/Egypt_Species_Li_occurrences_15645.csv_sorted
[CMD] [2019-10-12 16:30:13] /usr/bin/sort /app/public/converted_csv/Egypt_Species_Li_measurements_15646.csv > /app/public/converted_csv/Egypt_Species_Li_measurements_15646.csv_sorted
[STOP] [2019-10-12 16:30:13] convert_to_csv
[START] [2019-10-12 16:30:13] calculate_delta
[CMD] [2019-10-12 16:30:13] echo "0a" > /app/public/diff/Egypt_Species_Li_refs_15643.diff
[CMD] [2019-10-12 16:30:13] tail -n +1 /app/public/converted_csv/Egypt_Species_Li_refs_15643.csv >> /app/public/diff/Egypt_Species_Li_refs_15643.diff
[CMD] [2019-10-12 16:30:13] echo "." >> /app/public/diff/Egypt_Species_Li_refs_15643.diff
[CMD] [2019-10-12 16:30:13] echo "0a" > /app/public/diff/Egypt_Species_Li_nodes_15644.diff
[CMD] [2019-10-12 16:30:13] tail -n +1 /app/public/converted_csv/Egypt_Species_Li_nodes_15644.csv >> /app/public/diff/Egypt_Species_Li_nodes_15644.diff
[CMD] [2019-10-12 16:30:13] echo "." >> /app/public/diff/Egypt_Species_Li_nodes_15644.diff
[CMD] [2019-10-12 16:30:13] echo "0a" > /app/public/diff/Egypt_Species_Li_occurrences_15645.diff
[CMD] [2019-10-12 16:30:13] tail -n +1 /app/public/converted_csv/Egypt_Species_Li_occurrences_15645.csv >> /app/public/diff/Egypt_Species_Li_occurrences_15645.diff
[CMD] [2019-10-12 16:30:13] echo "." >> /app/public/diff/Egypt_Species_Li_occurrences_15645.diff
[CMD] [2019-10-12 16:30:14] echo "0a" > /app/public/diff/Egypt_Species_Li_measurements_15646.diff
[CMD] [2019-10-12 16:30:14] tail -n +1 /app/public/converted_csv/Egypt_Species_Li_measurements_15646.csv >> /app/public/diff/Egypt_Species_Li_measurements_15646.diff
[CMD] [2019-10-12 16:30:14] echo "." >> /app/public/diff/Egypt_Species_Li_measurements_15646.diff
[STOP] [2019-10-12 16:30:14] calculate_delta
[START] [2019-10-12 16:30:14] parse_diff_and_store
[INFO] [2019-10-12 16:30:14] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-12 16:30:14] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-12 16:30:18] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-12 16:30:19] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-12 16:30:51] Storing 2 References
[INFO] [2019-10-12 16:30:51] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-12 16:30:51] Average Time: 0.0
[INFO] [2019-10-12 16:30:51] Total Time: 1s
[INFO] [2019-10-12 16:30:51] Storing 9787 ScientificNames
[INFO] [2019-10-12 16:30:51] Processing group of 9787 in 10 groups of 1000
[INFO] [2019-10-12 16:30:55] Average Time: 0.346
[INFO] [2019-10-12 16:30:55] Total Time: 4s
[INFO] [2019-10-12 16:30:55] last 3 / first 3: 0.84
[INFO] [2019-10-12 16:30:55] Std.Dev: 0.05477225575051661; Max: 0.44
[INFO] [2019-10-12 16:30:55] Storing 9787 Nodes
[INFO] [2019-10-12 16:30:55] Processing group of 9787 in 10 groups of 1000
[INFO] [2019-10-12 16:30:58] Average Time: 0.288
[INFO] [2019-10-12 16:30:58] Total Time: 3s
[INFO] [2019-10-12 16:30:58] last 3 / first 3: 0.95
[INFO] [2019-10-12 16:30:58] Std.Dev: 0.03162277660168379; Max: 0.38
[INFO] [2019-10-12 16:30:58] Storing 5306 Occurrences
[INFO] [2019-10-12 16:30:58] Processing group of 5306 in 6 groups of 1000
[INFO] [2019-10-12 16:30:58] Average Time: 0.09
[INFO] [2019-10-12 16:30:58] Total Time: 1s
[INFO] [2019-10-12 16:30:58] Storing 10612 TraitsReferences
[INFO] [2019-10-12 16:30:58] Processing group of 10612 in 11 groups of 1000
[INFO] [2019-10-12 16:30:59] Average Time: 0.082
[INFO] [2019-10-12 16:30:59] Total Time: 1s
[INFO] [2019-10-12 16:30:59] last 3 / first 3: 0.64
[INFO] [2019-10-12 16:30:59] Std.Dev: 0.044721359549995794; Max: 0.19
[INFO] [2019-10-12 16:30:59] Storing 10612 Traits
[INFO] [2019-10-12 16:30:59] Processing group of 10612 in 11 groups of 1000
[INFO] [2019-10-12 16:31:03] Average Time: 0.318
[INFO] [2019-10-12 16:31:03] Total Time: 4s
[INFO] [2019-10-12 16:31:03] last 3 / first 3: 0.8
[INFO] [2019-10-12 16:31:03] Std.Dev: 0.06324555320336758; Max: 0.41
[INFO] [2019-10-12 16:31:03] Storing 10600 MetaTraits
[INFO] [2019-10-12 16:31:03] Processing group of 10600 in 11 groups of 1000
[INFO] [2019-10-12 16:31:05] Average Time: 0.185
[INFO] [2019-10-12 16:31:05] Total Time: 3s
[INFO] [2019-10-12 16:31:05] last 3 / first 3: 2.22
[INFO] [2019-10-12 16:31:05] Std.Dev: 0.17320508075688773; Max: 0.7
[STOP] [2019-10-12 16:31:05] parse_diff_and_store
[START] [2019-10-12 16:31:05] resolve_keys
[INFO] [2019-10-12 16:31:42] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-12 16:31:45] traits to occurrences...
[INFO] [2019-10-12 16:31:47] traits to nodes (through occurrences)...
[INFO] [2019-10-12 16:31:48] Traits to sex term...
[INFO] [2019-10-12 16:31:50] Traits to lifestage term...
[INFO] [2019-10-12 16:31:52] MetaTraits to traits...
[INFO] [2019-10-12 16:31:52] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-12 16:31:54] Assocs to occurrences...
[INFO] [2019-10-12 16:31:54] Assocs to nodes...
[INFO] [2019-10-12 16:31:54] Assoc to sex term...
[INFO] [2019-10-12 16:31:54] Assoc to lifestage term...
[STOP] [2019-10-12 16:31:54] resolve_keys
[START] [2019-10-12 16:31:54] hold_for_later_1
[STOP] [2019-10-12 16:31:54] hold_for_later_1
[START] [2019-10-12 16:31:54] hold_for_later_2
[STOP] [2019-10-12 16:31:54] hold_for_later_2
[START] [2019-10-12 16:31:54] resolve_missing_parents
[STOP] [2019-10-12 16:32:12] resolve_missing_parents
[START] [2019-10-12 16:32:12] rebuild_nodes
[START] [2019-10-12 16:32:12] Flattener#flatten
[START] [2019-10-12 16:32:12] Flattener#study_resource
[START] [2019-10-12 16:32:12] Flattener#build_ancestry
[STOP] [2019-10-12 16:32:13] Flattener#build_ancestry
[INFO] [2019-10-12 16:32:13] 9787 ancestry keys
[START] [2019-10-12 16:32:13] build_node_ancestors
[INFO] [2019-10-12 16:32:13] old ancestors deleted.
[STOP] [2019-10-12 16:32:17] build_node_ancestors
[START] [2019-10-12 16:32:21] Flattener#propagate_ancestor_ids
[STOP] [2019-10-12 16:32:22] Flattener#propagate_ancestor_ids
[STOP] [2019-10-12 16:32:22] Flattener#flatten
[STOP] [2019-10-12 16:32:22] rebuild_nodes
[START] [2019-10-12 16:32:22] resolve_missing_media_owners
[STOP] [2019-10-12 16:32:22] resolve_missing_media_owners
[START] [2019-10-12 16:32:22] sanitize_media_verbatims
[STOP] [2019-10-12 16:32:22] sanitize_media_verbatims
[START] [2019-10-12 16:32:22] queue_downloads
[STOP] [2019-10-12 16:32:22] queue_downloads
[START] [2019-10-12 16:32:22] parse_names
[WARN] [2019-10-12 16:32:22] I see 9787 names which still need to be parsed.
[STOP] [2019-10-12 16:32:31] parse_names
[START] [2019-10-12 16:32:31] denormalize_canonical_names_to_nodes
[STOP] [2019-10-12 16:32:31] denormalize_canonical_names_to_nodes
[START] [2019-10-12 16:32:31] match_nodes
[START] [2019-10-12 16:32:31] map_all_nodes_to_pages
[STOP] [2019-10-12 16:46:25] map_all_nodes_to_pages
[INFO] [2019-10-12 16:46:25] 632 Unmatched nodes (of 9787)! That's too many to output. First 10: Ochlerotatus caspius (#49530301); Richardidae (#49524142); Monomorium dentatum (#49523386); Euodynerus hottentotum (#49529621); Stilbum (#49523809); Stilbum cyanurum (#49523808); Diglyphus crassinervis (#49529757); Deroplax (#49524167); Deroplax silphoides (#49524166); Limnogeton (#49529955)
[START] [2019-10-12 16:46:25] update_nodes
[STOP] [2019-10-12 16:46:28] update_nodes
[STOP] [2019-10-12 16:46:28] match_nodes
[START] [2019-10-12 16:46:28] reindex_search
[STOP] [2019-10-12 16:46:52] reindex_search
[START] [2019-10-12 16:46:52] normalize_units
[STOP] [2019-10-12 16:46:52] normalize_units
[START] [2019-10-12 16:46:52] calculate_statistics
[STOP] [2019-10-12 16:46:52] calculate_statistics
[START] [2019-10-12 16:46:52] complete_harvest_instance
[START] [2019-10-12 16:46:52] overall_tsv_creation
[INFO] [2019-10-12 16:46:52] Processing group of 9787 in 1 batches of 10000
[INFO] [2019-10-12 16:48:22] 5306 Traits (unfiltered)...
[INFO] [2019-10-12 16:48:36] 5306 Traits (filtered)...
[INFO] [2019-10-12 16:48:36] 0 Associations (filtered)...
[INFO] [2019-10-12 16:49:24] 26518 metadata added.
[INFO] [2019-10-12 16:49:24] 0 metadata added.
[INFO] [2019-10-12 16:49:25] Average Time: 124.39
[INFO] [2019-10-12 16:49:25] Total Time: 2m33s
[STOP] [2019-10-12 16:49:25] overall_tsv_creation
[INFO] [2019-10-12 16:49:25] Done. Check your files:
[INFO] [2019-10-12 16:49:25] (9787 lines) /app/public/data/Egypt_Species_Li/publish_nodes.tsv
[INFO] [2019-10-12 16:49:25] (51636 lines) /app/public/data/Egypt_Species_Li/publish_node_ancestors.tsv
[INFO] [2019-10-12 16:49:25] (9787 lines) /app/public/data/Egypt_Species_Li/publish_scientific_names.tsv
[INFO] [2019-10-12 16:49:25] (5307 lines) /app/public/data/Egypt_Species_Li/publish_traits.tsv
[INFO] [2019-10-12 16:49:25] (26519 lines) /app/public/data/Egypt_Species_Li/publish_metadata.tsv
[STOP] [2019-10-12 16:49:25] complete_harvest_instance
[START] [2019-10-12 16:49:25] completed
[STOP] [2019-10-12 16:49:25] completed
[STOP] [2019-10-12 16:49:25] logged process, took 1154.82
Latest Process

12 Oct 16:30 EXITED currently processing 0