Harvest for Mauritania Species List Created 14 Oct 05:21

Stage: completed
Fetched: 14 Oct 05:21
Validated: 14 Oct 05:21
Deltas Created 14 Oct 05:21
Units Normalized: 14 Oct 05:24
Ancestry Built: 14 Oct 05:21
Nodes Matched: 14 Oct 05:24
Names Parsed: 14 Oct 05:21
New Models Stored: 14 Oct 05:21
Indexed: 14 Oct 05:24
Completed: 14 Oct 05:26
Time to Harvest: less than a minute
Harvesting Log

(139 lines)
# Logfile created on 2019-10-14 05:21:15 -0400 by logger.rb/56815
[START] [2019-10-14 05:21:15] logged process
[START] [2019-10-14 05:21:15] create_harvest_instance
[STOP] [2019-10-14 05:21:15] create_harvest_instance
[START] [2019-10-14 05:21:15] fetch_files
[STOP] [2019-10-14 05:21:15] fetch_files
[START] [2019-10-14 05:21:15] validate_each_file
[STOP] [2019-10-14 05:21:16] validate_each_file
[START] [2019-10-14 05:21:16] convert_to_csv
[CMD] [2019-10-14 05:21:16] /usr/bin/sort /app/public/converted_csv/mauritania_sp_li_refs_16555.csv > /app/public/converted_csv/mauritania_sp_li_refs_16555.csv_sorted
[CMD] [2019-10-14 05:21:16] /usr/bin/sort /app/public/converted_csv/mauritania_sp_li_nodes_16556.csv > /app/public/converted_csv/mauritania_sp_li_nodes_16556.csv_sorted
[CMD] [2019-10-14 05:21:16] /usr/bin/sort /app/public/converted_csv/mauritania_sp_li_occurrences_16557.csv > /app/public/converted_csv/mauritania_sp_li_occurrences_16557.csv_sorted
[CMD] [2019-10-14 05:21:16] /usr/bin/sort /app/public/converted_csv/mauritania_sp_li_measurements_16558.csv > /app/public/converted_csv/mauritania_sp_li_measurements_16558.csv_sorted
[STOP] [2019-10-14 05:21:16] convert_to_csv
[START] [2019-10-14 05:21:16] calculate_delta
[CMD] [2019-10-14 05:21:16] echo "0a" > /app/public/diff/mauritania_sp_li_refs_16555.diff
[CMD] [2019-10-14 05:21:16] tail -n +1 /app/public/converted_csv/mauritania_sp_li_refs_16555.csv >> /app/public/diff/mauritania_sp_li_refs_16555.diff
[CMD] [2019-10-14 05:21:16] echo "." >> /app/public/diff/mauritania_sp_li_refs_16555.diff
[CMD] [2019-10-14 05:21:17] echo "0a" > /app/public/diff/mauritania_sp_li_nodes_16556.diff
[CMD] [2019-10-14 05:21:17] tail -n +1 /app/public/converted_csv/mauritania_sp_li_nodes_16556.csv >> /app/public/diff/mauritania_sp_li_nodes_16556.diff
[CMD] [2019-10-14 05:21:17] echo "." >> /app/public/diff/mauritania_sp_li_nodes_16556.diff
[CMD] [2019-10-14 05:21:17] echo "0a" > /app/public/diff/mauritania_sp_li_occurrences_16557.diff
[CMD] [2019-10-14 05:21:17] tail -n +1 /app/public/converted_csv/mauritania_sp_li_occurrences_16557.csv >> /app/public/diff/mauritania_sp_li_occurrences_16557.diff
[CMD] [2019-10-14 05:21:17] echo "." >> /app/public/diff/mauritania_sp_li_occurrences_16557.diff
[CMD] [2019-10-14 05:21:17] echo "0a" > /app/public/diff/mauritania_sp_li_measurements_16558.diff
[CMD] [2019-10-14 05:21:17] tail -n +1 /app/public/converted_csv/mauritania_sp_li_measurements_16558.csv >> /app/public/diff/mauritania_sp_li_measurements_16558.diff
[CMD] [2019-10-14 05:21:17] echo "." >> /app/public/diff/mauritania_sp_li_measurements_16558.diff
[STOP] [2019-10-14 05:21:17] calculate_delta
[START] [2019-10-14 05:21:17] parse_diff_and_store
[INFO] [2019-10-14 05:21:17] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-14 05:21:18] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-14 05:21:19] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-14 05:21:19] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-14 05:21:28] Storing 2 References
[INFO] [2019-10-14 05:21:28] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-14 05:21:28] Average Time: 0.0
[INFO] [2019-10-14 05:21:28] Total Time: 1s
[INFO] [2019-10-14 05:21:28] Storing 2807 ScientificNames
[INFO] [2019-10-14 05:21:28] Processing group of 2807 in 3 groups of 1000
[INFO] [2019-10-14 05:21:29] Average Time: 0.35
[INFO] [2019-10-14 05:21:29] Total Time: 2s
[INFO] [2019-10-14 05:21:29] Storing 2807 Nodes
[INFO] [2019-10-14 05:21:29] Processing group of 2807 in 3 groups of 1000
[INFO] [2019-10-14 05:21:30] Average Time: 0.283
[INFO] [2019-10-14 05:21:30] Total Time: 1s
[INFO] [2019-10-14 05:21:30] Storing 1446 Occurrences
[INFO] [2019-10-14 05:21:30] Processing group of 1446 in 2 groups of 1000
[INFO] [2019-10-14 05:21:30] Average Time: 0.095
[INFO] [2019-10-14 05:21:30] Total Time: 1s
[INFO] [2019-10-14 05:21:30] Storing 3120 TraitsReferences
[INFO] [2019-10-14 05:21:30] Processing group of 3120 in 4 groups of 1000
[INFO] [2019-10-14 05:21:30] Average Time: 0.073
[INFO] [2019-10-14 05:21:30] Total Time: 1s
[INFO] [2019-10-14 05:21:30] Storing 3119 Traits
[INFO] [2019-10-14 05:21:30] Processing group of 3119 in 4 groups of 1000
[INFO] [2019-10-14 05:21:31] Average Time: 0.265
[INFO] [2019-10-14 05:21:31] Total Time: 2s
[INFO] [2019-10-14 05:21:31] Storing 3120 MetaTraits
[INFO] [2019-10-14 05:21:31] Processing group of 3120 in 4 groups of 1000
[INFO] [2019-10-14 05:21:32] Average Time: 0.103
[INFO] [2019-10-14 05:21:32] Total Time: 1s
[STOP] [2019-10-14 05:21:32] parse_diff_and_store
[START] [2019-10-14 05:21:32] resolve_keys
[INFO] [2019-10-14 05:21:46] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-14 05:21:47] traits to occurrences...
[INFO] [2019-10-14 05:21:48] traits to nodes (through occurrences)...
[INFO] [2019-10-14 05:21:48] Traits to sex term...
[INFO] [2019-10-14 05:21:48] Traits to lifestage term...
[INFO] [2019-10-14 05:21:49] MetaTraits to traits...
[INFO] [2019-10-14 05:21:49] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-14 05:21:49] Assocs to occurrences...
[INFO] [2019-10-14 05:21:50] Assocs to nodes...
[INFO] [2019-10-14 05:21:50] Assoc to sex term...
[INFO] [2019-10-14 05:21:50] Assoc to lifestage term...
[STOP] [2019-10-14 05:21:50] resolve_keys
[START] [2019-10-14 05:21:50] hold_for_later_1
[STOP] [2019-10-14 05:21:50] hold_for_later_1
[START] [2019-10-14 05:21:50] hold_for_later_2
[STOP] [2019-10-14 05:21:50] hold_for_later_2
[START] [2019-10-14 05:21:50] resolve_missing_parents
[STOP] [2019-10-14 05:21:55] resolve_missing_parents
[START] [2019-10-14 05:21:55] rebuild_nodes
[START] [2019-10-14 05:21:55] Flattener#flatten
[START] [2019-10-14 05:21:55] Flattener#study_resource
[START] [2019-10-14 05:21:55] Flattener#build_ancestry
[STOP] [2019-10-14 05:21:55] Flattener#build_ancestry
[INFO] [2019-10-14 05:21:55] 2807 ancestry keys
[START] [2019-10-14 05:21:55] build_node_ancestors
[INFO] [2019-10-14 05:21:55] old ancestors deleted.
[STOP] [2019-10-14 05:21:55] build_node_ancestors
[START] [2019-10-14 05:21:56] Flattener#propagate_ancestor_ids
[STOP] [2019-10-14 05:21:56] Flattener#propagate_ancestor_ids
[STOP] [2019-10-14 05:21:56] Flattener#flatten
[STOP] [2019-10-14 05:21:56] rebuild_nodes
[START] [2019-10-14 05:21:56] resolve_missing_media_owners
[STOP] [2019-10-14 05:21:56] resolve_missing_media_owners
[START] [2019-10-14 05:21:56] sanitize_media_verbatims
[STOP] [2019-10-14 05:21:56] sanitize_media_verbatims
[START] [2019-10-14 05:21:56] queue_downloads
[STOP] [2019-10-14 05:21:56] queue_downloads
[START] [2019-10-14 05:21:56] parse_names
[WARN] [2019-10-14 05:21:56] I see 2807 names which still need to be parsed.
[STOP] [2019-10-14 05:21:59] parse_names
[START] [2019-10-14 05:21:59] denormalize_canonical_names_to_nodes
[STOP] [2019-10-14 05:21:59] denormalize_canonical_names_to_nodes
[START] [2019-10-14 05:21:59] match_nodes
[START] [2019-10-14 05:21:59] map_all_nodes_to_pages
[STOP] [2019-10-14 05:24:36] map_all_nodes_to_pages
[INFO] [2019-10-14 05:24:36] 132 Unmatched nodes (of 2807)! That's too many to output. First 10: Larus audouinii (#50664899); Larus melanocephalus (#50667434); Thalaseus (#50664968); Thalaseus sandvicensis (#50664967); Thalaseus maximus (#50665768); Thalaseus bengalensis (#50666067); Thalaseus maxima (#50667381); Turdoides fulva (#50665777); Phylloscopus sibillatrix (#50666322); Euplectes macrourus (#50667288)
[START] [2019-10-14 05:24:36] update_nodes
[STOP] [2019-10-14 05:24:37] update_nodes
[STOP] [2019-10-14 05:24:37] match_nodes
[START] [2019-10-14 05:24:37] reindex_search
[STOP] [2019-10-14 05:24:43] reindex_search
[START] [2019-10-14 05:24:43] normalize_units
[STOP] [2019-10-14 05:24:43] normalize_units
[START] [2019-10-14 05:24:43] calculate_statistics
[STOP] [2019-10-14 05:24:43] calculate_statistics
[START] [2019-10-14 05:24:43] complete_harvest_instance
[START] [2019-10-14 05:24:43] overall_tsv_creation
[INFO] [2019-10-14 05:24:43] Processing group of 2807 in 1 batches of 10000
[INFO] [2019-10-14 05:25:41] 1446 Traits (unfiltered)...
[INFO] [2019-10-14 05:25:54] 1446 Traits (filtered)...
[INFO] [2019-10-14 05:25:54] 0 Associations (filtered)...
[INFO] [2019-10-14 05:26:40] 7230 metadata added.
[INFO] [2019-10-14 05:26:40] 0 metadata added.
[INFO] [2019-10-14 05:26:40] Average Time: 94.02
[INFO] [2019-10-14 05:26:40] Total Time: 1m58s
[STOP] [2019-10-14 05:26:40] overall_tsv_creation
[INFO] [2019-10-14 05:26:40] Done. Check your files:
[INFO] [2019-10-14 05:26:40] (2807 lines) /app/public/data/mauritania_sp_li/publish_nodes.tsv
[INFO] [2019-10-14 05:26:40] (3847 lines) /app/public/data/mauritania_sp_li/publish_node_ancestors.tsv
[INFO] [2019-10-14 05:26:40] (2807 lines) /app/public/data/mauritania_sp_li/publish_scientific_names.tsv
[INFO] [2019-10-14 05:26:40] (1447 lines) /app/public/data/mauritania_sp_li/publish_traits.tsv
[INFO] [2019-10-14 05:26:41] (7231 lines) /app/public/data/mauritania_sp_li/publish_metadata.tsv
[STOP] [2019-10-14 05:26:41] complete_harvest_instance
[START] [2019-10-14 05:26:41] completed
[STOP] [2019-10-14 05:26:41] completed
[STOP] [2019-10-14 05:26:41] logged process, took 325.92
Latest Process

14 Oct 05:21 EXITED currently processing 0