Harvest for United Arab Emirates Species List Created 16 Oct 20:23

Stage: completed
Fetched: 16 Oct 20:23
Validated: 16 Oct 20:23
Deltas Created 16 Oct 20:23
Units Normalized: 16 Oct 20:26
Ancestry Built: 16 Oct 20:23
Nodes Matched: 16 Oct 20:26
Names Parsed: 16 Oct 20:23
New Models Stored: 16 Oct 20:23
Indexed: 16 Oct 20:26
Completed: 16 Oct 20:28
Time to Harvest: less than a minute
Harvesting Log

(139 lines)
# Logfile created on 2019-10-16 20:23:10 -0400 by logger.rb/56815
[START] [2019-10-16 20:23:10] logged process
[START] [2019-10-16 20:23:10] create_harvest_instance
[STOP] [2019-10-16 20:23:10] create_harvest_instance
[START] [2019-10-16 20:23:10] fetch_files
[STOP] [2019-10-16 20:23:10] fetch_files
[START] [2019-10-16 20:23:10] validate_each_file
[STOP] [2019-10-16 20:23:11] validate_each_file
[START] [2019-10-16 20:23:11] convert_to_csv
[CMD] [2019-10-16 20:23:11] /usr/bin/sort /app/public/converted_csv/uaesl_refs_17617.csv > /app/public/converted_csv/uaesl_refs_17617.csv_sorted
[CMD] [2019-10-16 20:23:11] /usr/bin/sort /app/public/converted_csv/uaesl_nodes_17618.csv > /app/public/converted_csv/uaesl_nodes_17618.csv_sorted
[CMD] [2019-10-16 20:23:11] /usr/bin/sort /app/public/converted_csv/uaesl_occurrences_17619.csv > /app/public/converted_csv/uaesl_occurrences_17619.csv_sorted
[CMD] [2019-10-16 20:23:11] /usr/bin/sort /app/public/converted_csv/uaesl_measurements_17620.csv > /app/public/converted_csv/uaesl_measurements_17620.csv_sorted
[STOP] [2019-10-16 20:23:11] convert_to_csv
[START] [2019-10-16 20:23:11] calculate_delta
[CMD] [2019-10-16 20:23:11] echo "0a" > /app/public/diff/uaesl_refs_17617.diff
[CMD] [2019-10-16 20:23:11] tail -n +1 /app/public/converted_csv/uaesl_refs_17617.csv >> /app/public/diff/uaesl_refs_17617.diff
[CMD] [2019-10-16 20:23:11] echo "." >> /app/public/diff/uaesl_refs_17617.diff
[CMD] [2019-10-16 20:23:11] echo "0a" > /app/public/diff/uaesl_nodes_17618.diff
[CMD] [2019-10-16 20:23:11] tail -n +1 /app/public/converted_csv/uaesl_nodes_17618.csv >> /app/public/diff/uaesl_nodes_17618.diff
[CMD] [2019-10-16 20:23:11] echo "." >> /app/public/diff/uaesl_nodes_17618.diff
[CMD] [2019-10-16 20:23:11] echo "0a" > /app/public/diff/uaesl_occurrences_17619.diff
[CMD] [2019-10-16 20:23:11] tail -n +1 /app/public/converted_csv/uaesl_occurrences_17619.csv >> /app/public/diff/uaesl_occurrences_17619.diff
[CMD] [2019-10-16 20:23:11] echo "." >> /app/public/diff/uaesl_occurrences_17619.diff
[CMD] [2019-10-16 20:23:11] echo "0a" > /app/public/diff/uaesl_measurements_17620.diff
[CMD] [2019-10-16 20:23:11] tail -n +1 /app/public/converted_csv/uaesl_measurements_17620.csv >> /app/public/diff/uaesl_measurements_17620.diff
[CMD] [2019-10-16 20:23:11] echo "." >> /app/public/diff/uaesl_measurements_17620.diff
[STOP] [2019-10-16 20:23:12] calculate_delta
[START] [2019-10-16 20:23:12] parse_diff_and_store
[INFO] [2019-10-16 20:23:12] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-16 20:23:12] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-16 20:23:13] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-16 20:23:13] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-16 20:23:20] Storing 2 References
[INFO] [2019-10-16 20:23:20] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-16 20:23:20] Average Time: 0.0
[INFO] [2019-10-16 20:23:20] Total Time: 1s
[INFO] [2019-10-16 20:23:20] Storing 2399 ScientificNames
[INFO] [2019-10-16 20:23:20] Processing group of 2399 in 3 groups of 1000
[INFO] [2019-10-16 20:23:21] Average Time: 0.32
[INFO] [2019-10-16 20:23:21] Total Time: 1s
[INFO] [2019-10-16 20:23:21] Storing 2399 Nodes
[INFO] [2019-10-16 20:23:21] Processing group of 2399 in 3 groups of 1000
[INFO] [2019-10-16 20:23:22] Average Time: 0.237
[INFO] [2019-10-16 20:23:22] Total Time: 1s
[INFO] [2019-10-16 20:23:22] Storing 964 Occurrences
[INFO] [2019-10-16 20:23:22] Processing group of 964 in 1 groups of 1000
[INFO] [2019-10-16 20:23:22] Average Time: 0.11
[INFO] [2019-10-16 20:23:22] Total Time: 1s
[INFO] [2019-10-16 20:23:22] Storing 2550 TraitsReferences
[INFO] [2019-10-16 20:23:22] Processing group of 2550 in 3 groups of 1000
[INFO] [2019-10-16 20:23:22] Average Time: 0.083
[INFO] [2019-10-16 20:23:22] Total Time: 1s
[INFO] [2019-10-16 20:23:22] Storing 2549 Traits
[INFO] [2019-10-16 20:23:22] Processing group of 2549 in 3 groups of 1000
[INFO] [2019-10-16 20:23:23] Average Time: 0.293
[INFO] [2019-10-16 20:23:23] Total Time: 1s
[INFO] [2019-10-16 20:23:23] Storing 2549 MetaTraits
[INFO] [2019-10-16 20:23:23] Processing group of 2549 in 3 groups of 1000
[INFO] [2019-10-16 20:23:23] Average Time: 0.11
[INFO] [2019-10-16 20:23:23] Total Time: 1s
[STOP] [2019-10-16 20:23:23] parse_diff_and_store
[START] [2019-10-16 20:23:23] resolve_keys
[INFO] [2019-10-16 20:23:36] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-16 20:23:36] traits to occurrences...
[INFO] [2019-10-16 20:23:37] traits to nodes (through occurrences)...
[INFO] [2019-10-16 20:23:37] Traits to sex term...
[INFO] [2019-10-16 20:23:38] Traits to lifestage term...
[INFO] [2019-10-16 20:23:39] MetaTraits to traits...
[INFO] [2019-10-16 20:23:39] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-16 20:23:40] Assocs to occurrences...
[INFO] [2019-10-16 20:23:40] Assocs to nodes...
[INFO] [2019-10-16 20:23:40] Assoc to sex term...
[INFO] [2019-10-16 20:23:40] Assoc to lifestage term...
[STOP] [2019-10-16 20:23:40] resolve_keys
[START] [2019-10-16 20:23:40] hold_for_later_1
[STOP] [2019-10-16 20:23:40] hold_for_later_1
[START] [2019-10-16 20:23:40] hold_for_later_2
[STOP] [2019-10-16 20:23:40] hold_for_later_2
[START] [2019-10-16 20:23:40] resolve_missing_parents
[STOP] [2019-10-16 20:23:43] resolve_missing_parents
[START] [2019-10-16 20:23:43] rebuild_nodes
[START] [2019-10-16 20:23:43] Flattener#flatten
[START] [2019-10-16 20:23:43] Flattener#study_resource
[START] [2019-10-16 20:23:43] Flattener#build_ancestry
[STOP] [2019-10-16 20:23:43] Flattener#build_ancestry
[INFO] [2019-10-16 20:23:43] 2399 ancestry keys
[START] [2019-10-16 20:23:43] build_node_ancestors
[INFO] [2019-10-16 20:23:43] old ancestors deleted.
[STOP] [2019-10-16 20:23:43] build_node_ancestors
[START] [2019-10-16 20:23:44] Flattener#propagate_ancestor_ids
[STOP] [2019-10-16 20:23:44] Flattener#propagate_ancestor_ids
[STOP] [2019-10-16 20:23:44] Flattener#flatten
[STOP] [2019-10-16 20:23:44] rebuild_nodes
[START] [2019-10-16 20:23:44] resolve_missing_media_owners
[STOP] [2019-10-16 20:23:44] resolve_missing_media_owners
[START] [2019-10-16 20:23:44] sanitize_media_verbatims
[STOP] [2019-10-16 20:23:44] sanitize_media_verbatims
[START] [2019-10-16 20:23:44] queue_downloads
[STOP] [2019-10-16 20:23:44] queue_downloads
[START] [2019-10-16 20:23:44] parse_names
[WARN] [2019-10-16 20:23:44] I see 2399 names which still need to be parsed.
[STOP] [2019-10-16 20:23:47] parse_names
[START] [2019-10-16 20:23:47] denormalize_canonical_names_to_nodes
[STOP] [2019-10-16 20:23:47] denormalize_canonical_names_to_nodes
[START] [2019-10-16 20:23:47] match_nodes
[START] [2019-10-16 20:23:47] map_all_nodes_to_pages
[STOP] [2019-10-16 20:26:46] map_all_nodes_to_pages
[INFO] [2019-10-16 20:26:46] 177 Unmatched nodes (of 2399)! That's too many to output. First 10: Butorides striatus (#52560324); Petronia xanthocollis (#52559754); Petronia brachydactyla (#52560918); Streptopelia senegalensis (#52559251); Aquila fasciatus (#52561369); Hieraaetus pennata (#52561515); Torgos tracheliotus (#52559785); Calandrella rufescens (#52559699); Oenanthe alboniger (#52560008); Erythropygia (#52559545)
[START] [2019-10-16 20:26:46] update_nodes
[STOP] [2019-10-16 20:26:47] update_nodes
[STOP] [2019-10-16 20:26:47] match_nodes
[START] [2019-10-16 20:26:47] reindex_search
[STOP] [2019-10-16 20:26:52] reindex_search
[START] [2019-10-16 20:26:52] normalize_units
[STOP] [2019-10-16 20:26:52] normalize_units
[START] [2019-10-16 20:26:52] calculate_statistics
[STOP] [2019-10-16 20:26:52] calculate_statistics
[START] [2019-10-16 20:26:52] complete_harvest_instance
[START] [2019-10-16 20:26:52] overall_tsv_creation
[INFO] [2019-10-16 20:26:52] Processing group of 2399 in 1 batches of 10000
[INFO] [2019-10-16 20:27:45] 964 Traits (unfiltered)...
[INFO] [2019-10-16 20:27:58] 964 Traits (filtered)...
[INFO] [2019-10-16 20:27:58] 0 Associations (filtered)...
[INFO] [2019-10-16 20:28:37] 4820 metadata added.
[INFO] [2019-10-16 20:28:37] 0 metadata added.
[INFO] [2019-10-16 20:28:37] Average Time: 82.25
[INFO] [2019-10-16 20:28:37] Total Time: 1m45s
[STOP] [2019-10-16 20:28:37] overall_tsv_creation
[INFO] [2019-10-16 20:28:37] Done. Check your files:
[INFO] [2019-10-16 20:28:37] (2399 lines) /app/public/data/uaesl/publish_nodes.tsv
[INFO] [2019-10-16 20:28:37] (3242 lines) /app/public/data/uaesl/publish_node_ancestors.tsv
[INFO] [2019-10-16 20:28:37] (2399 lines) /app/public/data/uaesl/publish_scientific_names.tsv
[INFO] [2019-10-16 20:28:37] (965 lines) /app/public/data/uaesl/publish_traits.tsv
[INFO] [2019-10-16 20:28:37] (4821 lines) /app/public/data/uaesl/publish_metadata.tsv
[STOP] [2019-10-16 20:28:37] complete_harvest_instance
[START] [2019-10-16 20:28:37] completed
[STOP] [2019-10-16 20:28:37] completed
[STOP] [2019-10-16 20:28:37] logged process, took 327.51
Latest Process

16 Oct 20:23 EXITED currently processing 0