Harvest for Nigeria Species List Created 14 Oct 13:56

Stage: completed
Fetched: 14 Oct 13:56
Validated: 14 Oct 13:56
Deltas Created 14 Oct 13:56
Units Normalized: 14 Oct 14:06
Ancestry Built: 14 Oct 13:58
Nodes Matched: 14 Oct 14:06
Names Parsed: 14 Oct 13:58
New Models Stored: 14 Oct 13:57
Indexed: 14 Oct 14:06
Completed: 14 Oct 14:10
Time to Harvest: less than a minute
Harvesting Log

(156 lines)
# Logfile created on 2019-10-14 13:56:15 -0400 by logger.rb/56815
[START] [2019-10-14 13:56:15] logged process
[START] [2019-10-14 13:56:15] create_harvest_instance
[STOP] [2019-10-14 13:56:15] create_harvest_instance
[START] [2019-10-14 13:56:15] fetch_files
[STOP] [2019-10-14 13:56:16] fetch_files
[START] [2019-10-14 13:56:16] validate_each_file
[STOP] [2019-10-14 13:56:17] validate_each_file
[START] [2019-10-14 13:56:17] convert_to_csv
[CMD] [2019-10-14 13:56:17] /usr/bin/sort /app/public/converted_csv/nigeria_sp_list_refs_16723.csv > /app/public/converted_csv/nigeria_sp_list_refs_16723.csv_sorted
[CMD] [2019-10-14 13:56:17] /usr/bin/sort /app/public/converted_csv/nigeria_sp_list_nodes_16724.csv > /app/public/converted_csv/nigeria_sp_list_nodes_16724.csv_sorted
[CMD] [2019-10-14 13:56:17] /usr/bin/sort /app/public/converted_csv/nigeria_sp_list_occurrences_16725.csv > /app/public/converted_csv/nigeria_sp_list_occurrences_16725.csv_sorted
[CMD] [2019-10-14 13:56:17] /usr/bin/sort /app/public/converted_csv/nigeria_sp_list_measurements_16726.csv > /app/public/converted_csv/nigeria_sp_list_measurements_16726.csv_sorted
[STOP] [2019-10-14 13:56:17] convert_to_csv
[START] [2019-10-14 13:56:17] calculate_delta
[CMD] [2019-10-14 13:56:17] echo "0a" > /app/public/diff/nigeria_sp_list_refs_16723.diff
[CMD] [2019-10-14 13:56:17] tail -n +1 /app/public/converted_csv/nigeria_sp_list_refs_16723.csv >> /app/public/diff/nigeria_sp_list_refs_16723.diff
[CMD] [2019-10-14 13:56:17] echo "." >> /app/public/diff/nigeria_sp_list_refs_16723.diff
[CMD] [2019-10-14 13:56:18] echo "0a" > /app/public/diff/nigeria_sp_list_nodes_16724.diff
[CMD] [2019-10-14 13:56:18] tail -n +1 /app/public/converted_csv/nigeria_sp_list_nodes_16724.csv >> /app/public/diff/nigeria_sp_list_nodes_16724.diff
[CMD] [2019-10-14 13:56:18] echo "." >> /app/public/diff/nigeria_sp_list_nodes_16724.diff
[CMD] [2019-10-14 13:56:18] echo "0a" > /app/public/diff/nigeria_sp_list_occurrences_16725.diff
[CMD] [2019-10-14 13:56:18] tail -n +1 /app/public/converted_csv/nigeria_sp_list_occurrences_16725.csv >> /app/public/diff/nigeria_sp_list_occurrences_16725.diff
[CMD] [2019-10-14 13:56:18] echo "." >> /app/public/diff/nigeria_sp_list_occurrences_16725.diff
[CMD] [2019-10-14 13:56:18] echo "0a" > /app/public/diff/nigeria_sp_list_measurements_16726.diff
[CMD] [2019-10-14 13:56:18] tail -n +1 /app/public/converted_csv/nigeria_sp_list_measurements_16726.csv >> /app/public/diff/nigeria_sp_list_measurements_16726.diff
[CMD] [2019-10-14 13:56:18] echo "." >> /app/public/diff/nigeria_sp_list_measurements_16726.diff
[STOP] [2019-10-14 13:56:18] calculate_delta
[START] [2019-10-14 13:56:18] parse_diff_and_store
[INFO] [2019-10-14 13:56:19] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-14 13:56:19] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-14 13:56:22] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-14 13:56:23] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-14 13:57:00] Storing 2 References
[INFO] [2019-10-14 13:57:00] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-14 13:57:00] Average Time: 0.0
[INFO] [2019-10-14 13:57:00] Total Time: 1s
[INFO] [2019-10-14 13:57:00] Storing 10207 ScientificNames
[INFO] [2019-10-14 13:57:00] Processing group of 10207 in 11 groups of 1000
[INFO] [2019-10-14 13:57:04] Average Time: 0.342
[INFO] [2019-10-14 13:57:04] Total Time: 4s
[INFO] [2019-10-14 13:57:04] last 3 / first 3: 0.68
[INFO] [2019-10-14 13:57:04] Std.Dev: 0.09486832980505137; Max: 0.44
[INFO] [2019-10-14 13:57:04] Storing 10207 Nodes
[INFO] [2019-10-14 13:57:04] Processing group of 10207 in 11 groups of 1000
[INFO] [2019-10-14 13:57:07] Average Time: 0.275
[INFO] [2019-10-14 13:57:07] Total Time: 4s
[INFO] [2019-10-14 13:57:07] last 3 / first 3: 0.75
[INFO] [2019-10-14 13:57:07] Std.Dev: 0.07071067811865475; Max: 0.34
[INFO] [2019-10-14 13:57:07] Storing 6266 Occurrences
[INFO] [2019-10-14 13:57:07] Processing group of 6266 in 7 groups of 1000
[INFO] [2019-10-14 13:57:08] Average Time: 0.101
[INFO] [2019-10-14 13:57:08] Total Time: 1s
[INFO] [2019-10-14 13:57:08] last 3 / first 3: 0.62
[INFO] [2019-10-14 13:57:08] Std.Dev: 0.03162277660168379; Max: 0.15
[INFO] [2019-10-14 13:57:08] Storing 12690 TraitsReferences
[INFO] [2019-10-14 13:57:08] Processing group of 12690 in 13 groups of 1000
[INFO] [2019-10-14 13:57:09] Average Time: 0.075
[INFO] [2019-10-14 13:57:09] Total Time: 2s
[INFO] [2019-10-14 13:57:09] last 3 / first 3: 0.66
[INFO] [2019-10-14 13:57:09] Std.Dev: 0.03162277660168379; Max: 0.15
[INFO] [2019-10-14 13:57:09] Storing 12689 Traits
[INFO] [2019-10-14 13:57:09] Processing group of 12689 in 13 groups of 1000
[INFO] [2019-10-14 13:57:14] Average Time: 0.357
[INFO] [2019-10-14 13:57:14] Total Time: 5s
[INFO] [2019-10-14 13:57:14] last 3 / first 3: 1.1
[INFO] [2019-10-14 13:57:14] Std.Dev: 0.07071067811865475; Max: 0.45
[INFO] [2019-10-14 13:57:14] Storing 12680 MetaTraits
[INFO] [2019-10-14 13:57:14] Processing group of 12680 in 13 groups of 1000
[INFO] [2019-10-14 13:57:15] Average Time: 0.108
[INFO] [2019-10-14 13:57:15] Total Time: 2s
[INFO] [2019-10-14 13:57:15] last 3 / first 3: 0.82
[INFO] [2019-10-14 13:57:15] Std.Dev: 0.0; Max: 0.14
[STOP] [2019-10-14 13:57:15] parse_diff_and_store
[START] [2019-10-14 13:57:15] resolve_keys
[INFO] [2019-10-14 13:57:51] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-14 13:57:55] traits to occurrences...
[INFO] [2019-10-14 13:58:00] traits to nodes (through occurrences)...
[INFO] [2019-10-14 13:58:00] Traits to sex term...
[INFO] [2019-10-14 13:58:04] Traits to lifestage term...
[INFO] [2019-10-14 13:58:08] MetaTraits to traits...
[INFO] [2019-10-14 13:58:09] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-14 13:58:11] Assocs to occurrences...
[INFO] [2019-10-14 13:58:11] Assocs to nodes...
[INFO] [2019-10-14 13:58:11] Assoc to sex term...
[INFO] [2019-10-14 13:58:11] Assoc to lifestage term...
[STOP] [2019-10-14 13:58:11] resolve_keys
[START] [2019-10-14 13:58:11] hold_for_later_1
[STOP] [2019-10-14 13:58:11] hold_for_later_1
[START] [2019-10-14 13:58:11] hold_for_later_2
[STOP] [2019-10-14 13:58:11] hold_for_later_2
[START] [2019-10-14 13:58:11] resolve_missing_parents
[STOP] [2019-10-14 13:58:32] resolve_missing_parents
[START] [2019-10-14 13:58:32] rebuild_nodes
[START] [2019-10-14 13:58:32] Flattener#flatten
[START] [2019-10-14 13:58:32] Flattener#study_resource
[START] [2019-10-14 13:58:32] Flattener#build_ancestry
[STOP] [2019-10-14 13:58:32] Flattener#build_ancestry
[INFO] [2019-10-14 13:58:32] 10207 ancestry keys
[START] [2019-10-14 13:58:32] build_node_ancestors
[INFO] [2019-10-14 13:58:32] old ancestors deleted.
[STOP] [2019-10-14 13:58:35] build_node_ancestors
[START] [2019-10-14 13:58:37] Flattener#propagate_ancestor_ids
[STOP] [2019-10-14 13:58:38] Flattener#propagate_ancestor_ids
[STOP] [2019-10-14 13:58:38] Flattener#flatten
[STOP] [2019-10-14 13:58:38] rebuild_nodes
[START] [2019-10-14 13:58:38] resolve_missing_media_owners
[STOP] [2019-10-14 13:58:38] resolve_missing_media_owners
[START] [2019-10-14 13:58:38] sanitize_media_verbatims
[STOP] [2019-10-14 13:58:38] sanitize_media_verbatims
[START] [2019-10-14 13:58:38] queue_downloads
[STOP] [2019-10-14 13:58:38] queue_downloads
[START] [2019-10-14 13:58:38] parse_names
[WARN] [2019-10-14 13:58:38] I see 10207 names which still need to be parsed.
[STOP] [2019-10-14 13:58:47] parse_names
[START] [2019-10-14 13:58:47] denormalize_canonical_names_to_nodes
[STOP] [2019-10-14 13:58:47] denormalize_canonical_names_to_nodes
[START] [2019-10-14 13:58:47] match_nodes
[START] [2019-10-14 13:58:47] map_all_nodes_to_pages
[STOP] [2019-10-14 14:06:21] map_all_nodes_to_pages
[INFO] [2019-10-14 14:06:21] 552 Unmatched nodes (of 10207)! That's too many to output. First 10: Vigna adenanthus (#50953698); Crotalaria vogelii (#50952888); Desmodium mauritianum (#50951548); Acacia dudgeoni (#50945563); Acacia hockii (#50945857); Bauhinia reticulatum (#50946734); Anthonotha nigericum (#50949688); Ormocarpum bibracteatum (#50946707); Dolichos falcatus (#50952358); Lonchocarpus laxiflora (#50947524)
[START] [2019-10-14 14:06:21] update_nodes
[STOP] [2019-10-14 14:06:25] update_nodes
[STOP] [2019-10-14 14:06:25] match_nodes
[START] [2019-10-14 14:06:25] reindex_search
[STOP] [2019-10-14 14:06:45] reindex_search
[START] [2019-10-14 14:06:45] normalize_units
[STOP] [2019-10-14 14:06:45] normalize_units
[START] [2019-10-14 14:06:45] calculate_statistics
[STOP] [2019-10-14 14:06:45] calculate_statistics
[START] [2019-10-14 14:06:45] complete_harvest_instance
[START] [2019-10-14 14:06:45] overall_tsv_creation
[INFO] [2019-10-14 14:06:45] Processing group of 10207 in 2 batches of 10000
[INFO] [2019-10-14 14:08:15] 6206 Traits (unfiltered)...
[INFO] [2019-10-14 14:08:29] 6206 Traits (filtered)...
[INFO] [2019-10-14 14:08:29] 0 Associations (filtered)...
[INFO] [2019-10-14 14:09:18] 31020 metadata added.
[INFO] [2019-10-14 14:09:18] 0 metadata added.
[INFO] [2019-10-14 14:10:03] 60 Traits (unfiltered)...
[INFO] [2019-10-14 14:10:16] 60 Traits (filtered)...
[INFO] [2019-10-14 14:10:16] 0 Associations (filtered)...
[INFO] [2019-10-14 14:10:54] 300 metadata added.
[INFO] [2019-10-14 14:10:54] 0 metadata added.
[INFO] [2019-10-14 14:10:54] Average Time: 99.785
[INFO] [2019-10-14 14:10:54] Total Time: 4m9s
[STOP] [2019-10-14 14:10:54] overall_tsv_creation
[INFO] [2019-10-14 14:10:54] Done. Check your files:
[INFO] [2019-10-14 14:10:54] (10207 lines) /app/public/data/nigeria_sp_list/publish_nodes.tsv
[INFO] [2019-10-14 14:10:54] (34752 lines) /app/public/data/nigeria_sp_list/publish_node_ancestors.tsv
[INFO] [2019-10-14 14:10:54] (10207 lines) /app/public/data/nigeria_sp_list/publish_scientific_names.tsv
[INFO] [2019-10-14 14:10:54] (6267 lines) /app/public/data/nigeria_sp_list/publish_traits.tsv
[INFO] [2019-10-14 14:10:54] (31321 lines) /app/public/data/nigeria_sp_list/publish_metadata.tsv
[STOP] [2019-10-14 14:10:54] complete_harvest_instance
[START] [2019-10-14 14:10:54] completed
[STOP] [2019-10-14 14:10:54] completed
[STOP] [2019-10-14 14:10:54] logged process, took 879.44
Latest Process

14 Oct 13:56 EXITED currently processing 0