Harvest for Guinea Bissau Species List Created 13 Oct 06:44

Stage: completed
Fetched: 13 Oct 06:44
Validated: 13 Oct 06:44
Deltas Created 13 Oct 06:44
Units Normalized: 13 Oct 06:47
Ancestry Built: 13 Oct 06:44
Nodes Matched: 13 Oct 06:47
Names Parsed: 13 Oct 06:44
New Models Stored: 13 Oct 06:44
Indexed: 13 Oct 06:47
Completed: 13 Oct 06:49
Time to Harvest: less than a minute
Harvesting Log

(140 lines)
# Logfile created on 2019-10-13 06:44:13 -0400 by logger.rb/56815
[START] [2019-10-13 06:44:13] logged process
[START] [2019-10-13 06:44:13] create_harvest_instance
[STOP] [2019-10-13 06:44:14] create_harvest_instance
[START] [2019-10-13 06:44:14] fetch_files
[STOP] [2019-10-13 06:44:14] fetch_files
[START] [2019-10-13 06:44:14] validate_each_file
[STOP] [2019-10-13 06:44:14] validate_each_file
[START] [2019-10-13 06:44:14] convert_to_csv
[CMD] [2019-10-13 06:44:14] /usr/bin/sort /app/public/converted_csv/guinea_bissau_sp_refs_15931.csv > /app/public/converted_csv/guinea_bissau_sp_refs_15931.csv_sorted
[CMD] [2019-10-13 06:44:15] /usr/bin/sort /app/public/converted_csv/guinea_bissau_sp_nodes_15932.csv > /app/public/converted_csv/guinea_bissau_sp_nodes_15932.csv_sorted
[CMD] [2019-10-13 06:44:15] /usr/bin/sort /app/public/converted_csv/guinea_bissau_sp_occurrences_15933.csv > /app/public/converted_csv/guinea_bissau_sp_occurrences_15933.csv_sorted
[CMD] [2019-10-13 06:44:15] /usr/bin/sort /app/public/converted_csv/guinea_bissau_sp_measurements_15934.csv > /app/public/converted_csv/guinea_bissau_sp_measurements_15934.csv_sorted
[STOP] [2019-10-13 06:44:15] convert_to_csv
[START] [2019-10-13 06:44:15] calculate_delta
[CMD] [2019-10-13 06:44:15] echo "0a" > /app/public/diff/guinea_bissau_sp_refs_15931.diff
[CMD] [2019-10-13 06:44:15] tail -n +1 /app/public/converted_csv/guinea_bissau_sp_refs_15931.csv >> /app/public/diff/guinea_bissau_sp_refs_15931.diff
[CMD] [2019-10-13 06:44:15] echo "." >> /app/public/diff/guinea_bissau_sp_refs_15931.diff
[CMD] [2019-10-13 06:44:15] echo "0a" > /app/public/diff/guinea_bissau_sp_nodes_15932.diff
[CMD] [2019-10-13 06:44:15] tail -n +1 /app/public/converted_csv/guinea_bissau_sp_nodes_15932.csv >> /app/public/diff/guinea_bissau_sp_nodes_15932.diff
[CMD] [2019-10-13 06:44:15] echo "." >> /app/public/diff/guinea_bissau_sp_nodes_15932.diff
[CMD] [2019-10-13 06:44:15] echo "0a" > /app/public/diff/guinea_bissau_sp_occurrences_15933.diff
[CMD] [2019-10-13 06:44:15] tail -n +1 /app/public/converted_csv/guinea_bissau_sp_occurrences_15933.csv >> /app/public/diff/guinea_bissau_sp_occurrences_15933.diff
[CMD] [2019-10-13 06:44:16] echo "." >> /app/public/diff/guinea_bissau_sp_occurrences_15933.diff
[CMD] [2019-10-13 06:44:16] echo "0a" > /app/public/diff/guinea_bissau_sp_measurements_15934.diff
[CMD] [2019-10-13 06:44:16] tail -n +1 /app/public/converted_csv/guinea_bissau_sp_measurements_15934.csv >> /app/public/diff/guinea_bissau_sp_measurements_15934.diff
[CMD] [2019-10-13 06:44:16] echo "." >> /app/public/diff/guinea_bissau_sp_measurements_15934.diff
[STOP] [2019-10-13 06:44:16] calculate_delta
[START] [2019-10-13 06:44:16] parse_diff_and_store
[INFO] [2019-10-13 06:44:16] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-13 06:44:16] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-13 06:44:17] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-13 06:44:18] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-13 06:44:24] Storing 2 References
[INFO] [2019-10-13 06:44:24] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-13 06:44:24] Average Time: 0.0
[INFO] [2019-10-13 06:44:24] Total Time: 1s
[INFO] [2019-10-13 06:44:24] Storing 2269 ScientificNames
[INFO] [2019-10-13 06:44:24] Processing group of 2269 in 3 groups of 1000
[INFO] [2019-10-13 06:44:25] Average Time: 0.287
[INFO] [2019-10-13 06:44:25] Total Time: 1s
[INFO] [2019-10-13 06:44:25] Storing 2269 Nodes
[INFO] [2019-10-13 06:44:25] Processing group of 2269 in 3 groups of 1000
[INFO] [2019-10-13 06:44:26] Average Time: 0.213
[INFO] [2019-10-13 06:44:26] Total Time: 1s
[INFO] [2019-10-13 06:44:26] Storing 1306 Occurrences
[INFO] [2019-10-13 06:44:26] Processing group of 1306 in 2 groups of 1000
[INFO] [2019-10-13 06:44:26] Average Time: 0.07
[INFO] [2019-10-13 06:44:26] Total Time: 1s
[INFO] [2019-10-13 06:44:26] Storing 2202 TraitsReferences
[INFO] [2019-10-13 06:44:26] Processing group of 2202 in 3 groups of 1000
[INFO] [2019-10-13 06:44:26] Average Time: 0.103
[INFO] [2019-10-13 06:44:26] Total Time: 1s
[INFO] [2019-10-13 06:44:26] Storing 2202 Traits
[INFO] [2019-10-13 06:44:26] Processing group of 2202 in 3 groups of 1000
[INFO] [2019-10-13 06:44:27] Average Time: 0.27
[INFO] [2019-10-13 06:44:27] Total Time: 1s
[INFO] [2019-10-13 06:44:27] Storing 2202 MetaTraits
[INFO] [2019-10-13 06:44:27] Processing group of 2202 in 3 groups of 1000
[INFO] [2019-10-13 06:44:27] Average Time: 0.13
[INFO] [2019-10-13 06:44:27] Total Time: 1s
[STOP] [2019-10-13 06:44:27] parse_diff_and_store
[START] [2019-10-13 06:44:27] resolve_keys
[INFO] [2019-10-13 06:44:38] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-13 06:44:39] traits to occurrences...
[INFO] [2019-10-13 06:44:40] traits to nodes (through occurrences)...
[INFO] [2019-10-13 06:44:40] Traits to sex term...
[INFO] [2019-10-13 06:44:40] Traits to lifestage term...
[INFO] [2019-10-13 06:44:40] MetaTraits to traits...
[INFO] [2019-10-13 06:44:41] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-13 06:44:41] Assocs to occurrences...
[INFO] [2019-10-13 06:44:41] Assocs to nodes...
[INFO] [2019-10-13 06:44:41] Assoc to sex term...
[INFO] [2019-10-13 06:44:41] Assoc to lifestage term...
[STOP] [2019-10-13 06:44:41] resolve_keys
[START] [2019-10-13 06:44:41] hold_for_later_1
[STOP] [2019-10-13 06:44:41] hold_for_later_1
[START] [2019-10-13 06:44:41] hold_for_later_2
[STOP] [2019-10-13 06:44:41] hold_for_later_2
[START] [2019-10-13 06:44:41] resolve_missing_parents
[STOP] [2019-10-13 06:44:44] resolve_missing_parents
[START] [2019-10-13 06:44:44] rebuild_nodes
[START] [2019-10-13 06:44:44] Flattener#flatten
[START] [2019-10-13 06:44:44] Flattener#study_resource
[START] [2019-10-13 06:44:44] Flattener#build_ancestry
[STOP] [2019-10-13 06:44:44] Flattener#build_ancestry
[INFO] [2019-10-13 06:44:44] 2269 ancestry keys
[START] [2019-10-13 06:44:44] build_node_ancestors
[INFO] [2019-10-13 06:44:44] old ancestors deleted.
[STOP] [2019-10-13 06:44:45] build_node_ancestors
[START] [2019-10-13 06:44:46] Flattener#propagate_ancestor_ids
[STOP] [2019-10-13 06:44:46] Flattener#propagate_ancestor_ids
[STOP] [2019-10-13 06:44:46] Flattener#flatten
[STOP] [2019-10-13 06:44:46] rebuild_nodes
[START] [2019-10-13 06:44:46] resolve_missing_media_owners
[STOP] [2019-10-13 06:44:46] resolve_missing_media_owners
[START] [2019-10-13 06:44:46] sanitize_media_verbatims
[STOP] [2019-10-13 06:44:46] sanitize_media_verbatims
[START] [2019-10-13 06:44:46] queue_downloads
[STOP] [2019-10-13 06:44:46] queue_downloads
[START] [2019-10-13 06:44:46] parse_names
[WARN] [2019-10-13 06:44:46] I see 2269 names which still need to be parsed.
[WARN] [2019-10-13 06:44:49] I see 5 names which still need to be parsed.
[STOP] [2019-10-13 06:44:50] parse_names
[START] [2019-10-13 06:44:50] denormalize_canonical_names_to_nodes
[STOP] [2019-10-13 06:44:50] denormalize_canonical_names_to_nodes
[START] [2019-10-13 06:44:50] match_nodes
[START] [2019-10-13 06:44:50] map_all_nodes_to_pages
[STOP] [2019-10-13 06:47:36] map_all_nodes_to_pages
[INFO] [2019-10-13 06:47:36] 69 Unmatched nodes (of 2269)! That's too many to output. First 10: Pennisetum (#49966501); Kylinga (#49968349); Kylinga odorata (#49968348); Maxillopoda (#49968565); Stylochiton (#49967563); Magnoliopsida (#49966385); Albizia rhombifolium (#49968418); Caesalpinia benthamianum (#49967386); Aeschynomene pulchellum (#49968345); Nauclea latifolius (#49966477)
[START] [2019-10-13 06:47:36] update_nodes
[STOP] [2019-10-13 06:47:37] update_nodes
[STOP] [2019-10-13 06:47:37] match_nodes
[START] [2019-10-13 06:47:37] reindex_search
[STOP] [2019-10-13 06:47:42] reindex_search
[START] [2019-10-13 06:47:42] normalize_units
[STOP] [2019-10-13 06:47:42] normalize_units
[START] [2019-10-13 06:47:42] calculate_statistics
[STOP] [2019-10-13 06:47:42] calculate_statistics
[START] [2019-10-13 06:47:42] complete_harvest_instance
[START] [2019-10-13 06:47:42] overall_tsv_creation
[INFO] [2019-10-13 06:47:42] Processing group of 2269 in 1 batches of 10000
[INFO] [2019-10-13 06:48:37] 1100 Traits (unfiltered)...
[INFO] [2019-10-13 06:48:51] 1100 Traits (filtered)...
[INFO] [2019-10-13 06:48:51] 0 Associations (filtered)...
[INFO] [2019-10-13 06:49:30] 5500 metadata added.
[INFO] [2019-10-13 06:49:30] 0 metadata added.
[INFO] [2019-10-13 06:49:30] Average Time: 85.53
[INFO] [2019-10-13 06:49:30] Total Time: 1m48s
[STOP] [2019-10-13 06:49:30] overall_tsv_creation
[INFO] [2019-10-13 06:49:30] Done. Check your files:
[INFO] [2019-10-13 06:49:30] (2269 lines) /app/public/data/guinea_bissau_sp/publish_nodes.tsv
[INFO] [2019-10-13 06:49:30] (11551 lines) /app/public/data/guinea_bissau_sp/publish_node_ancestors.tsv
[INFO] [2019-10-13 06:49:30] (2269 lines) /app/public/data/guinea_bissau_sp/publish_scientific_names.tsv
[INFO] [2019-10-13 06:49:30] (1101 lines) /app/public/data/guinea_bissau_sp/publish_traits.tsv
[INFO] [2019-10-13 06:49:30] (5501 lines) /app/public/data/guinea_bissau_sp/publish_metadata.tsv
[STOP] [2019-10-13 06:49:30] complete_harvest_instance
[START] [2019-10-13 06:49:30] completed
[STOP] [2019-10-13 06:49:30] completed
[STOP] [2019-10-13 06:49:30] logged process, took 316.96
Latest Process

13 Oct 06:44 EXITED currently processing 0