Harvest for Bangladesh Species List Created 02 Oct 01:48

Stage: completed
Fetched: 02 Oct 01:48
Validated: 02 Oct 01:48
Deltas Created 02 Oct 01:48
Units Normalized: 02 Oct 01:51
Ancestry Built: 02 Oct 01:49
Nodes Matched: 02 Oct 01:51
Names Parsed: 02 Oct 01:49
New Models Stored: 02 Oct 01:48
Indexed: 02 Oct 01:51
Completed: 02 Oct 01:53
Time to Harvest: less than a minute
Harvesting Log

(139 lines)
# Logfile created on 2019-10-02 01:48:07 -0400 by logger.rb/56815
[START] [2019-10-02 01:48:07] logged process
[START] [2019-10-02 01:48:07] create_harvest_instance
[STOP] [2019-10-02 01:48:08] create_harvest_instance
[START] [2019-10-02 01:48:08] fetch_files
[STOP] [2019-10-02 01:48:08] fetch_files
[START] [2019-10-02 01:48:08] validate_each_file
[STOP] [2019-10-02 01:48:08] validate_each_file
[START] [2019-10-02 01:48:08] convert_to_csv
[CMD] [2019-10-02 01:48:08] /usr/bin/sort /app/public/converted_csv/bangladesh_sp_li_refs_14920.csv > /app/public/converted_csv/bangladesh_sp_li_refs_14920.csv_sorted
[CMD] [2019-10-02 01:48:10] /usr/bin/sort /app/public/converted_csv/bangladesh_sp_li_nodes_14921.csv > /app/public/converted_csv/bangladesh_sp_li_nodes_14921.csv_sorted
[CMD] [2019-10-02 01:48:11] /usr/bin/sort /app/public/converted_csv/bangladesh_sp_li_occurrences_14922.csv > /app/public/converted_csv/bangladesh_sp_li_occurrences_14922.csv_sorted
[CMD] [2019-10-02 01:48:13] /usr/bin/sort /app/public/converted_csv/bangladesh_sp_li_measurements_14923.csv > /app/public/converted_csv/bangladesh_sp_li_measurements_14923.csv_sorted
[STOP] [2019-10-02 01:48:14] convert_to_csv
[START] [2019-10-02 01:48:14] calculate_delta
[CMD] [2019-10-02 01:48:14] echo "0a" > /app/public/diff/bangladesh_sp_li_refs_14920.diff
[CMD] [2019-10-02 01:48:16] tail -n +1 /app/public/converted_csv/bangladesh_sp_li_refs_14920.csv >> /app/public/diff/bangladesh_sp_li_refs_14920.diff
[CMD] [2019-10-02 01:48:17] echo "." >> /app/public/diff/bangladesh_sp_li_refs_14920.diff
[CMD] [2019-10-02 01:48:19] echo "0a" > /app/public/diff/bangladesh_sp_li_nodes_14921.diff
[CMD] [2019-10-02 01:48:20] tail -n +1 /app/public/converted_csv/bangladesh_sp_li_nodes_14921.csv >> /app/public/diff/bangladesh_sp_li_nodes_14921.diff
[CMD] [2019-10-02 01:48:22] echo "." >> /app/public/diff/bangladesh_sp_li_nodes_14921.diff
[CMD] [2019-10-02 01:48:23] echo "0a" > /app/public/diff/bangladesh_sp_li_occurrences_14922.diff
[CMD] [2019-10-02 01:48:25] tail -n +1 /app/public/converted_csv/bangladesh_sp_li_occurrences_14922.csv >> /app/public/diff/bangladesh_sp_li_occurrences_14922.diff
[CMD] [2019-10-02 01:48:26] echo "." >> /app/public/diff/bangladesh_sp_li_occurrences_14922.diff
[CMD] [2019-10-02 01:48:28] echo "0a" > /app/public/diff/bangladesh_sp_li_measurements_14923.diff
[CMD] [2019-10-02 01:48:29] tail -n +1 /app/public/converted_csv/bangladesh_sp_li_measurements_14923.csv >> /app/public/diff/bangladesh_sp_li_measurements_14923.diff
[CMD] [2019-10-02 01:48:31] echo "." >> /app/public/diff/bangladesh_sp_li_measurements_14923.diff
[STOP] [2019-10-02 01:48:32] calculate_delta
[START] [2019-10-02 01:48:32] parse_diff_and_store
[INFO] [2019-10-02 01:48:34] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-02 01:48:35] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-02 01:48:38] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-02 01:48:39] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-02 01:48:47] Storing 2 References
[INFO] [2019-10-02 01:48:47] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-02 01:48:47] Average Time: 0.0
[INFO] [2019-10-02 01:48:47] Total Time: 1s
[INFO] [2019-10-02 01:48:47] Storing 2837 ScientificNames
[INFO] [2019-10-02 01:48:47] Processing group of 2837 in 3 groups of 1000
[INFO] [2019-10-02 01:48:48] Average Time: 0.317
[INFO] [2019-10-02 01:48:48] Total Time: 1s
[INFO] [2019-10-02 01:48:48] Storing 2837 Nodes
[INFO] [2019-10-02 01:48:48] Processing group of 2837 in 3 groups of 1000
[INFO] [2019-10-02 01:48:49] Average Time: 0.263
[INFO] [2019-10-02 01:48:49] Total Time: 1s
[INFO] [2019-10-02 01:48:49] Storing 1446 Occurrences
[INFO] [2019-10-02 01:48:49] Processing group of 1446 in 2 groups of 1000
[INFO] [2019-10-02 01:48:49] Average Time: 0.065
[INFO] [2019-10-02 01:48:49] Total Time: 1s
[INFO] [2019-10-02 01:48:49] Storing 3030 TraitsReferences
[INFO] [2019-10-02 01:48:49] Processing group of 3030 in 4 groups of 1000
[INFO] [2019-10-02 01:48:50] Average Time: 0.068
[INFO] [2019-10-02 01:48:50] Total Time: 1s
[INFO] [2019-10-02 01:48:50] Storing 3029 Traits
[INFO] [2019-10-02 01:48:50] Processing group of 3029 in 4 groups of 1000
[INFO] [2019-10-02 01:48:51] Average Time: 0.283
[INFO] [2019-10-02 01:48:51] Total Time: 2s
[INFO] [2019-10-02 01:48:51] Storing 3020 MetaTraits
[INFO] [2019-10-02 01:48:51] Processing group of 3020 in 4 groups of 1000
[INFO] [2019-10-02 01:48:51] Average Time: 0.15
[INFO] [2019-10-02 01:48:51] Total Time: 1s
[STOP] [2019-10-02 01:48:51] parse_diff_and_store
[START] [2019-10-02 01:48:51] resolve_keys
[INFO] [2019-10-02 01:49:06] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-02 01:49:08] traits to occurrences...
[INFO] [2019-10-02 01:49:09] traits to nodes (through occurrences)...
[INFO] [2019-10-02 01:49:09] Traits to sex term...
[INFO] [2019-10-02 01:49:11] Traits to lifestage term...
[INFO] [2019-10-02 01:49:12] MetaTraits to traits...
[INFO] [2019-10-02 01:49:12] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-02 01:49:12] Assocs to occurrences...
[INFO] [2019-10-02 01:49:12] Assocs to nodes...
[INFO] [2019-10-02 01:49:12] Assoc to sex term...
[INFO] [2019-10-02 01:49:12] Assoc to lifestage term...
[STOP] [2019-10-02 01:49:12] resolve_keys
[START] [2019-10-02 01:49:12] hold_for_later_1
[STOP] [2019-10-02 01:49:12] hold_for_later_1
[START] [2019-10-02 01:49:12] hold_for_later_2
[STOP] [2019-10-02 01:49:12] hold_for_later_2
[START] [2019-10-02 01:49:12] resolve_missing_parents
[STOP] [2019-10-02 01:49:17] resolve_missing_parents
[START] [2019-10-02 01:49:17] rebuild_nodes
[START] [2019-10-02 01:49:17] Flattener#flatten
[START] [2019-10-02 01:49:17] Flattener#study_resource
[START] [2019-10-02 01:49:17] Flattener#build_ancestry
[STOP] [2019-10-02 01:49:17] Flattener#build_ancestry
[INFO] [2019-10-02 01:49:17] 2837 ancestry keys
[START] [2019-10-02 01:49:17] build_node_ancestors
[INFO] [2019-10-02 01:49:17] old ancestors deleted.
[STOP] [2019-10-02 01:49:17] build_node_ancestors
[START] [2019-10-02 01:49:18] Flattener#propagate_ancestor_ids
[STOP] [2019-10-02 01:49:18] Flattener#propagate_ancestor_ids
[STOP] [2019-10-02 01:49:18] Flattener#flatten
[STOP] [2019-10-02 01:49:18] rebuild_nodes
[START] [2019-10-02 01:49:18] resolve_missing_media_owners
[STOP] [2019-10-02 01:49:18] resolve_missing_media_owners
[START] [2019-10-02 01:49:18] sanitize_media_verbatims
[STOP] [2019-10-02 01:49:18] sanitize_media_verbatims
[START] [2019-10-02 01:49:18] queue_downloads
[STOP] [2019-10-02 01:49:18] queue_downloads
[START] [2019-10-02 01:49:18] parse_names
[WARN] [2019-10-02 01:49:18] I see 2837 names which still need to be parsed.
[STOP] [2019-10-02 01:49:21] parse_names
[START] [2019-10-02 01:49:21] denormalize_canonical_names_to_nodes
[STOP] [2019-10-02 01:49:21] denormalize_canonical_names_to_nodes
[START] [2019-10-02 01:49:21] match_nodes
[START] [2019-10-02 01:49:21] map_all_nodes_to_pages
[STOP] [2019-10-02 01:51:35] map_all_nodes_to_pages
[INFO] [2019-10-02 01:51:35] 230 Unmatched nodes (of 2837)! That's too many to output. First 10: Oryza hybr (#47690959); Eriophorum comosus (#47690761); Paraderris cuneifolia (#47689715); Caesalpinia enneaphyllum (#47690031); Caesalpinia cucullatum (#47691168); Crotalaria sericea (#47690269); Neohydrocoptus (#47688721); Neohydrocoptus subvittulus (#47688720); Nisaetus limnaeetus (#47690007); Icthyophaga (#47690948)
[START] [2019-10-02 01:51:35] update_nodes
[STOP] [2019-10-02 01:51:36] update_nodes
[STOP] [2019-10-02 01:51:36] match_nodes
[START] [2019-10-02 01:51:36] reindex_search
[STOP] [2019-10-02 01:51:43] reindex_search
[START] [2019-10-02 01:51:43] normalize_units
[STOP] [2019-10-02 01:51:43] normalize_units
[START] [2019-10-02 01:51:43] calculate_statistics
[STOP] [2019-10-02 01:51:43] calculate_statistics
[START] [2019-10-02 01:51:43] complete_harvest_instance
[START] [2019-10-02 01:51:43] overall_tsv_creation
[INFO] [2019-10-02 01:51:43] Processing group of 2837 in 1 batches of 10000
[INFO] [2019-10-02 01:52:38] 1446 Traits (unfiltered)...
[INFO] [2019-10-02 01:52:51] 1446 Traits (filtered)...
[INFO] [2019-10-02 01:52:51] 0 Associations (filtered)...
[INFO] [2019-10-02 01:53:29] 7221 metadata added.
[INFO] [2019-10-02 01:53:29] 0 metadata added.
[INFO] [2019-10-02 01:53:29] Average Time: 82.76
[INFO] [2019-10-02 01:53:29] Total Time: 1m47s
[STOP] [2019-10-02 01:53:29] overall_tsv_creation
[INFO] [2019-10-02 01:53:29] Done. Check your files:
[INFO] [2019-10-02 01:53:30] (2837 lines) /app/public/data/bangladesh_sp_li/publish_nodes.tsv
[INFO] [2019-10-02 01:53:32] (4158 lines) /app/public/data/bangladesh_sp_li/publish_node_ancestors.tsv
[INFO] [2019-10-02 01:53:33] (2837 lines) /app/public/data/bangladesh_sp_li/publish_scientific_names.tsv
[INFO] [2019-10-02 01:53:35] (1447 lines) /app/public/data/bangladesh_sp_li/publish_traits.tsv
[INFO] [2019-10-02 01:53:36] (7222 lines) /app/public/data/bangladesh_sp_li/publish_metadata.tsv
[STOP] [2019-10-02 01:53:36] complete_harvest_instance
[START] [2019-10-02 01:53:36] completed
[STOP] [2019-10-02 01:53:36] completed
[STOP] [2019-10-02 01:53:36] logged process, took 329.24
Latest Process

02 Oct 01:48 EXITED currently processing 0