Stage:
completed
Fetched:
16 Oct 01:36
Validated:
16 Oct 01:36
Deltas Created
16 Oct 01:36
Units Normalized:
16 Oct 01:39
Ancestry Built:
16 Oct 01:37
Nodes Matched:
16 Oct 01:39
Names Parsed:
16 Oct 01:37
New Models Stored:
16 Oct 01:36
Indexed:
16 Oct 01:39
Completed:
16 Oct 01:41
Time to Harvest:
less than a minute
Harvesting Log
(139 lines)
# Logfile created on 2019-10-16 01:36:37 -0400 by logger.rb/56815
[START] [2019-10-16 01:36:37] logged process
[START] [2019-10-16 01:36:37] create_harvest_instance
[STOP] [2019-10-16 01:36:37] create_harvest_instance
[START] [2019-10-16 01:36:37] fetch_files
[STOP] [2019-10-16 01:36:37] fetch_files
[START] [2019-10-16 01:36:37] validate_each_file
[STOP] [2019-10-16 01:36:37] validate_each_file
[START] [2019-10-16 01:36:37] convert_to_csv
[CMD] [2019-10-16 01:36:37] /usr/bin/sort /app/public/converted_csv/sgssi_refs_17285.csv > /app/public/converted_csv/sgssi_refs_17285.csv_sorted
[CMD] [2019-10-16 01:36:38] /usr/bin/sort /app/public/converted_csv/sgssi_nodes_17286.csv > /app/public/converted_csv/sgssi_nodes_17286.csv_sorted
[CMD] [2019-10-16 01:36:38] /usr/bin/sort /app/public/converted_csv/sgssi_occurrences_17287.csv > /app/public/converted_csv/sgssi_occurrences_17287.csv_sorted
[CMD] [2019-10-16 01:36:38] /usr/bin/sort /app/public/converted_csv/sgssi_measurements_17288.csv > /app/public/converted_csv/sgssi_measurements_17288.csv_sorted
[STOP] [2019-10-16 01:36:39] convert_to_csv
[START] [2019-10-16 01:36:39] calculate_delta
[CMD] [2019-10-16 01:36:39] echo "0a" > /app/public/diff/sgssi_refs_17285.diff
[CMD] [2019-10-16 01:36:39] tail -n +1 /app/public/converted_csv/sgssi_refs_17285.csv >> /app/public/diff/sgssi_refs_17285.diff
[CMD] [2019-10-16 01:36:39] echo "." >> /app/public/diff/sgssi_refs_17285.diff
[CMD] [2019-10-16 01:36:39] echo "0a" > /app/public/diff/sgssi_nodes_17286.diff
[CMD] [2019-10-16 01:36:40] tail -n +1 /app/public/converted_csv/sgssi_nodes_17286.csv >> /app/public/diff/sgssi_nodes_17286.diff
[CMD] [2019-10-16 01:36:40] echo "." >> /app/public/diff/sgssi_nodes_17286.diff
[CMD] [2019-10-16 01:36:40] echo "0a" > /app/public/diff/sgssi_occurrences_17287.diff
[CMD] [2019-10-16 01:36:41] tail -n +1 /app/public/converted_csv/sgssi_occurrences_17287.csv >> /app/public/diff/sgssi_occurrences_17287.diff
[CMD] [2019-10-16 01:36:41] echo "." >> /app/public/diff/sgssi_occurrences_17287.diff
[CMD] [2019-10-16 01:36:41] echo "0a" > /app/public/diff/sgssi_measurements_17288.diff
[CMD] [2019-10-16 01:36:41] tail -n +1 /app/public/converted_csv/sgssi_measurements_17288.csv >> /app/public/diff/sgssi_measurements_17288.diff
[CMD] [2019-10-16 01:36:42] echo "." >> /app/public/diff/sgssi_measurements_17288.diff
[STOP] [2019-10-16 01:36:42] calculate_delta
[START] [2019-10-16 01:36:42] parse_diff_and_store
[INFO] [2019-10-16 01:36:42] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-16 01:36:43] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-16 01:36:44] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-16 01:36:44] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-16 01:36:52] Storing 2 References
[INFO] [2019-10-16 01:36:52] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-16 01:36:52] Average Time: 0.0
[INFO] [2019-10-16 01:36:52] Total Time: 1s
[INFO] [2019-10-16 01:36:52] Storing 2802 ScientificNames
[INFO] [2019-10-16 01:36:52] Processing group of 2802 in 3 groups of 1000
[INFO] [2019-10-16 01:36:53] Average Time: 0.34
[INFO] [2019-10-16 01:36:53] Total Time: 2s
[INFO] [2019-10-16 01:36:53] Storing 2802 Nodes
[INFO] [2019-10-16 01:36:53] Processing group of 2802 in 3 groups of 1000
[INFO] [2019-10-16 01:36:54] Average Time: 0.357
[INFO] [2019-10-16 01:36:54] Total Time: 2s
[INFO] [2019-10-16 01:36:54] Storing 964 Occurrences
[INFO] [2019-10-16 01:36:54] Processing group of 964 in 1 groups of 1000
[INFO] [2019-10-16 01:36:54] Average Time: 0.13
[INFO] [2019-10-16 01:36:54] Total Time: 1s
[INFO] [2019-10-16 01:36:54] Storing 2730 TraitsReferences
[INFO] [2019-10-16 01:36:54] Processing group of 2730 in 3 groups of 1000
[INFO] [2019-10-16 01:36:55] Average Time: 0.097
[INFO] [2019-10-16 01:36:55] Total Time: 1s
[INFO] [2019-10-16 01:36:55] Storing 2729 Traits
[INFO] [2019-10-16 01:36:55] Processing group of 2729 in 3 groups of 1000
[INFO] [2019-10-16 01:36:56] Average Time: 0.387
[INFO] [2019-10-16 01:36:56] Total Time: 2s
[INFO] [2019-10-16 01:36:56] Storing 2727 MetaTraits
[INFO] [2019-10-16 01:36:56] Processing group of 2727 in 3 groups of 1000
[INFO] [2019-10-16 01:36:56] Average Time: 0.117
[INFO] [2019-10-16 01:36:56] Total Time: 1s
[STOP] [2019-10-16 01:36:56] parse_diff_and_store
[START] [2019-10-16 01:36:56] resolve_keys
[INFO] [2019-10-16 01:37:09] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-16 01:37:10] traits to occurrences...
[INFO] [2019-10-16 01:37:11] traits to nodes (through occurrences)...
[INFO] [2019-10-16 01:37:11] Traits to sex term...
[INFO] [2019-10-16 01:37:12] Traits to lifestage term...
[INFO] [2019-10-16 01:37:13] MetaTraits to traits...
[INFO] [2019-10-16 01:37:13] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-16 01:37:13] Assocs to occurrences...
[INFO] [2019-10-16 01:37:13] Assocs to nodes...
[INFO] [2019-10-16 01:37:13] Assoc to sex term...
[INFO] [2019-10-16 01:37:13] Assoc to lifestage term...
[STOP] [2019-10-16 01:37:13] resolve_keys
[START] [2019-10-16 01:37:13] hold_for_later_1
[STOP] [2019-10-16 01:37:13] hold_for_later_1
[START] [2019-10-16 01:37:13] hold_for_later_2
[STOP] [2019-10-16 01:37:13] hold_for_later_2
[START] [2019-10-16 01:37:13] resolve_missing_parents
[STOP] [2019-10-16 01:37:18] resolve_missing_parents
[START] [2019-10-16 01:37:18] rebuild_nodes
[START] [2019-10-16 01:37:18] Flattener#flatten
[START] [2019-10-16 01:37:18] Flattener#study_resource
[START] [2019-10-16 01:37:18] Flattener#build_ancestry
[STOP] [2019-10-16 01:37:20] Flattener#build_ancestry
[INFO] [2019-10-16 01:37:20] 2802 ancestry keys
[START] [2019-10-16 01:37:20] build_node_ancestors
[INFO] [2019-10-16 01:37:20] old ancestors deleted.
[STOP] [2019-10-16 01:37:20] build_node_ancestors
[START] [2019-10-16 01:37:21] Flattener#propagate_ancestor_ids
[STOP] [2019-10-16 01:37:21] Flattener#propagate_ancestor_ids
[STOP] [2019-10-16 01:37:21] Flattener#flatten
[STOP] [2019-10-16 01:37:21] rebuild_nodes
[START] [2019-10-16 01:37:21] resolve_missing_media_owners
[STOP] [2019-10-16 01:37:21] resolve_missing_media_owners
[START] [2019-10-16 01:37:21] sanitize_media_verbatims
[STOP] [2019-10-16 01:37:21] sanitize_media_verbatims
[START] [2019-10-16 01:37:21] queue_downloads
[STOP] [2019-10-16 01:37:21] queue_downloads
[START] [2019-10-16 01:37:21] parse_names
[WARN] [2019-10-16 01:37:21] I see 2802 names which still need to be parsed.
[STOP] [2019-10-16 01:37:24] parse_names
[START] [2019-10-16 01:37:24] denormalize_canonical_names_to_nodes
[STOP] [2019-10-16 01:37:24] denormalize_canonical_names_to_nodes
[START] [2019-10-16 01:37:24] match_nodes
[START] [2019-10-16 01:37:24] map_all_nodes_to_pages
[STOP] [2019-10-16 01:39:53] map_all_nodes_to_pages
[INFO] [2019-10-16 01:39:53] 224 Unmatched nodes (of 2802)! That's too many to output. First 10: Eudyptes crestatus (#52032837); Puffinus griseus (#52032072); Pelecanoididae (#52031322); Pelecanoides urinator (#52032188); Anas discors (#52033763); Phalacrocorax georgianus (#52031205); Phalacrocorax atriceps (#52033312); Phalacrocorax bransfieldensis (#52033601); Racomitrium willii (#52031459); Racomitrium striatipila (#52033716)
[START] [2019-10-16 01:39:53] update_nodes
[STOP] [2019-10-16 01:39:54] update_nodes
[STOP] [2019-10-16 01:39:54] match_nodes
[START] [2019-10-16 01:39:54] reindex_search
[STOP] [2019-10-16 01:39:59] reindex_search
[START] [2019-10-16 01:39:59] normalize_units
[STOP] [2019-10-16 01:39:59] normalize_units
[START] [2019-10-16 01:39:59] calculate_statistics
[STOP] [2019-10-16 01:39:59] calculate_statistics
[START] [2019-10-16 01:39:59] complete_harvest_instance
[START] [2019-10-16 01:39:59] overall_tsv_creation
[INFO] [2019-10-16 01:39:59] Processing group of 2802 in 1 batches of 10000
[INFO] [2019-10-16 01:40:56] 964 Traits (unfiltered)...
[INFO] [2019-10-16 01:41:10] 964 Traits (filtered)...
[INFO] [2019-10-16 01:41:10] 0 Associations (filtered)...
[INFO] [2019-10-16 01:41:48] 4818 metadata added.
[INFO] [2019-10-16 01:41:48] 0 metadata added.
[INFO] [2019-10-16 01:41:48] Average Time: 86.52
[INFO] [2019-10-16 01:41:48] Total Time: 1m49s
[STOP] [2019-10-16 01:41:48] overall_tsv_creation
[INFO] [2019-10-16 01:41:48] Done. Check your files:
[INFO] [2019-10-16 01:41:49] (2802 lines) /app/public/data/sgssi/publish_nodes.tsv
[INFO] [2019-10-16 01:41:49] (8626 lines) /app/public/data/sgssi/publish_node_ancestors.tsv
[INFO] [2019-10-16 01:41:49] (2802 lines) /app/public/data/sgssi/publish_scientific_names.tsv
[INFO] [2019-10-16 01:41:50] (965 lines) /app/public/data/sgssi/publish_traits.tsv
[INFO] [2019-10-16 01:41:50] (4819 lines) /app/public/data/sgssi/publish_metadata.tsv
[STOP] [2019-10-16 01:41:50] complete_harvest_instance
[START] [2019-10-16 01:41:50] completed
[STOP] [2019-10-16 01:41:50] completed
[STOP] [2019-10-16 01:41:50] logged process, took 313.61
Latest Process