Harvest for Pitcairn Henderson Ducie and Oeno Islands Created 15 Oct 12:19

Stage: completed
Fetched: 15 Oct 12:19
Validated: 15 Oct 12:19
Deltas Created 15 Oct 12:19
Units Normalized: 15 Oct 12:20
Ancestry Built: 15 Oct 12:19
Nodes Matched: 15 Oct 12:20
Names Parsed: 15 Oct 12:19
New Models Stored: 15 Oct 12:19
Indexed: 15 Oct 12:20
Completed: 15 Oct 12:21
Time to Harvest: less than a minute
Harvesting Log

(139 lines)
# Logfile created on 2019-10-15 12:19:29 -0400 by logger.rb/56815
[START] [2019-10-15 12:19:29] logged process
[START] [2019-10-15 12:19:29] create_harvest_instance
[STOP] [2019-10-15 12:19:30] create_harvest_instance
[START] [2019-10-15 12:19:30] fetch_files
[STOP] [2019-10-15 12:19:30] fetch_files
[START] [2019-10-15 12:19:30] validate_each_file
[STOP] [2019-10-15 12:19:30] validate_each_file
[START] [2019-10-15 12:19:30] convert_to_csv
[CMD] [2019-10-15 12:19:30] /usr/bin/sort /app/public/converted_csv/phdoi_refs_16925.csv > /app/public/converted_csv/phdoi_refs_16925.csv_sorted
[CMD] [2019-10-15 12:19:30] /usr/bin/sort /app/public/converted_csv/phdoi_nodes_16926.csv > /app/public/converted_csv/phdoi_nodes_16926.csv_sorted
[CMD] [2019-10-15 12:19:30] /usr/bin/sort /app/public/converted_csv/phdoi_occurrences_16927.csv > /app/public/converted_csv/phdoi_occurrences_16927.csv_sorted
[CMD] [2019-10-15 12:19:31] /usr/bin/sort /app/public/converted_csv/phdoi_measurements_16928.csv > /app/public/converted_csv/phdoi_measurements_16928.csv_sorted
[STOP] [2019-10-15 12:19:31] convert_to_csv
[START] [2019-10-15 12:19:31] calculate_delta
[CMD] [2019-10-15 12:19:31] echo "0a" > /app/public/diff/phdoi_refs_16925.diff
[CMD] [2019-10-15 12:19:31] tail -n +1 /app/public/converted_csv/phdoi_refs_16925.csv >> /app/public/diff/phdoi_refs_16925.diff
[CMD] [2019-10-15 12:19:31] echo "." >> /app/public/diff/phdoi_refs_16925.diff
[CMD] [2019-10-15 12:19:32] echo "0a" > /app/public/diff/phdoi_nodes_16926.diff
[CMD] [2019-10-15 12:19:32] tail -n +1 /app/public/converted_csv/phdoi_nodes_16926.csv >> /app/public/diff/phdoi_nodes_16926.diff
[CMD] [2019-10-15 12:19:32] echo "." >> /app/public/diff/phdoi_nodes_16926.diff
[CMD] [2019-10-15 12:19:33] echo "0a" > /app/public/diff/phdoi_occurrences_16927.diff
[CMD] [2019-10-15 12:19:33] tail -n +1 /app/public/converted_csv/phdoi_occurrences_16927.csv >> /app/public/diff/phdoi_occurrences_16927.diff
[CMD] [2019-10-15 12:19:33] echo "." >> /app/public/diff/phdoi_occurrences_16927.diff
[CMD] [2019-10-15 12:19:33] echo "0a" > /app/public/diff/phdoi_measurements_16928.diff
[CMD] [2019-10-15 12:19:34] tail -n +1 /app/public/converted_csv/phdoi_measurements_16928.csv >> /app/public/diff/phdoi_measurements_16928.diff
[CMD] [2019-10-15 12:19:34] echo "." >> /app/public/diff/phdoi_measurements_16928.diff
[STOP] [2019-10-15 12:19:34] calculate_delta
[START] [2019-10-15 12:19:34] parse_diff_and_store
[INFO] [2019-10-15 12:19:35] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-15 12:19:35] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-15 12:19:35] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-15 12:19:36] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-15 12:19:36] Storing 2 References
[INFO] [2019-10-15 12:19:36] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-15 12:19:36] Average Time: 0.0
[INFO] [2019-10-15 12:19:36] Total Time: 1s
[INFO] [2019-10-15 12:19:36] Storing 333 ScientificNames
[INFO] [2019-10-15 12:19:36] Processing group of 333 in 1 groups of 1000
[INFO] [2019-10-15 12:19:37] Average Time: 0.2
[INFO] [2019-10-15 12:19:37] Total Time: 1s
[INFO] [2019-10-15 12:19:37] Storing 333 Nodes
[INFO] [2019-10-15 12:19:37] Processing group of 333 in 1 groups of 1000
[INFO] [2019-10-15 12:19:37] Average Time: 0.09
[INFO] [2019-10-15 12:19:37] Total Time: 1s
[INFO] [2019-10-15 12:19:37] Storing 109 Occurrences
[INFO] [2019-10-15 12:19:37] Processing group of 109 in 1 groups of 1000
[INFO] [2019-10-15 12:19:37] Average Time: 0.02
[INFO] [2019-10-15 12:19:37] Total Time: 1s
[INFO] [2019-10-15 12:19:37] Storing 218 TraitsReferences
[INFO] [2019-10-15 12:19:37] Processing group of 218 in 1 groups of 1000
[INFO] [2019-10-15 12:19:37] Average Time: 0.1
[INFO] [2019-10-15 12:19:37] Total Time: 1s
[INFO] [2019-10-15 12:19:37] Storing 218 Traits
[INFO] [2019-10-15 12:19:37] Processing group of 218 in 1 groups of 1000
[INFO] [2019-10-15 12:19:37] Average Time: 0.18
[INFO] [2019-10-15 12:19:37] Total Time: 1s
[INFO] [2019-10-15 12:19:37] Storing 218 MetaTraits
[INFO] [2019-10-15 12:19:37] Processing group of 218 in 1 groups of 1000
[INFO] [2019-10-15 12:19:37] Average Time: 0.06
[INFO] [2019-10-15 12:19:37] Total Time: 1s
[STOP] [2019-10-15 12:19:37] parse_diff_and_store
[START] [2019-10-15 12:19:37] resolve_keys
[INFO] [2019-10-15 12:19:41] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-15 12:19:41] traits to occurrences...
[INFO] [2019-10-15 12:19:41] traits to nodes (through occurrences)...
[INFO] [2019-10-15 12:19:41] Traits to sex term...
[INFO] [2019-10-15 12:19:41] Traits to lifestage term...
[INFO] [2019-10-15 12:19:41] MetaTraits to traits...
[INFO] [2019-10-15 12:19:42] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-15 12:19:42] Assocs to occurrences...
[INFO] [2019-10-15 12:19:42] Assocs to nodes...
[INFO] [2019-10-15 12:19:42] Assoc to sex term...
[INFO] [2019-10-15 12:19:42] Assoc to lifestage term...
[STOP] [2019-10-15 12:19:42] resolve_keys
[START] [2019-10-15 12:19:42] hold_for_later_1
[STOP] [2019-10-15 12:19:42] hold_for_later_1
[START] [2019-10-15 12:19:42] hold_for_later_2
[STOP] [2019-10-15 12:19:42] hold_for_later_2
[START] [2019-10-15 12:19:42] resolve_missing_parents
[STOP] [2019-10-15 12:19:42] resolve_missing_parents
[START] [2019-10-15 12:19:42] rebuild_nodes
[START] [2019-10-15 12:19:42] Flattener#flatten
[START] [2019-10-15 12:19:42] Flattener#study_resource
[START] [2019-10-15 12:19:42] Flattener#build_ancestry
[STOP] [2019-10-15 12:19:42] Flattener#build_ancestry
[INFO] [2019-10-15 12:19:42] 333 ancestry keys
[START] [2019-10-15 12:19:42] build_node_ancestors
[INFO] [2019-10-15 12:19:42] old ancestors deleted.
[STOP] [2019-10-15 12:19:42] build_node_ancestors
[START] [2019-10-15 12:19:42] Flattener#propagate_ancestor_ids
[STOP] [2019-10-15 12:19:42] Flattener#propagate_ancestor_ids
[STOP] [2019-10-15 12:19:42] Flattener#flatten
[STOP] [2019-10-15 12:19:42] rebuild_nodes
[START] [2019-10-15 12:19:42] resolve_missing_media_owners
[STOP] [2019-10-15 12:19:42] resolve_missing_media_owners
[START] [2019-10-15 12:19:42] sanitize_media_verbatims
[STOP] [2019-10-15 12:19:42] sanitize_media_verbatims
[START] [2019-10-15 12:19:42] queue_downloads
[STOP] [2019-10-15 12:19:42] queue_downloads
[START] [2019-10-15 12:19:42] parse_names
[WARN] [2019-10-15 12:19:42] I see 333 names which still need to be parsed.
[STOP] [2019-10-15 12:19:44] parse_names
[START] [2019-10-15 12:19:44] denormalize_canonical_names_to_nodes
[STOP] [2019-10-15 12:19:44] denormalize_canonical_names_to_nodes
[START] [2019-10-15 12:19:44] match_nodes
[START] [2019-10-15 12:19:44] map_all_nodes_to_pages
[STOP] [2019-10-15 12:20:15] map_all_nodes_to_pages
[INFO] [2019-10-15 12:20:15] 15 Unmatched nodes (of 333)! That's too many to output. First 10: Philonesia (#51625824); Philonesia pitcairnensis (#51625823); Philonesia filiceti (#51625836); Neotaenioglossa (#51626005); Veneroida (#51626020); Tridacnidae (#51626019); Ostreoida (#51626098); Aspidochirotida (#51625878); Ophiomyxa dubia (#51626043); Hadromerida (#51625852)
[START] [2019-10-15 12:20:15] update_nodes
[STOP] [2019-10-15 12:20:15] update_nodes
[STOP] [2019-10-15 12:20:15] match_nodes
[START] [2019-10-15 12:20:15] reindex_search
[STOP] [2019-10-15 12:20:16] reindex_search
[START] [2019-10-15 12:20:16] normalize_units
[STOP] [2019-10-15 12:20:16] normalize_units
[START] [2019-10-15 12:20:16] calculate_statistics
[STOP] [2019-10-15 12:20:16] calculate_statistics
[START] [2019-10-15 12:20:16] complete_harvest_instance
[START] [2019-10-15 12:20:16] overall_tsv_creation
[INFO] [2019-10-15 12:20:16] Processing group of 333 in 1 batches of 10000
[INFO] [2019-10-15 12:21:02] 109 Traits (unfiltered)...
[INFO] [2019-10-15 12:21:15] 109 Traits (filtered)...
[INFO] [2019-10-15 12:21:15] 0 Associations (filtered)...
[INFO] [2019-10-15 12:21:53] 545 metadata added.
[INFO] [2019-10-15 12:21:53] 0 metadata added.
[INFO] [2019-10-15 12:21:53] Average Time: 75.75
[INFO] [2019-10-15 12:21:53] Total Time: 1m37s
[STOP] [2019-10-15 12:21:53] overall_tsv_creation
[INFO] [2019-10-15 12:21:53] Done. Check your files:
[INFO] [2019-10-15 12:21:53] (333 lines) /app/public/data/phdoi/publish_nodes.tsv
[INFO] [2019-10-15 12:21:53] (1473 lines) /app/public/data/phdoi/publish_node_ancestors.tsv
[INFO] [2019-10-15 12:21:54] (333 lines) /app/public/data/phdoi/publish_scientific_names.tsv
[INFO] [2019-10-15 12:21:54] (110 lines) /app/public/data/phdoi/publish_traits.tsv
[INFO] [2019-10-15 12:21:54] (546 lines) /app/public/data/phdoi/publish_metadata.tsv
[STOP] [2019-10-15 12:21:54] complete_harvest_instance
[START] [2019-10-15 12:21:54] completed
[STOP] [2019-10-15 12:21:54] completed
[STOP] [2019-10-15 12:21:54] logged process, took 145.13
Latest Process

15 Oct 12:19 EXITED currently processing 0