Harvest for Territory of Heard Island and McDonald Islands Created 16 Oct 12:03

Stage: completed
Fetched: 16 Oct 12:03
Validated: 16 Oct 12:03
Deltas Created 16 Oct 12:03
Units Normalized: 16 Oct 12:05
Ancestry Built: 16 Oct 12:03
Nodes Matched: 16 Oct 12:05
Names Parsed: 16 Oct 12:03
New Models Stored: 16 Oct 12:03
Indexed: 16 Oct 12:05
Completed: 16 Oct 12:06
Time to Harvest: less than a minute
Harvesting Log

(139 lines)
# Logfile created on 2019-10-16 12:03:35 -0400 by logger.rb/56815
[START] [2019-10-16 12:03:35] logged process
[START] [2019-10-16 12:03:35] create_harvest_instance
[STOP] [2019-10-16 12:03:36] create_harvest_instance
[START] [2019-10-16 12:03:36] fetch_files
[STOP] [2019-10-16 12:03:36] fetch_files
[START] [2019-10-16 12:03:36] validate_each_file
[STOP] [2019-10-16 12:03:36] validate_each_file
[START] [2019-10-16 12:03:36] convert_to_csv
[CMD] [2019-10-16 12:03:36] /usr/bin/sort /app/public/converted_csv/thimi_refs_17409.csv > /app/public/converted_csv/thimi_refs_17409.csv_sorted
[CMD] [2019-10-16 12:03:37] /usr/bin/sort /app/public/converted_csv/thimi_nodes_17410.csv > /app/public/converted_csv/thimi_nodes_17410.csv_sorted
[CMD] [2019-10-16 12:03:37] /usr/bin/sort /app/public/converted_csv/thimi_occurrences_17411.csv > /app/public/converted_csv/thimi_occurrences_17411.csv_sorted
[CMD] [2019-10-16 12:03:37] /usr/bin/sort /app/public/converted_csv/thimi_measurements_17412.csv > /app/public/converted_csv/thimi_measurements_17412.csv_sorted
[STOP] [2019-10-16 12:03:37] convert_to_csv
[START] [2019-10-16 12:03:37] calculate_delta
[CMD] [2019-10-16 12:03:37] echo "0a" > /app/public/diff/thimi_refs_17409.diff
[CMD] [2019-10-16 12:03:38] tail -n +1 /app/public/converted_csv/thimi_refs_17409.csv >> /app/public/diff/thimi_refs_17409.diff
[CMD] [2019-10-16 12:03:38] echo "." >> /app/public/diff/thimi_refs_17409.diff
[CMD] [2019-10-16 12:03:38] echo "0a" > /app/public/diff/thimi_nodes_17410.diff
[CMD] [2019-10-16 12:03:39] tail -n +1 /app/public/converted_csv/thimi_nodes_17410.csv >> /app/public/diff/thimi_nodes_17410.diff
[CMD] [2019-10-16 12:03:39] echo "." >> /app/public/diff/thimi_nodes_17410.diff
[CMD] [2019-10-16 12:03:39] echo "0a" > /app/public/diff/thimi_occurrences_17411.diff
[CMD] [2019-10-16 12:03:39] tail -n +1 /app/public/converted_csv/thimi_occurrences_17411.csv >> /app/public/diff/thimi_occurrences_17411.diff
[CMD] [2019-10-16 12:03:40] echo "." >> /app/public/diff/thimi_occurrences_17411.diff
[CMD] [2019-10-16 12:03:40] echo "0a" > /app/public/diff/thimi_measurements_17412.diff
[CMD] [2019-10-16 12:03:40] tail -n +1 /app/public/converted_csv/thimi_measurements_17412.csv >> /app/public/diff/thimi_measurements_17412.diff
[CMD] [2019-10-16 12:03:40] echo "." >> /app/public/diff/thimi_measurements_17412.diff
[STOP] [2019-10-16 12:03:41] calculate_delta
[START] [2019-10-16 12:03:41] parse_diff_and_store
[INFO] [2019-10-16 12:03:41] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-16 12:03:41] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-16 12:03:42] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-16 12:03:42] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-16 12:03:44] Storing 2 References
[INFO] [2019-10-16 12:03:44] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-16 12:03:44] Average Time: 0.0
[INFO] [2019-10-16 12:03:44] Total Time: 1s
[INFO] [2019-10-16 12:03:44] Storing 692 ScientificNames
[INFO] [2019-10-16 12:03:44] Processing group of 692 in 1 groups of 1000
[INFO] [2019-10-16 12:03:44] Average Time: 0.38
[INFO] [2019-10-16 12:03:44] Total Time: 1s
[INFO] [2019-10-16 12:03:44] Storing 692 Nodes
[INFO] [2019-10-16 12:03:44] Processing group of 692 in 1 groups of 1000
[INFO] [2019-10-16 12:03:44] Average Time: 0.24
[INFO] [2019-10-16 12:03:44] Total Time: 1s
[INFO] [2019-10-16 12:03:44] Storing 239 Occurrences
[INFO] [2019-10-16 12:03:44] Processing group of 239 in 1 groups of 1000
[INFO] [2019-10-16 12:03:44] Average Time: 0.04
[INFO] [2019-10-16 12:03:44] Total Time: 1s
[INFO] [2019-10-16 12:03:44] Storing 478 TraitsReferences
[INFO] [2019-10-16 12:03:44] Processing group of 478 in 1 groups of 1000
[INFO] [2019-10-16 12:03:45] Average Time: 0.1
[INFO] [2019-10-16 12:03:45] Total Time: 1s
[INFO] [2019-10-16 12:03:45] Storing 478 Traits
[INFO] [2019-10-16 12:03:45] Processing group of 478 in 1 groups of 1000
[INFO] [2019-10-16 12:03:45] Average Time: 0.25
[INFO] [2019-10-16 12:03:45] Total Time: 1s
[INFO] [2019-10-16 12:03:45] Storing 477 MetaTraits
[INFO] [2019-10-16 12:03:45] Processing group of 477 in 1 groups of 1000
[INFO] [2019-10-16 12:03:45] Average Time: 0.08
[INFO] [2019-10-16 12:03:45] Total Time: 1s
[STOP] [2019-10-16 12:03:45] parse_diff_and_store
[START] [2019-10-16 12:03:45] resolve_keys
[INFO] [2019-10-16 12:03:50] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-16 12:03:50] traits to occurrences...
[INFO] [2019-10-16 12:03:50] traits to nodes (through occurrences)...
[INFO] [2019-10-16 12:03:51] Traits to sex term...
[INFO] [2019-10-16 12:03:51] Traits to lifestage term...
[INFO] [2019-10-16 12:03:51] MetaTraits to traits...
[INFO] [2019-10-16 12:03:51] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-16 12:03:51] Assocs to occurrences...
[INFO] [2019-10-16 12:03:51] Assocs to nodes...
[INFO] [2019-10-16 12:03:51] Assoc to sex term...
[INFO] [2019-10-16 12:03:51] Assoc to lifestage term...
[STOP] [2019-10-16 12:03:51] resolve_keys
[START] [2019-10-16 12:03:51] hold_for_later_1
[STOP] [2019-10-16 12:03:51] hold_for_later_1
[START] [2019-10-16 12:03:51] hold_for_later_2
[STOP] [2019-10-16 12:03:51] hold_for_later_2
[START] [2019-10-16 12:03:51] resolve_missing_parents
[STOP] [2019-10-16 12:03:52] resolve_missing_parents
[START] [2019-10-16 12:03:52] rebuild_nodes
[START] [2019-10-16 12:03:52] Flattener#flatten
[START] [2019-10-16 12:03:52] Flattener#study_resource
[START] [2019-10-16 12:03:52] Flattener#build_ancestry
[STOP] [2019-10-16 12:03:52] Flattener#build_ancestry
[INFO] [2019-10-16 12:03:52] 692 ancestry keys
[START] [2019-10-16 12:03:52] build_node_ancestors
[INFO] [2019-10-16 12:03:52] old ancestors deleted.
[STOP] [2019-10-16 12:03:52] build_node_ancestors
[START] [2019-10-16 12:03:52] Flattener#propagate_ancestor_ids
[STOP] [2019-10-16 12:03:52] Flattener#propagate_ancestor_ids
[STOP] [2019-10-16 12:03:52] Flattener#flatten
[STOP] [2019-10-16 12:03:52] rebuild_nodes
[START] [2019-10-16 12:03:52] resolve_missing_media_owners
[STOP] [2019-10-16 12:03:52] resolve_missing_media_owners
[START] [2019-10-16 12:03:52] sanitize_media_verbatims
[STOP] [2019-10-16 12:03:52] sanitize_media_verbatims
[START] [2019-10-16 12:03:52] queue_downloads
[STOP] [2019-10-16 12:03:52] queue_downloads
[START] [2019-10-16 12:03:52] parse_names
[WARN] [2019-10-16 12:03:52] I see 692 names which still need to be parsed.
[STOP] [2019-10-16 12:03:54] parse_names
[START] [2019-10-16 12:03:54] denormalize_canonical_names_to_nodes
[STOP] [2019-10-16 12:03:54] denormalize_canonical_names_to_nodes
[START] [2019-10-16 12:03:54] match_nodes
[START] [2019-10-16 12:03:54] map_all_nodes_to_pages
[STOP] [2019-10-16 12:05:01] map_all_nodes_to_pages
[INFO] [2019-10-16 12:05:01] 30 Unmatched nodes (of 692)! That's too many to output. First 10: Pygoscelis antarctica (#52367417); Pelecanoididae (#52366863); Enterogona (#52367368); Scorpaeniformes (#52367488); Maxillopoda (#52367226); Arcoida (#52367081); Adoncholaimus crassicaudatus (#52367054); Spionida (#52367336); Sabellida (#52367484); Harpidium uncinatus (#52366966)
[START] [2019-10-16 12:05:01] update_nodes
[STOP] [2019-10-16 12:05:02] update_nodes
[STOP] [2019-10-16 12:05:02] match_nodes
[START] [2019-10-16 12:05:02] reindex_search
[STOP] [2019-10-16 12:05:03] reindex_search
[START] [2019-10-16 12:05:03] normalize_units
[STOP] [2019-10-16 12:05:03] normalize_units
[START] [2019-10-16 12:05:03] calculate_statistics
[STOP] [2019-10-16 12:05:03] calculate_statistics
[START] [2019-10-16 12:05:03] complete_harvest_instance
[START] [2019-10-16 12:05:03] overall_tsv_creation
[INFO] [2019-10-16 12:05:03] Processing group of 692 in 1 batches of 10000
[INFO] [2019-10-16 12:05:50] 239 Traits (unfiltered)...
[INFO] [2019-10-16 12:06:03] 239 Traits (filtered)...
[INFO] [2019-10-16 12:06:03] 0 Associations (filtered)...
[INFO] [2019-10-16 12:06:41] 1194 metadata added.
[INFO] [2019-10-16 12:06:41] 0 metadata added.
[INFO] [2019-10-16 12:06:41] Average Time: 76.81
[INFO] [2019-10-16 12:06:41] Total Time: 1m38s
[STOP] [2019-10-16 12:06:41] overall_tsv_creation
[INFO] [2019-10-16 12:06:41] Done. Check your files:
[INFO] [2019-10-16 12:06:41] (692 lines) /app/public/data/thimi/publish_nodes.tsv
[INFO] [2019-10-16 12:06:42] (3160 lines) /app/public/data/thimi/publish_node_ancestors.tsv
[INFO] [2019-10-16 12:06:42] (692 lines) /app/public/data/thimi/publish_scientific_names.tsv
[INFO] [2019-10-16 12:06:42] (240 lines) /app/public/data/thimi/publish_traits.tsv
[INFO] [2019-10-16 12:06:43] (1195 lines) /app/public/data/thimi/publish_metadata.tsv
[STOP] [2019-10-16 12:06:43] complete_harvest_instance
[START] [2019-10-16 12:06:43] completed
[STOP] [2019-10-16 12:06:43] completed
[STOP] [2019-10-16 12:06:43] logged process, took 187.15
Latest Process

16 Oct 12:03 EXITED currently processing 0