Harvest for Mariana Islands Species List Created 14 Oct 05:01

Stage: completed
Fetched: 14 Oct 05:01
Validated: 14 Oct 05:01
Deltas Created 14 Oct 05:01
Units Normalized: 14 Oct 05:05
Ancestry Built: 14 Oct 05:02
Nodes Matched: 14 Oct 05:05
Names Parsed: 14 Oct 05:02
New Models Stored: 14 Oct 05:01
Indexed: 14 Oct 05:05
Completed: 14 Oct 05:07
Time to Harvest: less than a minute
Harvesting Log

(139 lines)
# Logfile created on 2019-10-14 05:01:35 -0400 by logger.rb/56815
[START] [2019-10-14 05:01:35] logged process
[START] [2019-10-14 05:01:35] create_harvest_instance
[STOP] [2019-10-14 05:01:35] create_harvest_instance
[START] [2019-10-14 05:01:35] fetch_files
[STOP] [2019-10-14 05:01:35] fetch_files
[START] [2019-10-14 05:01:35] validate_each_file
[STOP] [2019-10-14 05:01:36] validate_each_file
[START] [2019-10-14 05:01:36] convert_to_csv
[CMD] [2019-10-14 05:01:36] /usr/bin/sort /app/public/converted_csv/mariana_islands__refs_16531.csv > /app/public/converted_csv/mariana_islands__refs_16531.csv_sorted
[CMD] [2019-10-14 05:01:36] /usr/bin/sort /app/public/converted_csv/mariana_islands__nodes_16532.csv > /app/public/converted_csv/mariana_islands__nodes_16532.csv_sorted
[CMD] [2019-10-14 05:01:36] /usr/bin/sort /app/public/converted_csv/mariana_islands__occurrences_16533.csv > /app/public/converted_csv/mariana_islands__occurrences_16533.csv_sorted
[CMD] [2019-10-14 05:01:36] /usr/bin/sort /app/public/converted_csv/mariana_islands__measurements_16534.csv > /app/public/converted_csv/mariana_islands__measurements_16534.csv_sorted
[STOP] [2019-10-14 05:01:36] convert_to_csv
[START] [2019-10-14 05:01:36] calculate_delta
[CMD] [2019-10-14 05:01:36] echo "0a" > /app/public/diff/mariana_islands__refs_16531.diff
[CMD] [2019-10-14 05:01:36] tail -n +1 /app/public/converted_csv/mariana_islands__refs_16531.csv >> /app/public/diff/mariana_islands__refs_16531.diff
[CMD] [2019-10-14 05:01:36] echo "." >> /app/public/diff/mariana_islands__refs_16531.diff
[CMD] [2019-10-14 05:01:36] echo "0a" > /app/public/diff/mariana_islands__nodes_16532.diff
[CMD] [2019-10-14 05:01:36] tail -n +1 /app/public/converted_csv/mariana_islands__nodes_16532.csv >> /app/public/diff/mariana_islands__nodes_16532.diff
[CMD] [2019-10-14 05:01:37] echo "." >> /app/public/diff/mariana_islands__nodes_16532.diff
[CMD] [2019-10-14 05:01:37] echo "0a" > /app/public/diff/mariana_islands__occurrences_16533.diff
[CMD] [2019-10-14 05:01:37] tail -n +1 /app/public/converted_csv/mariana_islands__occurrences_16533.csv >> /app/public/diff/mariana_islands__occurrences_16533.diff
[CMD] [2019-10-14 05:01:37] echo "." >> /app/public/diff/mariana_islands__occurrences_16533.diff
[CMD] [2019-10-14 05:01:37] echo "0a" > /app/public/diff/mariana_islands__measurements_16534.diff
[CMD] [2019-10-14 05:01:37] tail -n +1 /app/public/converted_csv/mariana_islands__measurements_16534.csv >> /app/public/diff/mariana_islands__measurements_16534.diff
[CMD] [2019-10-14 05:01:37] echo "." >> /app/public/diff/mariana_islands__measurements_16534.diff
[STOP] [2019-10-14 05:01:37] calculate_delta
[START] [2019-10-14 05:01:37] parse_diff_and_store
[INFO] [2019-10-14 05:01:37] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-14 05:01:37] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-14 05:01:39] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-14 05:01:39] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-14 05:01:50] Storing 2 References
[INFO] [2019-10-14 05:01:50] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-14 05:01:50] Average Time: 0.0
[INFO] [2019-10-14 05:01:50] Total Time: 1s
[INFO] [2019-10-14 05:01:50] Storing 3443 ScientificNames
[INFO] [2019-10-14 05:01:50] Processing group of 3443 in 4 groups of 1000
[INFO] [2019-10-14 05:01:51] Average Time: 0.35
[INFO] [2019-10-14 05:01:51] Total Time: 2s
[INFO] [2019-10-14 05:01:51] Storing 3443 Nodes
[INFO] [2019-10-14 05:01:51] Processing group of 3443 in 4 groups of 1000
[INFO] [2019-10-14 05:01:52] Average Time: 0.272
[INFO] [2019-10-14 05:01:52] Total Time: 2s
[INFO] [2019-10-14 05:01:52] Storing 1446 Occurrences
[INFO] [2019-10-14 05:01:52] Processing group of 1446 in 2 groups of 1000
[INFO] [2019-10-14 05:01:53] Average Time: 0.145
[INFO] [2019-10-14 05:01:53] Total Time: 1s
[INFO] [2019-10-14 05:01:53] Storing 3600 TraitsReferences
[INFO] [2019-10-14 05:01:53] Processing group of 3600 in 4 groups of 1000
[INFO] [2019-10-14 05:01:53] Average Time: 0.085
[INFO] [2019-10-14 05:01:53] Total Time: 1s
[INFO] [2019-10-14 05:01:53] Storing 3599 Traits
[INFO] [2019-10-14 05:01:53] Processing group of 3599 in 4 groups of 1000
[INFO] [2019-10-14 05:01:54] Average Time: 0.31
[INFO] [2019-10-14 05:01:54] Total Time: 2s
[INFO] [2019-10-14 05:01:54] Storing 3597 MetaTraits
[INFO] [2019-10-14 05:01:54] Processing group of 3597 in 4 groups of 1000
[INFO] [2019-10-14 05:01:55] Average Time: 0.118
[INFO] [2019-10-14 05:01:55] Total Time: 1s
[STOP] [2019-10-14 05:01:55] parse_diff_and_store
[START] [2019-10-14 05:01:55] resolve_keys
[INFO] [2019-10-14 05:02:11] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-14 05:02:12] traits to occurrences...
[INFO] [2019-10-14 05:02:13] traits to nodes (through occurrences)...
[INFO] [2019-10-14 05:02:13] Traits to sex term...
[INFO] [2019-10-14 05:02:13] Traits to lifestage term...
[INFO] [2019-10-14 05:02:14] MetaTraits to traits...
[INFO] [2019-10-14 05:02:14] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-14 05:02:15] Assocs to occurrences...
[INFO] [2019-10-14 05:02:15] Assocs to nodes...
[INFO] [2019-10-14 05:02:15] Assoc to sex term...
[INFO] [2019-10-14 05:02:15] Assoc to lifestage term...
[STOP] [2019-10-14 05:02:15] resolve_keys
[START] [2019-10-14 05:02:15] hold_for_later_1
[STOP] [2019-10-14 05:02:15] hold_for_later_1
[START] [2019-10-14 05:02:15] hold_for_later_2
[STOP] [2019-10-14 05:02:15] hold_for_later_2
[START] [2019-10-14 05:02:15] resolve_missing_parents
[STOP] [2019-10-14 05:02:21] resolve_missing_parents
[START] [2019-10-14 05:02:21] rebuild_nodes
[START] [2019-10-14 05:02:21] Flattener#flatten
[START] [2019-10-14 05:02:21] Flattener#study_resource
[START] [2019-10-14 05:02:21] Flattener#build_ancestry
[STOP] [2019-10-14 05:02:21] Flattener#build_ancestry
[INFO] [2019-10-14 05:02:21] 3443 ancestry keys
[START] [2019-10-14 05:02:21] build_node_ancestors
[INFO] [2019-10-14 05:02:21] old ancestors deleted.
[STOP] [2019-10-14 05:02:22] build_node_ancestors
[START] [2019-10-14 05:02:22] Flattener#propagate_ancestor_ids
[STOP] [2019-10-14 05:02:22] Flattener#propagate_ancestor_ids
[STOP] [2019-10-14 05:02:22] Flattener#flatten
[STOP] [2019-10-14 05:02:22] rebuild_nodes
[START] [2019-10-14 05:02:22] resolve_missing_media_owners
[STOP] [2019-10-14 05:02:22] resolve_missing_media_owners
[START] [2019-10-14 05:02:22] sanitize_media_verbatims
[STOP] [2019-10-14 05:02:22] sanitize_media_verbatims
[START] [2019-10-14 05:02:22] queue_downloads
[STOP] [2019-10-14 05:02:22] queue_downloads
[START] [2019-10-14 05:02:22] parse_names
[WARN] [2019-10-14 05:02:22] I see 3443 names which still need to be parsed.
[STOP] [2019-10-14 05:02:26] parse_names
[START] [2019-10-14 05:02:26] denormalize_canonical_names_to_nodes
[STOP] [2019-10-14 05:02:26] denormalize_canonical_names_to_nodes
[START] [2019-10-14 05:02:26] match_nodes
[START] [2019-10-14 05:02:26] map_all_nodes_to_pages
[STOP] [2019-10-14 05:05:35] map_all_nodes_to_pages
[INFO] [2019-10-14 05:05:35] 248 Unmatched nodes (of 3443)! That's too many to output. First 10: Thalaseus (#50655369); Thalaseus bergii (#50655368); Tringa incanus (#50656878); Limnodromus (#50654258); Philomachus pugnax (#50655328); Streptopelia dusumieri (#50653803); Egretta intermedia (#50653901); Chondria simpliciuscula (#50653939); Chondria minutula (#50656159); Herposiphonia secunda (#50654027)
[START] [2019-10-14 05:05:35] update_nodes
[STOP] [2019-10-14 05:05:36] update_nodes
[STOP] [2019-10-14 05:05:36] match_nodes
[START] [2019-10-14 05:05:36] reindex_search
[STOP] [2019-10-14 05:05:43] reindex_search
[START] [2019-10-14 05:05:43] normalize_units
[STOP] [2019-10-14 05:05:43] normalize_units
[START] [2019-10-14 05:05:43] calculate_statistics
[STOP] [2019-10-14 05:05:43] calculate_statistics
[START] [2019-10-14 05:05:43] complete_harvest_instance
[START] [2019-10-14 05:05:43] overall_tsv_creation
[INFO] [2019-10-14 05:05:43] Processing group of 3443 in 1 batches of 10000
[INFO] [2019-10-14 05:06:43] 1446 Traits (unfiltered)...
[INFO] [2019-10-14 05:06:57] 1446 Traits (filtered)...
[INFO] [2019-10-14 05:06:57] 0 Associations (filtered)...
[INFO] [2019-10-14 05:07:40] 7228 metadata added.
[INFO] [2019-10-14 05:07:40] 0 metadata added.
[INFO] [2019-10-14 05:07:40] Average Time: 93.18
[INFO] [2019-10-14 05:07:40] Total Time: 1m57s
[STOP] [2019-10-14 05:07:40] overall_tsv_creation
[INFO] [2019-10-14 05:07:40] Done. Check your files:
[INFO] [2019-10-14 05:07:40] (3443 lines) /app/public/data/mariana_islands_/publish_nodes.tsv
[INFO] [2019-10-14 05:07:40] (5762 lines) /app/public/data/mariana_islands_/publish_node_ancestors.tsv
[INFO] [2019-10-14 05:07:40] (3443 lines) /app/public/data/mariana_islands_/publish_scientific_names.tsv
[INFO] [2019-10-14 05:07:40] (1447 lines) /app/public/data/mariana_islands_/publish_traits.tsv
[INFO] [2019-10-14 05:07:40] (7229 lines) /app/public/data/mariana_islands_/publish_metadata.tsv
[STOP] [2019-10-14 05:07:40] complete_harvest_instance
[START] [2019-10-14 05:07:40] completed
[STOP] [2019-10-14 05:07:40] completed
[STOP] [2019-10-14 05:07:40] logged process, took 365.78
Latest Process

14 Oct 05:01 EXITED currently processing 0