Stage:
completed
Fetched:
13 Nov 15:48
Validated:
13 Nov 15:48
Deltas Created
13 Nov 15:48
Units Normalized:
13 Nov 15:57
Ancestry Built:
13 Nov 15:55
Nodes Matched:
13 Nov 15:57
Names Parsed:
13 Nov 15:55
New Models Stored:
13 Nov 15:55
Indexed:
13 Nov 15:57
Completed:
13 Nov 16:03
Time to Harvest:
less than a minute
Harvesting Log
(155 lines)
# Logfile created on 2019-11-13 15:48:38 -0500 by logger.rb/56815
[START] [2019-11-13 15:48:38] logged process
[START] [2019-11-13 15:48:38] create_harvest_instance
[STOP] [2019-11-13 15:48:39] create_harvest_instance
[START] [2019-11-13 15:48:39] fetch_files
[STOP] [2019-11-13 15:48:39] fetch_files
[START] [2019-11-13 15:48:39] validate_each_file
[STOP] [2019-11-13 15:48:44] validate_each_file
[START] [2019-11-13 15:48:44] convert_to_csv
[CMD] [2019-11-13 15:48:44] /usr/bin/sort /app/public/converted_csv/nwpln_nodes_18532.csv > /app/public/converted_csv/nwpln_nodes_18532.csv_sorted
[CMD] [2019-11-13 15:48:44] /usr/bin/sort /app/public/converted_csv/nwpln_vernaculars_18533.csv > /app/public/converted_csv/nwpln_vernaculars_18533.csv_sorted
[CMD] [2019-11-13 15:48:44] /usr/bin/sort /app/public/converted_csv/nwpln_occurrences_18534.csv > /app/public/converted_csv/nwpln_occurrences_18534.csv_sorted
[CMD] [2019-11-13 15:48:44] /usr/bin/sort /app/public/converted_csv/nwpln_measurements_18535.csv > /app/public/converted_csv/nwpln_measurements_18535.csv_sorted
[STOP] [2019-11-13 15:48:44] convert_to_csv
[START] [2019-11-13 15:48:44] calculate_delta
[CMD] [2019-11-13 15:48:44] echo "0a" > /app/public/diff/nwpln_nodes_18532.diff
[CMD] [2019-11-13 15:48:44] tail -n +1 /app/public/converted_csv/nwpln_nodes_18532.csv >> /app/public/diff/nwpln_nodes_18532.diff
[CMD] [2019-11-13 15:48:44] echo "." >> /app/public/diff/nwpln_nodes_18532.diff
[CMD] [2019-11-13 15:48:44] echo "0a" > /app/public/diff/nwpln_vernaculars_18533.diff
[CMD] [2019-11-13 15:48:44] tail -n +1 /app/public/converted_csv/nwpln_vernaculars_18533.csv >> /app/public/diff/nwpln_vernaculars_18533.diff
[CMD] [2019-11-13 15:48:44] echo "." >> /app/public/diff/nwpln_vernaculars_18533.diff
[CMD] [2019-11-13 15:48:44] echo "0a" > /app/public/diff/nwpln_occurrences_18534.diff
[CMD] [2019-11-13 15:48:44] tail -n +1 /app/public/converted_csv/nwpln_occurrences_18534.csv >> /app/public/diff/nwpln_occurrences_18534.diff
[CMD] [2019-11-13 15:48:44] echo "." >> /app/public/diff/nwpln_occurrences_18534.diff
[CMD] [2019-11-13 15:48:44] echo "0a" > /app/public/diff/nwpln_measurements_18535.diff
[CMD] [2019-11-13 15:48:44] tail -n +1 /app/public/converted_csv/nwpln_measurements_18535.csv >> /app/public/diff/nwpln_measurements_18535.diff
[CMD] [2019-11-13 15:48:44] echo "." >> /app/public/diff/nwpln_measurements_18535.diff
[STOP] [2019-11-13 15:48:44] calculate_delta
[START] [2019-11-13 15:48:44] parse_diff_and_store
[INFO] [2019-11-13 15:48:44] Loading nodes diff file into memory (true lines)...
[INFO] [2019-11-13 15:48:48] Loading vernaculars diff file into memory (true lines)...
[INFO] [2019-11-13 15:48:50] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-11-13 15:49:54] Loading measurements diff file into memory (true lines)...
[INFO] [2019-11-13 15:54:27] Storing 8185 ScientificNames
[INFO] [2019-11-13 15:54:27] Processing group of 8185 in 9 groups of 1000
[INFO] [2019-11-13 15:54:30] Average Time: 0.336
[INFO] [2019-11-13 15:54:30] Total Time: 4s
[INFO] [2019-11-13 15:54:30] last 3 / first 3: 0.69
[INFO] [2019-11-13 15:54:30] Std.Dev: 0.10488088481701516; Max: 0.44
[INFO] [2019-11-13 15:54:30] Storing 8185 Nodes
[INFO] [2019-11-13 15:54:30] Processing group of 8185 in 9 groups of 1000
[INFO] [2019-11-13 15:54:33] Average Time: 0.272
[INFO] [2019-11-13 15:54:33] Total Time: 3s
[INFO] [2019-11-13 15:54:33] last 3 / first 3: 0.7
[INFO] [2019-11-13 15:54:33] Std.Dev: 0.08366600265340755; Max: 0.34
[INFO] [2019-11-13 15:54:33] Storing 8086 Vernaculars
[INFO] [2019-11-13 15:54:33] Processing group of 8086 in 9 groups of 1000
[INFO] [2019-11-13 15:54:35] Average Time: 0.178
[INFO] [2019-11-13 15:54:35] Total Time: 2s
[INFO] [2019-11-13 15:54:35] last 3 / first 3: 0.65
[INFO] [2019-11-13 15:54:35] Std.Dev: 0.06324555320336758; Max: 0.24
[INFO] [2019-11-13 15:54:35] Storing 55830 Occurrences
[INFO] [2019-11-13 15:54:35] Processing group of 55830 in 56 groups of 1000
[INFO] [2019-11-13 15:54:41] Average Time: 0.11
[INFO] [2019-11-13 15:54:41] Total Time: 7s
[INFO] [2019-11-13 15:54:41] last 3 / first 3: 0.91
[INFO] [2019-11-13 15:54:41] Std.Dev: 0.0; Max: 0.17
[INFO] [2019-11-13 15:54:41] Storing 27915 OccurrenceMetadata
[INFO] [2019-11-13 15:54:41] Processing group of 27915 in 28 groups of 1000
[INFO] [2019-11-13 15:54:46] Average Time: 0.16
[INFO] [2019-11-13 15:54:46] Total Time: 5s
[INFO] [2019-11-13 15:54:46] last 3 / first 3: 0.52
[INFO] [2019-11-13 15:54:46] Std.Dev: 0.07745966692414834; Max: 0.39
[INFO] [2019-11-13 15:54:46] Storing 55830 Traits
[INFO] [2019-11-13 15:54:46] Processing group of 55830 in 56 groups of 1000
[INFO] [2019-11-13 15:55:02] Average Time: 0.292
[INFO] [2019-11-13 15:55:02] Total Time: 17s
[INFO] [2019-11-13 15:55:02] last 3 / first 3: 0.8
[INFO] [2019-11-13 15:55:02] Std.Dev: 0.03162277660168379; Max: 0.42
[INFO] [2019-11-13 15:55:02] Storing 139575 MetaTraits
[INFO] [2019-11-13 15:55:02] Processing group of 139575 in 140 groups of 1000
[INFO] [2019-11-13 15:55:20] Average Time: 0.126
[INFO] [2019-11-13 15:55:20] Total Time: 19s
[INFO] [2019-11-13 15:55:20] last 3 / first 3: 0.89
[INFO] [2019-11-13 15:55:20] Std.Dev: 0.08944271909999159; Max: 1.17
[STOP] [2019-11-13 15:55:20] parse_diff_and_store
[START] [2019-11-13 15:55:20] resolve_keys
[INFO] [2019-11-13 15:55:27] Occurrences to nodes (through scientific_names)...
[INFO] [2019-11-13 15:55:29] traits to occurrences...
[INFO] [2019-11-13 15:55:34] traits to nodes (through occurrences)...
[INFO] [2019-11-13 15:55:35] Traits to sex term...
[INFO] [2019-11-13 15:55:36] Traits to lifestage term...
[INFO] [2019-11-13 15:55:37] MetaTraits to traits...
[INFO] [2019-11-13 15:55:46] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-11-13 15:55:46] Assocs to occurrences...
[INFO] [2019-11-13 15:55:46] Assocs to nodes...
[INFO] [2019-11-13 15:55:46] Assoc to sex term...
[INFO] [2019-11-13 15:55:46] Assoc to lifestage term...
[STOP] [2019-11-13 15:55:46] resolve_keys
[START] [2019-11-13 15:55:46] hold_for_later_1
[STOP] [2019-11-13 15:55:46] hold_for_later_1
[START] [2019-11-13 15:55:46] hold_for_later_2
[STOP] [2019-11-13 15:55:46] hold_for_later_2
[START] [2019-11-13 15:55:46] resolve_missing_parents
[STOP] [2019-11-13 15:55:46] resolve_missing_parents
[START] [2019-11-13 15:55:46] rebuild_nodes
[START] [2019-11-13 15:55:46] Flattener#flatten
[START] [2019-11-13 15:55:46] Flattener#study_resource
[START] [2019-11-13 15:55:46] Flattener#build_ancestry
[STOP] [2019-11-13 15:55:48] Flattener#build_ancestry
[INFO] [2019-11-13 15:55:48] 8185 ancestry keys
[START] [2019-11-13 15:55:48] build_node_ancestors
[INFO] [2019-11-13 15:55:48] old ancestors deleted.
[STOP] [2019-11-13 15:55:48] build_node_ancestors
[START] [2019-11-13 15:55:49] Flattener#propagate_ancestor_ids
[STOP] [2019-11-13 15:55:49] Flattener#propagate_ancestor_ids
[STOP] [2019-11-13 15:55:49] Flattener#flatten
[STOP] [2019-11-13 15:55:49] rebuild_nodes
[START] [2019-11-13 15:55:49] resolve_missing_media_owners
[STOP] [2019-11-13 15:55:49] resolve_missing_media_owners
[START] [2019-11-13 15:55:49] sanitize_media_verbatims
[STOP] [2019-11-13 15:55:49] sanitize_media_verbatims
[START] [2019-11-13 15:55:49] queue_downloads
[STOP] [2019-11-13 15:55:49] queue_downloads
[START] [2019-11-13 15:55:49] parse_names
[WARN] [2019-11-13 15:55:49] I see 8185 names which still need to be parsed.
[WARN] [2019-11-13 15:55:57] I see 1 names which still need to be parsed.
[STOP] [2019-11-13 15:55:58] parse_names
[START] [2019-11-13 15:55:58] denormalize_canonical_names_to_nodes
[STOP] [2019-11-13 15:55:58] denormalize_canonical_names_to_nodes
[START] [2019-11-13 15:55:58] match_nodes
[START] [2019-11-13 15:55:58] map_all_nodes_to_pages
[STOP] [2019-11-13 15:57:39] map_all_nodes_to_pages
[INFO] [2019-11-13 15:57:39] 163 Unmatched nodes (of 8185)! That's too many to output. First 10: Abies bifolia (#54788619); Aconitum delphiniifolium (#54788669); Adiantum pedatum (#54788694); Agalinis flexicaulis (#54788721); Agoseris agrestis (#54788747); Alnus viridis (#54788822); Amelanchier nantucketensis (#54788886); Andropogon glaucopsis (#54788929); Andropogon gracilis (#54788931); Anthenantia rufa (#54788978)
[START] [2019-11-13 15:57:39] update_nodes
[STOP] [2019-11-13 15:57:40] update_nodes
[STOP] [2019-11-13 15:57:40] match_nodes
[START] [2019-11-13 15:57:40] reindex_search
[STOP] [2019-11-13 15:57:50] reindex_search
[START] [2019-11-13 15:57:50] normalize_units
[STOP] [2019-11-13 15:57:50] normalize_units
[START] [2019-11-13 15:57:50] calculate_statistics
[STOP] [2019-11-13 15:57:51] calculate_statistics
[START] [2019-11-13 15:57:51] complete_harvest_instance
[START] [2019-11-13 15:57:51] overall_tsv_creation
[INFO] [2019-11-13 15:57:51] Processing group of 8185 in 1 batches of 10000
[INFO] [2019-11-13 15:59:10] 55830 Traits (unfiltered)...
[INFO] [2019-11-13 15:59:23] 55830 Traits (filtered)...
[INFO] [2019-11-13 15:59:23] 0 Associations (filtered)...
[INFO] [2019-11-13 16:03:16] 167490 metadata added.
[INFO] [2019-11-13 16:03:16] 0 metadata added.
[INFO] [2019-11-13 16:03:16] Average Time: 299.5
[INFO] [2019-11-13 16:03:16] Total Time: 5m25s
[STOP] [2019-11-13 16:03:16] overall_tsv_creation
[INFO] [2019-11-13 16:03:16] Done. Check your files:
[INFO] [2019-11-13 16:03:16] (8184 lines) /app/public/data/nwpln/publish_nodes.tsv
[INFO] [2019-11-13 16:03:16] (3495 lines) /app/public/data/nwpln/publish_node_ancestors.tsv
[INFO] [2019-11-13 16:03:16] (8185 lines) /app/public/data/nwpln/publish_scientific_names.tsv
[INFO] [2019-11-13 16:03:16] (8086 lines) /app/public/data/nwpln/publish_vernaculars.tsv
[INFO] [2019-11-13 16:03:16] (55831 lines) /app/public/data/nwpln/publish_traits.tsv
[INFO] [2019-11-13 16:03:16] (167491 lines) /app/public/data/nwpln/publish_metadata.tsv
[STOP] [2019-11-13 16:03:16] complete_harvest_instance
[START] [2019-11-13 16:03:16] completed
[STOP] [2019-11-13 16:03:16] completed
[STOP] [2019-11-13 16:03:16] logged process, took 877.57
Latest Process