Harvest for French Polynesia Species List Created 13 Oct 00:34

Stage: completed
Fetched: 13 Oct 00:34
Validated: 13 Oct 00:34
Deltas Created 13 Oct 00:34
Units Normalized: 13 Oct 00:49
Ancestry Built: 13 Oct 00:37
Nodes Matched: 13 Oct 00:48
Names Parsed: 13 Oct 00:38
New Models Stored: 13 Oct 00:36
Indexed: 13 Oct 00:49
Completed: 13 Oct 00:53
Time to Harvest: less than a minute
Harvesting Log

(156 lines)
# Logfile created on 2019-10-13 00:34:56 -0400 by logger.rb/56815
[START] [2019-10-13 00:34:56] logged process
[START] [2019-10-13 00:34:56] create_harvest_instance
[STOP] [2019-10-13 00:34:56] create_harvest_instance
[START] [2019-10-13 00:34:56] fetch_files
[STOP] [2019-10-13 00:34:56] fetch_files
[START] [2019-10-13 00:34:56] validate_each_file
[STOP] [2019-10-13 00:34:58] validate_each_file
[START] [2019-10-13 00:34:58] convert_to_csv
[CMD] [2019-10-13 00:34:58] /usr/bin/sort /app/public/converted_csv/french_polynesia_refs_15795.csv > /app/public/converted_csv/french_polynesia_refs_15795.csv_sorted
[CMD] [2019-10-13 00:34:58] /usr/bin/sort /app/public/converted_csv/french_polynesia_nodes_15796.csv > /app/public/converted_csv/french_polynesia_nodes_15796.csv_sorted
[CMD] [2019-10-13 00:34:58] /usr/bin/sort /app/public/converted_csv/french_polynesia_occurrences_15797.csv > /app/public/converted_csv/french_polynesia_occurrences_15797.csv_sorted
[CMD] [2019-10-13 00:34:58] /usr/bin/sort /app/public/converted_csv/french_polynesia_measurements_15798.csv > /app/public/converted_csv/french_polynesia_measurements_15798.csv_sorted
[STOP] [2019-10-13 00:34:58] convert_to_csv
[START] [2019-10-13 00:34:58] calculate_delta
[CMD] [2019-10-13 00:34:58] echo "0a" > /app/public/diff/french_polynesia_refs_15795.diff
[CMD] [2019-10-13 00:34:58] tail -n +1 /app/public/converted_csv/french_polynesia_refs_15795.csv >> /app/public/diff/french_polynesia_refs_15795.diff
[CMD] [2019-10-13 00:34:58] echo "." >> /app/public/diff/french_polynesia_refs_15795.diff
[CMD] [2019-10-13 00:34:59] echo "0a" > /app/public/diff/french_polynesia_nodes_15796.diff
[CMD] [2019-10-13 00:34:59] tail -n +1 /app/public/converted_csv/french_polynesia_nodes_15796.csv >> /app/public/diff/french_polynesia_nodes_15796.diff
[CMD] [2019-10-13 00:34:59] echo "." >> /app/public/diff/french_polynesia_nodes_15796.diff
[CMD] [2019-10-13 00:34:59] echo "0a" > /app/public/diff/french_polynesia_occurrences_15797.diff
[CMD] [2019-10-13 00:34:59] tail -n +1 /app/public/converted_csv/french_polynesia_occurrences_15797.csv >> /app/public/diff/french_polynesia_occurrences_15797.diff
[CMD] [2019-10-13 00:34:59] echo "." >> /app/public/diff/french_polynesia_occurrences_15797.diff
[CMD] [2019-10-13 00:34:59] echo "0a" > /app/public/diff/french_polynesia_measurements_15798.diff
[CMD] [2019-10-13 00:34:59] tail -n +1 /app/public/converted_csv/french_polynesia_measurements_15798.csv >> /app/public/diff/french_polynesia_measurements_15798.diff
[CMD] [2019-10-13 00:34:59] echo "." >> /app/public/diff/french_polynesia_measurements_15798.diff
[STOP] [2019-10-13 00:34:59] calculate_delta
[START] [2019-10-13 00:34:59] parse_diff_and_store
[INFO] [2019-10-13 00:35:00] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-13 00:35:00] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-13 00:35:04] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-13 00:35:06] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-13 00:35:54] Storing 2 References
[INFO] [2019-10-13 00:35:54] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-13 00:35:54] Average Time: 0.0
[INFO] [2019-10-13 00:35:54] Total Time: 1s
[INFO] [2019-10-13 00:35:54] Storing 13472 ScientificNames
[INFO] [2019-10-13 00:35:54] Processing group of 13472 in 14 groups of 1000
[INFO] [2019-10-13 00:35:59] Average Time: 0.349
[INFO] [2019-10-13 00:35:59] Total Time: 5s
[INFO] [2019-10-13 00:35:59] last 3 / first 3: 0.72
[INFO] [2019-10-13 00:35:59] Std.Dev: 0.06324555320336758; Max: 0.44
[INFO] [2019-10-13 00:35:59] Storing 13472 Nodes
[INFO] [2019-10-13 00:35:59] Processing group of 13472 in 14 groups of 1000
[INFO] [2019-10-13 00:36:04] Average Time: 0.33
[INFO] [2019-10-13 00:36:04] Total Time: 5s
[INFO] [2019-10-13 00:36:04] last 3 / first 3: 0.78
[INFO] [2019-10-13 00:36:04] Std.Dev: 0.10954451150103323; Max: 0.64
[INFO] [2019-10-13 00:36:04] Storing 8194 Occurrences
[INFO] [2019-10-13 00:36:04] Processing group of 8194 in 9 groups of 1000
[INFO] [2019-10-13 00:36:04] Average Time: 0.092
[INFO] [2019-10-13 00:36:04] Total Time: 1s
[INFO] [2019-10-13 00:36:04] last 3 / first 3: 0.71
[INFO] [2019-10-13 00:36:04] Std.Dev: 0.03162277660168379; Max: 0.12
[INFO] [2019-10-13 00:36:04] Storing 16940 TraitsReferences
[INFO] [2019-10-13 00:36:04] Processing group of 16940 in 17 groups of 1000
[INFO] [2019-10-13 00:36:06] Average Time: 0.075
[INFO] [2019-10-13 00:36:06] Total Time: 2s
[INFO] [2019-10-13 00:36:06] last 3 / first 3: 0.93
[INFO] [2019-10-13 00:36:06] Std.Dev: 0.03162277660168379; Max: 0.15
[INFO] [2019-10-13 00:36:06] Storing 16939 Traits
[INFO] [2019-10-13 00:36:06] Processing group of 16939 in 17 groups of 1000
[INFO] [2019-10-13 00:36:11] Average Time: 0.291
[INFO] [2019-10-13 00:36:11] Total Time: 6s
[INFO] [2019-10-13 00:36:11] last 3 / first 3: 0.91
[INFO] [2019-10-13 00:36:11] Std.Dev: 0.03162277660168379; Max: 0.38
[INFO] [2019-10-13 00:36:11] Storing 16923 MetaTraits
[INFO] [2019-10-13 00:36:11] Processing group of 16923 in 17 groups of 1000
[INFO] [2019-10-13 00:36:13] Average Time: 0.111
[INFO] [2019-10-13 00:36:13] Total Time: 2s
[INFO] [2019-10-13 00:36:13] last 3 / first 3: 0.86
[INFO] [2019-10-13 00:36:13] Std.Dev: 0.0; Max: 0.15
[STOP] [2019-10-13 00:36:13] parse_diff_and_store
[START] [2019-10-13 00:36:13] resolve_keys
[INFO] [2019-10-13 00:37:01] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-13 00:37:05] traits to occurrences...
[INFO] [2019-10-13 00:37:11] traits to nodes (through occurrences)...
[INFO] [2019-10-13 00:37:11] Traits to sex term...
[INFO] [2019-10-13 00:37:16] Traits to lifestage term...
[INFO] [2019-10-13 00:37:20] MetaTraits to traits...
[INFO] [2019-10-13 00:37:21] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-13 00:37:24] Assocs to occurrences...
[INFO] [2019-10-13 00:37:24] Assocs to nodes...
[INFO] [2019-10-13 00:37:24] Assoc to sex term...
[INFO] [2019-10-13 00:37:24] Assoc to lifestage term...
[STOP] [2019-10-13 00:37:24] resolve_keys
[START] [2019-10-13 00:37:24] hold_for_later_1
[STOP] [2019-10-13 00:37:24] hold_for_later_1
[START] [2019-10-13 00:37:24] hold_for_later_2
[STOP] [2019-10-13 00:37:24] hold_for_later_2
[START] [2019-10-13 00:37:24] resolve_missing_parents
[STOP] [2019-10-13 00:37:49] resolve_missing_parents
[START] [2019-10-13 00:37:49] rebuild_nodes
[START] [2019-10-13 00:37:49] Flattener#flatten
[START] [2019-10-13 00:37:49] Flattener#study_resource
[START] [2019-10-13 00:37:49] Flattener#build_ancestry
[STOP] [2019-10-13 00:37:50] Flattener#build_ancestry
[INFO] [2019-10-13 00:37:50] 13472 ancestry keys
[START] [2019-10-13 00:37:50] build_node_ancestors
[INFO] [2019-10-13 00:37:50] old ancestors deleted.
[STOP] [2019-10-13 00:37:53] build_node_ancestors
[START] [2019-10-13 00:37:55] Flattener#propagate_ancestor_ids
[STOP] [2019-10-13 00:37:56] Flattener#propagate_ancestor_ids
[STOP] [2019-10-13 00:37:56] Flattener#flatten
[STOP] [2019-10-13 00:37:56] rebuild_nodes
[START] [2019-10-13 00:37:56] resolve_missing_media_owners
[STOP] [2019-10-13 00:37:56] resolve_missing_media_owners
[START] [2019-10-13 00:37:56] sanitize_media_verbatims
[STOP] [2019-10-13 00:37:56] sanitize_media_verbatims
[START] [2019-10-13 00:37:56] queue_downloads
[STOP] [2019-10-13 00:37:56] queue_downloads
[START] [2019-10-13 00:37:56] parse_names
[WARN] [2019-10-13 00:37:56] I see 13472 names which still need to be parsed.
[STOP] [2019-10-13 00:38:07] parse_names
[START] [2019-10-13 00:38:07] denormalize_canonical_names_to_nodes
[STOP] [2019-10-13 00:38:07] denormalize_canonical_names_to_nodes
[START] [2019-10-13 00:38:07] match_nodes
[START] [2019-10-13 00:38:07] map_all_nodes_to_pages
[STOP] [2019-10-13 00:48:30] map_all_nodes_to_pages
[INFO] [2019-10-13 00:48:30] 1071 Unmatched nodes (of 13472)! That's too many to output. First 10: Thalaseus (#49765774); Thalaseus bergii (#49765773); Onychoprion fuscata (#49767607); Onychoprion lunata (#49775610); Procelsterna cerulea (#49766836); Tringa incanus (#49767123); Ramphocelus bresilius (#49775969); Pitta granatina (#49774413); Linckia guildingii (#49774474); Cephalopholis guttatus (#49774872)
[START] [2019-10-13 00:48:30] update_nodes
[STOP] [2019-10-13 00:48:34] update_nodes
[STOP] [2019-10-13 00:48:34] match_nodes
[START] [2019-10-13 00:48:34] reindex_search
[STOP] [2019-10-13 00:49:01] reindex_search
[START] [2019-10-13 00:49:01] normalize_units
[STOP] [2019-10-13 00:49:01] normalize_units
[START] [2019-10-13 00:49:02] calculate_statistics
[STOP] [2019-10-13 00:49:02] calculate_statistics
[START] [2019-10-13 00:49:02] complete_harvest_instance
[START] [2019-10-13 00:49:02] overall_tsv_creation
[INFO] [2019-10-13 00:49:02] Processing group of 13472 in 2 batches of 10000
[INFO] [2019-10-13 00:50:32] 6043 Traits (unfiltered)...
[INFO] [2019-10-13 00:50:46] 6043 Traits (filtered)...
[INFO] [2019-10-13 00:50:46] 0 Associations (filtered)...
[INFO] [2019-10-13 00:51:38] 30208 metadata added.
[INFO] [2019-10-13 00:51:38] 0 metadata added.
[INFO] [2019-10-13 00:52:41] 2151 Traits (unfiltered)...
[INFO] [2019-10-13 00:52:57] 2151 Traits (filtered)...
[INFO] [2019-10-13 00:52:57] 0 Associations (filtered)...
[INFO] [2019-10-13 00:53:46] 10745 metadata added.
[INFO] [2019-10-13 00:53:46] 0 metadata added.
[INFO] [2019-10-13 00:53:46] Average Time: 117.055
[INFO] [2019-10-13 00:53:46] Total Time: 4m45s
[STOP] [2019-10-13 00:53:46] overall_tsv_creation
[INFO] [2019-10-13 00:53:46] Done. Check your files:
[INFO] [2019-10-13 00:53:46] (13472 lines) /app/public/data/french_polynesia/publish_nodes.tsv
[INFO] [2019-10-13 00:53:46] (32870 lines) /app/public/data/french_polynesia/publish_node_ancestors.tsv
[INFO] [2019-10-13 00:53:46] (13472 lines) /app/public/data/french_polynesia/publish_scientific_names.tsv
[INFO] [2019-10-13 00:53:46] (8195 lines) /app/public/data/french_polynesia/publish_traits.tsv
[INFO] [2019-10-13 00:53:46] (40954 lines) /app/public/data/french_polynesia/publish_metadata.tsv
[STOP] [2019-10-13 00:53:47] complete_harvest_instance
[START] [2019-10-13 00:53:47] completed
[STOP] [2019-10-13 00:53:47] completed
[STOP] [2019-10-13 00:53:47] logged process, took 1130.93
Latest Process

13 Oct 00:34 EXITED currently processing 0