Harvest for Singapore Species List Created 15 Oct 18:26

Stage: completed
Fetched: 15 Oct 18:26
Validated: 15 Oct 18:26
Deltas Created 15 Oct 18:26
Units Normalized: 15 Oct 18:36
Ancestry Built: 15 Oct 18:27
Nodes Matched: 15 Oct 18:36
Names Parsed: 15 Oct 18:28
New Models Stored: 15 Oct 18:26
Indexed: 15 Oct 18:36
Completed: 15 Oct 18:38
Time to Harvest: less than a minute
Harvesting Log

(151 lines)
# Logfile created on 2019-10-15 18:26:14 -0400 by logger.rb/56815
[START] [2019-10-15 18:26:14] logged process
[START] [2019-10-15 18:26:14] create_harvest_instance
[STOP] [2019-10-15 18:26:14] create_harvest_instance
[START] [2019-10-15 18:26:14] fetch_files
[STOP] [2019-10-15 18:26:14] fetch_files
[START] [2019-10-15 18:26:14] validate_each_file
[STOP] [2019-10-15 18:26:15] validate_each_file
[START] [2019-10-15 18:26:15] convert_to_csv
[CMD] [2019-10-15 18:26:15] /usr/bin/sort /app/public/converted_csv/singapore_sp_lis_refs_17189.csv > /app/public/converted_csv/singapore_sp_lis_refs_17189.csv_sorted
[CMD] [2019-10-15 18:26:15] /usr/bin/sort /app/public/converted_csv/singapore_sp_lis_nodes_17190.csv > /app/public/converted_csv/singapore_sp_lis_nodes_17190.csv_sorted
[CMD] [2019-10-15 18:26:16] /usr/bin/sort /app/public/converted_csv/singapore_sp_lis_occurrences_17191.csv > /app/public/converted_csv/singapore_sp_lis_occurrences_17191.csv_sorted
[CMD] [2019-10-15 18:26:16] /usr/bin/sort /app/public/converted_csv/singapore_sp_lis_measurements_17192.csv > /app/public/converted_csv/singapore_sp_lis_measurements_17192.csv_sorted
[STOP] [2019-10-15 18:26:16] convert_to_csv
[START] [2019-10-15 18:26:16] calculate_delta
[CMD] [2019-10-15 18:26:16] echo "0a" > /app/public/diff/singapore_sp_lis_refs_17189.diff
[CMD] [2019-10-15 18:26:16] tail -n +1 /app/public/converted_csv/singapore_sp_lis_refs_17189.csv >> /app/public/diff/singapore_sp_lis_refs_17189.diff
[CMD] [2019-10-15 18:26:17] echo "." >> /app/public/diff/singapore_sp_lis_refs_17189.diff
[CMD] [2019-10-15 18:26:17] echo "0a" > /app/public/diff/singapore_sp_lis_nodes_17190.diff
[CMD] [2019-10-15 18:26:17] tail -n +1 /app/public/converted_csv/singapore_sp_lis_nodes_17190.csv >> /app/public/diff/singapore_sp_lis_nodes_17190.diff
[CMD] [2019-10-15 18:26:18] echo "." >> /app/public/diff/singapore_sp_lis_nodes_17190.diff
[CMD] [2019-10-15 18:26:18] echo "0a" > /app/public/diff/singapore_sp_lis_occurrences_17191.diff
[CMD] [2019-10-15 18:26:18] tail -n +1 /app/public/converted_csv/singapore_sp_lis_occurrences_17191.csv >> /app/public/diff/singapore_sp_lis_occurrences_17191.diff
[CMD] [2019-10-15 18:26:18] echo "." >> /app/public/diff/singapore_sp_lis_occurrences_17191.diff
[CMD] [2019-10-15 18:26:19] echo "0a" > /app/public/diff/singapore_sp_lis_measurements_17192.diff
[CMD] [2019-10-15 18:26:19] tail -n +1 /app/public/converted_csv/singapore_sp_lis_measurements_17192.csv >> /app/public/diff/singapore_sp_lis_measurements_17192.diff
[CMD] [2019-10-15 18:26:19] echo "." >> /app/public/diff/singapore_sp_lis_measurements_17192.diff
[STOP] [2019-10-15 18:26:20] calculate_delta
[START] [2019-10-15 18:26:20] parse_diff_and_store
[INFO] [2019-10-15 18:26:20] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-15 18:26:20] Loading nodes diff file into memory (true lines)...
[WARN] [2019-10-15 18:26:21] Filtered Scientific Name `Mimegralla  albimana` to `Mimegralla albimana`
[INFO] [2019-10-15 18:26:23] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-15 18:26:24] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-15 18:26:47] Storing 2 References
[INFO] [2019-10-15 18:26:47] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-15 18:26:47] Average Time: 0.0
[INFO] [2019-10-15 18:26:47] Total Time: 1s
[INFO] [2019-10-15 18:26:47] Storing 7351 ScientificNames
[INFO] [2019-10-15 18:26:47] Processing group of 7351 in 8 groups of 1000
[INFO] [2019-10-15 18:26:50] Average Time: 0.38
[INFO] [2019-10-15 18:26:50] Total Time: 4s
[INFO] [2019-10-15 18:26:50] last 3 / first 3: 0.57
[INFO] [2019-10-15 18:26:50] Std.Dev: 0.1224744871391589; Max: 0.52
[INFO] [2019-10-15 18:26:50] Storing 7351 Nodes
[INFO] [2019-10-15 18:26:50] Processing group of 7351 in 8 groups of 1000
[INFO] [2019-10-15 18:26:52] Average Time: 0.31
[INFO] [2019-10-15 18:26:52] Total Time: 3s
[INFO] [2019-10-15 18:26:52] last 3 / first 3: 0.87
[INFO] [2019-10-15 18:26:52] Std.Dev: 0.05477225575051661; Max: 0.34
[INFO] [2019-10-15 18:26:52] Storing 3856 Occurrences
[INFO] [2019-10-15 18:26:52] Processing group of 3856 in 4 groups of 1000
[INFO] [2019-10-15 18:26:53] Average Time: 0.115
[INFO] [2019-10-15 18:26:53] Total Time: 1s
[INFO] [2019-10-15 18:26:53] Storing 7810 TraitsReferences
[INFO] [2019-10-15 18:26:53] Processing group of 7810 in 8 groups of 1000
[INFO] [2019-10-15 18:26:53] Average Time: 0.076
[INFO] [2019-10-15 18:26:53] Total Time: 1s
[INFO] [2019-10-15 18:26:53] last 3 / first 3: 0.62
[INFO] [2019-10-15 18:26:53] Std.Dev: 0.03162277660168379; Max: 0.15
[INFO] [2019-10-15 18:26:53] Storing 7809 Traits
[INFO] [2019-10-15 18:26:53] Processing group of 7809 in 8 groups of 1000
[INFO] [2019-10-15 18:26:56] Average Time: 0.295
[INFO] [2019-10-15 18:26:56] Total Time: 3s
[INFO] [2019-10-15 18:26:56] last 3 / first 3: 0.87
[INFO] [2019-10-15 18:26:56] Std.Dev: 0.044721359549995794; Max: 0.39
[INFO] [2019-10-15 18:26:56] Storing 7802 MetaTraits
[INFO] [2019-10-15 18:26:56] Processing group of 7802 in 8 groups of 1000
[INFO] [2019-10-15 18:26:57] Average Time: 0.124
[INFO] [2019-10-15 18:26:57] Total Time: 2s
[INFO] [2019-10-15 18:26:57] last 3 / first 3: 0.69
[INFO] [2019-10-15 18:26:57] Std.Dev: 0.03162277660168379; Max: 0.2
[STOP] [2019-10-15 18:26:57] parse_diff_and_store
[START] [2019-10-15 18:26:57] resolve_keys
[INFO] [2019-10-15 18:27:28] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-15 18:27:31] traits to occurrences...
[INFO] [2019-10-15 18:27:32] traits to nodes (through occurrences)...
[INFO] [2019-10-15 18:27:32] Traits to sex term...
[INFO] [2019-10-15 18:27:34] Traits to lifestage term...
[INFO] [2019-10-15 18:27:36] MetaTraits to traits...
[INFO] [2019-10-15 18:27:36] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-15 18:27:37] Assocs to occurrences...
[INFO] [2019-10-15 18:27:37] Assocs to nodes...
[INFO] [2019-10-15 18:27:37] Assoc to sex term...
[INFO] [2019-10-15 18:27:37] Assoc to lifestage term...
[STOP] [2019-10-15 18:27:37] resolve_keys
[START] [2019-10-15 18:27:37] hold_for_later_1
[STOP] [2019-10-15 18:27:37] hold_for_later_1
[START] [2019-10-15 18:27:37] hold_for_later_2
[STOP] [2019-10-15 18:27:37] hold_for_later_2
[START] [2019-10-15 18:27:37] resolve_missing_parents
[STOP] [2019-10-15 18:27:54] resolve_missing_parents
[START] [2019-10-15 18:27:54] rebuild_nodes
[START] [2019-10-15 18:27:54] Flattener#flatten
[START] [2019-10-15 18:27:54] Flattener#study_resource
[START] [2019-10-15 18:27:54] Flattener#build_ancestry
[STOP] [2019-10-15 18:27:54] Flattener#build_ancestry
[INFO] [2019-10-15 18:27:54] 7351 ancestry keys
[START] [2019-10-15 18:27:54] build_node_ancestors
[INFO] [2019-10-15 18:27:54] old ancestors deleted.
[STOP] [2019-10-15 18:27:55] build_node_ancestors
[START] [2019-10-15 18:27:57] Flattener#propagate_ancestor_ids
[STOP] [2019-10-15 18:27:57] Flattener#propagate_ancestor_ids
[STOP] [2019-10-15 18:27:57] Flattener#flatten
[STOP] [2019-10-15 18:27:57] rebuild_nodes
[START] [2019-10-15 18:27:57] resolve_missing_media_owners
[STOP] [2019-10-15 18:27:57] resolve_missing_media_owners
[START] [2019-10-15 18:27:57] sanitize_media_verbatims
[STOP] [2019-10-15 18:27:57] sanitize_media_verbatims
[START] [2019-10-15 18:27:57] queue_downloads
[STOP] [2019-10-15 18:27:57] queue_downloads
[START] [2019-10-15 18:27:57] parse_names
[WARN] [2019-10-15 18:27:57] I see 7351 names which still need to be parsed.
[WARN] [2019-10-15 18:28:03] I see 2 names which still need to be parsed.
[STOP] [2019-10-15 18:28:05] parse_names
[START] [2019-10-15 18:28:05] denormalize_canonical_names_to_nodes
[STOP] [2019-10-15 18:28:05] denormalize_canonical_names_to_nodes
[START] [2019-10-15 18:28:05] match_nodes
[START] [2019-10-15 18:28:05] map_all_nodes_to_pages
[STOP] [2019-10-15 18:36:12] map_all_nodes_to_pages
[INFO] [2019-10-15 18:36:12] 541 Unmatched nodes (of 7351)! That's too many to output. First 10: Dicrurus annectans (#51806724); Nectarinia jugularis (#51807425); Nectarinia calcostetha (#51809273); Muscicapa latirostris (#51806510); Luscinia cyane (#51809664); Rhinomyias brunneata (#51813678); Mixornis (#51806477); Mixornis gularis (#51806476); Turdinus (#51807526); Turdinus abbotti (#51807525)
[START] [2019-10-15 18:36:12] update_nodes
[STOP] [2019-10-15 18:36:15] update_nodes
[STOP] [2019-10-15 18:36:15] match_nodes
[START] [2019-10-15 18:36:15] reindex_search
[STOP] [2019-10-15 18:36:38] reindex_search
[START] [2019-10-15 18:36:38] normalize_units
[STOP] [2019-10-15 18:36:38] normalize_units
[START] [2019-10-15 18:36:38] calculate_statistics
[STOP] [2019-10-15 18:36:38] calculate_statistics
[START] [2019-10-15 18:36:38] complete_harvest_instance
[START] [2019-10-15 18:36:38] overall_tsv_creation
[INFO] [2019-10-15 18:36:38] Processing group of 7351 in 1 batches of 10000
[INFO] [2019-10-15 18:37:55] 3856 Traits (unfiltered)...
[INFO] [2019-10-15 18:38:08] 3856 Traits (filtered)...
[INFO] [2019-10-15 18:38:08] 0 Associations (filtered)...
[INFO] [2019-10-15 18:38:53] 19273 metadata added.
[INFO] [2019-10-15 18:38:53] 0 metadata added.
[INFO] [2019-10-15 18:38:53] Average Time: 109.8
[INFO] [2019-10-15 18:38:53] Total Time: 2m16s
[STOP] [2019-10-15 18:38:53] overall_tsv_creation
[INFO] [2019-10-15 18:38:53] Done. Check your files:
[INFO] [2019-10-15 18:38:53] (7351 lines) /app/public/data/singapore_sp_lis/publish_nodes.tsv
[INFO] [2019-10-15 18:38:53] (17645 lines) /app/public/data/singapore_sp_lis/publish_node_ancestors.tsv
[INFO] [2019-10-15 18:38:54] (7351 lines) /app/public/data/singapore_sp_lis/publish_scientific_names.tsv
[INFO] [2019-10-15 18:38:54] (3857 lines) /app/public/data/singapore_sp_lis/publish_traits.tsv
[INFO] [2019-10-15 18:38:54] (19274 lines) /app/public/data/singapore_sp_lis/publish_metadata.tsv
[STOP] [2019-10-15 18:38:54] complete_harvest_instance
[START] [2019-10-15 18:38:54] completed
[STOP] [2019-10-15 18:38:54] completed
[STOP] [2019-10-15 18:38:54] logged process, took 760.65
Latest Process

15 Oct 18:26 EXITED currently processing 0