Harvest for Macedonia Species List Created 14 Oct 02:57

Stage: completed
Fetched: 14 Oct 02:57
Validated: 14 Oct 02:57
Deltas Created 14 Oct 02:57
Units Normalized: 14 Oct 02:59
Ancestry Built: 14 Oct 02:57
Nodes Matched: 14 Oct 02:59
Names Parsed: 14 Oct 02:57
New Models Stored: 14 Oct 02:57
Indexed: 14 Oct 02:59
Completed: 14 Oct 03:01
Time to Harvest: less than a minute
Harvesting Log

(139 lines)
# Logfile created on 2019-10-14 02:57:03 -0400 by logger.rb/56815
[START] [2019-10-14 02:57:03] logged process
[START] [2019-10-14 02:57:03] create_harvest_instance
[STOP] [2019-10-14 02:57:03] create_harvest_instance
[START] [2019-10-14 02:57:03] fetch_files
[STOP] [2019-10-14 02:57:03] fetch_files
[START] [2019-10-14 02:57:03] validate_each_file
[STOP] [2019-10-14 02:57:03] validate_each_file
[START] [2019-10-14 02:57:03] convert_to_csv
[CMD] [2019-10-14 02:57:03] /usr/bin/sort /app/public/converted_csv/macedonia_sp_lis_refs_16459.csv > /app/public/converted_csv/macedonia_sp_lis_refs_16459.csv_sorted
[CMD] [2019-10-14 02:57:04] /usr/bin/sort /app/public/converted_csv/macedonia_sp_lis_nodes_16460.csv > /app/public/converted_csv/macedonia_sp_lis_nodes_16460.csv_sorted
[CMD] [2019-10-14 02:57:04] /usr/bin/sort /app/public/converted_csv/macedonia_sp_lis_occurrences_16461.csv > /app/public/converted_csv/macedonia_sp_lis_occurrences_16461.csv_sorted
[CMD] [2019-10-14 02:57:04] /usr/bin/sort /app/public/converted_csv/macedonia_sp_lis_measurements_16462.csv > /app/public/converted_csv/macedonia_sp_lis_measurements_16462.csv_sorted
[STOP] [2019-10-14 02:57:04] convert_to_csv
[START] [2019-10-14 02:57:04] calculate_delta
[CMD] [2019-10-14 02:57:04] echo "0a" > /app/public/diff/macedonia_sp_lis_refs_16459.diff
[CMD] [2019-10-14 02:57:04] tail -n +1 /app/public/converted_csv/macedonia_sp_lis_refs_16459.csv >> /app/public/diff/macedonia_sp_lis_refs_16459.diff
[CMD] [2019-10-14 02:57:04] echo "." >> /app/public/diff/macedonia_sp_lis_refs_16459.diff
[CMD] [2019-10-14 02:57:04] echo "0a" > /app/public/diff/macedonia_sp_lis_nodes_16460.diff
[CMD] [2019-10-14 02:57:04] tail -n +1 /app/public/converted_csv/macedonia_sp_lis_nodes_16460.csv >> /app/public/diff/macedonia_sp_lis_nodes_16460.diff
[CMD] [2019-10-14 02:57:04] echo "." >> /app/public/diff/macedonia_sp_lis_nodes_16460.diff
[CMD] [2019-10-14 02:57:04] echo "0a" > /app/public/diff/macedonia_sp_lis_occurrences_16461.diff
[CMD] [2019-10-14 02:57:04] tail -n +1 /app/public/converted_csv/macedonia_sp_lis_occurrences_16461.csv >> /app/public/diff/macedonia_sp_lis_occurrences_16461.diff
[CMD] [2019-10-14 02:57:05] echo "." >> /app/public/diff/macedonia_sp_lis_occurrences_16461.diff
[CMD] [2019-10-14 02:57:05] echo "0a" > /app/public/diff/macedonia_sp_lis_measurements_16462.diff
[CMD] [2019-10-14 02:57:05] tail -n +1 /app/public/converted_csv/macedonia_sp_lis_measurements_16462.csv >> /app/public/diff/macedonia_sp_lis_measurements_16462.diff
[CMD] [2019-10-14 02:57:05] echo "." >> /app/public/diff/macedonia_sp_lis_measurements_16462.diff
[STOP] [2019-10-14 02:57:05] calculate_delta
[START] [2019-10-14 02:57:05] parse_diff_and_store
[INFO] [2019-10-14 02:57:05] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-14 02:57:05] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-14 02:57:06] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-14 02:57:06] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-14 02:57:14] Storing 2 References
[INFO] [2019-10-14 02:57:14] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-14 02:57:14] Average Time: 0.0
[INFO] [2019-10-14 02:57:14] Total Time: 1s
[INFO] [2019-10-14 02:57:14] Storing 2382 ScientificNames
[INFO] [2019-10-14 02:57:14] Processing group of 2382 in 3 groups of 1000
[INFO] [2019-10-14 02:57:15] Average Time: 0.287
[INFO] [2019-10-14 02:57:15] Total Time: 1s
[INFO] [2019-10-14 02:57:15] Storing 2382 Nodes
[INFO] [2019-10-14 02:57:15] Processing group of 2382 in 3 groups of 1000
[INFO] [2019-10-14 02:57:16] Average Time: 0.217
[INFO] [2019-10-14 02:57:16] Total Time: 1s
[INFO] [2019-10-14 02:57:16] Storing 964 Occurrences
[INFO] [2019-10-14 02:57:16] Processing group of 964 in 1 groups of 1000
[INFO] [2019-10-14 02:57:16] Average Time: 0.11
[INFO] [2019-10-14 02:57:16] Total Time: 1s
[INFO] [2019-10-14 02:57:16] Storing 2720 TraitsReferences
[INFO] [2019-10-14 02:57:16] Processing group of 2720 in 3 groups of 1000
[INFO] [2019-10-14 02:57:16] Average Time: 0.08
[INFO] [2019-10-14 02:57:16] Total Time: 1s
[INFO] [2019-10-14 02:57:16] Storing 2719 Traits
[INFO] [2019-10-14 02:57:16] Processing group of 2719 in 3 groups of 1000
[INFO] [2019-10-14 02:57:17] Average Time: 0.383
[INFO] [2019-10-14 02:57:17] Total Time: 2s
[INFO] [2019-10-14 02:57:17] Storing 2719 MetaTraits
[INFO] [2019-10-14 02:57:17] Processing group of 2719 in 3 groups of 1000
[INFO] [2019-10-14 02:57:18] Average Time: 0.12
[INFO] [2019-10-14 02:57:18] Total Time: 1s
[STOP] [2019-10-14 02:57:18] parse_diff_and_store
[START] [2019-10-14 02:57:18] resolve_keys
[INFO] [2019-10-14 02:57:29] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-14 02:57:30] traits to occurrences...
[INFO] [2019-10-14 02:57:31] traits to nodes (through occurrences)...
[INFO] [2019-10-14 02:57:31] Traits to sex term...
[INFO] [2019-10-14 02:57:32] Traits to lifestage term...
[INFO] [2019-10-14 02:57:33] MetaTraits to traits...
[INFO] [2019-10-14 02:57:33] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-14 02:57:33] Assocs to occurrences...
[INFO] [2019-10-14 02:57:33] Assocs to nodes...
[INFO] [2019-10-14 02:57:33] Assoc to sex term...
[INFO] [2019-10-14 02:57:33] Assoc to lifestage term...
[STOP] [2019-10-14 02:57:33] resolve_keys
[START] [2019-10-14 02:57:33] hold_for_later_1
[STOP] [2019-10-14 02:57:33] hold_for_later_1
[START] [2019-10-14 02:57:33] hold_for_later_2
[STOP] [2019-10-14 02:57:33] hold_for_later_2
[START] [2019-10-14 02:57:33] resolve_missing_parents
[STOP] [2019-10-14 02:57:37] resolve_missing_parents
[START] [2019-10-14 02:57:37] rebuild_nodes
[START] [2019-10-14 02:57:37] Flattener#flatten
[START] [2019-10-14 02:57:37] Flattener#study_resource
[START] [2019-10-14 02:57:37] Flattener#build_ancestry
[STOP] [2019-10-14 02:57:37] Flattener#build_ancestry
[INFO] [2019-10-14 02:57:37] 2382 ancestry keys
[START] [2019-10-14 02:57:37] build_node_ancestors
[INFO] [2019-10-14 02:57:37] old ancestors deleted.
[STOP] [2019-10-14 02:57:37] build_node_ancestors
[START] [2019-10-14 02:57:37] Flattener#propagate_ancestor_ids
[STOP] [2019-10-14 02:57:38] Flattener#propagate_ancestor_ids
[STOP] [2019-10-14 02:57:38] Flattener#flatten
[STOP] [2019-10-14 02:57:38] rebuild_nodes
[START] [2019-10-14 02:57:38] resolve_missing_media_owners
[STOP] [2019-10-14 02:57:38] resolve_missing_media_owners
[START] [2019-10-14 02:57:38] sanitize_media_verbatims
[STOP] [2019-10-14 02:57:38] sanitize_media_verbatims
[START] [2019-10-14 02:57:38] queue_downloads
[STOP] [2019-10-14 02:57:38] queue_downloads
[START] [2019-10-14 02:57:38] parse_names
[WARN] [2019-10-14 02:57:38] I see 2382 names which still need to be parsed.
[STOP] [2019-10-14 02:57:40] parse_names
[START] [2019-10-14 02:57:40] denormalize_canonical_names_to_nodes
[STOP] [2019-10-14 02:57:40] denormalize_canonical_names_to_nodes
[START] [2019-10-14 02:57:40] match_nodes
[START] [2019-10-14 02:57:40] map_all_nodes_to_pages
[STOP] [2019-10-14 02:59:52] map_all_nodes_to_pages
[INFO] [2019-10-14 02:59:52] 187 Unmatched nodes (of 2382)! That's too many to output. First 10: Cypria lacustris (#50577290); Bradleycypris (#50577375); Bradleycypris obliqua (#50577374); Heterocypris incongruens (#50577902); Corvus monedula (#50577365); Anas querquedula (#50577565); Anas penelope (#50577727); Anas strepera (#50577994); Anas clypeata (#50578324); Parus lugubris (#50579309)
[START] [2019-10-14 02:59:52] update_nodes
[STOP] [2019-10-14 02:59:53] update_nodes
[STOP] [2019-10-14 02:59:53] match_nodes
[START] [2019-10-14 02:59:53] reindex_search
[STOP] [2019-10-14 02:59:58] reindex_search
[START] [2019-10-14 02:59:58] normalize_units
[STOP] [2019-10-14 02:59:58] normalize_units
[START] [2019-10-14 02:59:58] calculate_statistics
[STOP] [2019-10-14 02:59:58] calculate_statistics
[START] [2019-10-14 02:59:58] complete_harvest_instance
[START] [2019-10-14 02:59:58] overall_tsv_creation
[INFO] [2019-10-14 02:59:58] Processing group of 2382 in 1 batches of 10000
[INFO] [2019-10-14 03:00:53] 964 Traits (unfiltered)...
[INFO] [2019-10-14 03:01:07] 964 Traits (filtered)...
[INFO] [2019-10-14 03:01:07] 0 Associations (filtered)...
[INFO] [2019-10-14 03:01:45] 4820 metadata added.
[INFO] [2019-10-14 03:01:45] 0 metadata added.
[INFO] [2019-10-14 03:01:45] Average Time: 85.23
[INFO] [2019-10-14 03:01:45] Total Time: 1m48s
[STOP] [2019-10-14 03:01:45] overall_tsv_creation
[INFO] [2019-10-14 03:01:45] Done. Check your files:
[INFO] [2019-10-14 03:01:45] (2382 lines) /app/public/data/macedonia_sp_lis/publish_nodes.tsv
[INFO] [2019-10-14 03:01:45] (3299 lines) /app/public/data/macedonia_sp_lis/publish_node_ancestors.tsv
[INFO] [2019-10-14 03:01:45] (2382 lines) /app/public/data/macedonia_sp_lis/publish_scientific_names.tsv
[INFO] [2019-10-14 03:01:45] (965 lines) /app/public/data/macedonia_sp_lis/publish_traits.tsv
[INFO] [2019-10-14 03:01:45] (4821 lines) /app/public/data/macedonia_sp_lis/publish_metadata.tsv
[STOP] [2019-10-14 03:01:45] complete_harvest_instance
[START] [2019-10-14 03:01:45] completed
[STOP] [2019-10-14 03:01:45] completed
[STOP] [2019-10-14 03:01:45] logged process, took 282.95
Latest Process

14 Oct 02:57 EXITED currently processing 0