Harvest for Malta Species List Created 14 Oct 04:57

Stage: completed
Fetched: 14 Oct 04:57
Validated: 14 Oct 04:57
Deltas Created 14 Oct 04:57
Units Normalized: 14 Oct 04:59
Ancestry Built: 14 Oct 04:57
Nodes Matched: 14 Oct 04:59
Names Parsed: 14 Oct 04:57
New Models Stored: 14 Oct 04:57
Indexed: 14 Oct 04:59
Completed: 14 Oct 05:01
Time to Harvest: less than a minute
Harvesting Log

(139 lines)
# Logfile created on 2019-10-14 04:57:27 -0400 by logger.rb/56815
[START] [2019-10-14 04:57:27] logged process
[START] [2019-10-14 04:57:27] create_harvest_instance
[STOP] [2019-10-14 04:57:27] create_harvest_instance
[START] [2019-10-14 04:57:27] fetch_files
[STOP] [2019-10-14 04:57:27] fetch_files
[START] [2019-10-14 04:57:27] validate_each_file
[STOP] [2019-10-14 04:57:27] validate_each_file
[START] [2019-10-14 04:57:27] convert_to_csv
[CMD] [2019-10-14 04:57:27] /usr/bin/sort /app/public/converted_csv/malta_sp_list_refs_16523.csv > /app/public/converted_csv/malta_sp_list_refs_16523.csv_sorted
[CMD] [2019-10-14 04:57:28] /usr/bin/sort /app/public/converted_csv/malta_sp_list_nodes_16524.csv > /app/public/converted_csv/malta_sp_list_nodes_16524.csv_sorted
[CMD] [2019-10-14 04:57:28] /usr/bin/sort /app/public/converted_csv/malta_sp_list_occurrences_16525.csv > /app/public/converted_csv/malta_sp_list_occurrences_16525.csv_sorted
[CMD] [2019-10-14 04:57:28] /usr/bin/sort /app/public/converted_csv/malta_sp_list_measurements_16526.csv > /app/public/converted_csv/malta_sp_list_measurements_16526.csv_sorted
[STOP] [2019-10-14 04:57:28] convert_to_csv
[START] [2019-10-14 04:57:28] calculate_delta
[CMD] [2019-10-14 04:57:28] echo "0a" > /app/public/diff/malta_sp_list_refs_16523.diff
[CMD] [2019-10-14 04:57:28] tail -n +1 /app/public/converted_csv/malta_sp_list_refs_16523.csv >> /app/public/diff/malta_sp_list_refs_16523.diff
[CMD] [2019-10-14 04:57:28] echo "." >> /app/public/diff/malta_sp_list_refs_16523.diff
[CMD] [2019-10-14 04:57:28] echo "0a" > /app/public/diff/malta_sp_list_nodes_16524.diff
[CMD] [2019-10-14 04:57:28] tail -n +1 /app/public/converted_csv/malta_sp_list_nodes_16524.csv >> /app/public/diff/malta_sp_list_nodes_16524.diff
[CMD] [2019-10-14 04:57:28] echo "." >> /app/public/diff/malta_sp_list_nodes_16524.diff
[CMD] [2019-10-14 04:57:28] echo "0a" > /app/public/diff/malta_sp_list_occurrences_16525.diff
[CMD] [2019-10-14 04:57:28] tail -n +1 /app/public/converted_csv/malta_sp_list_occurrences_16525.csv >> /app/public/diff/malta_sp_list_occurrences_16525.diff
[CMD] [2019-10-14 04:57:29] echo "." >> /app/public/diff/malta_sp_list_occurrences_16525.diff
[CMD] [2019-10-14 04:57:29] echo "0a" > /app/public/diff/malta_sp_list_measurements_16526.diff
[CMD] [2019-10-14 04:57:29] tail -n +1 /app/public/converted_csv/malta_sp_list_measurements_16526.csv >> /app/public/diff/malta_sp_list_measurements_16526.diff
[CMD] [2019-10-14 04:57:29] echo "." >> /app/public/diff/malta_sp_list_measurements_16526.diff
[STOP] [2019-10-14 04:57:29] calculate_delta
[START] [2019-10-14 04:57:29] parse_diff_and_store
[INFO] [2019-10-14 04:57:29] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-14 04:57:29] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-14 04:57:30] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-14 04:57:30] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-14 04:57:32] Storing 2 References
[INFO] [2019-10-14 04:57:32] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-14 04:57:32] Average Time: 0.0
[INFO] [2019-10-14 04:57:32] Total Time: 1s
[INFO] [2019-10-14 04:57:32] Storing 1110 ScientificNames
[INFO] [2019-10-14 04:57:32] Processing group of 1110 in 2 groups of 1000
[INFO] [2019-10-14 04:57:33] Average Time: 0.26
[INFO] [2019-10-14 04:57:33] Total Time: 1s
[INFO] [2019-10-14 04:57:33] Storing 1110 Nodes
[INFO] [2019-10-14 04:57:33] Processing group of 1110 in 2 groups of 1000
[INFO] [2019-10-14 04:57:33] Average Time: 0.17
[INFO] [2019-10-14 04:57:33] Total Time: 1s
[INFO] [2019-10-14 04:57:33] Storing 458 Occurrences
[INFO] [2019-10-14 04:57:33] Processing group of 458 in 1 groups of 1000
[INFO] [2019-10-14 04:57:33] Average Time: 0.07
[INFO] [2019-10-14 04:57:33] Total Time: 1s
[INFO] [2019-10-14 04:57:33] Storing 916 TraitsReferences
[INFO] [2019-10-14 04:57:33] Processing group of 916 in 1 groups of 1000
[INFO] [2019-10-14 04:57:34] Average Time: 0.13
[INFO] [2019-10-14 04:57:34] Total Time: 1s
[INFO] [2019-10-14 04:57:34] Storing 916 Traits
[INFO] [2019-10-14 04:57:34] Processing group of 916 in 1 groups of 1000
[INFO] [2019-10-14 04:57:34] Average Time: 0.41
[INFO] [2019-10-14 04:57:34] Total Time: 1s
[INFO] [2019-10-14 04:57:34] Storing 916 MetaTraits
[INFO] [2019-10-14 04:57:34] Processing group of 916 in 1 groups of 1000
[INFO] [2019-10-14 04:57:34] Average Time: 0.13
[INFO] [2019-10-14 04:57:34] Total Time: 1s
[STOP] [2019-10-14 04:57:34] parse_diff_and_store
[START] [2019-10-14 04:57:34] resolve_keys
[INFO] [2019-10-14 04:57:40] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-14 04:57:41] traits to occurrences...
[INFO] [2019-10-14 04:57:41] traits to nodes (through occurrences)...
[INFO] [2019-10-14 04:57:41] Traits to sex term...
[INFO] [2019-10-14 04:57:41] Traits to lifestage term...
[INFO] [2019-10-14 04:57:41] MetaTraits to traits...
[INFO] [2019-10-14 04:57:41] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-14 04:57:41] Assocs to occurrences...
[INFO] [2019-10-14 04:57:41] Assocs to nodes...
[INFO] [2019-10-14 04:57:41] Assoc to sex term...
[INFO] [2019-10-14 04:57:41] Assoc to lifestage term...
[STOP] [2019-10-14 04:57:41] resolve_keys
[START] [2019-10-14 04:57:41] hold_for_later_1
[STOP] [2019-10-14 04:57:41] hold_for_later_1
[START] [2019-10-14 04:57:41] hold_for_later_2
[STOP] [2019-10-14 04:57:41] hold_for_later_2
[START] [2019-10-14 04:57:41] resolve_missing_parents
[STOP] [2019-10-14 04:57:42] resolve_missing_parents
[START] [2019-10-14 04:57:42] rebuild_nodes
[START] [2019-10-14 04:57:42] Flattener#flatten
[START] [2019-10-14 04:57:42] Flattener#study_resource
[START] [2019-10-14 04:57:42] Flattener#build_ancestry
[STOP] [2019-10-14 04:57:42] Flattener#build_ancestry
[INFO] [2019-10-14 04:57:42] 1110 ancestry keys
[START] [2019-10-14 04:57:42] build_node_ancestors
[INFO] [2019-10-14 04:57:42] old ancestors deleted.
[STOP] [2019-10-14 04:57:43] build_node_ancestors
[START] [2019-10-14 04:57:43] Flattener#propagate_ancestor_ids
[STOP] [2019-10-14 04:57:43] Flattener#propagate_ancestor_ids
[STOP] [2019-10-14 04:57:43] Flattener#flatten
[STOP] [2019-10-14 04:57:43] rebuild_nodes
[START] [2019-10-14 04:57:43] resolve_missing_media_owners
[STOP] [2019-10-14 04:57:43] resolve_missing_media_owners
[START] [2019-10-14 04:57:43] sanitize_media_verbatims
[STOP] [2019-10-14 04:57:43] sanitize_media_verbatims
[START] [2019-10-14 04:57:43] queue_downloads
[STOP] [2019-10-14 04:57:43] queue_downloads
[START] [2019-10-14 04:57:43] parse_names
[WARN] [2019-10-14 04:57:43] I see 1110 names which still need to be parsed.
[STOP] [2019-10-14 04:57:45] parse_names
[START] [2019-10-14 04:57:45] denormalize_canonical_names_to_nodes
[STOP] [2019-10-14 04:57:45] denormalize_canonical_names_to_nodes
[START] [2019-10-14 04:57:45] match_nodes
[START] [2019-10-14 04:57:45] map_all_nodes_to_pages
[STOP] [2019-10-14 04:59:41] map_all_nodes_to_pages
[INFO] [2019-10-14 04:59:41] 46 Unmatched nodes (of 1110)! That's too many to output. First 10: Polyommatus coridon (#50653709); Syntarucus (#50653000); Argynnis aglaja (#50653721); Omicron regulus (#50653490); Halidamia (#50653176); Halidamia affinis (#50653175); Delichon urbica (#50653281); Phylloscopus sibillatrix (#50653006); Anas strepera (#50653258); Phorcus turbinata (#50653742)
[START] [2019-10-14 04:59:41] update_nodes
[STOP] [2019-10-14 04:59:41] update_nodes
[STOP] [2019-10-14 04:59:41] match_nodes
[START] [2019-10-14 04:59:41] reindex_search
[STOP] [2019-10-14 04:59:44] reindex_search
[START] [2019-10-14 04:59:44] normalize_units
[STOP] [2019-10-14 04:59:44] normalize_units
[START] [2019-10-14 04:59:44] calculate_statistics
[STOP] [2019-10-14 04:59:44] calculate_statistics
[START] [2019-10-14 04:59:44] complete_harvest_instance
[START] [2019-10-14 04:59:44] overall_tsv_creation
[INFO] [2019-10-14 04:59:44] Processing group of 1110 in 1 batches of 10000
[INFO] [2019-10-14 05:00:35] 458 Traits (unfiltered)...
[INFO] [2019-10-14 05:00:48] 458 Traits (filtered)...
[INFO] [2019-10-14 05:00:48] 0 Associations (filtered)...
[INFO] [2019-10-14 05:01:29] 2290 metadata added.
[INFO] [2019-10-14 05:01:29] 0 metadata added.
[INFO] [2019-10-14 05:01:29] Average Time: 82.81
[INFO] [2019-10-14 05:01:29] Total Time: 1m45s
[STOP] [2019-10-14 05:01:29] overall_tsv_creation
[INFO] [2019-10-14 05:01:29] Done. Check your files:
[INFO] [2019-10-14 05:01:29] (1110 lines) /app/public/data/malta_sp_list/publish_nodes.tsv
[INFO] [2019-10-14 05:01:29] (5295 lines) /app/public/data/malta_sp_list/publish_node_ancestors.tsv
[INFO] [2019-10-14 05:01:29] (1110 lines) /app/public/data/malta_sp_list/publish_scientific_names.tsv
[INFO] [2019-10-14 05:01:29] (459 lines) /app/public/data/malta_sp_list/publish_traits.tsv
[INFO] [2019-10-14 05:01:29] (2291 lines) /app/public/data/malta_sp_list/publish_metadata.tsv
[STOP] [2019-10-14 05:01:29] complete_harvest_instance
[START] [2019-10-14 05:01:29] completed
[STOP] [2019-10-14 05:01:29] completed
[STOP] [2019-10-14 05:01:29] logged process, took 242.43
Latest Process

14 Oct 04:57 EXITED currently processing 0