Harvest for Mozambique Channel Species List Created 23 Dec 18:52

Stage: completed
Fetched: 23 Dec 18:52
Validated: 23 Dec 18:52
Deltas Created 23 Dec 18:52
Units Normalized: 23 Dec 20:03
Ancestry Built: 23 Dec 18:56
Nodes Matched: 23 Dec 20:01
Names Parsed: 23 Dec 18:56
New Models Stored: 23 Dec 18:54
Indexed: 23 Dec 20:03
Completed: 23 Dec 20:09
Time to Harvest: 1 minute
Harvesting Log

(156 lines)
# Logfile created on 2019-12-23 18:52:19 -0500 by logger.rb/56815
[START] [2019-12-23 18:52:19] logged process
[START] [2019-12-23 18:52:19] create_harvest_instance
[STOP] [2019-12-23 18:52:20] create_harvest_instance
[START] [2019-12-23 18:52:20] fetch_files
[STOP] [2019-12-23 18:52:20] fetch_files
[START] [2019-12-23 18:52:20] validate_each_file
[STOP] [2019-12-23 18:52:22] validate_each_file
[START] [2019-12-23 18:52:22] convert_to_csv
[CMD] [2019-12-23 18:52:22] /usr/bin/sort /app/public/converted_csv/mozambique_chann_refs_19490.csv > /app/public/converted_csv/mozambique_chann_refs_19490.csv_sorted
[CMD] [2019-12-23 18:52:22] /usr/bin/sort /app/public/converted_csv/mozambique_chann_nodes_19491.csv > /app/public/converted_csv/mozambique_chann_nodes_19491.csv_sorted
[CMD] [2019-12-23 18:52:23] /usr/bin/sort /app/public/converted_csv/mozambique_chann_occurrences_19492.csv > /app/public/converted_csv/mozambique_chann_occurrences_19492.csv_sorted
[CMD] [2019-12-23 18:52:23] /usr/bin/sort /app/public/converted_csv/mozambique_chann_measurements_19493.csv > /app/public/converted_csv/mozambique_chann_measurements_19493.csv_sorted
[STOP] [2019-12-23 18:52:24] convert_to_csv
[START] [2019-12-23 18:52:24] calculate_delta
[CMD] [2019-12-23 18:52:24] echo "0a" > /app/public/diff/mozambique_chann_refs_19490.diff
[CMD] [2019-12-23 18:52:25] tail -n +1 /app/public/converted_csv/mozambique_chann_refs_19490.csv >> /app/public/diff/mozambique_chann_refs_19490.diff
[CMD] [2019-12-23 18:52:25] echo "." >> /app/public/diff/mozambique_chann_refs_19490.diff
[CMD] [2019-12-23 18:52:26] echo "0a" > /app/public/diff/mozambique_chann_nodes_19491.diff
[CMD] [2019-12-23 18:52:27] tail -n +1 /app/public/converted_csv/mozambique_chann_nodes_19491.csv >> /app/public/diff/mozambique_chann_nodes_19491.diff
[CMD] [2019-12-23 18:52:27] echo "." >> /app/public/diff/mozambique_chann_nodes_19491.diff
[CMD] [2019-12-23 18:52:28] echo "0a" > /app/public/diff/mozambique_chann_occurrences_19492.diff
[CMD] [2019-12-23 18:52:28] tail -n +1 /app/public/converted_csv/mozambique_chann_occurrences_19492.csv >> /app/public/diff/mozambique_chann_occurrences_19492.diff
[CMD] [2019-12-23 18:52:29] echo "." >> /app/public/diff/mozambique_chann_occurrences_19492.diff
[CMD] [2019-12-23 18:52:30] echo "0a" > /app/public/diff/mozambique_chann_measurements_19493.diff
[CMD] [2019-12-23 18:52:30] tail -n +1 /app/public/converted_csv/mozambique_chann_measurements_19493.csv >> /app/public/diff/mozambique_chann_measurements_19493.diff
[CMD] [2019-12-23 18:52:31] echo "." >> /app/public/diff/mozambique_chann_measurements_19493.diff
[STOP] [2019-12-23 18:52:32] calculate_delta
[START] [2019-12-23 18:52:32] parse_diff_and_store
[INFO] [2019-12-23 18:52:32] Loading refs diff file into memory (true lines)...
[INFO] [2019-12-23 18:52:33] Loading nodes diff file into memory (true lines)...
[INFO] [2019-12-23 18:52:40] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-12-23 18:52:42] Loading measurements diff file into memory (true lines)...
[INFO] [2019-12-23 18:53:39] Storing 2 References
[INFO] [2019-12-23 18:53:39] Processing group of 2 in 1 groups of 1000
[INFO] [2019-12-23 18:53:39] Average Time: 0.0
[INFO] [2019-12-23 18:53:39] Total Time: 1s
[INFO] [2019-12-23 18:53:39] Storing 17271 ScientificNames
[INFO] [2019-12-23 18:53:39] Processing group of 17271 in 18 groups of 1000
[INFO] [2019-12-23 18:53:46] Average Time: 0.387
[INFO] [2019-12-23 18:53:46] Total Time: 8s
[INFO] [2019-12-23 18:53:46] last 3 / first 3: 1.19
[INFO] [2019-12-23 18:53:46] Std.Dev: 0.11832159566199232; Max: 0.76
[INFO] [2019-12-23 18:53:46] Storing 17271 Nodes
[INFO] [2019-12-23 18:53:46] Processing group of 17271 in 18 groups of 1000
[INFO] [2019-12-23 18:53:51] Average Time: 0.289
[INFO] [2019-12-23 18:53:51] Total Time: 6s
[INFO] [2019-12-23 18:53:51] last 3 / first 3: 0.78
[INFO] [2019-12-23 18:53:51] Std.Dev: 0.05477225575051661; Max: 0.35
[INFO] [2019-12-23 18:53:51] Storing 10413 Occurrences
[INFO] [2019-12-23 18:53:51] Processing group of 10413 in 11 groups of 1000
[INFO] [2019-12-23 18:53:52] Average Time: 0.105
[INFO] [2019-12-23 18:53:52] Total Time: 2s
[INFO] [2019-12-23 18:53:52] last 3 / first 3: 0.94
[INFO] [2019-12-23 18:53:52] Std.Dev: 0.03162277660168379; Max: 0.16
[INFO] [2019-12-23 18:53:52] Storing 20826 TraitsReferences
[INFO] [2019-12-23 18:53:52] Processing group of 20826 in 21 groups of 1000
[INFO] [2019-12-23 18:53:54] Average Time: 0.103
[INFO] [2019-12-23 18:53:54] Total Time: 3s
[INFO] [2019-12-23 18:53:54] last 3 / first 3: 0.94
[INFO] [2019-12-23 18:53:54] Std.Dev: 0.03162277660168379; Max: 0.18
[INFO] [2019-12-23 18:53:54] Storing 20826 Traits
[INFO] [2019-12-23 18:53:54] Processing group of 20826 in 21 groups of 1000
[INFO] [2019-12-23 18:54:01] Average Time: 0.32
[INFO] [2019-12-23 18:54:01] Total Time: 7s
[INFO] [2019-12-23 18:54:01] last 3 / first 3: 0.73
[INFO] [2019-12-23 18:54:01] Std.Dev: 0.07071067811865475; Max: 0.62
[INFO] [2019-12-23 18:54:01] Storing 20823 MetaTraits
[INFO] [2019-12-23 18:54:01] Processing group of 20823 in 21 groups of 1000
[INFO] [2019-12-23 18:54:04] Average Time: 0.138
[INFO] [2019-12-23 18:54:04] Total Time: 4s
[INFO] [2019-12-23 18:54:04] last 3 / first 3: 1.51
[INFO] [2019-12-23 18:54:04] Std.Dev: 0.03162277660168379; Max: 0.24
[STOP] [2019-12-23 18:54:04] parse_diff_and_store
[START] [2019-12-23 18:54:04] resolve_keys
[INFO] [2019-12-23 18:55:10] Occurrences to nodes (through scientific_names)...
[INFO] [2019-12-23 18:55:15] traits to occurrences...
[INFO] [2019-12-23 18:55:21] traits to nodes (through occurrences)...
[INFO] [2019-12-23 18:55:22] Traits to sex term...
[INFO] [2019-12-23 18:55:27] Traits to lifestage term...
[INFO] [2019-12-23 18:55:34] MetaTraits to traits...
[INFO] [2019-12-23 18:55:35] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-12-23 18:55:39] Assocs to occurrences...
[INFO] [2019-12-23 18:55:39] Assocs to nodes...
[INFO] [2019-12-23 18:55:39] Assoc to sex term...
[INFO] [2019-12-23 18:55:39] Assoc to lifestage term...
[STOP] [2019-12-23 18:55:39] resolve_keys
[START] [2019-12-23 18:55:39] hold_for_later_1
[STOP] [2019-12-23 18:55:39] hold_for_later_1
[START] [2019-12-23 18:55:39] hold_for_later_2
[STOP] [2019-12-23 18:55:39] hold_for_later_2
[START] [2019-12-23 18:55:39] resolve_missing_parents
[STOP] [2019-12-23 18:56:09] resolve_missing_parents
[START] [2019-12-23 18:56:09] rebuild_nodes
[START] [2019-12-23 18:56:09] Flattener#flatten
[START] [2019-12-23 18:56:09] Flattener#study_resource
[START] [2019-12-23 18:56:10] Flattener#build_ancestry
[STOP] [2019-12-23 18:56:12] Flattener#build_ancestry
[INFO] [2019-12-23 18:56:12] 17271 ancestry keys
[START] [2019-12-23 18:56:12] build_node_ancestors
[INFO] [2019-12-23 18:56:12] old ancestors deleted.
[STOP] [2019-12-23 18:56:19] build_node_ancestors
[START] [2019-12-23 18:56:27] Flattener#propagate_ancestor_ids
[STOP] [2019-12-23 18:56:30] Flattener#propagate_ancestor_ids
[STOP] [2019-12-23 18:56:30] Flattener#flatten
[STOP] [2019-12-23 18:56:30] rebuild_nodes
[START] [2019-12-23 18:56:30] resolve_missing_media_owners
[STOP] [2019-12-23 18:56:30] resolve_missing_media_owners
[START] [2019-12-23 18:56:30] sanitize_media_verbatims
[STOP] [2019-12-23 18:56:30] sanitize_media_verbatims
[START] [2019-12-23 18:56:30] queue_downloads
[STOP] [2019-12-23 18:56:30] queue_downloads
[START] [2019-12-23 18:56:30] parse_names
[WARN] [2019-12-23 18:56:30] I see 17271 names which still need to be parsed.
[STOP] [2019-12-23 18:56:45] parse_names
[START] [2019-12-23 18:56:45] denormalize_canonical_names_to_nodes
[STOP] [2019-12-23 18:56:45] denormalize_canonical_names_to_nodes
[START] [2019-12-23 18:56:45] match_nodes
[START] [2019-12-23 18:56:45] map_all_nodes_to_pages
[STOP] [2019-12-23 20:01:45] map_all_nodes_to_pages
[INFO] [2019-12-23 20:01:45] 861 Unmatched nodes (of 17271)! That's too many to output. First 10: Fowleria auritus (#62132851); Cheilodipterus lineatus (#62118679); Chrysiptera glaucus (#62130645); Neopomacentrus inhacae (#62123992); Naso cavallo (#62124708); Naso coryphaenoides (#62125897); Naso baixopindae (#62128408); Odontanthias ornatus (#62126080); Amblygobius stagon (#62122755); Gobius bonti (#62123039)
[START] [2019-12-23 20:01:45] update_nodes
[STOP] [2019-12-23 20:01:52] update_nodes
[STOP] [2019-12-23 20:01:52] match_nodes
[START] [2019-12-23 20:01:52] reindex_search
[STOP] [2019-12-23 20:03:04] reindex_search
[START] [2019-12-23 20:03:04] normalize_units
[STOP] [2019-12-23 20:03:04] normalize_units
[START] [2019-12-23 20:03:04] calculate_statistics
[STOP] [2019-12-23 20:03:04] calculate_statistics
[START] [2019-12-23 20:03:04] complete_harvest_instance
[START] [2019-12-23 20:03:04] overall_tsv_creation
[INFO] [2019-12-23 20:03:05] Processing group of 17271 in 2 batches of 10000
[INFO] [2019-12-23 20:05:35] 5716 Traits (unfiltered)...
[INFO] [2019-12-23 20:05:49] 5716 Traits (filtered)...
[INFO] [2019-12-23 20:05:49] 0 Associations (filtered)...
[INFO] [2019-12-23 20:06:40] 28579 metadata added.
[INFO] [2019-12-23 20:06:40] 0 metadata added.
[INFO] [2019-12-23 20:08:21] 4697 Traits (unfiltered)...
[INFO] [2019-12-23 20:08:34] 4697 Traits (filtered)...
[INFO] [2019-12-23 20:08:34] 0 Associations (filtered)...
[INFO] [2019-12-23 20:09:25] 23483 metadata added.
[INFO] [2019-12-23 20:09:25] 0 metadata added.
[INFO] [2019-12-23 20:09:25] Average Time: 131.22
[INFO] [2019-12-23 20:09:25] Total Time: 6m21s
[STOP] [2019-12-23 20:09:25] overall_tsv_creation
[INFO] [2019-12-23 20:09:25] Done. Check your files:
[INFO] [2019-12-23 20:09:26] (17271 lines) /app/public/data/mozambique_chann/publish_nodes.tsv
[INFO] [2019-12-23 20:09:27] (93001 lines) /app/public/data/mozambique_chann/publish_node_ancestors.tsv
[INFO] [2019-12-23 20:09:27] (17271 lines) /app/public/data/mozambique_chann/publish_scientific_names.tsv
[INFO] [2019-12-23 20:09:28] (10414 lines) /app/public/data/mozambique_chann/publish_traits.tsv
[INFO] [2019-12-23 20:09:29] (52063 lines) /app/public/data/mozambique_chann/publish_metadata.tsv
[STOP] [2019-12-23 20:09:29] complete_harvest_instance
[START] [2019-12-23 20:09:29] completed
[STOP] [2019-12-23 20:09:29] completed
[STOP] [2019-12-23 20:09:29] logged process, took 4629.93
Latest Process

23 Dec 18:52 EXITED currently processing 0