Harvest for Gabon Species List Created 22 Feb 12:15

Harvesting Log

# Logfile created on 2019-10-13 00:54:03 -0400 by logger.rb/56815
[START] [2019-10-13 00:54:03] logged process
[START] [2019-10-13 00:54:03] create_harvest_instance
[STOP] [2019-10-13 00:54:04] create_harvest_instance
[START] [2019-10-13 00:54:04] fetch_files
[STOP] [2019-10-13 00:54:04] fetch_files
[START] [2019-10-13 00:54:04] validate_each_file
[STOP] [2019-10-13 00:54:06] validate_each_file
[START] [2019-10-13 00:54:06] convert_to_csv
[CMD] [2019-10-13 00:54:06] /usr/bin/sort /app/public/converted_csv/gabon_sp_list_refs_15803.csv > /app/public/converted_csv/gabon_sp_list_refs_15803.csv_sorted
[CMD] [2019-10-13 00:54:06] /usr/bin/sort /app/public/converted_csv/gabon_sp_list_nodes_15804.csv > /app/public/converted_csv/gabon_sp_list_nodes_15804.csv_sorted
[CMD] [2019-10-13 00:54:06] /usr/bin/sort /app/public/converted_csv/gabon_sp_list_occurrences_15805.csv > /app/public/converted_csv/gabon_sp_list_occurrences_15805.csv_sorted
[CMD] [2019-10-13 00:54:06] /usr/bin/sort /app/public/converted_csv/gabon_sp_list_measurements_15806.csv > /app/public/converted_csv/gabon_sp_list_measurements_15806.csv_sorted
[STOP] [2019-10-13 00:54:06] convert_to_csv
[START] [2019-10-13 00:54:06] calculate_delta
[CMD] [2019-10-13 00:54:06] echo "0a" > /app/public/diff/gabon_sp_list_refs_15803.diff
[CMD] [2019-10-13 00:54:06] tail -n +1 /app/public/converted_csv/gabon_sp_list_refs_15803.csv >> /app/public/diff/gabon_sp_list_refs_15803.diff
[CMD] [2019-10-13 00:54:06] echo "." >> /app/public/diff/gabon_sp_list_refs_15803.diff
[CMD] [2019-10-13 00:54:06] echo "0a" > /app/public/diff/gabon_sp_list_nodes_15804.diff
[CMD] [2019-10-13 00:54:07] tail -n +1 /app/public/converted_csv/gabon_sp_list_nodes_15804.csv >> /app/public/diff/gabon_sp_list_nodes_15804.diff
[CMD] [2019-10-13 00:54:07] echo "." >> /app/public/diff/gabon_sp_list_nodes_15804.diff
[CMD] [2019-10-13 00:54:07] echo "0a" > /app/public/diff/gabon_sp_list_occurrences_15805.diff
[CMD] [2019-10-13 00:54:07] tail -n +1 /app/public/converted_csv/gabon_sp_list_occurrences_15805.csv >> /app/public/diff/gabon_sp_list_occurrences_15805.diff
[CMD] [2019-10-13 00:54:07] echo "." >> /app/public/diff/gabon_sp_list_occurrences_15805.diff
[CMD] [2019-10-13 00:54:07] echo "0a" > /app/public/diff/gabon_sp_list_measurements_15806.diff
[CMD] [2019-10-13 00:54:07] tail -n +1 /app/public/converted_csv/gabon_sp_list_measurements_15806.csv >> /app/public/diff/gabon_sp_list_measurements_15806.diff
[CMD] [2019-10-13 00:54:07] echo "." >> /app/public/diff/gabon_sp_list_measurements_15806.diff
[STOP] [2019-10-13 00:54:07] calculate_delta
[START] [2019-10-13 00:54:07] parse_diff_and_store
[INFO] [2019-10-13 00:54:07] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-13 00:54:07] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-13 00:54:12] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-13 00:54:15] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-13 00:55:03] Storing 2 References
[INFO] [2019-10-13 00:55:03] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-13 00:55:03] Average Time: 0.0
[INFO] [2019-10-13 00:55:03] Total Time: 1s
[INFO] [2019-10-13 00:55:03] Storing 11762 ScientificNames
[INFO] [2019-10-13 00:55:03] Processing group of 11762 in 12 groups of 1000
[INFO] [2019-10-13 00:55:08] Average Time: 0.365
[INFO] [2019-10-13 00:55:08] Total Time: 5s
[INFO] [2019-10-13 00:55:08] last 3 / first 3: 0.77
[INFO] [2019-10-13 00:55:08] Std.Dev: 0.044721359549995794; Max: 0.47
[INFO] [2019-10-13 00:55:08] Storing 11762 Nodes
[INFO] [2019-10-13 00:55:08] Processing group of 11762 in 12 groups of 1000
[INFO] [2019-10-13 00:55:12] Average Time: 0.336
[INFO] [2019-10-13 00:55:12] Total Time: 5s
[INFO] [2019-10-13 00:55:12] last 3 / first 3: 0.53
[INFO] [2019-10-13 00:55:12] Std.Dev: 0.16431676725154984; Max: 0.85
[INFO] [2019-10-13 00:55:12] Storing 8194 Occurrences
[INFO] [2019-10-13 00:55:12] Processing group of 8194 in 9 groups of 1000
[INFO] [2019-10-13 00:55:13] Average Time: 0.101
[INFO] [2019-10-13 00:55:13] Total Time: 1s
[INFO] [2019-10-13 00:55:13] last 3 / first 3: 0.61
[INFO] [2019-10-13 00:55:13] Std.Dev: 0.03162277660168379; Max: 0.15
[INFO] [2019-10-13 00:55:13] Storing 17020 TraitsReferences
[INFO] [2019-10-13 00:55:13] Processing group of 17020 in 18 groups of 1000
[INFO] [2019-10-13 00:55:14] Average Time: 0.069
[INFO] [2019-10-13 00:55:14] Total Time: 2s
[INFO] [2019-10-13 00:55:14] last 3 / first 3: 0.5
[INFO] [2019-10-13 00:55:14] Std.Dev: 0.03162277660168379; Max: 0.14
[INFO] [2019-10-13 00:55:14] Storing 17019 Traits
[INFO] [2019-10-13 00:55:14] Processing group of 17019 in 18 groups of 1000
[INFO] [2019-10-13 00:55:20] Average Time: 0.321
[INFO] [2019-10-13 00:55:20] Total Time: 6s
[INFO] [2019-10-13 00:55:20] last 3 / first 3: 0.57
[INFO] [2019-10-13 00:55:20] Std.Dev: 0.08366600265340755; Max: 0.43
[INFO] [2019-10-13 00:55:20] Storing 16998 MetaTraits
[INFO] [2019-10-13 00:55:20] Processing group of 16998 in 17 groups of 1000
[INFO] [2019-10-13 00:55:24] Average Time: 0.222
[INFO] [2019-10-13 00:55:24] Total Time: 4s
[INFO] [2019-10-13 00:55:24] last 3 / first 3: 0.98
[INFO] [2019-10-13 00:55:24] Std.Dev: 0.21908902300206645; Max: 0.83
[STOP] [2019-10-13 00:55:24] parse_diff_and_store
[START] [2019-10-13 00:55:24] resolve_keys
[INFO] [2019-10-13 00:56:14] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-13 00:56:18] traits to occurrences...
[INFO] [2019-10-13 00:56:24] traits to nodes (through occurrences)...
[INFO] [2019-10-13 00:56:24] Traits to sex term...
[INFO] [2019-10-13 00:56:29] Traits to lifestage term...
[INFO] [2019-10-13 00:56:34] MetaTraits to traits...
[INFO] [2019-10-13 00:56:35] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-13 00:56:38] Assocs to occurrences...
[INFO] [2019-10-13 00:56:38] Assocs to nodes...
[INFO] [2019-10-13 00:56:38] Assoc to sex term...
[INFO] [2019-10-13 00:56:38] Assoc to lifestage term...
[STOP] [2019-10-13 00:56:38] resolve_keys
[START] [2019-10-13 00:56:38] hold_for_later_1
[STOP] [2019-10-13 00:56:38] hold_for_later_1
[START] [2019-10-13 00:56:38] hold_for_later_2
[STOP] [2019-10-13 00:56:38] hold_for_later_2
[START] [2019-10-13 00:56:38] resolve_missing_parents
[STOP] [2019-10-13 00:57:00] resolve_missing_parents
[START] [2019-10-13 00:57:00] rebuild_nodes
[START] [2019-10-13 00:57:00] Flattener#flatten
[START] [2019-10-13 00:57:00] Flattener#study_resource
[START] [2019-10-13 00:57:00] Flattener#build_ancestry
[STOP] [2019-10-13 00:57:00] Flattener#build_ancestry
[INFO] [2019-10-13 00:57:00] 11762 ancestry keys
[START] [2019-10-13 00:57:00] build_node_ancestors
[INFO] [2019-10-13 00:57:00] old ancestors deleted.
[STOP] [2019-10-13 00:57:02] build_node_ancestors
[START] [2019-10-13 00:57:05] Flattener#propagate_ancestor_ids
[STOP] [2019-10-13 00:57:06] Flattener#propagate_ancestor_ids
[STOP] [2019-10-13 00:57:06] Flattener#flatten
[STOP] [2019-10-13 00:57:06] rebuild_nodes
[START] [2019-10-13 00:57:06] resolve_missing_media_owners
[STOP] [2019-10-13 00:57:06] resolve_missing_media_owners
[START] [2019-10-13 00:57:06] sanitize_media_verbatims
[STOP] [2019-10-13 00:57:06] sanitize_media_verbatims
[START] [2019-10-13 00:57:06] queue_downloads
[STOP] [2019-10-13 00:57:06] queue_downloads
[START] [2019-10-13 00:57:06] parse_names
[WARN] [2019-10-13 00:57:06] I see 11762 names which still need to be parsed.
[STOP] [2019-10-13 00:57:15] parse_names
[START] [2019-10-13 00:57:15] denormalize_canonical_names_to_nodes
[STOP] [2019-10-13 00:57:15] denormalize_canonical_names_to_nodes
[START] [2019-10-13 00:57:15] match_nodes
[START] [2019-10-13 00:57:15] map_all_nodes_to_pages
[STOP] [2019-10-13 01:05:48] map_all_nodes_to_pages
[INFO] [2019-10-13 01:05:48] 604 Unmatched nodes (of 11762)! That's too many to output. First 10: Dacryodes heterotrichus (#49788899); Trichoscypha reygaerti (#49780930); Trichoscypha guineensis (#49790292); Spondias cytherea (#49786847); Quassia gabunensis (#49788661); Quassia africanum (#49789642); Odyendea (#49780160); Odyendea gabunensis (#49780159); Hannoa (#49784869); Neoguarea ngounyensis (#49787007)
[START] [2019-10-13 01:05:48] update_nodes
[STOP] [2019-10-13 01:05:52] update_nodes
[STOP] [2019-10-13 01:05:52] match_nodes
[START] [2019-10-13 01:05:52] reindex_search
[STOP] [2019-10-13 01:06:15] reindex_search
[START] [2019-10-13 01:06:15] normalize_units
[STOP] [2019-10-13 01:06:15] normalize_units
[START] [2019-10-13 01:06:15] calculate_statistics
[STOP] [2019-10-13 01:06:15] calculate_statistics
[START] [2019-10-13 01:06:15] complete_harvest_instance
[START] [2019-10-13 01:06:15] overall_tsv_creation
[INFO] [2019-10-13 01:06:15] Processing group of 11762 in 2 batches of 10000
[INFO] [2019-10-13 01:07:44] 7090 Traits (unfiltered)...
[INFO] [2019-10-13 01:07:59] 7090 Traits (filtered)...
[INFO] [2019-10-13 01:07:59] 0 Associations (filtered)...
[INFO] [2019-10-13 01:08:53] 35433 metadata added.
[INFO] [2019-10-13 01:08:53] 0 metadata added.
[INFO] [2019-10-13 01:09:46] 1104 Traits (unfiltered)...
[INFO] [2019-10-13 01:10:00] 1104 Traits (filtered)...
[INFO] [2019-10-13 01:10:00] 0 Associations (filtered)...
[INFO] [2019-10-13 01:10:39] 5516 metadata added.
[INFO] [2019-10-13 01:10:39] 0 metadata added.
[INFO] [2019-10-13 01:10:39] Average Time: 107.135
[INFO] [2019-10-13 01:10:39] Total Time: 4m24s
[STOP] [2019-10-13 01:10:39] overall_tsv_creation
[INFO] [2019-10-13 01:10:39] Done. Check your files:
[INFO] [2019-10-13 01:10:39] (11762 lines) /app/public/data/gabon_sp_list/publish_nodes.tsv
[INFO] [2019-10-13 01:10:39] (27115 lines) /app/public/data/gabon_sp_list/publish_node_ancestors.tsv
[INFO] [2019-10-13 01:10:40] (11762 lines) /app/public/data/gabon_sp_list/publish_scientific_names.tsv
[INFO] [2019-10-13 01:10:40] (8195 lines) /app/public/data/gabon_sp_list/publish_traits.tsv
[INFO] [2019-10-13 01:10:40] (40950 lines) /app/public/data/gabon_sp_list/publish_metadata.tsv
[STOP] [2019-10-13 01:10:40] complete_harvest_instance
[START] [2019-10-13 01:10:40] completed
[STOP] [2019-10-13 01:10:40] completed
[STOP] [2019-10-13 01:10:40] logged process, took 996.79
[INFO] [2021-02-22 12:15:38] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-02-22 12:15:41] ## remove_type: ScientificName
[INFO] [2021-02-22 12:15:41] ++ Calling delete_all on 11762 instances...
[INFO] [2021-02-22 12:15:42] [12:15:42.457] Removed 11762 Scientificnames
[INFO] [2021-02-22 12:15:42] ## remove_type: Vernacular
[INFO] [2021-02-22 12:15:42] ++ Calling delete_all on 0 instances...
[INFO] [2021-02-22 12:15:42] [12:15:42.460] Removed 0 Vernaculars
[INFO] [2021-02-22 12:15:42] ## remove_type: Article
[INFO] [2021-02-22 12:15:42] ++ Calling delete_all on 0 instances...
[INFO] [2021-02-22 12:15:42] [12:15:42.463] Removed 0 Articles
[INFO] [2021-02-22 12:15:42] ## remove_type: Medium
[INFO] [2021-02-22 12:15:42] ++ Calling delete_all on 0 instances...
[INFO] [2021-02-22 12:15:42] [12:15:42.467] Removed 0 Media
[INFO] [2021-02-22 12:15:42] ## remove_type: Trait
[INFO] [2021-02-22 12:15:42] ++ Calling delete_all on 17019 instances...
[INFO] [2021-02-22 12:15:45] [12:15:45.692] Removed 17019 Traits
[INFO] [2021-02-22 12:15:45] ## remove_type: MetaTrait
[INFO] [2021-02-22 12:15:45] ++ Calling delete_all on 16998 instances...
[INFO] [2021-02-22 12:15:46] [12:15:46.578] Removed 16998 Metatraits
[INFO] [2021-02-22 12:15:46] ## remove_type: OccurrenceMetadatum
[INFO] [2021-02-22 12:15:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-02-22 12:15:46] [12:15:46.628] Removed 0 Occurrencemetadata
[INFO] [2021-02-22 12:15:46] ## remove_type: Assoc
[INFO] [2021-02-22 12:15:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-02-22 12:15:46] [12:15:46.631] Removed 0 Assocs
[INFO] [2021-02-22 12:15:46] ## remove_type: MetaAssoc
[INFO] [2021-02-22 12:15:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-02-22 12:15:46] [12:15:46.634] Removed 0 Metaassocs
[INFO] [2021-02-22 12:15:46] ## remove_type: Identifier
[INFO] [2021-02-22 12:15:46] ++ Calling delete_all on 0 instances...
[INFO] [2021-02-22 12:15:46] [12:15:46.659] Removed 0 Identifiers
[INFO] [2021-02-22 12:15:46] ## remove_type: Reference
[INFO] [2021-02-22 12:15:46] ++ Calling delete_all on 2 instances...
[INFO] [2021-02-22 12:15:46] [12:15:46.663] Removed 2 References
[INFO] [2021-02-22 12:15:47] Starting batch with ID 49784044...
[INFO] [2021-02-22 12:15:48] Starting batch with ID 49784044...
[INFO] [2021-02-22 12:15:49] Starting batch with ID 49781953...
[INFO] [2021-02-22 12:15:50] Starting batch with ID 49781133...
[INFO] [2021-02-22 12:15:50] Starting batch with ID 49781133...
[INFO] [2021-02-22 12:15:50] Starting batch with ID 49781133...
[INFO] [2021-02-22 12:15:51] Starting batch with ID 49781133...
[INFO] [2021-02-22 12:15:51] ## remove_type: Node
[INFO] [2021-02-22 12:15:51] ++ Calling delete_all on 11762 instances...
[INFO] [2021-02-22 12:15:52] [12:15:52.031] Removed 11762 Nodes
[START] [2021-02-22 12:15:59] logged process: cdc14cc7e396e8d099dbdcb2b6584e4ad6934982

[START] [2021-02-22 12:15:59] Creating resource from OpenData
[START] [2021-02-22 12:15:59] logged process: cdc14cc7e396e8d099dbdcb2b6584e4ad6934982

[START] [2021-02-22 12:15:59] Parse meta.xml file and create formats with fields
[STOP] [2021-02-22 12:15:59] Parse meta.xml file and create formats with fields
[STOP] [2021-02-22 12:15:59] Creating resource from OpenData
[START] [2021-02-22 12:15:59] logged process: cdc14cc7e396e8d099dbdcb2b6584e4ad6934982

[START] [2021-02-22 12:15:59] create_harvest_instance
[STOP] [2021-02-22 12:16:01] create_harvest_instance
[START] [2021-02-22 12:16:01] fetch_files
[STOP] [2021-02-22 12:16:01] fetch_files
[START] [2021-02-22 12:16:01] validate_each_file
[STOP] [2021-02-22 12:16:02] validate_each_file
[START] [2021-02-22 12:16:02] convert_to_csv
[CMD] [2021-02-22 12:16:02] /usr/bin/sort /app/public/converted_csv/gabon_sp_list_refs_26932.csv > /app/public/converted_csv/gabon_sp_list_refs_26932.csv_sorted
[CMD] [2021-02-22 12:16:02] /usr/bin/sort /app/public/converted_csv/gabon_sp_list_nodes_26933.csv > /app/public/converted_csv/gabon_sp_list_nodes_26933.csv_sorted
[CMD] [2021-02-22 12:16:02] /usr/bin/sort /app/public/converted_csv/gabon_sp_list_occurrences_26934.csv > /app/public/converted_csv/gabon_sp_list_occurrences_26934.csv_sorted
[CMD] [2021-02-22 12:16:02] /usr/bin/sort /app/public/converted_csv/gabon_sp_list_measurements_26935.csv > /app/public/converted_csv/gabon_sp_list_measurements_26935.csv_sorted
[STOP] [2021-02-22 12:16:02] convert_to_csv
[START] [2021-02-22 12:16:02] calculate_delta
[CMD] [2021-02-22 12:16:02] echo "0a" > /app/public/diff/gabon_sp_list_refs_26932.diff
[CMD] [2021-02-22 12:16:02] tail -n +1 /app/public/converted_csv/gabon_sp_list_refs_26932.csv >> /app/public/diff/gabon_sp_list_refs_26932.diff
[CMD] [2021-02-22 12:16:02] echo "." >> /app/public/diff/gabon_sp_list_refs_26932.diff
[CMD] [2021-02-22 12:16:02] echo "0a" > /app/public/diff/gabon_sp_list_nodes_26933.diff
[CMD] [2021-02-22 12:16:02] tail -n +1 /app/public/converted_csv/gabon_sp_list_nodes_26933.csv >> /app/public/diff/gabon_sp_list_nodes_26933.diff
[CMD] [2021-02-22 12:16:02] echo "." >> /app/public/diff/gabon_sp_list_nodes_26933.diff
[CMD] [2021-02-22 12:16:02] echo "0a" > /app/public/diff/gabon_sp_list_occurrences_26934.diff
[CMD] [2021-02-22 12:16:02] tail -n +1 /app/public/converted_csv/gabon_sp_list_occurrences_26934.csv >> /app/public/diff/gabon_sp_list_occurrences_26934.diff
[CMD] [2021-02-22 12:16:02] echo "." >> /app/public/diff/gabon_sp_list_occurrences_26934.diff
[CMD] [2021-02-22 12:16:02] echo "0a" > /app/public/diff/gabon_sp_list_measurements_26935.diff
[CMD] [2021-02-22 12:16:02] tail -n +1 /app/public/converted_csv/gabon_sp_list_measurements_26935.csv >> /app/public/diff/gabon_sp_list_measurements_26935.diff
[CMD] [2021-02-22 12:16:02] echo "." >> /app/public/diff/gabon_sp_list_measurements_26935.diff
[STOP] [2021-02-22 12:16:02] calculate_delta
[START] [2021-02-22 12:16:02] parse_diff_and_store
[INFO] [2021-02-22 12:16:02] Loading refs diff file into memory (true lines)...
[INFO] [2021-02-22 12:16:02] Loading nodes diff file into memory (true lines)...
[INFO] [2021-02-22 12:16:05] Loading occurrences diff file into memory (true lines)...
[INFO] [2021-02-22 12:16:06] Loading measurements diff file into memory (true lines)...
[INFO] [2021-02-22 12:16:13] Storing 2 References
[INFO] [2021-02-22 12:16:13] Processing group of 2 in 1 groups of 1000
[INFO] [2021-02-22 12:16:13] Average Time: 0.0
[INFO] [2021-02-22 12:16:13] Total Time: 1s
[INFO] [2021-02-22 12:16:13] Storing 11762 ScientificNames
[INFO] [2021-02-22 12:16:13] Processing group of 11762 in 12 groups of 1000
[INFO] [2021-02-22 12:16:17] Average Time: 0.304
[INFO] [2021-02-22 12:16:17] Total Time: 4s
[INFO] [2021-02-22 12:16:17] last 3 / first 3: 1.01
[INFO] [2021-02-22 12:16:17] Std.Dev: 0.05477225575051661; Max: 0.38
[INFO] [2021-02-22 12:16:17] Storing 11762 Nodes
[INFO] [2021-02-22 12:16:17] Processing group of 11762 in 12 groups of 1000
[INFO] [2021-02-22 12:16:20] Average Time: 0.267
[INFO] [2021-02-22 12:16:20] Total Time: 4s
[INFO] [2021-02-22 12:16:20] last 3 / first 3: 0.74
[INFO] [2021-02-22 12:16:20] Std.Dev: 0.044721359549995794; Max: 0.36
[INFO] [2021-02-22 12:16:20] Storing 8194 Occurrences
[INFO] [2021-02-22 12:16:20] Processing group of 8194 in 9 groups of 1000
[INFO] [2021-02-22 12:16:21] Average Time: 0.121
[INFO] [2021-02-22 12:16:21] Total Time: 2s
[INFO] [2021-02-22 12:16:21] last 3 / first 3: 0.44
[INFO] [2021-02-22 12:16:21] Std.Dev: 0.05477225575051661; Max: 0.23
[INFO] [2021-02-22 12:16:21] Storing 17020 TraitsReferences
[INFO] [2021-02-22 12:16:21] Processing group of 17020 in 18 groups of 1000
[INFO] [2021-02-22 12:16:23] Average Time: 0.069
[INFO] [2021-02-22 12:16:23] Total Time: 2s
[INFO] [2021-02-22 12:16:23] last 3 / first 3: 0.6
[INFO] [2021-02-22 12:16:23] Std.Dev: 0.03162277660168379; Max: 0.18
[INFO] [2021-02-22 12:16:23] Storing 17019 Traits
[INFO] [2021-02-22 12:16:23] Processing group of 17019 in 18 groups of 1000
[INFO] [2021-02-22 12:16:30] Average Time: 0.424
[INFO] [2021-02-22 12:16:30] Total Time: 8s
[INFO] [2021-02-22 12:16:30] last 3 / first 3: 0.56
[INFO] [2021-02-22 12:16:30] Std.Dev: 0.2; Max: 1.02
[INFO] [2021-02-22 12:16:30] Storing 8510 MetaTraits
[INFO] [2021-02-22 12:16:30] Processing group of 8510 in 9 groups of 1000
[INFO] [2021-02-22 12:16:32] Average Time: 0.134
[INFO] [2021-02-22 12:16:32] Total Time: 2s
[INFO] [2021-02-22 12:16:32] last 3 / first 3: 0.45
[INFO] [2021-02-22 12:16:32] Std.Dev: 0.1; Max: 0.39
[STOP] [2021-02-22 12:16:32] parse_diff_and_store
[START] [2021-02-22 12:16:32] resolve_keys
[INFO] [2021-02-22 12:16:58] Occurrences to nodes (through scientific_names)...
[INFO] [2021-02-22 12:17:01] traits to occurrences...
[INFO] [2021-02-22 12:17:05] traits to nodes (through occurrences)...
[INFO] [2021-02-22 12:17:05] Traits to sex term...
[INFO] [2021-02-22 12:17:08] Traits to lifestage term...
[INFO] [2021-02-22 12:17:10] MetaTraits to traits...
[INFO] [2021-02-22 12:17:10] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-02-22 12:17:11] Assocs to occurrences...
[INFO] [2021-02-22 12:17:11] Assocs to nodes...
[INFO] [2021-02-22 12:17:11] Assoc to sex term...
[INFO] [2021-02-22 12:17:11] Assoc to lifestage term...
[INFO] [2021-02-22 12:17:11] MetaAssoc to assocs...
[STOP] [2021-02-22 12:17:11] resolve_keys
[START] [2021-02-22 12:17:11] hold_for_later_1
[STOP] [2021-02-22 12:17:11] hold_for_later_1
[START] [2021-02-22 12:17:11] hold_for_later_2
[STOP] [2021-02-22 12:17:11] hold_for_later_2
[START] [2021-02-22 12:17:11] resolve_missing_parents
[STOP] [2021-02-22 12:17:16] resolve_missing_parents
[START] [2021-02-22 12:17:16] rebuild_nodes
[START] [2021-02-22 12:17:16] Flattener#flatten
[START] [2021-02-22 12:17:16] Flattener#study_resource
[START] [2021-02-22 12:17:17] Flattener#build_ancestry
[STOP] [2021-02-22 12:17:17] Flattener#build_ancestry
[INFO] [2021-02-22 12:17:17] 11762 ancestry keys
[START] [2021-02-22 12:17:17] build_node_ancestors
[INFO] [2021-02-22 12:17:17] old ancestors deleted.
[STOP] [2021-02-22 12:17:18] build_node_ancestors
[START] [2021-02-22 12:17:20] Flattener#propagate_ancestor_ids
[STOP] [2021-02-22 12:17:21] Flattener#propagate_ancestor_ids
[STOP] [2021-02-22 12:17:21] Flattener#flatten
[STOP] [2021-02-22 12:17:21] rebuild_nodes
[START] [2021-02-22 12:17:21] resolve_missing_media_owners
[STOP] [2021-02-22 12:17:21] resolve_missing_media_owners
[START] [2021-02-22 12:17:21] sanitize_media_verbatims
[STOP] [2021-02-22 12:17:21] sanitize_media_verbatims
[START] [2021-02-22 12:17:21] queue_downloads
[STOP] [2021-02-22 12:17:21] queue_downloads
[START] [2021-02-22 12:17:21] parse_names
[WARN] [2021-02-22 12:17:21] I see 11762 names which still need to be parsed.
[STOP] [2021-02-22 12:17:29] parse_names
[START] [2021-02-22 12:17:29] denormalize_canonical_names_to_nodes
[STOP] [2021-02-22 12:17:29] denormalize_canonical_names_to_nodes
[START] [2021-02-22 12:17:29] match_nodes
[START] [2021-02-22 12:17:29] map_all_nodes_to_pages
[STOP] [2021-02-22 12:28:59] map_all_nodes_to_pages
[INFO] [2021-02-22 12:28:59] 622 Unmatched nodes (of 11762)! That's too many to output. Full list in /app/public/data/gabon_sp_list/unmatched_nodes.txt ; First 10: Dacryodes heterotrichus (#89274008); Trichoscypha reygaerti (#89266039); Trichoscypha guineensis (#89275401); Spondias cytherea (#89271956); Quassia gabunensis (#89273770); Quassia africanum (#89274751); Odyendea (#89265269); Odyendea gabunensis (#89265268); Hannoa (#89269978); Neoguarea ngounyensis (#89272116)
[START] [2021-02-22 12:28:59] update_nodes
[STOP] [2021-02-22 12:29:04] update_nodes
[STOP] [2021-02-22 12:29:04] match_nodes
[START] [2021-02-22 12:29:04] reindex_search
[STOP] [2021-02-22 12:29:18] reindex_search
[START] [2021-02-22 12:29:18] normalize_units
[STOP] [2021-02-22 12:29:18] normalize_units
[START] [2021-02-22 12:29:18] calculate_statistics
[STOP] [2021-02-22 12:29:19] calculate_statistics
[START] [2021-02-22 12:29:19] complete_harvest_instance
[START] [2021-02-22 12:29:19] overall_tsv_creation
[INFO] [2021-02-22 12:29:19] Processing group of 11762 in 2 batches of 10000
[INFO] [2021-02-22 12:30:42] 7090 Traits (unfiltered)...
[INFO] [2021-02-22 12:32:05] 7090 Traits (filtered)...
[INFO] [2021-02-22 12:32:05] 0 Associations (filtered)...
[INFO] [2021-02-22 12:32:07] 21270 metadata added.
[INFO] [2021-02-22 12:32:07] 0 metadata added.
[INFO] [2021-02-22 12:33:20] 1104 Traits (unfiltered)...
[INFO] [2021-02-22 12:34:01] 1104 Traits (filtered)...
[INFO] [2021-02-22 12:34:01] 0 Associations (filtered)...
[INFO] [2021-02-22 12:34:01] 3312 metadata added.
[INFO] [2021-02-22 12:34:01] 0 metadata added.
[INFO] [2021-02-22 12:34:28] Average Time: 116.93
[INFO] [2021-02-22 12:34:28] Total Time: 5m9s
[STOP] [2021-02-22 12:34:28] overall_tsv_creation
[INFO] [2021-02-22 12:34:28] Done. Check your files:
[INFO] [2021-02-22 12:34:28] (11762 lines) /app/public/data/gabon_sp_list/publish_nodes.tsv
[INFO] [2021-02-22 12:34:28] (27115 lines) /app/public/data/gabon_sp_list/publish_node_ancestors.tsv
[INFO] [2021-02-22 12:34:28] (11762 lines) /app/public/data/gabon_sp_list/publish_scientific_names.tsv
[INFO] [2021-02-22 12:34:28] (8195 lines) /app/public/data/gabon_sp_list/publish_traits.tsv
[INFO] [2021-02-22 12:34:28] (24583 lines) /app/public/data/gabon_sp_list/publish_metadata.tsv
[STOP] [2021-02-22 12:34:28] complete_harvest_instance
[START] [2021-02-22 12:34:28] completed
[STOP] [2021-02-22 12:34:28] completed
[STOP] [2021-02-22 12:34:28] logged process, took 1108.78

Latest Process

22 Feb 12:15 EXITED currently processing 0