Stage:
completed
Fetched:
01 Oct 16:04
Validated:
01 Oct 16:04
Deltas Created
01 Oct 16:04
Units Normalized:
01 Oct 16:10
Ancestry Built:
01 Oct 16:06
Nodes Matched:
01 Oct 16:10
Names Parsed:
01 Oct 16:06
New Models Stored:
01 Oct 16:05
Indexed:
01 Oct 16:10
Completed:
01 Oct 16:12
Time to Harvest:
less than a minute
Harvesting Log
(149 lines)
# Logfile created on 2019-10-01 16:04:05 -0400 by logger.rb/56815
[START] [2019-10-01 16:04:05] logged process
[START] [2019-10-01 16:04:05] create_harvest_instance
[STOP] [2019-10-01 16:04:06] create_harvest_instance
[START] [2019-10-01 16:04:06] fetch_files
[STOP] [2019-10-01 16:04:06] fetch_files
[START] [2019-10-01 16:04:06] validate_each_file
[STOP] [2019-10-01 16:04:06] validate_each_file
[START] [2019-10-01 16:04:06] convert_to_csv
[CMD] [2019-10-01 16:04:06] /usr/bin/sort /app/public/converted_csv/afghanistan_sp_l_refs_14734.csv > /app/public/converted_csv/afghanistan_sp_l_refs_14734.csv_sorted
[CMD] [2019-10-01 16:04:08] /usr/bin/sort /app/public/converted_csv/afghanistan_sp_l_nodes_14735.csv > /app/public/converted_csv/afghanistan_sp_l_nodes_14735.csv_sorted
[CMD] [2019-10-01 16:04:09] /usr/bin/sort /app/public/converted_csv/afghanistan_sp_l_occurrences_14736.csv > /app/public/converted_csv/afghanistan_sp_l_occurrences_14736.csv_sorted
[CMD] [2019-10-01 16:04:11] /usr/bin/sort /app/public/converted_csv/afghanistan_sp_l_measurements_14737.csv > /app/public/converted_csv/afghanistan_sp_l_measurements_14737.csv_sorted
[STOP] [2019-10-01 16:04:12] convert_to_csv
[START] [2019-10-01 16:04:12] calculate_delta
[CMD] [2019-10-01 16:04:12] echo "0a" > /app/public/diff/afghanistan_sp_l_refs_14734.diff
[CMD] [2019-10-01 16:04:14] tail -n +1 /app/public/converted_csv/afghanistan_sp_l_refs_14734.csv >> /app/public/diff/afghanistan_sp_l_refs_14734.diff
[CMD] [2019-10-01 16:04:15] echo "." >> /app/public/diff/afghanistan_sp_l_refs_14734.diff
[CMD] [2019-10-01 16:04:17] echo "0a" > /app/public/diff/afghanistan_sp_l_nodes_14735.diff
[CMD] [2019-10-01 16:04:18] tail -n +1 /app/public/converted_csv/afghanistan_sp_l_nodes_14735.csv >> /app/public/diff/afghanistan_sp_l_nodes_14735.diff
[CMD] [2019-10-01 16:04:20] echo "." >> /app/public/diff/afghanistan_sp_l_nodes_14735.diff
[CMD] [2019-10-01 16:04:21] echo "0a" > /app/public/diff/afghanistan_sp_l_occurrences_14736.diff
[CMD] [2019-10-01 16:04:23] tail -n +1 /app/public/converted_csv/afghanistan_sp_l_occurrences_14736.csv >> /app/public/diff/afghanistan_sp_l_occurrences_14736.diff
[CMD] [2019-10-01 16:04:24] echo "." >> /app/public/diff/afghanistan_sp_l_occurrences_14736.diff
[CMD] [2019-10-01 16:04:26] echo "0a" > /app/public/diff/afghanistan_sp_l_measurements_14737.diff
[CMD] [2019-10-01 16:04:27] tail -n +1 /app/public/converted_csv/afghanistan_sp_l_measurements_14737.csv >> /app/public/diff/afghanistan_sp_l_measurements_14737.diff
[CMD] [2019-10-01 16:04:29] echo "." >> /app/public/diff/afghanistan_sp_l_measurements_14737.diff
[STOP] [2019-10-01 16:04:30] calculate_delta
[START] [2019-10-01 16:04:30] parse_diff_and_store
[INFO] [2019-10-01 16:04:32] Loading refs diff file into memory (true lines)...
[INFO] [2019-10-01 16:04:33] Loading nodes diff file into memory (true lines)...
[INFO] [2019-10-01 16:04:37] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-10-01 16:04:39] Loading measurements diff file into memory (true lines)...
[INFO] [2019-10-01 16:05:01] Storing 2 References
[INFO] [2019-10-01 16:05:01] Processing group of 2 in 1 groups of 1000
[INFO] [2019-10-01 16:05:01] Average Time: 0.0
[INFO] [2019-10-01 16:05:01] Total Time: 1s
[INFO] [2019-10-01 16:05:01] Storing 6184 ScientificNames
[INFO] [2019-10-01 16:05:01] Processing group of 6184 in 7 groups of 1000
[INFO] [2019-10-01 16:05:04] Average Time: 0.321
[INFO] [2019-10-01 16:05:04] Total Time: 3s
[INFO] [2019-10-01 16:05:04] last 3 / first 3: 0.69
[INFO] [2019-10-01 16:05:04] Std.Dev: 0.1224744871391589; Max: 0.45
[INFO] [2019-10-01 16:05:04] Storing 6184 Nodes
[INFO] [2019-10-01 16:05:04] Processing group of 6184 in 7 groups of 1000
[INFO] [2019-10-01 16:05:06] Average Time: 0.31
[INFO] [2019-10-01 16:05:06] Total Time: 3s
[INFO] [2019-10-01 16:05:06] last 3 / first 3: 1.06
[INFO] [2019-10-01 16:05:06] Std.Dev: 0.14832396974191325; Max: 0.57
[INFO] [2019-10-01 16:05:06] Storing 3856 Occurrences
[INFO] [2019-10-01 16:05:06] Processing group of 3856 in 4 groups of 1000
[INFO] [2019-10-01 16:05:06] Average Time: 0.1
[INFO] [2019-10-01 16:05:06] Total Time: 1s
[INFO] [2019-10-01 16:05:06] Storing 8244 TraitsReferences
[INFO] [2019-10-01 16:05:06] Processing group of 8244 in 9 groups of 1000
[INFO] [2019-10-01 16:05:07] Average Time: 0.07
[INFO] [2019-10-01 16:05:07] Total Time: 1s
[INFO] [2019-10-01 16:05:07] last 3 / first 3: 0.59
[INFO] [2019-10-01 16:05:07] Std.Dev: 0.03162277660168379; Max: 0.15
[INFO] [2019-10-01 16:05:07] Storing 8243 Traits
[INFO] [2019-10-01 16:05:07] Processing group of 8243 in 9 groups of 1000
[INFO] [2019-10-01 16:05:10] Average Time: 0.293
[INFO] [2019-10-01 16:05:10] Total Time: 3s
[INFO] [2019-10-01 16:05:10] last 3 / first 3: 0.68
[INFO] [2019-10-01 16:05:10] Std.Dev: 0.08944271909999159; Max: 0.41
[INFO] [2019-10-01 16:05:10] Storing 8244 MetaTraits
[INFO] [2019-10-01 16:05:10] Processing group of 8244 in 9 groups of 1000
[INFO] [2019-10-01 16:05:11] Average Time: 0.163
[INFO] [2019-10-01 16:05:11] Total Time: 2s
[INFO] [2019-10-01 16:05:11] last 3 / first 3: 0.41
[INFO] [2019-10-01 16:05:11] Std.Dev: 0.11832159566199232; Max: 0.45
[STOP] [2019-10-01 16:05:11] parse_diff_and_store
[START] [2019-10-01 16:05:11] resolve_keys
[INFO] [2019-10-01 16:05:39] Occurrences to nodes (through scientific_names)...
[INFO] [2019-10-01 16:05:42] traits to occurrences...
[INFO] [2019-10-01 16:05:44] traits to nodes (through occurrences)...
[INFO] [2019-10-01 16:05:44] Traits to sex term...
[INFO] [2019-10-01 16:05:45] Traits to lifestage term...
[INFO] [2019-10-01 16:05:47] MetaTraits to traits...
[INFO] [2019-10-01 16:05:47] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-10-01 16:05:49] Assocs to occurrences...
[INFO] [2019-10-01 16:05:49] Assocs to nodes...
[INFO] [2019-10-01 16:05:49] Assoc to sex term...
[INFO] [2019-10-01 16:05:49] Assoc to lifestage term...
[STOP] [2019-10-01 16:05:49] resolve_keys
[START] [2019-10-01 16:05:49] hold_for_later_1
[STOP] [2019-10-01 16:05:49] hold_for_later_1
[START] [2019-10-01 16:05:49] hold_for_later_2
[STOP] [2019-10-01 16:05:49] hold_for_later_2
[START] [2019-10-01 16:05:49] resolve_missing_parents
[STOP] [2019-10-01 16:06:01] resolve_missing_parents
[START] [2019-10-01 16:06:01] rebuild_nodes
[START] [2019-10-01 16:06:01] Flattener#flatten
[START] [2019-10-01 16:06:01] Flattener#study_resource
[START] [2019-10-01 16:06:01] Flattener#build_ancestry
[STOP] [2019-10-01 16:06:02] Flattener#build_ancestry
[INFO] [2019-10-01 16:06:02] 6184 ancestry keys
[START] [2019-10-01 16:06:02] build_node_ancestors
[INFO] [2019-10-01 16:06:02] old ancestors deleted.
[STOP] [2019-10-01 16:06:03] build_node_ancestors
[START] [2019-10-01 16:06:04] Flattener#propagate_ancestor_ids
[STOP] [2019-10-01 16:06:04] Flattener#propagate_ancestor_ids
[STOP] [2019-10-01 16:06:04] Flattener#flatten
[STOP] [2019-10-01 16:06:04] rebuild_nodes
[START] [2019-10-01 16:06:04] resolve_missing_media_owners
[STOP] [2019-10-01 16:06:04] resolve_missing_media_owners
[START] [2019-10-01 16:06:04] sanitize_media_verbatims
[STOP] [2019-10-01 16:06:04] sanitize_media_verbatims
[START] [2019-10-01 16:06:04] queue_downloads
[STOP] [2019-10-01 16:06:04] queue_downloads
[START] [2019-10-01 16:06:04] parse_names
[WARN] [2019-10-01 16:06:04] I see 6184 names which still need to be parsed.
[STOP] [2019-10-01 16:06:09] parse_names
[START] [2019-10-01 16:06:09] denormalize_canonical_names_to_nodes
[STOP] [2019-10-01 16:06:10] denormalize_canonical_names_to_nodes
[START] [2019-10-01 16:06:10] match_nodes
[START] [2019-10-01 16:06:10] map_all_nodes_to_pages
[STOP] [2019-10-01 16:10:00] map_all_nodes_to_pages
[INFO] [2019-10-01 16:10:00] 447 Unmatched nodes (of 6184)! That's too many to output. First 10: Streptopelia senegalensis (#47336966); Calandrella rufescens (#47337754); Delichon urbica (#47343104); Sturnus pagodarum (#47341850); Oenanthe alboniger (#47341686); Phoenicurus erythronota (#47342171); Erythropygia (#47338371); Erythropygia galactotes (#47338370); Sylvia margelanica (#47341719); Garrulax lineatus (#47342511)
[START] [2019-10-01 16:10:00] update_nodes
[STOP] [2019-10-01 16:10:02] update_nodes
[STOP] [2019-10-01 16:10:02] match_nodes
[START] [2019-10-01 16:10:02] reindex_search
[STOP] [2019-10-01 16:10:14] reindex_search
[START] [2019-10-01 16:10:14] normalize_units
[STOP] [2019-10-01 16:10:14] normalize_units
[START] [2019-10-01 16:10:14] calculate_statistics
[STOP] [2019-10-01 16:10:14] calculate_statistics
[START] [2019-10-01 16:10:14] complete_harvest_instance
[START] [2019-10-01 16:10:14] overall_tsv_creation
[INFO] [2019-10-01 16:10:14] Processing group of 6184 in 1 batches of 10000
[INFO] [2019-10-01 16:11:22] 3856 Traits (unfiltered)...
[INFO] [2019-10-01 16:11:35] 3856 Traits (filtered)...
[INFO] [2019-10-01 16:11:36] 0 Associations (filtered)...
[INFO] [2019-10-01 16:12:18] 19280 metadata added.
[INFO] [2019-10-01 16:12:18] 0 metadata added.
[INFO] [2019-10-01 16:12:18] Average Time: 99.39
[INFO] [2019-10-01 16:12:18] Total Time: 2m4s
[STOP] [2019-10-01 16:12:18] overall_tsv_creation
[INFO] [2019-10-01 16:12:18] Done. Check your files:
[INFO] [2019-10-01 16:12:19] (6184 lines) /app/public/data/afghanistan_sp_l/publish_nodes.tsv
[INFO] [2019-10-01 16:12:21] (15304 lines) /app/public/data/afghanistan_sp_l/publish_node_ancestors.tsv
[INFO] [2019-10-01 16:12:22] (6184 lines) /app/public/data/afghanistan_sp_l/publish_scientific_names.tsv
[INFO] [2019-10-01 16:12:24] (3857 lines) /app/public/data/afghanistan_sp_l/publish_traits.tsv
[INFO] [2019-10-01 16:12:26] (19281 lines) /app/public/data/afghanistan_sp_l/publish_metadata.tsv
[STOP] [2019-10-01 16:12:26] complete_harvest_instance
[START] [2019-10-01 16:12:26] completed
[STOP] [2019-10-01 16:12:26] completed
[STOP] [2019-10-01 16:12:26] logged process, took 500.63
Latest Process