Harvest for Barton Finkel et al Created 06 Nov 19:04

Stage: completed
Fetched: 06 Nov 19:04
Validated: 06 Nov 19:04
Deltas Created 06 Nov 19:05
Units Normalized: 06 Nov 19:05
Ancestry Built: 06 Nov 19:05
Nodes Matched: 06 Nov 19:05
Names Parsed: 06 Nov 19:05
New Models Stored: 06 Nov 19:05
Indexed: 06 Nov 19:05
Completed: 06 Nov 19:07
Time to Harvest: less than a minute

Expected File Format Definitions

Harvesting Log (most recent first)

# Logfile created on 2019-11-06 19:04:44 -0500 by logger.rb/56815
[START] [2019-11-06 19:04:44] logged process
[START] [2019-11-06 19:04:44] create_harvest_instance
[STOP] [2019-11-06 19:04:48] create_harvest_instance
[START] [2019-11-06 19:04:48] fetch_files
[STOP] [2019-11-06 19:04:48] fetch_files
[START] [2019-11-06 19:04:48] validate_each_file
[STOP] [2019-11-06 19:04:48] validate_each_file
[START] [2019-11-06 19:04:48] convert_to_csv
[CMD] [2019-11-06 19:04:48] /usr/bin/sort /app/public/converted_csv/Barton_Finkel_et_agents_18172.csv > /app/public/converted_csv/Barton_Finkel_et_agents_18172.csv_sorted
[CMD] [2019-11-06 19:04:48] /usr/bin/sort /app/public/converted_csv/Barton_Finkel_et_refs_18173.csv > /app/public/converted_csv/Barton_Finkel_et_refs_18173.csv_sorted
[CMD] [2019-11-06 19:04:49] /usr/bin/sort /app/public/converted_csv/Barton_Finkel_et_nodes_18174.csv > /app/public/converted_csv/Barton_Finkel_et_nodes_18174.csv_sorted
[CMD] [2019-11-06 19:04:49] /usr/bin/sort /app/public/converted_csv/Barton_Finkel_et_media_18175.csv > /app/public/converted_csv/Barton_Finkel_et_media_18175.csv_sorted
[CMD] [2019-11-06 19:04:50] /usr/bin/sort /app/public/converted_csv/Barton_Finkel_et_vernaculars_18176.csv > /app/public/converted_csv/Barton_Finkel_et_vernaculars_18176.csv_sorted
[CMD] [2019-11-06 19:04:50] /usr/bin/sort /app/public/converted_csv/Barton_Finkel_et_occurrences_18177.csv > /app/public/converted_csv/Barton_Finkel_et_occurrences_18177.csv_sorted
[CMD] [2019-11-06 19:04:50] /usr/bin/sort /app/public/converted_csv/Barton_Finkel_et_assocs_18178.csv > /app/public/converted_csv/Barton_Finkel_et_assocs_18178.csv_sorted
[CMD] [2019-11-06 19:04:51] /usr/bin/sort /app/public/converted_csv/Barton_Finkel_et_measurements_18179.csv > /app/public/converted_csv/Barton_Finkel_et_measurements_18179.csv_sorted
[STOP] [2019-11-06 19:04:51] convert_to_csv
[START] [2019-11-06 19:04:51] calculate_delta
[CMD] [2019-11-06 19:04:51] echo "0a" > /app/public/diff/Barton_Finkel_et_agents_18172.diff
[CMD] [2019-11-06 19:04:52] tail -n +1 /app/public/converted_csv/Barton_Finkel_et_agents_18172.csv >> /app/public/diff/Barton_Finkel_et_agents_18172.diff
[CMD] [2019-11-06 19:04:52] echo "." >> /app/public/diff/Barton_Finkel_et_agents_18172.diff
[CMD] [2019-11-06 19:04:52] echo "0a" > /app/public/diff/Barton_Finkel_et_refs_18173.diff
[CMD] [2019-11-06 19:04:53] tail -n +1 /app/public/converted_csv/Barton_Finkel_et_refs_18173.csv >> /app/public/diff/Barton_Finkel_et_refs_18173.diff
[CMD] [2019-11-06 19:04:53] echo "." >> /app/public/diff/Barton_Finkel_et_refs_18173.diff
[CMD] [2019-11-06 19:04:54] echo "0a" > /app/public/diff/Barton_Finkel_et_nodes_18174.diff
[CMD] [2019-11-06 19:04:54] tail -n +1 /app/public/converted_csv/Barton_Finkel_et_nodes_18174.csv >> /app/public/diff/Barton_Finkel_et_nodes_18174.diff
[CMD] [2019-11-06 19:04:54] echo "." >> /app/public/diff/Barton_Finkel_et_nodes_18174.diff
[CMD] [2019-11-06 19:04:55] echo "0a" > /app/public/diff/Barton_Finkel_et_media_18175.diff
[CMD] [2019-11-06 19:04:55] tail -n +1 /app/public/converted_csv/Barton_Finkel_et_media_18175.csv >> /app/public/diff/Barton_Finkel_et_media_18175.diff
[CMD] [2019-11-06 19:04:56] echo "." >> /app/public/diff/Barton_Finkel_et_media_18175.diff
[CMD] [2019-11-06 19:04:56] echo "0a" > /app/public/diff/Barton_Finkel_et_vernaculars_18176.diff
[CMD] [2019-11-06 19:04:56] tail -n +1 /app/public/converted_csv/Barton_Finkel_et_vernaculars_18176.csv >> /app/public/diff/Barton_Finkel_et_vernaculars_18176.diff
[CMD] [2019-11-06 19:04:57] echo "." >> /app/public/diff/Barton_Finkel_et_vernaculars_18176.diff
[CMD] [2019-11-06 19:04:57] echo "0a" > /app/public/diff/Barton_Finkel_et_occurrences_18177.diff
[CMD] [2019-11-06 19:04:57] tail -n +1 /app/public/converted_csv/Barton_Finkel_et_occurrences_18177.csv >> /app/public/diff/Barton_Finkel_et_occurrences_18177.diff
[CMD] [2019-11-06 19:04:58] echo "." >> /app/public/diff/Barton_Finkel_et_occurrences_18177.diff
[CMD] [2019-11-06 19:04:58] echo "0a" > /app/public/diff/Barton_Finkel_et_assocs_18178.diff
[CMD] [2019-11-06 19:04:59] tail -n +1 /app/public/converted_csv/Barton_Finkel_et_assocs_18178.csv >> /app/public/diff/Barton_Finkel_et_assocs_18178.diff
[CMD] [2019-11-06 19:04:59] echo "." >> /app/public/diff/Barton_Finkel_et_assocs_18178.diff
[CMD] [2019-11-06 19:05:00] echo "0a" > /app/public/diff/Barton_Finkel_et_measurements_18179.diff
[CMD] [2019-11-06 19:05:00] tail -n +1 /app/public/converted_csv/Barton_Finkel_et_measurements_18179.csv >> /app/public/diff/Barton_Finkel_et_measurements_18179.diff
[CMD] [2019-11-06 19:05:00] echo "." >> /app/public/diff/Barton_Finkel_et_measurements_18179.diff
[STOP] [2019-11-06 19:05:01] calculate_delta
[START] [2019-11-06 19:05:01] parse_diff_and_store
[INFO] [2019-11-06 19:05:01] Loading agents diff file into memory (true lines)...
[INFO] [2019-11-06 19:05:02] Loading refs diff file into memory (true lines)...
[INFO] [2019-11-06 19:05:02] Loading nodes diff file into memory (true lines)...
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Asterionella glacialis    ` to `Asterionella glacialis `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Asterionella kariana      ` to `Asterionella kariana `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Bacillaria paxillifer     ` to `Bacillaria paxillifer `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Biddulphia granulata      ` to `Biddulphia granulata `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Biddulphia regia  ` to `Biddulphia regia `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Cerataulina pelagica      ` to `Cerataulina pelagica `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Ceratium belone   ` to `Ceratium belone `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Ceratium buceros  ` to `Ceratium buceros `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Ceratium candelabrum      ` to `Ceratium candelabrum `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Ceratium falcatiforme     ` to `Ceratium falcatiforme `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Ceratium furca    ` to `Ceratium furca `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Ceratium fusus    ` to `Ceratium fusus `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Ceratium hexacanthum      ` to `Ceratium hexacanthum `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Ceratium lamellicorne     ` to `Ceratium lamellicorne `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Ceratium longirostrum     ` to `Ceratium longirostrum `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Ceratium massiliense      ` to `Ceratium massiliense `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Ceratium minutum  ` to `Ceratium minutum `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Ceratium teres    ` to `Ceratium teres `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Ceratium trichoceros      ` to `Ceratium trichoceros `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Ceratium tripos   ` to `Ceratium tripos `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Ceratium vultur   ` to `Ceratium vultur `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Ceratocorys  ` to `Ceratocorys `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Chaetoceros (Hyalochaete)    ` to `Chaetoceros (Hyalochaete) `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Chaetoceros (Phaeoceros)     ` to `Chaetoceros (Phaeoceros) `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Cladopyxis   ` to `Cladopyxis `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Coscinodiscus concinnus   ` to `Coscinodiscus concinnus `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Detonula confervacea      ` to `Detonula confervacea `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Dinophysis   ` to `Dinophysis `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Ditylum brightwellii      ` to `Ditylum brightwellii `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Exuviaella   ` to `Exuviaella `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Fragilaria   ` to `Fragilaria `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Gonyaulax    ` to `Gonyaulax `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Gyrosigma    ` to `Gyrosigma `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Hemiaulus    ` to `Hemiaulus `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Leptocylindrus danicus    ` to `Leptocylindrus danicus `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Navicula     ` to `Navicula `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Nitzschia closterium      ` to `Nitzschia closterium `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Nitzschia delicatissima   ` to `Nitzschia delicatissima `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Nitzschia    ` to `Nitzschia `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Odontella mobiliensis     ` to `Odontella mobiliensis `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Oxytoxum     ` to `Oxytoxum `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Paralia sulcata   ` to `Paralia sulcata `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Podolampas   ` to `Podolampas `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Pronoctiluca pelagica     ` to `Pronoctiluca pelagica `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Protoperidinium      ` to `Protoperidinium `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Pyrophacus   ` to `Pyrophacus `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Rhaphoneis amphiceros     ` to `Rhaphoneis amphiceros `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Rhizosolenia acuminata    ` to `Rhizosolenia acuminata `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Rhizosolenia alata alata  ` to `Rhizosolenia alata alata `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Rhizosolenia bergonii     ` to `Rhizosolenia bergonii `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Rhizosolenia calcar avis  ` to `Rhizosolenia calcar avis `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Rhizosolenia delicatula   ` to `Rhizosolenia delicatula `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Rhizosolenia hebetata semispina   ` to `Rhizosolenia hebetata semispina `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Rhizosolenia setigera     ` to `Rhizosolenia setigera `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Rhizosolenia styliformis  ` to `Rhizosolenia styliformis `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Schroederella delicatula  ` to `Schroederella delicatula `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Skeletonema costatum      ` to `Skeletonema costatum `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Stauroneis membranacea    ` to `Stauroneis membranacea `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Streptotheca tamesis      ` to `Streptotheca tamesis `
[WARN] [2019-11-06 19:05:02] Filtered Scientific Name `Surirella    ` to `Surirella `
[INFO] [2019-11-06 19:05:02] Loading media diff file into memory (true lines)...
[INFO] [2019-11-06 19:05:03] Loading vernaculars diff file into memory (true lines)...
[INFO] [2019-11-06 19:05:03] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-11-06 19:05:04] Loading assocs diff file into memory (true lines)...
[INFO] [2019-11-06 19:05:04] Loading measurements diff file into memory (true lines)...
[INFO] [2019-11-06 19:05:07] Storing 90 References
[INFO] [2019-11-06 19:05:07] Processing group of 90 in 1 groups of 1000
[INFO] [2019-11-06 19:05:07] Average Time: 0.03
[INFO] [2019-11-06 19:05:07] Total Time: 1s
[INFO] [2019-11-06 19:05:07] Storing 227 ScientificNames
[INFO] [2019-11-06 19:05:07] Processing group of 227 in 1 groups of 1000
[INFO] [2019-11-06 19:05:08] Average Time: 0.11
[INFO] [2019-11-06 19:05:08] Total Time: 1s
[INFO] [2019-11-06 19:05:08] Storing 227 Nodes
[INFO] [2019-11-06 19:05:08] Processing group of 227 in 1 groups of 1000
[INFO] [2019-11-06 19:05:08] Average Time: 0.07
[INFO] [2019-11-06 19:05:08] Total Time: 1s
[INFO] [2019-11-06 19:05:08] Storing 113 Occurrences
[INFO] [2019-11-06 19:05:08] Processing group of 113 in 1 groups of 1000
[INFO] [2019-11-06 19:05:08] Average Time: 0.01
[INFO] [2019-11-06 19:05:08] Total Time: 1s
[INFO] [2019-11-06 19:05:08] Storing 559 Traits
[INFO] [2019-11-06 19:05:08] Processing group of 559 in 1 groups of 1000
[INFO] [2019-11-06 19:05:08] Average Time: 0.18
[INFO] [2019-11-06 19:05:08] Total Time: 1s
[INFO] [2019-11-06 19:05:08] Storing 1377 MetaTraits
[INFO] [2019-11-06 19:05:08] Processing group of 1377 in 2 groups of 1000
[INFO] [2019-11-06 19:05:08] Average Time: 0.075
[INFO] [2019-11-06 19:05:08] Total Time: 1s
[INFO] [2019-11-06 19:05:08] Storing 1486 TraitsReferences
[INFO] [2019-11-06 19:05:08] Processing group of 1486 in 2 groups of 1000
[INFO] [2019-11-06 19:05:08] Average Time: 0.075
[INFO] [2019-11-06 19:05:08] Total Time: 1s
[STOP] [2019-11-06 19:05:08] parse_diff_and_store
[START] [2019-11-06 19:05:08] resolve_keys
[INFO] [2019-11-06 19:05:12] Occurrences to nodes (through scientific_names)...
[INFO] [2019-11-06 19:05:12] traits to occurrences...
[INFO] [2019-11-06 19:05:12] traits to nodes (through occurrences)...
[INFO] [2019-11-06 19:05:12] Traits to sex term...
[INFO] [2019-11-06 19:05:12] Traits to lifestage term...
[INFO] [2019-11-06 19:05:12] MetaTraits to traits...
[INFO] [2019-11-06 19:05:12] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-11-06 19:05:12] Assocs to occurrences...
[INFO] [2019-11-06 19:05:12] Assocs to nodes...
[INFO] [2019-11-06 19:05:12] Assoc to sex term...
[INFO] [2019-11-06 19:05:12] Assoc to lifestage term...
[STOP] [2019-11-06 19:05:12] resolve_keys
[START] [2019-11-06 19:05:12] hold_for_later_1
[STOP] [2019-11-06 19:05:12] hold_for_later_1
[START] [2019-11-06 19:05:12] hold_for_later_2
[STOP] [2019-11-06 19:05:12] hold_for_later_2
[START] [2019-11-06 19:05:12] resolve_missing_parents
[STOP] [2019-11-06 19:05:12] resolve_missing_parents
[START] [2019-11-06 19:05:12] rebuild_nodes
[START] [2019-11-06 19:05:12] Flattener#flatten
[START] [2019-11-06 19:05:12] Flattener#study_resource
[START] [2019-11-06 19:05:12] Flattener#build_ancestry
[STOP] [2019-11-06 19:05:12] Flattener#build_ancestry
[INFO] [2019-11-06 19:05:12] 227 ancestry keys
[START] [2019-11-06 19:05:12] build_node_ancestors
[INFO] [2019-11-06 19:05:12] old ancestors deleted.
[STOP] [2019-11-06 19:05:12] build_node_ancestors
[START] [2019-11-06 19:05:12] Flattener#propagate_ancestor_ids
[STOP] [2019-11-06 19:05:12] Flattener#propagate_ancestor_ids
[STOP] [2019-11-06 19:05:12] Flattener#flatten
[STOP] [2019-11-06 19:05:12] rebuild_nodes
[START] [2019-11-06 19:05:12] resolve_missing_media_owners
[STOP] [2019-11-06 19:05:12] resolve_missing_media_owners
[START] [2019-11-06 19:05:12] sanitize_media_verbatims
[STOP] [2019-11-06 19:05:12] sanitize_media_verbatims
[START] [2019-11-06 19:05:13] queue_downloads
[STOP] [2019-11-06 19:05:13] queue_downloads
[START] [2019-11-06 19:05:13] parse_names
[WARN] [2019-11-06 19:05:13] I see 227 names which still need to be parsed.
[WARN] [2019-11-06 19:05:14] I see 24 names which still need to be parsed.
[STOP] [2019-11-06 19:05:15] parse_names
[START] [2019-11-06 19:05:15] denormalize_canonical_names_to_nodes
[STOP] [2019-11-06 19:05:15] denormalize_canonical_names_to_nodes
[START] [2019-11-06 19:05:15] match_nodes
[START] [2019-11-06 19:05:15] map_all_nodes_to_pages
[STOP] [2019-11-06 19:05:39] map_all_nodes_to_pages
[INFO] [2019-11-06 19:05:39] 19 Unmatched nodes (of 227)! That's too many to output. First 10: Biddulphia aurita (#54346778); Biddulphia granulata (#54346779); Ceratium carriense (#54346796); Ceratium falcatum (#54346801); Ceratium gibberum (#54346804); Ceratium karstenii (#54346808); Ceratium longirostrum (#54346813); Ceratium praelongum (#54346820); Ceratium setaceum (#54346822); Ceratium vultur (#54346826)
[START] [2019-11-06 19:05:39] update_nodes
[STOP] [2019-11-06 19:05:39] update_nodes
[STOP] [2019-11-06 19:05:39] match_nodes
[START] [2019-11-06 19:05:39] reindex_search
[STOP] [2019-11-06 19:05:39] reindex_search
[START] [2019-11-06 19:05:39] normalize_units
[STOP] [2019-11-06 19:05:40] normalize_units
[START] [2019-11-06 19:05:40] calculate_statistics
[STOP] [2019-11-06 19:05:40] calculate_statistics
[START] [2019-11-06 19:05:40] complete_harvest_instance
[START] [2019-11-06 19:05:40] overall_tsv_creation
[INFO] [2019-11-06 19:05:40] Processing group of 227 in 1 batches of 10000
[INFO] [2019-11-06 19:06:25] 338 Traits (unfiltered)...
[INFO] [2019-11-06 19:06:38] 338 Traits (filtered)...
[INFO] [2019-11-06 19:06:38] 0 Associations (filtered)...
[INFO] [2019-11-06 19:07:15] 2866 metadata added.
[INFO] [2019-11-06 19:07:15] 0 metadata added.
[INFO] [2019-11-06 19:07:15] Average Time: 72.77
[INFO] [2019-11-06 19:07:15] Total Time: 1m35s
[STOP] [2019-11-06 19:07:15] overall_tsv_creation
[INFO] [2019-11-06 19:07:15] Done. Check your files:
[INFO] [2019-11-06 19:07:15] (226 lines) /app/public/data/Barton_Finkel_et/publish_nodes.tsv
[INFO] [2019-11-06 19:07:16] (477 lines) /app/public/data/Barton_Finkel_et/publish_node_ancestors.tsv
[INFO] [2019-11-06 19:07:16] (227 lines) /app/public/data/Barton_Finkel_et/publish_scientific_names.tsv
[INFO] [2019-11-06 19:07:16] (339 lines) /app/public/data/Barton_Finkel_et/publish_traits.tsv
[INFO] [2019-11-06 19:07:17] (2867 lines) /app/public/data/Barton_Finkel_et/publish_metadata.tsv
[STOP] [2019-11-06 19:07:17] complete_harvest_instance
[START] [2019-11-06 19:07:17] completed
[STOP] [2019-11-06 19:07:17] completed
[STOP] [2019-11-06 19:07:17] logged process, took 152.41

Latest Process