Harvest for Barton Finkel et al Created 24 Mar 09:58

Stage: completed
Fetched: 24 Mar 09:58
Validated: 24 Mar 09:58
Deltas Created 24 Mar 09:59
Units Normalized: 24 Mar 09:59
Ancestry Built: 24 Mar 09:59
Nodes Matched: 24 Mar 09:59
Names Parsed: 24 Mar 09:59
New Models Stored: 24 Mar 09:59
Indexed: 24 Mar 09:59
Completed: 24 Mar 10:01
Time to Harvest: less than a minute

Harvesting Log

(221 lines)
# Logfile created on 2020-03-24 09:58:53 -0400 by logger.rb/56815
[INFO] [2020-03-24 09:58:53] ## HARVEST: type = -harvest
[START] [2020-03-24 09:58:55] logged process
[START] [2020-03-24 09:58:55] create_harvest_instance
[STOP] [2020-03-24 09:58:59] create_harvest_instance
[START] [2020-03-24 09:58:59] fetch_files
[STOP] [2020-03-24 09:58:59] fetch_files
[START] [2020-03-24 09:58:59] validate_each_file
[STOP] [2020-03-24 09:58:59] validate_each_file
[START] [2020-03-24 09:58:59] convert_to_csv
[CMD] [2020-03-24 09:58:59] /usr/bin/sort /app/public/converted_csv/Barton_Finkel_et_agents_20473.csv > /app/public/converted_csv/Barton_Finkel_et_agents_20473.csv_sorted
[CMD] [2020-03-24 09:59:00] /usr/bin/sort /app/public/converted_csv/Barton_Finkel_et_refs_20474.csv > /app/public/converted_csv/Barton_Finkel_et_refs_20474.csv_sorted
[CMD] [2020-03-24 09:59:01] /usr/bin/sort /app/public/converted_csv/Barton_Finkel_et_nodes_20475.csv > /app/public/converted_csv/Barton_Finkel_et_nodes_20475.csv_sorted
[CMD] [2020-03-24 09:59:01] /usr/bin/sort /app/public/converted_csv/Barton_Finkel_et_media_20476.csv > /app/public/converted_csv/Barton_Finkel_et_media_20476.csv_sorted
[CMD] [2020-03-24 09:59:02] /usr/bin/sort /app/public/converted_csv/Barton_Finkel_et_vernaculars_20477.csv > /app/public/converted_csv/Barton_Finkel_et_vernaculars_20477.csv_sorted
[CMD] [2020-03-24 09:59:03] /usr/bin/sort /app/public/converted_csv/Barton_Finkel_et_occurrences_20478.csv > /app/public/converted_csv/Barton_Finkel_et_occurrences_20478.csv_sorted
[CMD] [2020-03-24 09:59:04] /usr/bin/sort /app/public/converted_csv/Barton_Finkel_et_assocs_20479.csv > /app/public/converted_csv/Barton_Finkel_et_assocs_20479.csv_sorted
[CMD] [2020-03-24 09:59:05] /usr/bin/sort /app/public/converted_csv/Barton_Finkel_et_measurements_20480.csv > /app/public/converted_csv/Barton_Finkel_et_measurements_20480.csv_sorted
[STOP] [2020-03-24 09:59:05] convert_to_csv
[START] [2020-03-24 09:59:05] calculate_delta
[CMD] [2020-03-24 09:59:05] echo "0a" > /app/public/diff/Barton_Finkel_et_agents_20473.diff
[CMD] [2020-03-24 09:59:06] tail -n +1 /app/public/converted_csv/Barton_Finkel_et_agents_20473.csv >> /app/public/diff/Barton_Finkel_et_agents_20473.diff
[CMD] [2020-03-24 09:59:07] echo "." >> /app/public/diff/Barton_Finkel_et_agents_20473.diff
[CMD] [2020-03-24 09:59:08] echo "0a" > /app/public/diff/Barton_Finkel_et_refs_20474.diff
[CMD] [2020-03-24 09:59:08] tail -n +1 /app/public/converted_csv/Barton_Finkel_et_refs_20474.csv >> /app/public/diff/Barton_Finkel_et_refs_20474.diff
[CMD] [2020-03-24 09:59:09] echo "." >> /app/public/diff/Barton_Finkel_et_refs_20474.diff
[CMD] [2020-03-24 09:59:10] echo "0a" > /app/public/diff/Barton_Finkel_et_nodes_20475.diff
[CMD] [2020-03-24 09:59:11] tail -n +1 /app/public/converted_csv/Barton_Finkel_et_nodes_20475.csv >> /app/public/diff/Barton_Finkel_et_nodes_20475.diff
[CMD] [2020-03-24 09:59:11] echo "." >> /app/public/diff/Barton_Finkel_et_nodes_20475.diff
[CMD] [2020-03-24 09:59:12] echo "0a" > /app/public/diff/Barton_Finkel_et_media_20476.diff
[CMD] [2020-03-24 09:59:13] tail -n +1 /app/public/converted_csv/Barton_Finkel_et_media_20476.csv >> /app/public/diff/Barton_Finkel_et_media_20476.diff
[CMD] [2020-03-24 09:59:14] echo "." >> /app/public/diff/Barton_Finkel_et_media_20476.diff
[CMD] [2020-03-24 09:59:15] echo "0a" > /app/public/diff/Barton_Finkel_et_vernaculars_20477.diff
[CMD] [2020-03-24 09:59:15] tail -n +1 /app/public/converted_csv/Barton_Finkel_et_vernaculars_20477.csv >> /app/public/diff/Barton_Finkel_et_vernaculars_20477.diff
[CMD] [2020-03-24 09:59:16] echo "." >> /app/public/diff/Barton_Finkel_et_vernaculars_20477.diff
[CMD] [2020-03-24 09:59:17] echo "0a" > /app/public/diff/Barton_Finkel_et_occurrences_20478.diff
[CMD] [2020-03-24 09:59:18] tail -n +1 /app/public/converted_csv/Barton_Finkel_et_occurrences_20478.csv >> /app/public/diff/Barton_Finkel_et_occurrences_20478.diff
[CMD] [2020-03-24 09:59:18] echo "." >> /app/public/diff/Barton_Finkel_et_occurrences_20478.diff
[CMD] [2020-03-24 09:59:19] echo "0a" > /app/public/diff/Barton_Finkel_et_assocs_20479.diff
[CMD] [2020-03-24 09:59:20] tail -n +1 /app/public/converted_csv/Barton_Finkel_et_assocs_20479.csv >> /app/public/diff/Barton_Finkel_et_assocs_20479.diff
[CMD] [2020-03-24 09:59:21] echo "." >> /app/public/diff/Barton_Finkel_et_assocs_20479.diff
[CMD] [2020-03-24 09:59:22] echo "0a" > /app/public/diff/Barton_Finkel_et_measurements_20480.diff
[CMD] [2020-03-24 09:59:22] tail -n +1 /app/public/converted_csv/Barton_Finkel_et_measurements_20480.csv >> /app/public/diff/Barton_Finkel_et_measurements_20480.diff
[CMD] [2020-03-24 09:59:23] echo "." >> /app/public/diff/Barton_Finkel_et_measurements_20480.diff
[STOP] [2020-03-24 09:59:24] calculate_delta
[START] [2020-03-24 09:59:24] parse_diff_and_store
[INFO] [2020-03-24 09:59:25] Loading agents diff file into memory (true lines)...
[INFO] [2020-03-24 09:59:25] Loading refs diff file into memory (true lines)...
[INFO] [2020-03-24 09:59:26] Loading nodes diff file into memory (true lines)...
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Asterionella glacialis    ` to `Asterionella glacialis `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Asterionella kariana      ` to `Asterionella kariana `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Bacillaria paxillifer     ` to `Bacillaria paxillifer `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Biddulphia granulata      ` to `Biddulphia granulata `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Biddulphia regia  ` to `Biddulphia regia `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Cerataulina pelagica      ` to `Cerataulina pelagica `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Ceratium belone   ` to `Ceratium belone `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Ceratium buceros  ` to `Ceratium buceros `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Ceratium candelabrum      ` to `Ceratium candelabrum `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Ceratium falcatiforme     ` to `Ceratium falcatiforme `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Ceratium furca    ` to `Ceratium furca `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Ceratium fusus    ` to `Ceratium fusus `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Ceratium hexacanthum      ` to `Ceratium hexacanthum `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Ceratium lamellicorne     ` to `Ceratium lamellicorne `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Ceratium longirostrum     ` to `Ceratium longirostrum `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Ceratium massiliense      ` to `Ceratium massiliense `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Ceratium minutum  ` to `Ceratium minutum `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Ceratium teres    ` to `Ceratium teres `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Ceratium trichoceros      ` to `Ceratium trichoceros `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Ceratium tripos   ` to `Ceratium tripos `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Ceratium vultur   ` to `Ceratium vultur `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Ceratocorys  ` to `Ceratocorys `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Chaetoceros (Hyalochaete)    ` to `Chaetoceros (Hyalochaete) `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Chaetoceros (Phaeoceros)     ` to `Chaetoceros (Phaeoceros) `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Cladopyxis   ` to `Cladopyxis `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Coscinodiscus concinnus   ` to `Coscinodiscus concinnus `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Detonula confervacea      ` to `Detonula confervacea `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Dinophysis   ` to `Dinophysis `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Ditylum brightwellii      ` to `Ditylum brightwellii `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Exuviaella   ` to `Exuviaella `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Fragilaria   ` to `Fragilaria `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Gonyaulax    ` to `Gonyaulax `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Gyrosigma    ` to `Gyrosigma `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Hemiaulus    ` to `Hemiaulus `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Leptocylindrus danicus    ` to `Leptocylindrus danicus `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Navicula     ` to `Navicula `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Nitzschia closterium      ` to `Nitzschia closterium `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Nitzschia delicatissima   ` to `Nitzschia delicatissima `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Nitzschia    ` to `Nitzschia `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Odontella mobiliensis     ` to `Odontella mobiliensis `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Oxytoxum     ` to `Oxytoxum `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Paralia sulcata   ` to `Paralia sulcata `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Podolampas   ` to `Podolampas `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Pronoctiluca pelagica     ` to `Pronoctiluca pelagica `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Protoperidinium      ` to `Protoperidinium `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Pyrophacus   ` to `Pyrophacus `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Rhaphoneis amphiceros     ` to `Rhaphoneis amphiceros `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Rhizosolenia acuminata    ` to `Rhizosolenia acuminata `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Rhizosolenia alata alata  ` to `Rhizosolenia alata alata `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Rhizosolenia bergonii     ` to `Rhizosolenia bergonii `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Rhizosolenia calcar avis  ` to `Rhizosolenia calcar avis `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Rhizosolenia delicatula   ` to `Rhizosolenia delicatula `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Rhizosolenia hebetata semispina   ` to `Rhizosolenia hebetata semispina `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Rhizosolenia setigera     ` to `Rhizosolenia setigera `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Rhizosolenia styliformis  ` to `Rhizosolenia styliformis `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Schroederella delicatula  ` to `Schroederella delicatula `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Skeletonema costatum      ` to `Skeletonema costatum `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Stauroneis membranacea    ` to `Stauroneis membranacea `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Streptotheca tamesis      ` to `Streptotheca tamesis `
[WARN] [2020-03-24 09:59:26] Filtered Scientific Name `Surirella    ` to `Surirella `
[INFO] [2020-03-24 09:59:27] Loading media diff file into memory (true lines)...
[INFO] [2020-03-24 09:59:28] Loading vernaculars diff file into memory (true lines)...
[INFO] [2020-03-24 09:59:29] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-03-24 09:59:29] Loading assocs diff file into memory (true lines)...
[INFO] [2020-03-24 09:59:30] Loading measurements diff file into memory (true lines)...
[INFO] [2020-03-24 09:59:34] Storing 90 References
[INFO] [2020-03-24 09:59:34] Processing group of 90 in 1 groups of 1000
[INFO] [2020-03-24 09:59:34] Average Time: 0.02
[INFO] [2020-03-24 09:59:34] Total Time: 1s
[INFO] [2020-03-24 09:59:34] Storing 227 ScientificNames
[INFO] [2020-03-24 09:59:34] Processing group of 227 in 1 groups of 1000
[INFO] [2020-03-24 09:59:34] Average Time: 0.14
[INFO] [2020-03-24 09:59:34] Total Time: 1s
[INFO] [2020-03-24 09:59:34] Storing 227 Nodes
[INFO] [2020-03-24 09:59:34] Processing group of 227 in 1 groups of 1000
[INFO] [2020-03-24 09:59:34] Average Time: 0.1
[INFO] [2020-03-24 09:59:34] Total Time: 1s
[INFO] [2020-03-24 09:59:34] Storing 113 Occurrences
[INFO] [2020-03-24 09:59:34] Processing group of 113 in 1 groups of 1000
[INFO] [2020-03-24 09:59:34] Average Time: 0.03
[INFO] [2020-03-24 09:59:34] Total Time: 1s
[INFO] [2020-03-24 09:59:34] Storing 559 Traits
[INFO] [2020-03-24 09:59:34] Processing group of 559 in 1 groups of 1000
[INFO] [2020-03-24 09:59:34] Average Time: 0.32
[INFO] [2020-03-24 09:59:34] Total Time: 1s
[INFO] [2020-03-24 09:59:34] Storing 1377 MetaTraits
[INFO] [2020-03-24 09:59:34] Processing group of 1377 in 2 groups of 1000
[INFO] [2020-03-24 09:59:35] Average Time: 0.1
[INFO] [2020-03-24 09:59:35] Total Time: 1s
[INFO] [2020-03-24 09:59:35] Storing 1486 TraitsReferences
[INFO] [2020-03-24 09:59:35] Processing group of 1486 in 2 groups of 1000
[INFO] [2020-03-24 09:59:35] Average Time: 0.11
[INFO] [2020-03-24 09:59:35] Total Time: 1s
[STOP] [2020-03-24 09:59:35] parse_diff_and_store
[START] [2020-03-24 09:59:35] resolve_keys
[INFO] [2020-03-24 09:59:41] Occurrences to nodes (through scientific_names)...
[INFO] [2020-03-24 09:59:41] traits to occurrences...
[INFO] [2020-03-24 09:59:41] traits to nodes (through occurrences)...
[INFO] [2020-03-24 09:59:41] Traits to sex term...
[INFO] [2020-03-24 09:59:41] Traits to lifestage term...
[INFO] [2020-03-24 09:59:41] MetaTraits to traits...
[INFO] [2020-03-24 09:59:41] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-03-24 09:59:41] Assocs to occurrences...
[INFO] [2020-03-24 09:59:41] Assocs to nodes...
[INFO] [2020-03-24 09:59:41] Assoc to sex term...
[INFO] [2020-03-24 09:59:41] Assoc to lifestage term...
[STOP] [2020-03-24 09:59:41] resolve_keys
[START] [2020-03-24 09:59:41] hold_for_later_1
[STOP] [2020-03-24 09:59:41] hold_for_later_1
[START] [2020-03-24 09:59:41] hold_for_later_2
[STOP] [2020-03-24 09:59:41] hold_for_later_2
[START] [2020-03-24 09:59:41] resolve_missing_parents
[STOP] [2020-03-24 09:59:41] resolve_missing_parents
[START] [2020-03-24 09:59:41] rebuild_nodes
[START] [2020-03-24 09:59:41] Flattener#flatten
[START] [2020-03-24 09:59:41] Flattener#study_resource
[START] [2020-03-24 09:59:41] Flattener#build_ancestry
[STOP] [2020-03-24 09:59:41] Flattener#build_ancestry
[INFO] [2020-03-24 09:59:41] 227 ancestry keys
[START] [2020-03-24 09:59:41] build_node_ancestors
[INFO] [2020-03-24 09:59:41] old ancestors deleted.
[STOP] [2020-03-24 09:59:41] build_node_ancestors
[START] [2020-03-24 09:59:41] Flattener#propagate_ancestor_ids
[STOP] [2020-03-24 09:59:41] Flattener#propagate_ancestor_ids
[STOP] [2020-03-24 09:59:41] Flattener#flatten
[STOP] [2020-03-24 09:59:41] rebuild_nodes
[START] [2020-03-24 09:59:41] resolve_missing_media_owners
[STOP] [2020-03-24 09:59:41] resolve_missing_media_owners
[START] [2020-03-24 09:59:41] sanitize_media_verbatims
[STOP] [2020-03-24 09:59:41] sanitize_media_verbatims
[START] [2020-03-24 09:59:41] queue_downloads
[STOP] [2020-03-24 09:59:41] queue_downloads
[START] [2020-03-24 09:59:41] parse_names
[WARN] [2020-03-24 09:59:41] I see 227 names which still need to be parsed.
[WARN] [2020-03-24 09:59:43] I see 24 names which still need to be parsed.
[STOP] [2020-03-24 09:59:44] parse_names
[START] [2020-03-24 09:59:44] denormalize_canonical_names_to_nodes
[STOP] [2020-03-24 09:59:44] denormalize_canonical_names_to_nodes
[START] [2020-03-24 09:59:44] match_nodes
[START] [2020-03-24 09:59:44] map_all_nodes_to_pages
[STOP] [2020-03-24 09:59:57] map_all_nodes_to_pages
[INFO] [2020-03-24 09:59:57] 20 Unmatched nodes (of 227)! That's too many to output. First 10: Biddulphia aurita (#67411083); Biddulphia granulata (#67411084); Ceratium carriense (#67411101); Ceratium falcatum (#67411106); Ceratium gibberum (#67411109); Ceratium karstenii (#67411113); Ceratium longirostrum (#67411118); Ceratium praelongum (#67411125); Ceratium setaceum (#67411127); Ceratium vultur (#67411131)
[START] [2020-03-24 09:59:57] update_nodes
[STOP] [2020-03-24 09:59:57] update_nodes
[STOP] [2020-03-24 09:59:57] match_nodes
[START] [2020-03-24 09:59:57] reindex_search
[STOP] [2020-03-24 09:59:57] reindex_search
[START] [2020-03-24 09:59:57] normalize_units
[STOP] [2020-03-24 09:59:58] normalize_units
[START] [2020-03-24 09:59:58] calculate_statistics
[STOP] [2020-03-24 09:59:58] calculate_statistics
[START] [2020-03-24 09:59:58] complete_harvest_instance
[START] [2020-03-24 09:59:58] overall_tsv_creation
[INFO] [2020-03-24 09:59:58] Processing group of 227 in 1 batches of 10000
[INFO] [2020-03-24 10:00:53] 338 Traits (unfiltered)...
[INFO] [2020-03-24 10:01:06] 338 Traits (filtered)...
[INFO] [2020-03-24 10:01:06] 0 Associations (filtered)...
[INFO] [2020-03-24 10:01:44] 2866 metadata added.
[INFO] [2020-03-24 10:01:44] 0 metadata added.
[INFO] [2020-03-24 10:01:44] Average Time: 74.2
[INFO] [2020-03-24 10:01:44] Total Time: 1m46s
[STOP] [2020-03-24 10:01:44] overall_tsv_creation
[INFO] [2020-03-24 10:01:44] Done. Check your files:
[INFO] [2020-03-24 10:01:45] (226 lines) /app/public/data/Barton_Finkel_et/publish_nodes.tsv
[INFO] [2020-03-24 10:01:45] (477 lines) /app/public/data/Barton_Finkel_et/publish_node_ancestors.tsv
[INFO] [2020-03-24 10:01:46] (227 lines) /app/public/data/Barton_Finkel_et/publish_scientific_names.tsv
[INFO] [2020-03-24 10:01:47] (339 lines) /app/public/data/Barton_Finkel_et/publish_traits.tsv
[INFO] [2020-03-24 10:01:48] (2867 lines) /app/public/data/Barton_Finkel_et/publish_metadata.tsv
[STOP] [2020-03-24 10:01:48] complete_harvest_instance
[START] [2020-03-24 10:01:48] completed
[STOP] [2020-03-24 10:01:48] completed
[STOP] [2020-03-24 10:01:48] logged process, took 172.79

Latest Process