Stage:
completed
Fetched:
28 May 07:48
Validated:
28 May 07:50
Deltas Created
28 May 07:50
Units Normalized:
01 Jun 15:08
Ancestry Built:
01 Jun 12:30
Nodes Matched:
01 Jun 15:03
Names Parsed:
01 Jun 12:33
New Models Stored:
01 Jun 11:39
Indexed:
01 Jun 15:07
Completed:
01 Jun 17:00
Time to Harvest:
7 minutes
Harvesting Log
(358 lines)
[INFO] [2021-05-28 07:48:24] Created harvest instance #3900
[STOP] [2021-05-28 07:48:24] create_harvest_instance
[START] [2021-05-28 07:48:24] fetch_files
[STOP] [2021-05-28 07:48:24] fetch_files
[START] [2021-05-28 07:48:24] validate_each_file
[INFO] [2021-05-28 07:48:24] Created new folder: /app/public/converted_csv
[INFO] [2021-05-28 07:48:24] Looping over 3 formats...
[INFO] [2021-05-28 07:48:24] ...nodes (/app/public/data/gnntrg2/taxon.tab)
[INFO] [2021-05-28 07:48:32] Valid: /app/public/converted_csv/gnntrg2_nodes_3900.csv (225863 lines)
[INFO] [2021-05-28 07:48:32] ...occurrences (/app/public/data/gnntrg2/occurrence.tab)
[INFO] [2021-05-28 07:48:55] Valid: /app/public/converted_csv/gnntrg2_occurrences_3900.csv (472519 lines)
[INFO] [2021-05-28 07:48:55] ...measurements (/app/public/data/gnntrg2/measurement_or_fact.tab)
[INFO] [2021-05-28 07:50:51] Valid: /app/public/converted_csv/gnntrg2_measurements_3900.csv (3222653 lines)
[STOP] [2021-05-28 07:50:51] validate_each_file
[START] [2021-05-28 07:50:51] convert_to_csv
[INFO] [2021-05-28 07:50:51] Looping over 3 formats...
[INFO] [2021-05-28 07:50:51] ...nodes (/app/public/data/gnntrg2/taxon.tab)
[CMD] [2021-05-28 07:50:51] /usr/bin/sort /app/public/converted_csv/gnntrg2_nodes_3900.csv > /app/public/converted_csv/gnntrg2_nodes_3900.csv_sorted
[INFO] [2021-05-28 07:50:52] Converted: /app/public/converted_csv/gnntrg2_nodes_3900.csv (225863 lines)
[INFO] [2021-05-28 07:50:52] ...occurrences (/app/public/data/gnntrg2/occurrence.tab)
[CMD] [2021-05-28 07:50:52] /usr/bin/sort /app/public/converted_csv/gnntrg2_occurrences_3900.csv > /app/public/converted_csv/gnntrg2_occurrences_3900.csv_sorted
[INFO] [2021-05-28 07:50:52] Converted: /app/public/converted_csv/gnntrg2_occurrences_3900.csv (472519 lines)
[INFO] [2021-05-28 07:50:52] ...measurements (/app/public/data/gnntrg2/measurement_or_fact.tab)
[CMD] [2021-05-28 07:50:52] /usr/bin/sort /app/public/converted_csv/gnntrg2_measurements_3900.csv > /app/public/converted_csv/gnntrg2_measurements_3900.csv_sorted
[INFO] [2021-05-28 07:50:56] Converted: /app/public/converted_csv/gnntrg2_measurements_3900.csv (3222653 lines)
[STOP] [2021-05-28 07:50:56] convert_to_csv
[START] [2021-05-28 07:50:56] calculate_delta
[INFO] [2021-05-28 07:50:56] Created diff dir: /app/public/diff
[INFO] [2021-05-28 07:50:56] Looping over 3 formats...
[INFO] [2021-05-28 07:50:56] ...nodes (/app/public/data/gnntrg2/taxon.tab)
[CMD] [2021-05-28 07:50:56] echo "0a" > /app/public/diff/gnntrg2_nodes_3900.diff
[CMD] [2021-05-28 07:50:56] tail -n +1 /app/public/converted_csv/gnntrg2_nodes_3900.csv >> /app/public/diff/gnntrg2_nodes_3900.diff
[CMD] [2021-05-28 07:50:56] echo "." >> /app/public/diff/gnntrg2_nodes_3900.diff
[INFO] [2021-05-28 07:50:56] Created diff: /app/public/diff/gnntrg2_nodes_3900.diff (225865 lines)
[INFO] [2021-05-28 07:50:56] ...occurrences (/app/public/data/gnntrg2/occurrence.tab)
[CMD] [2021-05-28 07:50:56] echo "0a" > /app/public/diff/gnntrg2_occurrences_3900.diff
[CMD] [2021-05-28 07:50:56] tail -n +1 /app/public/converted_csv/gnntrg2_occurrences_3900.csv >> /app/public/diff/gnntrg2_occurrences_3900.diff
[CMD] [2021-05-28 07:50:56] echo "." >> /app/public/diff/gnntrg2_occurrences_3900.diff
[INFO] [2021-05-28 07:50:57] Created diff: /app/public/diff/gnntrg2_occurrences_3900.diff (472521 lines)
[INFO] [2021-05-28 07:50:57] ...measurements (/app/public/data/gnntrg2/measurement_or_fact.tab)
[CMD] [2021-05-28 07:50:57] echo "0a" > /app/public/diff/gnntrg2_measurements_3900.diff
[CMD] [2021-05-28 07:50:57] tail -n +1 /app/public/converted_csv/gnntrg2_measurements_3900.csv >> /app/public/diff/gnntrg2_measurements_3900.diff
[CMD] [2021-05-28 07:50:57] echo "." >> /app/public/diff/gnntrg2_measurements_3900.diff
[INFO] [2021-05-28 07:50:59] Created diff: /app/public/diff/gnntrg2_measurements_3900.diff (3222655 lines)
[STOP] [2021-05-28 07:50:59] calculate_delta
[START] [2021-05-28 07:50:59] parse_diff_and_store
[INFO] [2021-05-28 07:50:59] Handling diff: /app/public/diff/gnntrg2_nodes_3900.diff (225865 lines)
[INFO] [2021-05-28 07:50:59] Loading nodes diff file into memory (225865 /app/public/diff/gnntrg2_nodes_3900.diff lines)...
[WARN] [2021-05-28 07:51:12] Filtered Scientific Name `Callarge occidentalis Leech, 1892/94` to `Callarge occidentalis Leech, 189294`
[WARN] [2021-05-28 07:51:15] Filtered Scientific Name `Charaxes brutus Cramer, 1779/80` to `Charaxes brutus Cramer, 177980`
[WARN] [2021-05-28 07:51:15] Filtered Scientific Name `Charaxes castor Cramer, 1775/76` to `Charaxes castor Cramer, 177576`
[WARN] [2021-05-28 07:51:19] Filtered Scientific Name `Coscinodiscus hustedtii Müller-Melchers, 1953/1955` to `Coscinodiscus hustedtii Müller-Melchers, 19531955`
[WARN] [2021-05-28 07:52:40] Filtered Scientific Name `Zophoessa elwesi Moore, 1890/92` to `Zophoessa elwesi Moore, 189092`
[INFO] [2021-05-28 07:52:42] Handling diff: /app/public/diff/gnntrg2_occurrences_3900.diff (472521 lines)
[INFO] [2021-05-28 07:52:43] Loading occurrences diff file into memory (472521 /app/public/diff/gnntrg2_occurrences_3900.diff lines)...
[INFO] [2021-05-28 07:59:52] Handling diff: /app/public/diff/gnntrg2_measurements_3900.diff (3222655 lines)
[INFO] [2021-05-28 07:59:52] Loading measurements diff file into memory (3222655 /app/public/diff/gnntrg2_measurements_3900.diff lines)...
[STOP] [2021-05-28 07:59:54] parse_diff_and_store
[ERR] [2021-05-28 07:59:54] RuntimeError
[ERR] [2021-05-28 07:59:54] Missing Term for URI `http://eol.org/schema/terms/hypotype`, must be added!
[ERR] [2021-05-28 07:59:54] ../models/store/model_builder.rb:653:in `fail_on_bad_uri'
[ERR] [2021-05-28 07:59:54] ../models/store/model_builder.rb:607:in `convert_trait_value'
[ERR] [2021-05-28 07:59:54] ../models/store/model_builder.rb:425:in `build_trait'
[ERR] [2021-05-28 07:59:54] ../models/store/model_builder.rb:28:in `build_models'
[ERR] [2021-05-28 07:59:54] ../models/resource_harvester.rb:353:in `block (3 levels) in parse_diff_and_store'
[ERR] [2021-05-28 07:59:54] ../models/csv_parser.rb:111:in `block in diff_as_hashes'
[ERR] [2021-05-28 07:59:54] ../models/csv_parser.rb:28:in `block in line_at_a_time'
[ERR] [2021-05-28 07:59:54] ../models/csv_parser.rb:25:in `line_at_a_time'
[ERR] [2021-05-28 07:59:54] ../models/csv_parser.rb:96:in `diff_as_hashes'
[ERR] [2021-05-28 07:59:54] ../models/resource_harvester.rb:307:in `block (2 levels) in parse_diff_and_store'
[ERR] [2021-05-28 07:59:54] ../models/logged_process.rb:86:in `enter_group'
[ERR] [2021-05-28 07:59:54] ../models/resource_harvester.rb:306:in `block in parse_diff_and_store'
[ERR] [2021-05-28 07:59:54] ../models/resource_harvester.rb:708:in `block in each_diff'
[ERR] [2021-05-28 07:59:54] ../models/resource_harvester.rb:695:in `each_diff'
[ERR] [2021-05-28 07:59:54] ../models/resource_harvester.rb:301:in `parse_diff_and_store'
[ERR] [2021-05-28 07:59:54] ../models/resource_harvester.rb:86:in `block (2 levels) in start'
[ERR] [2021-05-28 07:59:54] ../models/logged_process.rb:43:in `run_step'
[ERR] [2021-05-28 07:59:54] ../models/resource_harvester.rb:86:in `block in start'
[ERR] [2021-05-28 07:59:54] ../models/resource_harvester.rb:75:in `each_key'
[ERR] [2021-05-28 07:59:54] ../models/resource_harvester.rb:75:in `start'
[ERR] [2021-05-28 07:59:54] ../models/resource.rb:269:in `harvest'
[ERR] [2021-05-28 07:59:54] ../models/resource.rb:245:in `re_download_opendata_and_harvest'
[ERR] [2021-05-28 07:59:54] bin/rails:4:in `require'
[ERR] [2021-05-28 07:59:54] bin/rails:4:in `<main>'
[STOP] [2021-05-28 07:59:54] logged process, took 690.34
[INFO] [2021-06-01 10:10:27] ## HARVEST: type = resume_-harvest
[START] [2021-06-01 10:10:30] logged process: 35d8f449de20889d009b727cc37c78323a1d58ab always delete compiler, contributer, determined_by from trait attributes, even if they're destined for OccurrenceMetadata
[INFO] [2021-06-01 10:10:30] Already completed stage create_harvest_instance, skipping...
[INFO] [2021-06-01 10:10:30] Already completed stage fetch_files, skipping...
[INFO] [2021-06-01 10:10:30] Already completed stage validate_each_file, skipping...
[INFO] [2021-06-01 10:10:30] Already completed stage convert_to_csv, skipping...
[INFO] [2021-06-01 10:10:30] Already completed stage calculate_delta, skipping...
[START] [2021-06-01 10:10:30] parse_diff_and_store
[INFO] [2021-06-01 10:10:30] Handling diff: /app/public/diff/gnntrg2_nodes_3900.diff (225865 lines)
[INFO] [2021-06-01 10:10:30] Loading nodes diff file into memory (225865 /app/public/diff/gnntrg2_nodes_3900.diff lines)...
[WARN] [2021-06-01 10:10:43] Filtered Scientific Name `Callarge occidentalis Leech, 1892/94` to `Callarge occidentalis Leech, 189294`
[WARN] [2021-06-01 10:10:46] Filtered Scientific Name `Charaxes brutus Cramer, 1779/80` to `Charaxes brutus Cramer, 177980`
[WARN] [2021-06-01 10:10:46] Filtered Scientific Name `Charaxes castor Cramer, 1775/76` to `Charaxes castor Cramer, 177576`
[WARN] [2021-06-01 10:10:51] Filtered Scientific Name `Coscinodiscus hustedtii Müller-Melchers, 1953/1955` to `Coscinodiscus hustedtii Müller-Melchers, 19531955`
[WARN] [2021-06-01 10:12:11] Filtered Scientific Name `Zophoessa elwesi Moore, 1890/92` to `Zophoessa elwesi Moore, 189092`
[INFO] [2021-06-01 10:12:12] Handling diff: /app/public/diff/gnntrg2_occurrences_3900.diff (472521 lines)
[INFO] [2021-06-01 10:12:12] Loading occurrences diff file into memory (472521 /app/public/diff/gnntrg2_occurrences_3900.diff lines)...
[INFO] [2021-06-01 10:19:32] Handling diff: /app/public/diff/gnntrg2_measurements_3900.diff (3222655 lines)
[INFO] [2021-06-01 10:19:33] Loading measurements diff file into memory (3222655 /app/public/diff/gnntrg2_measurements_3900.diff lines)...
[INFO] [2021-06-01 11:01:35] Storing 269498 ScientificNames
[INFO] [2021-06-01 11:01:35] Processing group of 269498 in 270 groups of 1000
[INFO] [2021-06-01 11:04:13] Average Time: 0.577
[INFO] [2021-06-01 11:04:13] Total Time: 2m38s
[INFO] [2021-06-01 11:04:13] last 3 / first 3: 0.97
[INFO] [2021-06-01 11:04:13] Std.Dev: 1.3457340004622014; Max: 9.8
[INFO] [2021-06-01 11:04:13] Storing 269498 Nodes
[INFO] [2021-06-01 11:04:13] Processing group of 269498 in 270 groups of 1000
[INFO] [2021-06-01 11:07:51] Average Time: 0.802
[INFO] [2021-06-01 11:07:51] Total Time: 3m39s
[INFO] [2021-06-01 11:07:51] last 3 / first 3: 0.64
[INFO] [2021-06-01 11:07:51] Std.Dev: 3.1399044571451533; Max: 33.37
[INFO] [2021-06-01 11:07:51] Storing 472519 Occurrences
[INFO] [2021-06-01 11:07:51] Processing group of 472519 in 473 groups of 1000
[INFO] [2021-06-01 11:10:47] Average Time: 0.367
[INFO] [2021-06-01 11:10:47] Total Time: 2m57s
[INFO] [2021-06-01 11:10:47] last 3 / first 3: 0.72
[INFO] [2021-06-01 11:10:47] Std.Dev: 0.8927485648266258; Max: 5.86
[INFO] [2021-06-01 11:10:47] Storing 2642794 OccurrenceMetadata
[INFO] [2021-06-01 11:10:47] Processing group of 2642794 in 2643 groups of 1000
[INFO] [2021-06-01 11:20:04] Average Time: 0.204
[INFO] [2021-06-01 11:20:04] Total Time: 9m17s
[INFO] [2021-06-01 11:20:04] last 3 / first 3: 2.02
[INFO] [2021-06-01 11:20:04] Std.Dev: 0.5594640292279746; Max: 6.66
[INFO] [2021-06-01 11:20:04] Storing 1808453 Traits
[INFO] [2021-06-01 11:20:04] Processing group of 1808453 in 1809 groups of 1000
[INFO] [2021-06-01 11:38:32] Average Time: 0.605
[INFO] [2021-06-01 11:38:32] Total Time: 18m28s
[INFO] [2021-06-01 11:38:32] last 3 / first 3: 0.79
[INFO] [2021-06-01 11:38:32] Std.Dev: 1.5703502793962882; Max: 13.13
[INFO] [2021-06-01 11:38:32] Storing 109667 MetaTraits
[INFO] [2021-06-01 11:38:32] Processing group of 109667 in 110 groups of 1000
[INFO] [2021-06-01 11:39:32] Average Time: 0.53
[INFO] [2021-06-01 11:39:32] Total Time: 1m1s
[INFO] [2021-06-01 11:39:32] last 3 / first 3: 0.68
[INFO] [2021-06-01 11:39:32] Std.Dev: 1.6757088052522728; Max: 10.57
[STOP] [2021-06-01 11:39:32] parse_diff_and_store
[START] [2021-06-01 11:39:32] resolve_keys
[INFO] [2021-06-01 11:40:53] Occurrences to nodes (through scientific_names)...
[INFO] [2021-06-01 11:41:53] traits to occurrences...
[INFO] [2021-06-01 11:42:12] traits to nodes (through occurrences)...
[INFO] [2021-06-01 11:42:24] Traits to sex term...
[INFO] [2021-06-01 11:42:33] Traits to lifestage term...
[INFO] [2021-06-01 11:42:42] MetaTraits to traits...
[INFO] [2021-06-01 11:42:45] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-06-01 11:43:33] Assocs to occurrences...
[INFO] [2021-06-01 11:43:33] Assocs to nodes...
[INFO] [2021-06-01 11:43:33] Assoc to sex term...
[INFO] [2021-06-01 11:43:33] Assoc to lifestage term...
[INFO] [2021-06-01 11:43:33] MetaAssoc to assocs...
[STOP] [2021-06-01 11:43:33] resolve_keys
[START] [2021-06-01 11:43:33] hold_for_later_1
[STOP] [2021-06-01 11:43:33] hold_for_later_1
[START] [2021-06-01 11:43:33] hold_for_later_2
[STOP] [2021-06-01 11:43:33] hold_for_later_2
[START] [2021-06-01 11:43:33] resolve_missing_parents
[STOP] [2021-06-01 11:43:41] resolve_missing_parents
[START] [2021-06-01 11:43:41] rebuild_nodes
[START] [2021-06-01 11:43:41] Flattener#flatten
[START] [2021-06-01 11:43:41] Flattener#study_resource
[START] [2021-06-01 11:44:09] Flattener#build_ancestry
[STOP] [2021-06-01 12:25:47] Flattener#build_ancestry
[INFO] [2021-06-01 12:25:47] 269498 ancestry keys
[START] [2021-06-01 12:25:47] build_node_ancestors
[INFO] [2021-06-01 12:25:47] old ancestors deleted.
[STOP] [2021-06-01 12:29:21] build_node_ancestors
[START] [2021-06-01 12:29:24] Flattener#propagate_ancestor_ids
[STOP] [2021-06-01 12:30:01] Flattener#propagate_ancestor_ids
[STOP] [2021-06-01 12:30:01] Flattener#flatten
[STOP] [2021-06-01 12:30:01] rebuild_nodes
[START] [2021-06-01 12:30:01] resolve_missing_media_owners
[STOP] [2021-06-01 12:30:01] resolve_missing_media_owners
[START] [2021-06-01 12:30:01] sanitize_media_verbatims
[STOP] [2021-06-01 12:30:01] sanitize_media_verbatims
[START] [2021-06-01 12:30:01] queue_downloads
[STOP] [2021-06-01 12:30:01] queue_downloads
[START] [2021-06-01 12:30:01] parse_names
[WARN] [2021-06-01 12:30:01] I see 269498 names which still need to be parsed.
[WARN] [2021-06-01 12:33:29] I see 251 names which still need to be parsed.
[WARN] [2021-06-01 12:33:31] I see 11 names which still need to be parsed.
[WARN] [2021-06-01 12:33:33] I see 5 names which still need to be parsed.
[WARN] [2021-06-01 12:33:35] I see 4 names which still need to be parsed.
[WARN] [2021-06-01 12:33:36] I see 3 names which still need to be parsed.
[WARN] [2021-06-01 12:33:38] I see 2 names which still need to be parsed.
[WARN] [2021-06-01 12:33:40] I see 1 names which still need to be parsed.
[STOP] [2021-06-01 12:33:42] parse_names
[START] [2021-06-01 12:33:42] denormalize_canonical_names_to_nodes
[STOP] [2021-06-01 12:33:46] denormalize_canonical_names_to_nodes
[START] [2021-06-01 12:33:46] match_nodes
[START] [2021-06-01 12:33:46] map_all_nodes_to_pages
[STOP] [2021-06-01 15:02:40] map_all_nodes_to_pages
[INFO] [2021-06-01 15:02:40] 21232 Unmatched nodes (of 269498)! That's too many to output. Full list in /app/public/data/gnntrg2/unmatched_nodes.txt ; First 10: Canonical: Valeriana apiifolia; Node#95397776; ResourceID: Valeriana_apiifolia; Canonical: Valeriana officinalis officinalis; Node#95397845; ResourceID: Valeriana_officinalis_officinalis; Canonical: Valeriana organensis; Node#95397848; ResourceID: Valeriana_organensis; Canonical: Valeriana paterioides; Node#95397852; ResourceID: Valeriana_paterioides; Canonical: Valeriana polyclada; Node#95397855; ResourceID: Valeriana_polyclada; Canonical: Valeriana psychrophila; Node#95397857; ResourceID: Valeriana_psychrophila; Canonical: Valeriana punctata; Node#95397859; ResourceID: Valeriana_punctata; Canonical: Valeriana stenophylla; Node#95397873; ResourceID: Valeriana_stenophylla; Canonical: Valeriana stracheyi; Node#95397874; ResourceID: Valeriana_stracheyi; Canonical: Valeriana subincisa; Node#95397876; ResourceID: Valeriana_subincisa
[START] [2021-06-01 15:02:40] update_nodes
[STOP] [2021-06-01 15:03:12] update_nodes
[STOP] [2021-06-01 15:03:12] match_nodes
[START] [2021-06-01 15:03:12] reindex_search
[STOP] [2021-06-01 15:07:38] reindex_search
[START] [2021-06-01 15:07:38] normalize_units
[STOP] [2021-06-01 15:08:03] normalize_units
[START] [2021-06-01 15:08:03] calculate_statistics
[STOP] [2021-06-01 15:08:13] calculate_statistics
[START] [2021-06-01 15:08:13] complete_harvest_instance
[START] [2021-06-01 15:08:13] overall_tsv_creation
[INFO] [2021-06-01 15:08:14] Processing group of 269498 in 27 batches of 10000
[INFO] [2021-06-01 15:09:38] 17901 Traits (unfiltered)...
[INFO] [2021-06-01 15:11:34] 17901 Traits (filtered)...
[INFO] [2021-06-01 15:11:39] 0 Associations (filtered)...
[INFO] [2021-06-01 15:11:57] 147634 metadata added.
[INFO] [2021-06-01 15:11:57] 0 metadata added.
[INFO] [2021-06-01 15:13:39] 16011 Traits (unfiltered)...
[INFO] [2021-06-01 15:15:34] 16011 Traits (filtered)...
[INFO] [2021-06-01 15:15:37] 0 Associations (filtered)...
[INFO] [2021-06-01 15:15:51] 128753 metadata added.
[INFO] [2021-06-01 15:15:51] 0 metadata added.
[INFO] [2021-06-01 15:17:34] 17440 Traits (unfiltered)...
[INFO] [2021-06-01 15:19:37] 17440 Traits (filtered)...
[INFO] [2021-06-01 15:19:40] 0 Associations (filtered)...
[INFO] [2021-06-01 15:19:53] 146202 metadata added.
[INFO] [2021-06-01 15:19:53] 0 metadata added.
[INFO] [2021-06-01 15:21:41] 17994 Traits (unfiltered)...
[INFO] [2021-06-01 15:23:45] 17994 Traits (filtered)...
[INFO] [2021-06-01 15:23:48] 0 Associations (filtered)...
[INFO] [2021-06-01 15:24:04] 152977 metadata added.
[INFO] [2021-06-01 15:24:04] 0 metadata added.
[INFO] [2021-06-01 15:25:50] 16920 Traits (unfiltered)...
[INFO] [2021-06-01 15:27:49] 16920 Traits (filtered)...
[INFO] [2021-06-01 15:27:51] 0 Associations (filtered)...
[INFO] [2021-06-01 15:28:04] 139844 metadata added.
[INFO] [2021-06-01 15:28:04] 0 metadata added.
[INFO] [2021-06-01 15:29:52] 16543 Traits (unfiltered)...
[INFO] [2021-06-01 15:31:48] 16543 Traits (filtered)...
[INFO] [2021-06-01 15:31:50] 0 Associations (filtered)...
[INFO] [2021-06-01 15:32:05] 132965 metadata added.
[INFO] [2021-06-01 15:32:05] 0 metadata added.
[INFO] [2021-06-01 15:33:52] 16681 Traits (unfiltered)...
[INFO] [2021-06-01 15:35:48] 16681 Traits (filtered)...
[INFO] [2021-06-01 15:35:51] 0 Associations (filtered)...
[INFO] [2021-06-01 15:36:06] 134953 metadata added.
[INFO] [2021-06-01 15:36:06] 0 metadata added.
[INFO] [2021-06-01 15:37:52] 18482 Traits (unfiltered)...
[INFO] [2021-06-01 15:40:00] 18482 Traits (filtered)...
[INFO] [2021-06-01 15:40:02] 0 Associations (filtered)...
[INFO] [2021-06-01 15:40:21] 153679 metadata added.
[INFO] [2021-06-01 15:40:21] 0 metadata added.
[INFO] [2021-06-01 15:42:08] 18203 Traits (unfiltered)...
[INFO] [2021-06-01 15:44:16] 18203 Traits (filtered)...
[INFO] [2021-06-01 15:44:19] 0 Associations (filtered)...
[INFO] [2021-06-01 15:44:38] 154065 metadata added.
[INFO] [2021-06-01 15:44:38] 0 metadata added.
[INFO] [2021-06-01 15:46:26] 17006 Traits (unfiltered)...
[INFO] [2021-06-01 15:48:28] 17006 Traits (filtered)...
[INFO] [2021-06-01 15:48:31] 0 Associations (filtered)...
[INFO] [2021-06-01 15:48:50] 145994 metadata added.
[INFO] [2021-06-01 15:48:50] 0 metadata added.
[INFO] [2021-06-01 15:50:35] 17421 Traits (unfiltered)...
[INFO] [2021-06-01 15:52:41] 17421 Traits (filtered)...
[INFO] [2021-06-01 15:52:43] 0 Associations (filtered)...
[INFO] [2021-06-01 15:52:57] 143235 metadata added.
[INFO] [2021-06-01 15:52:57] 0 metadata added.
[INFO] [2021-06-01 15:54:45] 18072 Traits (unfiltered)...
[INFO] [2021-06-01 15:56:54] 18072 Traits (filtered)...
[INFO] [2021-06-01 15:56:56] 0 Associations (filtered)...
[INFO] [2021-06-01 15:57:15] 150063 metadata added.
[INFO] [2021-06-01 15:57:15] 0 metadata added.
[INFO] [2021-06-01 15:59:02] 16780 Traits (unfiltered)...
[INFO] [2021-06-01 16:01:00] 16780 Traits (filtered)...
[INFO] [2021-06-01 16:01:03] 0 Associations (filtered)...
[INFO] [2021-06-01 16:01:16] 135408 metadata added.
[INFO] [2021-06-01 16:01:16] 0 metadata added.
[INFO] [2021-06-01 16:03:04] 16261 Traits (unfiltered)...
[INFO] [2021-06-01 16:05:00] 16261 Traits (filtered)...
[INFO] [2021-06-01 16:05:06] 0 Associations (filtered)...
[INFO] [2021-06-01 16:05:19] 137171 metadata added.
[INFO] [2021-06-01 16:05:19] 0 metadata added.
[INFO] [2021-06-01 16:07:06] 17455 Traits (unfiltered)...
[INFO] [2021-06-01 16:09:09] 17455 Traits (filtered)...
[INFO] [2021-06-01 16:09:12] 0 Associations (filtered)...
[INFO] [2021-06-01 16:09:25] 141685 metadata added.
[INFO] [2021-06-01 16:09:25] 0 metadata added.
[INFO] [2021-06-01 16:11:14] 17600 Traits (unfiltered)...
[INFO] [2021-06-01 16:13:17] 17600 Traits (filtered)...
[INFO] [2021-06-01 16:13:20] 0 Associations (filtered)...
[INFO] [2021-06-01 16:13:38] 146276 metadata added.
[INFO] [2021-06-01 16:13:38] 0 metadata added.
[INFO] [2021-06-01 16:15:22] 18507 Traits (unfiltered)...
[INFO] [2021-06-01 16:17:30] 18507 Traits (filtered)...
[INFO] [2021-06-01 16:17:33] 0 Associations (filtered)...
[INFO] [2021-06-01 16:17:52] 154804 metadata added.
[INFO] [2021-06-01 16:17:52] 0 metadata added.
[INFO] [2021-06-01 16:19:37] 16211 Traits (unfiltered)...
[INFO] [2021-06-01 16:21:34] 16211 Traits (filtered)...
[INFO] [2021-06-01 16:21:36] 0 Associations (filtered)...
[INFO] [2021-06-01 16:21:53] 128602 metadata added.
[INFO] [2021-06-01 16:21:53] 0 metadata added.
[INFO] [2021-06-01 16:23:36] 17986 Traits (unfiltered)...
[INFO] [2021-06-01 16:25:40] 17986 Traits (filtered)...
[INFO] [2021-06-01 16:25:42] 0 Associations (filtered)...
[INFO] [2021-06-01 16:26:00] 144327 metadata added.
[INFO] [2021-06-01 16:26:00] 0 metadata added.
[INFO] [2021-06-01 16:27:48] 16261 Traits (unfiltered)...
[INFO] [2021-06-01 16:29:45] 16261 Traits (filtered)...
[INFO] [2021-06-01 16:29:48] 0 Associations (filtered)...
[INFO] [2021-06-01 16:30:03] 135004 metadata added.
[INFO] [2021-06-01 16:30:03] 0 metadata added.
[INFO] [2021-06-01 16:31:47] 19512 Traits (unfiltered)...
[INFO] [2021-06-01 16:34:03] 19512 Traits (filtered)...
[INFO] [2021-06-01 16:34:06] 0 Associations (filtered)...
[INFO] [2021-06-01 16:34:26] 163597 metadata added.
[INFO] [2021-06-01 16:34:26] 0 metadata added.
[INFO] [2021-06-01 16:36:13] 17336 Traits (unfiltered)...
[INFO] [2021-06-01 16:38:15] 17336 Traits (filtered)...
[INFO] [2021-06-01 16:38:18] 0 Associations (filtered)...
[INFO] [2021-06-01 16:38:35] 140681 metadata added.
[INFO] [2021-06-01 16:38:35] 0 metadata added.
[INFO] [2021-06-01 16:40:22] 17457 Traits (unfiltered)...
[INFO] [2021-06-01 16:42:27] 17457 Traits (filtered)...
[INFO] [2021-06-01 16:42:30] 0 Associations (filtered)...
[INFO] [2021-06-01 16:42:49] 147292 metadata added.
[INFO] [2021-06-01 16:42:49] 0 metadata added.
[INFO] [2021-06-01 16:44:36] 16397 Traits (unfiltered)...
[INFO] [2021-06-01 16:46:36] 16397 Traits (filtered)...
[INFO] [2021-06-01 16:46:39] 0 Associations (filtered)...
[INFO] [2021-06-01 16:46:52] 138912 metadata added.
[INFO] [2021-06-01 16:46:52] 0 metadata added.
[INFO] [2021-06-01 16:48:40] 17297 Traits (unfiltered)...
[INFO] [2021-06-01 16:50:44] 17297 Traits (filtered)...
[INFO] [2021-06-01 16:50:47] 0 Associations (filtered)...
[INFO] [2021-06-01 16:51:01] 145810 metadata added.
[INFO] [2021-06-01 16:51:01] 0 metadata added.
[INFO] [2021-06-01 16:52:51] 18230 Traits (unfiltered)...
[INFO] [2021-06-01 16:54:59] 18230 Traits (filtered)...
[INFO] [2021-06-01 16:55:02] 0 Associations (filtered)...
[INFO] [2021-06-01 16:55:21] 149818 metadata added.
[INFO] [2021-06-01 16:55:21] 0 metadata added.
[INFO] [2021-06-01 16:57:04] 20555 Traits (unfiltered)...
[INFO] [2021-06-01 16:59:22] 20555 Traits (filtered)...
[INFO] [2021-06-01 16:59:24] 0 Associations (filtered)...
[INFO] [2021-06-01 16:59:45] 164013 metadata added.
[INFO] [2021-06-01 16:59:45] 0 metadata added.
[INFO] [2021-06-01 17:00:12] Average Time: 215.208
[INFO] [2021-06-01 17:00:12] Total Time: 1h51m59s
[INFO] [2021-06-01 17:00:12] last 3 / first 3: 1.09
[INFO] [2021-06-01 17:00:12] Std.Dev: 7.739509028355739; Max: 230.6
[STOP] [2021-06-01 17:00:12] overall_tsv_creation
[INFO] [2021-06-01 17:00:12] Done. Check your files:
[INFO] [2021-06-01 17:00:14] (265303 lines) /app/public/data/gnntrg2/publish_nodes.tsv
[INFO] [2021-06-01 17:00:15] (1523767 lines) /app/public/data/gnntrg2/publish_node_ancestors.tsv
[INFO] [2021-06-01 17:00:16] (269498 lines) /app/public/data/gnntrg2/publish_scientific_names.tsv
[INFO] [2021-06-01 17:00:17] (472520 lines) /app/public/data/gnntrg2/publish_traits.tsv
[INFO] [2021-06-01 17:00:18] (3903765 lines) /app/public/data/gnntrg2/publish_metadata.tsv
[STOP] [2021-06-01 17:00:19] complete_harvest_instance
[START] [2021-06-01 17:00:19] completed
[STOP] [2021-06-01 17:00:19] completed
[STOP] [2021-06-01 17:00:19] logged process, took 24589.1
Latest Process