Stage:
completed
Fetched:
24 Dec 08:15
Validated:
24 Dec 08:15
Deltas Created
24 Dec 08:16
Units Normalized:
24 Dec 15:22
Ancestry Built:
24 Dec 08:40
Nodes Matched:
24 Dec 15:16
Names Parsed:
24 Dec 08:42
New Models Stored:
24 Dec 08:31
Indexed:
24 Dec 15:21
Completed:
24 Dec 15:56
Time to Harvest:
8 minutes
Harvesting Log
(217 lines)
# Logfile created on 2019-12-24 08:15:42 -0500 by logger.rb/56815
[START] [2019-12-24 08:15:42] logged process
[START] [2019-12-24 08:15:42] create_harvest_instance
[STOP] [2019-12-24 08:15:43] create_harvest_instance
[START] [2019-12-24 08:15:43] fetch_files
[STOP] [2019-12-24 08:15:43] fetch_files
[START] [2019-12-24 08:15:43] validate_each_file
[STOP] [2019-12-24 08:15:59] validate_each_file
[START] [2019-12-24 08:15:59] convert_to_csv
[CMD] [2019-12-24 08:15:59] /usr/bin/sort /app/public/converted_csv/North_Pacific_Sp_refs_19538.csv > /app/public/converted_csv/North_Pacific_Sp_refs_19538.csv_sorted
[CMD] [2019-12-24 08:15:59] /usr/bin/sort /app/public/converted_csv/North_Pacific_Sp_nodes_19539.csv > /app/public/converted_csv/North_Pacific_Sp_nodes_19539.csv_sorted
[CMD] [2019-12-24 08:15:59] /usr/bin/sort /app/public/converted_csv/North_Pacific_Sp_occurrences_19540.csv > /app/public/converted_csv/North_Pacific_Sp_occurrences_19540.csv_sorted
[CMD] [2019-12-24 08:15:59] /usr/bin/sort /app/public/converted_csv/North_Pacific_Sp_measurements_19541.csv > /app/public/converted_csv/North_Pacific_Sp_measurements_19541.csv_sorted
[STOP] [2019-12-24 08:16:00] convert_to_csv
[START] [2019-12-24 08:16:00] calculate_delta
[CMD] [2019-12-24 08:16:00] echo "0a" > /app/public/diff/North_Pacific_Sp_refs_19538.diff
[CMD] [2019-12-24 08:16:00] tail -n +1 /app/public/converted_csv/North_Pacific_Sp_refs_19538.csv >> /app/public/diff/North_Pacific_Sp_refs_19538.diff
[CMD] [2019-12-24 08:16:00] echo "." >> /app/public/diff/North_Pacific_Sp_refs_19538.diff
[CMD] [2019-12-24 08:16:00] echo "0a" > /app/public/diff/North_Pacific_Sp_nodes_19539.diff
[CMD] [2019-12-24 08:16:00] tail -n +1 /app/public/converted_csv/North_Pacific_Sp_nodes_19539.csv >> /app/public/diff/North_Pacific_Sp_nodes_19539.diff
[CMD] [2019-12-24 08:16:00] echo "." >> /app/public/diff/North_Pacific_Sp_nodes_19539.diff
[CMD] [2019-12-24 08:16:01] echo "0a" > /app/public/diff/North_Pacific_Sp_occurrences_19540.diff
[CMD] [2019-12-24 08:16:01] tail -n +1 /app/public/converted_csv/North_Pacific_Sp_occurrences_19540.csv >> /app/public/diff/North_Pacific_Sp_occurrences_19540.diff
[CMD] [2019-12-24 08:16:01] echo "." >> /app/public/diff/North_Pacific_Sp_occurrences_19540.diff
[CMD] [2019-12-24 08:16:01] echo "0a" > /app/public/diff/North_Pacific_Sp_measurements_19541.diff
[CMD] [2019-12-24 08:16:01] tail -n +1 /app/public/converted_csv/North_Pacific_Sp_measurements_19541.csv >> /app/public/diff/North_Pacific_Sp_measurements_19541.diff
[CMD] [2019-12-24 08:16:01] echo "." >> /app/public/diff/North_Pacific_Sp_measurements_19541.diff
[STOP] [2019-12-24 08:16:02] calculate_delta
[START] [2019-12-24 08:16:02] parse_diff_and_store
[INFO] [2019-12-24 08:16:02] Loading refs diff file into memory (true lines)...
[INFO] [2019-12-24 08:16:02] Loading nodes diff file into memory (true lines)...
[WARN] [2019-12-24 08:16:24] Filtered Scientific Name `Entzia macrescens` to `Entzia macrescens`
[WARN] [2019-12-24 08:16:37] Filtered Scientific Name `Magnoliopsida ""` to `Magnoliopsida `
[WARN] [2019-12-24 08:16:39] Filtered Scientific Name `Isopoda rounded."` to `Isopoda rounded.`
[INFO] [2019-12-24 08:16:46] Loading occurrences diff file into memory (true lines)...
[INFO] [2019-12-24 08:16:59] Loading measurements diff file into memory (true lines)...
[INFO] [2019-12-24 08:26:09] Storing 2 References
[INFO] [2019-12-24 08:26:09] Processing group of 2 in 1 groups of 1000
[INFO] [2019-12-24 08:26:09] Average Time: 0.0
[INFO] [2019-12-24 08:26:09] Total Time: 1s
[INFO] [2019-12-24 08:26:09] Storing 122681 ScientificNames
[INFO] [2019-12-24 08:26:09] Processing group of 122681 in 123 groups of 1000
[INFO] [2019-12-24 08:27:17] Average Time: 0.55
[INFO] [2019-12-24 08:27:17] Total Time: 1m9s
[INFO] [2019-12-24 08:27:17] last 3 / first 3: 0.95
[INFO] [2019-12-24 08:27:17] Std.Dev: 0.6964194138592059; Max: 4.59
[INFO] [2019-12-24 08:27:17] Storing 122681 Nodes
[INFO] [2019-12-24 08:27:17] Processing group of 122681 in 123 groups of 1000
[INFO] [2019-12-24 08:28:17] Average Time: 0.488
[INFO] [2019-12-24 08:28:17] Total Time: 1m1s
[INFO] [2019-12-24 08:28:17] last 3 / first 3: 0.98
[INFO] [2019-12-24 08:28:17] Std.Dev: 0.844393273303382; Max: 5.41
[INFO] [2019-12-24 08:28:17] Storing 90594 Occurrences
[INFO] [2019-12-24 08:28:17] Processing group of 90594 in 91 groups of 1000
[INFO] [2019-12-24 08:28:40] Average Time: 0.239
[INFO] [2019-12-24 08:28:40] Total Time: 23s
[INFO] [2019-12-24 08:28:40] last 3 / first 3: 1.45
[INFO] [2019-12-24 08:28:40] Std.Dev: 0.7655063683601855; Max: 5.41
[INFO] [2019-12-24 08:28:40] Storing 181188 TraitsReferences
[INFO] [2019-12-24 08:28:40] Processing group of 181188 in 182 groups of 1000
[INFO] [2019-12-24 08:28:53] Average Time: 0.069
[INFO] [2019-12-24 08:28:53] Total Time: 14s
[INFO] [2019-12-24 08:28:53] last 3 / first 3: 0.54
[INFO] [2019-12-24 08:28:53] Std.Dev: 0.03162277660168379; Max: 0.33
[INFO] [2019-12-24 08:28:53] Storing 181188 Traits
[INFO] [2019-12-24 08:28:53] Processing group of 181188 in 182 groups of 1000
[INFO] [2019-12-24 08:30:28] Average Time: 0.52
[INFO] [2019-12-24 08:30:28] Total Time: 1m36s
[INFO] [2019-12-24 08:30:28] last 3 / first 3: 0.58
[INFO] [2019-12-24 08:30:28] Std.Dev: 0.9813256340277675; Max: 6.21
[INFO] [2019-12-24 08:30:28] Storing 180941 MetaTraits
[INFO] [2019-12-24 08:30:28] Processing group of 180941 in 181 groups of 1000
[INFO] [2019-12-24 08:31:13] Average Time: 0.242
[INFO] [2019-12-24 08:31:13] Total Time: 45s
[INFO] [2019-12-24 08:31:13] last 3 / first 3: 0.93
[INFO] [2019-12-24 08:31:13] Std.Dev: 0.763544366752843; Max: 6.23
[STOP] [2019-12-24 08:31:13] parse_diff_and_store
[START] [2019-12-24 08:31:13] resolve_keys
[INFO] [2019-12-24 08:34:19] Occurrences to nodes (through scientific_names)...
[INFO] [2019-12-24 08:34:29] traits to occurrences...
[INFO] [2019-12-24 08:34:41] traits to nodes (through occurrences)...
[INFO] [2019-12-24 08:34:43] Traits to sex term...
[INFO] [2019-12-24 08:34:51] Traits to lifestage term...
[INFO] [2019-12-24 08:35:00] MetaTraits to traits...
[INFO] [2019-12-24 08:35:11] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2019-12-24 08:35:35] Assocs to occurrences...
[INFO] [2019-12-24 08:35:35] Assocs to nodes...
[INFO] [2019-12-24 08:35:35] Assoc to sex term...
[INFO] [2019-12-24 08:35:35] Assoc to lifestage term...
[STOP] [2019-12-24 08:35:36] resolve_keys
[START] [2019-12-24 08:35:36] hold_for_later_1
[STOP] [2019-12-24 08:35:36] hold_for_later_1
[START] [2019-12-24 08:35:36] hold_for_later_2
[STOP] [2019-12-24 08:35:36] hold_for_later_2
[START] [2019-12-24 08:35:36] resolve_missing_parents
[STOP] [2019-12-24 08:37:48] resolve_missing_parents
[START] [2019-12-24 08:37:48] rebuild_nodes
[START] [2019-12-24 08:37:48] Flattener#flatten
[START] [2019-12-24 08:37:48] Flattener#study_resource
[START] [2019-12-24 08:37:55] Flattener#build_ancestry
[STOP] [2019-12-24 08:38:29] Flattener#build_ancestry
[INFO] [2019-12-24 08:38:29] 122681 ancestry keys
[START] [2019-12-24 08:38:29] build_node_ancestors
[INFO] [2019-12-24 08:38:29] old ancestors deleted.
[STOP] [2019-12-24 08:40:03] build_node_ancestors
[START] [2019-12-24 08:40:09] Flattener#propagate_ancestor_ids
[STOP] [2019-12-24 08:40:42] Flattener#propagate_ancestor_ids
[STOP] [2019-12-24 08:40:42] Flattener#flatten
[STOP] [2019-12-24 08:40:42] rebuild_nodes
[START] [2019-12-24 08:40:42] resolve_missing_media_owners
[STOP] [2019-12-24 08:40:42] resolve_missing_media_owners
[START] [2019-12-24 08:40:42] sanitize_media_verbatims
[STOP] [2019-12-24 08:40:42] sanitize_media_verbatims
[START] [2019-12-24 08:40:42] queue_downloads
[STOP] [2019-12-24 08:40:42] queue_downloads
[START] [2019-12-24 08:40:42] parse_names
[WARN] [2019-12-24 08:40:42] I see 122681 names which still need to be parsed.
[WARN] [2019-12-24 08:42:14] I see 1 names which still need to be parsed.
[STOP] [2019-12-24 08:42:16] parse_names
[START] [2019-12-24 08:42:16] denormalize_canonical_names_to_nodes
[STOP] [2019-12-24 08:42:18] denormalize_canonical_names_to_nodes
[START] [2019-12-24 08:42:18] match_nodes
[START] [2019-12-24 08:42:18] map_all_nodes_to_pages
[STOP] [2019-12-24 15:16:35] map_all_nodes_to_pages
[INFO] [2019-12-24 15:16:35] 9002 Unmatched nodes (of 122681)! That's too many to output. First 10: Mycetozoa (#62254047); Protosteliomycetes (#62254046); Protosteliales (#62254045); Protosteliaceae (#62254044); Schizoplasmodiopsis micropunctata (#62278135); Protostelium pyriforme (#62255098); Schizoplasmodium sechellarum (#62273402); Schizoplasmodium obovatum (#62294167); Microglomus (#62256722); Cavosteliaceae (#62254494)
[START] [2019-12-24 15:16:35] update_nodes
[STOP] [2019-12-24 15:16:43] update_nodes
[STOP] [2019-12-24 15:16:43] match_nodes
[START] [2019-12-24 15:16:43] reindex_search
[STOP] [2019-12-24 15:21:51] reindex_search
[START] [2019-12-24 15:21:51] normalize_units
[STOP] [2019-12-24 15:22:22] normalize_units
[START] [2019-12-24 15:22:22] calculate_statistics
[STOP] [2019-12-24 15:22:23] calculate_statistics
[START] [2019-12-24 15:22:23] complete_harvest_instance
[START] [2019-12-24 15:22:23] overall_tsv_creation
[INFO] [2019-12-24 15:22:23] Processing group of 122681 in 13 batches of 10000
[INFO] [2019-12-24 15:23:51] 5334 Traits (unfiltered)...
[INFO] [2019-12-24 15:24:04] 5334 Traits (filtered)...
[INFO] [2019-12-24 15:24:04] 0 Associations (filtered)...
[INFO] [2019-12-24 15:24:51] 26658 metadata added.
[INFO] [2019-12-24 15:24:51] 0 metadata added.
[INFO] [2019-12-24 15:26:24] 6352 Traits (unfiltered)...
[INFO] [2019-12-24 15:26:37] 6352 Traits (filtered)...
[INFO] [2019-12-24 15:26:37] 0 Associations (filtered)...
[INFO] [2019-12-24 15:27:28] 31746 metadata added.
[INFO] [2019-12-24 15:27:28] 0 metadata added.
[INFO] [2019-12-24 15:29:04] 6887 Traits (unfiltered)...
[INFO] [2019-12-24 15:29:18] 6887 Traits (filtered)...
[INFO] [2019-12-24 15:29:18] 0 Associations (filtered)...
[INFO] [2019-12-24 15:30:14] 34419 metadata added.
[INFO] [2019-12-24 15:30:14] 0 metadata added.
[INFO] [2019-12-24 15:31:49] 7181 Traits (unfiltered)...
[INFO] [2019-12-24 15:32:02] 7181 Traits (filtered)...
[INFO] [2019-12-24 15:32:02] 0 Associations (filtered)...
[INFO] [2019-12-24 15:32:56] 35887 metadata added.
[INFO] [2019-12-24 15:32:56] 0 metadata added.
[INFO] [2019-12-24 15:34:32] 7458 Traits (unfiltered)...
[INFO] [2019-12-24 15:34:45] 7458 Traits (filtered)...
[INFO] [2019-12-24 15:34:46] 0 Associations (filtered)...
[INFO] [2019-12-24 15:35:41] 37277 metadata added.
[INFO] [2019-12-24 15:35:41] 0 metadata added.
[INFO] [2019-12-24 15:37:18] 7587 Traits (unfiltered)...
[INFO] [2019-12-24 15:37:31] 7587 Traits (filtered)...
[INFO] [2019-12-24 15:37:31] 0 Associations (filtered)...
[INFO] [2019-12-24 15:38:26] 37923 metadata added.
[INFO] [2019-12-24 15:38:26] 0 metadata added.
[INFO] [2019-12-24 15:40:02] 7591 Traits (unfiltered)...
[INFO] [2019-12-24 15:40:15] 7591 Traits (filtered)...
[INFO] [2019-12-24 15:40:16] 0 Associations (filtered)...
[INFO] [2019-12-24 15:41:12] 37936 metadata added.
[INFO] [2019-12-24 15:41:12] 0 metadata added.
[INFO] [2019-12-24 15:42:49] 7778 Traits (unfiltered)...
[INFO] [2019-12-24 15:43:02] 7778 Traits (filtered)...
[INFO] [2019-12-24 15:43:02] 0 Associations (filtered)...
[INFO] [2019-12-24 15:43:57] 38860 metadata added.
[INFO] [2019-12-24 15:43:57] 0 metadata added.
[INFO] [2019-12-24 15:45:32] 7859 Traits (unfiltered)...
[INFO] [2019-12-24 15:45:45] 7859 Traits (filtered)...
[INFO] [2019-12-24 15:45:45] 0 Associations (filtered)...
[INFO] [2019-12-24 15:46:41] 39276 metadata added.
[INFO] [2019-12-24 15:46:41] 0 metadata added.
[INFO] [2019-12-24 15:48:14] 7870 Traits (unfiltered)...
[INFO] [2019-12-24 15:48:27] 7870 Traits (filtered)...
[INFO] [2019-12-24 15:48:28] 0 Associations (filtered)...
[INFO] [2019-12-24 15:49:23] 39328 metadata added.
[INFO] [2019-12-24 15:49:23] 0 metadata added.
[INFO] [2019-12-24 15:50:59] 8070 Traits (unfiltered)...
[INFO] [2019-12-24 15:51:12] 8070 Traits (filtered)...
[INFO] [2019-12-24 15:51:12] 0 Associations (filtered)...
[INFO] [2019-12-24 15:52:08] 40308 metadata added.
[INFO] [2019-12-24 15:52:08] 0 metadata added.
[INFO] [2019-12-24 15:53:42] 8403 Traits (unfiltered)...
[INFO] [2019-12-24 15:53:55] 8403 Traits (filtered)...
[INFO] [2019-12-24 15:53:55] 0 Associations (filtered)...
[INFO] [2019-12-24 15:54:52] 41988 metadata added.
[INFO] [2019-12-24 15:54:52] 0 metadata added.
[INFO] [2019-12-24 15:55:52] 2224 Traits (unfiltered)...
[INFO] [2019-12-24 15:56:05] 2224 Traits (filtered)...
[INFO] [2019-12-24 15:56:05] 0 Associations (filtered)...
[INFO] [2019-12-24 15:56:46] 11117 metadata added.
[INFO] [2019-12-24 15:56:46] 0 metadata added.
[INFO] [2019-12-24 15:56:46] Average Time: 129.95
[INFO] [2019-12-24 15:56:46] Total Time: 34m24s
[INFO] [2019-12-24 15:56:46] last 3 / first 3: 0.95
[INFO] [2019-12-24 15:56:46] Std.Dev: 12.666688596472245; Max: 136.83
[STOP] [2019-12-24 15:56:46] overall_tsv_creation
[INFO] [2019-12-24 15:56:46] Done. Check your files:
[INFO] [2019-12-24 15:56:46] (122681 lines) /app/public/data/North_Pacific_Sp/publish_nodes.tsv
[INFO] [2019-12-24 15:56:47] (684122 lines) /app/public/data/North_Pacific_Sp/publish_node_ancestors.tsv
[INFO] [2019-12-24 15:56:47] (122681 lines) /app/public/data/North_Pacific_Sp/publish_scientific_names.tsv
[INFO] [2019-12-24 15:56:47] (90595 lines) /app/public/data/North_Pacific_Sp/publish_traits.tsv
[INFO] [2019-12-24 15:56:47] (452724 lines) /app/public/data/North_Pacific_Sp/publish_metadata.tsv
[STOP] [2019-12-24 15:56:47] complete_harvest_instance
[START] [2019-12-24 15:56:47] completed
[STOP] [2019-12-24 15:56:47] completed
[STOP] [2019-12-24 15:56:47] logged process, took 27665.03
Latest Process