Harvest for Simpson et al 2020 Created 08 Apr 11:55

Stage: completed
Fetched: 08 Apr 11:55
Validated: 08 Apr 11:55
Deltas Created 08 Apr 11:56
Units Normalized: 08 Apr 12:06
Ancestry Built: 08 Apr 11:57
Nodes Matched: 08 Apr 12:06
Names Parsed: 08 Apr 11:57
New Models Stored: 08 Apr 11:56
Indexed: 08 Apr 12:06
Completed: 08 Apr 12:14
Time to Harvest: less than a minute

Harvesting Log

(1414 lines) (showing only the last 1000 lines, see /app/public/data/simpson_et_al_si/process.log for the full file)
[INFO] [2021-04-04 09:51:19] ++ Calling delete_all on 39874 instances...
[INFO] [2021-04-04 09:51:21] [09:51:21.156] Removed 39874 Metatraits
[INFO] [2021-04-04 09:51:21] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-04 09:51:21] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-04 09:51:21] [09:51:21.166] Removed 0 Occurrencemetadata
[INFO] [2021-04-04 09:51:21] ## remove_type: Assoc
[INFO] [2021-04-04 09:51:21] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-04 09:51:21] [09:51:21.168] Removed 0 Assocs
[INFO] [2021-04-04 09:51:21] ## remove_type: MetaAssoc
[INFO] [2021-04-04 09:51:21] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-04 09:51:21] [09:51:21.169] Removed 0 Metaassocs
[INFO] [2021-04-04 09:51:21] ## remove_type: Identifier
[INFO] [2021-04-04 09:51:21] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-04 09:51:21] [09:51:21.170] Removed 0 Identifiers
[INFO] [2021-04-04 09:51:21] ## remove_type: Reference
[INFO] [2021-04-04 09:51:21] ++ Calling delete_all on 1293 instances...
[INFO] [2021-04-04 09:51:21] [09:51:21.207] Removed 1293 References
[INFO] [2021-04-04 09:51:21] Starting batch with ID 80815580...
[INFO] [2021-04-04 09:51:22] Starting batch with ID 80815580...
[INFO] [2021-04-04 09:51:23] Starting batch with ID 80831114...
[INFO] [2021-04-04 09:51:23] Starting batch with ID 80831114...
[INFO] [2021-04-04 09:51:23] Starting batch with ID 80828880...
[INFO] [2021-04-04 09:51:24] Starting batch with ID 80828880...
[INFO] [2021-04-04 09:51:24] Starting batch with ID 80818828...
[INFO] [2021-04-04 09:51:25] Starting batch with ID 80818828...
[INFO] [2021-04-04 09:51:25] Starting batch with ID 80818828...
[INFO] [2021-04-04 09:51:25] ## remove_type: Node
[INFO] [2021-04-04 09:51:25] ++ Calling delete_all on 18353 instances...
[INFO] [2021-04-04 09:51:26] [09:51:26.964] Removed 18353 Nodes
[START] [2021-04-04 09:51:43] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-04 09:51:43] Creating resource from OpenData
[START] [2021-04-04 09:51:45] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-04 09:51:45] Parse meta.xml file and create formats with fields
[WARN] [2021-04-04 09:52:01] (common) IGNORED  (vernaculars) field header: CountryCode term: http://rs.tdwg.org/dwc/terms/countryCode
[WARN] [2021-04-04 09:52:01] (common) IGNORED  (refs) field header: Source Type term: http://eol.org/schema/reference/publicationType
[STOP] [2021-04-04 09:52:01] Parse meta.xml file and create formats with fields
[STOP] [2021-04-04 09:52:01] Creating resource from OpenData
[START] [2021-04-04 09:52:02] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-04 09:52:02] create_harvest_instance
[INFO] [2021-04-04 09:52:02] Created harvest instance #3628
[STOP] [2021-04-04 09:52:02] create_harvest_instance
[START] [2021-04-04 09:52:02] fetch_files
[STOP] [2021-04-04 09:52:02] fetch_files
[START] [2021-04-04 09:52:02] validate_each_file
[INFO] [2021-04-04 09:52:02] Looping over 5 formats...
[INFO] [2021-04-04 09:52:02] ...refs (/app/public/data/simpson_et_al_si/references.tsv)
[INFO] [2021-04-04 09:52:02] Valid: /app/public/converted_csv/simpson_et_al_si_refs_3628.csv (1293 lines)
[INFO] [2021-04-04 09:52:02] ...nodes (/app/public/data/simpson_et_al_si/taxa.tsv)
[INFO] [2021-04-04 09:52:02] Valid: /app/public/converted_csv/simpson_et_al_si_nodes_3628.csv (11184 lines)
[INFO] [2021-04-04 09:52:02] ...vernaculars (/app/public/data/simpson_et_al_si/common_names.txt)
[INFO] [2021-04-04 09:52:03] Valid: /app/public/converted_csv/simpson_et_al_si_vernaculars_3628.csv (8190 lines)
[INFO] [2021-04-04 09:52:03] ...occurrences (/app/public/data/simpson_et_al_si/occurrences.tsv)
[INFO] [2021-04-04 09:52:03] Valid: /app/public/converted_csv/simpson_et_al_si_occurrences_3628.csv (12985 lines)
[INFO] [2021-04-04 09:52:03] ...measurements (/app/public/data/simpson_et_al_si/measurementOrFact.tsv)
[INFO] [2021-04-04 09:52:04] Valid: /app/public/converted_csv/simpson_et_al_si_measurements_3628.csv (12986 lines)
[STOP] [2021-04-04 09:52:04] validate_each_file
[START] [2021-04-04 09:52:04] convert_to_csv
[INFO] [2021-04-04 09:52:04] Looping over 5 formats...
[INFO] [2021-04-04 09:52:04] ...refs (/app/public/data/simpson_et_al_si/references.tsv)
[CMD] [2021-04-04 09:52:04] /usr/bin/sort /app/public/converted_csv/simpson_et_al_si_refs_3628.csv > /app/public/converted_csv/simpson_et_al_si_refs_3628.csv_sorted
[INFO] [2021-04-04 09:52:05] Converted: /app/public/converted_csv/simpson_et_al_si_refs_3628.csv (1293 lines)
[INFO] [2021-04-04 09:52:05] ...nodes (/app/public/data/simpson_et_al_si/taxa.tsv)
[CMD] [2021-04-04 09:52:05] /usr/bin/sort /app/public/converted_csv/simpson_et_al_si_nodes_3628.csv > /app/public/converted_csv/simpson_et_al_si_nodes_3628.csv_sorted
[INFO] [2021-04-04 09:52:06] Converted: /app/public/converted_csv/simpson_et_al_si_nodes_3628.csv (11184 lines)
[INFO] [2021-04-04 09:52:06] ...vernaculars (/app/public/data/simpson_et_al_si/common_names.txt)
[CMD] [2021-04-04 09:52:06] /usr/bin/sort /app/public/converted_csv/simpson_et_al_si_vernaculars_3628.csv > /app/public/converted_csv/simpson_et_al_si_vernaculars_3628.csv_sorted
[INFO] [2021-04-04 09:52:07] Converted: /app/public/converted_csv/simpson_et_al_si_vernaculars_3628.csv (8190 lines)
[INFO] [2021-04-04 09:52:07] ...occurrences (/app/public/data/simpson_et_al_si/occurrences.tsv)
[CMD] [2021-04-04 09:52:07] /usr/bin/sort /app/public/converted_csv/simpson_et_al_si_occurrences_3628.csv > /app/public/converted_csv/simpson_et_al_si_occurrences_3628.csv_sorted
[INFO] [2021-04-04 09:52:08] Converted: /app/public/converted_csv/simpson_et_al_si_occurrences_3628.csv (12985 lines)
[INFO] [2021-04-04 09:52:08] ...measurements (/app/public/data/simpson_et_al_si/measurementOrFact.tsv)
[CMD] [2021-04-04 09:52:08] /usr/bin/sort /app/public/converted_csv/simpson_et_al_si_measurements_3628.csv > /app/public/converted_csv/simpson_et_al_si_measurements_3628.csv_sorted
[INFO] [2021-04-04 09:52:09] Converted: /app/public/converted_csv/simpson_et_al_si_measurements_3628.csv (12986 lines)
[STOP] [2021-04-04 09:52:09] convert_to_csv
[START] [2021-04-04 09:52:09] calculate_delta
[INFO] [2021-04-04 09:52:09] Looping over 5 formats...
[INFO] [2021-04-04 09:52:09] ...refs (/app/public/data/simpson_et_al_si/references.tsv)
[CMD] [2021-04-04 09:52:09] echo "0a" > /app/public/diff/simpson_et_al_si_refs_3628.diff
[CMD] [2021-04-04 09:52:10] tail -n +1 /app/public/converted_csv/simpson_et_al_si_refs_3628.csv >> /app/public/diff/simpson_et_al_si_refs_3628.diff
[CMD] [2021-04-04 09:52:11] echo "." >> /app/public/diff/simpson_et_al_si_refs_3628.diff
[INFO] [2021-04-04 09:52:12] Created diff: /app/public/diff/simpson_et_al_si_refs_3628.diff (1295 lines)
[INFO] [2021-04-04 09:52:12] ...nodes (/app/public/data/simpson_et_al_si/taxa.tsv)
[CMD] [2021-04-04 09:52:12] echo "0a" > /app/public/diff/simpson_et_al_si_nodes_3628.diff
[CMD] [2021-04-04 09:52:13] tail -n +1 /app/public/converted_csv/simpson_et_al_si_nodes_3628.csv >> /app/public/diff/simpson_et_al_si_nodes_3628.diff
[CMD] [2021-04-04 09:52:14] echo "." >> /app/public/diff/simpson_et_al_si_nodes_3628.diff
[INFO] [2021-04-04 09:52:15] Created diff: /app/public/diff/simpson_et_al_si_nodes_3628.diff (11186 lines)
[INFO] [2021-04-04 09:52:15] ...vernaculars (/app/public/data/simpson_et_al_si/common_names.txt)
[CMD] [2021-04-04 09:52:15] echo "0a" > /app/public/diff/simpson_et_al_si_vernaculars_3628.diff
[CMD] [2021-04-04 09:52:16] tail -n +1 /app/public/converted_csv/simpson_et_al_si_vernaculars_3628.csv >> /app/public/diff/simpson_et_al_si_vernaculars_3628.diff
[CMD] [2021-04-04 09:52:17] echo "." >> /app/public/diff/simpson_et_al_si_vernaculars_3628.diff
[INFO] [2021-04-04 09:52:18] Created diff: /app/public/diff/simpson_et_al_si_vernaculars_3628.diff (8192 lines)
[INFO] [2021-04-04 09:52:18] ...occurrences (/app/public/data/simpson_et_al_si/occurrences.tsv)
[CMD] [2021-04-04 09:52:18] echo "0a" > /app/public/diff/simpson_et_al_si_occurrences_3628.diff
[CMD] [2021-04-04 09:52:19] tail -n +1 /app/public/converted_csv/simpson_et_al_si_occurrences_3628.csv >> /app/public/diff/simpson_et_al_si_occurrences_3628.diff
[CMD] [2021-04-04 09:52:20] echo "." >> /app/public/diff/simpson_et_al_si_occurrences_3628.diff
[INFO] [2021-04-04 09:52:21] Created diff: /app/public/diff/simpson_et_al_si_occurrences_3628.diff (12987 lines)
[INFO] [2021-04-04 09:52:21] ...measurements (/app/public/data/simpson_et_al_si/measurementOrFact.tsv)
[CMD] [2021-04-04 09:52:21] echo "0a" > /app/public/diff/simpson_et_al_si_measurements_3628.diff
[CMD] [2021-04-04 09:52:22] tail -n +1 /app/public/converted_csv/simpson_et_al_si_measurements_3628.csv >> /app/public/diff/simpson_et_al_si_measurements_3628.diff
[CMD] [2021-04-04 09:52:23] echo "." >> /app/public/diff/simpson_et_al_si_measurements_3628.diff
[INFO] [2021-04-04 09:52:24] Created diff: /app/public/diff/simpson_et_al_si_measurements_3628.diff (12988 lines)
[STOP] [2021-04-04 09:52:24] calculate_delta
[START] [2021-04-04 09:52:24] parse_diff_and_store
[INFO] [2021-04-04 09:52:24] Handling diff: /app/public/diff/simpson_et_al_si_refs_3628.diff (1295 lines)
[INFO] [2021-04-04 09:52:25] Loading refs diff file into memory (1295 /app/public/diff/simpson_et_al_si_refs_3628.diff lines)...
[INFO] [2021-04-04 09:52:27] Handling diff: /app/public/diff/simpson_et_al_si_nodes_3628.diff (11186 lines)
[INFO] [2021-04-04 09:52:28] Loading nodes diff file into memory (11186 /app/public/diff/simpson_et_al_si_nodes_3628.diff lines)...
[INFO] [2021-04-04 09:52:33] Handling diff: /app/public/diff/simpson_et_al_si_vernaculars_3628.diff (8192 lines)
[INFO] [2021-04-04 09:52:34] Loading vernaculars diff file into memory (8192 /app/public/diff/simpson_et_al_si_vernaculars_3628.diff lines)...
[INFO] [2021-04-04 09:52:37] Handling diff: /app/public/diff/simpson_et_al_si_occurrences_3628.diff (12987 lines)
[INFO] [2021-04-04 09:52:37] Loading occurrences diff file into memory (12987 /app/public/diff/simpson_et_al_si_occurrences_3628.diff lines)...
[INFO] [2021-04-04 09:52:40] Handling diff: /app/public/diff/simpson_et_al_si_measurements_3628.diff (12988 lines)
[INFO] [2021-04-04 09:52:41] Loading measurements diff file into memory (12988 /app/public/diff/simpson_et_al_si_measurements_3628.diff lines)...
[INFO] [2021-04-04 09:52:50] Storing 1293 References
[INFO] [2021-04-04 09:52:50] Processing group of 1293 in 2 groups of 1000
[INFO] [2021-04-04 09:52:50] Average Time: 0.13
[INFO] [2021-04-04 09:52:50] Total Time: 1s
[INFO] [2021-04-04 09:52:50] Storing 18353 ScientificNames
[INFO] [2021-04-04 09:52:50] Processing group of 18353 in 19 groups of 1000
[INFO] [2021-04-04 09:52:56] Average Time: 0.287
[INFO] [2021-04-04 09:52:56] Total Time: 6s
[INFO] [2021-04-04 09:52:56] last 3 / first 3: 0.81
[INFO] [2021-04-04 09:52:56] Std.Dev: 0.06324555320336758; Max: 0.46
[INFO] [2021-04-04 09:52:56] Storing 18353 Nodes
[INFO] [2021-04-04 09:52:56] Processing group of 18353 in 19 groups of 1000
[INFO] [2021-04-04 09:53:01] Average Time: 0.255
[INFO] [2021-04-04 09:53:01] Total Time: 5s
[INFO] [2021-04-04 09:53:01] last 3 / first 3: 0.86
[INFO] [2021-04-04 09:53:01] Std.Dev: 0.044721359549995794; Max: 0.34
[INFO] [2021-04-04 09:53:01] Storing 8190 Vernaculars
[INFO] [2021-04-04 09:53:01] Processing group of 8190 in 9 groups of 1000
[INFO] [2021-04-04 09:53:02] Average Time: 0.147
[INFO] [2021-04-04 09:53:02] Total Time: 2s
[INFO] [2021-04-04 09:53:02] last 3 / first 3: 0.65
[INFO] [2021-04-04 09:53:02] Std.Dev: 0.044721359549995794; Max: 0.2
[INFO] [2021-04-04 09:53:02] Storing 12985 Occurrences
[INFO] [2021-04-04 09:53:02] Processing group of 12985 in 13 groups of 1000
[INFO] [2021-04-04 09:53:04] Average Time: 0.107
[INFO] [2021-04-04 09:53:04] Total Time: 2s
[INFO] [2021-04-04 09:53:04] last 3 / first 3: 1.0
[INFO] [2021-04-04 09:53:04] Std.Dev: 0.03162277660168379; Max: 0.19
[INFO] [2021-04-04 09:53:04] Storing 12986 TraitsReferences
[INFO] [2021-04-04 09:53:04] Processing group of 12986 in 13 groups of 1000
[INFO] [2021-04-04 09:53:05] Average Time: 0.076
[INFO] [2021-04-04 09:53:05] Total Time: 2s
[INFO] [2021-04-04 09:53:05] last 3 / first 3: 1.48
[INFO] [2021-04-04 09:53:05] Std.Dev: 0.03162277660168379; Max: 0.17
[INFO] [2021-04-04 09:53:05] Storing 12986 Traits
[INFO] [2021-04-04 09:53:05] Processing group of 12986 in 13 groups of 1000
[INFO] [2021-04-04 09:53:09] Average Time: 0.342
[INFO] [2021-04-04 09:53:09] Total Time: 5s
[INFO] [2021-04-04 09:53:09] last 3 / first 3: 1.75
[INFO] [2021-04-04 09:53:09] Std.Dev: 0.1; Max: 0.51
[INFO] [2021-04-04 09:53:09] Storing 26888 MetaTraits
[INFO] [2021-04-04 09:53:09] Processing group of 26888 in 27 groups of 1000
[INFO] [2021-04-04 09:53:12] Average Time: 0.115
[INFO] [2021-04-04 09:53:12] Total Time: 4s
[INFO] [2021-04-04 09:53:12] last 3 / first 3: 0.94
[INFO] [2021-04-04 09:53:12] Std.Dev: 0.0; Max: 0.18
[STOP] [2021-04-04 09:53:12] parse_diff_and_store
[START] [2021-04-04 09:53:12] resolve_keys
[INFO] [2021-04-04 09:53:26] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-04 09:53:27] traits to occurrences...
[INFO] [2021-04-04 09:53:27] traits to nodes (through occurrences)...
[INFO] [2021-04-04 09:53:28] Traits to sex term...
[INFO] [2021-04-04 09:53:28] Traits to lifestage term...
[INFO] [2021-04-04 09:53:28] MetaTraits to traits...
[INFO] [2021-04-04 09:53:28] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-04 09:53:29] Assocs to occurrences...
[INFO] [2021-04-04 09:53:29] Assocs to nodes...
[INFO] [2021-04-04 09:53:29] Assoc to sex term...
[INFO] [2021-04-04 09:53:29] Assoc to lifestage term...
[INFO] [2021-04-04 09:53:29] MetaAssoc to assocs...
[STOP] [2021-04-04 09:53:29] resolve_keys
[START] [2021-04-04 09:53:29] hold_for_later_1
[STOP] [2021-04-04 09:53:29] hold_for_later_1
[START] [2021-04-04 09:53:29] hold_for_later_2
[STOP] [2021-04-04 09:53:29] hold_for_later_2
[START] [2021-04-04 09:53:29] resolve_missing_parents
[STOP] [2021-04-04 09:53:30] resolve_missing_parents
[START] [2021-04-04 09:53:30] rebuild_nodes
[START] [2021-04-04 09:53:30] Flattener#flatten
[START] [2021-04-04 09:53:30] Flattener#study_resource
[START] [2021-04-04 09:53:30] Flattener#build_ancestry
[STOP] [2021-04-04 09:53:31] Flattener#build_ancestry
[INFO] [2021-04-04 09:53:31] 18353 ancestry keys
[START] [2021-04-04 09:53:31] build_node_ancestors
[INFO] [2021-04-04 09:53:31] old ancestors deleted.
[STOP] [2021-04-04 09:53:42] build_node_ancestors
[START] [2021-04-04 09:53:42] Flattener#propagate_ancestor_ids
[STOP] [2021-04-04 09:53:44] Flattener#propagate_ancestor_ids
[STOP] [2021-04-04 09:53:44] Flattener#flatten
[STOP] [2021-04-04 09:53:44] rebuild_nodes
[START] [2021-04-04 09:53:44] resolve_missing_media_owners
[STOP] [2021-04-04 09:53:44] resolve_missing_media_owners
[START] [2021-04-04 09:53:44] sanitize_media_verbatims
[STOP] [2021-04-04 09:53:44] sanitize_media_verbatims
[START] [2021-04-04 09:53:44] queue_downloads
[STOP] [2021-04-04 09:53:44] queue_downloads
[START] [2021-04-04 09:53:44] parse_names
[WARN] [2021-04-04 09:53:44] I see 18353 names which still need to be parsed.
[WARN] [2021-04-04 09:53:57] I see 85 names which still need to be parsed.
[WARN] [2021-04-04 09:53:59] I see 2 names which still need to be parsed.
[STOP] [2021-04-04 09:54:00] parse_names
[START] [2021-04-04 09:54:00] denormalize_canonical_names_to_nodes
[STOP] [2021-04-04 09:54:00] denormalize_canonical_names_to_nodes
[START] [2021-04-04 09:54:00] match_nodes
[START] [2021-04-04 09:54:00] map_all_nodes_to_pages
[STOP] [2021-04-04 10:01:48] map_all_nodes_to_pages
[INFO] [2021-04-04 10:01:48] 1335 Unmatched nodes (of 18353)! That's too many to output. Full list in /app/public/data/simpson_et_al_si/unmatched_nodes.txt ; First 10: Aceria hibisci (#91660049); Aceria malherbae (#91660051); Aculops fuchsiae (#91660247); Colomerus gardeniella (#91664554); Eriophyes (#91666718); Eriophyes cynodoniensis (#91666719); Eriophyes fici (#91666720); Eriophyes gardeniella (#91666721); Eriophyes granati (#91666722); Eriophyes hibisci (#91666723)
[START] [2021-04-04 10:01:48] update_nodes
[STOP] [2021-04-04 10:01:55] update_nodes
[STOP] [2021-04-04 10:01:55] match_nodes
[START] [2021-04-04 10:01:55] reindex_search
[STOP] [2021-04-04 10:02:12] reindex_search
[START] [2021-04-04 10:02:12] normalize_units
[STOP] [2021-04-04 10:02:12] normalize_units
[START] [2021-04-04 10:02:12] calculate_statistics
[STOP] [2021-04-04 10:02:13] calculate_statistics
[START] [2021-04-04 10:02:13] complete_harvest_instance
[START] [2021-04-04 10:02:13] overall_tsv_creation
[INFO] [2021-04-04 10:02:13] Processing group of 18353 in 2 batches of 10000
[INFO] [2021-04-04 10:03:24] 6900 Traits (unfiltered)...
[INFO] [2021-04-04 10:04:37] 6900 Traits (filtered)...
[INFO] [2021-04-04 10:04:40] 0 Associations (filtered)...
[INFO] [2021-04-04 10:04:41] 13800 metadata added.
[INFO] [2021-04-04 10:04:41] 0 metadata added.
[INFO] [2021-04-04 10:06:16] 6086 Traits (unfiltered)...
[INFO] [2021-04-04 10:07:29] 6086 Traits (filtered)...
[INFO] [2021-04-04 10:07:32] 0 Associations (filtered)...
[INFO] [2021-04-04 10:07:33] 12172 metadata added.
[INFO] [2021-04-04 10:07:33] 0 metadata added.
[INFO] [2021-04-04 10:08:01] Average Time: 142.58
[INFO] [2021-04-04 10:08:01] Total Time: 5m48s
[STOP] [2021-04-04 10:08:01] overall_tsv_creation
[INFO] [2021-04-04 10:08:01] Done. Check your files:
[INFO] [2021-04-04 10:08:02] (18270 lines) /app/public/data/simpson_et_al_si/publish_nodes.tsv
[INFO] [2021-04-04 10:08:03] (99765 lines) /app/public/data/simpson_et_al_si/publish_node_ancestors.tsv
[INFO] [2021-04-04 10:08:04] (18353 lines) /app/public/data/simpson_et_al_si/publish_scientific_names.tsv
[INFO] [2021-04-04 10:08:05] (7906 lines) /app/public/data/simpson_et_al_si/publish_vernaculars.tsv
[INFO] [2021-04-04 10:08:06] (12987 lines) /app/public/data/simpson_et_al_si/publish_traits.tsv
[INFO] [2021-04-04 10:08:07] (25973 lines) /app/public/data/simpson_et_al_si/publish_metadata.tsv
[STOP] [2021-04-04 10:08:07] complete_harvest_instance
[START] [2021-04-04 10:08:07] completed
[STOP] [2021-04-04 10:08:07] completed
[STOP] [2021-04-04 10:08:07] logged process, took 966.72
[INFO] [2021-04-08 11:01:42] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-08 11:01:45] ## remove_type: ScientificName
[INFO] [2021-04-08 11:01:45] ++ Calling delete_all on 18353 instances...
[INFO] [2021-04-08 11:01:46] [11:01:46.975] Removed 18353 Scientificnames
[INFO] [2021-04-08 11:01:46] ## remove_type: Vernacular
[INFO] [2021-04-08 11:01:46] ++ Calling delete_all on 8190 instances...
[INFO] [2021-04-08 11:01:47] [11:01:47.120] Removed 8190 Vernaculars
[INFO] [2021-04-08 11:01:47] ## remove_type: Article
[INFO] [2021-04-08 11:01:47] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:01:47] [11:01:47.122] Removed 0 Articles
[INFO] [2021-04-08 11:01:47] ## remove_type: Medium
[INFO] [2021-04-08 11:01:47] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:01:47] [11:01:47.123] Removed 0 Media
[INFO] [2021-04-08 11:01:47] ## remove_type: Trait
[INFO] [2021-04-08 11:01:47] ++ Calling delete_all on 12986 instances...
[INFO] [2021-04-08 11:01:49] [11:01:49.432] Removed 12986 Traits
[INFO] [2021-04-08 11:01:49] ## remove_type: MetaTrait
[INFO] [2021-04-08 11:01:49] ++ Calling delete_all on 26888 instances...
[INFO] [2021-04-08 11:01:50] [11:01:50.659] Removed 26888 Metatraits
[INFO] [2021-04-08 11:01:50] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-08 11:01:50] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:01:50] [11:01:50.660] Removed 0 Occurrencemetadata
[INFO] [2021-04-08 11:01:50] ## remove_type: Assoc
[INFO] [2021-04-08 11:01:50] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:01:50] [11:01:50.662] Removed 0 Assocs
[INFO] [2021-04-08 11:01:50] ## remove_type: MetaAssoc
[INFO] [2021-04-08 11:01:50] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:01:50] [11:01:50.663] Removed 0 Metaassocs
[INFO] [2021-04-08 11:01:50] ## remove_type: Identifier
[INFO] [2021-04-08 11:01:50] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:01:50] [11:01:50.664] Removed 0 Identifiers
[INFO] [2021-04-08 11:01:50] ## remove_type: Reference
[INFO] [2021-04-08 11:01:50] ++ Calling delete_all on 1293 instances...
[INFO] [2021-04-08 11:01:50] [11:01:50.705] Removed 1293 References
[INFO] [2021-04-08 11:01:51] Starting batch with ID 91676883...
[INFO] [2021-04-08 11:01:52] Starting batch with ID 91676883...
[INFO] [2021-04-08 11:01:52] Starting batch with ID 91668815...
[INFO] [2021-04-08 11:01:53] Starting batch with ID 91668815...
[INFO] [2021-04-08 11:01:54] Starting batch with ID 91662426...
[INFO] [2021-04-08 11:01:54] Starting batch with ID 91662426...
[INFO] [2021-04-08 11:01:54] Starting batch with ID 91669934...
[INFO] [2021-04-08 11:01:55] Starting batch with ID 91669934...
[INFO] [2021-04-08 11:01:55] Starting batch with ID 91669934...
[INFO] [2021-04-08 11:01:55] ## remove_type: Node
[INFO] [2021-04-08 11:01:55] ++ Calling delete_all on 18353 instances...
[INFO] [2021-04-08 11:02:03] [11:02:03.167] Removed 18353 Nodes
[START] [2021-04-08 11:02:20] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-08 11:02:20] Creating resource from OpenData
[START] [2021-04-08 11:02:21] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-08 11:02:21] Parse meta.xml file and create formats with fields
[WARN] [2021-04-08 11:02:30] (common) IGNORED  (refs) field header: publicationType term: http://eol.org/schema/reference/publicationType
[STOP] [2021-04-08 11:02:30] Parse meta.xml file and create formats with fields
[STOP] [2021-04-08 11:02:30] Creating resource from OpenData
[START] [2021-04-08 11:02:31] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-08 11:02:31] create_harvest_instance
[INFO] [2021-04-08 11:02:31] Created harvest instance #3701
[STOP] [2021-04-08 11:02:31] create_harvest_instance
[START] [2021-04-08 11:02:31] fetch_files
[STOP] [2021-04-08 11:02:31] fetch_files
[START] [2021-04-08 11:02:31] validate_each_file
[INFO] [2021-04-08 11:02:31] Looping over 5 formats...
[INFO] [2021-04-08 11:02:31] ...refs (/app/public/data/simpson_et_al_si/references.tsv)
[INFO] [2021-04-08 11:02:31] Valid: /app/public/converted_csv/simpson_et_al_si_refs_3701.csv (13380 lines)
[INFO] [2021-04-08 11:02:31] ...nodes (/app/public/data/simpson_et_al_si/taxa.tsv)
[INFO] [2021-04-08 11:02:32] Valid: /app/public/converted_csv/simpson_et_al_si_nodes_3701.csv (13351 lines)
[INFO] [2021-04-08 11:02:32] ...vernaculars (/app/public/data/simpson_et_al_si/common_names.tsv)
[INFO] [2021-04-08 11:02:32] Valid: /app/public/converted_csv/simpson_et_al_si_vernaculars_3701.csv (13352 lines)
[INFO] [2021-04-08 11:02:32] ...occurrences (/app/public/data/simpson_et_al_si/occurrences.tsv)
[INFO] [2021-04-08 11:02:32] Valid: /app/public/converted_csv/simpson_et_al_si_occurrences_3701.csv (13351 lines)
[INFO] [2021-04-08 11:02:32] ...measurements (/app/public/data/simpson_et_al_si/measurement_or_fact.tsv)
[INFO] [2021-04-08 11:02:33] Valid: /app/public/converted_csv/simpson_et_al_si_measurements_3701.csv (13351 lines)
[STOP] [2021-04-08 11:02:33] validate_each_file
[START] [2021-04-08 11:02:33] convert_to_csv
[INFO] [2021-04-08 11:02:33] Looping over 5 formats...
[INFO] [2021-04-08 11:02:33] ...refs (/app/public/data/simpson_et_al_si/references.tsv)
[CMD] [2021-04-08 11:02:33] /usr/bin/sort /app/public/converted_csv/simpson_et_al_si_refs_3701.csv > /app/public/converted_csv/simpson_et_al_si_refs_3701.csv_sorted
[INFO] [2021-04-08 11:02:34] Converted: /app/public/converted_csv/simpson_et_al_si_refs_3701.csv (13380 lines)
[INFO] [2021-04-08 11:02:34] ...nodes (/app/public/data/simpson_et_al_si/taxa.tsv)
[CMD] [2021-04-08 11:02:34] /usr/bin/sort /app/public/converted_csv/simpson_et_al_si_nodes_3701.csv > /app/public/converted_csv/simpson_et_al_si_nodes_3701.csv_sorted
[INFO] [2021-04-08 11:02:35] Converted: /app/public/converted_csv/simpson_et_al_si_nodes_3701.csv (13351 lines)
[INFO] [2021-04-08 11:02:35] ...vernaculars (/app/public/data/simpson_et_al_si/common_names.tsv)
[CMD] [2021-04-08 11:02:35] /usr/bin/sort /app/public/converted_csv/simpson_et_al_si_vernaculars_3701.csv > /app/public/converted_csv/simpson_et_al_si_vernaculars_3701.csv_sorted
[INFO] [2021-04-08 11:02:35] Converted: /app/public/converted_csv/simpson_et_al_si_vernaculars_3701.csv (13352 lines)
[INFO] [2021-04-08 11:02:35] ...occurrences (/app/public/data/simpson_et_al_si/occurrences.tsv)
[CMD] [2021-04-08 11:02:35] /usr/bin/sort /app/public/converted_csv/simpson_et_al_si_occurrences_3701.csv > /app/public/converted_csv/simpson_et_al_si_occurrences_3701.csv_sorted
[INFO] [2021-04-08 11:02:36] Converted: /app/public/converted_csv/simpson_et_al_si_occurrences_3701.csv (13351 lines)
[INFO] [2021-04-08 11:02:36] ...measurements (/app/public/data/simpson_et_al_si/measurement_or_fact.tsv)
[CMD] [2021-04-08 11:02:36] /usr/bin/sort /app/public/converted_csv/simpson_et_al_si_measurements_3701.csv > /app/public/converted_csv/simpson_et_al_si_measurements_3701.csv_sorted
[INFO] [2021-04-08 11:02:36] Converted: /app/public/converted_csv/simpson_et_al_si_measurements_3701.csv (13351 lines)
[STOP] [2021-04-08 11:02:36] convert_to_csv
[START] [2021-04-08 11:02:36] calculate_delta
[INFO] [2021-04-08 11:02:36] Looping over 5 formats...
[INFO] [2021-04-08 11:02:36] ...refs (/app/public/data/simpson_et_al_si/references.tsv)
[CMD] [2021-04-08 11:02:36] echo "0a" > /app/public/diff/simpson_et_al_si_refs_3701.diff
[CMD] [2021-04-08 11:02:37] tail -n +1 /app/public/converted_csv/simpson_et_al_si_refs_3701.csv >> /app/public/diff/simpson_et_al_si_refs_3701.diff
[CMD] [2021-04-08 11:02:37] echo "." >> /app/public/diff/simpson_et_al_si_refs_3701.diff
[INFO] [2021-04-08 11:02:38] Created diff: /app/public/diff/simpson_et_al_si_refs_3701.diff (13382 lines)
[INFO] [2021-04-08 11:02:38] ...nodes (/app/public/data/simpson_et_al_si/taxa.tsv)
[CMD] [2021-04-08 11:02:38] echo "0a" > /app/public/diff/simpson_et_al_si_nodes_3701.diff
[CMD] [2021-04-08 11:02:38] tail -n +1 /app/public/converted_csv/simpson_et_al_si_nodes_3701.csv >> /app/public/diff/simpson_et_al_si_nodes_3701.diff
[CMD] [2021-04-08 11:02:39] echo "." >> /app/public/diff/simpson_et_al_si_nodes_3701.diff
[INFO] [2021-04-08 11:02:40] Created diff: /app/public/diff/simpson_et_al_si_nodes_3701.diff (13353 lines)
[INFO] [2021-04-08 11:02:40] ...vernaculars (/app/public/data/simpson_et_al_si/common_names.tsv)
[CMD] [2021-04-08 11:02:40] echo "0a" > /app/public/diff/simpson_et_al_si_vernaculars_3701.diff
[CMD] [2021-04-08 11:02:40] tail -n +1 /app/public/converted_csv/simpson_et_al_si_vernaculars_3701.csv >> /app/public/diff/simpson_et_al_si_vernaculars_3701.diff
[CMD] [2021-04-08 11:02:41] echo "." >> /app/public/diff/simpson_et_al_si_vernaculars_3701.diff
[INFO] [2021-04-08 11:02:41] Created diff: /app/public/diff/simpson_et_al_si_vernaculars_3701.diff (13354 lines)
[INFO] [2021-04-08 11:02:41] ...occurrences (/app/public/data/simpson_et_al_si/occurrences.tsv)
[CMD] [2021-04-08 11:02:41] echo "0a" > /app/public/diff/simpson_et_al_si_occurrences_3701.diff
[CMD] [2021-04-08 11:02:42] tail -n +1 /app/public/converted_csv/simpson_et_al_si_occurrences_3701.csv >> /app/public/diff/simpson_et_al_si_occurrences_3701.diff
[CMD] [2021-04-08 11:02:43] echo "." >> /app/public/diff/simpson_et_al_si_occurrences_3701.diff
[INFO] [2021-04-08 11:02:43] Created diff: /app/public/diff/simpson_et_al_si_occurrences_3701.diff (13353 lines)
[INFO] [2021-04-08 11:02:43] ...measurements (/app/public/data/simpson_et_al_si/measurement_or_fact.tsv)
[CMD] [2021-04-08 11:02:43] echo "0a" > /app/public/diff/simpson_et_al_si_measurements_3701.diff
[CMD] [2021-04-08 11:02:44] tail -n +1 /app/public/converted_csv/simpson_et_al_si_measurements_3701.csv >> /app/public/diff/simpson_et_al_si_measurements_3701.diff
[CMD] [2021-04-08 11:02:44] echo "." >> /app/public/diff/simpson_et_al_si_measurements_3701.diff
[INFO] [2021-04-08 11:02:45] Created diff: /app/public/diff/simpson_et_al_si_measurements_3701.diff (13353 lines)
[STOP] [2021-04-08 11:02:45] calculate_delta
[START] [2021-04-08 11:02:45] parse_diff_and_store
[INFO] [2021-04-08 11:02:45] Handling diff: /app/public/diff/simpson_et_al_si_refs_3701.diff (13382 lines)
[INFO] [2021-04-08 11:02:45] Loading refs diff file into memory (13382 /app/public/diff/simpson_et_al_si_refs_3701.diff lines)...
[INFO] [2021-04-08 11:02:51] Handling diff: /app/public/diff/simpson_et_al_si_nodes_3701.diff (13353 lines)
[INFO] [2021-04-08 11:02:52] Loading nodes diff file into memory (13353 /app/public/diff/simpson_et_al_si_nodes_3701.diff lines)...
[INFO] [2021-04-08 11:02:58] Handling diff: /app/public/diff/simpson_et_al_si_vernaculars_3701.diff (13354 lines)
[INFO] [2021-04-08 11:02:58] Loading vernaculars diff file into memory (13354 /app/public/diff/simpson_et_al_si_vernaculars_3701.diff lines)...
[INFO] [2021-04-08 11:03:00] Handling diff: /app/public/diff/simpson_et_al_si_occurrences_3701.diff (13353 lines)
[INFO] [2021-04-08 11:03:01] Loading occurrences diff file into memory (13353 /app/public/diff/simpson_et_al_si_occurrences_3701.diff lines)...
[INFO] [2021-04-08 11:03:03] Handling diff: /app/public/diff/simpson_et_al_si_measurements_3701.diff (13353 lines)
[INFO] [2021-04-08 11:03:04] Loading measurements diff file into memory (13353 /app/public/diff/simpson_et_al_si_measurements_3701.diff lines)...
[INFO] [2021-04-08 11:03:13] Storing 13380 References
[INFO] [2021-04-08 11:03:13] Processing group of 13380 in 14 groups of 1000
[INFO] [2021-04-08 11:03:14] Average Time: NaN
[INFO] [2021-04-08 11:03:14] Total Time: 1s
[STOP] [2021-04-08 11:03:14] parse_diff_and_store
[ERR] [2021-04-08 11:03:14] RuntimeError
[ERR] [2021-04-08 11:03:14] ActiveRecord::ValueTooLong while parsing something around here: [{"id":null,"body":"\"First North American Records of the Eastasian Metallic WoodBoring","resource_id":701,"harvest_id":3701,"resource_pk":"Beetle Agrilus smaragdifrons Ganglbauer (Coleoptera:Buprestidae: Agrilinae), a Specialist on Tree of Heaven (Ailanthus altissima, Simaroubaceae). Proceedings of the Entomological Society of Washington. pp. 408-422. Entomological Society of Washington. Washington, DC https://doi.org/10.4289/0013-8797.119.3.408\"","url":null,"doi":null,"removed_by_harvest_id":null,"created_at":null,"updated_at":null},{"id":null,"body":"Florida Department of Agriculture and Consumer Services","resource_id":701,"harvest_id":3701,"resource_pk":"Theraphosidae), an Exotic Tarantula Established in Florida\"","url":null,"doi":null,"removed_by_harvest_id":null,"created_at":null,"updated_at":null},{"id":null,"body":"\"The Mexican Redrump, Brachypelma vagans (Araneae:","resource_id":701,"harvest_id":3701,"resource_pk":"Theraphosidae), an Exotic Tarantula Established in Florida. Entomology Circular No. 394. pp. 2 p. Florida Department of Agriculture and Consumer Services. Gainesville, Florida https://www.fdacs.gov/content/download/10780/file/ent394.pdf\"","url":null,"doi":null,"removed_by_harvest_id":null,"created_at":null,"updated_at":null}]
[ERR] [2021-04-08 11:03:14] ../models/resource_harvester.rb:405:in `rescue in block (2 levels) in store_new'
[ERR] [2021-04-08 11:03:14] ../models/resource_harvester.rb:396:in `block (2 levels) in store_new'
[ERR] [2021-04-08 11:03:14] ../models/logged_process.rb:49:in `block in in_groups'
[ERR] [2021-04-08 11:03:14] ../models/logged_process.rb:47:in `in_groups'
[ERR] [2021-04-08 11:03:14] ../models/resource_harvester.rb:392:in `block in store_new'
[ERR] [2021-04-08 11:03:14] ../models/resource_harvester.rb:386:in `each'
[ERR] [2021-04-08 11:03:14] ../models/resource_harvester.rb:386:in `store_new'
[ERR] [2021-04-08 11:03:14] ../models/resource_harvester.rb:360:in `parse_diff_and_store'
[ERR] [2021-04-08 11:03:14] ../models/resource_harvester.rb:86:in `block (2 levels) in start'
[ERR] [2021-04-08 11:03:14] ../models/logged_process.rb:34:in `run_step'
[ERR] [2021-04-08 11:03:14] ../models/resource_harvester.rb:86:in `block in start'
[ERR] [2021-04-08 11:03:14] ../models/resource_harvester.rb:75:in `each_key'
[ERR] [2021-04-08 11:03:14] ../models/resource_harvester.rb:75:in `start'
[ERR] [2021-04-08 11:03:14] ../models/resource.rb:261:in `harvest'
[ERR] [2021-04-08 11:03:14] ../models/resource.rb:237:in `re_download_opendata_and_harvest'
[ERR] [2021-04-08 11:03:14] bin/rails:4:in `require'
[ERR] [2021-04-08 11:03:14] bin/rails:4:in `<main>'
[STOP] [2021-04-08 11:03:14] logged process, took 43.57
[INFO] [2021-04-08 11:16:11] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-08 11:16:15] ## remove_type: ScientificName
[INFO] [2021-04-08 11:16:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:16:15] [11:16:15.110] Removed 0 Scientificnames
[INFO] [2021-04-08 11:16:15] ## remove_type: Vernacular
[INFO] [2021-04-08 11:16:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:16:15] [11:16:15.112] Removed 0 Vernaculars
[INFO] [2021-04-08 11:16:15] ## remove_type: Article
[INFO] [2021-04-08 11:16:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:16:15] [11:16:15.113] Removed 0 Articles
[INFO] [2021-04-08 11:16:15] ## remove_type: Medium
[INFO] [2021-04-08 11:16:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:16:15] [11:16:15.115] Removed 0 Media
[INFO] [2021-04-08 11:16:15] ## remove_type: Trait
[INFO] [2021-04-08 11:16:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:16:15] [11:16:15.116] Removed 0 Traits
[INFO] [2021-04-08 11:16:15] ## remove_type: MetaTrait
[INFO] [2021-04-08 11:16:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:16:15] [11:16:15.118] Removed 0 Metatraits
[INFO] [2021-04-08 11:16:15] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-08 11:16:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:16:15] [11:16:15.119] Removed 0 Occurrencemetadata
[INFO] [2021-04-08 11:16:15] ## remove_type: Assoc
[INFO] [2021-04-08 11:16:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:16:15] [11:16:15.121] Removed 0 Assocs
[INFO] [2021-04-08 11:16:15] ## remove_type: MetaAssoc
[INFO] [2021-04-08 11:16:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:16:15] [11:16:15.123] Removed 0 Metaassocs
[INFO] [2021-04-08 11:16:15] ## remove_type: Identifier
[INFO] [2021-04-08 11:16:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:16:15] [11:16:15.124] Removed 0 Identifiers
[INFO] [2021-04-08 11:16:15] ## remove_type: Reference
[INFO] [2021-04-08 11:16:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:16:15] [11:16:15.126] Removed 0 References
[INFO] [2021-04-08 11:16:15] ## remove_type: Node
[INFO] [2021-04-08 11:16:15] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:16:15] [11:16:15.136] Removed 0 Nodes
[START] [2021-04-08 11:16:15] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-08 11:16:15] Creating resource from OpenData
[START] [2021-04-08 11:16:17] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-08 11:16:17] Parse meta.xml file and create formats with fields
[WARN] [2021-04-08 11:16:25] (common) IGNORED  (refs) field header: publicationType term: http://eol.org/schema/reference/publicationType
[STOP] [2021-04-08 11:16:25] Parse meta.xml file and create formats with fields
[STOP] [2021-04-08 11:16:25] Creating resource from OpenData
[START] [2021-04-08 11:16:26] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-08 11:16:26] create_harvest_instance
[INFO] [2021-04-08 11:16:26] Created harvest instance #3702
[STOP] [2021-04-08 11:16:26] create_harvest_instance
[START] [2021-04-08 11:16:26] fetch_files
[STOP] [2021-04-08 11:16:26] fetch_files
[START] [2021-04-08 11:16:26] validate_each_file
[INFO] [2021-04-08 11:16:26] Looping over 5 formats...
[INFO] [2021-04-08 11:16:26] ...refs (/app/public/data/simpson_et_al_si/references.tsv)
[INFO] [2021-04-08 11:16:26] Valid: /app/public/converted_csv/simpson_et_al_si_refs_3702.csv (13351 lines)
[INFO] [2021-04-08 11:16:26] ...nodes (/app/public/data/simpson_et_al_si/taxa.tsv)
[INFO] [2021-04-08 11:16:27] Valid: /app/public/converted_csv/simpson_et_al_si_nodes_3702.csv (13351 lines)
[INFO] [2021-04-08 11:16:27] ...vernaculars (/app/public/data/simpson_et_al_si/common_names.tsv)
[INFO] [2021-04-08 11:16:27] Valid: /app/public/converted_csv/simpson_et_al_si_vernaculars_3702.csv (13352 lines)
[INFO] [2021-04-08 11:16:27] ...occurrences (/app/public/data/simpson_et_al_si/occurrences.tsv)
[INFO] [2021-04-08 11:16:27] Valid: /app/public/converted_csv/simpson_et_al_si_occurrences_3702.csv (13351 lines)
[INFO] [2021-04-08 11:16:27] ...measurements (/app/public/data/simpson_et_al_si/measurement_or_fact.tsv)
[INFO] [2021-04-08 11:16:28] Valid: /app/public/converted_csv/simpson_et_al_si_measurements_3702.csv (13351 lines)
[STOP] [2021-04-08 11:16:28] validate_each_file
[START] [2021-04-08 11:16:28] convert_to_csv
[INFO] [2021-04-08 11:16:28] Looping over 5 formats...
[INFO] [2021-04-08 11:16:28] ...refs (/app/public/data/simpson_et_al_si/references.tsv)
[CMD] [2021-04-08 11:16:28] /usr/bin/sort /app/public/converted_csv/simpson_et_al_si_refs_3702.csv > /app/public/converted_csv/simpson_et_al_si_refs_3702.csv_sorted
[INFO] [2021-04-08 11:16:29] Converted: /app/public/converted_csv/simpson_et_al_si_refs_3702.csv (13351 lines)
[INFO] [2021-04-08 11:16:29] ...nodes (/app/public/data/simpson_et_al_si/taxa.tsv)
[CMD] [2021-04-08 11:16:29] /usr/bin/sort /app/public/converted_csv/simpson_et_al_si_nodes_3702.csv > /app/public/converted_csv/simpson_et_al_si_nodes_3702.csv_sorted
[INFO] [2021-04-08 11:16:29] Converted: /app/public/converted_csv/simpson_et_al_si_nodes_3702.csv (13351 lines)
[INFO] [2021-04-08 11:16:29] ...vernaculars (/app/public/data/simpson_et_al_si/common_names.tsv)
[CMD] [2021-04-08 11:16:29] /usr/bin/sort /app/public/converted_csv/simpson_et_al_si_vernaculars_3702.csv > /app/public/converted_csv/simpson_et_al_si_vernaculars_3702.csv_sorted
[INFO] [2021-04-08 11:16:30] Converted: /app/public/converted_csv/simpson_et_al_si_vernaculars_3702.csv (13352 lines)
[INFO] [2021-04-08 11:16:30] ...occurrences (/app/public/data/simpson_et_al_si/occurrences.tsv)
[CMD] [2021-04-08 11:16:30] /usr/bin/sort /app/public/converted_csv/simpson_et_al_si_occurrences_3702.csv > /app/public/converted_csv/simpson_et_al_si_occurrences_3702.csv_sorted
[INFO] [2021-04-08 11:16:30] Converted: /app/public/converted_csv/simpson_et_al_si_occurrences_3702.csv (13351 lines)
[INFO] [2021-04-08 11:16:30] ...measurements (/app/public/data/simpson_et_al_si/measurement_or_fact.tsv)
[CMD] [2021-04-08 11:16:30] /usr/bin/sort /app/public/converted_csv/simpson_et_al_si_measurements_3702.csv > /app/public/converted_csv/simpson_et_al_si_measurements_3702.csv_sorted
[INFO] [2021-04-08 11:16:31] Converted: /app/public/converted_csv/simpson_et_al_si_measurements_3702.csv (13351 lines)
[STOP] [2021-04-08 11:16:31] convert_to_csv
[START] [2021-04-08 11:16:31] calculate_delta
[INFO] [2021-04-08 11:16:31] Looping over 5 formats...
[INFO] [2021-04-08 11:16:31] ...refs (/app/public/data/simpson_et_al_si/references.tsv)
[CMD] [2021-04-08 11:16:31] echo "0a" > /app/public/diff/simpson_et_al_si_refs_3702.diff
[CMD] [2021-04-08 11:16:31] tail -n +1 /app/public/converted_csv/simpson_et_al_si_refs_3702.csv >> /app/public/diff/simpson_et_al_si_refs_3702.diff
[CMD] [2021-04-08 11:16:32] echo "." >> /app/public/diff/simpson_et_al_si_refs_3702.diff
[INFO] [2021-04-08 11:16:33] Created diff: /app/public/diff/simpson_et_al_si_refs_3702.diff (13353 lines)
[INFO] [2021-04-08 11:16:33] ...nodes (/app/public/data/simpson_et_al_si/taxa.tsv)
[CMD] [2021-04-08 11:16:33] echo "0a" > /app/public/diff/simpson_et_al_si_nodes_3702.diff
[CMD] [2021-04-08 11:16:33] tail -n +1 /app/public/converted_csv/simpson_et_al_si_nodes_3702.csv >> /app/public/diff/simpson_et_al_si_nodes_3702.diff
[CMD] [2021-04-08 11:16:34] echo "." >> /app/public/diff/simpson_et_al_si_nodes_3702.diff
[INFO] [2021-04-08 11:16:34] Created diff: /app/public/diff/simpson_et_al_si_nodes_3702.diff (13353 lines)
[INFO] [2021-04-08 11:16:34] ...vernaculars (/app/public/data/simpson_et_al_si/common_names.tsv)
[CMD] [2021-04-08 11:16:34] echo "0a" > /app/public/diff/simpson_et_al_si_vernaculars_3702.diff
[CMD] [2021-04-08 11:16:35] tail -n +1 /app/public/converted_csv/simpson_et_al_si_vernaculars_3702.csv >> /app/public/diff/simpson_et_al_si_vernaculars_3702.diff
[CMD] [2021-04-08 11:16:35] echo "." >> /app/public/diff/simpson_et_al_si_vernaculars_3702.diff
[INFO] [2021-04-08 11:16:36] Created diff: /app/public/diff/simpson_et_al_si_vernaculars_3702.diff (13354 lines)
[INFO] [2021-04-08 11:16:36] ...occurrences (/app/public/data/simpson_et_al_si/occurrences.tsv)
[CMD] [2021-04-08 11:16:36] echo "0a" > /app/public/diff/simpson_et_al_si_occurrences_3702.diff
[CMD] [2021-04-08 11:16:36] tail -n +1 /app/public/converted_csv/simpson_et_al_si_occurrences_3702.csv >> /app/public/diff/simpson_et_al_si_occurrences_3702.diff
[CMD] [2021-04-08 11:16:37] echo "." >> /app/public/diff/simpson_et_al_si_occurrences_3702.diff
[INFO] [2021-04-08 11:16:38] Created diff: /app/public/diff/simpson_et_al_si_occurrences_3702.diff (13353 lines)
[INFO] [2021-04-08 11:16:38] ...measurements (/app/public/data/simpson_et_al_si/measurement_or_fact.tsv)
[CMD] [2021-04-08 11:16:38] echo "0a" > /app/public/diff/simpson_et_al_si_measurements_3702.diff
[CMD] [2021-04-08 11:16:38] tail -n +1 /app/public/converted_csv/simpson_et_al_si_measurements_3702.csv >> /app/public/diff/simpson_et_al_si_measurements_3702.diff
[CMD] [2021-04-08 11:16:39] echo "." >> /app/public/diff/simpson_et_al_si_measurements_3702.diff
[INFO] [2021-04-08 11:16:39] Created diff: /app/public/diff/simpson_et_al_si_measurements_3702.diff (13353 lines)
[STOP] [2021-04-08 11:16:39] calculate_delta
[START] [2021-04-08 11:16:39] parse_diff_and_store
[INFO] [2021-04-08 11:16:39] Handling diff: /app/public/diff/simpson_et_al_si_refs_3702.diff (13353 lines)
[INFO] [2021-04-08 11:16:40] Loading refs diff file into memory (13353 /app/public/diff/simpson_et_al_si_refs_3702.diff lines)...
[INFO] [2021-04-08 11:16:45] Handling diff: /app/public/diff/simpson_et_al_si_nodes_3702.diff (13353 lines)
[INFO] [2021-04-08 11:16:46] Loading nodes diff file into memory (13353 /app/public/diff/simpson_et_al_si_nodes_3702.diff lines)...
[INFO] [2021-04-08 11:16:53] Handling diff: /app/public/diff/simpson_et_al_si_vernaculars_3702.diff (13354 lines)
[INFO] [2021-04-08 11:16:53] Loading vernaculars diff file into memory (13354 /app/public/diff/simpson_et_al_si_vernaculars_3702.diff lines)...
[INFO] [2021-04-08 11:16:55] Handling diff: /app/public/diff/simpson_et_al_si_occurrences_3702.diff (13353 lines)
[INFO] [2021-04-08 11:16:56] Loading occurrences diff file into memory (13353 /app/public/diff/simpson_et_al_si_occurrences_3702.diff lines)...
[INFO] [2021-04-08 11:16:58] Handling diff: /app/public/diff/simpson_et_al_si_measurements_3702.diff (13353 lines)
[INFO] [2021-04-08 11:16:58] Loading measurements diff file into memory (13353 /app/public/diff/simpson_et_al_si_measurements_3702.diff lines)...
[INFO] [2021-04-08 11:17:08] Storing 13351 References
[INFO] [2021-04-08 11:17:08] Processing group of 13351 in 14 groups of 1000
[INFO] [2021-04-08 11:17:12] Average Time: 0.267
[INFO] [2021-04-08 11:17:12] Total Time: 4s
[INFO] [2021-04-08 11:17:12] last 3 / first 3: 0.45
[INFO] [2021-04-08 11:17:12] Std.Dev: 0.10954451150103323; Max: 0.46
[INFO] [2021-04-08 11:17:12] Storing 20705 ScientificNames
[INFO] [2021-04-08 11:17:12] Processing group of 20705 in 21 groups of 1000
[INFO] [2021-04-08 11:17:18] Average Time: 0.284
[INFO] [2021-04-08 11:17:18] Total Time: 7s
[INFO] [2021-04-08 11:17:18] last 3 / first 3: 0.87
[INFO] [2021-04-08 11:17:18] Std.Dev: 0.03162277660168379; Max: 0.37
[INFO] [2021-04-08 11:17:18] Storing 20705 Nodes
[INFO] [2021-04-08 11:17:18] Processing group of 20705 in 21 groups of 1000
[INFO] [2021-04-08 11:17:24] Average Time: 0.255
[INFO] [2021-04-08 11:17:24] Total Time: 6s
[INFO] [2021-04-08 11:17:24] last 3 / first 3: 0.94
[INFO] [2021-04-08 11:17:24] Std.Dev: 0.03162277660168379; Max: 0.33
[INFO] [2021-04-08 11:17:24] Storing 13352 Vernaculars
[INFO] [2021-04-08 11:17:24] Processing group of 13352 in 14 groups of 1000
[INFO] [2021-04-08 11:17:26] Average Time: 0.147
[INFO] [2021-04-08 11:17:26] Total Time: 3s
[INFO] [2021-04-08 11:17:26] last 3 / first 3: 0.75
[INFO] [2021-04-08 11:17:26] Std.Dev: 0.03162277660168379; Max: 0.16
[INFO] [2021-04-08 11:17:26] Storing 13351 Occurrences
[INFO] [2021-04-08 11:17:26] Processing group of 13351 in 14 groups of 1000
[INFO] [2021-04-08 11:17:27] Average Time: 0.104
[INFO] [2021-04-08 11:17:27] Total Time: 2s
[INFO] [2021-04-08 11:17:27] last 3 / first 3: 0.59
[INFO] [2021-04-08 11:17:27] Std.Dev: 0.03162277660168379; Max: 0.2
[INFO] [2021-04-08 11:17:27] Storing 13351 TraitsReferences
[INFO] [2021-04-08 11:17:27] Processing group of 13351 in 14 groups of 1000
[INFO] [2021-04-08 11:17:28] Average Time: 0.069
[INFO] [2021-04-08 11:17:28] Total Time: 1s
[INFO] [2021-04-08 11:17:28] last 3 / first 3: 1.09
[INFO] [2021-04-08 11:17:28] Std.Dev: 0.03162277660168379; Max: 0.17
[INFO] [2021-04-08 11:17:28] Storing 13351 Traits
[INFO] [2021-04-08 11:17:28] Processing group of 13351 in 14 groups of 1000
[INFO] [2021-04-08 11:17:32] Average Time: 0.27
[INFO] [2021-04-08 11:17:32] Total Time: 4s
[INFO] [2021-04-08 11:17:32] last 3 / first 3: 0.78
[INFO] [2021-04-08 11:17:32] Std.Dev: 0.05477225575051661; Max: 0.35
[INFO] [2021-04-08 11:17:32] Storing 40053 MetaTraits
[INFO] [2021-04-08 11:17:32] Processing group of 40053 in 41 groups of 1000
[INFO] [2021-04-08 11:17:37] Average Time: 0.113
[INFO] [2021-04-08 11:17:37] Total Time: 5s
[INFO] [2021-04-08 11:17:37] last 3 / first 3: 0.7
[INFO] [2021-04-08 11:17:37] Std.Dev: 0.03162277660168379; Max: 0.19
[STOP] [2021-04-08 11:17:37] parse_diff_and_store
[START] [2021-04-08 11:17:37] resolve_keys
[INFO] [2021-04-08 11:17:56] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-08 11:17:57] traits to occurrences...
[INFO] [2021-04-08 11:17:58] traits to nodes (through occurrences)...
[INFO] [2021-04-08 11:17:58] Traits to sex term...
[INFO] [2021-04-08 11:17:58] Traits to lifestage term...
[INFO] [2021-04-08 11:17:58] MetaTraits to traits...
[INFO] [2021-04-08 11:17:59] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-08 11:18:00] Assocs to occurrences...
[INFO] [2021-04-08 11:18:00] Assocs to nodes...
[INFO] [2021-04-08 11:18:00] Assoc to sex term...
[INFO] [2021-04-08 11:18:00] Assoc to lifestage term...
[INFO] [2021-04-08 11:18:00] MetaAssoc to assocs...
[STOP] [2021-04-08 11:18:00] resolve_keys
[START] [2021-04-08 11:18:00] hold_for_later_1
[STOP] [2021-04-08 11:18:00] hold_for_later_1
[START] [2021-04-08 11:18:00] hold_for_later_2
[STOP] [2021-04-08 11:18:00] hold_for_later_2
[START] [2021-04-08 11:18:00] resolve_missing_parents
[STOP] [2021-04-08 11:18:01] resolve_missing_parents
[START] [2021-04-08 11:18:01] rebuild_nodes
[START] [2021-04-08 11:18:01] Flattener#flatten
[START] [2021-04-08 11:18:01] Flattener#study_resource
[START] [2021-04-08 11:18:01] Flattener#build_ancestry
[STOP] [2021-04-08 11:18:02] Flattener#build_ancestry
[INFO] [2021-04-08 11:18:02] 20705 ancestry keys
[START] [2021-04-08 11:18:02] build_node_ancestors
[INFO] [2021-04-08 11:18:02] old ancestors deleted.
[STOP] [2021-04-08 11:18:13] build_node_ancestors
[START] [2021-04-08 11:18:14] Flattener#propagate_ancestor_ids
[STOP] [2021-04-08 11:18:17] Flattener#propagate_ancestor_ids
[STOP] [2021-04-08 11:18:17] Flattener#flatten
[STOP] [2021-04-08 11:18:17] rebuild_nodes
[START] [2021-04-08 11:18:17] resolve_missing_media_owners
[STOP] [2021-04-08 11:18:17] resolve_missing_media_owners
[START] [2021-04-08 11:18:17] sanitize_media_verbatims
[STOP] [2021-04-08 11:18:17] sanitize_media_verbatims
[START] [2021-04-08 11:18:17] queue_downloads
[STOP] [2021-04-08 11:18:17] queue_downloads
[START] [2021-04-08 11:18:17] parse_names
[WARN] [2021-04-08 11:18:17] I see 20705 names which still need to be parsed.
[WARN] [2021-04-08 11:18:31] I see 1960 names which still need to be parsed.
[WARN] [2021-04-08 11:18:33] I see 184 names which still need to be parsed.
[WARN] [2021-04-08 11:18:34] I see 23 names which still need to be parsed.
[WARN] [2021-04-08 11:18:36] I see 21 names which still need to be parsed.
[WARN] [2021-04-08 11:18:37] I see 20 names which still need to be parsed.
[WARN] [2021-04-08 11:18:38] I see 19 names which still need to be parsed.
[WARN] [2021-04-08 11:18:39] I see 18 names which still need to be parsed.
[WARN] [2021-04-08 11:18:40] I see 17 names which still need to be parsed.
[WARN] [2021-04-08 11:18:41] I see 16 names which still need to be parsed.
[WARN] [2021-04-08 11:18:43] I see 15 names which still need to be parsed.
[STOP] [2021-04-08 11:18:44] parse_names
[START] [2021-04-08 11:18:44] denormalize_canonical_names_to_nodes
[STOP] [2021-04-08 11:18:44] denormalize_canonical_names_to_nodes
[START] [2021-04-08 11:18:44] match_nodes
[START] [2021-04-08 11:18:44] map_all_nodes_to_pages
[WARN] [2021-04-08 11:20:09] cannot match node with blank canonical: Node#92696289
[WARN] [2021-04-08 11:22:11] cannot match node with blank canonical: Node#92693573
[WARN] [2021-04-08 11:23:34] cannot match node with blank canonical: Node#92692413
[WARN] [2021-04-08 11:23:46] cannot match node with blank canonical: Node#92689174
[WARN] [2021-04-08 11:24:31] cannot match node with blank canonical: Node#92691565
[WARN] [2021-04-08 11:25:17] cannot match node with blank canonical: Node#92694781
[WARN] [2021-04-08 11:25:21] cannot match node with blank canonical: Node#92693112
[WARN] [2021-04-08 11:25:33] cannot match node with blank canonical: Node#92686035
[WARN] [2021-04-08 11:25:38] cannot match node with blank canonical: Node#92694247
[WARN] [2021-04-08 11:25:42] cannot match node with blank canonical: Node#92699722
[WARN] [2021-04-08 11:25:49] cannot match node with blank canonical: Node#92687661
[WARN] [2021-04-08 11:25:50] cannot match node with blank canonical: Node#92697208
[WARN] [2021-04-08 11:27:15] cannot match node with blank canonical: Node#92683688
[WARN] [2021-04-08 11:27:15] cannot match node with blank canonical: Node#92684762
[WARN] [2021-04-08 11:27:15] cannot match node with blank canonical: Node#92684763
[WARN] [2021-04-08 11:27:15] cannot match node with blank canonical: Node#92688927
[WARN] [2021-04-08 11:27:15] cannot match node with blank canonical: Node#92688928
[WARN] [2021-04-08 11:27:15] cannot match node with blank canonical: Node#92697209
[WARN] [2021-04-08 11:27:15] cannot match node with blank canonical: Node#92697210
[WARN] [2021-04-08 11:27:16] cannot match node with blank canonical: Node#92693767
[WARN] [2021-04-08 11:27:17] cannot match node with blank canonical: Node#92690105
[WARN] [2021-04-08 11:27:18] cannot match node with blank canonical: Node#92697190
[WARN] [2021-04-08 11:27:18] cannot match node with blank canonical: Node#92688926
[WARN] [2021-04-08 11:27:19] cannot match node with blank canonical: Node#92684953
[WARN] [2021-04-08 11:27:20] cannot match node with blank canonical: Node#92698800
[WARN] [2021-04-08 11:27:20] cannot match node with blank canonical: Node#92684761
[STOP] [2021-04-08 11:27:21] map_all_nodes_to_pages
[INFO] [2021-04-08 11:27:21] 1349 Unmatched nodes (of 20705)! That's too many to output. Full list in /app/public/data/simpson_et_al_si/unmatched_nodes.txt ; First 10: Aceria guerreronis (#92682457); Aceria litchii (#92682459); Aceria litchii (#92682460); Aceria malherbae (#92682461); Aculops fuchsiae (#92682676); Colomerus gardeniella (#92687532); Eriophyes (#92689993); Eriophyes cynodoniensis (#92689994); Eriophyes fici (#92689995); Eriophyes granati (#92689997)
[START] [2021-04-08 11:27:21] update_nodes
[STOP] [2021-04-08 11:27:29] update_nodes
[STOP] [2021-04-08 11:27:29] match_nodes
[START] [2021-04-08 11:27:29] reindex_search
[STOP] [2021-04-08 11:27:47] reindex_search
[START] [2021-04-08 11:27:47] normalize_units
[STOP] [2021-04-08 11:27:47] normalize_units
[START] [2021-04-08 11:27:47] calculate_statistics
[STOP] [2021-04-08 11:27:48] calculate_statistics
[START] [2021-04-08 11:27:48] complete_harvest_instance
[START] [2021-04-08 11:27:48] overall_tsv_creation
[INFO] [2021-04-08 11:27:48] Processing group of 20705 in 3 batches of 10000
[INFO] [2021-04-08 11:30:15] 6235 Traits (unfiltered)...
[INFO] [2021-04-08 11:31:29] 6235 Traits (filtered)...
[INFO] [2021-04-08 11:31:29] 0 Associations (filtered)...
[INFO] [2021-04-08 11:31:31] 12470 metadata added.
[INFO] [2021-04-08 11:31:31] 0 metadata added.
[INFO] [2021-04-08 11:33:17] 6637 Traits (unfiltered)...
[INFO] [2021-04-08 11:34:30] 6637 Traits (filtered)...
[INFO] [2021-04-08 11:34:30] 0 Associations (filtered)...
[INFO] [2021-04-08 11:34:32] 13274 metadata added.
[INFO] [2021-04-08 11:34:32] 0 metadata added.
[INFO] [2021-04-08 11:35:41] 479 Traits (unfiltered)...
[INFO] [2021-04-08 11:36:16] 479 Traits (filtered)...
[INFO] [2021-04-08 11:36:16] 0 Associations (filtered)...
[INFO] [2021-04-08 11:36:16] 958 metadata added.
[INFO] [2021-04-08 11:36:16] 0 metadata added.
[INFO] [2021-04-08 11:36:42] Average Time: 121.157
[INFO] [2021-04-08 11:36:42] Total Time: 8m55s
[STOP] [2021-04-08 11:36:42] overall_tsv_creation
[INFO] [2021-04-08 11:36:42] Done. Check your files:
[INFO] [2021-04-08 11:36:43] (18744 lines) /app/public/data/simpson_et_al_si/publish_nodes.tsv
[INFO] [2021-04-08 11:36:43] (95161 lines) /app/public/data/simpson_et_al_si/publish_node_ancestors.tsv
[INFO] [2021-04-08 11:36:44] (20705 lines) /app/public/data/simpson_et_al_si/publish_scientific_names.tsv
[INFO] [2021-04-08 11:36:44] (13351 lines) /app/public/data/simpson_et_al_si/publish_vernaculars.tsv
[INFO] [2021-04-08 11:36:45] (13352 lines) /app/public/data/simpson_et_al_si/publish_traits.tsv
[INFO] [2021-04-08 11:36:45] (26703 lines) /app/public/data/simpson_et_al_si/publish_metadata.tsv
[STOP] [2021-04-08 11:36:46] complete_harvest_instance
[START] [2021-04-08 11:36:46] completed
[STOP] [2021-04-08 11:36:46] completed
[STOP] [2021-04-08 11:36:46] logged process, took 1220.67
[INFO] [2021-04-08 11:55:19] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2021-04-08 11:55:22] ## remove_type: ScientificName
[INFO] [2021-04-08 11:55:22] ++ Calling delete_all on 20705 instances...
[INFO] [2021-04-08 11:55:23] [11:55:23.186] Removed 20705 Scientificnames
[INFO] [2021-04-08 11:55:23] ## remove_type: Vernacular
[INFO] [2021-04-08 11:55:23] ++ Calling delete_all on 13352 instances...
[INFO] [2021-04-08 11:55:23] [11:55:23.318] Removed 13352 Vernaculars
[INFO] [2021-04-08 11:55:23] ## remove_type: Article
[INFO] [2021-04-08 11:55:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:55:23] [11:55:23.320] Removed 0 Articles
[INFO] [2021-04-08 11:55:23] ## remove_type: Medium
[INFO] [2021-04-08 11:55:23] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:55:23] [11:55:23.321] Removed 0 Media
[INFO] [2021-04-08 11:55:23] ## remove_type: Trait
[INFO] [2021-04-08 11:55:23] ++ Calling delete_all on 13351 instances...
[INFO] [2021-04-08 11:55:23] [11:55:23.673] Removed 13351 Traits
[INFO] [2021-04-08 11:55:23] ## remove_type: MetaTrait
[INFO] [2021-04-08 11:55:23] ++ Calling delete_all on 40053 instances...
[INFO] [2021-04-08 11:55:24] [11:55:24.068] Removed 40053 Metatraits
[INFO] [2021-04-08 11:55:24] ## remove_type: OccurrenceMetadatum
[INFO] [2021-04-08 11:55:24] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:55:24] [11:55:24.070] Removed 0 Occurrencemetadata
[INFO] [2021-04-08 11:55:24] ## remove_type: Assoc
[INFO] [2021-04-08 11:55:24] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:55:24] [11:55:24.071] Removed 0 Assocs
[INFO] [2021-04-08 11:55:24] ## remove_type: MetaAssoc
[INFO] [2021-04-08 11:55:24] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:55:24] [11:55:24.073] Removed 0 Metaassocs
[INFO] [2021-04-08 11:55:24] ## remove_type: Identifier
[INFO] [2021-04-08 11:55:24] ++ Calling delete_all on 0 instances...
[INFO] [2021-04-08 11:55:24] [11:55:24.074] Removed 0 Identifiers
[INFO] [2021-04-08 11:55:24] ## remove_type: Reference
[INFO] [2021-04-08 11:55:24] ++ Calling delete_all on 13351 instances...
[INFO] [2021-04-08 11:55:24] [11:55:24.202] Removed 13351 References
[INFO] [2021-04-08 11:55:24] Starting batch with ID 92685276...
[INFO] [2021-04-08 11:55:24] Starting batch with ID 92685276...
[INFO] [2021-04-08 11:55:25] Starting batch with ID 92685276...
[INFO] [2021-04-08 11:55:26] Starting batch with ID 92684609...
[INFO] [2021-04-08 11:55:26] Starting batch with ID 92684609...
[INFO] [2021-04-08 11:55:27] Starting batch with ID 92685908...
[INFO] [2021-04-08 11:55:27] Starting batch with ID 92692086...
[INFO] [2021-04-08 11:55:28] Starting batch with ID 92692086...
[INFO] [2021-04-08 11:55:28] Starting batch with ID 92692086...
[INFO] [2021-04-08 11:55:29] Starting batch with ID 92688791...
[INFO] [2021-04-08 11:55:29] Starting batch with ID 92688791...
[INFO] [2021-04-08 11:55:29] Starting batch with ID 92688791...
[INFO] [2021-04-08 11:55:29] Starting batch with ID 92688791...
[INFO] [2021-04-08 11:55:29] Starting batch with ID 92688791...
[INFO] [2021-04-08 11:55:29] ## remove_type: Node
[INFO] [2021-04-08 11:55:29] ++ Calling delete_all on 20705 instances...
[INFO] [2021-04-08 11:55:30] [11:55:30.665] Removed 20705 Nodes
[START] [2021-04-08 11:55:39] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-08 11:55:40] Creating resource from OpenData
[START] [2021-04-08 11:55:41] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-08 11:55:41] Parse meta.xml file and create formats with fields
[WARN] [2021-04-08 11:55:51] (common) IGNORED  (refs) field header: publicationType term: http://eol.org/schema/reference/publicationType
[STOP] [2021-04-08 11:55:51] Parse meta.xml file and create formats with fields
[STOP] [2021-04-08 11:55:51] Creating resource from OpenData
[START] [2021-04-08 11:55:51] logged process: 5ecc716a6a5541910d0c854f5a0c8d1651b82ad0 Improved MetaXml.ignore and added publisher to media (ignored)
[START] [2021-04-08 11:55:51] create_harvest_instance
[INFO] [2021-04-08 11:55:51] Created harvest instance #3703
[STOP] [2021-04-08 11:55:51] create_harvest_instance
[START] [2021-04-08 11:55:51] fetch_files
[STOP] [2021-04-08 11:55:51] fetch_files
[START] [2021-04-08 11:55:51] validate_each_file
[INFO] [2021-04-08 11:55:51] Looping over 5 formats...
[INFO] [2021-04-08 11:55:51] ...refs (/app/public/data/simpson_et_al_si/references.tsv)
[INFO] [2021-04-08 11:55:51] Valid: /app/public/converted_csv/simpson_et_al_si_refs_3703.csv (1381 lines)
[INFO] [2021-04-08 11:55:51] ...nodes (/app/public/data/simpson_et_al_si/taxa.tsv)
[INFO] [2021-04-08 11:55:52] Valid: /app/public/converted_csv/simpson_et_al_si_nodes_3703.csv (13351 lines)
[INFO] [2021-04-08 11:55:52] ...vernaculars (/app/public/data/simpson_et_al_si/common_names.tsv)
[INFO] [2021-04-08 11:55:52] Valid: /app/public/converted_csv/simpson_et_al_si_vernaculars_3703.csv (13352 lines)
[INFO] [2021-04-08 11:55:52] ...occurrences (/app/public/data/simpson_et_al_si/occurrences.tsv)
[INFO] [2021-04-08 11:55:52] Valid: /app/public/converted_csv/simpson_et_al_si_occurrences_3703.csv (13351 lines)
[INFO] [2021-04-08 11:55:52] ...measurements (/app/public/data/simpson_et_al_si/measurement_or_fact.tsv)
[INFO] [2021-04-08 11:55:53] Valid: /app/public/converted_csv/simpson_et_al_si_measurements_3703.csv (13351 lines)
[STOP] [2021-04-08 11:55:53] validate_each_file
[START] [2021-04-08 11:55:53] convert_to_csv
[INFO] [2021-04-08 11:55:53] Looping over 5 formats...
[INFO] [2021-04-08 11:55:53] ...refs (/app/public/data/simpson_et_al_si/references.tsv)
[CMD] [2021-04-08 11:55:53] /usr/bin/sort /app/public/converted_csv/simpson_et_al_si_refs_3703.csv > /app/public/converted_csv/simpson_et_al_si_refs_3703.csv_sorted
[INFO] [2021-04-08 11:55:54] Converted: /app/public/converted_csv/simpson_et_al_si_refs_3703.csv (1381 lines)
[INFO] [2021-04-08 11:55:54] ...nodes (/app/public/data/simpson_et_al_si/taxa.tsv)
[CMD] [2021-04-08 11:55:54] /usr/bin/sort /app/public/converted_csv/simpson_et_al_si_nodes_3703.csv > /app/public/converted_csv/simpson_et_al_si_nodes_3703.csv_sorted
[INFO] [2021-04-08 11:55:54] Converted: /app/public/converted_csv/simpson_et_al_si_nodes_3703.csv (13351 lines)
[INFO] [2021-04-08 11:55:54] ...vernaculars (/app/public/data/simpson_et_al_si/common_names.tsv)
[CMD] [2021-04-08 11:55:54] /usr/bin/sort /app/public/converted_csv/simpson_et_al_si_vernaculars_3703.csv > /app/public/converted_csv/simpson_et_al_si_vernaculars_3703.csv_sorted
[INFO] [2021-04-08 11:55:55] Converted: /app/public/converted_csv/simpson_et_al_si_vernaculars_3703.csv (13352 lines)
[INFO] [2021-04-08 11:55:55] ...occurrences (/app/public/data/simpson_et_al_si/occurrences.tsv)
[CMD] [2021-04-08 11:55:55] /usr/bin/sort /app/public/converted_csv/simpson_et_al_si_occurrences_3703.csv > /app/public/converted_csv/simpson_et_al_si_occurrences_3703.csv_sorted
[INFO] [2021-04-08 11:55:55] Converted: /app/public/converted_csv/simpson_et_al_si_occurrences_3703.csv (13351 lines)
[INFO] [2021-04-08 11:55:55] ...measurements (/app/public/data/simpson_et_al_si/measurement_or_fact.tsv)
[CMD] [2021-04-08 11:55:55] /usr/bin/sort /app/public/converted_csv/simpson_et_al_si_measurements_3703.csv > /app/public/converted_csv/simpson_et_al_si_measurements_3703.csv_sorted
[INFO] [2021-04-08 11:55:56] Converted: /app/public/converted_csv/simpson_et_al_si_measurements_3703.csv (13351 lines)
[STOP] [2021-04-08 11:55:56] convert_to_csv
[START] [2021-04-08 11:55:56] calculate_delta
[INFO] [2021-04-08 11:55:56] Looping over 5 formats...
[INFO] [2021-04-08 11:55:56] ...refs (/app/public/data/simpson_et_al_si/references.tsv)
[CMD] [2021-04-08 11:55:56] echo "0a" > /app/public/diff/simpson_et_al_si_refs_3703.diff
[CMD] [2021-04-08 11:55:56] tail -n +1 /app/public/converted_csv/simpson_et_al_si_refs_3703.csv >> /app/public/diff/simpson_et_al_si_refs_3703.diff
[CMD] [2021-04-08 11:55:57] echo "." >> /app/public/diff/simpson_et_al_si_refs_3703.diff
[INFO] [2021-04-08 11:55:58] Created diff: /app/public/diff/simpson_et_al_si_refs_3703.diff (1383 lines)
[INFO] [2021-04-08 11:55:58] ...nodes (/app/public/data/simpson_et_al_si/taxa.tsv)
[CMD] [2021-04-08 11:55:58] echo "0a" > /app/public/diff/simpson_et_al_si_nodes_3703.diff
[CMD] [2021-04-08 11:55:58] tail -n +1 /app/public/converted_csv/simpson_et_al_si_nodes_3703.csv >> /app/public/diff/simpson_et_al_si_nodes_3703.diff
[CMD] [2021-04-08 11:55:59] echo "." >> /app/public/diff/simpson_et_al_si_nodes_3703.diff
[INFO] [2021-04-08 11:55:59] Created diff: /app/public/diff/simpson_et_al_si_nodes_3703.diff (13353 lines)
[INFO] [2021-04-08 11:55:59] ...vernaculars (/app/public/data/simpson_et_al_si/common_names.tsv)
[CMD] [2021-04-08 11:55:59] echo "0a" > /app/public/diff/simpson_et_al_si_vernaculars_3703.diff
[CMD] [2021-04-08 11:56:00] tail -n +1 /app/public/converted_csv/simpson_et_al_si_vernaculars_3703.csv >> /app/public/diff/simpson_et_al_si_vernaculars_3703.diff
[CMD] [2021-04-08 11:56:00] echo "." >> /app/public/diff/simpson_et_al_si_vernaculars_3703.diff
[INFO] [2021-04-08 11:56:01] Created diff: /app/public/diff/simpson_et_al_si_vernaculars_3703.diff (13354 lines)
[INFO] [2021-04-08 11:56:01] ...occurrences (/app/public/data/simpson_et_al_si/occurrences.tsv)
[CMD] [2021-04-08 11:56:01] echo "0a" > /app/public/diff/simpson_et_al_si_occurrences_3703.diff
[CMD] [2021-04-08 11:56:01] tail -n +1 /app/public/converted_csv/simpson_et_al_si_occurrences_3703.csv >> /app/public/diff/simpson_et_al_si_occurrences_3703.diff
[CMD] [2021-04-08 11:56:02] echo "." >> /app/public/diff/simpson_et_al_si_occurrences_3703.diff
[INFO] [2021-04-08 11:56:03] Created diff: /app/public/diff/simpson_et_al_si_occurrences_3703.diff (13353 lines)
[INFO] [2021-04-08 11:56:03] ...measurements (/app/public/data/simpson_et_al_si/measurement_or_fact.tsv)
[CMD] [2021-04-08 11:56:03] echo "0a" > /app/public/diff/simpson_et_al_si_measurements_3703.diff
[CMD] [2021-04-08 11:56:03] tail -n +1 /app/public/converted_csv/simpson_et_al_si_measurements_3703.csv >> /app/public/diff/simpson_et_al_si_measurements_3703.diff
[CMD] [2021-04-08 11:56:04] echo "." >> /app/public/diff/simpson_et_al_si_measurements_3703.diff
[INFO] [2021-04-08 11:56:04] Created diff: /app/public/diff/simpson_et_al_si_measurements_3703.diff (13353 lines)
[STOP] [2021-04-08 11:56:04] calculate_delta
[START] [2021-04-08 11:56:04] parse_diff_and_store
[INFO] [2021-04-08 11:56:04] Handling diff: /app/public/diff/simpson_et_al_si_refs_3703.diff (1383 lines)
[INFO] [2021-04-08 11:56:05] Loading refs diff file into memory (1383 /app/public/diff/simpson_et_al_si_refs_3703.diff lines)...
[INFO] [2021-04-08 11:56:06] Handling diff: /app/public/diff/simpson_et_al_si_nodes_3703.diff (13353 lines)
[INFO] [2021-04-08 11:56:06] Loading nodes diff file into memory (13353 /app/public/diff/simpson_et_al_si_nodes_3703.diff lines)...
[INFO] [2021-04-08 11:56:13] Handling diff: /app/public/diff/simpson_et_al_si_vernaculars_3703.diff (13354 lines)
[INFO] [2021-04-08 11:56:13] Loading vernaculars diff file into memory (13354 /app/public/diff/simpson_et_al_si_vernaculars_3703.diff lines)...
[INFO] [2021-04-08 11:56:16] Handling diff: /app/public/diff/simpson_et_al_si_occurrences_3703.diff (13353 lines)
[INFO] [2021-04-08 11:56:16] Loading occurrences diff file into memory (13353 /app/public/diff/simpson_et_al_si_occurrences_3703.diff lines)...
[INFO] [2021-04-08 11:56:18] Handling diff: /app/public/diff/simpson_et_al_si_measurements_3703.diff (13353 lines)
[INFO] [2021-04-08 11:56:18] Loading measurements diff file into memory (13353 /app/public/diff/simpson_et_al_si_measurements_3703.diff lines)...
[INFO] [2021-04-08 11:56:30] Storing 1381 References
[INFO] [2021-04-08 11:56:30] Processing group of 1381 in 2 groups of 1000
[INFO] [2021-04-08 11:56:30] Average Time: 0.135
[INFO] [2021-04-08 11:56:30] Total Time: 1s
[INFO] [2021-04-08 11:56:30] Storing 20705 ScientificNames
[INFO] [2021-04-08 11:56:30] Processing group of 20705 in 21 groups of 1000
[INFO] [2021-04-08 11:56:37] Average Time: 0.334
[INFO] [2021-04-08 11:56:37] Total Time: 8s
[INFO] [2021-04-08 11:56:37] last 3 / first 3: 0.83
[INFO] [2021-04-08 11:56:37] Std.Dev: 0.19493588689617927; Max: 1.17
[INFO] [2021-04-08 11:56:37] Storing 20705 Nodes
[INFO] [2021-04-08 11:56:37] Processing group of 20705 in 21 groups of 1000
[INFO] [2021-04-08 11:56:43] Average Time: 0.294
[INFO] [2021-04-08 11:56:43] Total Time: 7s
[INFO] [2021-04-08 11:56:43] last 3 / first 3: 0.4
[INFO] [2021-04-08 11:56:43] Std.Dev: 0.2073644135332772; Max: 1.18
[INFO] [2021-04-08 11:56:43] Storing 13352 Vernaculars
[INFO] [2021-04-08 11:56:43] Processing group of 13352 in 14 groups of 1000
[INFO] [2021-04-08 11:56:45] Average Time: 0.15
[INFO] [2021-04-08 11:56:45] Total Time: 3s
[INFO] [2021-04-08 11:56:45] last 3 / first 3: 1.0
[INFO] [2021-04-08 11:56:45] Std.Dev: 0.03162277660168379; Max: 0.23
[INFO] [2021-04-08 11:56:45] Storing 13351 Occurrences
[INFO] [2021-04-08 11:56:45] Processing group of 13351 in 14 groups of 1000
[INFO] [2021-04-08 11:56:47] Average Time: 0.109
[INFO] [2021-04-08 11:56:47] Total Time: 2s
[INFO] [2021-04-08 11:56:47] last 3 / first 3: 0.8
[INFO] [2021-04-08 11:56:47] Std.Dev: 0.05477225575051661; Max: 0.28
[INFO] [2021-04-08 11:56:47] Storing 13351 TraitsReferences
[INFO] [2021-04-08 11:56:47] Processing group of 13351 in 14 groups of 1000
[INFO] [2021-04-08 11:56:48] Average Time: 0.065
[INFO] [2021-04-08 11:56:48] Total Time: 1s
[INFO] [2021-04-08 11:56:48] last 3 / first 3: 0.95
[INFO] [2021-04-08 11:56:48] Std.Dev: 0.0; Max: 0.11
[INFO] [2021-04-08 11:56:48] Storing 13351 Traits
[INFO] [2021-04-08 11:56:48] Processing group of 13351 in 14 groups of 1000
[INFO] [2021-04-08 11:56:52] Average Time: 0.274
[INFO] [2021-04-08 11:56:52] Total Time: 4s
[INFO] [2021-04-08 11:56:52] last 3 / first 3: 0.81
[INFO] [2021-04-08 11:56:52] Std.Dev: 0.06324555320336758; Max: 0.42
[INFO] [2021-04-08 11:56:52] Storing 40053 MetaTraits
[INFO] [2021-04-08 11:56:52] Processing group of 40053 in 41 groups of 1000
[INFO] [2021-04-08 11:56:57] Average Time: 0.126
[INFO] [2021-04-08 11:56:57] Total Time: 6s
[INFO] [2021-04-08 11:56:57] last 3 / first 3: 0.7
[INFO] [2021-04-08 11:56:57] Std.Dev: 0.044721359549995794; Max: 0.31
[STOP] [2021-04-08 11:56:57] parse_diff_and_store
[START] [2021-04-08 11:56:57] resolve_keys
[INFO] [2021-04-08 11:57:08] Occurrences to nodes (through scientific_names)...
[INFO] [2021-04-08 11:57:08] traits to occurrences...
[INFO] [2021-04-08 11:57:09] traits to nodes (through occurrences)...
[INFO] [2021-04-08 11:57:09] Traits to sex term...
[INFO] [2021-04-08 11:57:09] Traits to lifestage term...
[INFO] [2021-04-08 11:57:09] MetaTraits to traits...
[INFO] [2021-04-08 11:57:10] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-04-08 11:57:11] Assocs to occurrences...
[INFO] [2021-04-08 11:57:11] Assocs to nodes...
[INFO] [2021-04-08 11:57:11] Assoc to sex term...
[INFO] [2021-04-08 11:57:11] Assoc to lifestage term...
[INFO] [2021-04-08 11:57:11] MetaAssoc to assocs...
[STOP] [2021-04-08 11:57:11] resolve_keys
[START] [2021-04-08 11:57:11] hold_for_later_1
[STOP] [2021-04-08 11:57:11] hold_for_later_1
[START] [2021-04-08 11:57:11] hold_for_later_2
[STOP] [2021-04-08 11:57:11] hold_for_later_2
[START] [2021-04-08 11:57:11] resolve_missing_parents
[STOP] [2021-04-08 11:57:12] resolve_missing_parents
[START] [2021-04-08 11:57:12] rebuild_nodes
[START] [2021-04-08 11:57:12] Flattener#flatten
[START] [2021-04-08 11:57:12] Flattener#study_resource
[START] [2021-04-08 11:57:12] Flattener#build_ancestry
[STOP] [2021-04-08 11:57:14] Flattener#build_ancestry
[INFO] [2021-04-08 11:57:14] 20705 ancestry keys
[START] [2021-04-08 11:57:14] build_node_ancestors
[INFO] [2021-04-08 11:57:14] old ancestors deleted.
[STOP] [2021-04-08 11:57:26] build_node_ancestors
[START] [2021-04-08 11:57:26] Flattener#propagate_ancestor_ids
[STOP] [2021-04-08 11:57:29] Flattener#propagate_ancestor_ids
[STOP] [2021-04-08 11:57:29] Flattener#flatten
[STOP] [2021-04-08 11:57:29] rebuild_nodes
[START] [2021-04-08 11:57:29] resolve_missing_media_owners
[STOP] [2021-04-08 11:57:29] resolve_missing_media_owners
[START] [2021-04-08 11:57:29] sanitize_media_verbatims
[STOP] [2021-04-08 11:57:29] sanitize_media_verbatims
[START] [2021-04-08 11:57:29] queue_downloads
[STOP] [2021-04-08 11:57:29] queue_downloads
[START] [2021-04-08 11:57:29] parse_names
[WARN] [2021-04-08 11:57:29] I see 20705 names which still need to be parsed.
[WARN] [2021-04-08 11:57:43] I see 1960 names which still need to be parsed.
[WARN] [2021-04-08 11:57:46] I see 184 names which still need to be parsed.
[WARN] [2021-04-08 11:57:47] I see 23 names which still need to be parsed.
[WARN] [2021-04-08 11:57:48] I see 21 names which still need to be parsed.
[WARN] [2021-04-08 11:57:49] I see 20 names which still need to be parsed.
[WARN] [2021-04-08 11:57:50] I see 19 names which still need to be parsed.
[WARN] [2021-04-08 11:57:51] I see 18 names which still need to be parsed.
[WARN] [2021-04-08 11:57:52] I see 17 names which still need to be parsed.
[WARN] [2021-04-08 11:57:54] I see 16 names which still need to be parsed.
[WARN] [2021-04-08 11:57:55] I see 15 names which still need to be parsed.
[STOP] [2021-04-08 11:57:56] parse_names
[START] [2021-04-08 11:57:56] denormalize_canonical_names_to_nodes
[STOP] [2021-04-08 11:57:56] denormalize_canonical_names_to_nodes
[START] [2021-04-08 11:57:56] match_nodes
[START] [2021-04-08 11:57:56] map_all_nodes_to_pages
[WARN] [2021-04-08 11:59:07] cannot match node with blank canonical: Node#92716994
[WARN] [2021-04-08 12:01:08] cannot match node with blank canonical: Node#92714278
[WARN] [2021-04-08 12:02:31] cannot match node with blank canonical: Node#92713118
[WARN] [2021-04-08 12:02:43] cannot match node with blank canonical: Node#92709879
[WARN] [2021-04-08 12:03:26] cannot match node with blank canonical: Node#92712270
[WARN] [2021-04-08 12:04:12] cannot match node with blank canonical: Node#92715486
[WARN] [2021-04-08 12:04:16] cannot match node with blank canonical: Node#92713817
[WARN] [2021-04-08 12:04:27] cannot match node with blank canonical: Node#92706740
[WARN] [2021-04-08 12:04:33] cannot match node with blank canonical: Node#92714952
[WARN] [2021-04-08 12:04:37] cannot match node with blank canonical: Node#92720427
[WARN] [2021-04-08 12:04:44] cannot match node with blank canonical: Node#92708366
[WARN] [2021-04-08 12:04:44] cannot match node with blank canonical: Node#92717913
[WARN] [2021-04-08 12:06:07] cannot match node with blank canonical: Node#92704393
[WARN] [2021-04-08 12:06:07] cannot match node with blank canonical: Node#92705467
[WARN] [2021-04-08 12:06:07] cannot match node with blank canonical: Node#92705468
[WARN] [2021-04-08 12:06:07] cannot match node with blank canonical: Node#92709632
[WARN] [2021-04-08 12:06:07] cannot match node with blank canonical: Node#92709633
[WARN] [2021-04-08 12:06:08] cannot match node with blank canonical: Node#92717914
[WARN] [2021-04-08 12:06:08] cannot match node with blank canonical: Node#92717915
[WARN] [2021-04-08 12:06:08] cannot match node with blank canonical: Node#92714472
[WARN] [2021-04-08 12:06:09] cannot match node with blank canonical: Node#92710810
[WARN] [2021-04-08 12:06:10] cannot match node with blank canonical: Node#92717895
[WARN] [2021-04-08 12:06:10] cannot match node with blank canonical: Node#92709631
[WARN] [2021-04-08 12:06:11] cannot match node with blank canonical: Node#92705658
[WARN] [2021-04-08 12:06:12] cannot match node with blank canonical: Node#92719505
[WARN] [2021-04-08 12:06:12] cannot match node with blank canonical: Node#92705466
[STOP] [2021-04-08 12:06:13] map_all_nodes_to_pages
[INFO] [2021-04-08 12:06:13] 1349 Unmatched nodes (of 20705)! That's too many to output. Full list in /app/public/data/simpson_et_al_si/unmatched_nodes.txt ; First 10: Aceria guerreronis (#92703162); Aceria litchii (#92703164); Aceria litchii (#92703165); Aceria malherbae (#92703166); Aculops fuchsiae (#92703381); Colomerus gardeniella (#92708237); Eriophyes (#92710698); Eriophyes cynodoniensis (#92710699); Eriophyes fici (#92710700); Eriophyes granati (#92710702)
[START] [2021-04-08 12:06:13] update_nodes
[STOP] [2021-04-08 12:06:21] update_nodes
[STOP] [2021-04-08 12:06:21] match_nodes
[START] [2021-04-08 12:06:21] reindex_search
[STOP] [2021-04-08 12:06:39] reindex_search
[START] [2021-04-08 12:06:39] normalize_units
[STOP] [2021-04-08 12:06:39] normalize_units
[START] [2021-04-08 12:06:39] calculate_statistics
[STOP] [2021-04-08 12:06:40] calculate_statistics
[START] [2021-04-08 12:06:40] complete_harvest_instance
[START] [2021-04-08 12:06:40] overall_tsv_creation
[INFO] [2021-04-08 12:06:40] Processing group of 20705 in 3 batches of 10000
[INFO] [2021-04-08 12:07:47] 6235 Traits (unfiltered)...
[INFO] [2021-04-08 12:08:58] 6235 Traits (filtered)...
[INFO] [2021-04-08 12:08:58] 0 Associations (filtered)...
[INFO] [2021-04-08 12:09:00] 12470 metadata added.
[INFO] [2021-04-08 12:09:00] 0 metadata added.
[INFO] [2021-04-08 12:10:36] 6637 Traits (unfiltered)...
[INFO] [2021-04-08 12:11:48] 6637 Traits (filtered)...
[INFO] [2021-04-08 12:11:48] 0 Associations (filtered)...
[INFO] [2021-04-08 12:11:49] 13274 metadata added.
[INFO] [2021-04-08 12:11:49] 0 metadata added.
[INFO] [2021-04-08 12:12:57] 479 Traits (unfiltered)...
[INFO] [2021-04-08 12:13:31] 479 Traits (filtered)...
[INFO] [2021-04-08 12:13:31] 0 Associations (filtered)...
[INFO] [2021-04-08 12:13:31] 958 metadata added.
[INFO] [2021-04-08 12:13:31] 0 metadata added.
[INFO] [2021-04-08 12:13:58] Average Time: 117.66
[INFO] [2021-04-08 12:13:58] Total Time: 7m18s
[STOP] [2021-04-08 12:13:58] overall_tsv_creation
[INFO] [2021-04-08 12:13:58] Done. Check your files:
[INFO] [2021-04-08 12:13:58] (18744 lines) /app/public/data/simpson_et_al_si/publish_nodes.tsv
[INFO] [2021-04-08 12:13:59] (95161 lines) /app/public/data/simpson_et_al_si/publish_node_ancestors.tsv
[INFO] [2021-04-08 12:13:59] (20705 lines) /app/public/data/simpson_et_al_si/publish_scientific_names.tsv
[INFO] [2021-04-08 12:14:00] (13351 lines) /app/public/data/simpson_et_al_si/publish_vernaculars.tsv
[INFO] [2021-04-08 12:14:00] (13352 lines) /app/public/data/simpson_et_al_si/publish_traits.tsv
[INFO] [2021-04-08 12:14:01] (26703 lines) /app/public/data/simpson_et_al_si/publish_metadata.tsv
[STOP] [2021-04-08 12:14:01] complete_harvest_instance
[START] [2021-04-08 12:14:01] completed
[STOP] [2021-04-08 12:14:01] completed
[STOP] [2021-04-08 12:14:01] logged process, took 1090.7

Latest Process