Harvest for Christensen et al, 2015 Created 31 May 19:30

Stage: completed
Fetched: 31 May 19:30
Validated: 31 May 19:30
Deltas Created 31 May 19:30
Units Normalized: 31 May 19:30
Ancestry Built: 31 May 19:30
Nodes Matched: 31 May 19:30
Names Parsed: 31 May 19:30
New Models Stored: 31 May 19:30
Indexed: 31 May 19:30
Completed: 31 May 19:32
Time to Harvest: less than a minute
Harvesting Log

(160 lines)
[INFO] [2021-05-31 19:30:41] Created harvest instance #3952
[STOP] [2021-05-31 19:30:41] create_harvest_instance
[START] [2021-05-31 19:30:41] fetch_files
[STOP] [2021-05-31 19:30:41] fetch_files
[START] [2021-05-31 19:30:41] validate_each_file
[INFO] [2021-05-31 19:30:41] Looping over 4 formats...
[INFO] [2021-05-31 19:30:41] ...refs (/app/public/data/bech_et_al_bech_/references.txt)
[INFO] [2021-05-31 19:30:41] Valid: /app/public/converted_csv/bech_et_al_bech__refs_3952.csv (0 lines)
[INFO] [2021-05-31 19:30:41] ...nodes (/app/public/data/bech_et_al_bech_/taxa.txt)
[INFO] [2021-05-31 19:30:41] Valid: /app/public/converted_csv/bech_et_al_bech__nodes_3952.csv (1 lines)
[INFO] [2021-05-31 19:30:41] ...occurrences (/app/public/data/bech_et_al_bech_/occurrences.txt)
[INFO] [2021-05-31 19:30:41] Valid: /app/public/converted_csv/bech_et_al_bech__occurrences_3952.csv (1 lines)
[INFO] [2021-05-31 19:30:41] ...measurements (/app/public/data/bech_et_al_bech_/measurementsorfacts.txt)
[INFO] [2021-05-31 19:30:41] Valid: /app/public/converted_csv/bech_et_al_bech__measurements_3952.csv (21 lines)
[STOP] [2021-05-31 19:30:41] validate_each_file
[START] [2021-05-31 19:30:41] convert_to_csv
[INFO] [2021-05-31 19:30:41] Looping over 4 formats...
[INFO] [2021-05-31 19:30:41] ...refs (/app/public/data/bech_et_al_bech_/references.txt)
[CMD] [2021-05-31 19:30:41] /usr/bin/sort /app/public/converted_csv/bech_et_al_bech__refs_3952.csv > /app/public/converted_csv/bech_et_al_bech__refs_3952.csv_sorted
[INFO] [2021-05-31 19:30:42] Converted: /app/public/converted_csv/bech_et_al_bech__refs_3952.csv (0 lines)
[INFO] [2021-05-31 19:30:42] ...nodes (/app/public/data/bech_et_al_bech_/taxa.txt)
[CMD] [2021-05-31 19:30:42] /usr/bin/sort /app/public/converted_csv/bech_et_al_bech__nodes_3952.csv > /app/public/converted_csv/bech_et_al_bech__nodes_3952.csv_sorted
[INFO] [2021-05-31 19:30:42] Converted: /app/public/converted_csv/bech_et_al_bech__nodes_3952.csv (1 lines)
[INFO] [2021-05-31 19:30:42] ...occurrences (/app/public/data/bech_et_al_bech_/occurrences.txt)
[CMD] [2021-05-31 19:30:42] /usr/bin/sort /app/public/converted_csv/bech_et_al_bech__occurrences_3952.csv > /app/public/converted_csv/bech_et_al_bech__occurrences_3952.csv_sorted
[INFO] [2021-05-31 19:30:43] Converted: /app/public/converted_csv/bech_et_al_bech__occurrences_3952.csv (1 lines)
[INFO] [2021-05-31 19:30:43] ...measurements (/app/public/data/bech_et_al_bech_/measurementsorfacts.txt)
[CMD] [2021-05-31 19:30:43] /usr/bin/sort /app/public/converted_csv/bech_et_al_bech__measurements_3952.csv > /app/public/converted_csv/bech_et_al_bech__measurements_3952.csv_sorted
[INFO] [2021-05-31 19:30:43] Converted: /app/public/converted_csv/bech_et_al_bech__measurements_3952.csv (21 lines)
[STOP] [2021-05-31 19:30:43] convert_to_csv
[START] [2021-05-31 19:30:43] calculate_delta
[INFO] [2021-05-31 19:30:43] Looping over 4 formats...
[INFO] [2021-05-31 19:30:43] ...refs (/app/public/data/bech_et_al_bech_/references.txt)
[CMD] [2021-05-31 19:30:43] echo "0a" > /app/public/diff/bech_et_al_bech__refs_3952.diff
[CMD] [2021-05-31 19:30:43] tail -n +1 /app/public/converted_csv/bech_et_al_bech__refs_3952.csv >> /app/public/diff/bech_et_al_bech__refs_3952.diff
[CMD] [2021-05-31 19:30:44] echo "." >> /app/public/diff/bech_et_al_bech__refs_3952.diff
[INFO] [2021-05-31 19:30:44] Created diff: /app/public/diff/bech_et_al_bech__refs_3952.diff (2 lines)
[INFO] [2021-05-31 19:30:44] ...nodes (/app/public/data/bech_et_al_bech_/taxa.txt)
[CMD] [2021-05-31 19:30:44] echo "0a" > /app/public/diff/bech_et_al_bech__nodes_3952.diff
[CMD] [2021-05-31 19:30:45] tail -n +1 /app/public/converted_csv/bech_et_al_bech__nodes_3952.csv >> /app/public/diff/bech_et_al_bech__nodes_3952.diff
[CMD] [2021-05-31 19:30:45] echo "." >> /app/public/diff/bech_et_al_bech__nodes_3952.diff
[INFO] [2021-05-31 19:30:45] Created diff: /app/public/diff/bech_et_al_bech__nodes_3952.diff (3 lines)
[INFO] [2021-05-31 19:30:45] ...occurrences (/app/public/data/bech_et_al_bech_/occurrences.txt)
[CMD] [2021-05-31 19:30:45] echo "0a" > /app/public/diff/bech_et_al_bech__occurrences_3952.diff
[CMD] [2021-05-31 19:30:46] tail -n +1 /app/public/converted_csv/bech_et_al_bech__occurrences_3952.csv >> /app/public/diff/bech_et_al_bech__occurrences_3952.diff
[CMD] [2021-05-31 19:30:46] echo "." >> /app/public/diff/bech_et_al_bech__occurrences_3952.diff
[INFO] [2021-05-31 19:30:46] Created diff: /app/public/diff/bech_et_al_bech__occurrences_3952.diff (3 lines)
[INFO] [2021-05-31 19:30:46] ...measurements (/app/public/data/bech_et_al_bech_/measurementsorfacts.txt)
[CMD] [2021-05-31 19:30:46] echo "0a" > /app/public/diff/bech_et_al_bech__measurements_3952.diff
[CMD] [2021-05-31 19:30:47] tail -n +1 /app/public/converted_csv/bech_et_al_bech__measurements_3952.csv >> /app/public/diff/bech_et_al_bech__measurements_3952.diff
[CMD] [2021-05-31 19:30:47] echo "." >> /app/public/diff/bech_et_al_bech__measurements_3952.diff
[INFO] [2021-05-31 19:30:48] Created diff: /app/public/diff/bech_et_al_bech__measurements_3952.diff (23 lines)
[STOP] [2021-05-31 19:30:48] calculate_delta
[START] [2021-05-31 19:30:48] parse_diff_and_store
[INFO] [2021-05-31 19:30:48] Handling diff: /app/public/diff/bech_et_al_bech__refs_3952.diff (2 lines)
[INFO] [2021-05-31 19:30:48] Loading refs diff file into memory (2 /app/public/diff/bech_et_al_bech__refs_3952.diff lines)...
[INFO] [2021-05-31 19:30:49] Handling diff: /app/public/diff/bech_et_al_bech__nodes_3952.diff (3 lines)
[INFO] [2021-05-31 19:30:49] Loading nodes diff file into memory (3 /app/public/diff/bech_et_al_bech__nodes_3952.diff lines)...
[INFO] [2021-05-31 19:30:49] Handling diff: /app/public/diff/bech_et_al_bech__occurrences_3952.diff (3 lines)
[INFO] [2021-05-31 19:30:50] Loading occurrences diff file into memory (3 /app/public/diff/bech_et_al_bech__occurrences_3952.diff lines)...
[INFO] [2021-05-31 19:30:50] Handling diff: /app/public/diff/bech_et_al_bech__measurements_3952.diff (23 lines)
[INFO] [2021-05-31 19:30:50] Loading measurements diff file into memory (23 /app/public/diff/bech_et_al_bech__measurements_3952.diff lines)...
[INFO] [2021-05-31 19:30:51] Storing 1 ScientificNames
[INFO] [2021-05-31 19:30:51] Processing group of 1 in 1 groups of 1000
[INFO] [2021-05-31 19:30:51] Average Time: 0.0
[INFO] [2021-05-31 19:30:51] Total Time: 1s
[INFO] [2021-05-31 19:30:51] Storing 1 Nodes
[INFO] [2021-05-31 19:30:51] Processing group of 1 in 1 groups of 1000
[INFO] [2021-05-31 19:30:51] Average Time: 0.0
[INFO] [2021-05-31 19:30:51] Total Time: 1s
[INFO] [2021-05-31 19:30:51] Storing 1 Occurrences
[INFO] [2021-05-31 19:30:51] Processing group of 1 in 1 groups of 1000
[INFO] [2021-05-31 19:30:51] Average Time: 0.0
[INFO] [2021-05-31 19:30:51] Total Time: 1s
[INFO] [2021-05-31 19:30:51] Storing 21 Traits
[INFO] [2021-05-31 19:30:51] Processing group of 21 in 1 groups of 1000
[INFO] [2021-05-31 19:30:51] Average Time: 0.01
[INFO] [2021-05-31 19:30:51] Total Time: 1s
[INFO] [2021-05-31 19:30:51] Storing 8 MetaTraits
[INFO] [2021-05-31 19:30:51] Processing group of 8 in 1 groups of 1000
[INFO] [2021-05-31 19:30:51] Average Time: 0.0
[INFO] [2021-05-31 19:30:51] Total Time: 1s
[STOP] [2021-05-31 19:30:51] parse_diff_and_store
[START] [2021-05-31 19:30:51] resolve_keys
[INFO] [2021-05-31 19:30:57] Occurrences to nodes (through scientific_names)...
[INFO] [2021-05-31 19:30:57] traits to occurrences...
[INFO] [2021-05-31 19:30:57] traits to nodes (through occurrences)...
[INFO] [2021-05-31 19:30:57] Traits to sex term...
[INFO] [2021-05-31 19:30:57] Traits to lifestage term...
[INFO] [2021-05-31 19:30:57] MetaTraits to traits...
[INFO] [2021-05-31 19:30:57] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-05-31 19:30:57] Assocs to occurrences...
[INFO] [2021-05-31 19:30:57] Assocs to nodes...
[INFO] [2021-05-31 19:30:57] Assoc to sex term...
[INFO] [2021-05-31 19:30:57] Assoc to lifestage term...
[INFO] [2021-05-31 19:30:57] MetaAssoc to assocs...
[STOP] [2021-05-31 19:30:57] resolve_keys
[START] [2021-05-31 19:30:57] hold_for_later_1
[STOP] [2021-05-31 19:30:57] hold_for_later_1
[START] [2021-05-31 19:30:57] hold_for_later_2
[STOP] [2021-05-31 19:30:57] hold_for_later_2
[START] [2021-05-31 19:30:57] resolve_missing_parents
[STOP] [2021-05-31 19:30:57] resolve_missing_parents
[START] [2021-05-31 19:30:57] rebuild_nodes
[START] [2021-05-31 19:30:57] Flattener#flatten
[START] [2021-05-31 19:30:57] Flattener#study_resource
[START] [2021-05-31 19:30:57] Flattener#build_ancestry
[STOP] [2021-05-31 19:30:57] Flattener#build_ancestry
[INFO] [2021-05-31 19:30:57] 1 ancestry keys
[START] [2021-05-31 19:30:57] build_node_ancestors
[INFO] [2021-05-31 19:30:57] old ancestors deleted.
[STOP] [2021-05-31 19:30:57] build_node_ancestors
[WARN] [2021-05-31 19:30:57] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2021-05-31 19:30:57] Flattener#flatten
[STOP] [2021-05-31 19:30:57] rebuild_nodes
[START] [2021-05-31 19:30:57] resolve_missing_media_owners
[STOP] [2021-05-31 19:30:57] resolve_missing_media_owners
[START] [2021-05-31 19:30:57] sanitize_media_verbatims
[STOP] [2021-05-31 19:30:57] sanitize_media_verbatims
[START] [2021-05-31 19:30:57] queue_downloads
[STOP] [2021-05-31 19:30:57] queue_downloads
[START] [2021-05-31 19:30:57] parse_names
[WARN] [2021-05-31 19:30:57] I see 1 names which still need to be parsed.
[STOP] [2021-05-31 19:30:58] parse_names
[START] [2021-05-31 19:30:58] denormalize_canonical_names_to_nodes
[STOP] [2021-05-31 19:30:58] denormalize_canonical_names_to_nodes
[START] [2021-05-31 19:30:58] match_nodes
[START] [2021-05-31 19:30:58] map_all_nodes_to_pages
[STOP] [2021-05-31 19:30:58] map_all_nodes_to_pages
[INFO] [2021-05-31 19:30:58] ZERO unmatched nodes (of 1)! Nicely done.
[START] [2021-05-31 19:30:58] update_nodes
[STOP] [2021-05-31 19:30:58] update_nodes
[STOP] [2021-05-31 19:30:58] match_nodes
[START] [2021-05-31 19:30:58] reindex_search
[STOP] [2021-05-31 19:30:58] reindex_search
[START] [2021-05-31 19:30:58] normalize_units
[STOP] [2021-05-31 19:30:58] normalize_units
[START] [2021-05-31 19:30:58] calculate_statistics
[2021-05-31 19:30:58] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2021-05-31 19:30:58] calculate_statistics
[START] [2021-05-31 19:30:58] complete_harvest_instance
[START] [2021-05-31 19:30:58] overall_tsv_creation
[INFO] [2021-05-31 19:30:58] Processing group of 1 in 1 batches of 10000
[INFO] [2021-05-31 19:31:30] 7 Traits (unfiltered)...
[INFO] [2021-05-31 19:31:54] 7 Traits (filtered)...
[INFO] [2021-05-31 19:31:54] 0 Associations (filtered)...
[INFO] [2021-05-31 19:31:54] 14 metadata added.
[INFO] [2021-05-31 19:31:54] 0 metadata added.
[INFO] [2021-05-31 19:32:17] Average Time: 57.27
[INFO] [2021-05-31 19:32:17] Total Time: 1m19s
[STOP] [2021-05-31 19:32:17] overall_tsv_creation
[INFO] [2021-05-31 19:32:17] Done. Check your files:
[INFO] [2021-05-31 19:32:17] (1 lines) /app/public/data/bech_et_al_bech_/publish_nodes.tsv
[INFO] [2021-05-31 19:32:18] (1 lines) /app/public/data/bech_et_al_bech_/publish_scientific_names.tsv
[INFO] [2021-05-31 19:32:18] (8 lines) /app/public/data/bech_et_al_bech_/publish_traits.tsv
[INFO] [2021-05-31 19:32:18] (15 lines) /app/public/data/bech_et_al_bech_/publish_metadata.tsv
[STOP] [2021-05-31 19:32:18] complete_harvest_instance
[START] [2021-05-31 19:32:18] completed
[STOP] [2021-05-31 19:32:18] completed
[STOP] [2021-05-31 19:32:18] logged process, took 97.53
Latest Process

31 May 19:30 EXITED currently processing 0