Harvest for Christensen, Christensen-Dalsgaard and Madsen 2015 Created 31 May 19:27

Stage: completed
Fetched: 31 May 19:27
Validated: 31 May 19:27
Deltas Created 31 May 19:27
Units Normalized: 31 May 19:27
Ancestry Built: 31 May 19:27
Nodes Matched: 31 May 19:27
Names Parsed: 31 May 19:27
New Models Stored: 31 May 19:27
Indexed: 31 May 19:27
Completed: 31 May 19:29
Time to Harvest: less than a minute
Harvesting Log

(160 lines)
[INFO] [2021-05-31 19:27:25] Created harvest instance #3950
[STOP] [2021-05-31 19:27:25] create_harvest_instance
[START] [2021-05-31 19:27:25] fetch_files
[STOP] [2021-05-31 19:27:25] fetch_files
[START] [2021-05-31 19:27:25] validate_each_file
[INFO] [2021-05-31 19:27:25] Looping over 4 formats...
[INFO] [2021-05-31 19:27:25] ...refs (/app/public/data/christensen_et_a/references.txt)
[INFO] [2021-05-31 19:27:25] Valid: /app/public/converted_csv/christensen_et_a_refs_3950.csv (0 lines)
[INFO] [2021-05-31 19:27:25] ...nodes (/app/public/data/christensen_et_a/taxa.txt)
[INFO] [2021-05-31 19:27:25] Valid: /app/public/converted_csv/christensen_et_a_nodes_3950.csv (1 lines)
[INFO] [2021-05-31 19:27:25] ...occurrences (/app/public/data/christensen_et_a/occurrences.txt)
[INFO] [2021-05-31 19:27:25] Valid: /app/public/converted_csv/christensen_et_a_occurrences_3950.csv (1 lines)
[INFO] [2021-05-31 19:27:25] ...measurements (/app/public/data/christensen_et_a/measurementsorfacts.txt)
[INFO] [2021-05-31 19:27:25] Valid: /app/public/converted_csv/christensen_et_a_measurements_3950.csv (17 lines)
[STOP] [2021-05-31 19:27:25] validate_each_file
[START] [2021-05-31 19:27:25] convert_to_csv
[INFO] [2021-05-31 19:27:25] Looping over 4 formats...
[INFO] [2021-05-31 19:27:25] ...refs (/app/public/data/christensen_et_a/references.txt)
[CMD] [2021-05-31 19:27:25] /usr/bin/sort /app/public/converted_csv/christensen_et_a_refs_3950.csv > /app/public/converted_csv/christensen_et_a_refs_3950.csv_sorted
[INFO] [2021-05-31 19:27:25] Converted: /app/public/converted_csv/christensen_et_a_refs_3950.csv (0 lines)
[INFO] [2021-05-31 19:27:25] ...nodes (/app/public/data/christensen_et_a/taxa.txt)
[CMD] [2021-05-31 19:27:25] /usr/bin/sort /app/public/converted_csv/christensen_et_a_nodes_3950.csv > /app/public/converted_csv/christensen_et_a_nodes_3950.csv_sorted
[INFO] [2021-05-31 19:27:25] Converted: /app/public/converted_csv/christensen_et_a_nodes_3950.csv (1 lines)
[INFO] [2021-05-31 19:27:25] ...occurrences (/app/public/data/christensen_et_a/occurrences.txt)
[CMD] [2021-05-31 19:27:25] /usr/bin/sort /app/public/converted_csv/christensen_et_a_occurrences_3950.csv > /app/public/converted_csv/christensen_et_a_occurrences_3950.csv_sorted
[INFO] [2021-05-31 19:27:26] Converted: /app/public/converted_csv/christensen_et_a_occurrences_3950.csv (1 lines)
[INFO] [2021-05-31 19:27:26] ...measurements (/app/public/data/christensen_et_a/measurementsorfacts.txt)
[CMD] [2021-05-31 19:27:26] /usr/bin/sort /app/public/converted_csv/christensen_et_a_measurements_3950.csv > /app/public/converted_csv/christensen_et_a_measurements_3950.csv_sorted
[INFO] [2021-05-31 19:27:26] Converted: /app/public/converted_csv/christensen_et_a_measurements_3950.csv (17 lines)
[STOP] [2021-05-31 19:27:26] convert_to_csv
[START] [2021-05-31 19:27:26] calculate_delta
[INFO] [2021-05-31 19:27:26] Looping over 4 formats...
[INFO] [2021-05-31 19:27:26] ...refs (/app/public/data/christensen_et_a/references.txt)
[CMD] [2021-05-31 19:27:26] echo "0a" > /app/public/diff/christensen_et_a_refs_3950.diff
[CMD] [2021-05-31 19:27:26] tail -n +1 /app/public/converted_csv/christensen_et_a_refs_3950.csv >> /app/public/diff/christensen_et_a_refs_3950.diff
[CMD] [2021-05-31 19:27:27] echo "." >> /app/public/diff/christensen_et_a_refs_3950.diff
[INFO] [2021-05-31 19:27:27] Created diff: /app/public/diff/christensen_et_a_refs_3950.diff (2 lines)
[INFO] [2021-05-31 19:27:27] ...nodes (/app/public/data/christensen_et_a/taxa.txt)
[CMD] [2021-05-31 19:27:27] echo "0a" > /app/public/diff/christensen_et_a_nodes_3950.diff
[CMD] [2021-05-31 19:27:27] tail -n +1 /app/public/converted_csv/christensen_et_a_nodes_3950.csv >> /app/public/diff/christensen_et_a_nodes_3950.diff
[CMD] [2021-05-31 19:27:28] echo "." >> /app/public/diff/christensen_et_a_nodes_3950.diff
[INFO] [2021-05-31 19:27:28] Created diff: /app/public/diff/christensen_et_a_nodes_3950.diff (3 lines)
[INFO] [2021-05-31 19:27:28] ...occurrences (/app/public/data/christensen_et_a/occurrences.txt)
[CMD] [2021-05-31 19:27:28] echo "0a" > /app/public/diff/christensen_et_a_occurrences_3950.diff
[CMD] [2021-05-31 19:27:29] tail -n +1 /app/public/converted_csv/christensen_et_a_occurrences_3950.csv >> /app/public/diff/christensen_et_a_occurrences_3950.diff
[CMD] [2021-05-31 19:27:29] echo "." >> /app/public/diff/christensen_et_a_occurrences_3950.diff
[INFO] [2021-05-31 19:27:30] Created diff: /app/public/diff/christensen_et_a_occurrences_3950.diff (3 lines)
[INFO] [2021-05-31 19:27:30] ...measurements (/app/public/data/christensen_et_a/measurementsorfacts.txt)
[CMD] [2021-05-31 19:27:30] echo "0a" > /app/public/diff/christensen_et_a_measurements_3950.diff
[CMD] [2021-05-31 19:27:30] tail -n +1 /app/public/converted_csv/christensen_et_a_measurements_3950.csv >> /app/public/diff/christensen_et_a_measurements_3950.diff
[CMD] [2021-05-31 19:27:30] echo "." >> /app/public/diff/christensen_et_a_measurements_3950.diff
[INFO] [2021-05-31 19:27:31] Created diff: /app/public/diff/christensen_et_a_measurements_3950.diff (19 lines)
[STOP] [2021-05-31 19:27:31] calculate_delta
[START] [2021-05-31 19:27:31] parse_diff_and_store
[INFO] [2021-05-31 19:27:31] Handling diff: /app/public/diff/christensen_et_a_refs_3950.diff (2 lines)
[INFO] [2021-05-31 19:27:31] Loading refs diff file into memory (2 /app/public/diff/christensen_et_a_refs_3950.diff lines)...
[INFO] [2021-05-31 19:27:31] Handling diff: /app/public/diff/christensen_et_a_nodes_3950.diff (3 lines)
[INFO] [2021-05-31 19:27:32] Loading nodes diff file into memory (3 /app/public/diff/christensen_et_a_nodes_3950.diff lines)...
[INFO] [2021-05-31 19:27:32] Handling diff: /app/public/diff/christensen_et_a_occurrences_3950.diff (3 lines)
[INFO] [2021-05-31 19:27:32] Loading occurrences diff file into memory (3 /app/public/diff/christensen_et_a_occurrences_3950.diff lines)...
[INFO] [2021-05-31 19:27:33] Handling diff: /app/public/diff/christensen_et_a_measurements_3950.diff (19 lines)
[INFO] [2021-05-31 19:27:33] Loading measurements diff file into memory (19 /app/public/diff/christensen_et_a_measurements_3950.diff lines)...
[INFO] [2021-05-31 19:27:34] Storing 1 ScientificNames
[INFO] [2021-05-31 19:27:34] Processing group of 1 in 1 groups of 1000
[INFO] [2021-05-31 19:27:34] Average Time: 0.0
[INFO] [2021-05-31 19:27:34] Total Time: 1s
[INFO] [2021-05-31 19:27:34] Storing 1 Nodes
[INFO] [2021-05-31 19:27:34] Processing group of 1 in 1 groups of 1000
[INFO] [2021-05-31 19:27:34] Average Time: 0.0
[INFO] [2021-05-31 19:27:34] Total Time: 1s
[INFO] [2021-05-31 19:27:34] Storing 1 Occurrences
[INFO] [2021-05-31 19:27:34] Processing group of 1 in 1 groups of 1000
[INFO] [2021-05-31 19:27:34] Average Time: 0.0
[INFO] [2021-05-31 19:27:34] Total Time: 1s
[INFO] [2021-05-31 19:27:34] Storing 17 Traits
[INFO] [2021-05-31 19:27:34] Processing group of 17 in 1 groups of 1000
[INFO] [2021-05-31 19:27:34] Average Time: 0.01
[INFO] [2021-05-31 19:27:34] Total Time: 1s
[INFO] [2021-05-31 19:27:34] Storing 13 MetaTraits
[INFO] [2021-05-31 19:27:34] Processing group of 13 in 1 groups of 1000
[INFO] [2021-05-31 19:27:34] Average Time: 0.0
[INFO] [2021-05-31 19:27:34] Total Time: 1s
[STOP] [2021-05-31 19:27:34] parse_diff_and_store
[START] [2021-05-31 19:27:34] resolve_keys
[INFO] [2021-05-31 19:27:39] Occurrences to nodes (through scientific_names)...
[INFO] [2021-05-31 19:27:39] traits to occurrences...
[INFO] [2021-05-31 19:27:39] traits to nodes (through occurrences)...
[INFO] [2021-05-31 19:27:39] Traits to sex term...
[INFO] [2021-05-31 19:27:39] Traits to lifestage term...
[INFO] [2021-05-31 19:27:39] MetaTraits to traits...
[INFO] [2021-05-31 19:27:39] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-05-31 19:27:39] Assocs to occurrences...
[INFO] [2021-05-31 19:27:39] Assocs to nodes...
[INFO] [2021-05-31 19:27:39] Assoc to sex term...
[INFO] [2021-05-31 19:27:39] Assoc to lifestage term...
[INFO] [2021-05-31 19:27:39] MetaAssoc to assocs...
[STOP] [2021-05-31 19:27:39] resolve_keys
[START] [2021-05-31 19:27:39] hold_for_later_1
[STOP] [2021-05-31 19:27:39] hold_for_later_1
[START] [2021-05-31 19:27:39] hold_for_later_2
[STOP] [2021-05-31 19:27:39] hold_for_later_2
[START] [2021-05-31 19:27:39] resolve_missing_parents
[STOP] [2021-05-31 19:27:39] resolve_missing_parents
[START] [2021-05-31 19:27:39] rebuild_nodes
[START] [2021-05-31 19:27:39] Flattener#flatten
[START] [2021-05-31 19:27:39] Flattener#study_resource
[START] [2021-05-31 19:27:39] Flattener#build_ancestry
[STOP] [2021-05-31 19:27:39] Flattener#build_ancestry
[INFO] [2021-05-31 19:27:39] 1 ancestry keys
[START] [2021-05-31 19:27:39] build_node_ancestors
[INFO] [2021-05-31 19:27:39] old ancestors deleted.
[STOP] [2021-05-31 19:27:40] build_node_ancestors
[WARN] [2021-05-31 19:27:40] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2021-05-31 19:27:40] Flattener#flatten
[STOP] [2021-05-31 19:27:40] rebuild_nodes
[START] [2021-05-31 19:27:40] resolve_missing_media_owners
[STOP] [2021-05-31 19:27:40] resolve_missing_media_owners
[START] [2021-05-31 19:27:40] sanitize_media_verbatims
[STOP] [2021-05-31 19:27:40] sanitize_media_verbatims
[START] [2021-05-31 19:27:40] queue_downloads
[STOP] [2021-05-31 19:27:40] queue_downloads
[START] [2021-05-31 19:27:40] parse_names
[WARN] [2021-05-31 19:27:40] I see 1 names which still need to be parsed.
[STOP] [2021-05-31 19:27:41] parse_names
[START] [2021-05-31 19:27:41] denormalize_canonical_names_to_nodes
[STOP] [2021-05-31 19:27:41] denormalize_canonical_names_to_nodes
[START] [2021-05-31 19:27:41] match_nodes
[START] [2021-05-31 19:27:41] map_all_nodes_to_pages
[STOP] [2021-05-31 19:27:41] map_all_nodes_to_pages
[INFO] [2021-05-31 19:27:41] ZERO unmatched nodes (of 1)! Nicely done.
[START] [2021-05-31 19:27:41] update_nodes
[STOP] [2021-05-31 19:27:41] update_nodes
[STOP] [2021-05-31 19:27:41] match_nodes
[START] [2021-05-31 19:27:41] reindex_search
[STOP] [2021-05-31 19:27:41] reindex_search
[START] [2021-05-31 19:27:41] normalize_units
[STOP] [2021-05-31 19:27:41] normalize_units
[START] [2021-05-31 19:27:41] calculate_statistics
[2021-05-31 19:27:41] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2021-05-31 19:27:41] calculate_statistics
[START] [2021-05-31 19:27:41] complete_harvest_instance
[START] [2021-05-31 19:27:41] overall_tsv_creation
[INFO] [2021-05-31 19:27:41] Processing group of 1 in 1 batches of 10000
[INFO] [2021-05-31 19:28:12] 10 Traits (unfiltered)...
[INFO] [2021-05-31 19:28:37] 10 Traits (filtered)...
[INFO] [2021-05-31 19:28:37] 0 Associations (filtered)...
[INFO] [2021-05-31 19:28:37] 7 metadata added.
[INFO] [2021-05-31 19:28:37] 0 metadata added.
[INFO] [2021-05-31 19:28:59] Average Time: 56.71
[INFO] [2021-05-31 19:28:59] Total Time: 1m19s
[STOP] [2021-05-31 19:28:59] overall_tsv_creation
[INFO] [2021-05-31 19:28:59] Done. Check your files:
[INFO] [2021-05-31 19:28:59] (1 lines) /app/public/data/christensen_et_a/publish_nodes.tsv
[INFO] [2021-05-31 19:29:00] (1 lines) /app/public/data/christensen_et_a/publish_scientific_names.tsv
[INFO] [2021-05-31 19:29:00] (11 lines) /app/public/data/christensen_et_a/publish_traits.tsv
[INFO] [2021-05-31 19:29:00] (8 lines) /app/public/data/christensen_et_a/publish_metadata.tsv
[STOP] [2021-05-31 19:29:00] complete_harvest_instance
[START] [2021-05-31 19:29:00] completed
[STOP] [2021-05-31 19:29:00] completed
[STOP] [2021-05-31 19:29:00] logged process, took 96.33
Latest Process

31 May 19:27 EXITED currently processing 0