Harvest for Boistel et al 2013 Created 31 May 19:40

Harvesting Log

[INFO] [2021-05-31 19:40:34] Created harvest instance #3957
[STOP] [2021-05-31 19:40:34] create_harvest_instance
[START] [2021-05-31 19:40:34] fetch_files
[STOP] [2021-05-31 19:40:34] fetch_files
[START] [2021-05-31 19:40:34] validate_each_file
[INFO] [2021-05-31 19:40:34] Looping over 4 formats...
[INFO] [2021-05-31 19:40:34] ...refs (/app/public/data/boistel_et_al_bo/references.txt)
[INFO] [2021-05-31 19:40:34] Valid: /app/public/converted_csv/boistel_et_al_bo_refs_3957.csv (0 lines)
[INFO] [2021-05-31 19:40:34] ...nodes (/app/public/data/boistel_et_al_bo/taxa.txt)
[INFO] [2021-05-31 19:40:34] Valid: /app/public/converted_csv/boistel_et_al_bo_nodes_3957.csv (1 lines)
[INFO] [2021-05-31 19:40:34] ...occurrences (/app/public/data/boistel_et_al_bo/occurrences.txt)
[INFO] [2021-05-31 19:40:34] Valid: /app/public/converted_csv/boistel_et_al_bo_occurrences_3957.csv (1 lines)
[INFO] [2021-05-31 19:40:34] ...measurements (/app/public/data/boistel_et_al_bo/measurementsorfacts.txt)
[INFO] [2021-05-31 19:40:34] Valid: /app/public/converted_csv/boistel_et_al_bo_measurements_3957.csv (3 lines)
[STOP] [2021-05-31 19:40:34] validate_each_file
[START] [2021-05-31 19:40:34] convert_to_csv
[INFO] [2021-05-31 19:40:34] Looping over 4 formats...
[INFO] [2021-05-31 19:40:34] ...refs (/app/public/data/boistel_et_al_bo/references.txt)
[CMD] [2021-05-31 19:40:34] /usr/bin/sort /app/public/converted_csv/boistel_et_al_bo_refs_3957.csv > /app/public/converted_csv/boistel_et_al_bo_refs_3957.csv_sorted
[INFO] [2021-05-31 19:40:34] Converted: /app/public/converted_csv/boistel_et_al_bo_refs_3957.csv (0 lines)
[INFO] [2021-05-31 19:40:34] ...nodes (/app/public/data/boistel_et_al_bo/taxa.txt)
[CMD] [2021-05-31 19:40:34] /usr/bin/sort /app/public/converted_csv/boistel_et_al_bo_nodes_3957.csv > /app/public/converted_csv/boistel_et_al_bo_nodes_3957.csv_sorted
[INFO] [2021-05-31 19:40:35] Converted: /app/public/converted_csv/boistel_et_al_bo_nodes_3957.csv (1 lines)
[INFO] [2021-05-31 19:40:35] ...occurrences (/app/public/data/boistel_et_al_bo/occurrences.txt)
[CMD] [2021-05-31 19:40:35] /usr/bin/sort /app/public/converted_csv/boistel_et_al_bo_occurrences_3957.csv > /app/public/converted_csv/boistel_et_al_bo_occurrences_3957.csv_sorted
[INFO] [2021-05-31 19:40:35] Converted: /app/public/converted_csv/boistel_et_al_bo_occurrences_3957.csv (1 lines)
[INFO] [2021-05-31 19:40:35] ...measurements (/app/public/data/boistel_et_al_bo/measurementsorfacts.txt)
[CMD] [2021-05-31 19:40:35] /usr/bin/sort /app/public/converted_csv/boistel_et_al_bo_measurements_3957.csv > /app/public/converted_csv/boistel_et_al_bo_measurements_3957.csv_sorted
[INFO] [2021-05-31 19:40:35] Converted: /app/public/converted_csv/boistel_et_al_bo_measurements_3957.csv (3 lines)
[STOP] [2021-05-31 19:40:35] convert_to_csv
[START] [2021-05-31 19:40:35] calculate_delta
[INFO] [2021-05-31 19:40:35] Looping over 4 formats...
[INFO] [2021-05-31 19:40:35] ...refs (/app/public/data/boistel_et_al_bo/references.txt)
[CMD] [2021-05-31 19:40:35] echo "0a" > /app/public/diff/boistel_et_al_bo_refs_3957.diff
[CMD] [2021-05-31 19:40:36] tail -n +1 /app/public/converted_csv/boistel_et_al_bo_refs_3957.csv >> /app/public/diff/boistel_et_al_bo_refs_3957.diff
[CMD] [2021-05-31 19:40:36] echo "." >> /app/public/diff/boistel_et_al_bo_refs_3957.diff
[INFO] [2021-05-31 19:40:36] Created diff: /app/public/diff/boistel_et_al_bo_refs_3957.diff (2 lines)
[INFO] [2021-05-31 19:40:36] ...nodes (/app/public/data/boistel_et_al_bo/taxa.txt)
[CMD] [2021-05-31 19:40:36] echo "0a" > /app/public/diff/boistel_et_al_bo_nodes_3957.diff
[CMD] [2021-05-31 19:40:37] tail -n +1 /app/public/converted_csv/boistel_et_al_bo_nodes_3957.csv >> /app/public/diff/boistel_et_al_bo_nodes_3957.diff
[CMD] [2021-05-31 19:40:37] echo "." >> /app/public/diff/boistel_et_al_bo_nodes_3957.diff
[INFO] [2021-05-31 19:40:38] Created diff: /app/public/diff/boistel_et_al_bo_nodes_3957.diff (3 lines)
[INFO] [2021-05-31 19:40:38] ...occurrences (/app/public/data/boistel_et_al_bo/occurrences.txt)
[CMD] [2021-05-31 19:40:38] echo "0a" > /app/public/diff/boistel_et_al_bo_occurrences_3957.diff
[CMD] [2021-05-31 19:40:38] tail -n +1 /app/public/converted_csv/boistel_et_al_bo_occurrences_3957.csv >> /app/public/diff/boistel_et_al_bo_occurrences_3957.diff
[CMD] [2021-05-31 19:40:38] echo "." >> /app/public/diff/boistel_et_al_bo_occurrences_3957.diff
[INFO] [2021-05-31 19:40:39] Created diff: /app/public/diff/boistel_et_al_bo_occurrences_3957.diff (3 lines)
[INFO] [2021-05-31 19:40:39] ...measurements (/app/public/data/boistel_et_al_bo/measurementsorfacts.txt)
[CMD] [2021-05-31 19:40:39] echo "0a" > /app/public/diff/boistel_et_al_bo_measurements_3957.diff
[CMD] [2021-05-31 19:40:39] tail -n +1 /app/public/converted_csv/boistel_et_al_bo_measurements_3957.csv >> /app/public/diff/boistel_et_al_bo_measurements_3957.diff
[CMD] [2021-05-31 19:40:40] echo "." >> /app/public/diff/boistel_et_al_bo_measurements_3957.diff
[INFO] [2021-05-31 19:40:40] Created diff: /app/public/diff/boistel_et_al_bo_measurements_3957.diff (5 lines)
[STOP] [2021-05-31 19:40:40] calculate_delta
[START] [2021-05-31 19:40:40] parse_diff_and_store
[INFO] [2021-05-31 19:40:40] Handling diff: /app/public/diff/boistel_et_al_bo_refs_3957.diff (2 lines)
[INFO] [2021-05-31 19:40:40] Loading refs diff file into memory (2 /app/public/diff/boistel_et_al_bo_refs_3957.diff lines)...
[INFO] [2021-05-31 19:40:41] Handling diff: /app/public/diff/boistel_et_al_bo_nodes_3957.diff (3 lines)
[INFO] [2021-05-31 19:40:41] Loading nodes diff file into memory (3 /app/public/diff/boistel_et_al_bo_nodes_3957.diff lines)...
[INFO] [2021-05-31 19:40:41] Handling diff: /app/public/diff/boistel_et_al_bo_occurrences_3957.diff (3 lines)
[INFO] [2021-05-31 19:40:42] Loading occurrences diff file into memory (3 /app/public/diff/boistel_et_al_bo_occurrences_3957.diff lines)...
[INFO] [2021-05-31 19:40:42] Handling diff: /app/public/diff/boistel_et_al_bo_measurements_3957.diff (5 lines)
[INFO] [2021-05-31 19:40:43] Loading measurements diff file into memory (5 /app/public/diff/boistel_et_al_bo_measurements_3957.diff lines)...
[INFO] [2021-05-31 19:40:43] Storing 1 ScientificNames
[INFO] [2021-05-31 19:40:43] Processing group of 1 in 1 groups of 1000
[INFO] [2021-05-31 19:40:43] Average Time: 0.0
[INFO] [2021-05-31 19:40:43] Total Time: 1s
[INFO] [2021-05-31 19:40:43] Storing 1 Nodes
[INFO] [2021-05-31 19:40:43] Processing group of 1 in 1 groups of 1000
[INFO] [2021-05-31 19:40:43] Average Time: 0.0
[INFO] [2021-05-31 19:40:43] Total Time: 1s
[INFO] [2021-05-31 19:40:43] Storing 1 Occurrences
[INFO] [2021-05-31 19:40:43] Processing group of 1 in 1 groups of 1000
[INFO] [2021-05-31 19:40:43] Average Time: 0.0
[INFO] [2021-05-31 19:40:43] Total Time: 1s
[INFO] [2021-05-31 19:40:43] Storing 1 OccurrenceMetadata
[INFO] [2021-05-31 19:40:43] Processing group of 1 in 1 groups of 1000
[INFO] [2021-05-31 19:40:43] Average Time: 0.0
[INFO] [2021-05-31 19:40:43] Total Time: 1s
[INFO] [2021-05-31 19:40:43] Storing 3 Traits
[INFO] [2021-05-31 19:40:43] Processing group of 3 in 1 groups of 1000
[INFO] [2021-05-31 19:40:43] Average Time: 0.0
[INFO] [2021-05-31 19:40:43] Total Time: 1s
[INFO] [2021-05-31 19:40:43] Storing 5 MetaTraits
[INFO] [2021-05-31 19:40:43] Processing group of 5 in 1 groups of 1000
[INFO] [2021-05-31 19:40:43] Average Time: 0.0
[INFO] [2021-05-31 19:40:43] Total Time: 1s
[STOP] [2021-05-31 19:40:43] parse_diff_and_store
[START] [2021-05-31 19:40:43] resolve_keys
[INFO] [2021-05-31 19:40:49] Occurrences to nodes (through scientific_names)...
[INFO] [2021-05-31 19:40:49] traits to occurrences...
[INFO] [2021-05-31 19:40:49] traits to nodes (through occurrences)...
[INFO] [2021-05-31 19:40:49] Traits to sex term...
[INFO] [2021-05-31 19:40:49] Traits to lifestage term...
[INFO] [2021-05-31 19:40:49] MetaTraits to traits...
[INFO] [2021-05-31 19:40:49] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-05-31 19:40:49] Assocs to occurrences...
[INFO] [2021-05-31 19:40:49] Assocs to nodes...
[INFO] [2021-05-31 19:40:49] Assoc to sex term...
[INFO] [2021-05-31 19:40:49] Assoc to lifestage term...
[INFO] [2021-05-31 19:40:49] MetaAssoc to assocs...
[STOP] [2021-05-31 19:40:49] resolve_keys
[START] [2021-05-31 19:40:49] hold_for_later_1
[STOP] [2021-05-31 19:40:49] hold_for_later_1
[START] [2021-05-31 19:40:49] hold_for_later_2
[STOP] [2021-05-31 19:40:49] hold_for_later_2
[START] [2021-05-31 19:40:49] resolve_missing_parents
[STOP] [2021-05-31 19:40:49] resolve_missing_parents
[START] [2021-05-31 19:40:49] rebuild_nodes
[START] [2021-05-31 19:40:49] Flattener#flatten
[START] [2021-05-31 19:40:49] Flattener#study_resource
[START] [2021-05-31 19:40:49] Flattener#build_ancestry
[STOP] [2021-05-31 19:40:49] Flattener#build_ancestry
[INFO] [2021-05-31 19:40:49] 1 ancestry keys
[START] [2021-05-31 19:40:49] build_node_ancestors
[INFO] [2021-05-31 19:40:49] old ancestors deleted.
[STOP] [2021-05-31 19:40:49] build_node_ancestors
[WARN] [2021-05-31 19:40:49] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2021-05-31 19:40:49] Flattener#flatten
[STOP] [2021-05-31 19:40:49] rebuild_nodes
[START] [2021-05-31 19:40:49] resolve_missing_media_owners
[STOP] [2021-05-31 19:40:49] resolve_missing_media_owners
[START] [2021-05-31 19:40:49] sanitize_media_verbatims
[STOP] [2021-05-31 19:40:49] sanitize_media_verbatims
[START] [2021-05-31 19:40:49] queue_downloads
[STOP] [2021-05-31 19:40:49] queue_downloads
[START] [2021-05-31 19:40:49] parse_names
[WARN] [2021-05-31 19:40:49] I see 1 names which still need to be parsed.
[STOP] [2021-05-31 19:40:50] parse_names
[START] [2021-05-31 19:40:50] denormalize_canonical_names_to_nodes
[STOP] [2021-05-31 19:40:50] denormalize_canonical_names_to_nodes
[START] [2021-05-31 19:40:50] match_nodes
[START] [2021-05-31 19:40:50] map_all_nodes_to_pages
[STOP] [2021-05-31 19:40:50] map_all_nodes_to_pages
[INFO] [2021-05-31 19:40:50] ZERO unmatched nodes (of 1)! Nicely done.
[START] [2021-05-31 19:40:50] update_nodes
[STOP] [2021-05-31 19:40:50] update_nodes
[STOP] [2021-05-31 19:40:50] match_nodes
[START] [2021-05-31 19:40:50] reindex_search
[STOP] [2021-05-31 19:40:50] reindex_search
[START] [2021-05-31 19:40:50] normalize_units
[STOP] [2021-05-31 19:40:50] normalize_units
[START] [2021-05-31 19:40:50] calculate_statistics
[2021-05-31 19:40:50] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2021-05-31 19:40:50] calculate_statistics
[START] [2021-05-31 19:40:50] complete_harvest_instance
[START] [2021-05-31 19:40:50] overall_tsv_creation
[INFO] [2021-05-31 19:40:50] Processing group of 1 in 1 batches of 10000
[INFO] [2021-05-31 19:41:22] 3 Traits (unfiltered)...
[INFO] [2021-05-31 19:41:47] 3 Traits (filtered)...
[INFO] [2021-05-31 19:41:47] 0 Associations (filtered)...
[INFO] [2021-05-31 19:41:47] 0 metadata added.
[INFO] [2021-05-31 19:41:47] 0 metadata added.
[INFO] [2021-05-31 19:42:11] Average Time: 59.08
[INFO] [2021-05-31 19:42:11] Total Time: 1m21s
[STOP] [2021-05-31 19:42:11] overall_tsv_creation
[INFO] [2021-05-31 19:42:11] Done. Check your files:
[INFO] [2021-05-31 19:42:11] (1 lines) /app/public/data/boistel_et_al_bo/publish_nodes.tsv
[INFO] [2021-05-31 19:42:11] (1 lines) /app/public/data/boistel_et_al_bo/publish_scientific_names.tsv
[INFO] [2021-05-31 19:42:12] (4 lines) /app/public/data/boistel_et_al_bo/publish_traits.tsv
[INFO] [2021-05-31 19:42:12] (1 lines) /app/public/data/boistel_et_al_bo/publish_metadata.tsv
[STOP] [2021-05-31 19:42:12] complete_harvest_instance
[START] [2021-05-31 19:42:12] completed
[STOP] [2021-05-31 19:42:12] completed
[STOP] [2021-05-31 19:42:12] logged process, took 98.8

Latest Process

31 May 19:40 EXITED currently processing 0