Harvest for Ladich and Popper 2004 Created 31 May 19:43

Stage: completed
Fetched: 31 May 19:43
Validated: 31 May 19:43
Deltas Created 31 May 19:43
Units Normalized: 31 May 19:44
Ancestry Built: 31 May 19:44
Nodes Matched: 31 May 19:44
Names Parsed: 31 May 19:44
New Models Stored: 31 May 19:44
Indexed: 31 May 19:44
Completed: 31 May 19:45
Time to Harvest: less than a minute

Harvesting Log

(168 lines)
[INFO] [2021-05-31 19:43:52] Created harvest instance #3959
[STOP] [2021-05-31 19:43:52] create_harvest_instance
[START] [2021-05-31 19:43:52] fetch_files
[STOP] [2021-05-31 19:43:52] fetch_files
[START] [2021-05-31 19:43:53] validate_each_file
[INFO] [2021-05-31 19:43:53] Looping over 4 formats...
[INFO] [2021-05-31 19:43:53] ...refs (/app/public/data/ladich_popper_la/references.txt)
[INFO] [2021-05-31 19:43:53] Valid: /app/public/converted_csv/ladich_popper_la_refs_3959.csv (8 lines)
[INFO] [2021-05-31 19:43:53] ...nodes (/app/public/data/ladich_popper_la/taxa.txt)
[INFO] [2021-05-31 19:43:53] Valid: /app/public/converted_csv/ladich_popper_la_nodes_3959.csv (12 lines)
[INFO] [2021-05-31 19:43:53] ...occurrences (/app/public/data/ladich_popper_la/occurrences.txt)
[INFO] [2021-05-31 19:43:53] Valid: /app/public/converted_csv/ladich_popper_la_occurrences_3959.csv (12 lines)
[INFO] [2021-05-31 19:43:53] ...measurements (/app/public/data/ladich_popper_la/measurementsorfacts.txt)
[INFO] [2021-05-31 19:43:53] Valid: /app/public/converted_csv/ladich_popper_la_measurements_3959.csv (111 lines)
[STOP] [2021-05-31 19:43:53] validate_each_file
[START] [2021-05-31 19:43:53] convert_to_csv
[INFO] [2021-05-31 19:43:53] Looping over 4 formats...
[INFO] [2021-05-31 19:43:53] ...refs (/app/public/data/ladich_popper_la/references.txt)
[CMD] [2021-05-31 19:43:53] /usr/bin/sort /app/public/converted_csv/ladich_popper_la_refs_3959.csv > /app/public/converted_csv/ladich_popper_la_refs_3959.csv_sorted
[INFO] [2021-05-31 19:43:53] Converted: /app/public/converted_csv/ladich_popper_la_refs_3959.csv (8 lines)
[INFO] [2021-05-31 19:43:53] ...nodes (/app/public/data/ladich_popper_la/taxa.txt)
[CMD] [2021-05-31 19:43:53] /usr/bin/sort /app/public/converted_csv/ladich_popper_la_nodes_3959.csv > /app/public/converted_csv/ladich_popper_la_nodes_3959.csv_sorted
[INFO] [2021-05-31 19:43:53] Converted: /app/public/converted_csv/ladich_popper_la_nodes_3959.csv (12 lines)
[INFO] [2021-05-31 19:43:53] ...occurrences (/app/public/data/ladich_popper_la/occurrences.txt)
[CMD] [2021-05-31 19:43:53] /usr/bin/sort /app/public/converted_csv/ladich_popper_la_occurrences_3959.csv > /app/public/converted_csv/ladich_popper_la_occurrences_3959.csv_sorted
[INFO] [2021-05-31 19:43:54] Converted: /app/public/converted_csv/ladich_popper_la_occurrences_3959.csv (12 lines)
[INFO] [2021-05-31 19:43:54] ...measurements (/app/public/data/ladich_popper_la/measurementsorfacts.txt)
[CMD] [2021-05-31 19:43:54] /usr/bin/sort /app/public/converted_csv/ladich_popper_la_measurements_3959.csv > /app/public/converted_csv/ladich_popper_la_measurements_3959.csv_sorted
[INFO] [2021-05-31 19:43:54] Converted: /app/public/converted_csv/ladich_popper_la_measurements_3959.csv (111 lines)
[STOP] [2021-05-31 19:43:54] convert_to_csv
[START] [2021-05-31 19:43:54] calculate_delta
[INFO] [2021-05-31 19:43:54] Looping over 4 formats...
[INFO] [2021-05-31 19:43:54] ...refs (/app/public/data/ladich_popper_la/references.txt)
[CMD] [2021-05-31 19:43:54] echo "0a" > /app/public/diff/ladich_popper_la_refs_3959.diff
[CMD] [2021-05-31 19:43:55] tail -n +1 /app/public/converted_csv/ladich_popper_la_refs_3959.csv >> /app/public/diff/ladich_popper_la_refs_3959.diff
[CMD] [2021-05-31 19:43:55] echo "." >> /app/public/diff/ladich_popper_la_refs_3959.diff
[INFO] [2021-05-31 19:43:55] Created diff: /app/public/diff/ladich_popper_la_refs_3959.diff (10 lines)
[INFO] [2021-05-31 19:43:55] ...nodes (/app/public/data/ladich_popper_la/taxa.txt)
[CMD] [2021-05-31 19:43:55] echo "0a" > /app/public/diff/ladich_popper_la_nodes_3959.diff
[CMD] [2021-05-31 19:43:56] tail -n +1 /app/public/converted_csv/ladich_popper_la_nodes_3959.csv >> /app/public/diff/ladich_popper_la_nodes_3959.diff
[CMD] [2021-05-31 19:43:56] echo "." >> /app/public/diff/ladich_popper_la_nodes_3959.diff
[INFO] [2021-05-31 19:43:56] Created diff: /app/public/diff/ladich_popper_la_nodes_3959.diff (14 lines)
[INFO] [2021-05-31 19:43:56] ...occurrences (/app/public/data/ladich_popper_la/occurrences.txt)
[CMD] [2021-05-31 19:43:56] echo "0a" > /app/public/diff/ladich_popper_la_occurrences_3959.diff
[CMD] [2021-05-31 19:43:57] tail -n +1 /app/public/converted_csv/ladich_popper_la_occurrences_3959.csv >> /app/public/diff/ladich_popper_la_occurrences_3959.diff
[CMD] [2021-05-31 19:43:57] echo "." >> /app/public/diff/ladich_popper_la_occurrences_3959.diff
[INFO] [2021-05-31 19:43:58] Created diff: /app/public/diff/ladich_popper_la_occurrences_3959.diff (14 lines)
[INFO] [2021-05-31 19:43:58] ...measurements (/app/public/data/ladich_popper_la/measurementsorfacts.txt)
[CMD] [2021-05-31 19:43:58] echo "0a" > /app/public/diff/ladich_popper_la_measurements_3959.diff
[CMD] [2021-05-31 19:43:58] tail -n +1 /app/public/converted_csv/ladich_popper_la_measurements_3959.csv >> /app/public/diff/ladich_popper_la_measurements_3959.diff
[CMD] [2021-05-31 19:43:58] echo "." >> /app/public/diff/ladich_popper_la_measurements_3959.diff
[INFO] [2021-05-31 19:43:59] Created diff: /app/public/diff/ladich_popper_la_measurements_3959.diff (113 lines)
[STOP] [2021-05-31 19:43:59] calculate_delta
[START] [2021-05-31 19:43:59] parse_diff_and_store
[INFO] [2021-05-31 19:43:59] Handling diff: /app/public/diff/ladich_popper_la_refs_3959.diff (10 lines)
[INFO] [2021-05-31 19:43:59] Loading refs diff file into memory (10 /app/public/diff/ladich_popper_la_refs_3959.diff lines)...
[INFO] [2021-05-31 19:44:00] Handling diff: /app/public/diff/ladich_popper_la_nodes_3959.diff (14 lines)
[INFO] [2021-05-31 19:44:00] Loading nodes diff file into memory (14 /app/public/diff/ladich_popper_la_nodes_3959.diff lines)...
[INFO] [2021-05-31 19:44:00] Handling diff: /app/public/diff/ladich_popper_la_occurrences_3959.diff (14 lines)
[INFO] [2021-05-31 19:44:01] Loading occurrences diff file into memory (14 /app/public/diff/ladich_popper_la_occurrences_3959.diff lines)...
[INFO] [2021-05-31 19:44:01] Handling diff: /app/public/diff/ladich_popper_la_measurements_3959.diff (113 lines)
[INFO] [2021-05-31 19:44:01] Loading measurements diff file into memory (113 /app/public/diff/ladich_popper_la_measurements_3959.diff lines)...
[INFO] [2021-05-31 19:44:02] Storing 8 References
[INFO] [2021-05-31 19:44:02] Processing group of 8 in 1 groups of 1000
[INFO] [2021-05-31 19:44:02] Average Time: 0.0
[INFO] [2021-05-31 19:44:02] Total Time: 1s
[INFO] [2021-05-31 19:44:02] Storing 12 ScientificNames
[INFO] [2021-05-31 19:44:02] Processing group of 12 in 1 groups of 1000
[INFO] [2021-05-31 19:44:02] Average Time: 0.01
[INFO] [2021-05-31 19:44:02] Total Time: 1s
[INFO] [2021-05-31 19:44:02] Storing 12 Nodes
[INFO] [2021-05-31 19:44:02] Processing group of 12 in 1 groups of 1000
[INFO] [2021-05-31 19:44:02] Average Time: 0.0
[INFO] [2021-05-31 19:44:02] Total Time: 1s
[INFO] [2021-05-31 19:44:02] Storing 12 Occurrences
[INFO] [2021-05-31 19:44:02] Processing group of 12 in 1 groups of 1000
[INFO] [2021-05-31 19:44:02] Average Time: 0.0
[INFO] [2021-05-31 19:44:02] Total Time: 1s
[INFO] [2021-05-31 19:44:02] Storing 40 TraitsReferences
[INFO] [2021-05-31 19:44:02] Processing group of 40 in 1 groups of 1000
[INFO] [2021-05-31 19:44:02] Average Time: 0.0
[INFO] [2021-05-31 19:44:02] Total Time: 1s
[INFO] [2021-05-31 19:44:02] Storing 111 Traits
[INFO] [2021-05-31 19:44:02] Processing group of 111 in 1 groups of 1000
[INFO] [2021-05-31 19:44:02] Average Time: 0.04
[INFO] [2021-05-31 19:44:02] Total Time: 1s
[INFO] [2021-05-31 19:44:02] Storing 70 MetaTraits
[INFO] [2021-05-31 19:44:02] Processing group of 70 in 1 groups of 1000
[INFO] [2021-05-31 19:44:02] Average Time: 0.01
[INFO] [2021-05-31 19:44:02] Total Time: 1s
[STOP] [2021-05-31 19:44:02] parse_diff_and_store
[START] [2021-05-31 19:44:02] resolve_keys
[INFO] [2021-05-31 19:44:08] Occurrences to nodes (through scientific_names)...
[INFO] [2021-05-31 19:44:08] traits to occurrences...
[INFO] [2021-05-31 19:44:08] traits to nodes (through occurrences)...
[INFO] [2021-05-31 19:44:08] Traits to sex term...
[INFO] [2021-05-31 19:44:08] Traits to lifestage term...
[INFO] [2021-05-31 19:44:08] MetaTraits to traits...
[INFO] [2021-05-31 19:44:08] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-05-31 19:44:08] Assocs to occurrences...
[INFO] [2021-05-31 19:44:08] Assocs to nodes...
[INFO] [2021-05-31 19:44:08] Assoc to sex term...
[INFO] [2021-05-31 19:44:08] Assoc to lifestage term...
[INFO] [2021-05-31 19:44:08] MetaAssoc to assocs...
[STOP] [2021-05-31 19:44:08] resolve_keys
[START] [2021-05-31 19:44:08] hold_for_later_1
[STOP] [2021-05-31 19:44:08] hold_for_later_1
[START] [2021-05-31 19:44:08] hold_for_later_2
[STOP] [2021-05-31 19:44:08] hold_for_later_2
[START] [2021-05-31 19:44:08] resolve_missing_parents
[STOP] [2021-05-31 19:44:08] resolve_missing_parents
[START] [2021-05-31 19:44:08] rebuild_nodes
[START] [2021-05-31 19:44:08] Flattener#flatten
[START] [2021-05-31 19:44:08] Flattener#study_resource
[START] [2021-05-31 19:44:08] Flattener#build_ancestry
[STOP] [2021-05-31 19:44:08] Flattener#build_ancestry
[INFO] [2021-05-31 19:44:08] 12 ancestry keys
[START] [2021-05-31 19:44:08] build_node_ancestors
[INFO] [2021-05-31 19:44:08] old ancestors deleted.
[STOP] [2021-05-31 19:44:08] build_node_ancestors
[WARN] [2021-05-31 19:44:08] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2021-05-31 19:44:08] Flattener#flatten
[STOP] [2021-05-31 19:44:08] rebuild_nodes
[START] [2021-05-31 19:44:08] resolve_missing_media_owners
[STOP] [2021-05-31 19:44:08] resolve_missing_media_owners
[START] [2021-05-31 19:44:08] sanitize_media_verbatims
[STOP] [2021-05-31 19:44:08] sanitize_media_verbatims
[START] [2021-05-31 19:44:08] queue_downloads
[STOP] [2021-05-31 19:44:08] queue_downloads
[START] [2021-05-31 19:44:08] parse_names
[WARN] [2021-05-31 19:44:08] I see 12 names which still need to be parsed.
[STOP] [2021-05-31 19:44:09] parse_names
[START] [2021-05-31 19:44:09] denormalize_canonical_names_to_nodes
[STOP] [2021-05-31 19:44:09] denormalize_canonical_names_to_nodes
[START] [2021-05-31 19:44:09] match_nodes
[START] [2021-05-31 19:44:09] map_all_nodes_to_pages
[STOP] [2021-05-31 19:44:09] map_all_nodes_to_pages
[INFO] [2021-05-31 19:44:09] ZERO unmatched nodes (of 12)! Nicely done.
[START] [2021-05-31 19:44:09] update_nodes
[STOP] [2021-05-31 19:44:09] update_nodes
[STOP] [2021-05-31 19:44:09] match_nodes
[START] [2021-05-31 19:44:09] reindex_search
[STOP] [2021-05-31 19:44:09] reindex_search
[START] [2021-05-31 19:44:09] normalize_units
[STOP] [2021-05-31 19:44:09] normalize_units
[START] [2021-05-31 19:44:09] calculate_statistics
[2021-05-31 19:44:09] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[STOP] [2021-05-31 19:44:09] calculate_statistics
[START] [2021-05-31 19:44:09] complete_harvest_instance
[START] [2021-05-31 19:44:09] overall_tsv_creation
[INFO] [2021-05-31 19:44:09] Processing group of 12 in 1 batches of 10000
[INFO] [2021-05-31 19:44:42] 39 Traits (unfiltered)...
[INFO] [2021-05-31 19:45:08] 39 Traits (filtered)...
[INFO] [2021-05-31 19:45:08] 0 Associations (filtered)...
[INFO] [2021-05-31 19:45:08] 112 metadata added.
[INFO] [2021-05-31 19:45:08] 0 metadata added.
[INFO] [2021-05-31 19:45:31] Average Time: 59.64
[INFO] [2021-05-31 19:45:31] Total Time: 1m22s
[STOP] [2021-05-31 19:45:31] overall_tsv_creation
[INFO] [2021-05-31 19:45:31] Done. Check your files:
[INFO] [2021-05-31 19:45:31] (12 lines) /app/public/data/ladich_popper_la/publish_nodes.tsv
[INFO] [2021-05-31 19:45:32] (12 lines) /app/public/data/ladich_popper_la/publish_scientific_names.tsv
[INFO] [2021-05-31 19:45:32] (40 lines) /app/public/data/ladich_popper_la/publish_traits.tsv
[INFO] [2021-05-31 19:45:32] (113 lines) /app/public/data/ladich_popper_la/publish_metadata.tsv
[STOP] [2021-05-31 19:45:32] complete_harvest_instance
[START] [2021-05-31 19:45:32] completed
[STOP] [2021-05-31 19:45:32] completed
[STOP] [2021-05-31 19:45:32] logged process, took 100.43

Latest Process