Harvest for Hershkovitz et al Created 17 May 15:33

Stage: completed
Fetched: 17 May 15:33
Validated: 17 May 15:33
Deltas Created 17 May 15:33
Units Normalized: 17 May 15:33
Ancestry Built: 17 May 15:33
Nodes Matched: 17 May 15:33
Names Parsed: 17 May 15:33
New Models Stored: 17 May 15:33
Indexed: 17 May 15:33
Completed: 17 May 15:36
Time to Harvest: less than a minute
Harvesting Log

(151 lines)
[INFO] [2021-05-17 15:33:15] Created harvest instance #3875
[STOP] [2021-05-17 15:33:15] create_harvest_instance
[START] [2021-05-17 15:33:15] fetch_files
[STOP] [2021-05-17 15:33:15] fetch_files
[START] [2021-05-17 15:33:15] validate_each_file
[INFO] [2021-05-17 15:33:15] Looping over 3 formats...
[INFO] [2021-05-17 15:33:15] ...nodes (/app/public/data/hershkovitz_et_3/taxon.tab)
[INFO] [2021-05-17 15:33:15] Valid: /app/public/converted_csv/hershkovitz_et_3_nodes_3875.csv (1940 lines)
[INFO] [2021-05-17 15:33:15] ...occurrences (/app/public/data/hershkovitz_et_3/occurrence.tab)
[INFO] [2021-05-17 15:33:15] Valid: /app/public/converted_csv/hershkovitz_et_3_occurrences_3875.csv (3880 lines)
[INFO] [2021-05-17 15:33:15] ...measurements (/app/public/data/hershkovitz_et_3/measurement_or_fact.tsv)
[INFO] [2021-05-17 15:33:16] Valid: /app/public/converted_csv/hershkovitz_et_3_measurements_3875.csv (5820 lines)
[STOP] [2021-05-17 15:33:16] validate_each_file
[START] [2021-05-17 15:33:16] convert_to_csv
[INFO] [2021-05-17 15:33:16] Looping over 3 formats...
[INFO] [2021-05-17 15:33:16] ...nodes (/app/public/data/hershkovitz_et_3/taxon.tab)
[CMD] [2021-05-17 15:33:16] /usr/bin/sort /app/public/converted_csv/hershkovitz_et_3_nodes_3875.csv > /app/public/converted_csv/hershkovitz_et_3_nodes_3875.csv_sorted
[INFO] [2021-05-17 15:33:16] Converted: /app/public/converted_csv/hershkovitz_et_3_nodes_3875.csv (1940 lines)
[INFO] [2021-05-17 15:33:16] ...occurrences (/app/public/data/hershkovitz_et_3/occurrence.tab)
[CMD] [2021-05-17 15:33:16] /usr/bin/sort /app/public/converted_csv/hershkovitz_et_3_occurrences_3875.csv > /app/public/converted_csv/hershkovitz_et_3_occurrences_3875.csv_sorted
[INFO] [2021-05-17 15:33:16] Converted: /app/public/converted_csv/hershkovitz_et_3_occurrences_3875.csv (3880 lines)
[INFO] [2021-05-17 15:33:16] ...measurements (/app/public/data/hershkovitz_et_3/measurement_or_fact.tsv)
[CMD] [2021-05-17 15:33:16] /usr/bin/sort /app/public/converted_csv/hershkovitz_et_3_measurements_3875.csv > /app/public/converted_csv/hershkovitz_et_3_measurements_3875.csv_sorted
[INFO] [2021-05-17 15:33:16] Converted: /app/public/converted_csv/hershkovitz_et_3_measurements_3875.csv (5820 lines)
[STOP] [2021-05-17 15:33:16] convert_to_csv
[START] [2021-05-17 15:33:16] calculate_delta
[INFO] [2021-05-17 15:33:16] Looping over 3 formats...
[INFO] [2021-05-17 15:33:16] ...nodes (/app/public/data/hershkovitz_et_3/taxon.tab)
[CMD] [2021-05-17 15:33:16] echo "0a" > /app/public/diff/hershkovitz_et_3_nodes_3875.diff
[CMD] [2021-05-17 15:33:16] tail -n +1 /app/public/converted_csv/hershkovitz_et_3_nodes_3875.csv >> /app/public/diff/hershkovitz_et_3_nodes_3875.diff
[CMD] [2021-05-17 15:33:16] echo "." >> /app/public/diff/hershkovitz_et_3_nodes_3875.diff
[INFO] [2021-05-17 15:33:16] Created diff: /app/public/diff/hershkovitz_et_3_nodes_3875.diff (1942 lines)
[INFO] [2021-05-17 15:33:16] ...occurrences (/app/public/data/hershkovitz_et_3/occurrence.tab)
[CMD] [2021-05-17 15:33:16] echo "0a" > /app/public/diff/hershkovitz_et_3_occurrences_3875.diff
[CMD] [2021-05-17 15:33:16] tail -n +1 /app/public/converted_csv/hershkovitz_et_3_occurrences_3875.csv >> /app/public/diff/hershkovitz_et_3_occurrences_3875.diff
[CMD] [2021-05-17 15:33:16] echo "." >> /app/public/diff/hershkovitz_et_3_occurrences_3875.diff
[INFO] [2021-05-17 15:33:16] Created diff: /app/public/diff/hershkovitz_et_3_occurrences_3875.diff (3882 lines)
[INFO] [2021-05-17 15:33:16] ...measurements (/app/public/data/hershkovitz_et_3/measurement_or_fact.tsv)
[CMD] [2021-05-17 15:33:16] echo "0a" > /app/public/diff/hershkovitz_et_3_measurements_3875.diff
[CMD] [2021-05-17 15:33:16] tail -n +1 /app/public/converted_csv/hershkovitz_et_3_measurements_3875.csv >> /app/public/diff/hershkovitz_et_3_measurements_3875.diff
[CMD] [2021-05-17 15:33:16] echo "." >> /app/public/diff/hershkovitz_et_3_measurements_3875.diff
[INFO] [2021-05-17 15:33:16] Created diff: /app/public/diff/hershkovitz_et_3_measurements_3875.diff (5822 lines)
[STOP] [2021-05-17 15:33:16] calculate_delta
[START] [2021-05-17 15:33:16] parse_diff_and_store
[INFO] [2021-05-17 15:33:16] Handling diff: /app/public/diff/hershkovitz_et_3_nodes_3875.diff (1942 lines)
[INFO] [2021-05-17 15:33:16] Loading nodes diff file into memory (1942 /app/public/diff/hershkovitz_et_3_nodes_3875.diff lines)...
[INFO] [2021-05-17 15:33:17] Handling diff: /app/public/diff/hershkovitz_et_3_occurrences_3875.diff (3882 lines)
[INFO] [2021-05-17 15:33:17] Loading occurrences diff file into memory (3882 /app/public/diff/hershkovitz_et_3_occurrences_3875.diff lines)...
[INFO] [2021-05-17 15:33:17] Handling diff: /app/public/diff/hershkovitz_et_3_measurements_3875.diff (5822 lines)
[INFO] [2021-05-17 15:33:17] Loading measurements diff file into memory (5822 /app/public/diff/hershkovitz_et_3_measurements_3875.diff lines)...
[INFO] [2021-05-17 15:33:20] Storing 2197 ScientificNames
[INFO] [2021-05-17 15:33:20] Processing group of 2197 in 3 groups of 1000
[INFO] [2021-05-17 15:33:21] Average Time: 0.223
[INFO] [2021-05-17 15:33:21] Total Time: 1s
[INFO] [2021-05-17 15:33:21] Storing 2197 Nodes
[INFO] [2021-05-17 15:33:21] Processing group of 2197 in 3 groups of 1000
[INFO] [2021-05-17 15:33:21] Average Time: 0.233
[INFO] [2021-05-17 15:33:21] Total Time: 1s
[INFO] [2021-05-17 15:33:21] Storing 3880 Occurrences
[INFO] [2021-05-17 15:33:21] Processing group of 3880 in 4 groups of 1000
[INFO] [2021-05-17 15:33:22] Average Time: 0.198
[INFO] [2021-05-17 15:33:22] Total Time: 1s
[INFO] [2021-05-17 15:33:22] Storing 5820 Traits
[INFO] [2021-05-17 15:33:22] Processing group of 5820 in 6 groups of 1000
[INFO] [2021-05-17 15:33:24] Average Time: 0.277
[INFO] [2021-05-17 15:33:24] Total Time: 2s
[INFO] [2021-05-17 15:33:24] Storing 7760 MetaTraits
[INFO] [2021-05-17 15:33:24] Processing group of 7760 in 8 groups of 1000
[INFO] [2021-05-17 15:33:25] Average Time: 0.123
[INFO] [2021-05-17 15:33:25] Total Time: 2s
[INFO] [2021-05-17 15:33:25] last 3 / first 3: 1.11
[INFO] [2021-05-17 15:33:25] Std.Dev: 0.0; Max: 0.15
[STOP] [2021-05-17 15:33:25] parse_diff_and_store
[START] [2021-05-17 15:33:25] resolve_keys
[INFO] [2021-05-17 15:33:31] Occurrences to nodes (through scientific_names)...
[INFO] [2021-05-17 15:33:31] traits to occurrences...
[INFO] [2021-05-17 15:33:31] traits to nodes (through occurrences)...
[INFO] [2021-05-17 15:33:32] Traits to sex term...
[INFO] [2021-05-17 15:33:32] Traits to lifestage term...
[INFO] [2021-05-17 15:33:32] MetaTraits to traits...
[INFO] [2021-05-17 15:33:32] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-05-17 15:33:32] Assocs to occurrences...
[INFO] [2021-05-17 15:33:32] Assocs to nodes...
[INFO] [2021-05-17 15:33:32] Assoc to sex term...
[INFO] [2021-05-17 15:33:32] Assoc to lifestage term...
[INFO] [2021-05-17 15:33:32] MetaAssoc to assocs...
[STOP] [2021-05-17 15:33:32] resolve_keys
[START] [2021-05-17 15:33:32] hold_for_later_1
[STOP] [2021-05-17 15:33:32] hold_for_later_1
[START] [2021-05-17 15:33:32] hold_for_later_2
[STOP] [2021-05-17 15:33:32] hold_for_later_2
[START] [2021-05-17 15:33:32] resolve_missing_parents
[STOP] [2021-05-17 15:33:32] resolve_missing_parents
[START] [2021-05-17 15:33:32] rebuild_nodes
[START] [2021-05-17 15:33:32] Flattener#flatten
[START] [2021-05-17 15:33:32] Flattener#study_resource
[START] [2021-05-17 15:33:32] Flattener#build_ancestry
[STOP] [2021-05-17 15:33:32] Flattener#build_ancestry
[INFO] [2021-05-17 15:33:32] 2197 ancestry keys
[START] [2021-05-17 15:33:32] build_node_ancestors
[INFO] [2021-05-17 15:33:32] old ancestors deleted.
[STOP] [2021-05-17 15:33:32] build_node_ancestors
[START] [2021-05-17 15:33:33] Flattener#propagate_ancestor_ids
[STOP] [2021-05-17 15:33:33] Flattener#propagate_ancestor_ids
[STOP] [2021-05-17 15:33:33] Flattener#flatten
[STOP] [2021-05-17 15:33:33] rebuild_nodes
[START] [2021-05-17 15:33:33] resolve_missing_media_owners
[STOP] [2021-05-17 15:33:33] resolve_missing_media_owners
[START] [2021-05-17 15:33:33] sanitize_media_verbatims
[STOP] [2021-05-17 15:33:33] sanitize_media_verbatims
[START] [2021-05-17 15:33:33] queue_downloads
[STOP] [2021-05-17 15:33:33] queue_downloads
[START] [2021-05-17 15:33:33] parse_names
[WARN] [2021-05-17 15:33:33] I see 2197 names which still need to be parsed.
[STOP] [2021-05-17 15:33:35] parse_names
[START] [2021-05-17 15:33:35] denormalize_canonical_names_to_nodes
[STOP] [2021-05-17 15:33:35] denormalize_canonical_names_to_nodes
[START] [2021-05-17 15:33:35] match_nodes
[START] [2021-05-17 15:33:35] map_all_nodes_to_pages
[STOP] [2021-05-17 15:33:56] map_all_nodes_to_pages
[INFO] [2021-05-17 15:33:56] 163 Unmatched nodes (of 2197)! That's too many to output. Full list in /app/public/data/hershkovitz_et_3/unmatched_nodes.txt ; First 10: Canonical: Baetis atrebatinus; Node#94032236; ResourceID: Baetis_atrebatinus; Canonical: Baetis calcaratus; Node#94032243; ResourceID: Baetis_calcaratus; Canonical: Baetis digitatus; Node#94032247; ResourceID: Baetis_digitatus; Canonical: Baetis neglectus; Node#94032264; ResourceID: Baetis_neglectus; Canonical: Baetis niger; Node#94032267; ResourceID: Baetis_niger; Canonical: Baetis strugensis; Node#94032276; ResourceID: Baetis_strugensis; Canonical: Baetis tricolor; Node#94032279; ResourceID: Baetis_tricolor; Canonical: Baetopus tenellus; Node#94032283; ResourceID: Baetopus_tenellus; Canonical: Centroptilum pirinense; Node#94032418; ResourceID: Centroptilum_pirinense; Canonical: Procloeon lacustre; Node#94033629; ResourceID: Procloeon_lacustre
[START] [2021-05-17 15:33:56] update_nodes
[STOP] [2021-05-17 15:33:57] update_nodes
[STOP] [2021-05-17 15:33:57] match_nodes
[START] [2021-05-17 15:33:57] reindex_search
[STOP] [2021-05-17 15:33:59] reindex_search
[START] [2021-05-17 15:33:59] normalize_units
[STOP] [2021-05-17 15:33:59] normalize_units
[START] [2021-05-17 15:33:59] calculate_statistics
[STOP] [2021-05-17 15:33:59] calculate_statistics
[START] [2021-05-17 15:33:59] complete_harvest_instance
[START] [2021-05-17 15:33:59] overall_tsv_creation
[INFO] [2021-05-17 15:33:59] Processing group of 2197 in 1 batches of 10000
[INFO] [2021-05-17 15:34:42] 3880 Traits (unfiltered)...
[INFO] [2021-05-17 15:35:35] 3880 Traits (filtered)...
[INFO] [2021-05-17 15:35:35] 0 Associations (filtered)...
[INFO] [2021-05-17 15:35:36] 1940 metadata added.
[INFO] [2021-05-17 15:35:36] 0 metadata added.
[INFO] [2021-05-17 15:36:03] Average Time: 97.99
[INFO] [2021-05-17 15:36:03] Total Time: 2m5s
[STOP] [2021-05-17 15:36:03] overall_tsv_creation
[INFO] [2021-05-17 15:36:03] Done. Check your files:
[INFO] [2021-05-17 15:36:04] (2197 lines) /app/public/data/hershkovitz_et_3/publish_nodes.tsv
[INFO] [2021-05-17 15:36:04] (6245 lines) /app/public/data/hershkovitz_et_3/publish_node_ancestors.tsv
[INFO] [2021-05-17 15:36:04] (2197 lines) /app/public/data/hershkovitz_et_3/publish_scientific_names.tsv
[INFO] [2021-05-17 15:36:04] (3881 lines) /app/public/data/hershkovitz_et_3/publish_traits.tsv
[INFO] [2021-05-17 15:36:04] (1941 lines) /app/public/data/hershkovitz_et_3/publish_metadata.tsv
[STOP] [2021-05-17 15:36:04] complete_harvest_instance
[START] [2021-05-17 15:36:04] completed
[STOP] [2021-05-17 15:36:04] completed
[STOP] [2021-05-17 15:36:04] logged process, took 168.97
Latest Process

17 May 15:33 EXITED currently processing 0