Harvest for Braconid wasps caterpillars and biocontrol Created 04 Jun 14:27

Stage: completed
Fetched: 04 Jun 14:27
Validated: 04 Jun 14:27
Deltas Created 04 Jun 14:27
Units Normalized: 04 Jun 14:28
Ancestry Built: 04 Jun 14:28
Nodes Matched: 04 Jun 14:28
Names Parsed: 04 Jun 14:28
New Models Stored: 04 Jun 14:27
Indexed: 04 Jun 14:28
Completed: 04 Jun 14:30
Time to Harvest: less than a minute
Harvesting Log

(179 lines)
[INFO] [2021-06-04 14:27:50] Created harvest instance #3991
[STOP] [2021-06-04 14:27:50] create_harvest_instance
[START] [2021-06-04 14:27:50] fetch_files
[STOP] [2021-06-04 14:27:50] fetch_files
[START] [2021-06-04 14:27:50] validate_each_file
[INFO] [2021-06-04 14:27:50] Looping over 4 formats...
[INFO] [2021-06-04 14:27:50] ...agents (/app/public/data/braconid_wasps_2/agent.tab)
[INFO] [2021-06-04 14:27:50] Valid: /app/public/converted_csv/braconid_wasps_2_agents_3991.csv (2 lines)
[INFO] [2021-06-04 14:27:50] ...refs (/app/public/data/braconid_wasps_2/reference.tab)
[INFO] [2021-06-04 14:27:50] Valid: /app/public/converted_csv/braconid_wasps_2_refs_3991.csv (2 lines)
[INFO] [2021-06-04 14:27:50] ...nodes (/app/public/data/braconid_wasps_2/taxon.tab)
[INFO] [2021-06-04 14:27:50] Valid: /app/public/converted_csv/braconid_wasps_2_nodes_3991.csv (14 lines)
[INFO] [2021-06-04 14:27:50] ...media (/app/public/data/braconid_wasps_2/media_resource.tab)
[INFO] [2021-06-04 14:27:50] Valid: /app/public/converted_csv/braconid_wasps_2_media_3991.csv (44 lines)
[STOP] [2021-06-04 14:27:50] validate_each_file
[START] [2021-06-04 14:27:50] convert_to_csv
[INFO] [2021-06-04 14:27:50] Looping over 4 formats...
[INFO] [2021-06-04 14:27:50] ...agents (/app/public/data/braconid_wasps_2/agent.tab)
[CMD] [2021-06-04 14:27:50] /usr/bin/sort /app/public/converted_csv/braconid_wasps_2_agents_3991.csv > /app/public/converted_csv/braconid_wasps_2_agents_3991.csv_sorted
[INFO] [2021-06-04 14:27:50] Converted: /app/public/converted_csv/braconid_wasps_2_agents_3991.csv (2 lines)
[INFO] [2021-06-04 14:27:50] ...refs (/app/public/data/braconid_wasps_2/reference.tab)
[CMD] [2021-06-04 14:27:50] /usr/bin/sort /app/public/converted_csv/braconid_wasps_2_refs_3991.csv > /app/public/converted_csv/braconid_wasps_2_refs_3991.csv_sorted
[INFO] [2021-06-04 14:27:50] Converted: /app/public/converted_csv/braconid_wasps_2_refs_3991.csv (2 lines)
[INFO] [2021-06-04 14:27:50] ...nodes (/app/public/data/braconid_wasps_2/taxon.tab)
[CMD] [2021-06-04 14:27:50] /usr/bin/sort /app/public/converted_csv/braconid_wasps_2_nodes_3991.csv > /app/public/converted_csv/braconid_wasps_2_nodes_3991.csv_sorted
[INFO] [2021-06-04 14:27:50] Converted: /app/public/converted_csv/braconid_wasps_2_nodes_3991.csv (14 lines)
[INFO] [2021-06-04 14:27:50] ...media (/app/public/data/braconid_wasps_2/media_resource.tab)
[CMD] [2021-06-04 14:27:50] /usr/bin/sort /app/public/converted_csv/braconid_wasps_2_media_3991.csv > /app/public/converted_csv/braconid_wasps_2_media_3991.csv_sorted
[INFO] [2021-06-04 14:27:50] Converted: /app/public/converted_csv/braconid_wasps_2_media_3991.csv (44 lines)
[STOP] [2021-06-04 14:27:50] convert_to_csv
[START] [2021-06-04 14:27:50] calculate_delta
[INFO] [2021-06-04 14:27:50] Looping over 4 formats...
[INFO] [2021-06-04 14:27:50] ...agents (/app/public/data/braconid_wasps_2/agent.tab)
[CMD] [2021-06-04 14:27:50] echo "0a" > /app/public/diff/braconid_wasps_2_agents_3991.diff
[CMD] [2021-06-04 14:27:50] tail -n +1 /app/public/converted_csv/braconid_wasps_2_agents_3991.csv >> /app/public/diff/braconid_wasps_2_agents_3991.diff
[CMD] [2021-06-04 14:27:50] echo "." >> /app/public/diff/braconid_wasps_2_agents_3991.diff
[INFO] [2021-06-04 14:27:50] Created diff: /app/public/diff/braconid_wasps_2_agents_3991.diff (4 lines)
[INFO] [2021-06-04 14:27:50] ...refs (/app/public/data/braconid_wasps_2/reference.tab)
[CMD] [2021-06-04 14:27:50] echo "0a" > /app/public/diff/braconid_wasps_2_refs_3991.diff
[CMD] [2021-06-04 14:27:50] tail -n +1 /app/public/converted_csv/braconid_wasps_2_refs_3991.csv >> /app/public/diff/braconid_wasps_2_refs_3991.diff
[CMD] [2021-06-04 14:27:50] echo "." >> /app/public/diff/braconid_wasps_2_refs_3991.diff
[INFO] [2021-06-04 14:27:50] Created diff: /app/public/diff/braconid_wasps_2_refs_3991.diff (4 lines)
[INFO] [2021-06-04 14:27:50] ...nodes (/app/public/data/braconid_wasps_2/taxon.tab)
[CMD] [2021-06-04 14:27:50] echo "0a" > /app/public/diff/braconid_wasps_2_nodes_3991.diff
[CMD] [2021-06-04 14:27:50] tail -n +1 /app/public/converted_csv/braconid_wasps_2_nodes_3991.csv >> /app/public/diff/braconid_wasps_2_nodes_3991.diff
[CMD] [2021-06-04 14:27:50] echo "." >> /app/public/diff/braconid_wasps_2_nodes_3991.diff
[INFO] [2021-06-04 14:27:50] Created diff: /app/public/diff/braconid_wasps_2_nodes_3991.diff (16 lines)
[INFO] [2021-06-04 14:27:50] ...media (/app/public/data/braconid_wasps_2/media_resource.tab)
[CMD] [2021-06-04 14:27:50] echo "0a" > /app/public/diff/braconid_wasps_2_media_3991.diff
[CMD] [2021-06-04 14:27:50] tail -n +1 /app/public/converted_csv/braconid_wasps_2_media_3991.csv >> /app/public/diff/braconid_wasps_2_media_3991.diff
[CMD] [2021-06-04 14:27:50] echo "." >> /app/public/diff/braconid_wasps_2_media_3991.diff
[INFO] [2021-06-04 14:27:50] Created diff: /app/public/diff/braconid_wasps_2_media_3991.diff (46 lines)
[STOP] [2021-06-04 14:27:50] calculate_delta
[START] [2021-06-04 14:27:50] parse_diff_and_store
[INFO] [2021-06-04 14:27:50] Handling diff: /app/public/diff/braconid_wasps_2_agents_3991.diff (4 lines)
[INFO] [2021-06-04 14:27:50] Loading agents diff file into memory (4 /app/public/diff/braconid_wasps_2_agents_3991.diff lines)...
[INFO] [2021-06-04 14:27:50] Handling diff: /app/public/diff/braconid_wasps_2_refs_3991.diff (4 lines)
[INFO] [2021-06-04 14:27:50] Loading refs diff file into memory (4 /app/public/diff/braconid_wasps_2_refs_3991.diff lines)...
[INFO] [2021-06-04 14:27:50] Handling diff: /app/public/diff/braconid_wasps_2_nodes_3991.diff (16 lines)
[INFO] [2021-06-04 14:27:50] Loading nodes diff file into memory (16 /app/public/diff/braconid_wasps_2_nodes_3991.diff lines)...
[INFO] [2021-06-04 14:27:50] Handling diff: /app/public/diff/braconid_wasps_2_media_3991.diff (46 lines)
[INFO] [2021-06-04 14:27:50] Loading media diff file into memory (46 /app/public/diff/braconid_wasps_2_media_3991.diff lines)...
[INFO] [2021-06-04 14:27:50] Storing 2 Attributions
[INFO] [2021-06-04 14:27:50] Processing group of 2 in 1 groups of 1000
[INFO] [2021-06-04 14:27:50] Average Time: 0.01
[INFO] [2021-06-04 14:27:50] Total Time: 1s
[INFO] [2021-06-04 14:27:50] Storing 2 References
[INFO] [2021-06-04 14:27:50] Processing group of 2 in 1 groups of 1000
[INFO] [2021-06-04 14:27:50] Average Time: 0.0
[INFO] [2021-06-04 14:27:50] Total Time: 1s
[INFO] [2021-06-04 14:27:50] Storing 27 ScientificNames
[INFO] [2021-06-04 14:27:50] Processing group of 27 in 1 groups of 1000
[INFO] [2021-06-04 14:27:50] Average Time: 0.01
[INFO] [2021-06-04 14:27:50] Total Time: 1s
[INFO] [2021-06-04 14:27:50] Storing 27 Nodes
[INFO] [2021-06-04 14:27:50] Processing group of 27 in 1 groups of 1000
[INFO] [2021-06-04 14:27:50] Average Time: 0.01
[INFO] [2021-06-04 14:27:50] Total Time: 1s
[INFO] [2021-06-04 14:27:50] Storing 1 MediaReferences
[INFO] [2021-06-04 14:27:50] Processing group of 1 in 1 groups of 1000
[INFO] [2021-06-04 14:27:50] Average Time: 0.02
[INFO] [2021-06-04 14:27:50] Total Time: 1s
[INFO] [2021-06-04 14:27:50] Storing 88 ContentAttributions
[INFO] [2021-06-04 14:27:50] Processing group of 88 in 1 groups of 1000
[INFO] [2021-06-04 14:27:51] Average Time: 0.1
[INFO] [2021-06-04 14:27:51] Total Time: 1s
[INFO] [2021-06-04 14:27:51] Storing 43 Media
[INFO] [2021-06-04 14:27:51] Processing group of 43 in 1 groups of 1000
[INFO] [2021-06-04 14:27:51] Average Time: 0.04
[INFO] [2021-06-04 14:27:51] Total Time: 1s
[INFO] [2021-06-04 14:27:51] Storing 1 ArticlesReferences
[INFO] [2021-06-04 14:27:51] Processing group of 1 in 1 groups of 1000
[INFO] [2021-06-04 14:27:51] Average Time: 0.04
[INFO] [2021-06-04 14:27:51] Total Time: 1s
[INFO] [2021-06-04 14:27:51] Storing 1 ArticlesSections
[INFO] [2021-06-04 14:27:51] Processing group of 1 in 1 groups of 1000
[INFO] [2021-06-04 14:27:51] Average Time: 0.02
[INFO] [2021-06-04 14:27:51] Total Time: 1s
[INFO] [2021-06-04 14:27:51] Storing 1 Articles
[INFO] [2021-06-04 14:27:51] Processing group of 1 in 1 groups of 1000
[INFO] [2021-06-04 14:27:51] Average Time: 0.01
[INFO] [2021-06-04 14:27:51] Total Time: 1s
[STOP] [2021-06-04 14:27:51] parse_diff_and_store
[START] [2021-06-04 14:27:51] resolve_keys
[INFO] [2021-06-04 14:28:14] Occurrences to nodes (through scientific_names)...
[INFO] [2021-06-04 14:28:14] traits to occurrences...
[INFO] [2021-06-04 14:28:14] traits to nodes (through occurrences)...
[INFO] [2021-06-04 14:28:14] Traits to sex term...
[INFO] [2021-06-04 14:28:14] Traits to lifestage term...
[INFO] [2021-06-04 14:28:14] MetaTraits to traits...
[INFO] [2021-06-04 14:28:14] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-06-04 14:28:14] Assocs to occurrences...
[INFO] [2021-06-04 14:28:14] Assocs to nodes...
[INFO] [2021-06-04 14:28:14] Assoc to sex term...
[INFO] [2021-06-04 14:28:14] Assoc to lifestage term...
[INFO] [2021-06-04 14:28:14] MetaAssoc to assocs...
[STOP] [2021-06-04 14:28:14] resolve_keys
[START] [2021-06-04 14:28:14] hold_for_later_1
[STOP] [2021-06-04 14:28:14] hold_for_later_1
[START] [2021-06-04 14:28:14] hold_for_later_2
[STOP] [2021-06-04 14:28:14] hold_for_later_2
[START] [2021-06-04 14:28:14] resolve_missing_parents
[STOP] [2021-06-04 14:28:14] resolve_missing_parents
[START] [2021-06-04 14:28:14] rebuild_nodes
[START] [2021-06-04 14:28:14] Flattener#flatten
[START] [2021-06-04 14:28:14] Flattener#study_resource
[START] [2021-06-04 14:28:14] Flattener#build_ancestry
[STOP] [2021-06-04 14:28:14] Flattener#build_ancestry
[INFO] [2021-06-04 14:28:14] 27 ancestry keys
[START] [2021-06-04 14:28:14] build_node_ancestors
[INFO] [2021-06-04 14:28:14] old ancestors deleted.
[STOP] [2021-06-04 14:28:14] build_node_ancestors
[START] [2021-06-04 14:28:14] Flattener#propagate_ancestor_ids
[STOP] [2021-06-04 14:28:14] Flattener#propagate_ancestor_ids
[STOP] [2021-06-04 14:28:14] Flattener#flatten
[STOP] [2021-06-04 14:28:14] rebuild_nodes
[START] [2021-06-04 14:28:14] resolve_missing_media_owners
[STOP] [2021-06-04 14:28:14] resolve_missing_media_owners
[START] [2021-06-04 14:28:14] sanitize_media_verbatims
[STOP] [2021-06-04 14:28:14] sanitize_media_verbatims
[START] [2021-06-04 14:28:14] queue_downloads
[STOP] [2021-06-04 14:28:14] queue_downloads
[START] [2021-06-04 14:28:14] parse_names
[WARN] [2021-06-04 14:28:14] I see 27 names which still need to be parsed.
[STOP] [2021-06-04 14:28:15] parse_names
[START] [2021-06-04 14:28:15] denormalize_canonical_names_to_nodes
[STOP] [2021-06-04 14:28:15] denormalize_canonical_names_to_nodes
[START] [2021-06-04 14:28:15] match_nodes
[START] [2021-06-04 14:28:15] map_all_nodes_to_pages
[STOP] [2021-06-04 14:28:19] map_all_nodes_to_pages
[INFO] [2021-06-04 14:28:19] Unmatched nodes (6 of 27): Canonical: Apanteles samarshalli; Node#95616630; ResourceID: Apanteles samarshalli; Canonical: Dolichogenidea; Node#95616635; ResourceID: Dolichogenidea; Canonical: Dolichogenidea clavata; Node#95616636; ResourceID: Dolichogenidea clavata; Canonical: Glyptapanteles; Node#95616637; ResourceID: Glyptapanteles; Canonical: Glyptapanteles compressiventris; Node#95616638; ResourceID: Glyptapanteles compressiventris; Canonical: Protapanteles alaskensis; Node#95616649; ResourceID: Protapanteles alaskensis
[START] [2021-06-04 14:28:19] update_nodes
[STOP] [2021-06-04 14:28:19] update_nodes
[STOP] [2021-06-04 14:28:19] match_nodes
[START] [2021-06-04 14:28:19] reindex_search
[STOP] [2021-06-04 14:28:19] reindex_search
[START] [2021-06-04 14:28:19] normalize_units
[STOP] [2021-06-04 14:28:19] normalize_units
[START] [2021-06-04 14:28:19] calculate_statistics
[STOP] [2021-06-04 14:28:19] calculate_statistics
[START] [2021-06-04 14:28:19] complete_harvest_instance
[START] [2021-06-04 14:28:19] overall_tsv_creation
[INFO] [2021-06-04 14:28:19] Processing group of 27 in 1 batches of 10000
[INFO] [2021-06-04 14:30:38] Average Time: 7.7
[INFO] [2021-06-04 14:30:38] Total Time: 2m20s
[STOP] [2021-06-04 14:30:38] overall_tsv_creation
[INFO] [2021-06-04 14:30:38] Done. Check your files:
[INFO] [2021-06-04 14:30:38] (27 lines) /app/public/data/braconid_wasps_2/publish_nodes.tsv
[INFO] [2021-06-04 14:30:38] (134 lines) /app/public/data/braconid_wasps_2/publish_node_ancestors.tsv
[INFO] [2021-06-04 14:30:38] (27 lines) /app/public/data/braconid_wasps_2/publish_scientific_names.tsv
[INFO] [2021-06-04 14:30:38] (43 lines) /app/public/data/braconid_wasps_2/publish_media.tsv
[INFO] [2021-06-04 14:30:38] (1 lines) /app/public/data/braconid_wasps_2/publish_articles.tsv
[INFO] [2021-06-04 14:30:38] (19 lines) /app/public/data/braconid_wasps_2/publish_image_info.tsv
[INFO] [2021-06-04 14:30:38] (88 lines) /app/public/data/braconid_wasps_2/publish_attributions.tsv
[INFO] [2021-06-04 14:30:38] (1 lines) /app/public/data/braconid_wasps_2/publish_content_sections.tsv
[STOP] [2021-06-04 14:30:38] complete_harvest_instance
[START] [2021-06-04 14:30:38] completed
[STOP] [2021-06-04 14:30:38] completed
[STOP] [2021-06-04 14:30:38] logged process, took 168.78
Latest Process

04 Jun 14:27 EXITED currently processing 0