Harvest for Microbe ecomorphological guilds Created 31 Oct 13:55

Stage: completed
Fetched: 31 Oct 13:55
Validated: 31 Oct 13:55
Deltas Created 31 Oct 13:55
Units Normalized: 31 Oct 13:55
Ancestry Built: 31 Oct 13:55
Nodes Matched: 31 Oct 13:55
Names Parsed: 31 Oct 13:55
New Models Stored: 31 Oct 13:55
Indexed: 31 Oct 13:55
Completed: 31 Oct 13:58
Time to Harvest: less than a minute
Harvesting Log

(141 lines)
[INFO] [2022-10-31 13:55:00] Created harvest instance #4228
[STOP] [2022-10-31 13:55:00] create_harvest_instance
[START] [2022-10-31 13:55:00] fetch_files
[STOP] [2022-10-31 13:55:00] fetch_files
[START] [2022-10-31 13:55:00] validate_each_file
[INFO] [2022-10-31 13:55:00] Looping over 3 formats...
[INFO] [2022-10-31 13:55:00] ...nodes (/app/public/data/microbe_ecomorp2/taxon.tab)
[INFO] [2022-10-31 13:55:00] Valid: /app/public/data/microbe_ecomorp2/converted_csv/microbe_ecomorp2_nodes_29860.csv (482 lines)
[INFO] [2022-10-31 13:55:00] ...occurrences (/app/public/data/microbe_ecomorp2/occurrence_specific.tab)
[INFO] [2022-10-31 13:55:00] Valid: /app/public/data/microbe_ecomorp2/converted_csv/microbe_ecomorp2_occurrences_29861.csv (572 lines)
[INFO] [2022-10-31 13:55:00] ...measurements (/app/public/data/microbe_ecomorp2/measurement_or_fact_specific.tab)
[INFO] [2022-10-31 13:55:01] Valid: /app/public/data/microbe_ecomorp2/converted_csv/microbe_ecomorp2_measurements_29862.csv (1003 lines)
[STOP] [2022-10-31 13:55:01] validate_each_file
[START] [2022-10-31 13:55:01] convert_to_csv
[INFO] [2022-10-31 13:55:01] Looping over 3 formats...
[INFO] [2022-10-31 13:55:01] ...nodes (/app/public/data/microbe_ecomorp2/taxon.tab)
[CMD] [2022-10-31 13:55:01] /usr/bin/sort /app/public/data/microbe_ecomorp2/converted_csv/microbe_ecomorp2_nodes_29860.csv > /app/public/data/microbe_ecomorp2/converted_csv/microbe_ecomorp2_nodes_29860.csv_sorted
[INFO] [2022-10-31 13:55:01] Converted: /app/public/data/microbe_ecomorp2/converted_csv/microbe_ecomorp2_nodes_29860.csv (482 lines)
[INFO] [2022-10-31 13:55:01] ...occurrences (/app/public/data/microbe_ecomorp2/occurrence_specific.tab)
[CMD] [2022-10-31 13:55:01] /usr/bin/sort /app/public/data/microbe_ecomorp2/converted_csv/microbe_ecomorp2_occurrences_29861.csv > /app/public/data/microbe_ecomorp2/converted_csv/microbe_ecomorp2_occurrences_29861.csv_sorted
[INFO] [2022-10-31 13:55:01] Converted: /app/public/data/microbe_ecomorp2/converted_csv/microbe_ecomorp2_occurrences_29861.csv (572 lines)
[INFO] [2022-10-31 13:55:01] ...measurements (/app/public/data/microbe_ecomorp2/measurement_or_fact_specific.tab)
[CMD] [2022-10-31 13:55:01] /usr/bin/sort /app/public/data/microbe_ecomorp2/converted_csv/microbe_ecomorp2_measurements_29862.csv > /app/public/data/microbe_ecomorp2/converted_csv/microbe_ecomorp2_measurements_29862.csv_sorted
[INFO] [2022-10-31 13:55:01] Converted: /app/public/data/microbe_ecomorp2/converted_csv/microbe_ecomorp2_measurements_29862.csv (1003 lines)
[STOP] [2022-10-31 13:55:01] convert_to_csv
[START] [2022-10-31 13:55:01] calculate_delta
[INFO] [2022-10-31 13:55:01] Looping over 3 formats...
[INFO] [2022-10-31 13:55:01] ...nodes (/app/public/data/microbe_ecomorp2/taxon.tab)
[CMD] [2022-10-31 13:55:01] echo "0a" > /app/public/data/microbe_ecomorp2/diff/microbe_ecomorp2_nodes_29860.diff
[CMD] [2022-10-31 13:55:01] tail -n +1 /app/public/data/microbe_ecomorp2/converted_csv/microbe_ecomorp2_nodes_29860.csv >> /app/public/data/microbe_ecomorp2/diff/microbe_ecomorp2_nodes_29860.diff
[CMD] [2022-10-31 13:55:01] echo "." >> /app/public/data/microbe_ecomorp2/diff/microbe_ecomorp2_nodes_29860.diff
[INFO] [2022-10-31 13:55:01] Created diff: /app/public/data/microbe_ecomorp2/diff/microbe_ecomorp2_nodes_29860.diff (484 lines)
[INFO] [2022-10-31 13:55:01] ...occurrences (/app/public/data/microbe_ecomorp2/occurrence_specific.tab)
[CMD] [2022-10-31 13:55:01] echo "0a" > /app/public/data/microbe_ecomorp2/diff/microbe_ecomorp2_occurrences_29861.diff
[CMD] [2022-10-31 13:55:01] tail -n +1 /app/public/data/microbe_ecomorp2/converted_csv/microbe_ecomorp2_occurrences_29861.csv >> /app/public/data/microbe_ecomorp2/diff/microbe_ecomorp2_occurrences_29861.diff
[CMD] [2022-10-31 13:55:01] echo "." >> /app/public/data/microbe_ecomorp2/diff/microbe_ecomorp2_occurrences_29861.diff
[INFO] [2022-10-31 13:55:01] Created diff: /app/public/data/microbe_ecomorp2/diff/microbe_ecomorp2_occurrences_29861.diff (574 lines)
[INFO] [2022-10-31 13:55:01] ...measurements (/app/public/data/microbe_ecomorp2/measurement_or_fact_specific.tab)
[CMD] [2022-10-31 13:55:01] echo "0a" > /app/public/data/microbe_ecomorp2/diff/microbe_ecomorp2_measurements_29862.diff
[CMD] [2022-10-31 13:55:01] tail -n +1 /app/public/data/microbe_ecomorp2/converted_csv/microbe_ecomorp2_measurements_29862.csv >> /app/public/data/microbe_ecomorp2/diff/microbe_ecomorp2_measurements_29862.diff
[CMD] [2022-10-31 13:55:01] echo "." >> /app/public/data/microbe_ecomorp2/diff/microbe_ecomorp2_measurements_29862.diff
[INFO] [2022-10-31 13:55:01] Created diff: /app/public/data/microbe_ecomorp2/diff/microbe_ecomorp2_measurements_29862.diff (1005 lines)
[STOP] [2022-10-31 13:55:01] calculate_delta
[START] [2022-10-31 13:55:01] parse_diff_and_store
[INFO] [2022-10-31 13:55:01] Handling diff: /app/public/data/microbe_ecomorp2/diff/microbe_ecomorp2_nodes_29860.diff (484 lines)
[INFO] [2022-10-31 13:55:01] Loading nodes diff file into memory (484 lines)...
[INFO] [2022-10-31 13:55:01] Storing 482 ScientificNames (964/482/484)
[INFO] [2022-10-31 13:55:01] Storing 482 Nodes (964/482/484)
[INFO] [2022-10-31 13:55:01] Handling diff: /app/public/data/microbe_ecomorp2/diff/microbe_ecomorp2_occurrences_29861.diff (574 lines)
[INFO] [2022-10-31 13:55:01] Loading occurrences diff file into memory (574 lines)...
[INFO] [2022-10-31 13:55:01] Storing 572 Occurrences (576/572/574)
[INFO] [2022-10-31 13:55:02] Storing 4 OccurrenceMetadata (576/572/574)
[INFO] [2022-10-31 13:55:02] Handling diff: /app/public/data/microbe_ecomorp2/diff/microbe_ecomorp2_measurements_29862.diff (1005 lines)
[INFO] [2022-10-31 13:55:02] Loading measurements diff file into memory (1005 lines)...
[INFO] [2022-10-31 13:55:02] Storing 1003 Traits (1575/1003/1005)
[INFO] [2022-10-31 13:55:02] Storing 572 MetaTraits (1575/1003/1005)
[STOP] [2022-10-31 13:55:02] parse_diff_and_store
[START] [2022-10-31 13:55:02] resolve_keys
[2022-10-31 13:55:03] Resolving downloaded urls (this is not actually downloading them yet)
[INFO] [2022-10-31 13:55:10] Occurrences to nodes (through scientific_names)...
[INFO] [2022-10-31 13:55:10] traits to occurrences...
[INFO] [2022-10-31 13:55:10] traits to nodes (through occurrences)...
[INFO] [2022-10-31 13:55:10] Traits to sex term...
[INFO] [2022-10-31 13:55:10] Traits to lifestage term...
[INFO] [2022-10-31 13:55:10] MetaTraits to traits...
[INFO] [2022-10-31 13:55:10] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2022-10-31 13:55:10] Assocs to occurrences...
[INFO] [2022-10-31 13:55:10] Assocs to nodes...
[INFO] [2022-10-31 13:55:10] Assoc to sex term...
[INFO] [2022-10-31 13:55:10] Assoc to lifestage term...
[INFO] [2022-10-31 13:55:10] MetaAssoc to assocs...
[STOP] [2022-10-31 13:55:10] resolve_keys
[START] [2022-10-31 13:55:10] hold_for_later_1
[STOP] [2022-10-31 13:55:10] hold_for_later_1
[START] [2022-10-31 13:55:10] hold_for_later_2
[STOP] [2022-10-31 13:55:10] hold_for_later_2
[START] [2022-10-31 13:55:10] resolve_missing_parents
[STOP] [2022-10-31 13:55:10] resolve_missing_parents
[START] [2022-10-31 13:55:10] rebuild_nodes
[START] [2022-10-31 13:55:10] Flattener#flatten
[START] [2022-10-31 13:55:10] Flattener#study_resource
[START] [2022-10-31 13:55:10] Flattener#build_ancestry
[STOP] [2022-10-31 13:55:10] Flattener#build_ancestry
[INFO] [2022-10-31 13:55:10] 482 ancestry keys
[START] [2022-10-31 13:55:10] build_node_ancestors
[INFO] [2022-10-31 13:55:10] old ancestors deleted.
[STOP] [2022-10-31 13:55:10] build_node_ancestors
[WARN] [2022-10-31 13:55:10] Flattener: nothing to flatten! (Completely flat resource?)
[STOP] [2022-10-31 13:55:10] Flattener#flatten
[STOP] [2022-10-31 13:55:10] rebuild_nodes
[START] [2022-10-31 13:55:10] resolve_missing_media_owners
[STOP] [2022-10-31 13:55:10] resolve_missing_media_owners
[START] [2022-10-31 13:55:10] sanitize_media_verbatims
[STOP] [2022-10-31 13:55:10] sanitize_media_verbatims
[START] [2022-10-31 13:55:10] queue_downloads
[STOP] [2022-10-31 13:55:10] queue_downloads
[START] [2022-10-31 13:55:10] parse_names
[WARN] [2022-10-31 13:55:10] I see 482 names which still need to be parsed.
[WARN] [2022-10-31 13:55:10] Names to parse: 482 formatted: 482 learned: 482 parsed: 482
[STOP] [2022-10-31 13:55:12] parse_names
[START] [2022-10-31 13:55:12] denormalize_canonical_names_to_nodes
[STOP] [2022-10-31 13:55:12] denormalize_canonical_names_to_nodes
[START] [2022-10-31 13:55:12] match_nodes
[START] [2022-10-31 13:55:12] map_all_nodes_to_pages
[STOP] [2022-10-31 13:55:12] map_all_nodes_to_pages
[INFO] [2022-10-31 13:55:12] ZERO unmatched nodes (of 482)! Nicely done.
[START] [2022-10-31 13:55:12] update_nodes
[STOP] [2022-10-31 13:55:12] update_nodes
[STOP] [2022-10-31 13:55:12] match_nodes
[START] [2022-10-31 13:55:12] reindex_search
[STOP] [2022-10-31 13:55:13] reindex_search
[START] [2022-10-31 13:55:13] normalize_units
[STOP] [2022-10-31 13:55:13] normalize_units
[START] [2022-10-31 13:55:13] calculate_statistics
[2022-10-31 13:55:13] ZERO NODE ANCESTORS. Is this actually a completely flat resource?
[INFO] [2022-10-31 13:55:21] Duplicate page_id count: 0
[STOP] [2022-10-31 13:55:21] calculate_statistics
[START] [2022-10-31 13:55:21] complete_harvest_instance
[START] [2022-10-31 13:55:21] overall_tsv_creation
[INFO] [2022-10-31 13:55:21] Processing group of 482 in 1 batches of 10000
[INFO] [2022-10-31 13:56:17] 572 Traits (unfiltered)...
[INFO] [2022-10-31 13:56:17] Building Traits map (this can take a while)...
[INFO] [2022-10-31 13:57:21] Done. 572 traits mapped (572 meta).
[INFO] [2022-10-31 13:57:21] Building Associations map (this can take a while)...
[INFO] [2022-10-31 13:57:21] Done. 0 assocs mapped (0 meta).
[INFO] [2022-10-31 13:57:21] Adding 572 traits...
[INFO] [2022-10-31 13:57:21] 431 metadata added.
[INFO] [2022-10-31 13:57:21] Adding 0 assocs...
[INFO] [2022-10-31 13:57:21] 0 metadata added.
[INFO] [2022-10-31 13:58:11] Average Time: 142.01
[INFO] [2022-10-31 13:58:11] Total Time: 2m50s
[STOP] [2022-10-31 13:58:11] overall_tsv_creation
[INFO] [2022-10-31 13:58:11] Done. Check your files:
[INFO] [2022-10-31 13:58:11] (482 lines) /app/public/data/microbe_ecomorp2/publish_nodes.tsv
[INFO] [2022-10-31 13:58:11] (482 lines) /app/public/data/microbe_ecomorp2/publish_scientific_names.tsv
[INFO] [2022-10-31 13:58:11] (573 lines) /app/public/data/microbe_ecomorp2/publish_traits.tsv
[INFO] [2022-10-31 13:58:11] (432 lines) /app/public/data/microbe_ecomorp2/publish_metadata.tsv
[STOP] [2022-10-31 13:58:11] complete_harvest_instance
[START] [2022-10-31 13:58:11] completed
[STOP] [2022-10-31 13:58:11] completed
[STOP] [2022-10-31 13:58:11] logged process, took 190.65
Latest Process

31 Oct 13:55 EXITED currently processing 0