Harvest for Timor Sea Species List Created 09 Jun 11:18

Harvesting Log

[INFO] [2021-06-09 11:18:07] Created harvest instance #4008
[STOP] [2021-06-09 11:18:07] create_harvest_instance
[START] [2021-06-09 11:18:07] fetch_files
[STOP] [2021-06-09 11:18:07] fetch_files
[START] [2021-06-09 11:18:07] validate_each_file
[INFO] [2021-06-09 11:18:07] Looping over 4 formats...
[INFO] [2021-06-09 11:18:07] ...refs (/app/public/data/timor_sea_sp_lis/tb_references.txt)
[INFO] [2021-06-09 11:18:07] Valid: /app/public/converted_csv/timor_sea_sp_lis_refs_4008.csv (1 lines)
[INFO] [2021-06-09 11:18:07] ...nodes (/app/public/data/timor_sea_sp_lis/tb_taxon.txt)
[INFO] [2021-06-09 11:18:07] Valid: /app/public/converted_csv/timor_sea_sp_lis_nodes_4008.csv (11901 lines)
[INFO] [2021-06-09 11:18:07] ...occurrences (/app/public/data/timor_sea_sp_lis/tb_occurrence.txt)
[INFO] [2021-06-09 11:18:07] Valid: /app/public/converted_csv/timor_sea_sp_lis_occurrences_4008.csv (6991 lines)
[INFO] [2021-06-09 11:18:07] ...measurements (/app/public/data/timor_sea_sp_lis/tb_measurement.txt)
[INFO] [2021-06-09 11:18:08] Valid: /app/public/converted_csv/timor_sea_sp_lis_measurements_4008.csv (13982 lines)
[STOP] [2021-06-09 11:18:08] validate_each_file
[START] [2021-06-09 11:18:08] convert_to_csv
[INFO] [2021-06-09 11:18:08] Looping over 4 formats...
[INFO] [2021-06-09 11:18:08] ...refs (/app/public/data/timor_sea_sp_lis/tb_references.txt)
[CMD] [2021-06-09 11:18:08] /usr/bin/sort /app/public/converted_csv/timor_sea_sp_lis_refs_4008.csv > /app/public/converted_csv/timor_sea_sp_lis_refs_4008.csv_sorted
[INFO] [2021-06-09 11:18:08] Converted: /app/public/converted_csv/timor_sea_sp_lis_refs_4008.csv (1 lines)
[INFO] [2021-06-09 11:18:08] ...nodes (/app/public/data/timor_sea_sp_lis/tb_taxon.txt)
[CMD] [2021-06-09 11:18:08] /usr/bin/sort /app/public/converted_csv/timor_sea_sp_lis_nodes_4008.csv > /app/public/converted_csv/timor_sea_sp_lis_nodes_4008.csv_sorted
[INFO] [2021-06-09 11:18:08] Converted: /app/public/converted_csv/timor_sea_sp_lis_nodes_4008.csv (11901 lines)
[INFO] [2021-06-09 11:18:08] ...occurrences (/app/public/data/timor_sea_sp_lis/tb_occurrence.txt)
[CMD] [2021-06-09 11:18:08] /usr/bin/sort /app/public/converted_csv/timor_sea_sp_lis_occurrences_4008.csv > /app/public/converted_csv/timor_sea_sp_lis_occurrences_4008.csv_sorted
[INFO] [2021-06-09 11:18:08] Converted: /app/public/converted_csv/timor_sea_sp_lis_occurrences_4008.csv (6991 lines)
[INFO] [2021-06-09 11:18:08] ...measurements (/app/public/data/timor_sea_sp_lis/tb_measurement.txt)
[CMD] [2021-06-09 11:18:08] /usr/bin/sort /app/public/converted_csv/timor_sea_sp_lis_measurements_4008.csv > /app/public/converted_csv/timor_sea_sp_lis_measurements_4008.csv_sorted
[INFO] [2021-06-09 11:18:08] Converted: /app/public/converted_csv/timor_sea_sp_lis_measurements_4008.csv (13982 lines)
[STOP] [2021-06-09 11:18:08] convert_to_csv
[START] [2021-06-09 11:18:08] calculate_delta
[INFO] [2021-06-09 11:18:08] Looping over 4 formats...
[INFO] [2021-06-09 11:18:08] ...refs (/app/public/data/timor_sea_sp_lis/tb_references.txt)
[CMD] [2021-06-09 11:18:08] echo "0a" > /app/public/diff/timor_sea_sp_lis_refs_4008.diff
[CMD] [2021-06-09 11:18:09] tail -n +1 /app/public/converted_csv/timor_sea_sp_lis_refs_4008.csv >> /app/public/diff/timor_sea_sp_lis_refs_4008.diff
[CMD] [2021-06-09 11:18:09] echo "." >> /app/public/diff/timor_sea_sp_lis_refs_4008.diff
[INFO] [2021-06-09 11:18:09] Created diff: /app/public/diff/timor_sea_sp_lis_refs_4008.diff (3 lines)
[INFO] [2021-06-09 11:18:09] ...nodes (/app/public/data/timor_sea_sp_lis/tb_taxon.txt)
[CMD] [2021-06-09 11:18:09] echo "0a" > /app/public/diff/timor_sea_sp_lis_nodes_4008.diff
[CMD] [2021-06-09 11:18:09] tail -n +1 /app/public/converted_csv/timor_sea_sp_lis_nodes_4008.csv >> /app/public/diff/timor_sea_sp_lis_nodes_4008.diff
[CMD] [2021-06-09 11:18:09] echo "." >> /app/public/diff/timor_sea_sp_lis_nodes_4008.diff
[INFO] [2021-06-09 11:18:09] Created diff: /app/public/diff/timor_sea_sp_lis_nodes_4008.diff (11903 lines)
[INFO] [2021-06-09 11:18:09] ...occurrences (/app/public/data/timor_sea_sp_lis/tb_occurrence.txt)
[CMD] [2021-06-09 11:18:09] echo "0a" > /app/public/diff/timor_sea_sp_lis_occurrences_4008.diff
[CMD] [2021-06-09 11:18:09] tail -n +1 /app/public/converted_csv/timor_sea_sp_lis_occurrences_4008.csv >> /app/public/diff/timor_sea_sp_lis_occurrences_4008.diff
[CMD] [2021-06-09 11:18:09] echo "." >> /app/public/diff/timor_sea_sp_lis_occurrences_4008.diff
[INFO] [2021-06-09 11:18:09] Created diff: /app/public/diff/timor_sea_sp_lis_occurrences_4008.diff (6993 lines)
[INFO] [2021-06-09 11:18:09] ...measurements (/app/public/data/timor_sea_sp_lis/tb_measurement.txt)
[CMD] [2021-06-09 11:18:09] echo "0a" > /app/public/diff/timor_sea_sp_lis_measurements_4008.diff
[CMD] [2021-06-09 11:18:09] tail -n +1 /app/public/converted_csv/timor_sea_sp_lis_measurements_4008.csv >> /app/public/diff/timor_sea_sp_lis_measurements_4008.diff
[CMD] [2021-06-09 11:18:10] echo "." >> /app/public/diff/timor_sea_sp_lis_measurements_4008.diff
[INFO] [2021-06-09 11:18:10] Created diff: /app/public/diff/timor_sea_sp_lis_measurements_4008.diff (13984 lines)
[STOP] [2021-06-09 11:18:10] calculate_delta
[START] [2021-06-09 11:18:10] parse_diff_and_store
[INFO] [2021-06-09 11:18:10] Handling diff: /app/public/diff/timor_sea_sp_lis_refs_4008.diff (3 lines)
[INFO] [2021-06-09 11:18:10] Loading refs diff file into memory (3 /app/public/diff/timor_sea_sp_lis_refs_4008.diff lines)...
[INFO] [2021-06-09 11:18:10] Handling diff: /app/public/diff/timor_sea_sp_lis_nodes_4008.diff (11903 lines)
[INFO] [2021-06-09 11:18:10] Loading nodes diff file into memory (11903 /app/public/diff/timor_sea_sp_lis_nodes_4008.diff lines)...
[INFO] [2021-06-09 11:18:13] Handling diff: /app/public/diff/timor_sea_sp_lis_occurrences_4008.diff (6993 lines)
[INFO] [2021-06-09 11:18:13] Loading occurrences diff file into memory (6993 /app/public/diff/timor_sea_sp_lis_occurrences_4008.diff lines)...
[INFO] [2021-06-09 11:18:14] Handling diff: /app/public/diff/timor_sea_sp_lis_measurements_4008.diff (13984 lines)
[INFO] [2021-06-09 11:18:14] Loading measurements diff file into memory (13984 /app/public/diff/timor_sea_sp_lis_measurements_4008.diff lines)...
[INFO] [2021-06-09 11:18:19] Storing 1 References
[INFO] [2021-06-09 11:18:19] Processing group of 1 in 1 groups of 1000
[INFO] [2021-06-09 11:18:19] Average Time: 0.0
[INFO] [2021-06-09 11:18:19] Total Time: 1s
[INFO] [2021-06-09 11:18:19] Storing 11901 ScientificNames
[INFO] [2021-06-09 11:18:19] Processing group of 11901 in 12 groups of 1000
[INFO] [2021-06-09 11:18:23] Average Time: 0.298
[INFO] [2021-06-09 11:18:23] Total Time: 4s
[INFO] [2021-06-09 11:18:23] last 3 / first 3: 0.88
[INFO] [2021-06-09 11:18:23] Std.Dev: 0.03162277660168379; Max: 0.36
[INFO] [2021-06-09 11:18:23] Storing 11901 Nodes
[INFO] [2021-06-09 11:18:23] Processing group of 11901 in 12 groups of 1000
[INFO] [2021-06-09 11:18:26] Average Time: 0.263
[INFO] [2021-06-09 11:18:26] Total Time: 4s
[INFO] [2021-06-09 11:18:26] last 3 / first 3: 1.09
[INFO] [2021-06-09 11:18:26] Std.Dev: 0.03162277660168379; Max: 0.33
[INFO] [2021-06-09 11:18:26] Storing 6991 Occurrences
[INFO] [2021-06-09 11:18:26] Processing group of 6991 in 7 groups of 1000
[INFO] [2021-06-09 11:18:27] Average Time: 0.111
[INFO] [2021-06-09 11:18:27] Total Time: 1s
[INFO] [2021-06-09 11:18:27] last 3 / first 3: 1.0
[INFO] [2021-06-09 11:18:27] Std.Dev: 0.0; Max: 0.16
[INFO] [2021-06-09 11:18:27] Storing 13982 Traits
[INFO] [2021-06-09 11:18:27] Processing group of 13982 in 14 groups of 1000
[INFO] [2021-06-09 11:18:31] Average Time: 0.323
[INFO] [2021-06-09 11:18:31] Total Time: 5s
[INFO] [2021-06-09 11:18:31] last 3 / first 3: 1.22
[INFO] [2021-06-09 11:18:31] Std.Dev: 0.044721359549995794; Max: 0.39
[INFO] [2021-06-09 11:18:31] Storing 13982 TraitsReferences
[INFO] [2021-06-09 11:18:31] Processing group of 13982 in 14 groups of 1000
[INFO] [2021-06-09 11:18:32] Average Time: 0.064
[INFO] [2021-06-09 11:18:32] Total Time: 1s
[INFO] [2021-06-09 11:18:32] last 3 / first 3: 1.0
[INFO] [2021-06-09 11:18:32] Std.Dev: 0.0; Max: 0.11
[STOP] [2021-06-09 11:18:32] parse_diff_and_store
[START] [2021-06-09 11:18:32] resolve_keys
[INFO] [2021-06-09 11:18:55] Occurrences to nodes (through scientific_names)...
[INFO] [2021-06-09 11:18:57] traits to occurrences...
[INFO] [2021-06-09 11:19:00] traits to nodes (through occurrences)...
[INFO] [2021-06-09 11:19:00] Traits to sex term...
[INFO] [2021-06-09 11:19:02] Traits to lifestage term...
[INFO] [2021-06-09 11:19:04] MetaTraits to traits...
[INFO] [2021-06-09 11:19:04] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-06-09 11:19:05] Assocs to occurrences...
[INFO] [2021-06-09 11:19:05] Assocs to nodes...
[INFO] [2021-06-09 11:19:05] Assoc to sex term...
[INFO] [2021-06-09 11:19:05] Assoc to lifestage term...
[INFO] [2021-06-09 11:19:05] MetaAssoc to assocs...
[STOP] [2021-06-09 11:19:05] resolve_keys
[START] [2021-06-09 11:19:05] hold_for_later_1
[STOP] [2021-06-09 11:19:05] hold_for_later_1
[START] [2021-06-09 11:19:05] hold_for_later_2
[STOP] [2021-06-09 11:19:05] hold_for_later_2
[START] [2021-06-09 11:19:05] resolve_missing_parents
[STOP] [2021-06-09 11:19:09] resolve_missing_parents
[START] [2021-06-09 11:19:09] rebuild_nodes
[START] [2021-06-09 11:19:09] Flattener#flatten
[START] [2021-06-09 11:19:09] Flattener#study_resource
[START] [2021-06-09 11:19:09] Flattener#build_ancestry
[STOP] [2021-06-09 11:19:10] Flattener#build_ancestry
[INFO] [2021-06-09 11:19:10] 11901 ancestry keys
[START] [2021-06-09 11:19:10] build_node_ancestors
[INFO] [2021-06-09 11:19:10] old ancestors deleted.
[STOP] [2021-06-09 11:19:12] build_node_ancestors
[START] [2021-06-09 11:19:16] Flattener#propagate_ancestor_ids
[STOP] [2021-06-09 11:19:18] Flattener#propagate_ancestor_ids
[STOP] [2021-06-09 11:19:18] Flattener#flatten
[STOP] [2021-06-09 11:19:18] rebuild_nodes
[START] [2021-06-09 11:19:18] resolve_missing_media_owners
[STOP] [2021-06-09 11:19:18] resolve_missing_media_owners
[START] [2021-06-09 11:19:18] sanitize_media_verbatims
[STOP] [2021-06-09 11:19:18] sanitize_media_verbatims
[START] [2021-06-09 11:19:18] queue_downloads
[STOP] [2021-06-09 11:19:18] queue_downloads
[START] [2021-06-09 11:19:18] parse_names
[WARN] [2021-06-09 11:19:18] I see 11901 names which still need to be parsed.
[STOP] [2021-06-09 11:19:27] parse_names
[START] [2021-06-09 11:19:27] denormalize_canonical_names_to_nodes
[STOP] [2021-06-09 11:19:27] denormalize_canonical_names_to_nodes
[START] [2021-06-09 11:19:27] match_nodes
[START] [2021-06-09 11:19:27] map_all_nodes_to_pages
[STOP] [2021-06-09 11:28:44] map_all_nodes_to_pages
[INFO] [2021-06-09 11:28:44] 616 Unmatched nodes (of 11901)! That's too many to output. Full list in /app/public/data/timor_sea_sp_lis/unmatched_nodes.txt ; First 10: Canonical: Limicola; Node#95878475; ResourceID: T100357; Canonical: Limicola falcinellus; Node#95878474; ResourceID: T100356; Canonical: Philomachus; Node#95878636; ResourceID: T100518; Canonical: Philomachus pugnax; Node#95878635; ResourceID: T100517; Canonical: Limnodromus; Node#95879243; ResourceID: T101128; Canonical: Egretta intermedia; Node#95878168; ResourceID: T100050; Canonical: Anas querquedula; Node#95878340; ResourceID: T100222; Canonical: Puffinus pacifica; Node#95883199; ResourceID: T105098; Canonical: Puffinus pacificus; Node#95883266; ResourceID: T105165; Canonical: Eudynamys scolopacea; Node#95882521; ResourceID: T104417
[START] [2021-06-09 11:28:44] update_nodes
[STOP] [2021-06-09 11:28:49] update_nodes
[STOP] [2021-06-09 11:28:49] match_nodes
[START] [2021-06-09 11:28:49] reindex_search
[STOP] [2021-06-09 11:29:00] reindex_search
[START] [2021-06-09 11:29:00] normalize_units
[STOP] [2021-06-09 11:29:00] normalize_units
[START] [2021-06-09 11:29:00] calculate_statistics
[STOP] [2021-06-09 11:29:01] calculate_statistics
[START] [2021-06-09 11:29:01] complete_harvest_instance
[START] [2021-06-09 11:29:01] overall_tsv_creation
[INFO] [2021-06-09 11:29:01] Processing group of 11901 in 2 batches of 10000
[INFO] [2021-06-09 11:30:06] 5674 Traits (unfiltered)...
[INFO] [2021-06-09 11:31:13] 5674 Traits (filtered)...
[INFO] [2021-06-09 11:31:16] 0 Associations (filtered)...
[INFO] [2021-06-09 11:31:18] 5674 metadata added.
[INFO] [2021-06-09 11:31:18] 0 metadata added.
[INFO] [2021-06-09 11:32:27] 1317 Traits (unfiltered)...
[INFO] [2021-06-09 11:33:09] 1317 Traits (filtered)...
[INFO] [2021-06-09 11:33:09] 0 Associations (filtered)...
[INFO] [2021-06-09 11:33:10] 1317 metadata added.
[INFO] [2021-06-09 11:33:10] 0 metadata added.
[INFO] [2021-06-09 11:33:38] Average Time: 112.775
[INFO] [2021-06-09 11:33:38] Total Time: 4m38s
[STOP] [2021-06-09 11:33:38] overall_tsv_creation
[INFO] [2021-06-09 11:33:38] Done. Check your files:
[INFO] [2021-06-09 11:33:38] (11901 lines) /app/public/data/timor_sea_sp_lis/publish_nodes.tsv
[INFO] [2021-06-09 11:33:38] (61965 lines) /app/public/data/timor_sea_sp_lis/publish_node_ancestors.tsv
[INFO] [2021-06-09 11:33:38] (11901 lines) /app/public/data/timor_sea_sp_lis/publish_scientific_names.tsv
[INFO] [2021-06-09 11:33:38] (6992 lines) /app/public/data/timor_sea_sp_lis/publish_traits.tsv
[INFO] [2021-06-09 11:33:38] (6992 lines) /app/public/data/timor_sea_sp_lis/publish_metadata.tsv
[STOP] [2021-06-09 11:33:39] complete_harvest_instance
[START] [2021-06-09 11:33:39] completed
[STOP] [2021-06-09 11:33:39] completed
[STOP] [2021-06-09 11:33:39] logged process, took 931.76

Latest Process

09 Jun 11:18 EXITED currently processing 0