Harvest for Kāneʻohe Bay Bioblitz Created 31 May 14:35

Harvesting Log

[INFO] [2021-05-31 14:35:37] Created harvest instance #3938
[STOP] [2021-05-31 14:35:37] create_harvest_instance
[START] [2021-05-31 14:35:37] fetch_files
[STOP] [2021-05-31 14:35:37] fetch_files
[START] [2021-05-31 14:35:37] validate_each_file
[INFO] [2021-05-31 14:35:37] Looping over 8 formats...
[INFO] [2021-05-31 14:35:37] ...agents (/app/public/data/kaneohe_bay_biob/agents.txt)
[INFO] [2021-05-31 14:35:37] Valid: /app/public/converted_csv/kaneohe_bay_biob_agents_3938.csv (0 lines)
[INFO] [2021-05-31 14:35:37] ...refs (/app/public/data/kaneohe_bay_biob/references.txt)
[INFO] [2021-05-31 14:35:37] Valid: /app/public/converted_csv/kaneohe_bay_biob_refs_3938.csv (0 lines)
[INFO] [2021-05-31 14:35:37] ...nodes (/app/public/data/kaneohe_bay_biob/taxa.txt)
[INFO] [2021-05-31 14:35:37] Valid: /app/public/converted_csv/kaneohe_bay_biob_nodes_3938.csv (95 lines)
[INFO] [2021-05-31 14:35:37] ...media (/app/public/data/kaneohe_bay_biob/media.txt)
[INFO] [2021-05-31 14:35:37] Valid: /app/public/converted_csv/kaneohe_bay_biob_media_3938.csv (0 lines)
[INFO] [2021-05-31 14:35:37] ...vernaculars (/app/public/data/kaneohe_bay_biob/common names.txt)
[INFO] [2021-05-31 14:35:37] Valid: /app/public/converted_csv/kaneohe_bay_biob_vernaculars_3938.csv (0 lines)
[INFO] [2021-05-31 14:35:37] ...occurrences (/app/public/data/kaneohe_bay_biob/occurrences.txt)
[INFO] [2021-05-31 14:35:37] Valid: /app/public/converted_csv/kaneohe_bay_biob_occurrences_3938.csv (152 lines)
[INFO] [2021-05-31 14:35:37] ...assocs (/app/public/data/kaneohe_bay_biob/associations.txt)
[INFO] [2021-05-31 14:35:37] Valid: /app/public/converted_csv/kaneohe_bay_biob_assocs_3938.csv (0 lines)
[INFO] [2021-05-31 14:35:37] ...measurements (/app/public/data/kaneohe_bay_biob/measurements or facts.txt)
[INFO] [2021-05-31 14:35:37] Valid: /app/public/converted_csv/kaneohe_bay_biob_measurements_3938.csv (300 lines)
[STOP] [2021-05-31 14:35:37] validate_each_file
[START] [2021-05-31 14:35:37] convert_to_csv
[INFO] [2021-05-31 14:35:37] Looping over 8 formats...
[INFO] [2021-05-31 14:35:37] ...agents (/app/public/data/kaneohe_bay_biob/agents.txt)
[CMD] [2021-05-31 14:35:37] /usr/bin/sort /app/public/converted_csv/kaneohe_bay_biob_agents_3938.csv > /app/public/converted_csv/kaneohe_bay_biob_agents_3938.csv_sorted
[INFO] [2021-05-31 14:35:37] Converted: /app/public/converted_csv/kaneohe_bay_biob_agents_3938.csv (0 lines)
[INFO] [2021-05-31 14:35:37] ...refs (/app/public/data/kaneohe_bay_biob/references.txt)
[CMD] [2021-05-31 14:35:37] /usr/bin/sort /app/public/converted_csv/kaneohe_bay_biob_refs_3938.csv > /app/public/converted_csv/kaneohe_bay_biob_refs_3938.csv_sorted
[INFO] [2021-05-31 14:35:37] Converted: /app/public/converted_csv/kaneohe_bay_biob_refs_3938.csv (0 lines)
[INFO] [2021-05-31 14:35:37] ...nodes (/app/public/data/kaneohe_bay_biob/taxa.txt)
[CMD] [2021-05-31 14:35:37] /usr/bin/sort /app/public/converted_csv/kaneohe_bay_biob_nodes_3938.csv > /app/public/converted_csv/kaneohe_bay_biob_nodes_3938.csv_sorted
[INFO] [2021-05-31 14:35:38] Converted: /app/public/converted_csv/kaneohe_bay_biob_nodes_3938.csv (95 lines)
[INFO] [2021-05-31 14:35:38] ...media (/app/public/data/kaneohe_bay_biob/media.txt)
[CMD] [2021-05-31 14:35:38] /usr/bin/sort /app/public/converted_csv/kaneohe_bay_biob_media_3938.csv > /app/public/converted_csv/kaneohe_bay_biob_media_3938.csv_sorted
[INFO] [2021-05-31 14:35:38] Converted: /app/public/converted_csv/kaneohe_bay_biob_media_3938.csv (0 lines)
[INFO] [2021-05-31 14:35:38] ...vernaculars (/app/public/data/kaneohe_bay_biob/common names.txt)
[CMD] [2021-05-31 14:35:38] /usr/bin/sort /app/public/converted_csv/kaneohe_bay_biob_vernaculars_3938.csv > /app/public/converted_csv/kaneohe_bay_biob_vernaculars_3938.csv_sorted
[INFO] [2021-05-31 14:35:39] Converted: /app/public/converted_csv/kaneohe_bay_biob_vernaculars_3938.csv (0 lines)
[INFO] [2021-05-31 14:35:39] ...occurrences (/app/public/data/kaneohe_bay_biob/occurrences.txt)
[CMD] [2021-05-31 14:35:39] /usr/bin/sort /app/public/converted_csv/kaneohe_bay_biob_occurrences_3938.csv > /app/public/converted_csv/kaneohe_bay_biob_occurrences_3938.csv_sorted
[INFO] [2021-05-31 14:35:39] Converted: /app/public/converted_csv/kaneohe_bay_biob_occurrences_3938.csv (152 lines)
[INFO] [2021-05-31 14:35:39] ...assocs (/app/public/data/kaneohe_bay_biob/associations.txt)
[CMD] [2021-05-31 14:35:39] /usr/bin/sort /app/public/converted_csv/kaneohe_bay_biob_assocs_3938.csv > /app/public/converted_csv/kaneohe_bay_biob_assocs_3938.csv_sorted
[INFO] [2021-05-31 14:35:39] Converted: /app/public/converted_csv/kaneohe_bay_biob_assocs_3938.csv (0 lines)
[INFO] [2021-05-31 14:35:39] ...measurements (/app/public/data/kaneohe_bay_biob/measurements or facts.txt)
[CMD] [2021-05-31 14:35:39] /usr/bin/sort /app/public/converted_csv/kaneohe_bay_biob_measurements_3938.csv > /app/public/converted_csv/kaneohe_bay_biob_measurements_3938.csv_sorted
[INFO] [2021-05-31 14:35:40] Converted: /app/public/converted_csv/kaneohe_bay_biob_measurements_3938.csv (300 lines)
[STOP] [2021-05-31 14:35:40] convert_to_csv
[START] [2021-05-31 14:35:40] calculate_delta
[INFO] [2021-05-31 14:35:40] Looping over 8 formats...
[INFO] [2021-05-31 14:35:40] ...agents (/app/public/data/kaneohe_bay_biob/agents.txt)
[CMD] [2021-05-31 14:35:40] echo "0a" > /app/public/diff/kaneohe_bay_biob_agents_3938.diff
[CMD] [2021-05-31 14:35:40] tail -n +1 /app/public/converted_csv/kaneohe_bay_biob_agents_3938.csv >> /app/public/diff/kaneohe_bay_biob_agents_3938.diff
[CMD] [2021-05-31 14:35:41] echo "." >> /app/public/diff/kaneohe_bay_biob_agents_3938.diff
[INFO] [2021-05-31 14:35:41] Created diff: /app/public/diff/kaneohe_bay_biob_agents_3938.diff (2 lines)
[INFO] [2021-05-31 14:35:41] ...refs (/app/public/data/kaneohe_bay_biob/references.txt)
[CMD] [2021-05-31 14:35:41] echo "0a" > /app/public/diff/kaneohe_bay_biob_refs_3938.diff
[CMD] [2021-05-31 14:35:41] tail -n +1 /app/public/converted_csv/kaneohe_bay_biob_refs_3938.csv >> /app/public/diff/kaneohe_bay_biob_refs_3938.diff
[CMD] [2021-05-31 14:35:42] echo "." >> /app/public/diff/kaneohe_bay_biob_refs_3938.diff
[INFO] [2021-05-31 14:35:42] Created diff: /app/public/diff/kaneohe_bay_biob_refs_3938.diff (2 lines)
[INFO] [2021-05-31 14:35:42] ...nodes (/app/public/data/kaneohe_bay_biob/taxa.txt)
[CMD] [2021-05-31 14:35:42] echo "0a" > /app/public/diff/kaneohe_bay_biob_nodes_3938.diff
[CMD] [2021-05-31 14:35:42] tail -n +1 /app/public/converted_csv/kaneohe_bay_biob_nodes_3938.csv >> /app/public/diff/kaneohe_bay_biob_nodes_3938.diff
[CMD] [2021-05-31 14:35:43] echo "." >> /app/public/diff/kaneohe_bay_biob_nodes_3938.diff
[INFO] [2021-05-31 14:35:43] Created diff: /app/public/diff/kaneohe_bay_biob_nodes_3938.diff (97 lines)
[INFO] [2021-05-31 14:35:43] ...media (/app/public/data/kaneohe_bay_biob/media.txt)
[CMD] [2021-05-31 14:35:43] echo "0a" > /app/public/diff/kaneohe_bay_biob_media_3938.diff
[CMD] [2021-05-31 14:35:44] tail -n +1 /app/public/converted_csv/kaneohe_bay_biob_media_3938.csv >> /app/public/diff/kaneohe_bay_biob_media_3938.diff
[CMD] [2021-05-31 14:35:44] echo "." >> /app/public/diff/kaneohe_bay_biob_media_3938.diff
[INFO] [2021-05-31 14:35:44] Created diff: /app/public/diff/kaneohe_bay_biob_media_3938.diff (2 lines)
[INFO] [2021-05-31 14:35:44] ...vernaculars (/app/public/data/kaneohe_bay_biob/common names.txt)
[CMD] [2021-05-31 14:35:44] echo "0a" > /app/public/diff/kaneohe_bay_biob_vernaculars_3938.diff
[CMD] [2021-05-31 14:35:45] tail -n +1 /app/public/converted_csv/kaneohe_bay_biob_vernaculars_3938.csv >> /app/public/diff/kaneohe_bay_biob_vernaculars_3938.diff
[CMD] [2021-05-31 14:35:45] echo "." >> /app/public/diff/kaneohe_bay_biob_vernaculars_3938.diff
[INFO] [2021-05-31 14:35:45] Created diff: /app/public/diff/kaneohe_bay_biob_vernaculars_3938.diff (2 lines)
[INFO] [2021-05-31 14:35:45] ...occurrences (/app/public/data/kaneohe_bay_biob/occurrences.txt)
[CMD] [2021-05-31 14:35:45] echo "0a" > /app/public/diff/kaneohe_bay_biob_occurrences_3938.diff
[CMD] [2021-05-31 14:35:46] tail -n +1 /app/public/converted_csv/kaneohe_bay_biob_occurrences_3938.csv >> /app/public/diff/kaneohe_bay_biob_occurrences_3938.diff
[CMD] [2021-05-31 14:35:46] echo "." >> /app/public/diff/kaneohe_bay_biob_occurrences_3938.diff
[INFO] [2021-05-31 14:35:46] Created diff: /app/public/diff/kaneohe_bay_biob_occurrences_3938.diff (154 lines)
[INFO] [2021-05-31 14:35:46] ...assocs (/app/public/data/kaneohe_bay_biob/associations.txt)
[CMD] [2021-05-31 14:35:46] echo "0a" > /app/public/diff/kaneohe_bay_biob_assocs_3938.diff
[CMD] [2021-05-31 14:35:47] tail -n +1 /app/public/converted_csv/kaneohe_bay_biob_assocs_3938.csv >> /app/public/diff/kaneohe_bay_biob_assocs_3938.diff
[CMD] [2021-05-31 14:35:47] echo "." >> /app/public/diff/kaneohe_bay_biob_assocs_3938.diff
[INFO] [2021-05-31 14:35:48] Created diff: /app/public/diff/kaneohe_bay_biob_assocs_3938.diff (2 lines)
[INFO] [2021-05-31 14:35:48] ...measurements (/app/public/data/kaneohe_bay_biob/measurements or facts.txt)
[CMD] [2021-05-31 14:35:48] echo "0a" > /app/public/diff/kaneohe_bay_biob_measurements_3938.diff
[CMD] [2021-05-31 14:35:48] tail -n +1 /app/public/converted_csv/kaneohe_bay_biob_measurements_3938.csv >> /app/public/diff/kaneohe_bay_biob_measurements_3938.diff
[CMD] [2021-05-31 14:35:48] echo "." >> /app/public/diff/kaneohe_bay_biob_measurements_3938.diff
[INFO] [2021-05-31 14:35:49] Created diff: /app/public/diff/kaneohe_bay_biob_measurements_3938.diff (302 lines)
[STOP] [2021-05-31 14:35:49] calculate_delta
[START] [2021-05-31 14:35:49] parse_diff_and_store
[INFO] [2021-05-31 14:35:49] Handling diff: /app/public/diff/kaneohe_bay_biob_agents_3938.diff (2 lines)
[INFO] [2021-05-31 14:35:49] Loading agents diff file into memory (2 /app/public/diff/kaneohe_bay_biob_agents_3938.diff lines)...
[INFO] [2021-05-31 14:35:50] Handling diff: /app/public/diff/kaneohe_bay_biob_refs_3938.diff (2 lines)
[INFO] [2021-05-31 14:35:50] Loading refs diff file into memory (2 /app/public/diff/kaneohe_bay_biob_refs_3938.diff lines)...
[INFO] [2021-05-31 14:35:50] Handling diff: /app/public/diff/kaneohe_bay_biob_nodes_3938.diff (97 lines)
[INFO] [2021-05-31 14:35:51] Loading nodes diff file into memory (97 /app/public/diff/kaneohe_bay_biob_nodes_3938.diff lines)...
[INFO] [2021-05-31 14:35:51] Handling diff: /app/public/diff/kaneohe_bay_biob_media_3938.diff (2 lines)
[INFO] [2021-05-31 14:35:52] Loading media diff file into memory (2 /app/public/diff/kaneohe_bay_biob_media_3938.diff lines)...
[INFO] [2021-05-31 14:35:52] Handling diff: /app/public/diff/kaneohe_bay_biob_vernaculars_3938.diff (2 lines)
[INFO] [2021-05-31 14:35:52] Loading vernaculars diff file into memory (2 /app/public/diff/kaneohe_bay_biob_vernaculars_3938.diff lines)...
[INFO] [2021-05-31 14:35:53] Handling diff: /app/public/diff/kaneohe_bay_biob_occurrences_3938.diff (154 lines)
[INFO] [2021-05-31 14:35:53] Loading occurrences diff file into memory (154 /app/public/diff/kaneohe_bay_biob_occurrences_3938.diff lines)...
[INFO] [2021-05-31 14:35:54] Handling diff: /app/public/diff/kaneohe_bay_biob_assocs_3938.diff (2 lines)
[INFO] [2021-05-31 14:35:54] Loading assocs diff file into memory (2 /app/public/diff/kaneohe_bay_biob_assocs_3938.diff lines)...
[INFO] [2021-05-31 14:35:54] Handling diff: /app/public/diff/kaneohe_bay_biob_measurements_3938.diff (302 lines)
[INFO] [2021-05-31 14:35:55] Loading measurements diff file into memory (302 /app/public/diff/kaneohe_bay_biob_measurements_3938.diff lines)...
[INFO] [2021-05-31 14:35:55] Storing 226 ScientificNames
[INFO] [2021-05-31 14:35:55] Processing group of 226 in 1 groups of 1000
[INFO] [2021-05-31 14:35:55] Average Time: 0.07
[INFO] [2021-05-31 14:35:55] Total Time: 1s
[INFO] [2021-05-31 14:35:55] Storing 226 Nodes
[INFO] [2021-05-31 14:35:55] Processing group of 226 in 1 groups of 1000
[INFO] [2021-05-31 14:35:55] Average Time: 0.06
[INFO] [2021-05-31 14:35:55] Total Time: 1s
[INFO] [2021-05-31 14:35:55] Storing 152 Occurrences
[INFO] [2021-05-31 14:35:55] Processing group of 152 in 1 groups of 1000
[INFO] [2021-05-31 14:35:55] Average Time: 0.02
[INFO] [2021-05-31 14:35:55] Total Time: 1s
[INFO] [2021-05-31 14:35:55] Storing 911 OccurrenceMetadata
[INFO] [2021-05-31 14:35:55] Processing group of 911 in 1 groups of 1000
[INFO] [2021-05-31 14:35:56] Average Time: 0.1
[INFO] [2021-05-31 14:35:56] Total Time: 1s
[INFO] [2021-05-31 14:35:56] Storing 300 Traits
[INFO] [2021-05-31 14:35:56] Processing group of 300 in 1 groups of 1000
[INFO] [2021-05-31 14:35:56] Average Time: 0.09
[INFO] [2021-05-31 14:35:56] Total Time: 1s
[INFO] [2021-05-31 14:35:56] Storing 291 MetaTraits
[INFO] [2021-05-31 14:35:56] Processing group of 291 in 1 groups of 1000
[INFO] [2021-05-31 14:35:56] Average Time: 0.03
[INFO] [2021-05-31 14:35:56] Total Time: 1s
[STOP] [2021-05-31 14:35:56] parse_diff_and_store
[START] [2021-05-31 14:35:56] resolve_keys
[INFO] [2021-05-31 14:36:02] Occurrences to nodes (through scientific_names)...
[INFO] [2021-05-31 14:36:02] traits to occurrences...
[INFO] [2021-05-31 14:36:02] traits to nodes (through occurrences)...
[INFO] [2021-05-31 14:36:02] Traits to sex term...
[INFO] [2021-05-31 14:36:02] Traits to lifestage term...
[INFO] [2021-05-31 14:36:02] MetaTraits to traits...
[INFO] [2021-05-31 14:36:02] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-05-31 14:36:02] Assocs to occurrences...
[INFO] [2021-05-31 14:36:02] Assocs to nodes...
[INFO] [2021-05-31 14:36:02] Assoc to sex term...
[INFO] [2021-05-31 14:36:02] Assoc to lifestage term...
[INFO] [2021-05-31 14:36:02] MetaAssoc to assocs...
[STOP] [2021-05-31 14:36:02] resolve_keys
[START] [2021-05-31 14:36:02] hold_for_later_1
[STOP] [2021-05-31 14:36:02] hold_for_later_1
[START] [2021-05-31 14:36:02] hold_for_later_2
[STOP] [2021-05-31 14:36:02] hold_for_later_2
[START] [2021-05-31 14:36:02] resolve_missing_parents
[STOP] [2021-05-31 14:36:02] resolve_missing_parents
[START] [2021-05-31 14:36:02] rebuild_nodes
[START] [2021-05-31 14:36:02] Flattener#flatten
[START] [2021-05-31 14:36:02] Flattener#study_resource
[START] [2021-05-31 14:36:02] Flattener#build_ancestry
[STOP] [2021-05-31 14:36:02] Flattener#build_ancestry
[INFO] [2021-05-31 14:36:02] 226 ancestry keys
[START] [2021-05-31 14:36:02] build_node_ancestors
[INFO] [2021-05-31 14:36:02] old ancestors deleted.
[STOP] [2021-05-31 14:36:02] build_node_ancestors
[START] [2021-05-31 14:36:02] Flattener#propagate_ancestor_ids
[STOP] [2021-05-31 14:36:02] Flattener#propagate_ancestor_ids
[STOP] [2021-05-31 14:36:02] Flattener#flatten
[STOP] [2021-05-31 14:36:02] rebuild_nodes
[START] [2021-05-31 14:36:02] resolve_missing_media_owners
[STOP] [2021-05-31 14:36:02] resolve_missing_media_owners
[START] [2021-05-31 14:36:02] sanitize_media_verbatims
[STOP] [2021-05-31 14:36:02] sanitize_media_verbatims
[START] [2021-05-31 14:36:02] queue_downloads
[STOP] [2021-05-31 14:36:02] queue_downloads
[START] [2021-05-31 14:36:02] parse_names
[WARN] [2021-05-31 14:36:02] I see 226 names which still need to be parsed.
[WARN] [2021-05-31 14:36:03] I see 5 names which still need to be parsed.
[STOP] [2021-05-31 14:36:04] parse_names
[START] [2021-05-31 14:36:04] denormalize_canonical_names_to_nodes
[STOP] [2021-05-31 14:36:04] denormalize_canonical_names_to_nodes
[START] [2021-05-31 14:36:04] match_nodes
[START] [2021-05-31 14:36:04] map_all_nodes_to_pages
[STOP] [2021-05-31 14:36:07] map_all_nodes_to_pages
[INFO] [2021-05-31 14:36:07] Unmatched nodes (3 of 226): Canonical: Ovalentaria; Node#95121059; ResourceID: Ovalentaria; Canonical: Stegastes marginatus; Node#95121265; ResourceID: Stegastes marginatus; Canonical: Scorpaeniformes; Node#95121137; ResourceID: Scorpaeniformes
[START] [2021-05-31 14:36:07] update_nodes
[STOP] [2021-05-31 14:36:07] update_nodes
[STOP] [2021-05-31 14:36:07] match_nodes
[START] [2021-05-31 14:36:07] reindex_search
[STOP] [2021-05-31 14:36:07] reindex_search
[START] [2021-05-31 14:36:07] normalize_units
[STOP] [2021-05-31 14:36:08] normalize_units
[START] [2021-05-31 14:36:08] calculate_statistics
[STOP] [2021-05-31 14:36:08] calculate_statistics
[START] [2021-05-31 14:36:08] complete_harvest_instance
[START] [2021-05-31 14:36:08] overall_tsv_creation
[INFO] [2021-05-31 14:36:08] Processing group of 226 in 1 batches of 10000
[INFO] [2021-05-31 14:36:41] 300 Traits (unfiltered)...
[INFO] [2021-05-31 14:37:09] 300 Traits (filtered)...
[INFO] [2021-05-31 14:37:09] 0 Associations (filtered)...
[INFO] [2021-05-31 14:37:09] 1788 metadata added.
[INFO] [2021-05-31 14:37:09] 0 metadata added.
[INFO] [2021-05-31 14:37:33] Average Time: 62.5
[INFO] [2021-05-31 14:37:33] Total Time: 1m25s
[STOP] [2021-05-31 14:37:33] overall_tsv_creation
[INFO] [2021-05-31 14:37:33] Done. Check your files:
[INFO] [2021-05-31 14:37:33] (221 lines) /app/public/data/kaneohe_bay_biob/publish_nodes.tsv
[INFO] [2021-05-31 14:37:33] (884 lines) /app/public/data/kaneohe_bay_biob/publish_node_ancestors.tsv
[INFO] [2021-05-31 14:37:34] (226 lines) /app/public/data/kaneohe_bay_biob/publish_scientific_names.tsv
[INFO] [2021-05-31 14:37:34] (301 lines) /app/public/data/kaneohe_bay_biob/publish_traits.tsv
[INFO] [2021-05-31 14:37:35] (1789 lines) /app/public/data/kaneohe_bay_biob/publish_metadata.tsv
[STOP] [2021-05-31 14:37:35] complete_harvest_instance
[START] [2021-05-31 14:37:35] completed
[STOP] [2021-05-31 14:37:35] completed
[STOP] [2021-05-31 14:37:35] logged process, took 118.44

Latest Process

31 May 14:35 EXITED currently processing 0