Harvest for Noguchi et al 2006 Created 13 Oct 09:12

Harvesting Log

[INFO] [2023-10-13 09:12:05] Created harvest instance #4415
[STOP] [2023-10-13 09:12:05] create_harvest_instance
[START] [2023-10-13 09:12:05] fetch_files
[STOP] [2023-10-13 09:12:05] fetch_files
[START] [2023-10-13 09:12:05] validate_each_file
[INFO] [2023-10-13 09:12:05] Looping over 4 formats...
[INFO] [2023-10-13 09:12:05] ...refs (/app/public/data/noguchi_et_al_no/references.txt)
[INFO] [2023-10-13 09:12:05] Valid: /app/public/data/noguchi_et_al_no/converted_csv/noguchi_et_al_no_refs_30611.csv (21 lines)
[INFO] [2023-10-13 09:12:05] ...nodes (/app/public/data/noguchi_et_al_no/taxa.txt)
[INFO] [2023-10-13 09:12:05] Valid: /app/public/data/noguchi_et_al_no/converted_csv/noguchi_et_al_no_nodes_30608.csv (27 lines)
[INFO] [2023-10-13 09:12:05] ...occurrences (/app/public/data/noguchi_et_al_no/occurrences.txt)
[INFO] [2023-10-13 09:12:05] Valid: /app/public/data/noguchi_et_al_no/converted_csv/noguchi_et_al_no_occurrences_30609.csv (27 lines)
[INFO] [2023-10-13 09:12:05] ...measurements (/app/public/data/noguchi_et_al_no/measurementsorfacts.txt)
[INFO] [2023-10-13 09:12:05] Valid: /app/public/data/noguchi_et_al_no/converted_csv/noguchi_et_al_no_measurements_30610.csv (37 lines)
[STOP] [2023-10-13 09:12:05] validate_each_file
[START] [2023-10-13 09:12:05] convert_to_csv
[INFO] [2023-10-13 09:12:05] Looping over 4 formats...
[INFO] [2023-10-13 09:12:05] ...refs (/app/public/data/noguchi_et_al_no/references.txt)
[CMD] [2023-10-13 09:12:05] /usr/bin/sort /app/public/data/noguchi_et_al_no/converted_csv/noguchi_et_al_no_refs_30611.csv > /app/public/data/noguchi_et_al_no/converted_csv/noguchi_et_al_no_refs_30611.csv_sorted
[INFO] [2023-10-13 09:12:05] Converted: /app/public/data/noguchi_et_al_no/converted_csv/noguchi_et_al_no_refs_30611.csv (21 lines)
[INFO] [2023-10-13 09:12:05] ...nodes (/app/public/data/noguchi_et_al_no/taxa.txt)
[CMD] [2023-10-13 09:12:05] /usr/bin/sort /app/public/data/noguchi_et_al_no/converted_csv/noguchi_et_al_no_nodes_30608.csv > /app/public/data/noguchi_et_al_no/converted_csv/noguchi_et_al_no_nodes_30608.csv_sorted
[INFO] [2023-10-13 09:12:05] Converted: /app/public/data/noguchi_et_al_no/converted_csv/noguchi_et_al_no_nodes_30608.csv (27 lines)
[INFO] [2023-10-13 09:12:05] ...occurrences (/app/public/data/noguchi_et_al_no/occurrences.txt)
[CMD] [2023-10-13 09:12:05] /usr/bin/sort /app/public/data/noguchi_et_al_no/converted_csv/noguchi_et_al_no_occurrences_30609.csv > /app/public/data/noguchi_et_al_no/converted_csv/noguchi_et_al_no_occurrences_30609.csv_sorted
[INFO] [2023-10-13 09:12:06] Converted: /app/public/data/noguchi_et_al_no/converted_csv/noguchi_et_al_no_occurrences_30609.csv (27 lines)
[INFO] [2023-10-13 09:12:06] ...measurements (/app/public/data/noguchi_et_al_no/measurementsorfacts.txt)
[CMD] [2023-10-13 09:12:06] /usr/bin/sort /app/public/data/noguchi_et_al_no/converted_csv/noguchi_et_al_no_measurements_30610.csv > /app/public/data/noguchi_et_al_no/converted_csv/noguchi_et_al_no_measurements_30610.csv_sorted
[INFO] [2023-10-13 09:12:06] Converted: /app/public/data/noguchi_et_al_no/converted_csv/noguchi_et_al_no_measurements_30610.csv (37 lines)
[STOP] [2023-10-13 09:12:06] convert_to_csv
[START] [2023-10-13 09:12:06] calculate_delta
[INFO] [2023-10-13 09:12:06] Looping over 4 formats...
[INFO] [2023-10-13 09:12:06] ...refs (/app/public/data/noguchi_et_al_no/references.txt)
[CMD] [2023-10-13 09:12:06] echo "0a" > /app/public/data/noguchi_et_al_no/diff/noguchi_et_al_no_refs_30611.diff
[CMD] [2023-10-13 09:12:06] tail -n +1 /app/public/data/noguchi_et_al_no/converted_csv/noguchi_et_al_no_refs_30611.csv >> /app/public/data/noguchi_et_al_no/diff/noguchi_et_al_no_refs_30611.diff
[CMD] [2023-10-13 09:12:06] echo "." >> /app/public/data/noguchi_et_al_no/diff/noguchi_et_al_no_refs_30611.diff
[INFO] [2023-10-13 09:12:06] Created diff: /app/public/data/noguchi_et_al_no/diff/noguchi_et_al_no_refs_30611.diff (23 lines)
[INFO] [2023-10-13 09:12:06] ...nodes (/app/public/data/noguchi_et_al_no/taxa.txt)
[CMD] [2023-10-13 09:12:06] echo "0a" > /app/public/data/noguchi_et_al_no/diff/noguchi_et_al_no_nodes_30608.diff
[CMD] [2023-10-13 09:12:06] tail -n +1 /app/public/data/noguchi_et_al_no/converted_csv/noguchi_et_al_no_nodes_30608.csv >> /app/public/data/noguchi_et_al_no/diff/noguchi_et_al_no_nodes_30608.diff
[CMD] [2023-10-13 09:12:06] echo "." >> /app/public/data/noguchi_et_al_no/diff/noguchi_et_al_no_nodes_30608.diff
[INFO] [2023-10-13 09:12:06] Created diff: /app/public/data/noguchi_et_al_no/diff/noguchi_et_al_no_nodes_30608.diff (29 lines)
[INFO] [2023-10-13 09:12:06] ...occurrences (/app/public/data/noguchi_et_al_no/occurrences.txt)
[CMD] [2023-10-13 09:12:06] echo "0a" > /app/public/data/noguchi_et_al_no/diff/noguchi_et_al_no_occurrences_30609.diff
[CMD] [2023-10-13 09:12:06] tail -n +1 /app/public/data/noguchi_et_al_no/converted_csv/noguchi_et_al_no_occurrences_30609.csv >> /app/public/data/noguchi_et_al_no/diff/noguchi_et_al_no_occurrences_30609.diff
[CMD] [2023-10-13 09:12:06] echo "." >> /app/public/data/noguchi_et_al_no/diff/noguchi_et_al_no_occurrences_30609.diff
[INFO] [2023-10-13 09:12:06] Created diff: /app/public/data/noguchi_et_al_no/diff/noguchi_et_al_no_occurrences_30609.diff (29 lines)
[INFO] [2023-10-13 09:12:06] ...measurements (/app/public/data/noguchi_et_al_no/measurementsorfacts.txt)
[CMD] [2023-10-13 09:12:06] echo "0a" > /app/public/data/noguchi_et_al_no/diff/noguchi_et_al_no_measurements_30610.diff
[CMD] [2023-10-13 09:12:06] tail -n +1 /app/public/data/noguchi_et_al_no/converted_csv/noguchi_et_al_no_measurements_30610.csv >> /app/public/data/noguchi_et_al_no/diff/noguchi_et_al_no_measurements_30610.diff
[CMD] [2023-10-13 09:12:06] echo "." >> /app/public/data/noguchi_et_al_no/diff/noguchi_et_al_no_measurements_30610.diff
[INFO] [2023-10-13 09:12:06] Created diff: /app/public/data/noguchi_et_al_no/diff/noguchi_et_al_no_measurements_30610.diff (39 lines)
[STOP] [2023-10-13 09:12:06] calculate_delta
[START] [2023-10-13 09:12:07] parse_diff_and_store
[INFO] [2023-10-13 09:12:07] Handling diff: /app/public/data/noguchi_et_al_no/diff/noguchi_et_al_no_refs_30611.diff (23 lines)
[INFO] [2023-10-13 09:12:07] Loading refs diff file into memory (23 lines)...
[INFO] [2023-10-13 09:12:07] Storing 21 References (21/21/23)
[INFO] [2023-10-13 09:12:07] Handling diff: /app/public/data/noguchi_et_al_no/diff/noguchi_et_al_no_nodes_30608.diff (29 lines)
[INFO] [2023-10-13 09:12:07] Loading nodes diff file into memory (29 lines)...
[INFO] [2023-10-13 09:12:07] Storing 30 ScientificNames (60/27/29)
[INFO] [2023-10-13 09:12:07] Storing 30 Nodes (60/27/29)
[INFO] [2023-10-13 09:12:07] Handling diff: /app/public/data/noguchi_et_al_no/diff/noguchi_et_al_no_occurrences_30609.diff (29 lines)
[INFO] [2023-10-13 09:12:07] Loading occurrences diff file into memory (29 lines)...
[INFO] [2023-10-13 09:12:07] Storing 27 Occurrences (27/27/29)
[INFO] [2023-10-13 09:12:07] Handling diff: /app/public/data/noguchi_et_al_no/diff/noguchi_et_al_no_measurements_30610.diff (39 lines)
[INFO] [2023-10-13 09:12:07] Loading measurements diff file into memory (39 lines)...
[INFO] [2023-10-13 09:12:07] Storing 37 Traits (89/37/39)
[INFO] [2023-10-13 09:12:07] Storing 27 MetaTraits (89/37/39)
[INFO] [2023-10-13 09:12:07] Storing 25 TraitsReferences (89/37/39)
[STOP] [2023-10-13 09:12:07] parse_diff_and_store
[START] [2023-10-13 09:12:07] resolve_keys
[2023-10-13 09:12:07] Resolving downloaded urls (this is not actually downloading them yet)
[INFO] [2023-10-13 09:12:15] Occurrences to nodes (through scientific_names)...
[INFO] [2023-10-13 09:12:15] traits to occurrences...
[INFO] [2023-10-13 09:12:15] traits to nodes (through occurrences)...
[INFO] [2023-10-13 09:12:15] Traits to sex term...
[INFO] [2023-10-13 09:12:15] Traits to lifestage term...
[INFO] [2023-10-13 09:12:15] MetaTraits to traits...
[INFO] [2023-10-13 09:12:15] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2023-10-13 09:12:15] Assocs to occurrences...
[INFO] [2023-10-13 09:12:15] Assocs to nodes...
[INFO] [2023-10-13 09:12:15] Assoc to sex term...
[INFO] [2023-10-13 09:12:15] Assoc to lifestage term...
[INFO] [2023-10-13 09:12:15] MetaAssoc to assocs...
[STOP] [2023-10-13 09:12:15] resolve_keys
[START] [2023-10-13 09:12:15] hold_for_later_1
[STOP] [2023-10-13 09:12:15] hold_for_later_1
[START] [2023-10-13 09:12:15] hold_for_later_2
[STOP] [2023-10-13 09:12:15] hold_for_later_2
[START] [2023-10-13 09:12:15] resolve_missing_parents
[STOP] [2023-10-13 09:12:15] resolve_missing_parents
[START] [2023-10-13 09:12:15] rebuild_nodes
[START] [2023-10-13 09:12:15] Flattener#flatten
[START] [2023-10-13 09:12:15] Flattener#study_resource
[START] [2023-10-13 09:12:15] Flattener#build_ancestry
[STOP] [2023-10-13 09:12:15] Flattener#build_ancestry
[INFO] [2023-10-13 09:12:15] 30 ancestry keys
[START] [2023-10-13 09:12:15] build_node_ancestors
[INFO] [2023-10-13 09:12:15] old ancestors deleted.
[STOP] [2023-10-13 09:12:15] build_node_ancestors
[START] [2023-10-13 09:12:15] Flattener#propagate_ancestor_ids
[STOP] [2023-10-13 09:12:15] Flattener#propagate_ancestor_ids
[STOP] [2023-10-13 09:12:15] Flattener#flatten
[STOP] [2023-10-13 09:12:15] rebuild_nodes
[START] [2023-10-13 09:12:15] resolve_missing_media_owners
[STOP] [2023-10-13 09:12:15] resolve_missing_media_owners
[START] [2023-10-13 09:12:15] sanitize_media_verbatims
[STOP] [2023-10-13 09:12:15] sanitize_media_verbatims
[START] [2023-10-13 09:12:15] queue_downloads
[STOP] [2023-10-13 09:12:15] queue_downloads
[START] [2023-10-13 09:12:15] parse_names
[WARN] [2023-10-13 09:12:15] I see 30 names which still need to be parsed.
[WARN] [2023-10-13 09:12:15] Names to parse: 30 formatted: 30 learned: 30 parsed: 30
[STOP] [2023-10-13 09:12:16] parse_names
[START] [2023-10-13 09:12:16] denormalize_canonical_names_to_nodes
[STOP] [2023-10-13 09:12:16] denormalize_canonical_names_to_nodes
[START] [2023-10-13 09:12:16] match_nodes
[START] [2023-10-13 09:12:16] map_all_nodes_to_pages
[STOP] [2023-10-13 09:12:16] map_all_nodes_to_pages
[INFO] [2023-10-13 09:12:16] Unmatched nodes (2 of 30): Canonical: Pseudopolamilla occelata; Node#136982659; ResourceID: Pseudopolamilla occelata; Canonical: Pugilina ternotoma; Node#136982662; ResourceID: Pugilina ternotoma
[START] [2023-10-13 09:12:16] update_nodes
[STOP] [2023-10-13 09:12:16] update_nodes
[STOP] [2023-10-13 09:12:16] match_nodes
[START] [2023-10-13 09:12:16] reindex_search
[STOP] [2023-10-13 09:12:16] reindex_search
[START] [2023-10-13 09:12:16] normalize_units
[STOP] [2023-10-13 09:12:16] normalize_units
[START] [2023-10-13 09:12:16] calculate_statistics
[INFO] [2023-10-13 09:12:16] Duplicate page_id count: 0
[STOP] [2023-10-13 09:12:16] calculate_statistics
[START] [2023-10-13 09:12:16] complete_harvest_instance
[START] [2023-10-13 09:12:16] overall_tsv_creation
[INFO] [2023-10-13 09:12:17] Exporting 30 nodes as TSV in batches of 10000...
[INFO] [2023-10-13 09:12:17] Processing group of 30 in 1 batches of 10000
[INFO] [2023-10-13 09:12:17] 27 Traits (unfiltered) and 0 associations...
[INFO] [2023-10-13 09:12:17] Building Traits map for 30 nodes (this can take a while)...
[INFO] [2023-10-13 09:12:17] Mapped 27 traits (27 meta) for 30 nodes.
[INFO] [2023-10-13 09:12:17] Building Associations map (this can take a while)...
[INFO] [2023-10-13 09:12:17] Done. 0 assocs mapped (0 meta).
[INFO] [2023-10-13 09:12:17] Adding 27 traits...
[INFO] [2023-10-13 09:12:17] 35 metadata added.
[INFO] [2023-10-13 09:12:17] Adding 0 assocs...
[INFO] [2023-10-13 09:12:17] 0 metadata added.
[INFO] [2023-10-13 09:13:01] Processed 30/30 nodes
[INFO] [2023-10-13 09:13:01] Average Time: 44.58
[INFO] [2023-10-13 09:13:01] Total Time: 45s
[STOP] [2023-10-13 09:13:01] overall_tsv_creation
[INFO] [2023-10-13 09:13:01] Done. Check your files:
[INFO] [2023-10-13 09:13:01] (30 lines) /app/public/data/noguchi_et_al_no/publish_nodes.tsv
[INFO] [2023-10-13 09:13:01] (4 lines) /app/public/data/noguchi_et_al_no/publish_node_ancestors.tsv
[INFO] [2023-10-13 09:13:01] (30 lines) /app/public/data/noguchi_et_al_no/publish_scientific_names.tsv
[INFO] [2023-10-13 09:13:01] (28 lines) /app/public/data/noguchi_et_al_no/publish_traits.tsv
[INFO] [2023-10-13 09:13:02] (36 lines) /app/public/data/noguchi_et_al_no/publish_metadata.tsv
[STOP] [2023-10-13 09:13:02] complete_harvest_instance
[START] [2023-10-13 09:13:02] completed
[STOP] [2023-10-13 09:13:02] completed
[STOP] [2023-10-13 09:13:02] logged process, took 56.46

Latest Process

13 Oct 09:12 EXITED currently processing 0