Harvest for Calcification rate literature Created 31 May 17:37

Stage: completed
Fetched: 31 May 17:37
Validated: 31 May 17:37
Deltas Created 31 May 17:38
Units Normalized: 31 May 17:38
Ancestry Built: 31 May 17:38
Nodes Matched: 31 May 17:38
Names Parsed: 31 May 17:38
New Models Stored: 31 May 17:38
Indexed: 31 May 17:38
Completed: 31 May 17:39
Time to Harvest: less than a minute

Harvesting Log

(222 lines)
[INFO] [2021-05-31 17:37:55] Created harvest instance #3943
[STOP] [2021-05-31 17:37:55] create_harvest_instance
[START] [2021-05-31 17:37:55] fetch_files
[STOP] [2021-05-31 17:37:55] fetch_files
[START] [2021-05-31 17:37:55] validate_each_file
[INFO] [2021-05-31 17:37:55] Looping over 8 formats...
[INFO] [2021-05-31 17:37:55] ...agents (/app/public/data/Calcification_ra/agents.txt)
[INFO] [2021-05-31 17:37:55] Valid: /app/public/converted_csv/Calcification_ra_agents_3943.csv (0 lines)
[INFO] [2021-05-31 17:37:55] ...refs (/app/public/data/Calcification_ra/references.txt)
[INFO] [2021-05-31 17:37:55] Valid: /app/public/converted_csv/Calcification_ra_refs_3943.csv (6 lines)
[INFO] [2021-05-31 17:37:55] ...nodes (/app/public/data/Calcification_ra/taxa.txt)
[INFO] [2021-05-31 17:37:55] Valid: /app/public/converted_csv/Calcification_ra_nodes_3943.csv (20 lines)
[INFO] [2021-05-31 17:37:55] ...media (/app/public/data/Calcification_ra/media.txt)
[INFO] [2021-05-31 17:37:55] Valid: /app/public/converted_csv/Calcification_ra_media_3943.csv (0 lines)
[INFO] [2021-05-31 17:37:55] ...vernaculars (/app/public/data/Calcification_ra/common names.txt)
[INFO] [2021-05-31 17:37:55] Valid: /app/public/converted_csv/Calcification_ra_vernaculars_3943.csv (0 lines)
[INFO] [2021-05-31 17:37:55] ...occurrences (/app/public/data/Calcification_ra/occurrences.txt)
[INFO] [2021-05-31 17:37:55] Valid: /app/public/converted_csv/Calcification_ra_occurrences_3943.csv (25 lines)
[INFO] [2021-05-31 17:37:55] ...assocs (/app/public/data/Calcification_ra/associations.txt)
[INFO] [2021-05-31 17:37:55] Valid: /app/public/converted_csv/Calcification_ra_assocs_3943.csv (0 lines)
[INFO] [2021-05-31 17:37:55] ...measurements (/app/public/data/Calcification_ra/measurements or facts.txt)
[INFO] [2021-05-31 17:37:55] Valid: /app/public/converted_csv/Calcification_ra_measurements_3943.csv (116 lines)
[STOP] [2021-05-31 17:37:55] validate_each_file
[START] [2021-05-31 17:37:55] convert_to_csv
[INFO] [2021-05-31 17:37:55] Looping over 8 formats...
[INFO] [2021-05-31 17:37:55] ...agents (/app/public/data/Calcification_ra/agents.txt)
[CMD] [2021-05-31 17:37:55] /usr/bin/sort /app/public/converted_csv/Calcification_ra_agents_3943.csv > /app/public/converted_csv/Calcification_ra_agents_3943.csv_sorted
[INFO] [2021-05-31 17:37:55] Converted: /app/public/converted_csv/Calcification_ra_agents_3943.csv (0 lines)
[INFO] [2021-05-31 17:37:55] ...refs (/app/public/data/Calcification_ra/references.txt)
[CMD] [2021-05-31 17:37:55] /usr/bin/sort /app/public/converted_csv/Calcification_ra_refs_3943.csv > /app/public/converted_csv/Calcification_ra_refs_3943.csv_sorted
[INFO] [2021-05-31 17:37:56] Converted: /app/public/converted_csv/Calcification_ra_refs_3943.csv (6 lines)
[INFO] [2021-05-31 17:37:56] ...nodes (/app/public/data/Calcification_ra/taxa.txt)
[CMD] [2021-05-31 17:37:56] /usr/bin/sort /app/public/converted_csv/Calcification_ra_nodes_3943.csv > /app/public/converted_csv/Calcification_ra_nodes_3943.csv_sorted
[INFO] [2021-05-31 17:37:56] Converted: /app/public/converted_csv/Calcification_ra_nodes_3943.csv (20 lines)
[INFO] [2021-05-31 17:37:56] ...media (/app/public/data/Calcification_ra/media.txt)
[CMD] [2021-05-31 17:37:56] /usr/bin/sort /app/public/converted_csv/Calcification_ra_media_3943.csv > /app/public/converted_csv/Calcification_ra_media_3943.csv_sorted
[INFO] [2021-05-31 17:37:56] Converted: /app/public/converted_csv/Calcification_ra_media_3943.csv (0 lines)
[INFO] [2021-05-31 17:37:56] ...vernaculars (/app/public/data/Calcification_ra/common names.txt)
[CMD] [2021-05-31 17:37:56] /usr/bin/sort /app/public/converted_csv/Calcification_ra_vernaculars_3943.csv > /app/public/converted_csv/Calcification_ra_vernaculars_3943.csv_sorted
[INFO] [2021-05-31 17:37:57] Converted: /app/public/converted_csv/Calcification_ra_vernaculars_3943.csv (0 lines)
[INFO] [2021-05-31 17:37:57] ...occurrences (/app/public/data/Calcification_ra/occurrences.txt)
[CMD] [2021-05-31 17:37:57] /usr/bin/sort /app/public/converted_csv/Calcification_ra_occurrences_3943.csv > /app/public/converted_csv/Calcification_ra_occurrences_3943.csv_sorted
[INFO] [2021-05-31 17:37:57] Converted: /app/public/converted_csv/Calcification_ra_occurrences_3943.csv (25 lines)
[INFO] [2021-05-31 17:37:57] ...assocs (/app/public/data/Calcification_ra/associations.txt)
[CMD] [2021-05-31 17:37:57] /usr/bin/sort /app/public/converted_csv/Calcification_ra_assocs_3943.csv > /app/public/converted_csv/Calcification_ra_assocs_3943.csv_sorted
[INFO] [2021-05-31 17:37:57] Converted: /app/public/converted_csv/Calcification_ra_assocs_3943.csv (0 lines)
[INFO] [2021-05-31 17:37:57] ...measurements (/app/public/data/Calcification_ra/measurements or facts.txt)
[CMD] [2021-05-31 17:37:57] /usr/bin/sort /app/public/converted_csv/Calcification_ra_measurements_3943.csv > /app/public/converted_csv/Calcification_ra_measurements_3943.csv_sorted
[INFO] [2021-05-31 17:37:58] Converted: /app/public/converted_csv/Calcification_ra_measurements_3943.csv (116 lines)
[STOP] [2021-05-31 17:37:58] convert_to_csv
[START] [2021-05-31 17:37:58] calculate_delta
[INFO] [2021-05-31 17:37:58] Looping over 8 formats...
[INFO] [2021-05-31 17:37:58] ...agents (/app/public/data/Calcification_ra/agents.txt)
[CMD] [2021-05-31 17:37:58] echo "0a" > /app/public/diff/Calcification_ra_agents_3943.diff
[CMD] [2021-05-31 17:37:58] tail -n +1 /app/public/converted_csv/Calcification_ra_agents_3943.csv >> /app/public/diff/Calcification_ra_agents_3943.diff
[CMD] [2021-05-31 17:37:59] echo "." >> /app/public/diff/Calcification_ra_agents_3943.diff
[INFO] [2021-05-31 17:37:59] Created diff: /app/public/diff/Calcification_ra_agents_3943.diff (2 lines)
[INFO] [2021-05-31 17:37:59] ...refs (/app/public/data/Calcification_ra/references.txt)
[CMD] [2021-05-31 17:37:59] echo "0a" > /app/public/diff/Calcification_ra_refs_3943.diff
[CMD] [2021-05-31 17:37:59] tail -n +1 /app/public/converted_csv/Calcification_ra_refs_3943.csv >> /app/public/diff/Calcification_ra_refs_3943.diff
[CMD] [2021-05-31 17:38:00] echo "." >> /app/public/diff/Calcification_ra_refs_3943.diff
[INFO] [2021-05-31 17:38:00] Created diff: /app/public/diff/Calcification_ra_refs_3943.diff (8 lines)
[INFO] [2021-05-31 17:38:00] ...nodes (/app/public/data/Calcification_ra/taxa.txt)
[CMD] [2021-05-31 17:38:00] echo "0a" > /app/public/diff/Calcification_ra_nodes_3943.diff
[CMD] [2021-05-31 17:38:00] tail -n +1 /app/public/converted_csv/Calcification_ra_nodes_3943.csv >> /app/public/diff/Calcification_ra_nodes_3943.diff
[CMD] [2021-05-31 17:38:01] echo "." >> /app/public/diff/Calcification_ra_nodes_3943.diff
[INFO] [2021-05-31 17:38:01] Created diff: /app/public/diff/Calcification_ra_nodes_3943.diff (22 lines)
[INFO] [2021-05-31 17:38:01] ...media (/app/public/data/Calcification_ra/media.txt)
[CMD] [2021-05-31 17:38:01] echo "0a" > /app/public/diff/Calcification_ra_media_3943.diff
[CMD] [2021-05-31 17:38:02] tail -n +1 /app/public/converted_csv/Calcification_ra_media_3943.csv >> /app/public/diff/Calcification_ra_media_3943.diff
[CMD] [2021-05-31 17:38:02] echo "." >> /app/public/diff/Calcification_ra_media_3943.diff
[INFO] [2021-05-31 17:38:02] Created diff: /app/public/diff/Calcification_ra_media_3943.diff (2 lines)
[INFO] [2021-05-31 17:38:02] ...vernaculars (/app/public/data/Calcification_ra/common names.txt)
[CMD] [2021-05-31 17:38:02] echo "0a" > /app/public/diff/Calcification_ra_vernaculars_3943.diff
[CMD] [2021-05-31 17:38:03] tail -n +1 /app/public/converted_csv/Calcification_ra_vernaculars_3943.csv >> /app/public/diff/Calcification_ra_vernaculars_3943.diff
[CMD] [2021-05-31 17:38:03] echo "." >> /app/public/diff/Calcification_ra_vernaculars_3943.diff
[INFO] [2021-05-31 17:38:03] Created diff: /app/public/diff/Calcification_ra_vernaculars_3943.diff (2 lines)
[INFO] [2021-05-31 17:38:03] ...occurrences (/app/public/data/Calcification_ra/occurrences.txt)
[CMD] [2021-05-31 17:38:03] echo "0a" > /app/public/diff/Calcification_ra_occurrences_3943.diff
[CMD] [2021-05-31 17:38:04] tail -n +1 /app/public/converted_csv/Calcification_ra_occurrences_3943.csv >> /app/public/diff/Calcification_ra_occurrences_3943.diff
[CMD] [2021-05-31 17:38:04] echo "." >> /app/public/diff/Calcification_ra_occurrences_3943.diff
[INFO] [2021-05-31 17:38:05] Created diff: /app/public/diff/Calcification_ra_occurrences_3943.diff (27 lines)
[INFO] [2021-05-31 17:38:05] ...assocs (/app/public/data/Calcification_ra/associations.txt)
[CMD] [2021-05-31 17:38:05] echo "0a" > /app/public/diff/Calcification_ra_assocs_3943.diff
[CMD] [2021-05-31 17:38:05] tail -n +1 /app/public/converted_csv/Calcification_ra_assocs_3943.csv >> /app/public/diff/Calcification_ra_assocs_3943.diff
[CMD] [2021-05-31 17:38:05] echo "." >> /app/public/diff/Calcification_ra_assocs_3943.diff
[INFO] [2021-05-31 17:38:06] Created diff: /app/public/diff/Calcification_ra_assocs_3943.diff (2 lines)
[INFO] [2021-05-31 17:38:06] ...measurements (/app/public/data/Calcification_ra/measurements or facts.txt)
[CMD] [2021-05-31 17:38:06] echo "0a" > /app/public/diff/Calcification_ra_measurements_3943.diff
[CMD] [2021-05-31 17:38:06] tail -n +1 /app/public/converted_csv/Calcification_ra_measurements_3943.csv >> /app/public/diff/Calcification_ra_measurements_3943.diff
[CMD] [2021-05-31 17:38:06] echo "." >> /app/public/diff/Calcification_ra_measurements_3943.diff
[INFO] [2021-05-31 17:38:07] Created diff: /app/public/diff/Calcification_ra_measurements_3943.diff (118 lines)
[STOP] [2021-05-31 17:38:07] calculate_delta
[START] [2021-05-31 17:38:07] parse_diff_and_store
[INFO] [2021-05-31 17:38:07] Handling diff: /app/public/diff/Calcification_ra_agents_3943.diff (2 lines)
[INFO] [2021-05-31 17:38:07] Loading agents diff file into memory (2 /app/public/diff/Calcification_ra_agents_3943.diff lines)...
[INFO] [2021-05-31 17:38:08] Handling diff: /app/public/diff/Calcification_ra_refs_3943.diff (8 lines)
[INFO] [2021-05-31 17:38:08] Loading refs diff file into memory (8 /app/public/diff/Calcification_ra_refs_3943.diff lines)...
[INFO] [2021-05-31 17:38:08] Handling diff: /app/public/diff/Calcification_ra_nodes_3943.diff (22 lines)
[INFO] [2021-05-31 17:38:09] Loading nodes diff file into memory (22 /app/public/diff/Calcification_ra_nodes_3943.diff lines)...
[INFO] [2021-05-31 17:38:09] Handling diff: /app/public/diff/Calcification_ra_media_3943.diff (2 lines)
[INFO] [2021-05-31 17:38:09] Loading media diff file into memory (2 /app/public/diff/Calcification_ra_media_3943.diff lines)...
[INFO] [2021-05-31 17:38:10] Handling diff: /app/public/diff/Calcification_ra_vernaculars_3943.diff (2 lines)
[INFO] [2021-05-31 17:38:10] Loading vernaculars diff file into memory (2 /app/public/diff/Calcification_ra_vernaculars_3943.diff lines)...
[INFO] [2021-05-31 17:38:11] Handling diff: /app/public/diff/Calcification_ra_occurrences_3943.diff (27 lines)
[INFO] [2021-05-31 17:38:11] Loading occurrences diff file into memory (27 /app/public/diff/Calcification_ra_occurrences_3943.diff lines)...
[INFO] [2021-05-31 17:38:11] Handling diff: /app/public/diff/Calcification_ra_assocs_3943.diff (2 lines)
[INFO] [2021-05-31 17:38:12] Loading assocs diff file into memory (2 /app/public/diff/Calcification_ra_assocs_3943.diff lines)...
[INFO] [2021-05-31 17:38:12] Handling diff: /app/public/diff/Calcification_ra_measurements_3943.diff (118 lines)
[INFO] [2021-05-31 17:38:13] Loading measurements diff file into memory (118 /app/public/diff/Calcification_ra_measurements_3943.diff lines)...
[INFO] [2021-05-31 17:38:13] Storing 6 References
[INFO] [2021-05-31 17:38:13] Processing group of 6 in 1 groups of 1000
[INFO] [2021-05-31 17:38:13] Average Time: 0.0
[INFO] [2021-05-31 17:38:13] Total Time: 1s
[INFO] [2021-05-31 17:38:13] Storing 24 ScientificNames
[INFO] [2021-05-31 17:38:13] Processing group of 24 in 1 groups of 1000
[INFO] [2021-05-31 17:38:13] Average Time: 0.01
[INFO] [2021-05-31 17:38:13] Total Time: 1s
[INFO] [2021-05-31 17:38:13] Storing 24 Nodes
[INFO] [2021-05-31 17:38:13] Processing group of 24 in 1 groups of 1000
[INFO] [2021-05-31 17:38:13] Average Time: 0.01
[INFO] [2021-05-31 17:38:13] Total Time: 1s
[INFO] [2021-05-31 17:38:13] Storing 25 Occurrences
[INFO] [2021-05-31 17:38:13] Processing group of 25 in 1 groups of 1000
[INFO] [2021-05-31 17:38:13] Average Time: 0.0
[INFO] [2021-05-31 17:38:13] Total Time: 1s
[INFO] [2021-05-31 17:38:13] Storing 113 OccurrenceMetadata
[INFO] [2021-05-31 17:38:13] Processing group of 113 in 1 groups of 1000
[INFO] [2021-05-31 17:38:13] Average Time: 0.01
[INFO] [2021-05-31 17:38:13] Total Time: 1s
[INFO] [2021-05-31 17:38:13] Storing 8 TraitsReferences
[INFO] [2021-05-31 17:38:13] Processing group of 8 in 1 groups of 1000
[INFO] [2021-05-31 17:38:13] Average Time: 0.0
[INFO] [2021-05-31 17:38:13] Total Time: 1s
[INFO] [2021-05-31 17:38:13] Storing 26 Traits
[INFO] [2021-05-31 17:38:13] Processing group of 26 in 1 groups of 1000
[INFO] [2021-05-31 17:38:13] Average Time: 0.01
[INFO] [2021-05-31 17:38:13] Total Time: 1s
[INFO] [2021-05-31 17:38:13] Storing 51 MetaTraits
[INFO] [2021-05-31 17:38:13] Processing group of 51 in 1 groups of 1000
[INFO] [2021-05-31 17:38:13] Average Time: 0.01
[INFO] [2021-05-31 17:38:13] Total Time: 1s
[STOP] [2021-05-31 17:38:13] parse_diff_and_store
[START] [2021-05-31 17:38:13] resolve_keys
[INFO] [2021-05-31 17:38:19] Occurrences to nodes (through scientific_names)...
[INFO] [2021-05-31 17:38:19] traits to occurrences...
[INFO] [2021-05-31 17:38:19] traits to nodes (through occurrences)...
[INFO] [2021-05-31 17:38:19] Traits to sex term...
[INFO] [2021-05-31 17:38:19] Traits to lifestage term...
[INFO] [2021-05-31 17:38:19] MetaTraits to traits...
[INFO] [2021-05-31 17:38:19] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2021-05-31 17:38:19] Assocs to occurrences...
[INFO] [2021-05-31 17:38:19] Assocs to nodes...
[INFO] [2021-05-31 17:38:19] Assoc to sex term...
[INFO] [2021-05-31 17:38:19] Assoc to lifestage term...
[INFO] [2021-05-31 17:38:19] MetaAssoc to assocs...
[STOP] [2021-05-31 17:38:19] resolve_keys
[START] [2021-05-31 17:38:19] hold_for_later_1
[STOP] [2021-05-31 17:38:19] hold_for_later_1
[START] [2021-05-31 17:38:19] hold_for_later_2
[STOP] [2021-05-31 17:38:19] hold_for_later_2
[START] [2021-05-31 17:38:19] resolve_missing_parents
[STOP] [2021-05-31 17:38:19] resolve_missing_parents
[START] [2021-05-31 17:38:19] rebuild_nodes
[START] [2021-05-31 17:38:19] Flattener#flatten
[START] [2021-05-31 17:38:19] Flattener#study_resource
[START] [2021-05-31 17:38:19] Flattener#build_ancestry
[STOP] [2021-05-31 17:38:19] Flattener#build_ancestry
[INFO] [2021-05-31 17:38:19] 24 ancestry keys
[START] [2021-05-31 17:38:19] build_node_ancestors
[INFO] [2021-05-31 17:38:19] old ancestors deleted.
[STOP] [2021-05-31 17:38:19] build_node_ancestors
[START] [2021-05-31 17:38:19] Flattener#propagate_ancestor_ids
[STOP] [2021-05-31 17:38:19] Flattener#propagate_ancestor_ids
[STOP] [2021-05-31 17:38:19] Flattener#flatten
[STOP] [2021-05-31 17:38:19] rebuild_nodes
[START] [2021-05-31 17:38:19] resolve_missing_media_owners
[STOP] [2021-05-31 17:38:19] resolve_missing_media_owners
[START] [2021-05-31 17:38:19] sanitize_media_verbatims
[STOP] [2021-05-31 17:38:19] sanitize_media_verbatims
[START] [2021-05-31 17:38:19] queue_downloads
[STOP] [2021-05-31 17:38:19] queue_downloads
[START] [2021-05-31 17:38:19] parse_names
[WARN] [2021-05-31 17:38:19] I see 24 names which still need to be parsed.
[WARN] [2021-05-31 17:38:20] I see 1 names which still need to be parsed.
[STOP] [2021-05-31 17:38:21] parse_names
[START] [2021-05-31 17:38:21] denormalize_canonical_names_to_nodes
[STOP] [2021-05-31 17:38:21] denormalize_canonical_names_to_nodes
[START] [2021-05-31 17:38:21] match_nodes
[START] [2021-05-31 17:38:21] map_all_nodes_to_pages
[STOP] [2021-05-31 17:38:23] map_all_nodes_to_pages
[INFO] [2021-05-31 17:38:23] ZERO unmatched nodes (of 24)! Nicely done.
[START] [2021-05-31 17:38:23] update_nodes
[STOP] [2021-05-31 17:38:23] update_nodes
[STOP] [2021-05-31 17:38:23] match_nodes
[START] [2021-05-31 17:38:23] reindex_search
[STOP] [2021-05-31 17:38:23] reindex_search
[START] [2021-05-31 17:38:23] normalize_units
[STOP] [2021-05-31 17:38:23] normalize_units
[START] [2021-05-31 17:38:23] calculate_statistics
[STOP] [2021-05-31 17:38:23] calculate_statistics
[START] [2021-05-31 17:38:23] complete_harvest_instance
[START] [2021-05-31 17:38:23] overall_tsv_creation
[INFO] [2021-05-31 17:38:23] Processing group of 24 in 1 batches of 10000
[INFO] [2021-05-31 17:38:56] 26 Traits (unfiltered)...
[INFO] [2021-05-31 17:39:22] 26 Traits (filtered)...
[INFO] [2021-05-31 17:39:22] 0 Associations (filtered)...
[INFO] [2021-05-31 17:39:22] 112 metadata added.
[INFO] [2021-05-31 17:39:22] 0 metadata added.
[INFO] [2021-05-31 17:39:46] Average Time: 60.48
[INFO] [2021-05-31 17:39:46] Total Time: 1m23s
[STOP] [2021-05-31 17:39:46] overall_tsv_creation
[INFO] [2021-05-31 17:39:46] Done. Check your files:
[INFO] [2021-05-31 17:39:46] (23 lines) /app/public/data/Calcification_ra/publish_nodes.tsv
[INFO] [2021-05-31 17:39:46] (82 lines) /app/public/data/Calcification_ra/publish_node_ancestors.tsv
[INFO] [2021-05-31 17:39:47] (24 lines) /app/public/data/Calcification_ra/publish_scientific_names.tsv
[INFO] [2021-05-31 17:39:47] (27 lines) /app/public/data/Calcification_ra/publish_traits.tsv
[INFO] [2021-05-31 17:39:47] (113 lines) /app/public/data/Calcification_ra/publish_metadata.tsv
[STOP] [2021-05-31 17:39:47] complete_harvest_instance
[START] [2021-05-31 17:39:47] completed
[STOP] [2021-05-31 17:39:47] completed
[STOP] [2021-05-31 17:39:47] logged process, took 113.06

Latest Process