Harvest for FAOSTAT Created 09 Jul 15:13

Harvesting Log

(1119 lines) (showing only the last 1000 lines, see /app/public/data/FAOSTAT/process.log for the full file)

[START] [2020-06-19 12:02:36] reindex_search
[STOP] [2020-06-19 12:02:36] reindex_search
[START] [2020-06-19 12:02:36] normalize_units
[STOP] [2020-06-19 12:03:30] normalize_units
[START] [2020-06-19 12:03:30] calculate_statistics
[STOP] [2020-06-19 12:03:30] calculate_statistics
[START] [2020-06-19 12:03:30] complete_harvest_instance
[START] [2020-06-19 12:03:30] overall_tsv_creation
[INFO] [2020-06-19 12:03:30] Processing group of 150 in 1 batches of 10000
[INFO] [2020-06-19 12:05:40] 13202 Traits (unfiltered)...
[INFO] [2020-06-19 12:05:53] 13202 Traits (filtered)...
[INFO] [2020-06-19 12:05:53] 0 Associations (filtered)...
[INFO] [2020-06-19 12:06:43] 52808 metadata added.
[INFO] [2020-06-19 12:06:43] 0 metadata added.
[INFO] [2020-06-19 12:06:43] Average Time: 91.82
[INFO] [2020-06-19 12:06:43] Total Time: 3m14s
[STOP] [2020-06-19 12:06:43] overall_tsv_creation
[INFO] [2020-06-19 12:06:43] Done. Check your files:
[INFO] [2020-06-19 12:06:43] (150 lines) /app/public/data/FAOSTAT/publish_nodes.tsv
[INFO] [2020-06-19 12:06:43] (350 lines) /app/public/data/FAOSTAT/publish_node_ancestors.tsv
[INFO] [2020-06-19 12:06:43] (150 lines) /app/public/data/FAOSTAT/publish_scientific_names.tsv
[INFO] [2020-06-19 12:06:43] (13203 lines) /app/public/data/FAOSTAT/publish_traits.tsv
[INFO] [2020-06-19 12:06:43] (52809 lines) /app/public/data/FAOSTAT/publish_metadata.tsv
[STOP] [2020-06-19 12:06:43] complete_harvest_instance
[START] [2020-06-19 12:06:43] completed
[STOP] [2020-06-19 12:06:43] completed
[STOP] [2020-06-19 12:06:43] logged process, took 422.23
[INFO] [2020-06-19 12:28:13] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-06-19 12:28:15] ## remove_type: ScientificName
[INFO] [2020-06-19 12:28:15] ++ Calling delete_all on 150 instances...
[INFO] [2020-06-19 12:28:15] [12:28:15.122] Removed 150 Scientificnames
[INFO] [2020-06-19 12:28:15] ## remove_type: Vernacular
[INFO] [2020-06-19 12:28:15] ++ Calling delete_all on 0 instances...
[INFO] [2020-06-19 12:28:15] [12:28:15.124] Removed 0 Vernaculars
[INFO] [2020-06-19 12:28:15] ## remove_type: Article
[INFO] [2020-06-19 12:28:15] ++ Calling delete_all on 0 instances...
[INFO] [2020-06-19 12:28:15] [12:28:15.127] Removed 0 Articles
[INFO] [2020-06-19 12:28:15] ## remove_type: Medium
[INFO] [2020-06-19 12:28:15] ++ Calling delete_all on 0 instances...
[INFO] [2020-06-19 12:28:15] [12:28:15.130] Removed 0 Media
[INFO] [2020-06-19 12:28:15] ## remove_type: Trait
[INFO] [2020-06-19 12:28:15] ++ Calling delete_all on 26404 instances...
[INFO] [2020-06-19 12:28:18] [12:28:18.513] Removed 26404 Traits
[INFO] [2020-06-19 12:28:18] ## remove_type: MetaTrait
[INFO] [2020-06-19 12:28:18] ++ Calling delete_all on 39606 instances...
[INFO] [2020-06-19 12:28:19] [12:28:19.012] Removed 39606 Metatraits
[INFO] [2020-06-19 12:28:19] ## remove_type: OccurrenceMetadatum
[INFO] [2020-06-19 12:28:19] ++ Calling delete_all on 0 instances...
[INFO] [2020-06-19 12:28:19] [12:28:19.015] Removed 0 Occurrencemetadata
[INFO] [2020-06-19 12:28:19] ## remove_type: Assoc
[INFO] [2020-06-19 12:28:19] ++ Calling delete_all on 0 instances...
[INFO] [2020-06-19 12:28:19] [12:28:19.017] Removed 0 Assocs
[INFO] [2020-06-19 12:28:19] ## remove_type: MetaAssoc
[INFO] [2020-06-19 12:28:19] ++ Calling delete_all on 0 instances...
[INFO] [2020-06-19 12:28:19] [12:28:19.020] Removed 0 Metaassocs
[INFO] [2020-06-19 12:28:19] ## remove_type: Identifier
[INFO] [2020-06-19 12:28:19] ++ Calling delete_all on 0 instances...
[INFO] [2020-06-19 12:28:19] [12:28:19.023] Removed 0 Identifiers
[INFO] [2020-06-19 12:28:19] ## remove_type: Reference
[INFO] [2020-06-19 12:28:19] ++ Calling delete_all on 1 instances...
[INFO] [2020-06-19 12:28:19] [12:28:19.025] Removed 1 References
[INFO] [2020-06-19 12:28:19] Starting batch with ID 80104458...
[INFO] [2020-06-19 12:28:19] Starting batch with ID 80104458...
[INFO] [2020-06-19 12:28:19] Starting batch with ID 80104458...
[INFO] [2020-06-19 12:28:19] Starting batch with ID 80104458...
[INFO] [2020-06-19 12:28:19] Starting batch with ID 80104458...
[INFO] [2020-06-19 12:28:19] Starting batch with ID 80104458...
[INFO] [2020-06-19 12:28:19] Starting batch with ID 80104458...
[INFO] [2020-06-19 12:28:19] Starting batch with ID 80104458...
[INFO] [2020-06-19 12:28:19] Starting batch with ID 80104458...
[INFO] [2020-06-19 12:28:19] Starting batch with ID 80104458...
[INFO] [2020-06-19 12:28:19] Starting batch with ID 80104458...
[INFO] [2020-06-19 12:28:19] Starting batch with ID 80104458...
[INFO] [2020-06-19 12:28:19] Starting batch with ID 80104458...
[INFO] [2020-06-19 12:28:19] Starting batch with ID 80104458...
[INFO] [2020-06-19 12:28:19] Starting batch with ID 80104458...
[INFO] [2020-06-19 12:28:19] Starting batch with ID 80104458...
[INFO] [2020-06-19 12:28:19] Starting batch with ID 80104458...
[INFO] [2020-06-19 12:28:19] Starting batch with ID 80104458...
[INFO] [2020-06-19 12:28:19] Starting batch with ID 80104458...
[INFO] [2020-06-19 12:28:20] Starting batch with ID 80104458...
[INFO] [2020-06-19 12:28:20] Starting batch with ID 80104465...
[INFO] [2020-06-19 12:28:21] Starting batch with ID 80104463...
[INFO] [2020-06-19 12:28:21] ## remove_type: Node
[INFO] [2020-06-19 12:28:21] ++ Calling delete_all on 150 instances...
[INFO] [2020-06-19 12:28:21] [12:28:21.169] Removed 150 Nodes
[START] [2020-06-19 12:28:21] logged process
[START] [2020-06-19 12:28:21] Creating resource from OpenData
[START] [2020-06-19 12:28:21] logged process
[START] [2020-06-19 12:28:21] Parse meta.xml file and create formats with fields
[STOP] [2020-06-19 12:28:21] Parse meta.xml file and create formats with fields
[STOP] [2020-06-19 12:28:21] Creating resource from OpenData
[START] [2020-06-19 12:28:22] logged process
[START] [2020-06-19 12:28:22] create_harvest_instance
[STOP] [2020-06-19 12:28:23] create_harvest_instance
[START] [2020-06-19 12:28:23] fetch_files
[STOP] [2020-06-19 12:28:23] fetch_files
[START] [2020-06-19 12:28:23] validate_each_file
[STOP] [2020-06-19 12:28:24] validate_each_file
[START] [2020-06-19 12:28:24] convert_to_csv
[CMD] [2020-06-19 12:28:24] /usr/bin/sort /app/public/converted_csv/FAOSTAT_refs_21391.csv > /app/public/converted_csv/FAOSTAT_refs_21391.csv_sorted
[CMD] [2020-06-19 12:28:24] /usr/bin/sort /app/public/converted_csv/FAOSTAT_nodes_21392.csv > /app/public/converted_csv/FAOSTAT_nodes_21392.csv_sorted
[CMD] [2020-06-19 12:28:24] /usr/bin/sort /app/public/converted_csv/FAOSTAT_occurrences_21393.csv > /app/public/converted_csv/FAOSTAT_occurrences_21393.csv_sorted
[CMD] [2020-06-19 12:28:24] /usr/bin/sort /app/public/converted_csv/FAOSTAT_measurements_21394.csv > /app/public/converted_csv/FAOSTAT_measurements_21394.csv_sorted
[STOP] [2020-06-19 12:28:24] convert_to_csv
[START] [2020-06-19 12:28:24] calculate_delta
[CMD] [2020-06-19 12:28:24] echo "0a" > /app/public/diff/FAOSTAT_refs_21391.diff
[CMD] [2020-06-19 12:28:24] tail -n +1 /app/public/converted_csv/FAOSTAT_refs_21391.csv >> /app/public/diff/FAOSTAT_refs_21391.diff
[CMD] [2020-06-19 12:28:24] echo "." >> /app/public/diff/FAOSTAT_refs_21391.diff
[CMD] [2020-06-19 12:28:24] echo "0a" > /app/public/diff/FAOSTAT_nodes_21392.diff
[CMD] [2020-06-19 12:28:24] tail -n +1 /app/public/converted_csv/FAOSTAT_nodes_21392.csv >> /app/public/diff/FAOSTAT_nodes_21392.diff
[CMD] [2020-06-19 12:28:24] echo "." >> /app/public/diff/FAOSTAT_nodes_21392.diff
[CMD] [2020-06-19 12:28:24] echo "0a" > /app/public/diff/FAOSTAT_occurrences_21393.diff
[CMD] [2020-06-19 12:28:24] tail -n +1 /app/public/converted_csv/FAOSTAT_occurrences_21393.csv >> /app/public/diff/FAOSTAT_occurrences_21393.diff
[CMD] [2020-06-19 12:28:24] echo "." >> /app/public/diff/FAOSTAT_occurrences_21393.diff
[CMD] [2020-06-19 12:28:24] echo "0a" > /app/public/diff/FAOSTAT_measurements_21394.diff
[CMD] [2020-06-19 12:28:24] tail -n +1 /app/public/converted_csv/FAOSTAT_measurements_21394.csv >> /app/public/diff/FAOSTAT_measurements_21394.diff
[CMD] [2020-06-19 12:28:24] echo "." >> /app/public/diff/FAOSTAT_measurements_21394.diff
[STOP] [2020-06-19 12:28:24] calculate_delta
[START] [2020-06-19 12:28:24] parse_diff_and_store
[INFO] [2020-06-19 12:28:24] Loading refs diff file into memory (true lines)...
[INFO] [2020-06-19 12:28:24] Loading nodes diff file into memory (true lines)...
[INFO] [2020-06-19 12:28:24] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-06-19 12:28:24] Loading measurements diff file into memory (true lines)...
[INFO] [2020-06-19 12:30:15] Storing 1 References
[INFO] [2020-06-19 12:30:15] Processing group of 1 in 1 groups of 1000
[INFO] [2020-06-19 12:30:15] Average Time: 0.0
[INFO] [2020-06-19 12:30:15] Total Time: 1s
[INFO] [2020-06-19 12:30:15] Storing 150 ScientificNames
[INFO] [2020-06-19 12:30:15] Processing group of 150 in 1 groups of 1000
[INFO] [2020-06-19 12:30:15] Average Time: 0.08
[INFO] [2020-06-19 12:30:15] Total Time: 1s
[INFO] [2020-06-19 12:30:15] Storing 150 Nodes
[INFO] [2020-06-19 12:30:15] Processing group of 150 in 1 groups of 1000
[INFO] [2020-06-19 12:30:15] Average Time: 0.04
[INFO] [2020-06-19 12:30:15] Total Time: 1s
[INFO] [2020-06-19 12:30:15] Storing 134 Occurrences
[INFO] [2020-06-19 12:30:15] Processing group of 134 in 1 groups of 1000
[INFO] [2020-06-19 12:30:15] Average Time: 0.01
[INFO] [2020-06-19 12:30:15] Total Time: 1s
[INFO] [2020-06-19 12:30:15] Storing 13202 TraitsReferences
[INFO] [2020-06-19 12:30:15] Processing group of 13202 in 14 groups of 1000
[INFO] [2020-06-19 12:30:16] Average Time: 0.062
[INFO] [2020-06-19 12:30:16] Total Time: 1s
[INFO] [2020-06-19 12:30:16] last 3 / first 3: 0.54
[INFO] [2020-06-19 12:30:16] Std.Dev: 0.0; Max: 0.12
[INFO] [2020-06-19 12:30:16] Storing 26404 Traits
[INFO] [2020-06-19 12:30:16] Processing group of 26404 in 27 groups of 1000
[INFO] [2020-06-19 12:30:23] Average Time: 0.247
[INFO] [2020-06-19 12:30:23] Total Time: 7s
[INFO] [2020-06-19 12:30:23] last 3 / first 3: 1.1
[INFO] [2020-06-19 12:30:23] Std.Dev: 0.044721359549995794; Max: 0.43
[INFO] [2020-06-19 12:30:23] Storing 39606 MetaTraits
[INFO] [2020-06-19 12:30:23] Processing group of 39606 in 40 groups of 1000
[INFO] [2020-06-19 12:30:26] Average Time: 0.094
[INFO] [2020-06-19 12:30:26] Total Time: 4s
[INFO] [2020-06-19 12:30:26] last 3 / first 3: 0.51
[INFO] [2020-06-19 12:30:26] Std.Dev: 0.03162277660168379; Max: 0.22
[STOP] [2020-06-19 12:30:26] parse_diff_and_store
[START] [2020-06-19 12:30:26] resolve_keys
[INFO] [2020-06-19 12:30:34] Occurrences to nodes (through scientific_names)...
[INFO] [2020-06-19 12:30:34] traits to occurrences...
[INFO] [2020-06-19 12:30:34] traits to nodes (through occurrences)...
[INFO] [2020-06-19 12:30:35] Traits to sex term...
[INFO] [2020-06-19 12:30:35] Traits to lifestage term...
[INFO] [2020-06-19 12:30:35] MetaTraits to traits...
[INFO] [2020-06-19 12:30:36] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-06-19 12:30:37] Assocs to occurrences...
[INFO] [2020-06-19 12:30:37] Assocs to nodes...
[INFO] [2020-06-19 12:30:37] Assoc to sex term...
[INFO] [2020-06-19 12:30:37] Assoc to lifestage term...
[STOP] [2020-06-19 12:30:37] resolve_keys
[START] [2020-06-19 12:30:37] hold_for_later_1
[STOP] [2020-06-19 12:30:37] hold_for_later_1
[START] [2020-06-19 12:30:37] hold_for_later_2
[STOP] [2020-06-19 12:30:37] hold_for_later_2
[START] [2020-06-19 12:30:37] resolve_missing_parents
[STOP] [2020-06-19 12:30:37] resolve_missing_parents
[START] [2020-06-19 12:30:37] rebuild_nodes
[START] [2020-06-19 12:30:37] Flattener#flatten
[START] [2020-06-19 12:30:37] Flattener#study_resource
[START] [2020-06-19 12:30:37] Flattener#build_ancestry
[STOP] [2020-06-19 12:30:37] Flattener#build_ancestry
[INFO] [2020-06-19 12:30:37] 150 ancestry keys
[START] [2020-06-19 12:30:37] build_node_ancestors
[INFO] [2020-06-19 12:30:37] old ancestors deleted.
[STOP] [2020-06-19 12:30:37] build_node_ancestors
[START] [2020-06-19 12:30:37] Flattener#propagate_ancestor_ids
[STOP] [2020-06-19 12:30:37] Flattener#propagate_ancestor_ids
[STOP] [2020-06-19 12:30:37] Flattener#flatten
[STOP] [2020-06-19 12:30:37] rebuild_nodes
[START] [2020-06-19 12:30:37] resolve_missing_media_owners
[STOP] [2020-06-19 12:30:37] resolve_missing_media_owners
[START] [2020-06-19 12:30:37] sanitize_media_verbatims
[STOP] [2020-06-19 12:30:37] sanitize_media_verbatims
[START] [2020-06-19 12:30:37] queue_downloads
[STOP] [2020-06-19 12:30:37] queue_downloads
[START] [2020-06-19 12:30:37] parse_names
[WARN] [2020-06-19 12:30:37] I see 150 names which still need to be parsed.
[STOP] [2020-06-19 12:30:38] parse_names
[START] [2020-06-19 12:30:38] denormalize_canonical_names_to_nodes
[STOP] [2020-06-19 12:30:38] denormalize_canonical_names_to_nodes
[START] [2020-06-19 12:30:38] match_nodes
[START] [2020-06-19 12:30:38] map_all_nodes_to_pages
[STOP] [2020-06-19 12:30:47] map_all_nodes_to_pages
[INFO] [2020-06-19 12:30:47] ZERO unmatched nodes (of 150)! Nicely done.
[START] [2020-06-19 12:30:47] update_nodes
[STOP] [2020-06-19 12:30:47] update_nodes
[STOP] [2020-06-19 12:30:47] match_nodes
[START] [2020-06-19 12:30:47] reindex_search
[STOP] [2020-06-19 12:30:48] reindex_search
[START] [2020-06-19 12:30:48] normalize_units
[STOP] [2020-06-19 12:31:40] normalize_units
[START] [2020-06-19 12:31:40] calculate_statistics
[STOP] [2020-06-19 12:31:40] calculate_statistics
[START] [2020-06-19 12:31:40] complete_harvest_instance
[START] [2020-06-19 12:31:40] overall_tsv_creation
[INFO] [2020-06-19 12:31:41] Processing group of 150 in 1 batches of 10000
[INFO] [2020-06-19 12:32:45] 13202 Traits (unfiltered)...
[INFO] [2020-06-19 12:32:58] 13202 Traits (filtered)...
[INFO] [2020-06-19 12:32:58] 0 Associations (filtered)...
[INFO] [2020-06-19 12:33:48] 52808 metadata added.
[INFO] [2020-06-19 12:33:48] 0 metadata added.
[INFO] [2020-06-19 12:33:48] Average Time: 86.81
[INFO] [2020-06-19 12:33:48] Total Time: 2m8s
[STOP] [2020-06-19 12:33:48] overall_tsv_creation
[INFO] [2020-06-19 12:33:48] Done. Check your files:
[INFO] [2020-06-19 12:33:48] (150 lines) /app/public/data/FAOSTAT/publish_nodes.tsv
[INFO] [2020-06-19 12:33:48] (350 lines) /app/public/data/FAOSTAT/publish_node_ancestors.tsv
[INFO] [2020-06-19 12:33:48] (150 lines) /app/public/data/FAOSTAT/publish_scientific_names.tsv
[INFO] [2020-06-19 12:33:48] (13203 lines) /app/public/data/FAOSTAT/publish_traits.tsv
[INFO] [2020-06-19 12:33:48] (52809 lines) /app/public/data/FAOSTAT/publish_metadata.tsv
[STOP] [2020-06-19 12:33:48] complete_harvest_instance
[START] [2020-06-19 12:33:48] completed
[STOP] [2020-06-19 12:33:48] completed
[STOP] [2020-06-19 12:33:48] logged process, took 326.87
[INFO] [2020-06-19 12:50:55] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-06-19 12:50:57] ## remove_type: ScientificName
[INFO] [2020-06-19 12:50:57] ++ Calling delete_all on 150 instances...
[INFO] [2020-06-19 12:50:57] [12:50:57.037] Removed 150 Scientificnames
[INFO] [2020-06-19 12:50:57] ## remove_type: Vernacular
[INFO] [2020-06-19 12:50:57] ++ Calling delete_all on 0 instances...
[INFO] [2020-06-19 12:50:57] [12:50:57.040] Removed 0 Vernaculars
[INFO] [2020-06-19 12:50:57] ## remove_type: Article
[INFO] [2020-06-19 12:50:57] ++ Calling delete_all on 0 instances...
[INFO] [2020-06-19 12:50:57] [12:50:57.043] Removed 0 Articles
[INFO] [2020-06-19 12:50:57] ## remove_type: Medium
[INFO] [2020-06-19 12:50:57] ++ Calling delete_all on 0 instances...
[INFO] [2020-06-19 12:50:57] [12:50:57.051] Removed 0 Media
[INFO] [2020-06-19 12:50:57] ## remove_type: Trait
[INFO] [2020-06-19 12:50:57] ++ Calling delete_all on 26404 instances...
[INFO] [2020-06-19 12:51:00] [12:51:00.853] Removed 26404 Traits
[INFO] [2020-06-19 12:51:00] ## remove_type: MetaTrait
[INFO] [2020-06-19 12:51:00] ++ Calling delete_all on 39606 instances...
[INFO] [2020-06-19 12:51:02] [12:51:02.558] Removed 39606 Metatraits
[INFO] [2020-06-19 12:51:02] ## remove_type: OccurrenceMetadatum
[INFO] [2020-06-19 12:51:02] ++ Calling delete_all on 0 instances...
[INFO] [2020-06-19 12:51:02] [12:51:02.561] Removed 0 Occurrencemetadata
[INFO] [2020-06-19 12:51:02] ## remove_type: Assoc
[INFO] [2020-06-19 12:51:02] ++ Calling delete_all on 0 instances...
[INFO] [2020-06-19 12:51:02] [12:51:02.564] Removed 0 Assocs
[INFO] [2020-06-19 12:51:02] ## remove_type: MetaAssoc
[INFO] [2020-06-19 12:51:02] ++ Calling delete_all on 0 instances...
[INFO] [2020-06-19 12:51:02] [12:51:02.566] Removed 0 Metaassocs
[INFO] [2020-06-19 12:51:02] ## remove_type: Identifier
[INFO] [2020-06-19 12:51:02] ++ Calling delete_all on 0 instances...
[INFO] [2020-06-19 12:51:02] [12:51:02.568] Removed 0 Identifiers
[INFO] [2020-06-19 12:51:02] ## remove_type: Reference
[INFO] [2020-06-19 12:51:02] ++ Calling delete_all on 1 instances...
[INFO] [2020-06-19 12:51:02] [12:51:02.571] Removed 1 References
[INFO] [2020-06-19 12:51:02] Starting batch with ID 80104642...
[INFO] [2020-06-19 12:51:02] Starting batch with ID 80104642...
[INFO] [2020-06-19 12:51:02] Starting batch with ID 80104642...
[INFO] [2020-06-19 12:51:02] Starting batch with ID 80104642...
[INFO] [2020-06-19 12:51:02] Starting batch with ID 80104642...
[INFO] [2020-06-19 12:51:03] Starting batch with ID 80104642...
[INFO] [2020-06-19 12:51:03] ## remove_type: Node
[INFO] [2020-06-19 12:51:03] ++ Calling delete_all on 150 instances...
[INFO] [2020-06-19 12:51:03] [12:51:03.165] Removed 150 Nodes
[START] [2020-06-19 12:51:03] logged process
[START] [2020-06-19 12:51:03] Creating resource from OpenData
[START] [2020-06-19 12:51:03] logged process
[START] [2020-06-19 12:51:03] Parse meta.xml file and create formats with fields
[STOP] [2020-06-19 12:51:03] Parse meta.xml file and create formats with fields
[STOP] [2020-06-19 12:51:03] Creating resource from OpenData
[START] [2020-06-19 12:51:03] logged process
[START] [2020-06-19 12:51:03] create_harvest_instance
[STOP] [2020-06-19 12:51:05] create_harvest_instance
[START] [2020-06-19 12:51:05] fetch_files
[STOP] [2020-06-19 12:51:05] fetch_files
[START] [2020-06-19 12:51:05] validate_each_file
[STOP] [2020-06-19 12:51:06] validate_each_file
[START] [2020-06-19 12:51:06] convert_to_csv
[CMD] [2020-06-19 12:51:06] /usr/bin/sort /app/public/converted_csv/FAOSTAT_refs_21399.csv > /app/public/converted_csv/FAOSTAT_refs_21399.csv_sorted
[CMD] [2020-06-19 12:51:06] /usr/bin/sort /app/public/converted_csv/FAOSTAT_nodes_21400.csv > /app/public/converted_csv/FAOSTAT_nodes_21400.csv_sorted
[CMD] [2020-06-19 12:51:06] /usr/bin/sort /app/public/converted_csv/FAOSTAT_occurrences_21401.csv > /app/public/converted_csv/FAOSTAT_occurrences_21401.csv_sorted
[CMD] [2020-06-19 12:51:06] /usr/bin/sort /app/public/converted_csv/FAOSTAT_measurements_21402.csv > /app/public/converted_csv/FAOSTAT_measurements_21402.csv_sorted
[STOP] [2020-06-19 12:51:06] convert_to_csv
[START] [2020-06-19 12:51:06] calculate_delta
[CMD] [2020-06-19 12:51:06] echo "0a" > /app/public/diff/FAOSTAT_refs_21399.diff
[CMD] [2020-06-19 12:51:06] tail -n +1 /app/public/converted_csv/FAOSTAT_refs_21399.csv >> /app/public/diff/FAOSTAT_refs_21399.diff
[CMD] [2020-06-19 12:51:06] echo "." >> /app/public/diff/FAOSTAT_refs_21399.diff
[CMD] [2020-06-19 12:51:06] echo "0a" > /app/public/diff/FAOSTAT_nodes_21400.diff
[CMD] [2020-06-19 12:51:06] tail -n +1 /app/public/converted_csv/FAOSTAT_nodes_21400.csv >> /app/public/diff/FAOSTAT_nodes_21400.diff
[CMD] [2020-06-19 12:51:06] echo "." >> /app/public/diff/FAOSTAT_nodes_21400.diff
[CMD] [2020-06-19 12:51:06] echo "0a" > /app/public/diff/FAOSTAT_occurrences_21401.diff
[CMD] [2020-06-19 12:51:06] tail -n +1 /app/public/converted_csv/FAOSTAT_occurrences_21401.csv >> /app/public/diff/FAOSTAT_occurrences_21401.diff
[CMD] [2020-06-19 12:51:06] echo "." >> /app/public/diff/FAOSTAT_occurrences_21401.diff
[CMD] [2020-06-19 12:51:06] echo "0a" > /app/public/diff/FAOSTAT_measurements_21402.diff
[CMD] [2020-06-19 12:51:06] tail -n +1 /app/public/converted_csv/FAOSTAT_measurements_21402.csv >> /app/public/diff/FAOSTAT_measurements_21402.diff
[CMD] [2020-06-19 12:51:06] echo "." >> /app/public/diff/FAOSTAT_measurements_21402.diff
[STOP] [2020-06-19 12:51:06] calculate_delta
[START] [2020-06-19 12:51:06] parse_diff_and_store
[INFO] [2020-06-19 12:51:06] Loading refs diff file into memory (true lines)...
[INFO] [2020-06-19 12:51:06] Loading nodes diff file into memory (true lines)...
[INFO] [2020-06-19 12:51:06] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-06-19 12:51:06] Loading measurements diff file into memory (true lines)...
[INFO] [2020-06-19 12:52:53] Storing 1 References
[INFO] [2020-06-19 12:52:53] Processing group of 1 in 1 groups of 1000
[INFO] [2020-06-19 12:52:53] Average Time: 0.0
[INFO] [2020-06-19 12:52:53] Total Time: 1s
[INFO] [2020-06-19 12:52:53] Storing 150 ScientificNames
[INFO] [2020-06-19 12:52:53] Processing group of 150 in 1 groups of 1000
[INFO] [2020-06-19 12:52:53] Average Time: 0.1
[INFO] [2020-06-19 12:52:53] Total Time: 1s
[INFO] [2020-06-19 12:52:53] Storing 150 Nodes
[INFO] [2020-06-19 12:52:53] Processing group of 150 in 1 groups of 1000
[INFO] [2020-06-19 12:52:54] Average Time: 0.04
[INFO] [2020-06-19 12:52:54] Total Time: 1s
[INFO] [2020-06-19 12:52:54] Storing 134 Occurrences
[INFO] [2020-06-19 12:52:54] Processing group of 134 in 1 groups of 1000
[INFO] [2020-06-19 12:52:54] Average Time: 0.01
[INFO] [2020-06-19 12:52:54] Total Time: 1s
[INFO] [2020-06-19 12:52:54] Storing 13202 TraitsReferences
[INFO] [2020-06-19 12:52:54] Processing group of 13202 in 14 groups of 1000
[INFO] [2020-06-19 12:52:54] Average Time: 0.063
[INFO] [2020-06-19 12:52:54] Total Time: 1s
[INFO] [2020-06-19 12:52:54] last 3 / first 3: 0.48
[INFO] [2020-06-19 12:52:54] Std.Dev: 0.03162277660168379; Max: 0.15
[INFO] [2020-06-19 12:52:54] Storing 25691 Traits
[INFO] [2020-06-19 12:52:54] Processing group of 25691 in 26 groups of 1000
[INFO] [2020-06-19 12:53:01] Average Time: 0.242
[INFO] [2020-06-19 12:53:01] Total Time: 7s
[INFO] [2020-06-19 12:53:01] last 3 / first 3: 0.92
[INFO] [2020-06-19 12:53:01] Std.Dev: 0.03162277660168379; Max: 0.3
[INFO] [2020-06-19 12:53:01] Storing 38893 MetaTraits
[INFO] [2020-06-19 12:53:01] Processing group of 38893 in 39 groups of 1000
[INFO] [2020-06-19 12:53:05] Average Time: 0.102
[INFO] [2020-06-19 12:53:05] Total Time: 5s
[INFO] [2020-06-19 12:53:05] last 3 / first 3: 0.89
[INFO] [2020-06-19 12:53:05] Std.Dev: 0.03162277660168379; Max: 0.21
[STOP] [2020-06-19 12:53:05] parse_diff_and_store
[START] [2020-06-19 12:53:05] resolve_keys
[INFO] [2020-06-19 12:53:13] Occurrences to nodes (through scientific_names)...
[INFO] [2020-06-19 12:53:13] traits to occurrences...
[INFO] [2020-06-19 12:53:13] traits to nodes (through occurrences)...
[INFO] [2020-06-19 12:53:13] Traits to sex term...
[INFO] [2020-06-19 12:53:13] Traits to lifestage term...
[INFO] [2020-06-19 12:53:13] MetaTraits to traits...
[INFO] [2020-06-19 12:53:14] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-06-19 12:53:15] Assocs to occurrences...
[INFO] [2020-06-19 12:53:15] Assocs to nodes...
[INFO] [2020-06-19 12:53:15] Assoc to sex term...
[INFO] [2020-06-19 12:53:15] Assoc to lifestage term...
[STOP] [2020-06-19 12:53:15] resolve_keys
[START] [2020-06-19 12:53:15] hold_for_later_1
[STOP] [2020-06-19 12:53:15] hold_for_later_1
[START] [2020-06-19 12:53:15] hold_for_later_2
[STOP] [2020-06-19 12:53:15] hold_for_later_2
[START] [2020-06-19 12:53:15] resolve_missing_parents
[STOP] [2020-06-19 12:53:15] resolve_missing_parents
[START] [2020-06-19 12:53:15] rebuild_nodes
[START] [2020-06-19 12:53:15] Flattener#flatten
[START] [2020-06-19 12:53:15] Flattener#study_resource
[START] [2020-06-19 12:53:15] Flattener#build_ancestry
[STOP] [2020-06-19 12:53:15] Flattener#build_ancestry
[INFO] [2020-06-19 12:53:15] 150 ancestry keys
[START] [2020-06-19 12:53:15] build_node_ancestors
[INFO] [2020-06-19 12:53:15] old ancestors deleted.
[STOP] [2020-06-19 12:53:15] build_node_ancestors
[START] [2020-06-19 12:53:15] Flattener#propagate_ancestor_ids
[STOP] [2020-06-19 12:53:15] Flattener#propagate_ancestor_ids
[STOP] [2020-06-19 12:53:15] Flattener#flatten
[STOP] [2020-06-19 12:53:15] rebuild_nodes
[START] [2020-06-19 12:53:15] resolve_missing_media_owners
[STOP] [2020-06-19 12:53:15] resolve_missing_media_owners
[START] [2020-06-19 12:53:15] sanitize_media_verbatims
[STOP] [2020-06-19 12:53:15] sanitize_media_verbatims
[START] [2020-06-19 12:53:15] queue_downloads
[STOP] [2020-06-19 12:53:15] queue_downloads
[START] [2020-06-19 12:53:15] parse_names
[WARN] [2020-06-19 12:53:15] I see 150 names which still need to be parsed.
[STOP] [2020-06-19 12:53:17] parse_names
[START] [2020-06-19 12:53:17] denormalize_canonical_names_to_nodes
[STOP] [2020-06-19 12:53:17] denormalize_canonical_names_to_nodes
[START] [2020-06-19 12:53:17] match_nodes
[START] [2020-06-19 12:53:17] map_all_nodes_to_pages
[STOP] [2020-06-19 12:53:26] map_all_nodes_to_pages
[INFO] [2020-06-19 12:53:26] ZERO unmatched nodes (of 150)! Nicely done.
[START] [2020-06-19 12:53:26] update_nodes
[STOP] [2020-06-19 12:53:26] update_nodes
[STOP] [2020-06-19 12:53:26] match_nodes
[START] [2020-06-19 12:53:26] reindex_search
[STOP] [2020-06-19 12:53:26] reindex_search
[START] [2020-06-19 12:53:26] normalize_units
[STOP] [2020-06-19 12:54:19] normalize_units
[START] [2020-06-19 12:54:19] calculate_statistics
[STOP] [2020-06-19 12:54:19] calculate_statistics
[START] [2020-06-19 12:54:19] complete_harvest_instance
[START] [2020-06-19 12:54:19] overall_tsv_creation
[INFO] [2020-06-19 12:54:19] Processing group of 150 in 1 batches of 10000
[INFO] [2020-06-19 12:55:11] 13202 Traits (unfiltered)...
[INFO] [2020-06-19 12:55:24] 13202 Traits (filtered)...
[INFO] [2020-06-19 12:55:24] 0 Associations (filtered)...
[INFO] [2020-06-19 12:56:12] 52095 metadata added.
[INFO] [2020-06-19 12:56:12] 0 metadata added.
[INFO] [2020-06-19 12:56:12] Average Time: 84.98
[INFO] [2020-06-19 12:56:12] Total Time: 1m54s
[STOP] [2020-06-19 12:56:12] overall_tsv_creation
[INFO] [2020-06-19 12:56:12] Done. Check your files:
[INFO] [2020-06-19 12:56:12] (150 lines) /app/public/data/FAOSTAT/publish_nodes.tsv
[INFO] [2020-06-19 12:56:12] (350 lines) /app/public/data/FAOSTAT/publish_node_ancestors.tsv
[INFO] [2020-06-19 12:56:12] (150 lines) /app/public/data/FAOSTAT/publish_scientific_names.tsv
[INFO] [2020-06-19 12:56:12] (13203 lines) /app/public/data/FAOSTAT/publish_traits.tsv
[INFO] [2020-06-19 12:56:13] (52096 lines) /app/public/data/FAOSTAT/publish_metadata.tsv
[STOP] [2020-06-19 12:56:13] complete_harvest_instance
[START] [2020-06-19 12:56:13] completed
[STOP] [2020-06-19 12:56:13] completed
[STOP] [2020-06-19 12:56:13] logged process, took 309.16
[INFO] [2020-07-07 14:58:47] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-07-07 14:58:49] ## remove_type: ScientificName
[INFO] [2020-07-07 14:58:49] ++ Calling delete_all on 150 instances...
[INFO] [2020-07-07 14:58:49] [14:58:49.796] Removed 150 Scientificnames
[INFO] [2020-07-07 14:58:49] ## remove_type: Vernacular
[INFO] [2020-07-07 14:58:49] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 14:58:49] [14:58:49.800] Removed 0 Vernaculars
[INFO] [2020-07-07 14:58:49] ## remove_type: Article
[INFO] [2020-07-07 14:58:49] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 14:58:49] [14:58:49.803] Removed 0 Articles
[INFO] [2020-07-07 14:58:49] ## remove_type: Medium
[INFO] [2020-07-07 14:58:49] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 14:58:49] [14:58:49.807] Removed 0 Media
[INFO] [2020-07-07 14:58:49] ## remove_type: Trait
[INFO] [2020-07-07 14:58:49] ++ Calling delete_all on 25691 instances...
[INFO] [2020-07-07 14:58:54] [14:58:54.019] Removed 25691 Traits
[INFO] [2020-07-07 14:58:54] ## remove_type: MetaTrait
[INFO] [2020-07-07 14:58:54] ++ Calling delete_all on 38893 instances...
[INFO] [2020-07-07 14:58:55] [14:58:55.896] Removed 38893 Metatraits
[INFO] [2020-07-07 14:58:55] ## remove_type: OccurrenceMetadatum
[INFO] [2020-07-07 14:58:55] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 14:58:55] [14:58:55.904] Removed 0 Occurrencemetadata
[INFO] [2020-07-07 14:58:55] ## remove_type: Assoc
[INFO] [2020-07-07 14:58:55] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 14:58:55] [14:58:55.906] Removed 0 Assocs
[INFO] [2020-07-07 14:58:55] ## remove_type: MetaAssoc
[INFO] [2020-07-07 14:58:55] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 14:58:55] [14:58:55.914] Removed 0 Metaassocs
[INFO] [2020-07-07 14:58:55] ## remove_type: Identifier
[INFO] [2020-07-07 14:58:55] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 14:58:55] [14:58:55.928] Removed 0 Identifiers
[INFO] [2020-07-07 14:58:55] ## remove_type: Reference
[INFO] [2020-07-07 14:58:55] ++ Calling delete_all on 1 instances...
[INFO] [2020-07-07 14:58:55] [14:58:55.931] Removed 1 References
[INFO] [2020-07-07 14:58:56] Starting batch with ID 80104759...
[INFO] [2020-07-07 14:58:58] Starting batch with ID 80104759...
[INFO] [2020-07-07 14:58:58] Starting batch with ID 80104759...
[INFO] [2020-07-07 14:59:00] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:01] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:01] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:01] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:01] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:01] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:01] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:02] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:03] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:03] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:03] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:03] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:03] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:03] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:03] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:03] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:03] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:03] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:03] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:04] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:04] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:04] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:04] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:05] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:06] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:06] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:06] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:06] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:06] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:06] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:06] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:06] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:06] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:06] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:06] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:06] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:07] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:07] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:07] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:07] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:07] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:07] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:07] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:07] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:07] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:08] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:09] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:09] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:09] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:09] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:09] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:09] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:09] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:09] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:09] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:09] Starting batch with ID 80104883...
[INFO] [2020-07-07 14:59:09] ## remove_type: Node
[INFO] [2020-07-07 14:59:09] ++ Calling delete_all on 150 instances...
[INFO] [2020-07-07 14:59:09] [14:59:09.471] Removed 150 Nodes
[INFO] [2020-07-07 15:11:53] ## HARVEST: type = -harvest
[START] [2020-07-07 15:11:56] logged process
[INFO] [2020-07-07 15:12:30] ## HARVEST: type = -harvest
[START] [2020-07-07 15:12:31] logged process
[START] [2020-07-07 15:12:31] create_harvest_instance
[STOP] [2020-07-07 15:12:32] create_harvest_instance
[START] [2020-07-07 15:12:32] fetch_files
[STOP] [2020-07-07 15:12:32] fetch_files
[START] [2020-07-07 15:12:32] validate_each_file
[STOP] [2020-07-07 15:12:33] validate_each_file
[START] [2020-07-07 15:12:33] convert_to_csv
[CMD] [2020-07-07 15:12:33] /usr/bin/sort /app/public/converted_csv/FAOSTAT_refs_21517.csv > /app/public/converted_csv/FAOSTAT_refs_21517.csv_sorted
[CMD] [2020-07-07 15:12:33] /usr/bin/sort /app/public/converted_csv/FAOSTAT_nodes_21518.csv > /app/public/converted_csv/FAOSTAT_nodes_21518.csv_sorted
[CMD] [2020-07-07 15:12:33] /usr/bin/sort /app/public/converted_csv/FAOSTAT_occurrences_21519.csv > /app/public/converted_csv/FAOSTAT_occurrences_21519.csv_sorted
[CMD] [2020-07-07 15:12:33] /usr/bin/sort /app/public/converted_csv/FAOSTAT_measurements_21520.csv > /app/public/converted_csv/FAOSTAT_measurements_21520.csv_sorted
[STOP] [2020-07-07 15:12:33] convert_to_csv
[START] [2020-07-07 15:12:33] calculate_delta
[CMD] [2020-07-07 15:12:33] echo "0a" > /app/public/diff/FAOSTAT_refs_21517.diff
[CMD] [2020-07-07 15:12:33] tail -n +1 /app/public/converted_csv/FAOSTAT_refs_21517.csv >> /app/public/diff/FAOSTAT_refs_21517.diff
[CMD] [2020-07-07 15:12:33] echo "." >> /app/public/diff/FAOSTAT_refs_21517.diff
[CMD] [2020-07-07 15:12:33] echo "0a" > /app/public/diff/FAOSTAT_nodes_21518.diff
[CMD] [2020-07-07 15:12:33] tail -n +1 /app/public/converted_csv/FAOSTAT_nodes_21518.csv >> /app/public/diff/FAOSTAT_nodes_21518.diff
[CMD] [2020-07-07 15:12:33] echo "." >> /app/public/diff/FAOSTAT_nodes_21518.diff
[CMD] [2020-07-07 15:12:33] echo "0a" > /app/public/diff/FAOSTAT_occurrences_21519.diff
[CMD] [2020-07-07 15:12:33] tail -n +1 /app/public/converted_csv/FAOSTAT_occurrences_21519.csv >> /app/public/diff/FAOSTAT_occurrences_21519.diff
[CMD] [2020-07-07 15:12:33] echo "." >> /app/public/diff/FAOSTAT_occurrences_21519.diff
[CMD] [2020-07-07 15:12:33] echo "0a" > /app/public/diff/FAOSTAT_measurements_21520.diff
[CMD] [2020-07-07 15:12:33] tail -n +1 /app/public/converted_csv/FAOSTAT_measurements_21520.csv >> /app/public/diff/FAOSTAT_measurements_21520.diff
[CMD] [2020-07-07 15:12:33] echo "." >> /app/public/diff/FAOSTAT_measurements_21520.diff
[STOP] [2020-07-07 15:12:33] calculate_delta
[START] [2020-07-07 15:12:33] parse_diff_and_store
[INFO] [2020-07-07 15:12:33] Loading refs diff file into memory (true lines)...
[INFO] [2020-07-07 15:12:33] Loading nodes diff file into memory (true lines)...
[INFO] [2020-07-07 15:12:33] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-07-07 15:12:33] Loading measurements diff file into memory (true lines)...
[INFO] [2020-07-07 15:14:22] Storing 1 References
[INFO] [2020-07-07 15:14:22] Processing group of 1 in 1 groups of 1000
[INFO] [2020-07-07 15:14:22] Average Time: 0.0
[INFO] [2020-07-07 15:14:22] Total Time: 1s
[INFO] [2020-07-07 15:14:22] Storing 150 ScientificNames
[INFO] [2020-07-07 15:14:22] Processing group of 150 in 1 groups of 1000
[INFO] [2020-07-07 15:14:22] Average Time: 0.09
[INFO] [2020-07-07 15:14:22] Total Time: 1s
[INFO] [2020-07-07 15:14:22] Storing 150 Nodes
[INFO] [2020-07-07 15:14:22] Processing group of 150 in 1 groups of 1000
[INFO] [2020-07-07 15:14:22] Average Time: 0.07
[INFO] [2020-07-07 15:14:22] Total Time: 1s
[INFO] [2020-07-07 15:14:22] Storing 134 Occurrences
[INFO] [2020-07-07 15:14:22] Processing group of 134 in 1 groups of 1000
[INFO] [2020-07-07 15:14:22] Average Time: 0.03
[INFO] [2020-07-07 15:14:22] Total Time: 1s
[INFO] [2020-07-07 15:14:22] Storing 13202 TraitsReferences
[INFO] [2020-07-07 15:14:22] Processing group of 13202 in 14 groups of 1000
[INFO] [2020-07-07 15:14:23] Average Time: 0.065
[INFO] [2020-07-07 15:14:23] Total Time: 1s
[INFO] [2020-07-07 15:14:23] last 3 / first 3: 0.5
[INFO] [2020-07-07 15:14:23] Std.Dev: 0.03162277660168379; Max: 0.14
[INFO] [2020-07-07 15:14:23] Storing 25691 Traits
[INFO] [2020-07-07 15:14:23] Processing group of 25691 in 26 groups of 1000
[INFO] [2020-07-07 15:14:30] Average Time: 0.256
[INFO] [2020-07-07 15:14:30] Total Time: 7s
[INFO] [2020-07-07 15:14:30] last 3 / first 3: 0.79
[INFO] [2020-07-07 15:14:30] Std.Dev: 0.03162277660168379; Max: 0.39
[INFO] [2020-07-07 15:14:30] Storing 38893 MetaTraits
[INFO] [2020-07-07 15:14:30] Processing group of 38893 in 39 groups of 1000
[INFO] [2020-07-07 15:14:34] Average Time: 0.101
[INFO] [2020-07-07 15:14:34] Total Time: 5s
[INFO] [2020-07-07 15:14:34] last 3 / first 3: 0.67
[INFO] [2020-07-07 15:14:34] Std.Dev: 0.03162277660168379; Max: 0.22
[STOP] [2020-07-07 15:14:34] parse_diff_and_store
[START] [2020-07-07 15:14:34] resolve_keys
[INFO] [2020-07-07 15:14:42] Occurrences to nodes (through scientific_names)...
[INFO] [2020-07-07 15:14:42] traits to occurrences...
[INFO] [2020-07-07 15:14:43] traits to nodes (through occurrences)...
[INFO] [2020-07-07 15:14:43] Traits to sex term...
[INFO] [2020-07-07 15:14:43] Traits to lifestage term...
[INFO] [2020-07-07 15:14:43] MetaTraits to traits...
[INFO] [2020-07-07 15:14:44] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-07-07 15:14:45] Assocs to occurrences...
[INFO] [2020-07-07 15:14:45] Assocs to nodes...
[INFO] [2020-07-07 15:14:45] Assoc to sex term...
[INFO] [2020-07-07 15:14:45] Assoc to lifestage term...
[STOP] [2020-07-07 15:14:45] resolve_keys
[START] [2020-07-07 15:14:45] hold_for_later_1
[STOP] [2020-07-07 15:14:45] hold_for_later_1
[START] [2020-07-07 15:14:45] hold_for_later_2
[STOP] [2020-07-07 15:14:45] hold_for_later_2
[START] [2020-07-07 15:14:45] resolve_missing_parents
[STOP] [2020-07-07 15:14:45] resolve_missing_parents
[START] [2020-07-07 15:14:45] rebuild_nodes
[START] [2020-07-07 15:14:45] Flattener#flatten
[START] [2020-07-07 15:14:45] Flattener#study_resource
[START] [2020-07-07 15:14:45] Flattener#build_ancestry
[STOP] [2020-07-07 15:14:45] Flattener#build_ancestry
[INFO] [2020-07-07 15:14:45] 150 ancestry keys
[START] [2020-07-07 15:14:45] build_node_ancestors
[INFO] [2020-07-07 15:14:45] old ancestors deleted.
[STOP] [2020-07-07 15:14:45] build_node_ancestors
[START] [2020-07-07 15:14:45] Flattener#propagate_ancestor_ids
[STOP] [2020-07-07 15:14:45] Flattener#propagate_ancestor_ids
[STOP] [2020-07-07 15:14:45] Flattener#flatten
[STOP] [2020-07-07 15:14:45] rebuild_nodes
[START] [2020-07-07 15:14:45] resolve_missing_media_owners
[STOP] [2020-07-07 15:14:45] resolve_missing_media_owners
[START] [2020-07-07 15:14:45] sanitize_media_verbatims
[STOP] [2020-07-07 15:14:45] sanitize_media_verbatims
[START] [2020-07-07 15:14:45] queue_downloads
[STOP] [2020-07-07 15:14:45] queue_downloads
[START] [2020-07-07 15:14:45] parse_names
[WARN] [2020-07-07 15:14:45] I see 150 names which still need to be parsed.
[STOP] [2020-07-07 15:14:47] parse_names
[START] [2020-07-07 15:14:47] denormalize_canonical_names_to_nodes
[STOP] [2020-07-07 15:14:47] denormalize_canonical_names_to_nodes
[START] [2020-07-07 15:14:47] match_nodes
[START] [2020-07-07 15:14:47] map_all_nodes_to_pages
[STOP] [2020-07-07 15:15:11] map_all_nodes_to_pages
[INFO] [2020-07-07 15:15:11] Unmatched nodes (1 of 150): Lactuca sativa capitata (#80139537)
[START] [2020-07-07 15:15:11] update_nodes
[STOP] [2020-07-07 15:15:11] update_nodes
[STOP] [2020-07-07 15:15:11] match_nodes
[START] [2020-07-07 15:15:11] reindex_search
[STOP] [2020-07-07 15:15:11] reindex_search
[START] [2020-07-07 15:15:11] normalize_units
[STOP] [2020-07-07 15:16:05] normalize_units
[START] [2020-07-07 15:16:05] calculate_statistics
[STOP] [2020-07-07 15:16:05] calculate_statistics
[START] [2020-07-07 15:16:05] complete_harvest_instance
[START] [2020-07-07 15:16:05] overall_tsv_creation
[INFO] [2020-07-07 15:16:05] Processing group of 150 in 1 batches of 10000
[INFO] [2020-07-07 15:17:58] 13202 Traits (unfiltered)...
[INFO] [2020-07-07 15:18:11] 13202 Traits (filtered)...
[INFO] [2020-07-07 15:18:11] 0 Associations (filtered)...
[INFO] [2020-07-07 15:18:59] 52095 metadata added.
[INFO] [2020-07-07 15:18:59] 0 metadata added.
[INFO] [2020-07-07 15:18:59] Average Time: 84.74
[INFO] [2020-07-07 15:18:59] Total Time: 2m55s
[STOP] [2020-07-07 15:18:59] overall_tsv_creation
[INFO] [2020-07-07 15:18:59] Done. Check your files:
[INFO] [2020-07-07 15:18:59] (150 lines) /app/public/data/FAOSTAT/publish_nodes.tsv
[INFO] [2020-07-07 15:18:59] (350 lines) /app/public/data/FAOSTAT/publish_node_ancestors.tsv
[INFO] [2020-07-07 15:18:59] (150 lines) /app/public/data/FAOSTAT/publish_scientific_names.tsv
[INFO] [2020-07-07 15:18:59] (13203 lines) /app/public/data/FAOSTAT/publish_traits.tsv
[INFO] [2020-07-07 15:18:59] (52096 lines) /app/public/data/FAOSTAT/publish_metadata.tsv
[STOP] [2020-07-07 15:18:59] complete_harvest_instance
[START] [2020-07-07 15:18:59] completed
[STOP] [2020-07-07 15:18:59] completed
[STOP] [2020-07-07 15:18:59] logged process, took 388.39
[INFO] [2020-07-07 15:25:57] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-07-07 15:26:00] ## remove_type: ScientificName
[INFO] [2020-07-07 15:26:00] ++ Calling delete_all on 150 instances...
[INFO] [2020-07-07 15:26:00] [15:26:00.121] Removed 150 Scientificnames
[INFO] [2020-07-07 15:26:00] ## remove_type: Vernacular
[INFO] [2020-07-07 15:26:00] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 15:26:00] [15:26:00.123] Removed 0 Vernaculars
[INFO] [2020-07-07 15:26:00] ## remove_type: Article
[INFO] [2020-07-07 15:26:00] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 15:26:00] [15:26:00.125] Removed 0 Articles
[INFO] [2020-07-07 15:26:00] ## remove_type: Medium
[INFO] [2020-07-07 15:26:00] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 15:26:00] [15:26:00.127] Removed 0 Media
[INFO] [2020-07-07 15:26:00] ## remove_type: Trait
[INFO] [2020-07-07 15:26:00] ++ Calling delete_all on 25691 instances...
[INFO] [2020-07-07 15:26:04] [15:26:04.321] Removed 25691 Traits
[INFO] [2020-07-07 15:26:04] ## remove_type: MetaTrait
[INFO] [2020-07-07 15:26:04] ++ Calling delete_all on 38893 instances...
[INFO] [2020-07-07 15:26:04] [15:26:04.680] Removed 38893 Metatraits
[INFO] [2020-07-07 15:26:04] ## remove_type: OccurrenceMetadatum
[INFO] [2020-07-07 15:26:04] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 15:26:04] [15:26:04.683] Removed 0 Occurrencemetadata
[INFO] [2020-07-07 15:26:04] ## remove_type: Assoc
[INFO] [2020-07-07 15:26:04] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 15:26:04] [15:26:04.686] Removed 0 Assocs
[INFO] [2020-07-07 15:26:04] ## remove_type: MetaAssoc
[INFO] [2020-07-07 15:26:04] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 15:26:04] [15:26:04.689] Removed 0 Metaassocs
[INFO] [2020-07-07 15:26:04] ## remove_type: Identifier
[INFO] [2020-07-07 15:26:04] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 15:26:04] [15:26:04.691] Removed 0 Identifiers
[INFO] [2020-07-07 15:26:04] ## remove_type: Reference
[INFO] [2020-07-07 15:26:04] ++ Calling delete_all on 1 instances...
[INFO] [2020-07-07 15:26:04] [15:26:04.694] Removed 1 References
[INFO] [2020-07-07 15:26:04] Starting batch with ID 80139454...
[INFO] [2020-07-07 15:26:05] Starting batch with ID 80139454...
[INFO] [2020-07-07 15:26:05] ## remove_type: Node
[INFO] [2020-07-07 15:26:05] ++ Calling delete_all on 150 instances...
[INFO] [2020-07-07 15:26:05] [15:26:05.189] Removed 150 Nodes
[INFO] [2020-07-07 15:26:49] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-07-07 15:26:50] ## remove_type: ScientificName
[INFO] [2020-07-07 15:26:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 15:26:50] [15:26:50.594] Removed 0 Scientificnames
[INFO] [2020-07-07 15:26:50] ## remove_type: Vernacular
[INFO] [2020-07-07 15:26:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 15:26:50] [15:26:50.596] Removed 0 Vernaculars
[INFO] [2020-07-07 15:26:50] ## remove_type: Article
[INFO] [2020-07-07 15:26:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 15:26:50] [15:26:50.597] Removed 0 Articles
[INFO] [2020-07-07 15:26:50] ## remove_type: Medium
[INFO] [2020-07-07 15:26:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 15:26:50] [15:26:50.599] Removed 0 Media
[INFO] [2020-07-07 15:26:50] ## remove_type: Trait
[INFO] [2020-07-07 15:26:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 15:26:50] [15:26:50.601] Removed 0 Traits
[INFO] [2020-07-07 15:26:50] ## remove_type: MetaTrait
[INFO] [2020-07-07 15:26:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 15:26:50] [15:26:50.602] Removed 0 Metatraits
[INFO] [2020-07-07 15:26:50] ## remove_type: OccurrenceMetadatum
[INFO] [2020-07-07 15:26:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 15:26:50] [15:26:50.604] Removed 0 Occurrencemetadata
[INFO] [2020-07-07 15:26:50] ## remove_type: Assoc
[INFO] [2020-07-07 15:26:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 15:26:50] [15:26:50.606] Removed 0 Assocs
[INFO] [2020-07-07 15:26:50] ## remove_type: MetaAssoc
[INFO] [2020-07-07 15:26:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 15:26:50] [15:26:50.607] Removed 0 Metaassocs
[INFO] [2020-07-07 15:26:50] ## remove_type: Identifier
[INFO] [2020-07-07 15:26:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 15:26:50] [15:26:50.609] Removed 0 Identifiers
[INFO] [2020-07-07 15:26:50] ## remove_type: Reference
[INFO] [2020-07-07 15:26:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 15:26:50] [15:26:50.611] Removed 0 References
[INFO] [2020-07-07 15:26:50] ## remove_type: Node
[INFO] [2020-07-07 15:26:50] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-07 15:26:50] [15:26:50.622] Removed 0 Nodes
[START] [2020-07-07 15:27:05] logged process
[START] [2020-07-07 15:27:05] overall_tsv_creation
[INFO] [2020-07-07 15:27:05] Processing group of 0 in 0 batches of 10000
[INFO] [2020-07-07 15:27:05] Average Time: NaN
[INFO] [2020-07-07 15:27:05] Total Time: 1s
[STOP] [2020-07-07 15:27:05] overall_tsv_creation
[INFO] [2020-07-07 15:27:05] Done. Check your files:
[INFO] [2020-07-09 15:12:26] ## HARVEST: type = re_download_opendata_-harvest
[INFO] [2020-07-09 15:12:31] ## remove_type: ScientificName
[INFO] [2020-07-09 15:12:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-09 15:12:31] [15:12:31.544] Removed 0 Scientificnames
[INFO] [2020-07-09 15:12:31] ## remove_type: Vernacular
[INFO] [2020-07-09 15:12:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-09 15:12:31] [15:12:31.545] Removed 0 Vernaculars
[INFO] [2020-07-09 15:12:31] ## remove_type: Article
[INFO] [2020-07-09 15:12:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-09 15:12:31] [15:12:31.547] Removed 0 Articles
[INFO] [2020-07-09 15:12:31] ## remove_type: Medium
[INFO] [2020-07-09 15:12:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-09 15:12:31] [15:12:31.549] Removed 0 Media
[INFO] [2020-07-09 15:12:31] ## remove_type: Trait
[INFO] [2020-07-09 15:12:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-09 15:12:31] [15:12:31.551] Removed 0 Traits
[INFO] [2020-07-09 15:12:31] ## remove_type: MetaTrait
[INFO] [2020-07-09 15:12:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-09 15:12:31] [15:12:31.552] Removed 0 Metatraits
[INFO] [2020-07-09 15:12:31] ## remove_type: OccurrenceMetadatum
[INFO] [2020-07-09 15:12:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-09 15:12:31] [15:12:31.554] Removed 0 Occurrencemetadata
[INFO] [2020-07-09 15:12:31] ## remove_type: Assoc
[INFO] [2020-07-09 15:12:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-09 15:12:31] [15:12:31.555] Removed 0 Assocs
[INFO] [2020-07-09 15:12:31] ## remove_type: MetaAssoc
[INFO] [2020-07-09 15:12:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-09 15:12:31] [15:12:31.557] Removed 0 Metaassocs
[INFO] [2020-07-09 15:12:31] ## remove_type: Identifier
[INFO] [2020-07-09 15:12:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-09 15:12:31] [15:12:31.558] Removed 0 Identifiers
[INFO] [2020-07-09 15:12:31] ## remove_type: Reference
[INFO] [2020-07-09 15:12:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-09 15:12:31] [15:12:31.560] Removed 0 References
[INFO] [2020-07-09 15:12:31] ## remove_type: Node
[INFO] [2020-07-09 15:12:31] ++ Calling delete_all on 0 instances...
[INFO] [2020-07-09 15:12:31] [15:12:31.711] Removed 0 Nodes
[START] [2020-07-09 15:12:51] logged process
[START] [2020-07-09 15:12:51] overall_tsv_creation
[INFO] [2020-07-09 15:12:51] Processing group of 0 in 0 batches of 10000
[INFO] [2020-07-09 15:12:51] Average Time: NaN
[INFO] [2020-07-09 15:12:51] Total Time: 1s
[STOP] [2020-07-09 15:12:51] overall_tsv_creation
[INFO] [2020-07-09 15:12:51] Done. Check your files:
[INFO] [2020-07-09 15:13:00] ## HARVEST: type = -harvest
[START] [2020-07-09 15:13:01] logged process
[INFO] [2020-07-09 15:13:06] ## HARVEST: type = -harvest
[START] [2020-07-09 15:13:06] logged process
[START] [2020-07-09 15:13:06] create_harvest_instance
[STOP] [2020-07-09 15:13:08] create_harvest_instance
[START] [2020-07-09 15:13:08] fetch_files
[STOP] [2020-07-09 15:13:08] fetch_files
[START] [2020-07-09 15:13:08] validate_each_file
[STOP] [2020-07-09 15:13:09] validate_each_file
[START] [2020-07-09 15:13:09] convert_to_csv
[CMD] [2020-07-09 15:13:09] /usr/bin/sort /app/public/converted_csv/FAOSTAT_refs_21563.csv > /app/public/converted_csv/FAOSTAT_refs_21563.csv_sorted
[CMD] [2020-07-09 15:13:09] /usr/bin/sort /app/public/converted_csv/FAOSTAT_nodes_21564.csv > /app/public/converted_csv/FAOSTAT_nodes_21564.csv_sorted
[CMD] [2020-07-09 15:13:09] /usr/bin/sort /app/public/converted_csv/FAOSTAT_occurrences_21565.csv > /app/public/converted_csv/FAOSTAT_occurrences_21565.csv_sorted
[CMD] [2020-07-09 15:13:09] /usr/bin/sort /app/public/converted_csv/FAOSTAT_measurements_21566.csv > /app/public/converted_csv/FAOSTAT_measurements_21566.csv_sorted
[STOP] [2020-07-09 15:13:09] convert_to_csv
[START] [2020-07-09 15:13:09] calculate_delta
[CMD] [2020-07-09 15:13:09] echo "0a" > /app/public/diff/FAOSTAT_refs_21563.diff
[CMD] [2020-07-09 15:13:09] tail -n +1 /app/public/converted_csv/FAOSTAT_refs_21563.csv >> /app/public/diff/FAOSTAT_refs_21563.diff
[CMD] [2020-07-09 15:13:09] echo "." >> /app/public/diff/FAOSTAT_refs_21563.diff
[CMD] [2020-07-09 15:13:09] echo "0a" > /app/public/diff/FAOSTAT_nodes_21564.diff
[CMD] [2020-07-09 15:13:09] tail -n +1 /app/public/converted_csv/FAOSTAT_nodes_21564.csv >> /app/public/diff/FAOSTAT_nodes_21564.diff
[CMD] [2020-07-09 15:13:09] echo "." >> /app/public/diff/FAOSTAT_nodes_21564.diff
[CMD] [2020-07-09 15:13:09] echo "0a" > /app/public/diff/FAOSTAT_occurrences_21565.diff
[CMD] [2020-07-09 15:13:09] tail -n +1 /app/public/converted_csv/FAOSTAT_occurrences_21565.csv >> /app/public/diff/FAOSTAT_occurrences_21565.diff
[CMD] [2020-07-09 15:13:09] echo "." >> /app/public/diff/FAOSTAT_occurrences_21565.diff
[CMD] [2020-07-09 15:13:09] echo "0a" > /app/public/diff/FAOSTAT_measurements_21566.diff
[CMD] [2020-07-09 15:13:09] tail -n +1 /app/public/converted_csv/FAOSTAT_measurements_21566.csv >> /app/public/diff/FAOSTAT_measurements_21566.diff
[CMD] [2020-07-09 15:13:09] echo "." >> /app/public/diff/FAOSTAT_measurements_21566.diff
[STOP] [2020-07-09 15:13:09] calculate_delta
[START] [2020-07-09 15:13:09] parse_diff_and_store
[INFO] [2020-07-09 15:13:09] Loading refs diff file into memory (true lines)...
[INFO] [2020-07-09 15:13:09] Loading nodes diff file into memory (true lines)...
[INFO] [2020-07-09 15:13:09] Loading occurrences diff file into memory (true lines)...
[INFO] [2020-07-09 15:13:09] Loading measurements diff file into memory (true lines)...
[INFO] [2020-07-09 15:14:56] Storing 1 References
[INFO] [2020-07-09 15:14:56] Processing group of 1 in 1 groups of 1000
[INFO] [2020-07-09 15:14:56] Average Time: 0.0
[INFO] [2020-07-09 15:14:56] Total Time: 1s
[INFO] [2020-07-09 15:14:56] Storing 150 ScientificNames
[INFO] [2020-07-09 15:14:56] Processing group of 150 in 1 groups of 1000
[INFO] [2020-07-09 15:14:57] Average Time: 0.1
[INFO] [2020-07-09 15:14:57] Total Time: 1s
[INFO] [2020-07-09 15:14:57] Storing 150 Nodes
[INFO] [2020-07-09 15:14:57] Processing group of 150 in 1 groups of 1000
[INFO] [2020-07-09 15:14:57] Average Time: 0.07
[INFO] [2020-07-09 15:14:57] Total Time: 1s
[INFO] [2020-07-09 15:14:57] Storing 134 Occurrences
[INFO] [2020-07-09 15:14:57] Processing group of 134 in 1 groups of 1000
[INFO] [2020-07-09 15:14:57] Average Time: 0.03
[INFO] [2020-07-09 15:14:57] Total Time: 1s
[INFO] [2020-07-09 15:14:57] Storing 13202 TraitsReferences
[INFO] [2020-07-09 15:14:57] Processing group of 13202 in 14 groups of 1000
[INFO] [2020-07-09 15:14:58] Average Time: 0.065
[INFO] [2020-07-09 15:14:58] Total Time: 1s
[INFO] [2020-07-09 15:14:58] last 3 / first 3: 0.48
[INFO] [2020-07-09 15:14:58] Std.Dev: 0.03162277660168379; Max: 0.15
[INFO] [2020-07-09 15:14:58] Storing 25691 Traits
[INFO] [2020-07-09 15:14:58] Processing group of 25691 in 26 groups of 1000
[INFO] [2020-07-09 15:15:05] Average Time: 0.284
[INFO] [2020-07-09 15:15:05] Total Time: 8s
[INFO] [2020-07-09 15:15:05] last 3 / first 3: 0.5
[INFO] [2020-07-09 15:15:05] Std.Dev: 0.06324555320336758; Max: 0.49
[INFO] [2020-07-09 15:15:05] Storing 38893 MetaTraits
[INFO] [2020-07-09 15:15:05] Processing group of 38893 in 39 groups of 1000
[INFO] [2020-07-09 15:15:09] Average Time: 0.092
[INFO] [2020-07-09 15:15:09] Total Time: 4s
[INFO] [2020-07-09 15:15:09] last 3 / first 3: 0.87
[INFO] [2020-07-09 15:15:09] Std.Dev: 0.0; Max: 0.16
[STOP] [2020-07-09 15:15:09] parse_diff_and_store
[START] [2020-07-09 15:15:09] resolve_keys
[INFO] [2020-07-09 15:15:17] Occurrences to nodes (through scientific_names)...
[INFO] [2020-07-09 15:15:17] traits to occurrences...
[INFO] [2020-07-09 15:15:17] traits to nodes (through occurrences)...
[INFO] [2020-07-09 15:15:18] Traits to sex term...
[INFO] [2020-07-09 15:15:18] Traits to lifestage term...
[INFO] [2020-07-09 15:15:18] MetaTraits to traits...
[INFO] [2020-07-09 15:15:19] MetaTraits (simple, measurement row refers to parent) to traits...
[INFO] [2020-07-09 15:15:20] Assocs to occurrences...
[INFO] [2020-07-09 15:15:20] Assocs to nodes...
[INFO] [2020-07-09 15:15:20] Assoc to sex term...
[INFO] [2020-07-09 15:15:20] Assoc to lifestage term...
[STOP] [2020-07-09 15:15:20] resolve_keys
[START] [2020-07-09 15:15:20] hold_for_later_1
[STOP] [2020-07-09 15:15:20] hold_for_later_1
[START] [2020-07-09 15:15:20] hold_for_later_2
[STOP] [2020-07-09 15:15:20] hold_for_later_2
[START] [2020-07-09 15:15:20] resolve_missing_parents
[STOP] [2020-07-09 15:15:20] resolve_missing_parents
[START] [2020-07-09 15:15:20] rebuild_nodes
[START] [2020-07-09 15:15:20] Flattener#flatten
[START] [2020-07-09 15:15:20] Flattener#study_resource
[START] [2020-07-09 15:15:20] Flattener#build_ancestry
[STOP] [2020-07-09 15:15:20] Flattener#build_ancestry
[INFO] [2020-07-09 15:15:20] 150 ancestry keys
[START] [2020-07-09 15:15:20] build_node_ancestors
[INFO] [2020-07-09 15:15:20] old ancestors deleted.
[STOP] [2020-07-09 15:15:20] build_node_ancestors
[START] [2020-07-09 15:15:20] Flattener#propagate_ancestor_ids
[STOP] [2020-07-09 15:15:20] Flattener#propagate_ancestor_ids
[STOP] [2020-07-09 15:15:20] Flattener#flatten
[STOP] [2020-07-09 15:15:20] rebuild_nodes
[START] [2020-07-09 15:15:20] resolve_missing_media_owners
[STOP] [2020-07-09 15:15:20] resolve_missing_media_owners
[START] [2020-07-09 15:15:20] sanitize_media_verbatims
[STOP] [2020-07-09 15:15:20] sanitize_media_verbatims
[START] [2020-07-09 15:15:20] queue_downloads
[STOP] [2020-07-09 15:15:20] queue_downloads
[START] [2020-07-09 15:15:20] parse_names
[WARN] [2020-07-09 15:15:20] I see 150 names which still need to be parsed.
[STOP] [2020-07-09 15:15:21] parse_names
[START] [2020-07-09 15:15:21] denormalize_canonical_names_to_nodes
[STOP] [2020-07-09 15:15:21] denormalize_canonical_names_to_nodes
[START] [2020-07-09 15:15:21] match_nodes
[START] [2020-07-09 15:15:21] map_all_nodes_to_pages
[STOP] [2020-07-09 15:15:53] map_all_nodes_to_pages
[INFO] [2020-07-09 15:15:53] Unmatched nodes (1 of 150): Lactuca sativa capitata (#80139792)
[START] [2020-07-09 15:15:53] update_nodes
[STOP] [2020-07-09 15:15:53] update_nodes
[STOP] [2020-07-09 15:15:53] match_nodes
[START] [2020-07-09 15:15:53] reindex_search
[STOP] [2020-07-09 15:15:53] reindex_search
[START] [2020-07-09 15:15:53] normalize_units
[STOP] [2020-07-09 15:16:46] normalize_units
[START] [2020-07-09 15:16:46] calculate_statistics
[STOP] [2020-07-09 15:16:46] calculate_statistics
[START] [2020-07-09 15:16:46] complete_harvest_instance
[START] [2020-07-09 15:16:46] overall_tsv_creation
[INFO] [2020-07-09 15:16:46] Processing group of 150 in 1 batches of 10000
[INFO] [2020-07-09 15:17:36] 13202 Traits (unfiltered)...
[INFO] [2020-07-09 15:17:48] 13202 Traits (filtered)...
[INFO] [2020-07-09 15:17:48] 0 Associations (filtered)...
[INFO] [2020-07-09 15:18:36] 52095 metadata added.
[INFO] [2020-07-09 15:18:36] 0 metadata added.
[INFO] [2020-07-09 15:18:36] Average Time: 82.64
[INFO] [2020-07-09 15:18:36] Total Time: 1m50s
[STOP] [2020-07-09 15:18:36] overall_tsv_creation
[INFO] [2020-07-09 15:18:36] Done. Check your files:
[INFO] [2020-07-09 15:18:36] (150 lines) /app/public/data/FAOSTAT/publish_nodes.tsv
[INFO] [2020-07-09 15:18:36] (350 lines) /app/public/data/FAOSTAT/publish_node_ancestors.tsv
[INFO] [2020-07-09 15:18:36] (150 lines) /app/public/data/FAOSTAT/publish_scientific_names.tsv
[INFO] [2020-07-09 15:18:36] (13203 lines) /app/public/data/FAOSTAT/publish_traits.tsv
[INFO] [2020-07-09 15:18:36] (52096 lines) /app/public/data/FAOSTAT/publish_metadata.tsv
[STOP] [2020-07-09 15:18:36] complete_harvest_instance
[START] [2020-07-09 15:18:36] completed
[STOP] [2020-07-09 15:18:36] completed
[STOP] [2020-07-09 15:18:36] logged process, took 329.44

Latest Process

09 Jul 15:13 EXITED currently processing 0