AdvancedDatabase & Storage
Elasticsearch ILM: Hot-Warm-Cold Architecture
Optimize storage costs and search performance by automatically moving aging log data from expensive SSDs to cheaper HDD storage tiers.
Elasticsearch
Prerequisites
- Elasticsearch Cluster with node tiers assigned
Configuration Files
ilm_policy.jsonAPI PUT _ilm/policy/log_policy
{\n "policy": {\n "phases": {\n "hot": {\n "actions": { "rollover": { "max_size": "50gb", "max_age": "7d" } }\n },\n "warm": {\n "min_age": "7d",\n "actions": { "shrink": { "number_of_shards": 1 }, "allocate": { "require": { "data": "warm" } } }\n },\n "delete": {\n "min_age": "30d",\n "actions": { "delete": {} }\n }\n }\n }\n}Explanation:Rolls over indexes at 50GB, moves them to warm nodes after 7 days, and deletes them after 30 days.
Verification Steps
1
Verifies the policy was saved to the cluster.
$curl -X GET "localhost:9200/_ilm/policy/log_policy"
Expected Output"name" : "log_policy"
Production Gotchas
- Rollover requires an index alias to be configured. The application must write to the alias, not the direct index name.
Frequently Asked Questions
Why shrink shards in the warm phase?
Fewer shards consume less heap memory on the node. Old, read-only data doesn't need to be spread across many shards.