Quick use case : remove daily indices older than 90 days with ILM

In this short post i will not dive into considerations on why i use daily indices instead of using a global ILM policy that will rollover and organize the data ..., in my daily basis dealing with duplicates is a must and this is not supported by rollover or datastream, but i will talk about this on a other post.

So basically i import network metrics into daily indices :

snmp-devices-metrics-YYYY-MM-dd

The re quirements was to keep only last 90 days and remove any index older than 90 days, again i will dive into why not taking a snapshoot before deleting the data :)

• Step 1 : create an ILM policy

PUT _ilm/policy/delete_policy_90days
{
  "policy": {                       
    "phases": {
      "delete": {
        "min_age": "90d",           
        "actions": {
          "delete": {}              
        }
      }
    }
  }
}

• Step 2 : attach the ILM policy to the index template

PUT _template/snmp-devices-metrics
{
  "index_patterns": ["snmp-devices-metrics-*"],                 
  "settings": {
     ...
    "index.lifecycle.name": "delete_policy_90days"
  }
  ...
}

Right now any new index that meet the pattern snmp-devices-metrics-*` will be assigned the ILM policy `delete_policy_90dayswith status/phase new and after 90 days it will be deleted automatically

Step 3 : attach ILM policy to old indices

The following step will attach the new ILM policy to all existing index with pattern snmp-devices-metrics-*, and anyone older than 90 days will be in status deleted

PUT snmp-devices-metrics-*/_settings 
{
  "index": {
    "lifecycle": {
      "name": "delete_policy_90days"
    }
  }
}

Last step, not needed but as i needed to clean quickly disk spaces :) i decided to update a global setting indices.lifecycle.poll_interval at cluster level to force ILM policy

indices.lifecycle.poll_interval

(Dynamic, time unit value) How often index lifecycle management checks for indices that meet policy criteria. Defaults to 10m.

PUT _cluster/settings
{
  "transient": {
    "indices.lifecycle.poll_interval": "1m"
  }
}