How To Set Different Retention Periods to Different Indexes using Elasticsearch Curator

You must be already aware that Elasticsearch Curator is a great tool that can be used to manage indexes in Elasticsearch. That being said, at some point you’ll want to set different retention periods for the indexes as per the important of each indexes.

So in this article I’ll be sharing a curator action file (curator sample config file) which is going to setup different retention periods for different elasticsearch indexes.

---
actions:
  1:
    action: delete_indices
    description: delete activity/fluentd indices which are older than 60 days
    options:
      ignore_empty_list: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(activity-|fluentd-).*$'
      exclude: False
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 60
      exclude: False
    - filtertype: kibana
      exclude: True
  2:
    action: delete_indices
    description: delete server/exceptions indices which are older than 90 days
    options:
      ignore_empty_list: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(server-|exceptions-).*$'
      exclude: False
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 90
      exclude: False
  
  3:
    action: delete_indices
    description: delete metrics indices which are older than 120 days
    options:
      ignore_empty_list: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(metrics-).*$'
      exclude: False
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 120
      exclude: False
    - filtertype: kibana
      exclude: True

If you refer to the above action file you’ll be able to see three action blocks which go as 1: / 2: / 3:. These are the actions which the curator will be performing for the indexes in your elasticsearch.

Block 1: This will delete activity/fluentd indices which are older than 60 days.

Block 2: This will delete server/exception indices which are older than 90 days.

Block 3: This will delete metrics indices which are older than 120 days.

In each block these are the main filters that you need to focus on,

– filtertype: pattern – This means that you’re going to filter your indexes using a pattern, here we’ve used regex. Our indexes go as follows,

<index_name>-YYYY.MM.DD – so we filter the indexes using their names by regex.

ex – ‘^(server-|exceptions-).*$’

Above regex will get all the server-YYYY.MM-DD / exceptions-YYYY.MM.DD indexes.iltertype: pattern

– filtertype: age – This is where you set the retentions period, given below I have explained some of the main attributes we need to use there.

  • direction: older – X number of days OLDER indexes will be removed, hence direction should set as older.
  • timestring: ‘%Y.%m.%d’ – As our index dates goes as YYYY.MM.DD format we need the time string in the same format.
  • unit: days / unit_count: 90 – As we are deleting the indexes which are older than X number of days, unit should be days and count should set to X.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Website Powered by WordPress.com.

Up ↑

%d bloggers like this: