Curator

Goal:
In these tutorial we gonna cover deletion of old logs in ELK Stack. We gonna achive these by deleting old indices created by Logstash while dumping logs in Elasticsearch.

Prerequisites:
Old logs to delete... 😜😜

Let's Begin the exercise:

Install curator
Curator is a package in Elasticsearch  repository to delete old indices.

Create a file

sudo vi /etc/yum.repos.d/curator.repo

paste following lines

	[curator-5]
	name=CentOS/RHEL 7 repository for Elasticsearch Curator 5.x packages
	baseurl=http://packages.elastic.co/curator/5/centos/7
	gpgcheck=1
	gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
	enabled=1

view raw curator.repo hosted with ❤ by GitHub

Save and Exit file
Run yum install

sudo yum install elasticsearch-curator

Configure Curator
Create a directory

mkdir ~/.curator/

Open a file

sudo vi ~/.curator/curator.yml

paste following code

	# Remember, leave a key empty if there is no value. None will be a string,
	# not a Python "NoneType"
	client:
	hosts:
	- 127.0.0.1
	port: 9200
	url_prefix:
	use_ssl: False
	certificate:
	client_cert:
	client_key:
	ssl_no_validate: False
	http_auth:
	timeout: 30
	master_only: False
	logging:
	loglevel: INFO
	logfile: /var/log/curator
	logformat: default
	blacklist: []

view raw curator_conf hosted with ❤ by GitHub

Save and Exit file

Deletion pattern
Create file to define delete pattern in Elasticesearch

sudo vi ~/.curator/delete_indices.yml

paste following lines in file

	# Remember, leave a key empty if there is no value. None will be a string,
	# not a Python "NoneType"
	#
	# Also remember that all examples have 'disable_action' set to True. If you
	# want to use this action as a template, be sure to set this to False after
	# copying it.
	actions:
	1:
	action: delete_indices
	description: >-
	Delete indices older than 45 days (based on index name), for logstash-
	prefixed indices. Ignore the error if the filter does not result in an
	actionable list of indices (ignore_empty_list) and exit cleanly.
	options:
	ignore_empty_list: True
	disable_action: false
	filters:
	- filtertype: pattern
	kind: prefix
	value: logstash-
	- filtertype: age
	source: name
	direction: older
	timestring: '%Y.%m.%d'
	unit: days
	unit_count: 45

view raw curator_delete_indices hosted with ❤ by GitHub

Create a log file for curator on the location you defined in configuration, and assign permission to right into file.

sudo touch /var/log/curator
#to assign permission to write logs 
sudo chown ec2-user:ec2-user /var/log/curator

Logrotate on curator logs

# create file
sudo vi /etc/logrotate.d/curator
# paste these lines in logroate
/var/log/curator {
    missingok
    notifempty
    rotate 5
    daily
}
#save and exit

Cron to Run Curator (Log Deletion) job daily

#open crontab
crontab -e
#paste above line to run cron everyday 2 O'clock
0 2 * * * /usr/bin/curator ~/.curator/delete_indices.yml
#save and exit