Curator

Goal:
In these tutorial we gonna cover deletion of old logs in ELK Stack. We gonna achive these by deleting old indices created by Logstash while dumping logs in Elasticsearch.

Prerequisites:
Old logs to delete... 😜😜

Let's Begin the exercise:

Install curator
Curator is a package in Elasticsearch  repository to delete old indices.

Create a file

sudo vi /etc/yum.repos.d/curator.repo

paste following lines
Save and Exit file
Run yum install

sudo yum install elasticsearch-curator

Configure Curator
Create a directory

mkdir ~/.curator/

Open a file

sudo vi ~/.curator/curator.yml

paste following code

Save and Exit file

Deletion pattern
Create file to define delete pattern in Elasticesearch

sudo vi ~/.curator/delete_indices.yml

paste following lines in file
Create a log file for curator on the location you defined in configuration, and assign permission to right into file.

sudo touch /var/log/curator
#to assign permission to write logs 
sudo chown ec2-user:ec2-user /var/log/curator

Logrotate on curator logs

# create file
sudo vi /etc/logrotate.d/curator
# paste these lines in logroate
/var/log/curator {
    missingok
    notifempty
    rotate 5
    daily
}
#save and exit

Cron to Run Curator (Log Deletion) job daily

#open crontab
crontab -e
#paste above line to run cron everyday 2 O'clock
0 2 * * * /usr/bin/curator ~/.curator/delete_indices.yml
#save and exit