Skip to main content

Elastalert

Goal
Trigger alert on elastic search stream. 

Install Elastalert
sudo yum install gcc
sudo pip install elastalert
sudo yum install git

Just to get basic elastalert rules reference clone following git repository.
git clone https://github.com/Yelp/elastalert.git example_elastalert

Create your own rules 
sudo mkdir -p /etc/elastalert/rules_folder/
sudo cp ~/example_elastalert/example_rules/example_frequency.yaml /etc/elastalert/rules_folder/frequency.yaml
sudo cp ~/example_elastalert/config.yaml.example /etc/elastalert/config.yaml

Configure Elastalert 
sudo vi /etc/elastalert/config.yaml
Search for 'rules_folder', 'es_host' and change values in file 
rules_folder: "/etc/elastalert/rules_folder" 
es_host: localhost

Now Open File  and change conf
sudo vi /etc/elastalert/rules_folder/frequency.yaml

filter:
- term:
  _type: "apache-error"
- query:
  query_string:
  query: "host:HostName"

alert:
 - "email"
include: ["host", "message", "errormessage"]
top_count_keys: ["errormessage"]
alert_subject: 'Apache Error rate increased for hostname {0} at {1}'
alert_subject_args: ["host", "@timestamp"]
alert_text_type: exclude_fields
email: ["Email-1", "Email-2"]
from_addr: 'prod-elk@hostname'

To test rule 
elastalert-test-rule --config /etc/elastalert/config.yaml /etc/elastalert/rules_folder/frequency.yaml
If any conflict occur 
sudo pip install pip-conflict-checker
pipconflictchecker
Generally following package have issue so upgrade package version 
sudo pip install "six>=1.9"
sudo pip install "requests-oauthlib(>=0.8.0)"
Test again 
elastalert-test-rule --config /etc/elastalert/config.yaml /etc/elastalert/rules_folder/frequency.yaml
Now all thing should be okay

To run elastalert as service 
sudo vi /etc/init.d/elastalert
Gist
To make it executable 
sudo chmod +x /etc/init.d/elastalert

Further Reference
https://github.com/Yelp/elastalert

Hope You will be able to get basic alert from your elastic search stream, If any issue occur please lemme know in comments.
Labels:

Comments

Post a Comment

Popular posts from this blog

Curator

Goal: In these tutorial we gonna cover deletion of old logs in ELK Stack. We gonna achive these by deleting old indices created by Logstash while dumping logs in Elasticsearch. Prerequisites: Old logs to delete... 😜😜 Let's Begin the exercise: Install curator Curator is a package in Elasticsearch  repository to delete old indices. Create a file sudo vi /etc/yum.repos.d/curator.repo paste following lines Save and Exit file Run yum install sudo yum install elasticsearch-curator Configure Curator Create a directory mkdir ~/.curator/ Open a file sudo vi ~/.curator/curator.yml paste following code Save and Exit file Deletion pattern Create file to define delete pattern in Elasticesearch sudo vi ~/.curator/delete_indices.yml paste following lines in file Create a log file for curator on the location you defined in configuration, and assign permission to right into file. sudo touch /var/log/curator #to assign permission to write l...
Post a Comment
Read more

GoReplay - Testing Your Site with Actual Traffic

Goal:   In these article we gonna learn How to capture your Real Time traffic from production and reuse it at your testing/development environment. Prerequisite: One web server running, or If you are just playing around then you can run goreplay test ftp server. Let's Begin Load Testing for site serving millions user wasn't be that easy before I came to know GoReplay . Here I am not gonna explain you How great go replay is, You will automatically get to know after following steps above step to capture and replay your request logs. FYI GoReplay capture logs from tcpdump. Installation: Download zip file from there git repo and unzip it. # create a directory mkdir ~/goreplay # go to directory you created cd ~/goreplay # download tar file from goreplay git repo wget https://github.com/buger/goreplay/releases/download/v0.16.1/gor_0.16.1_x64.tar.gz # unzip it tar -xf gor_0.16.1_x64.tar.gz After Unzipping Check GoReplay binary File is available in directory. Ca...
Post a Comment
Read more

Install Central Logging on Amazon Linux

Goal: In these tutorial we gonna cover setup of central logging system on amazon linux (CentOs) in same aws vpc . We will setup one central log server to receive log using rsyslog, after that we will setup one client to forward apache & syslog to central server. we already covered forward logs from central log server to ELK stack for analyzing. Logging Stack Component: Central Log server Multiple logging client server/Any apache web server generating logs Rsyslog: we setup with rsyslog v8-stable. You can use any rsyslog  version after rsyslog-6, because we encountered rsyslog drop message in earlier version. Prerequisites: Rsyslog is quite light weight, we doesn't requirement any high configuration machine, aws t2.micro should be enough. We are running t2.micro in production for central log server to receive around 1000 log entry/second, server is using less then 2 percent/sec within same vpc. Now Let's Start we gonna break these tutorial in two pa...
Post a Comment
Read more