Piyush Tech Blog

Goal: In these tutorial we gonna cover deletion of old logs in ELK Stack. We gonna achive these by deleting old indices created by Logstash while dumping logs in Elasticsearch. Prerequisites: Old logs to delete... 😜😜 Let's Begin the exercise: Install curator Curator is a package in Elasticsearch  repository to delete old indices. Create a file sudo vi /etc/yum.repos.d/curator.repo paste following lines Save and Exit file Run yum install sudo yum install elasticsearch-curator Configure Curator Create a directory mkdir ~/.curator/ Open a file sudo vi ~/.curator/curator.yml paste following code Save and Exit file Deletion pattern Create file to define delete pattern in Elasticesearch sudo vi ~/.curator/delete_indices.yml paste following lines in file Create a log file for curator on the location you defined in configuration, and assign permission to right into file. sudo touch /var/log/curator #to assign permission to write l

Goal Trigger alert on elastic search stream.   Install Elastalert sudo yum install gcc sudo pip install elastalert sudo yum install git Just to get basic elastalert rules reference clone following git repository. git clone https://github.com/Yelp/elastalert.git example_elastalert Create your own rules sudo mkdir -p /etc/elastalert/rules_folder/ sudo cp ~/example_elastalert/example_rules/example_frequency.yaml /etc/elastalert/rules_folder/frequency.yaml sudo cp ~/example_elastalert/config.yaml.example /etc/elastalert/config.yaml Configure Elastalert sudo vi /etc/elastalert/config.yaml Search for 'rules_folder', 'es_host' and change values in file rules_folder: "/etc/elastalert/rules_folder" es_host: localhost Now Open File  and change conf sudo vi /etc/elastalert/rules_folder/frequency.yaml filter: - term: _type: "apache-error" - query: query_string: query: "host:HostName" alert: - "email" include: [

Goal: In these tutorial we gonna cover setup of central logging system on amazon linux (CentOs) in same aws vpc . We will setup one central log server to receive log using rsyslog, after that we will setup one client to forward apache & syslog to central server. we already covered forward logs from central log server to ELK stack for analyzing. Logging Stack Component: Central Log server Multiple logging client server/Any apache web server generating logs Rsyslog: we setup with rsyslog v8-stable. You can use any rsyslog  version after rsyslog-6, because we encountered rsyslog drop message in earlier version. Prerequisites: Rsyslog is quite light weight, we doesn't requirement any high configuration machine, aws t2.micro should be enough. We are running t2.micro in production for central log server to receive around 1000 log entry/second, server is using less then 2 percent/sec within same vpc. Now Let's Start we gonna break these tutorial in two pa

Goal: In these tutorial we gonna cover installation of ELK Stack on fresh amazon ec2 linux (CentOS). We will install Elasticsearch 5.x.x, Logstash 5.x.x, and Kibana 5.x.x. We will also show you how to configure filebeat to forwards apache logs collected by central rsyslog server to elk server using Filebeat 5.x.x. ELK stack components: Logstash: Transform incoming logs. Elasticsearch(ES): Stores logs transformed by logstash. Kibana: Web interface for searching and visualizing logs stored in elasticsearch, which is proxied through Nginx. Filebeat: Lightweight Shipper of Logs from client to logstash server. Prerequisites: Minimum size to run your ES cluster RAM --> 4GB CPU --> 2 core Disk --> 20 GB (highly varies on your log size) You many need to increase RAM, CPU, Disk size depending on your log size. Let's start on our main goal to setup ELK Server Install java 8 sudo yum install java-1.8.0-openjdk Change Java Home as Java 8 sudo sh -c "echo expo