Skip to main content

Install Elasticsearch, Logstash, and Kibana (ELK Stack) on Amazon Linux

Goal:
In these tutorial we gonna cover installation of ELK Stack on fresh amazon ec2 linux (CentOS). We will install Elasticsearch 5.x.x, Logstash 5.x.x, and Kibana 5.x.x. We will also show you how to configure filebeat to forwards apache logs collected by central rsyslog server to elk server using Filebeat 5.x.x.

ELK stack components:
Logstash: Transform incoming logs.
Elasticsearch(ES): Stores logs transformed by logstash.
Kibana: Web interface for searching and visualizing logs stored in elasticsearch, which is proxied through Nginx.
Filebeat: Lightweight Shipper of Logs from client to logstash server.

Prerequisites:
Minimum size to run your ES cluster
RAM --> 4GB
CPU --> 2 core
Disk --> 20 GB (highly varies on your log size)
You many need to increase RAM, CPU, Disk size depending on your log size.

Let's start on our main goal to setup ELK Server
Install java 8
sudo yum install java-1.8.0-openjdk
Change Java Home as Java 8
sudo sh -c "echo export JAVA_HOME=/usr/java/jdk1.8.0_60/jre >> /etc/environment"
Set Java 8 as system default java
sudo alternatives --config java
follow corresponding options to set java 8 as default system java.

Install ELK packages 
Import elastic-search repo key
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
Create yum repo file to install elastic-search packages
sudo vi /etc/yum.repos.d/elasticsearch.repo
paste following code in above file[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
save and exit file and then install elk packages.
Install Logstash
sudo yum install logstash
Install elasticsearch
sudo yum install elasticsearch
Install kibana
sudo yum install kibana

Install nginx
sudo yum install epel-release
sudo yum install nginx httpd-tools

Add all services on system reboot 
sudo chkconfig --add nginx
sudo chkconfig --add kibana
sudo chkconfig --add elasticsearch
sudo chkconfig --add logstash

Configure ELK stack
Configure Elasticsearch
Open file
sudo vi /etc/elasticsearch/elasticsearch.yml
Change 'network.host' value in above file 
network.host: private_ip_of_box

Configure Kibana
Open file
sudo vi /etc/kibana/kibana.yml
Change 'elasticsearch.url' value in above file 
elasticsearch.url: "http://elasticsearch_ip:9200"

Configure Nginx
Create and Open file
sudo vi /etc/nginx/conf.d/kibana.conf
And paste following code in above file
server {
  listen 80;

  server_name PUBLIC_IP_OF_SERVER;

  location / {
  proxy_pass http://localhost:5601;
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection 'upgrade';
  proxy_set_header Host $host;
  proxy_cache_bypass $http_upgrade;
  }
}

Configure Logstash
Create and open file to configure logstash to receive logs from filebeat clients
sudo vi /etc/nginx/conf.d/kibana.conf
Paste following code to receive combined Apache logs from central rsyslog server 
input {
  beats {
    port => 5044
    host => "current_server_private_ip"
  }
}

filter {
  if [type] == "central-apache-ssl-access"
  {
    grok {
      match => { "message" => "%{SYSLOGBASE} %{COMBINEDAPACHELOG} %{NUMBER:resptime_ms} %{QS:proxy_ip}" } }
      geoip {
      source => "clientip"}
      mutate {
      remove_field => [ "beat", "day", "host", "month", "tags", "source" ] }
  }

  else if [type] == "central-apache-access"
  {
    grok {
      match => { "message" => "%{SYSLOGBASE} %{COMBINEDAPACHELOG} %{NUMBER:resptime_ms} %{QS:proxy_ip}" } }
      geoip {
      source => "clientip"}
      mutate {
      remove_field => [ "beat", "day", "host", "month", "tags", "source" ] }
    }


  else  if [type] == "central-apache-error"
  {
    grok {
      match => { "message" => "%{SYSLOGBASE} \[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[.*:%{LOGLEVEL:loglevel}\] \[pid %{NUMBER:pid}] (?:\[client %{IPORHOST:clientip}:%{POSINT:port}\] ){0,1}(?<errormessage>(?:(?!, referer).)*)(?:, referer: %{GREEDYDATA:referer})?" } }
      geoip {
      source => "clientip"}
      mutate {
      remove_field => [ "beat", "day", "host", "month", "tags", "source" ] }
  }

  else  if [type] == "central-apache-ssl-error"
  {
    grok {
      match => { "message" => "%{SYSLOGBASE} \[(?<timestamp>%{DAY:day} %{MONTH:month} %{MONTHDAY} %{TIME} %{YEAR})\] \[.*:%{LOGLEVEL:loglevel}\] \[pid %{NUMBER:pid}] (?:\[client %{IPORHOST:clientip}:%{POSINT:port}\] ){0,1}(?<errormessage>(?:(?!, referer).)*)(?:, referer: %{GREEDYDATA:referer})?" } }
      geoip {
      source => "clientip"}
      mutate {
      remove_field => [ "beat", "day", "host", "month", "tags", "source" ] }
  }
}


output {
  elasticsearch {
    hosts => [ "elasticserach_ip:9200" ]
  }
}

If all good then Start services one by one
Start elasticsearch
sudo service elasticsearch start
Start kibana
sudo service kibana start
Start nginx 
sudo service nginx start
Start logstash 
sudo initctl start logstash
If logstash is not being started and you are getting 'unknown job logstash' then do following steps

 Create and open new file
sudo vi /etc/init/logstash.conf
And paste following code 
description "logstash"
start on filesystem or runlevel [2345]
stop on runlevel [!2345]

respawn
umask 022
nice 19
limit nofile 16384 16384
chroot /
chdir /

#limit core <softlimit> <hardlimit>
#limit cpu <softlimit> <hardlimit>
#limit data <softlimit> <hardlimit>
#limit fsize <softlimit> <hardlimit>
#limit memlock <softlimit> <hardlimit>
#limit msgqueue <softlimit> <hardlimit>
#limit nice <softlimit> <hardlimit>
#limit nofile <softlimit> <hardlimit>
#limit nproc <softlimit> <hardlimit>
#limit rss <softlimit> <hardlimit>
#limit rtprio <softlimit> <hardlimit>
#limit sigpending <softlimit> <hardlimit>
#limit stack <softlimit> <hardlimit>

script
  # When loading default and sysconfig files, we use `set -a` to make
  # all variables automatically into environment variables.
  set -a
  [ -r "/etc/default/logstash" ] && . "/etc/default/logstash"
  [ -r "/etc/sysconfig/logstash" ] && . "/etc/sysconfig/logstash"
  set +a
  exec chroot --userspec logstash:logstash / /usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash" >> /var/log/logstash-stdout.log 2>> /var/log/logstash-stderr.log
end script

Save and Exit file and start Logstash 
sudo initctl start logstash

Check if any error there 
sudo tail -f /var/log/logstash/logstash-plain.log

sudo tail -f /var/log/elasticsearch/elasticsearch.log
sudo tail -f /var/log/nginx/access.log
If all good then your elk stack is ready to receive logs.


Configure client to sent logs to elk server

Install and configure Filebeat
Create and open file
sudo vi /etc/yum.repos.d/elastic.repo
And paste following code 
[elastic-5.x]
name=Elastic repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
Save and exit file.

Install filebeat 
sudo yum install filebeat
Start filebeat on system reboot 
sudo chkconfig --add filebeat
Backup original file 
sudo cp /etc/filebeat/filebeat.yml /etc/filebeat/filebeat.yml.bak

Configure filebeat
Open file
sudo vi /etc/filebeat/filebeat.yml
Add following code near 'input_type'
- input_type: log

  paths:
  - /var/log/apache_access_log
  document_type: central-apache-access

  paths:
  - /var/log/apache_ssl_access_log
  document_type: central-apache-ssl-access

- input_type: log
  paths:
  - /var/log/apache_error_log
  document_type: central-apache-error

- input_type: log
  paths:
  - /var/log/apache_ssl_error_log
  document_type: central-apache-ssl-error

Save and exit and check filebeat syntax 
sudo filebeat.sh -configtest

Now by default filebeat forward log to Elasticsearch so change it to Logstash 
sudo vi /etc/filebeat/filebeat.yml

Search for 'Elasticsearch output' and comment ES output conf uncomment Logstash output conf and replace Logstash host IP with your elkbox private ip and check filebeat syntax.
sudo filebeat.sh -configtest

Start filebeat
sudo service filebeat start

Check if any error there 
sudo tail -f /var/log/filebeat/filebeat

If all good then your pipeline is started to ship log to elk sever.

Happy ELK Stack..!!
Labels:

Comments

Post a Comment

Popular posts from this blog

Curator

Goal: In these tutorial we gonna cover deletion of old logs in ELK Stack. We gonna achive these by deleting old indices created by Logstash while dumping logs in Elasticsearch. Prerequisites: Old logs to delete... 😜😜 Let's Begin the exercise: Install curator Curator is a package in Elasticsearch  repository to delete old indices. Create a file sudo vi /etc/yum.repos.d/curator.repo paste following lines Save and Exit file Run yum install sudo yum install elasticsearch-curator Configure Curator Create a directory mkdir ~/.curator/ Open a file sudo vi ~/.curator/curator.yml paste following code Save and Exit file Deletion pattern Create file to define delete pattern in Elasticesearch sudo vi ~/.curator/delete_indices.yml paste following lines in file Create a log file for curator on the location you defined in configuration, and assign permission to right into file. sudo touch /var/log/curator #to assign permission to write l...
Post a Comment
Read more

GoReplay - Testing Your Site with Actual Traffic

Goal:   In these article we gonna learn How to capture your Real Time traffic from production and reuse it at your testing/development environment. Prerequisite: One web server running, or If you are just playing around then you can run goreplay test ftp server. Let's Begin Load Testing for site serving millions user wasn't be that easy before I came to know GoReplay . Here I am not gonna explain you How great go replay is, You will automatically get to know after following steps above step to capture and replay your request logs. FYI GoReplay capture logs from tcpdump. Installation: Download zip file from there git repo and unzip it. # create a directory mkdir ~/goreplay # go to directory you created cd ~/goreplay # download tar file from goreplay git repo wget https://github.com/buger/goreplay/releases/download/v0.16.1/gor_0.16.1_x64.tar.gz # unzip it tar -xf gor_0.16.1_x64.tar.gz After Unzipping Check GoReplay binary File is available in directory. Ca...
Post a Comment
Read more

Install Central Logging on Amazon Linux

Goal: In these tutorial we gonna cover setup of central logging system on amazon linux (CentOs) in same aws vpc . We will setup one central log server to receive log using rsyslog, after that we will setup one client to forward apache & syslog to central server. we already covered forward logs from central log server to ELK stack for analyzing. Logging Stack Component: Central Log server Multiple logging client server/Any apache web server generating logs Rsyslog: we setup with rsyslog v8-stable. You can use any rsyslog  version after rsyslog-6, because we encountered rsyslog drop message in earlier version. Prerequisites: Rsyslog is quite light weight, we doesn't requirement any high configuration machine, aws t2.micro should be enough. We are running t2.micro in production for central log server to receive around 1000 log entry/second, server is using less then 2 percent/sec within same vpc. Now Let's Start we gonna break these tutorial in two pa...
Post a Comment
Read more