Piyush Tech Blog

Goal: In these tutorial we gonna cover setup of central logging system on amazon linux (CentOs) in same aws vpc . We will setup one central log server to receive log using rsyslog, after that we will setup one client to forward apache & syslog to central server. we already covered forward logs from central log server to ELK stack for analyzing. Logging Stack Component: Central Log server Multiple logging client server/Any apache web server generating logs Rsyslog: we setup with rsyslog v8-stable. You can use any rsyslog  version after rsyslog-6, because we encountered rsyslog drop message in earlier version. Prerequisites: Rsyslog is quite light weight, we doesn't requirement any high configuration machine, aws t2.micro should be enough. We are running t2.micro in production for central log server to receive around 1000 log entry/second, server is using less then 2 percent/sec within same vpc. Now Let's Start we gonna break these tutorial in two pa

Goal: In these tutorial we gonna cover installation of ELK Stack on fresh amazon ec2 linux (CentOS). We will install Elasticsearch 5.x.x, Logstash 5.x.x, and Kibana 5.x.x. We will also show you how to configure filebeat to forwards apache logs collected by central rsyslog server to elk server using Filebeat 5.x.x. ELK stack components: Logstash: Transform incoming logs. Elasticsearch(ES): Stores logs transformed by logstash. Kibana: Web interface for searching and visualizing logs stored in elasticsearch, which is proxied through Nginx. Filebeat: Lightweight Shipper of Logs from client to logstash server. Prerequisites: Minimum size to run your ES cluster RAM --> 4GB CPU --> 2 core Disk --> 20 GB (highly varies on your log size) You many need to increase RAM, CPU, Disk size depending on your log size. Let's start on our main goal to setup ELK Server Install java 8 sudo yum install java-1.8.0-openjdk Change Java Home as Java 8 sudo sh -c "echo expo