Skip to main content

Install Central Logging on Amazon Linux


Goal:
In these tutorial we gonna cover setup of central logging system on amazon linux (CentOs) in same aws vpc . We will setup one central log server to receive log using rsyslog, after that we will setup one client to forward apache & syslog to central server.
we already covered forward logs from central log server to ELK stack for analyzing.


 Logging Stack Component:
Central Log server
Multiple logging client server/Any apache web server generating logs
Rsyslog: we setup with rsyslog v8-stable. You can use any rsyslog  version after rsyslog-6, because we encountered rsyslog drop message in earlier version.

 Prerequisites:
Rsyslog is quite light weight, we doesn't requirement any high configuration machine, aws t2.micro should be enough.
We are running t2.micro in production for central log server to receive around 1000 log entry/second, server is using less then 2 percent/sec within same vpc.

 Now Let's Start we gonna break these tutorial in two parts
  1. Central Log Server Setup
  2. Client Server Setup (Log Forwarding on different aws machine)

Central Log Server Setup

 First create an ec2 Machine within same VPC where your logging client server exists. Launch ec2 instance with around 20 GB EBS ebs size may vary depending on your log size.

 As soon as your machine is up, then ssh to machine Mount EBS to a directory where you are planning to store logs on logging server.

 Process to Mount 
sudo mkdir -p /var/log/prod-group/
#check file system of your EBS 
sudo file -s /dev/xvdb
#if not ext4, then make it ext4
sudo mkfs -t ext4 /dev/xvdb
#check again file system should be ext4
sudo file -s /dev/xvdb
#mount EBS to your logging storage directory
sudo mount /dev/xvdb /var/log/prod-group/
#just to make sure directory is being mounted
df -h 

 Add Entry in fstab to auto mount on reboot
open fstab file 
sudo vi /etc/fstab
paste following line 
/dev/xvdb /var/log/prod-group/ ext4 defaults,nofail,comment=cloudconfig 0 2

reboot machine just to make sure EBS is being mount on reboot 
sudo reboot

just to make sure directory is auto mounted after reboot 
df -h

 Update rsyslog to avoid message drop bug at scale

 Create new file 
sudo vi /etc/yum.repos.d/rsyslog.repo

paste following line 
[rsyslog-v8-stable]
name=Adiscon Rsyslog v8-stable for CentOS-$releasever-$basearch
baseurl=http://rpms.adiscon.com/v8-stable/epel-6/$basearch
enabled=1
gpgcheck=0
protect=1

to upgrade rsyslog version 
sudo yum upgrade rsyslog --disablerepo=amzn-main

Now Configure Rsyslog To Listen at UDP port 514

Uncomment few lines in rsyslog conf file
Open rsyslog conf file 
sudo vi /etc/rsyslog.conf

Search for line 'module(load="imudp")'
Un-comment following two lines, after uncommenting it will look like
module(load="imudp") # needs to be done just once
input(type="imudp" port="514")
Save and Exit conf File
Now Check Syntax rsyslog 
rsyslogd -N1

 If All Good then restart rsyslog 
sudo service rsyslog restart
Check UDP port 514 is open for listing 
netstat -anp |grep 514

 Change few more things, In case you wanna store different clients log in different directory and different files for different type of messages
To achieve that open rsyslog conf file 
sudo vi /etc/rsyslog.conf

 Serach for '/var/log/messages' and comment that line
Serach for '/var/log/secure' and comment that line
Serach for '/var/log/maillog' and comment that line
Serach for '/var/log/cron' and comment that line
Serach for '/var/log/spooler' and comment that line
Serach for '/var/log/boot.log' and comment that line

 Now Add following template at the end of file to store logs in different directories and different files.
Save and Exit conf File Now Check Syntax rsyslog 
rsyslogd -N1

 If All Good then restart rsyslog 
sudo service rsyslog restart

Now Add Logrotate to rotate logs daily, as well auto sync on aws s3.

Create new file 
sudo vi /etc/logrotate.d/syslog-central

And Paste Following lines 
/var/log/prod-group/*/messages
/var/log/prod-group/*/secure
/var/log/prod-group/*/maillog
/var/log/prod-group/*/cron
/var/log/prod-group/*/spooler
/var/log/prod-group/*/boot.log
/var/log/prod-group/*/apache_access_log
/var/log/prod-group/*/apache_error_log
/var/log/prod-group/*/apache_ssl_access_log
/var/log/prod-group/*/apache_ssl_error_log
{
    rotate 4
    daily
    missingok
    dateext
    compress
    delaycompress
    notifempty
    sharedscripts
    postrotate
        sudo /etc/init.d/rsyslog restart
        sudo /usr/bin/aws s3 sync --exclude "*" --include "*.gz" /var/log/prod-group/ s3://{LOG_BUCKET}/ --profile AWS_S3SYNC_PROFILE
    endscript

}

Note:
I am assuming you have aws cli setup on root user and have permission to sync logs on relevant s3 bucket.

Central Logging Server Setup is Finished.

-------------------

Setup Log Forwarding on clients 

ssh to client server where you want to start log forwarding

Change Hostname so it will be easy to recognize logs at central log server.

hostnameNodot=xyz-com
#Run following Command
echo "127.0.0.1 <hostnameNodot> localhost localhost.localdomain" | tee /etc/hosts
#Open following file
sudo vi /etc/sysconfig/network
#change 
HOSTNAME=<hostnameNodot>
#Run command to change host before reboot
hostname <hostnameNodot>

Upgrade rsyslog to avoid message drop bug at scale

 Create new file 
sudo vi /etc/yum.repos.d/rsyslog.repo

paste following line 
[rsyslog-v8-stable]
name=Adiscon Rsyslog v8-stable for CentOS-$releasever-$basearch
baseurl=http://rpms.adiscon.com/v8-stable/epel-6/$basearch
enabled=1
gpgcheck=0
protect=1

to upgrade rsyslog version 
sudo yum upgrade rsyslog --disablerepo=amzn-main

Check version 
rsyslogd -v

Check syntax 
rsyslogd -N1

If you get following error
rsyslogd: error during parsing file /etc/rsyslog.conf, on or before line 52: warnings occurred in file '/etc/rsyslog.conf' around line 52 [v8.27.0 try http://www.rsyslog.com/e/2207 ]

Then Search '*.emerg' in rsyslog file and replace that line with

*.emerg               :omusrmsg:*

If cloud init file is there, Then open file

sudo vi /etc/rsyslog.d/21-cloudinit.conf


replace '~' with 'stop'
Note: '~' action is deprecated, us 'stop' statement instead.

Check syntax again
rsyslogd -N1

If all good restart rsyslog
sudo service rsyslog restart


Now to add Apache logs to syslog

Open a file
sudo vi /etc/httpd/conf/httpd.conf

Check for logformat to maintain same apache log format at all central log client server

Search for 'LogFormat', it should be look like following, otherwise you should replace with following
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %{ms}T \"%{X-Forwarded-For}i\"" myownlog

Search for 'CustomLog "logs/access_log"'
Comment that line and based on your requirement add the respective line

<to keep a local copy>
CustomLog "|$/usr/bin/tee -a /var/log/httpd/access_log | /usr/bin/logger -t apache -p local6.info" myownlog
<OR>
CustomLog "|/usr/bin/logger -t apache -p local6.info" myownlog

Search for 'ErrorLog "logs/error_log"'
Comment that line and based on your requirement add the respective line

<Keep a local copy>
ErrorLog "|$/usr/bin/tee -a /var/log/httpd/error_log | /usr/bin/logger -t apache -p local6.err"
<OR>
ErrorLog "|/usr/bin/logger -t apache -p local6.err"
Save and Exit file.

Open ssl conf file, in my case location of file is 
sudo vi /etc/httpd/conf.d/ssl.conf

Check Logformat is same as below or not, If not then replace with it. 
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %{ms}T \"%{X-Forwarded-For}i\""

Search for 'ErrorLog logs/ssl_error_log'
Comment that line and based on your requirement add the respective line

<Keep a local copy>
ErrorLog "|$/usr/bin/tee -a /var/log/httpd/ssl_error_log | /usr/bin/logger -t apachessl -p local7.err"
<OR>
ErrorLog "|/usr/bin/logger -t apachessl -p local7.err"

Search for 'TransferLog logs/ssl_access_log'
Comment that line and based on your requirement add the respective line

<Keep a local copy>
TransferLog "|$/usr/bin/tee -a /var/log/httpd/ssl_access_log | /usr/bin/logger -t apachessl -p local7.info"
<OR>
TransferLog "|/usr/bin/logger -t apachessl -p local7.info"

To check apache conf syntax 
httpd -t

If firewall is install then open udp out port 514 in csf.

Open file 
sudo vi /etc/csf/csf.conf

Add 514 to UDP_OUT port list. Restart csf/firewal 
sudo service lfd restart
sudo csf --restart
 
Set up rsyslog to send log streams to central log server.
Search for 'end of the forwarding rule' Add the following line above that...
*.* @cental_logging_server_private_ip:514

Save and exit file.

Before restarting rsyslog, please check client server is allowed to forward logs to central log server. (Security-Group/Firewall of both client as well as server)


#check rsyslog syntax
rsyslogd -N1

#If all good restart rsyslog
sudo service rsyslog restart

#Reload apache
sudo service httpd reload

Now your Logs should start streaming to central log server.


Hope this article helped you.....!!
Labels:

Comments

Post a Comment

Popular posts from this blog

Curator

Goal: In these tutorial we gonna cover deletion of old logs in ELK Stack. We gonna achive these by deleting old indices created by Logstash while dumping logs in Elasticsearch. Prerequisites: Old logs to delete... 😜😜 Let's Begin the exercise: Install curator Curator is a package in Elasticsearch  repository to delete old indices. Create a file sudo vi /etc/yum.repos.d/curator.repo paste following lines Save and Exit file Run yum install sudo yum install elasticsearch-curator Configure Curator Create a directory mkdir ~/.curator/ Open a file sudo vi ~/.curator/curator.yml paste following code Save and Exit file Deletion pattern Create file to define delete pattern in Elasticesearch sudo vi ~/.curator/delete_indices.yml paste following lines in file Create a log file for curator on the location you defined in configuration, and assign permission to right into file. sudo touch /var/log/curator #to assign permission to write l
Post a Comment
Read more

GoReplay - Testing Your Site with Actual Traffic

Goal:   In these article we gonna learn How to capture your Real Time traffic from production and reuse it at your testing/development environment. Prerequisite: One web server running, or If you are just playing around then you can run goreplay test ftp server. Let's Begin Load Testing for site serving millions user wasn't be that easy before I came to know GoReplay . Here I am not gonna explain you How great go replay is, You will automatically get to know after following steps above step to capture and replay your request logs. FYI GoReplay capture logs from tcpdump. Installation: Download zip file from there git repo and unzip it. # create a directory mkdir ~/goreplay # go to directory you created cd ~/goreplay # download tar file from goreplay git repo wget https://github.com/buger/goreplay/releases/download/v0.16.1/gor_0.16.1_x64.tar.gz # unzip it tar -xf gor_0.16.1_x64.tar.gz After Unzipping Check GoReplay binary File is available in directory. Ca
Post a Comment
Read more