Syslog-ng : log Consolidation

Application teams some times require to save their logs for a longer period so that they can analyze them after some time. In most cases the server that are running the application holds the logs files , but in production environment there is always a issues of Disk Space.

How can we send log files to a different location which can be used as a Log server?

Syslog-ng is a pacakge available in linux which can be used to send logs for a log server for storing them.

From WIKI

Syslog-ng is an open source implementation of the Syslog protocol for Unix and Unix-like systems. It extends the original syslogd model with content-based filtering, rich filtering capabilities.

So how can we use the syslog-ng.

1.Download the package from http://www.balabit.com/

2.For a Web Application ( On the Sender Side )

For a web application , we can use the log4j to do the Sysnog configuration which send a content to the syslog server. The configuration looks like this

<!-- ====================================== -->

<!-- Append messages to the a remote syslog -->

<!-- ====================================== -->

<appender name="ABC-A1_SYS" class="org.productivity.java.syslog4j.impl.log4j.Syslog4jAppender">

<param name="Facility" value="user"/>

<param name="Protocol" value="tcp"/>

<param name="Host" value="198.12.34.22"/>

<param name="port" value="59503" />

<param name="threshold" value="ALL"/>

<param name="ident" value="abc" />

<param name="maxMessageLength" value="1000000"/>

<layout class="org.apache.log4j.PatternLayout">

<param name="ConversionPattern" value="[%d{ISO8601}] [das] [$] [%p] [%c{3}] %m%n"/>

</layout>

<filter class="org.apache.log4j.varia.LevelRangeFilter">

<param name="LevelMin" value="DEBUG" />

<param name="LevelMax" value="FATAL" />

</filter>

</appender>

So in this log4j configuration we configured the syslog which can be used to send the log content to 198.12.34.22 IP address on port 59503.Ini the above configuration the ident value is important as we use that to send content to the Receiver.

Now configure your application logger using

<logger name="com.sample.app..common" additivity="false">

<level value="info" />

<appender-ref ref="file" />

<appender-ref ref="ABC-A1_SYS" />

</logger>

<!-- Scheduled Jobs Logs →

We need to add the line in root logger in the log4j configuration.

<root>

<level value="ERROR" />

<appender-ref ref="file" />

<appender-ref ref="ABC-A1_SYS" />

</root>

Once this is done , deploy you application with the above log4j configuration.

3.Now we need to configure the Receiver side where the logs are to be saved , we need to configure the Syslog-ng configuration like

#----------------------------------------------------------------------

# Options Which tells about the Owner , port information

#----------------------------------------------------------------------

options

{

owner(root);

group(root);

log_fifo_size(8192);

perm(0664);

sync(0);

use_dns(no);

};

#--------------------------------------------------------------------------------

# Sources from where the Content can come or the receiver should read

#--------------------------------------------------------------------------------

source source(s_crpchipper) { udp(ip(0.0.0.0) port(59503)); };

In the above line we use the same port as the one used in our web application log4j configuration

and next , we have

#Filters for event handlers

filter f_abc {match('\[abc\]');}; # Filter the content coming on the Port using the identi value

We use the identi value in here

#destinations

destination d_abc { file("/logs/syslog/conf/dev/abc/abc-$MONTH-$DAY.log");}; # the location of the file where the content is to be pushed

#Logging

log { source(s_crpchipper); filter(f_abc); destination(d_abc); flags(final); };

Configure these in the file

/syslog-ng/etc/syslog-ng/syslog-ng.conf

Once the Configuration is done , just start the Process using

/syslog-ng/sbin/syslog-ng -f /syslog-ng/etc/syslog-ng/syslog-ng.conf

Now the Process is up and running , so when ever the web application generates log content , the log will also be saved on the 198.12.34.22 server at location logs/syslog/conf/dev/abc/abc-$MONTH-$DAY.log.

There may be a small latency for the logs being updated.