rSyslog5 – centralized logging

The goal was to set up a simple syslog server for centralized logging, able to receive from any syslog-enabled device, but also able to talk to machines with more advanced capabilities, like logging over TCP/IP so messages wouldn’t be lost if connection dropped for a while or sending messages over encrypted channels. Of course, it should also be able to send those messages to any file based on sender, time, or any other arbitrary condition. rSyslog fits nicely. As a bonus it also sends them to a database to be picked up by log analyzers. syslog-ng is the alternative.

Why rSyslog and not syslog-ng? I’m not really sure, look for comparisons around the web. Major Linux distros chose to switch from syslog-ng to rSyslog, rSyslog’s conf files are backwards compatible with the stock BSD syslogd, some (old?) reports say that rSyslog performs better. But they are both under development, new features get added all the time and rough edges get smoothed out. Point is all comparisons I found are more or less outdated, rSyslog seems to have an edge, but that might change. My personal reasons were the backwards compatibility which enabled me to make the transition smoothly over time and the availability of Adiscon’s Log Analyzer for simple and fast setup along with the Windows logging tools if I would ever need to expand that way. All of these form a nice package and go back to the same man, Rainer Gerhards, the author of rSyslog, which means they are more likely to work together without making much fuss about it.

Installation on Linux depends on your distro, won’t be covered here, it might even be already there. For FreeBSD there are ports of different versions of rSyslog under sysutils along with their respective modules.

On to the config. The file is read from top to bottom, every instruction (filter) being executed as it’s encountered, like a firewall config file. This is important because you can discard a message at some point in the file and the following statements won’t affect it. It makes the conf have the following basic structure:

Global Directives - basically these are instructions read by the daemon when it's started the first time, they don't affect the way messages are filtered. What modules to load, what ports to listen on, stuff like that, general rSyslog settings. These start with a $ sign.
Templates - these are.. well, templates for different things, like file names that contain the IP address of the sender or log formats. These need to be declared before they are used.
Output channels - basically used for log rotation based on file size, but they might be replaced with something else so I didn't use them.
Rules - that's what does the actual logging. Rules consist of two parts, a filter condition and an action. They basically say "if the messages meets the criteria for this filtering condition then apply this action to it (i.e. log it to this file, in this form)".

Setting rSyslog up to do what the old syslogd was doing is easy. Copy the old config to rsyslog’s file and add these two lines at the top of the file:

$ModLoad imuxsock.so
$ModLoad imklog.so

First line loads the module that provides support for local system logging, as in logging for daemons that run on the local system. Second line loads the module that does logging for the kernel messages. These make for the GlobalDirectives part of the config. The rest of the lines are the Rules.

To exemplify, the simplest config file:

$ModLoad imuxsock.so
$ModLoad imklog.so

*.*            /var/log/all.log

All this does is send everything to /var/log/all.log. Nothing else. There’s no filtering done and the daemon doesn’t listen on any ports for messages sent to it from other machines. So let’s make it listen on the default port for UDP messages and add a DynamicFile template so that messages from different devices go to different files based on the sender’s IP address and year/month.

# GLOBALS
$ModLoad imuxsock.so
$ModLoad imklog.so
$ModLoad imudp.so    	# standard udp logging (syslog)
$UDPServerRun 514    	#

# TEMPLATES
$template DynAll,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_all.log"

# RULES
*.*                                                 ?DynAll

Module imudp.so provides the listening-on-udp functionality, with UDPServerRun being the setting that tells it what port to listen on, in this case the default 514 port.

The template follows the format $template TemplateName,"TemplateFormat". It will generate a file that will have a name similar to /usr/local/var/log/192.168.1.12/2011-12_all.log. I’m doing this on FreeBSD and I’ve chosen /usr/local as the location because that’s the big partition and there are gonna be a lot of files. The logs are in different directories according to the IP of the sending host. I could’ve also used %HOSTNAME% instead of %FROMHOST-IP%, but on our configuration with a host having multiple names this generated multiple directories for the same host.

Finally, the rule line now specifies the template name as the action to take, meaning “log to the file determined by expanding this template”

So far so good, wrapping it up with a config file that actually does something useful. What the old syslog did, but with the added benefit of TCP connections and different log files by host/month:

######################################################################
# GLOBALS
$ModLoad imuxsock.so 	# provides support for local system logging
$ModLoad imklog.so   	# kernel logging
$ModLoad imudp.so    	# standard udp logging (syslog)
$UDPServerRun 514    	#
$ModLoad imtcp	        # standard tcp logging
$InputTCPServerRun 514
# setting up permissions
$FileOwner root
$FileGroup wheel
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
## CAN'T DROP
#$PrivDropToUser syslog
#$PrivDropToGroup syslog
######################################################################

######################################################################
# TEMPLATES
# dynamic directory by host
$template DynAll,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_all.log"
$template DynMessages,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_messages.log"
$template DynSecurity,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_security.log"
$template DynAuth,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_auth.log"
$template DynMail,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_mail.log"
$template DynLpr,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_lpd-errs.log"
$template DynFtp,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_xfer.log"
$template DynCron,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_cron.log"
$template DynDebug,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_debug.log"
######################################################################

######################################################################
# RULES
*.err;kern.warning;auth.notice;mail.crit            /dev/console
*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err	-?DynMessages;RSYSLOG_SyslogProtocol23Format
security.*                                          -?DynSecurity;RSYSLOG_SyslogProtocol23Format
auth.info;authpriv.info                             -?DynAuth;RSYSLOG_SyslogProtocol23Format
mail.info                                           -?DynMail;RSYSLOG_SyslogProtocol23Format
lpr.info                                            -?DynLpr;RSYSLOG_SyslogProtocol23Format
ftp.info                                            -?DynFtp;RSYSLOG_SyslogProtocol23Format
cron.*                                              -?DynCron;RSYSLOG_SyslogProtocol23Format
*.=debug                                            -?DynDebug;RSYSLOG_SyslogProtocol23Format
*.emerg                                             *
*.*                                                 -?DynAll;RSYSLOG_SyslogProtocol23Format

A few additions to the config here:

  • lines 7 - 8 : enable TCP listener on default port
  • lines 10 - 14 : set the default permissions for new log files
  • lines 16 - 17 : allows the daemon to drop permissions to a different user after starting, but there are certain restrictions to that, so it’s just here as a side note, something to be aware of
  • lines 23 - 31 : templates for the log files used in the following rules
  • lines 36 - 46 : a set of more useful rules, this is the default set of rules on FreeBSD, adjust according to your needs. Notice the dash in front of the file names, it tells rSyslog not to force a file sync after every write. That improves performance but it means that you might lose some records if the power goes out suddenly. The “RSYSLOG_SyslogProtocol23Format” after the file name is a template name for a default template, “the format specified in IETF’s internet-draft ietf-syslog-protocol-23, which is assumed to become the new syslog standard RFC”. It has since turned into RFC5424.

APACHE REMOTE LOGGING

Apache has two logging mechanisms, one for errors, the error log, and one for everything else, the access log. Only the error log can be sent directly to syslog though, the access log needs to be piped through and external program.

For the error log it’s easy. Put this in apache’s conf file:

ErrorLog syslog:local3

It tells it to send everything to the local syslog daemon under log facility local3. The local syslogd will then forward the messages to remote like any other log. It has different log levels for the error log, so filtering on something like local3.crit is possible.

There’s a little trick that needs to be done for the access log. Adding this line to apache’s conf file:

CustomLog "|/usr/bin/logger -t httpd -p local4.info" combined

tells the daemon to pipe the access log through logger, which is a little program that sends anything it receives to syslog. This will log everything under local4.info, there are no levels, but that’s ok, rSyslog has a whole lot of filtering options, so no worries there. You can have multiple log lines in apache’s configuration, so you can send to syslog and also keep your local /var/log/http/ file.

Now the rSyslog conf on the remote server. First two log file templates for the two logs, access and error respectively:

$template DynHttpdAccess,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_httpd_access.log"
$template DynHttpdError,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_httpd_error.log"

and the rules to send the logs to the proper files:

local4.*                                            -?DynHttpdAccess
local3.*                                            -?DynHttpdError

On a small network this might be enough, but as local facilities aren’t reserved for anything there might be other devices, like Cisco routers for example, logging to them and the logs would get mangled, so we need to add a condition to the rules that is specific to the http daemon:

if $syslogfacility-text == 'local4' and $programname == 'httpd' then -?DynHttpdAccess;RSYSLOG_SyslogProtocol23Format
if $syslogfacility-text == 'local4' and $programname == 'httpd' then ~
if $syslogfacility-text == 'local3' and $programname == 'httpd' then -?DynHttpdError;RSYSLOG_SyslogProtocol23Format
if $syslogfacility-text == 'local3' and $programname == 'httpd' then ~

The tilde (~) sign on the second and fourth lines tells rSyslog to discard the message if the condition is met. Which means that it stops processing the following lines and moves on to the next message. Which means that the httpd messages only get logged to the proper files and won’t overlap with anything. So these rules go on the top, before other more general rules. The final config so far looks like this:

######################################################################
# GLOBALS
$ModLoad imuxsock.so 	# provides support for local system logging
$ModLoad imklog.so   	# kernel logging
$ModLoad imudp.so    	# standard udp logging (syslog)
$UDPServerRun 514    	#
$ModLoad imtcp	        # standard tcp logging
$InputTCPServerRun 514
# setting up permissions
$FileOwner root
$FileGroup wheel
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
## CAN'T DROP
#$PrivDropToUser syslog
#$PrivDropToGroup syslog
######################################################################

######################################################################
# TEMPLATES
# dynamic directory by host
$template DynAll,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_all.log"
$template DynMessages,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_messages.log"
$template DynSecurity,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_security.log"
$template DynAuth,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_auth.log"
$template DynMail,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_mail.log"
$template DynLpr,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_lpd-errs.log"
$template DynFtp,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_xfer.log"
$template DynCron,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_cron.log"
$template DynDebug,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_debug.log"
$template DynHttpdAccess,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_httpd_access.log"
$template DynHttpdError,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_httpd_error.log"
######################################################################

######################################################################
# RULES
if $syslogfacility-text == 'local4' and $programname == 'httpd' then -?DynHttpdAccess;RSYSLOG_SyslogProtocol23Format
if $syslogfacility-text == 'local4' and $programname == 'httpd' then ~
if $syslogfacility-text == 'local3' and $programname == 'httpd' then -?DynHttpdError;RSYSLOG_SyslogProtocol23Format
if $syslogfacility-text == 'local3' and $programname == 'httpd' then ~

*.err;kern.warning;auth.notice;mail.crit            /dev/console
*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err	-?DynMessages;RSYSLOG_SyslogProtocol23Format
security.*                                          -?DynSecurity;RSYSLOG_SyslogProtocol23Format
auth.info;authpriv.info                             -?DynAuth;RSYSLOG_SyslogProtocol23Format
mail.info                                           -?DynMail;RSYSLOG_SyslogProtocol23Format
lpr.info                                            -?DynLpr;RSYSLOG_SyslogProtocol23Format
ftp.info                                            -?DynFtp;RSYSLOG_SyslogProtocol23Format
cron.*                                              -?DynCron;RSYSLOG_SyslogProtocol23Format
*.=debug                                            -?DynDebug;RSYSLOG_SyslogProtocol23Format
*.emerg                                             *
*.*                                                 -?DynAll;RSYSLOG_SyslogProtocol23Format

Easy enough. When adding a new web server to the network one needs only add the proper lines to apache’s config and local syslog and it all gets automagically sent to it’s proper place. Doesn’t get much easier than this.

LOGGING TO A DATABASE AND LOGANALYZER

(yet to be explained)

$ModLoad immark.so   	# provides --MARK-- message capability
$ModLoad imuxsock.so 	# provides support for local system logging
$ModLoad imklog.so   	# kernel logging
$ModLoad imudp.so    	# standard udp logging (syslog)
$UDPServerRun 514    	#
$ModLoad imtcp	        # standard tcp logging
$InputTCPServerRun 514
$ModLoad ommysql        # load MySql output module

# setting up permissions
$FileOwner root
$FileGroup wheel
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
## CAN'T DROP
#$PrivDropToUser syslog
#$PrivDropToGroup syslog

# SEND EVERYTHING TO TEST SERVER
# TEMPORARY
#*.*	@@192.168.4.244

## TEMPLATES
#####################################################
# Log everything to a per host daily logfile        #
#####################################################
#$template DailyPerHostLogs,"/var/log/syslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/messages.log"

# dynamic directory by hostname
$template DynAll,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_all.log"
$template DynMessages,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_messages.log"
$template DynSecurity,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_security.log"
$template DynAuth,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_auth.log"
$template DynMail,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_mail.log"
$template DynLpr,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_lpd-errs.log"
$template DynFtp,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_xfer.log"
$template DynCron,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_cron.log"
$template DynDebug,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_debug.log"
$template DynHttpdAccess,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_httpd_access.log"
$template DynHttpdError,"/usr/local/var/log/%FROMHOST-IP%/%$YEAR%-%$MONTH%_httpd_error.log"

# apache logs, sent thorugh local3.* (error) and local4.info (access) - log and discard
#local4.*                                            -?DynHttpdAccess
#local3.*                                            -?DynHttpdError
if $syslogfacility-text == 'local4' and $programname == 'httpd' then -?DynHttpdAccess;RSYSLOG_SyslogProtocol23Format
if $syslogfacility-text == 'local4' and $programname == 'httpd' then :ommysql:127.0.0.1,dbWeb,syslogUser,syslogPassword
if $syslogfacility-text == 'local4' and $programname == 'httpd' then ~
if $syslogfacility-text == 'local3' and $programname == 'httpd' then -?DynHttpdError;RSYSLOG_SyslogProtocol23Format
if $syslogfacility-text == 'local3' and $programname == 'httpd' then ~

*.err;kern.warning;auth.notice;mail.crit            /dev/console
*.notice;authpriv.none;kern.debug;lpr.info;mail.crit;news.err	-?DynMessages;RSYSLOG_SyslogProtocol23Format
security.*                                          -?DynSecurity;RSYSLOG_SyslogProtocol23Format
auth.info;authpriv.info                             -?DynAuth;RSYSLOG_SyslogProtocol23Format
mail.info                                           -?DynMail;RSYSLOG_SyslogProtocol23Format
lpr.info                                            -?DynLpr;RSYSLOG_SyslogProtocol23Format
ftp.info                                            -?DynFtp;RSYSLOG_SyslogProtocol23Format
cron.*                                              -?DynCron;RSYSLOG_SyslogProtocol23Format
*.=debug                                            -?DynDebug;RSYSLOG_SyslogProtocol23Format
*.emerg                                             *
*.*                                                 -?DynAll;RSYSLOG_SyslogProtocol23Format

# logging to database starts here
# discard all mail.*
mail.* ~
# discard info messages from zimbramon, too spammy
if $programname == 'zimbramon' and $syslogseverity-text == 'info' then ~
# send everything that got this far to mysql database
*.*						    :ommysql:127.0.0.1,dbSyslog,syslogUser,syslogPassword