BOFH Hunter

Centralized logging with CentOS and rsyslog

While more IT shops are turning to centralized logging, the default syslog in centos lacks the feature set many admins want when it comes to centralized logging. Some people have moved to syslog-ng which boasts many features that system admins want for log generation, however it’s also got some substantial drawbacks.

With syslog-ng, there is a dual-licensing model which may not give you everything you want, or provide current code without some additional cash expenditure for the full paid version. This is actually one of the reasons Fedora listed for not choosing syslog-ng as syslog’s successor. Fedora ultimately chose to go with rsyslog, a decision which I feel is the right right one given the features available in rsyslog.

The rsyslog application offers a couple major selling points, including remote logging via TCP traffic in addition to UDP, optional database backend support (logging to mysql), regex support, RFC 3195 support, and encryption support. While we won’t be getting into all these goodies here, you can see the full list of supported features here.

• Installation

Update: Rsyslog is now in CentOS 5.2, though it is not the default application. To install it, simply ‘yum install rsyslog’.

Rsyslog is not available in the default CentOS repositories. You’ll have to get the rpm from the c.k.o repository. Repository directions are available here.

• Server Configuration

Okay, here’s the fun part for rsyslog: It’s a drop in replacement for the base syslog package. If you don’t want to configure it, simply start it up with service rsyslog start and you’re all set. To use it as a central logging facility though, we’ll have to define a few things.

While the stock config in /etc/rsyslog.conf looks very much like what you’re used to seeing, you can pretty much nuke the entire file, as we won’t be needing it for the log server.

To start, we want to declare some global directives. These directives will dictate which systems, ips or netblocks are allowed to log to this server.

# Global Directives

$AllowedSender UDP, 127.0.0.1, 192.168.1.0/24
$AllowedSender TCP, 127.0.0.1, 192.168.1.0/24
In the example above, we’re allowing the localhost, and the 192.168.1.0/24 block to write to this log server, over both TCP and UDP.

Next, we’re going to set up some basic templates. The goal here is to emulate the default log structure, with a directory for each machine. As a result, we’ll end up with /var/log/machine1/ and /var/log/machine2/ each with the usual messages, boot.log, secure, spooler, cron, etc.

Below are the template definitions you’ll want to do this:

# Template Section

# The authpriv file has restricted access.
$template DynAuth, “/var/log/%HOSTNAME%/secure.log”

# Log anything (except mail and cron) of level info or higher.
$template DynMSG, “/var/log/%HOSTNAME%/messages”

# Log all the mail messages in one place.
$template Dynmail, “/var/log/%HOSTNAME%/maillog”

# Log cron stuff
$template Dyncron,”/var/log/%HOSTNAME%/cron”

# Save news errors of level crit and higher in a special file.
$template Dynspool, “/var/log/%HOSTNAME%/spooler”

# Save boot messages also to boot.log
$template Dynboot, “/var/log/%HOSTNAME%/boot.log”
This sets up the templating section with everything we need. From here out, we can simply refer to the template we want to use when we actually need to log something.

Next we need to tell rsyslog to actually log everything we discussed in the templates above.

authpriv.* ?DynAuth
*.info,mail.none,authpriv.none,cron.none ?DynMSG
mail.* -?Dynmail
cron.* ?Dyncron
news.crit ?Dynspool
local7.* ?Dynboot


That’s it for the /etc/rsyslog.conf file. Now we need to tweak /etc/sysconfig/rsyslog to allow remote connections.

In /etc/sysconfig/rsyslog you will see a line for SYSLOGD_OPTIONS. If you want to enable UDP remote logging, simply add a -r within the line. (it will default to port 514). If you wish to use TCP logging instead or in addition to UDP, add a -t also. The resulting file should look like the one below.

# Options to syslogd
# -m 0 disables 'MARK' messages.
# -r enables logging from remote machines
# -x disables DNS lookups on messages recieved with -r
# See syslogd(8) for more details
SYSLOGD_OPTIONS="-r -t -m 0"
# Options to klogd
# -2 prints all kernel oops messages twice; once for klogd to decode, and
# once for processing with 'ksymoops'
# -x disables all klogd processing of oops messages entirely
# See klogd(8) for more details
KLOGD_OPTIONS="-x"
#
SYSLOG_UMASK=077
# set this to a umask value to use for all log files as in umask(1).
# By default, all permissions are removed for "group" and "other".


That’s it. Run ’service rsyslog (re)start’ as applicable, and make any necessary adjustments to your firewall to allow TCP/UDP traffic on port 514.

• Configuring the client

Okay, now that we have the rsyslog server running, we need to configure the clients to properly log to the server, rather than handling logging themselves. This is a two step process and pretty darn simple, so here goes.

Open the client /etc/sysconfig/rsyslog in your favorite text editor, and add a -h to the SYSLOGD_OPTIONS. The result should look like this:

SYSLOGD_OPTIONS="-h -m 0"


Then open up /etc/rsyslog.conf and change the contents so that you just have the appropriate line:


*.* @logserver-ip #for UDP logging
*.* @@logserver-ip # for TCP logging


These two lines tell the client to log everything to the server we defined earlier. One is via tcp, and one is via udp, so please pick the one you want for your environment. Once you’ve made these modifications, you’re all set. Simply (re)start rsyslog on the client, and check the server to ensure proper communication. Everything from here is dynamicly generated, so you can now add as many hosts as you like within the ip ranges we defined earlier for the server.

Next time I hope to be looking at rsyslog’s mysql backend for logging and statistics queries.