Jeff's Professional Side

Last night Bobby, Jason, and I returned to our old habit of attending a monthly meeting of the Atlanta UNIX Users Group. Returning speaker Ryan Matteson gave a rather informative presentation on the topic of Centralized Logging with syslog-ng. I’ve used syslog-ng on and off over the years, and we recommend the GPLv2 version to our services and support customers for use in conjunction with the syslog listener built into OpenNMS. Despite all this, I had never had a proper introduction to syslog-ng, so this meeting was one not to be missed.

Taking a cue from DJ’s recent post, I’m putting my notes from the talk online. Ryan has also put his slides online, so there’s not much marginal value in the following except for a couple of verbal comments that I rolled in.

AUUG 2009-10-05: Ryan Matteson on syslog-ng

* what is syslog-ng?
syslog-ng: a flexible, robust, open source syslog implementation
features:
	logging vai udp or tcp
	mutual auth via certificates
	tls encryption
	filters to sort traffic by host, facility, log level, message contents, etc.
	messages can be parsed and rewritten (especially useful for removing sensitive data)
	log to SQL database

* how does syslog-ng work?
configured through a single text file, broken into sections describing where to read messages from, how to process them, where to send them
sections broken down into:
	global options
	filter statements
	parser, rewrite statements
	traffic sources
	traffic destinations
	log statement

* syslog-ng global options
allow you to control the global behavior
options include:
	entries to resolve hosts via dns
	how many log entries to write(2) at a time - no mandatory synchronous writes on socket / fdsync() on files, saves on disk i/o
	permissions to assign to files
	buffer sizes
	whether or not to preserve names whentries are forwarded through another syslog process

* global options example
specified in an options block:
@version: 3.0
options{
	flush_lines(100);
	use_dns(no);
	owner(root);
	group(logs);
	perm(0640);
	dir_perm(0750);
	dir_owner(root);
	dir_group(logs);
	...
}

* traffic sources
uses traffic sources to define where syslog-ng should log messages from
types of sources:
	internal - internally generated
	file
	fifo (named pipe)
	program
	tcp/udp socket
	unix-dgram

* example sources
created by adding a source statement along with one or more configuration directives to a source block:
source local {
	file("/proc/kmsg" log_prefix("kernel: ");
	unix-stream("/dev/log");
	internal();
}

source network {
	udp(ip(0.0.0.0) port(514));
};

* log dests
specify where messages should be written / forwarded
types:
	file
	fifo
	program
	sql
	tcp / udp
	unix-dgram / unix-stream
	usertty
macros for flexible naming:
	$HOST
	$IP (source IP address)
	$MONTH, $DAY, $YEAR - date message created. other timestamps also available.
	syslog-ng 3.x man page has the complete list

* example destination
created by defining a destination {} with a log dest, and adding optional dest options:
destination d_unix_oom_msgs {
file("/log/unix/kernoom.$HOST.$YEAR.$MONTH.$DAY"
	owner(matty) group(matty) perm(0600)
	dir_owner(matty) dir_group(matty)
	dir_perm(0700);
};

* filters
allow you to route incoming msgs to dests based on criteria
matched using one or more filter funcs:
	facility
	level
	match - match against a string in message, headers
	message - matches a string against the message body
	host - match ip or hostname
	netmask
	additional funcs in man page
complex filters possible using POSIX, PCRE regexes plus boolean (or, and, not) operators

* example filter
lok for messages from 192.168.1.1, 1.2 that are part of kern facility and contain "Out of Memory":
filter f_kern_oom {
	((host(192.168.1.1") or
	host("192.168.1.2")) and
	facility(kern) and
	level(debug...emerg) and
	message("Out of Memory"));
};

* log statements
let you combine filters, sources, dests to control where messages are sent:
log { source(network);
	filter(f_kern_oom);
	destination(d_unix_oom_msgs);
	flags(final);
};

* monitoring syslog-ng usage
syslog-ng gathers stats for each log dest, and will write them out periodically -- interval controlled by the stats(time interval) directive -- to the system logs. look in system logs for "syslog-ng.* Log statistics"

* debugging syslog-ng issues
if a filter isn't working as expected, run syslog-ng with "-d" (debug) and "-e" (log to stderr) options to observe rule processing:
$ syslog-ng -e -d > /var/tmp/syslog.out 2>&1

* conclusion
offers a flexible and easy way to configure centralized logging solution
combined with tools like logwatch and swatch, ou will know exactly what is going on with your servers, and will have one place to look when things go wrong

* references
syslog-ng web site: http://www.balabit.com/network-security/syslog-ng
manual: http://www.balabit.com/dl/guides/syslog-ng-v3.0-guide-admin-en.pdf

discussion:
windows stuff; google: syslog-ng windows agent (looks to be a pay-for addon)

jeff Geeky atlanta, auug, Network Management, OpenNMS, opensource, syslog-ng