Taking a cue from DJ’s recent post, I’m putting my notes from the talk online. Ryan has also put his slides online, so there’s not much marginal value in the following except for a couple of verbal comments that I rolled in.

AUUG 2009-10-05: Ryan Matteson on syslog-ng

* what is syslog-ng?
syslog-ng: a flexible, robust, open source syslog implementation
features:
	logging vai udp or tcp
	mutual auth via certificates
	tls encryption
	filters to sort traffic by host, facility, log level, message contents, etc.
	messages can be parsed and rewritten (especially useful for removing sensitive data)
	log to SQL database

* how does syslog-ng work?
configured through a single text file, broken into sections describing where to read messages from, how to process them, where to send them
sections broken down into:
	global options
	filter statements
	parser, rewrite statements
	traffic sources
	traffic destinations
	log statement

* syslog-ng global options
allow you to control the global behavior
options include:
	entries to resolve hosts via dns
	how many log entries to write(2) at a time - no mandatory synchronous writes on socket / fdsync() on files, saves on disk i/o
	permissions to assign to files
	buffer sizes
	whether or not to preserve names whentries are forwarded through another syslog process

* global options example
specified in an options block:
@version: 3.0
options{
	flush_lines(100);
	use_dns(no);
	owner(root);
	group(logs);
	perm(0640);
	dir_perm(0750);
	dir_owner(root);
	dir_group(logs);
	...
}

* traffic sources
uses traffic sources to define where syslog-ng should log messages from
types of sources:
	internal - internally generated
	file
	fifo (named pipe)
	program
	tcp/udp socket
	unix-dgram

* example sources
created by adding a source statement along with one or more configuration directives to a source block:
source local {
	file("/proc/kmsg" log_prefix("kernel: ");
	unix-stream("/dev/log");
	internal();
}

source network {
	udp(ip(0.0.0.0) port(514));
};

* log dests
specify where messages should be written / forwarded
types:
	file
	fifo
	program
	sql
	tcp / udp
	unix-dgram / unix-stream
	usertty
macros for flexible naming:
	$HOST
	$IP (source IP address)
	$MONTH, $DAY, $YEAR - date message created. other timestamps also available.
	syslog-ng 3.x man page has the complete list

* example destination
created by defining a destination {} with a log dest, and adding optional dest options:
destination d_unix_oom_msgs {
file("/log/unix/kernoom.$HOST.$YEAR.$MONTH.$DAY"
	owner(matty) group(matty) perm(0600)
	dir_owner(matty) dir_group(matty)
	dir_perm(0700);
};

* filters
allow you to route incoming msgs to dests based on criteria
matched using one or more filter funcs:
	facility
	level
	match - match against a string in message, headers
	message - matches a string against the message body
	host - match ip or hostname
	netmask
	additional funcs in man page
complex filters possible using POSIX, PCRE regexes plus boolean (or, and, not) operators

* example filter
lok for messages from 192.168.1.1, 1.2 that are part of kern facility and contain "Out of Memory":
filter f_kern_oom {
	((host(192.168.1.1") or
	host("192.168.1.2")) and
	facility(kern) and
	level(debug...emerg) and
	message("Out of Memory"));
};

* log statements
let you combine filters, sources, dests to control where messages are sent:
log { source(network);
	filter(f_kern_oom);
	destination(d_unix_oom_msgs);
	flags(final);
};

* monitoring syslog-ng usage
syslog-ng gathers stats for each log dest, and will write them out periodically -- interval controlled by the stats(time interval) directive -- to the system logs. look in system logs for "syslog-ng.* Log statistics"

* debugging syslog-ng issues
if a filter isn't working as expected, run syslog-ng with "-d" (debug) and "-e" (log to stderr) options to observe rule processing:
$ syslog-ng -e -d > /var/tmp/syslog.out 2>&1

* conclusion
offers a flexible and easy way to configure centralized logging solution
combined with tools like logwatch and swatch, ou will know exactly what is going on with your servers, and will have one place to look when things go wrong

* references
syslog-ng web site: http://www.balabit.com/network-security/syslog-ng
manual: http://www.balabit.com/dl/guides/syslog-ng-v3.0-guide-admin-en.pdf

discussion:
windows stuff; google: syslog-ng windows agent (looks to be a pay-for addon)

At Nick’s request, I had convinced Tarus to give a talk, and I pretty much unilaterally decided that OpenNMS would also be an exhibitor. We scored an absolutely prime spot in the exhibit area and drew in lots of traffic. Our good friends Mita and Robbie at Presentation Rentals (which sponsors ANSMTUG monthly) got us a great deal on a big monitor on which we ran our loop. My amazing wife volunteered to help out with the booth, her first time ever “wearing the shirt” though she’d hung out with us last October at a show in London. We always like to have as many community members representing as we have OpenNMS Group employees, so I recruited local friends Bobby and Robert to help man the booth. I’m confident that each of them fully earned a 100% commission on the free software he sold that day.

A plethora of Papa John's pizza pies, constituting lunch at Atlanta Linux Fest 2009

This show was the project’s first in Atlanta, and as happens in every new city, people came out of the woodwork who use OpenNMS to keep the IT infrastructure running smoothly at local companies that we previously had no idea were part of our user community. It turns out that Georgia Tech, which hosted last year’s Dev-Jam, is a user, as is Georgia Power. Less widely known users that we learned about include the company that provides the data underlying Google Maps‘ and other providers’ integrated roadway traffic services. I hope that some of these people will join the Order of the Blue Polo, especially since both my wife and Bobby were modeling the shirt in question. It’s fitting, and a testament to the show organizers’ excellent taste, that lunch came from a company represented in the OBP.

It was such a busy day for us that nobody had time to snap any pictures of our booth! We were fortunate enough to have, in the booth adjacent to ours, shutterbug Jane Ullah, who captured three of us loafing during a lull in the show traffic. If some images emerge that paint us in a less idle light, I’ll update this post to include them

That is, until last weekend, when I visited family for a wedding. (Insert hillbilly joke here.) My uncle, who is a part-time competitive marksman (and father of the bride), took dad and me to the pistol range for a few hours of turning neatly cast bits of lead and brass into irregularly flat ones.

My guest pass to Bluegrass Sportsmens League

This trip was my first time doing practical shooting, in which time counts as much as accuracy. It turns out that from the holster at thirty yards I can reliably hit four rectangular steel plates, each about the size of an average Dell notebook computer, and then a larger fifth plate, in just over six seconds. That’s about a second slower than my dad and about three seconds slower than my uncle, but not a bad time for someone who hasn’t handled a loaded weapon in about 536 million seconds. More amazing is that I can do this with no damage to that flap of skin between the thumb and forefinger of the dominant hand, a part of the less experienced shooter that Glock actions seem to take particular delight in biting.

I’m now going to have to join my wife in applying for a Georgia firearms license (Georgia, my home these days, is one of the few United States that requires a license to carry openly) and see if I can get that time under six seconds before my next visit to BGSL.

If you use and love OpenNMS, you can help by nominating the project yourself — just click either of the logos below, then select “Best Project for the Enterprise” from the drop-down list. Thanks in advance!

If you’d like to pass along the word, here’s some HTML that you can paste into your own blog or other web page:


<a href="http://sourceforge.net/community/cca09/nominate/?project_name=OpenNMS&project_url=http://sourceforge.net/projects/opennms/">
<img src="http://sourceforge.net/images/cca/cca_nominate.png" border="0"/></a>



<protocol-plugin protocol="Twitter" class-name="org.opennms.netmgt.capsd.plugins.LoopPlugin" scan="off" />


The real magic lives in the poller-configuration.xml file. Define the Twitter service in the appropriate package (beware the default-empty “strafer” package):

<service name="Twitter" interval="600000" user-defined="false" status="on">
<parameter key="retry" value="1" />
<parameter key="timeout" value="5000" />
<parameter key="page-sequence">
<page-sequence>
<page path="/twitter" port="80" successMatch="(?s).*<title>Twitter \(twitter\) on Twitter.*"
virtual-host="twitter.com"
user-agent="OpenNMS/1.6 (JVM; I; Most Any Platform; en-us)" />
</page-sequence>
</parameter>
</service>


Finally, don’t forget to tell the poller which monitor class to use for the Twitter service:


<monitor service="Twitter" class-name="org.opennms.netmgt.poller.monitors.PageSequenceMonitor" />


I used the PageSequenceMonitor rather than the more basic HttpMonitor because I plan on coming back later and making the page sequence include logging in to a Twitter account, verifying that some more sophisticated tests pass, and logging out again.

Seems no one at digium is capable of implementing it so why not give some 18 year old kid the chance.

Russell identified the project suggestion as not necessarily fitting the spirit of GSoC, and the comment as completely unnecessary. The troll, of course, didn’t take the hint.

First, let me defend those insulted here. I know personally and/or have worked with professionally probably a dozen or more Digium employees, and every one of them is a first-class developer, engineer, product manager, or people manager. I have also been a GSoC mentor, and can say without reservation that as a rule, any student who has made the cut to participate in the Summer of Code is not “some 18 year old kid” but an adult (in more than just the legal sense) worthy of the same respect as any other colleague.

Second, for those who don’t instinctively understand why a third-party license server, even an open-source one, is outside the spirit of GSoC, let me refer to the 2009 GSoC FAQ on the first of several goals the program aims to accomplish:

Get more open source code created and released for the benefit of all;

As discussed above, such a project could arguably have an indirect benefit to the Asterisk project by easing one unpleasant development task and therefore making Asterisk more attractive to a wider range of developers. The problem is that the only developers who are currently saddled with that unpleasantness are, by definition, ones who are creating non-open-source products. I’ll stop short of saying that any commercial licensing libraries exist that are “perfectly good”, but there are ones out there that get the job done if you’re willing to pay to include them in your products. Apparently this troll thinks that Digium owes him a free alternative to those products, and that if Digium won’t spend its own money to write one for him, perhaps they could dupe Google into paying a $5,000 USD stipend to some hapless code-monkey kid who’ll write it for them.

That’s not what GSoC is about, and it’s not what open source software generally is about, and it’s not what Digium is about. Here’s the kind of company Digium is.

Around ten years ago, a friend of mine was running a business called Linux Support Services. Realizing that he needed a phone system, he looked around and saw that the options on the market that could meet his needs were impossibly expensive for the budget of his company, which he had bootstrapped in his parents’ garage before moving its offices to a bigger city. Since he was a telephony geek and a Linux geek and a brilliant guy who didn’t know to be afraid of the task, he sat down and started writing a software PBX that ran on Linux. When it got to the point of being usable, he decided to release it under an open-source license for anybody to download and use for free. Over the course of a couple years, the PBX software grew in maturity and popularity to the point that the company was doing more business around the PBX software than in the area of Linux support. The company found further success building and selling hardware for interfacing the PBX software with the traditional telephone network, and since the company had either sole or joint copyright on all the PBX code it was also able to sell a thoroughly quality-assured, supported version of the PBX software under a commercial license to customers with a need for such a thing. Never has the commercial product had any features that were missing from the open-source product, and the direction the company has chosen for the project has always been heavily informed by the wishes of the open-source development community outside the company. Since the company had moved fully into the digital telephony space, it changed its name. To Digium.

One of the things I’ve always admired about Digium as a company is that it’s very good, on balance, about recognizing that the community of open-source developers outside the company is an essential part of what makes Asterisk great. It’s an attribute that we try our best to share as we make money in support of our stewardship of the OpenNMS project. Digium also recognizes that the ecosystem of third-party commercial products and services that has grown up around Asterisk must be nourished if the platform is to remain innovative and relevant. The community and the ecosystem usually coexist peacefully even if they don’t entirely agree on all points, or entirely understand each other at all times. This morning’s incident, I think, serves as an extreme example of what can happen when somebody from the ecosystem with an inflated sense of his own entitlement metaphorically drives a truck through the front window of the room where the community is busy working. It’s a testament to the mettle of the Asterisk community that the troll got only a single, cool-headed reply to his overture, and then nothing but silence when he tried to stir things up again. The thread about GSoC, meanwhile, has yielded lots of good discussion.

When you first visit the site, it asks you to print out a Solar Panel Marker. Like, with your printer. Now that I think about it, there’s a small irony in asking me to do that as part of a “green” campaign, but I digress. Print yours on the back side of a piece of scrap paper if you want to indulge your conscience.

So I’ve got this piece of dead tree in my hand, and from the Smart Grid Augmented Reality page I launch the windmill demo. Hold up the paper to my webcam, which just looks like this:

Holding up the Solar Panel Marker - BORING

Once I grant the Flash gizmo access to my camera and speakers, though, it draws on top of the webcam input an animated, quasi-holographic plot of a scene with three wind turbines and that scarecrow that I still don’t understand because I didn’t watch any of the Superb Owl, not even the commercials. Move the piece of dead tree around, and the camera angle changes. It’s really the kind of thing you just have to try for yourself — as Taylor commented, ::mind is blown::. Here’s the windmill demo:

Wind Turbine demo - Spiffy

And here’s the solar panel one:

Solar Panel demo - Also spiffy

Without presuming to speak for Tarus (Ben and I both work with Tarus and so are quite familiar with his philosophies) I’d like to lay out a few places where I think Luke’s post missed the mark. These thoughts are ones that I really wanted to get across in person during our brief conversation in January after Luke’s talk at CloudCampATL, but I didn’t want to hog his time.

My points follow.

1. Open-core is not invalid as a business model, it’s just not the same as open-source

Tarus’ campaign is not against the open-core business model itself. It’s against open-core companies that want to have it both ways by playing the necessary-feature-withholding game but still calling themselves “open source companies”. It’s not about hate but about calling something what it is.

2. Joint ownership of copyright and why we as a project sponsor pursued it

As the maintainers of OpenNMS, we do require all contributors to execute an agreement that concerns copyright. It’s not about attribution or assignment, though, as assignment in particular is essentially impossible under current U.S. copyright law. It’s actually about declaring a shared or joint copyright. A contributor who executes the OpenNMS Contributor Agreement (which Tarus blogged about here) forfeits exactly zero rights while simultaneously granting us the same set of rights that the contributor has. The contributor is still free to do anything he wishes with his contribution — license it commercially, give it away under a non-GPL-compatible license, print it out and roll a joint with it, or (gasp) also contribute it to Hyperic. That point was key in our selecting this agreement, which as Ben pointed out earlier is literally the Sun Contributor Agreement with e-mail and fax information added and “Sun Microsystems”, “OpenSolaris”, and “California” changed to “The OpenNMS Group”, “OpenNMS”, and “North Carolina” respectively. We were incredibly hesitant to add this kind of requirement, and we didn’t do so unilaterally. When we went shopping for an agreement, we created a committee of non-employees from the Order of the Green Polo (the secret society of core OpenNMS contributors) to do the research and waited months for them to make a recommendation — they all have day jobs, after all. We knew that we ran the risk of pissing off and maybe even running off valuable community members, something that thankfully did not happen. Even with OCAs on file for all our contributors, we still don’t hold all the copyright since we inherited the code from the original creator, Oculan (formerly PlatformWorks, then Atipa, now part of Raritan through subsequent acquisitions). Raritan doesn’t want to sell us the intellectual property for one dollar, so we couldn’t do anything evil with the code today even if we had the desire to do so.

At the end of the day, our motivation for adding the OCA was about protecting the community, not about enriching ourselves. Having all the rights conferred under U.S. copyright law is the only way that we as a company can take legal action to protect the community from having its work (allegedly!) ripped off by people whose primary interest actually is self-enrichment. it’s incredibly tricky to pursue a GPL-violation lawsuit when you don’t hold copyright on your entire code base. Having an OCA for each contributor means that if the project ever has to appear in court, we don’t have to go broke flying everybody who’s ever contributed code to the venue. One other thing that a folder full of OCAs allows us to consider doing is offering patent indemnification to our support customers.

3. Making money selling software, and pragmatism

As far as development costing money, The OpenNMS Group is in exactly the same boat as Reductive Labs. We’re not running a charity either, and we’ve all got heads full of ideas that we just don’t have the resources to translate into code. It takes real discipline when you love the project as much as I do to put off work on an amazingly cool feature until we find somebody willing to fund it, and go work on weatherizing the damn house instead (which I should be doing right now). Would we ever sell some software? We’ve talked about it, but we would never sell OpenNMS to end users. If we held all the copyright would we maybe license the code commercially to, hypothetically, HP as the foundation of a completely redone OpenView Network Node Manager? Only if the community were on board, and the version HP got would be exactly the same as the GPL version, and we’d plow almost every penny of profit back into the project the same as we do today. This all in keeping with our promise that “OpenNMS will never suck, and it will always be free.”

4. Supported binaries and what we’re willing to support

Luke refers to a model of producing supported binaries — which we do, but we also are just as happy to support a customer who builds everything from source. There’s no shrink-wrapped version of OpenNMS is what I’m getting at here, and if there ever is, it won’t affect our willingness to support people who roll their own. We even support customers who make substantial changes to the code provided they purchase a Developer Support agreement in addition to Basic or Enterprise support. Digium is going through a transition to a similar model right now — they’re about to start offering support contracts for GPL Asterisk where they previously offered support only as part of the cost of the shrink-wrapped Asterisk Business Edition. While I wish we had something like Digium’s hardware revenue stream to draw from, I do think that their move is something of a validation of our model.

5. The greater sin is (C), both of the above

Last item, and this one’s actually way more than just the semantic distinction that I thought it would be when I wrote this paragraph the first time. Luke asks, “Is it a greater sin to only accept patches to your product if the contributor is willing to assign copyright to your commercial company, or to produce some closed-source code?” I would like to point out that the open-core guys, in our sector at least, are doing both. See Zenoss‘ and Hyperic’s terms. In so doing, these guys are accepting code from the community and reserving the right to roll it into their closed-source code.

John Willis on What is a Cloud?

Sudesh Oudi on EC2 and Windows

Unknown Speaker on Joyent User Experience 1/3

Unknown Speaker on Joyent User Experience 2/3

Unknown Speaker on Joyent User Experience 3/3

Luke Kanies on Puppet

Dave Wright on Jungle Disk

Russell Jurney on Scalr