Spread ‘em!
Lately I’ve been playing around with the Spread Toolkit, and I gotta say, its cool and fun.
I got started on this because at my day job, we use spread to collect the error log from our web cluster onto the machine we regularly log into to do our dev work. This gives you a bit of a firehose of errors, but, you don’t suffer from missing any info. With this basic idea in mind, I played around and built some fun little adaptations that should help smooth out the logging infrastructure at GeekISP.
First, I built an OpenBSD port of Spread - this was pretty easy since I was able to start with the FreeBSD port. Next, since I knew I was going to be working with Apache logs, I built a port of spreadlogd, a spread-to-logfile program designed to work with mod_log_spread (which I’m not using). So, I managed to get things up where I had a similar setup as at work, error logs collect on the machine where the users are.
Next I started looking over at Tomcat, and wanting to do the same thing. I didn’t want to mess with Tomcat’s config file and deal with log4j appenders and all that stuff, so I started in with the Perl bindings that ship with Spread. I also discovered that the File::Tail module has a select() interface. Hmmm… so I started hacking away, and about 60 LOC later I had multitail-spread.pl - a “multitail meets spread” program, not very different from my basic log sender for apache. But, it’s config-file driven, so I get to run a single instance and monitor as many files as I want. Well, ok, so now that takes care of all the Tomcat files, the PHP-5 error log (it runs via FastCGI), and the lighttpd error log in one shot. Woo!
That was yesterday. Today I was thinking about the 30-odd customer error log files that are generated from the webserver. So I added support for log directories to multitail-spread, and a small new feature - for lack of a better name I’ll call it “prefixing”. All log files that live under a “log directory” get sent to the same spread group, but if this group is a “prefix group”, then I simply tack on the basename of the file it came from before sending the message.
So for example (roughly in yaml):
– conf file
log_dir:
/var/www/logs/customers: prefix
When File::Tail detects a new line in /var/www/logs/customers/foo-error.log, say that line is “Hello there!”, it goes over the wire as:
[foo-error.log] [$hostname] Hello there!
Now, and here’s the fun part, I wrote a simple version of spreadlogd in Perl (again, maybe 75 LOC) to listen to the “prefix” group, and store the messages in the files specified in the message. I even got fancy use Perl’s Cache modules (specifically the Cache::Memory module) to ensure that I stayed below my open files limit via the LRU cache removal policy. In the end, this gives me the ability to write to an open-ended set of files with no real configuration required.
Spread’s documentation is so-so. Things would have been easier if it were better documented, but all in all, fun stuff!