Posts Tagged ‘icinga’

Automated monitoring? Easy!

Tuesday, February 11th, 2014

One of the things we take very serious here at Kumina is our monitoring. We’ve always done so, but even we must admit that during the starting years, we sometimes forgot to include all possible checks for a new service or host. And it sucks when you forget to setup the monitoring for a specific item, because you generally only find out about it when it’s actually down already…

We like to check as much as possible (if not everything). For example, we check if a service is up and running, we check if a vhost is returning the expected response, if an SSL certificate is still valid or if it will expire within 30 days, we check if OpenVPN certificates are close to expiration and if all loaded Apache modules actually come from a Debian package. And we check often, generally every 30 seconds, but we would prefer to do it even more often. However, these are not things you want to configure manually over and over again.

Automate everything

We’re using Icinga in two datacenters in failover mode, the second node takes over if the primary is unreachable. We currently monitor 319 hosts (including some failover virtual hosts) and a grand total of nearly 10000 checks. Although this fluctuates daily, since most changes on a server also adds or removes checks. It is all done automatically. This prevents us from forgetting to setup monitoring for a specific item or host and also allows us to quickly deploy new checks on the entire infrastructure. Consider the Fokirtor check we created last year, it’s very easy for us to simply deploy it on all those machines.

Using the tools at hand

We’re currently pretty heavy Puppet users, so we leverage the infrastructure we already have in place for that.

Since a puppet agent runs on our monitoring hosts every few hours, it’ll deploy new configuration a few times per day. It’s not exactly continuous delivery, but close enough for our needs for now. Equally important, it removes checks we no longer need. For instance, if we’ve create a redirect that was changed into a full-fledged site, the check is automatically changed to no longer expect a 301 response but a 200 with a correct string (that we provided, of course, it’s not that automated).

We started out using the power of puppet’s exported resources but over time as our config grew, it started to take way too long for Puppet to deploy new configuration on the monitoring hosts. We now deploy the configuration for both Icinga instances using a script that reads the stored config from the Puppet database.

Other uses

As you might imagine, we also do this for trending with Munin. We automatically deploy the Munin plugins on the clients when we deploy a new service and we automatically deploy the host configuration on the Munin server. As well as required firewall rules on the client side.

Icinga check for Linux.Fokirtor

Friday, November 15th, 2013

We were notified this morning of the specifics of the attack that struck Hetzner at the start of this year. Or rather, the backdoor software that was used to provide access to the machines. It does not detail what vulnerability was exploited to actually install the Trojan. But it’s still a good idea to make sure your current processes are not infected.

So we went ahead and created a check that can detect Linux.Fokirtor, based on the information provided by Hetzner and Symantec.

Check and detect Linux/CDorked.A infections

Wednesday, May 8th, 2013

We’ve been reading a lot about a Linux exploit targeting webservers and since we manage quite a lot of webservers, we’re keeping a close eye on it. We recently already deployed a check for rogue Apache modules (since we mainly use Apache), but now we’ve also created a check from the code provided by ESET on their security blog describing the Linux/CDorked.A exploit. All it does is check shared memory for a segment of a specific size, but it’s still better than nothing.

As usual, the Icinga check can be found in our GitHub repository and if you’re on Debian, you can find the nagios-plugins-kumina package in our repository. This check needs to be run on the local machine, so you need to setup nrpe or ssh access from Icinga for that.

Let us know if this helps you or if we should improve on it! All kudos to ESET, since they provided the actual script (and research!) for this check.

Checking for rogue Apache modules

Wednesday, April 3rd, 2013

We’ve read a lot recently about attacks in which an attacker loads a modified module into Apache to insert iframes in outgoing data. Pretty scary, especially since nobody really seems to know how the hacks are performed. Recently, Sucuri wrote a blog article about how to check for rogue Apache modules on Debian. We’ve decided to implement this into an Icinga/Nagios check.

You can find the source for the plugin here. We also publish all our plugins via the ‘nagios-plugins-kumina’ package, provided by our apt repository.

Hope this helps!

Update: I packaged and pushed the wrong version of the script… Silly me. Fixed now!