Archive for the ‘Innovation and open source releases’ Category

A Prometheus exporter for Dovecot

Monday, January 16th, 2017

To start this year’s series of contribution to the open source community, we’re proud to announce the release of yet another tool that we use to monitor our production setup, namely a Prometheus metrics exporter for the Dovecot POP/IMAP mail server.

One of the key features of Prometheus is that it is very well suited for white box monitoring, i.e. having graphs and alerts based on internal state of the program, as opposed to testing just the externally visible behaviour of the system. For Dovecot we’re very interested in using white box monitoring to be able to graph traffic and resource usage per customer, domain and user.

It turns out that we’re in luck, as Dovecot 2.1 and later ship with a statistics module that provides access to this kind information. When enabled, Dovecot binds to an additional UNIX or TCP socket on which metrics are exported. The Dovecot exporter that we’ve published on GitHub is basically a light-weight proxy that converts the metrics from Dovecot’s format into Prometheus metrics, exporting them over HTTP.

Below is a screenshot of our Dovecot exporter in action. The graph shows the rate of IMAP commands sent to our mail server, broken down by IMAP username.

If your email setup is also based on Dovecot and use Prometheus for monitoring, we’d like to invite you to give this exporter a try as well. Feel free to file issues or send pull requests on GitHub.

KumiNews 2016: The latest and the future at a glance

Monday, January 2nd, 2017

KumiNews: The latest and the future at a glance

Kumina can look back on a successful year. A lot happened at Kumina these past 12 months. We carried out challenging new projects and started new collaborations with amazing organisations. Behind the scenes, we worked hard on optimising and extending our services. We also welcomed new colleagues at our team.

Tim’s vision on the recent and future progression and expansion of our services:

In 2017, Kumina will be celebrating its 10th anniversary. A lot has changed since Bart, Kees and I decided to merge our companies in 2007. And we expect a lot more changes in the coming years as well, as the industry is evolving faster and faster.

Where in 2007 Kumina’s main business was providing “old-style” system administration, usually on hardware provided by the customers, we’ve since moved to our own hosting offering for most of our clients. Even when clients do not make use of the Virtual Private Servers we offer, a lot of them rent hardware with us directly, using either of the two datacenters Kumina is using herself. A lot of our business still revolves around maintaining the Virtual Private Servers and associated services.

Although we noticed the rise in people asking for ‘cloud’, we always refrained from calling our offering ‘cloud’, as it is not as flexible as the public cloud environments provided by the large players in the field.

This was never much of a problem, as most customers did not require this type of flexibility. Having a reliable platform is more important and in those starting years, everyone looked a bit askance at those big clouds. Most of our customers preferred a solid partner that they could actually communicate with.

The public cloud is picking up speed with our customers as well. The ease with which one can set up an environment on a public cloud and break it down again offers a whole lot of options to our customers. Scaling is becoming an issue more often, especially when a web application has to deal with a sudden peak in concurrent users. The public cloud works great for that. Where those public cloud companies lack in personal contact, it makes up for it with the range of possibilities the platforms offer. The pay-as-you-go constructions help a lot here as well.

At Kumina, we notice that a lot of customers are expressing more and more interest in those possibilities. Although we’ve been offering our services on AWS as well for quite a while (as we are still a maintenance company, not a hosting company), until recently most of our customer’s systems ended up on either their own hardware or our hosting cluster. This is changing, however, as our customers are interested in the scaling options and quick replication that a cloud environment provides. And we’re happy to provide.

We wouldn’t be Kumina if we didn’t have an opinion about the best way of doing that. Since 2016, we’ve been working a lot with containers and Kubernetes and we’re convinced that currently, there’s no better way of working with applications than running them containerized within a Kubernetes cluster. This provides a lot of additional possibilities to our customers, which in turn allows them to do more in less time. We love to support them in this.

So for the immediate future, we expect an uptick in the number of Kubernetes setups we administer, which includes more than “just” maintaining Kubernetes of course. We add monitoring and metrics collection via Prometheus, ElasticSearch for log aggregation and lots more. We even provide a full development stack, from a Gitlab instance to an automatically deploying Jenkins.

2017 is promising to become a very interesting year regarding all the new possibilities. Are you wondering if we can be of meaning to your organisation? We always offer a free consult and advice by phone, so don’t hesitate to contact us!

Meet our new colleagues

Last year, we welcomed two new colleagues to our team: Ed Schouten and Bart Vercoulen.

In 2011 during his studies, Ed worked as part-time employee at Kumina. Five years later, after working at Google, we welcomed him back to our team. In his combined function as system administration and software developer, he mainly focusses on process optimisation. He is currently working on replacing our monitoring system. With this new feature-rich system, we can get more insights and also share these with our customers. This enables us to solve potential alerts faster and gives us the ability to get the most performance out of our systems. With his experience with large systems, algorithms, developer techniques and extended knowledge on the tools we use, like Kubernetes, Prometheus and Cassandra, he is great addition to our team.

We also welcomed our new colleague Bart, who started as our part-time junior developer. He supports the team and our internal processes by developing tools that we need internally. Thanks to his commitment we were able to take great strides towards augmenting the coverage of our monitoring and trending the past year. With his Icinga checks and Prometheus collectors we are now able to detect potential problems even faster. In the near future he will work on making our customers set-ups more comprehensible. We hope we are able to offer this new service to all our customers in the year to come.

In 2017 Niek Geerts will start as our new system administrator, who will be introduced at some point in the future. We continue assuring and improving the quality of our services. For example, we just started the process to obtain ISO 27001 certification. We will keep you informed!

Optimisation and Innovation: Open Source Releases and sponsoring
Since the foundation of Kumina we have worked almost exclusively with Open Source software. Past year, we gladly contributed to the open source community by releasing several open source software improvements and initiatives. We also decided to sponsor a promising new open source project by the company Nuxi.

Open Source Releases
Once in a while we face a challenge, without there being a solution that lives up to our quality standards. This is also the case with our current project to reimplement our monitoring to be based on Prometheus. In some cases we want to be able monitor applications that cannot yet interface with Prometheus, which is why we’ve designed these components ourselves. Curious about these and other open source releases from last year? Have a look at our Business Github page or click around on this blog.

Cooperation and sponsoring Nuxi / CloudABI
In 2014, our colleague Ed started an open source project named CloudABI. CloudABI is a framework which allows software developers to build applications that are strongly sandboxed. Sandboxing massively reduces the impact of security problems. With the use of CloudABI it is also possible to test and manage software in a better way. Kumina decided to help Ed with this promising project by sponsoring him.

Open source release ‘promacct’: Network traffic accounting using Prometheus

Monday, November 14th, 2016

Network traffic accounting with pmacct
At Kumina we’ve been a long-time user of pmacct. pmacct is an easy to use daemon for UNIX-based systems to perform network traffic accounting. Under the hood, pmacct makes use of libpcap to capture network traffic from the operating system. In our current deployment we’ve configured pmacct to write its results into a MySQL database. At the end of every month we run various queries on this database, ranging from simple summation per host to computing the 95th percentile. These results may then be used for billing purposes.

Pmacct and MySQL database
Given that the number of servers, IP addresses and the amount of traffic at Kumina has increased steadily over the last couple of years, we’re at this point running into the problem that pmacct in combination with a MySQL database simply no longer scales. Not only are our search queries taking a long time to complete, even insertions of new data are becoming problematic. A SQL database server is not the right tool for storing and processing time series.

Improving monitoring and trending
Over the last couple of months we’ve been working on replacing and improving our existing monitoring and trending setup with Prometheus. So far our experiences using it have been very positive, which is why we’ve decided that we also want to use it as the basis for a new traffic accounting setup. Being able to create recording rules that use functions like quantile_over_time() is exactly what we need, as it allows us to compute traffic percentiles not just at the end of the month, but in real-time.

Alternative for pmacct: promacct
After searching online, we haven’t been able to find a Prometheus metrics exporter that could act as a drop-in replacement for pmacct, which is why we’ve decided to develop it ourselves, called promacct. Where proamcct differs from pmacct is that instead of periodically storing results to a database, it provides access to its metrics over HTTP, allowing Prometheus to scrape it directly.
Due to promacct supporting aggregation by source/destination IP addresses, we can now easily create traffic graphs for individual hosts:


Per-datacenter traffic quantiles are computed through recording rules, so that they can be inspected at real-time:

Today we’re glad to announce that we’re releasing promacct as Open Source Software. Its source code can be found on our company’s GitHub page. Be sure to give it a try and let us know whether it works for you.

Enjoy and feel free to share!

Birdwatcher: Accessing Calico/BIRD metrics through Prometheus

Friday, October 28th, 2016

At Kumina we maintain a Kubernetes setup running on Amazon EC2. For the low-level networking between containers, we make use of Calico. Calico configures all of our EC2 systems to form a mesh network. The systems in this mesh network all run an instance of the BIRD Internet Routing Daemon.

One of the problems we ran into with Calico is that it’s sometimes hard to get a holistic view of the state of the system. Calico ships with a utility called calicoctl that can be used to print the state of a single node in the mesh, but using this utility can easily become laborious as the number of EC2 instances increases.

Given that we already make strong use of Prometheus for our monitoring, we’ve solved this by writing a tool called Birdwatcher that exports the metrics generated by BIRD in Prometheus’ format. This allows us to put alerts in place for when an excessive number of changes to routes occur, or when routes simply fail to work for a prolonged period of time.

Today we’re happy to announce that Birdwatcher is now available on our company’s GitHub page. If you’re a user of both Calico and Prometheus, be sure to give it a try. Enjoy!



Kumina sponsoring CloudABI: practical sandboxing for UNIX

Friday, October 14th, 2016

Ed Schouten: “Almost exactly two years ago I started working on a project called CloudABI. In a nutshell, CloudABI is a UNIX-like programming environment for Linux and the BSDs that allows you to easily design sandboxed applications. It accomplishes this by making strong use of capability-based security, inspired by the University of Cambridge’s Capsicum. Compared to traditional UNIX applications, CloudABI applications are better resistent against security vulnerabilities, easier to test and easier to maintain. CloudABI is available as Open Source Software, free of charge. Feel free to watch my talk at 32C3 if you’re interested in all of the nitty-gritty details.

Some time ago I decided to visit the folks at Kumina, as I used to work there until early 2012. That’s why you’ll see my name next to some of the older posts on this blog. During my visit, Tim made me an offer I simply couldn’t refuse: a job at Kumina that allows me to spend a significant amount of time every week to continue the development of CloudABI. As you can see, I’ve accepted the offer. As of last month, I’m a member of the team once again!

What brings me joy is that this step makes the development of CloudABI sustainable. Over the last couple of weeks I’ve already managed to implement at least one large new feature: support for 32-bit hardware architectures. The CloudABI Development Blog now has an article describing the work that was needed to realise this.

At Kumina my job consists of a mixture between systems administration and software development. There are various pieces of software that we’re developing in-house. One of my tasks is to release some of these as Open Source Software, so stay tuned for my next posts!”