Posts Tagged ‘github’

A Prometheus exporter for Dovecot

Monday, January 16th, 2017

To start this year’s series of contribution to the open source community, we’re proud to announce the release of yet another tool that we use to monitor our production setup, namely a Prometheus metrics exporter for the Dovecot POP/IMAP mail server.

One of the key features of Prometheus is that it is very well suited for white box monitoring, i.e. having graphs and alerts based on internal state of the program, as opposed to testing just the externally visible behaviour of the system. For Dovecot we’re very interested in using white box monitoring to be able to graph traffic and resource usage per customer, domain and user.

It turns out that we’re in luck, as Dovecot 2.1 and later ship with a statistics module that provides access to this kind information. When enabled, Dovecot binds to an additional UNIX or TCP socket on which metrics are exported. The Dovecot exporter that we’ve published on GitHub is basically a light-weight proxy that converts the metrics from Dovecot’s format into Prometheus metrics, exporting them over HTTP.

Below is a screenshot of our Dovecot exporter in action. The graph shows the rate of IMAP commands sent to our mail server, broken down by IMAP username.

If your email setup is also based on Dovecot and use Prometheus for monitoring, we’d like to invite you to give this exporter a try as well. Feel free to file issues or send pull requests on GitHub.

KumiNews 2016: The latest and the future at a glance

Monday, January 2nd, 2017

KumiNews: The latest and the future at a glance

Kumina can look back on a successful year. A lot happened at Kumina these past 12 months. We carried out challenging new projects and started new collaborations with amazing organisations. Behind the scenes, we worked hard on optimising and extending our services. We also welcomed new colleagues at our team.

Tim’s vision on the recent and future progression and expansion of our services:

In 2017, Kumina will be celebrating its 10th anniversary. A lot has changed since Bart, Kees and I decided to merge our companies in 2007. And we expect a lot more changes in the coming years as well, as the industry is evolving faster and faster.

Where in 2007 Kumina’s main business was providing “old-style” system administration, usually on hardware provided by the customers, we’ve since moved to our own hosting offering for most of our clients. Even when clients do not make use of the Virtual Private Servers we offer, a lot of them rent hardware with us directly, using either of the two datacenters Kumina is using herself. A lot of our business still revolves around maintaining the Virtual Private Servers and associated services.

Although we noticed the rise in people asking for ‘cloud’, we always refrained from calling our offering ‘cloud’, as it is not as flexible as the public cloud environments provided by the large players in the field.

This was never much of a problem, as most customers did not require this type of flexibility. Having a reliable platform is more important and in those starting years, everyone looked a bit askance at those big clouds. Most of our customers preferred a solid partner that they could actually communicate with.

The public cloud is picking up speed with our customers as well. The ease with which one can set up an environment on a public cloud and break it down again offers a whole lot of options to our customers. Scaling is becoming an issue more often, especially when a web application has to deal with a sudden peak in concurrent users. The public cloud works great for that. Where those public cloud companies lack in personal contact, it makes up for it with the range of possibilities the platforms offer. The pay-as-you-go constructions help a lot here as well.

At Kumina, we notice that a lot of customers are expressing more and more interest in those possibilities. Although we’ve been offering our services on AWS as well for quite a while (as we are still a maintenance company, not a hosting company), until recently most of our customer’s systems ended up on either their own hardware or our hosting cluster. This is changing, however, as our customers are interested in the scaling options and quick replication that a cloud environment provides. And we’re happy to provide.

We wouldn’t be Kumina if we didn’t have an opinion about the best way of doing that. Since 2016, we’ve been working a lot with containers and Kubernetes and we’re convinced that currently, there’s no better way of working with applications than running them containerized within a Kubernetes cluster. This provides a lot of additional possibilities to our customers, which in turn allows them to do more in less time. We love to support them in this.

So for the immediate future, we expect an uptick in the number of Kubernetes setups we administer, which includes more than “just” maintaining Kubernetes of course. We add monitoring and metrics collection via Prometheus, ElasticSearch for log aggregation and lots more. We even provide a full development stack, from a Gitlab instance to an automatically deploying Jenkins.

2017 is promising to become a very interesting year regarding all the new possibilities. Are you wondering if we can be of meaning to your organisation? We always offer a free consult and advice by phone, so don’t hesitate to contact us!

Meet our new colleagues

Last year, we welcomed two new colleagues to our team: Ed Schouten and Bart Vercoulen.

In 2011 during his studies, Ed worked as part-time employee at Kumina. Five years later, after working at Google, we welcomed him back to our team. In his combined function as system administration and software developer, he mainly focusses on process optimisation. He is currently working on replacing our monitoring system. With this new feature-rich system, we can get more insights and also share these with our customers. This enables us to solve potential alerts faster and gives us the ability to get the most performance out of our systems. With his experience with large systems, algorithms, developer techniques and extended knowledge on the tools we use, like Kubernetes, Prometheus and Cassandra, he is great addition to our team.

We also welcomed our new colleague Bart, who started as our part-time junior developer. He supports the team and our internal processes by developing tools that we need internally. Thanks to his commitment we were able to take great strides towards augmenting the coverage of our monitoring and trending the past year. With his Icinga checks and Prometheus collectors we are now able to detect potential problems even faster. In the near future he will work on making our customers set-ups more comprehensible. We hope we are able to offer this new service to all our customers in the year to come.

In 2017 Niek Geerts will start as our new system administrator, who will be introduced at some point in the future. We continue assuring and improving the quality of our services. For example, we just started the process to obtain ISO 27001 certification. We will keep you informed!

Optimisation and Innovation: Open Source Releases and sponsoring
Since the foundation of Kumina we have worked almost exclusively with Open Source software. Past year, we gladly contributed to the open source community by releasing several open source software improvements and initiatives. We also decided to sponsor a promising new open source project by the company Nuxi.

Open Source Releases
Once in a while we face a challenge, without there being a solution that lives up to our quality standards. This is also the case with our current project to reimplement our monitoring to be based on Prometheus. In some cases we want to be able monitor applications that cannot yet interface with Prometheus, which is why we’ve designed these components ourselves. Curious about these and other open source releases from last year? Have a look at our Business Github page or click around on this blog.

Cooperation and sponsoring Nuxi / CloudABI
In 2014, our colleague Ed started an open source project named CloudABI. CloudABI is a framework which allows software developers to build applications that are strongly sandboxed. Sandboxing massively reduces the impact of security problems. With the use of CloudABI it is also possible to test and manage software in a better way. Kumina decided to help Ed with this promising project by sponsoring him.

awssyncer: an automatic syncer for Amazon S3 that makes use of inotify

Friday, September 16th, 2016

# awssyncer: Continuous syncing of local files into Amazon AWS S3.

At Kumina, we’re strong users of the Amazon AWS cloud computing platform. We’ve been using EC2 instances for quite some time and are currently working on expanding this by making use of Kubernetes.

While setting this up, we’ve noticed that we sometimes want to run jobs for which we want to keep track of small amounts of local state (i.e., files on disk). In this case we’ve decided that we want to store this data in S3, but do want to have it efficiently available through the local file system. The advantage of using S3 for this purpose is that it’s globally replicated, unlike EBS.

For this purpose we’ve developed a new utility called awssyncer, which is as of now available on GitHub! awssyncer is a utility written in C++ that uses Linux’s inotify to keep track of local modifications to a directory on disk. The purpose of this utility is to use these inotify events to determine which files need to be synced back into S3. This utility thus provides continuous one-way sychronisation from local disk to S3. A simple container startup script is used to sync files from S3 to local disk on startup.

Though we realise that this utility is fairly specific to our situation at hand, we do invite all of you to give it a try. Feel free to get in touch in case you have any questions or discover any bugs!

Published our repositories

Friday, April 15th, 2011

So we’ve finally gotten around to publishing our puppet generic and kbp repositories, as described in this previous post. They still need a lot of work (and a lot of documentation!), but it’s a start. And having them public gives us the extra push to add the documentation. You can find them here:

Not all in there is originally ours, but most of it is.

Feel free to send pull requests if you fix stuff!