On our new DNS infrastructure (and DNSSEC)

Recently we’ve been busy implementing a new DNS infrastructure for our resolvers as well as our authoritative servers. We wanted to be ready for future developments like DNSSEC and we wanted to re-new this important part of our infrastructure for a while. This blog-post gives an overview of our new setup.

Legacy Infrastructure
Our legacy authoritative infrastructure was based on a hidden master running PowerDNS 2.9.22 supplying our ns1 and ns2 running BIND 9.7.3 with the zones via AXFR. As, in our opinion, AXFRs are slow, unsecured (by default) and can cause inconsistency between the slaves when the AXFR fails, we opted for using native database replication as the means of distributing altered zone-data to the authoritative servers.

The legacy resolver was a PowerDNS authoritative server (2.9.22) that contained our private zones and allowed recursion for our (public) office IP (quelle horreur).

So we started this project to accomplish several goals:

• Simplify the infrastructure
• Design security into the infrastructure
• Be DNSSEC-ready
• Be fully dual-stack (IPv4 and IPv6)

Public Authoritative Infrastructure
For our authoritative nameservers we did away with the hidden master setup. We switched to PowerDNS 3.3 with the MySQL backend, we can use MySQL replication to keep the nameservers in sync and update records on all servers instantaneous. To ensure integrity, authentication and confidentiality of the data passed through MySQL (hey, there might be private keys for DNSSEC in there), we have enabled (and require) SSL for replication.

We now have enabled DNSSEC on several of our domains and are planning to sign some of our customer’s domains soon and both our authoritative servers are reachable over IPv4 and IPv6:

;; QUESTION SECTION:
;kumina.nl. IN NS

;; ANSWER SECTION:
kumina.nl. 600 IN NS ns4.kumina.nl.
kumina.nl. 600 IN NS ns3.kumina.nl.

;; ADDITIONAL SECTION:
ns4.kumina.nl. 600 IN AAAA 2a01:4f8:160:8001::3
ns3.kumina.nl. 600 IN A 212.153.88.206
ns4.kumina.nl. 600 IN A 5.9.10.23
ns3.kumina.nl. 600 IN AAAA 2a02:2aa8:10a:2003::6

Resolver Infrastructure for Our Office and Road-Warriors
As we are purists and don’t want to leak private zone-information – like RFC1918 addresses or reverse zones for RFC1918 addresses – onto the public internet, we have a resolver that is also authoritative for our private zones.

The resolver is Unbound 1.4.17 and listens only on the VPN interface, as our office network and road-warriors are all connected to the VPN.

The authoritative DNS is provided by a PowerDNS 3.3 instance running on the local loopback interface. These zones are configured in Unbound as stub-zones that query the PowerDNS instance. As a picture explains more than a 1000 blogposts, here’s a picture of the setup:

Resolver Infrastructure for Servers
The new resolver infrastructure has been set up using Unbound 1.4.17. Unbound is designed to be fast, secure and has DNSSEC validation built-in. In our datacenter where we host our and some of our customer’s machines, we have set up 2 resolver virtual machines. This was very easy with the puppet modules we created:

$allowed_networks = ['1.2.3.4/25', '3.4.5.6/23']
include kbp_unbound
kbp_unbound::allow { $allow_networks:; }


Puppet takes care of the installation, retrieval of the root DNSSEC key and the setup of the validation by Unbound. After a day of testing, we were confident that it did the job. We switched the resolv.conf on all servers to use these resolvers and haven’t had a single problem since (we did this 4 months ago). Any private zones or zone-data for our customers is deployed by puppet as /etc/hosts entries, so these resolvers are very straight-forward.

Zone Data Management
To manage our zones, we switched from PowerAdmin to django-powerdns-manager (with some patches) for a couple of reasons:

• It’s written in Python (we prefer that over PHP)
• It has support for DNSSEC-enabled database schemas
• It can import zones via AXFR and zonefiles

Monitoring DNSSEC
We would like to know whether or not our secured zones validate and are delegated correctly. To do this we created a Nagios check that validates the DNSSEC chain for the SOA record of the secured zone. With DNSSEC (and DNS data in general) it is a bad idea to have broken data, like CNAMEs with other data or out-of-zone data, in your zones. PowerDNS comes with the pdnssec tool that provides the check-all-zones command to check for these errors. We built another Nagios check around this tool to warn us when there is erroneous data in one of our zones.

Tags: Checks, DNS, DNSSEC, Infrastructure, nagios, PDNS, powerdns, puppet, python, Unbound