pdns-etcd3

A PowerDNS remote backend with ETCD v3 cluster as storage. It uses the official client to get the data from the cluster. Responses are authoritative for each zone found in the data. Only the DNS class IN is supported, but that's because of the limitation of PowerDNS.

There is no stable release yet, even no beta. The latest release is v0.2.0+0.1.1, the second development release, considered alpha quality. Any testing is appreciated.

Features

• Automatic serial for SOA records (based on the cluster revision).
• Replication is handled by the ETCD cluster, no additional configuration is needed for using multiple authoritative PowerDNS servers.
  • DNS responses are nearly instantly up-to-date (on every server instance!) after data changes by using a watcher into ETCD (multi-master)
• Multiple syntax possibilities for object-supported records
• Short syntax for single-value objects
  • or for the last value left when using defaults (e.g. target in SRV)
• Default prefix for IP addresses
  • overrideable per entry
• Support for custom records (types), like those supported by PowerDNS but unimplemented in pdns-etcd3
• Support for automatically appending zone name to unqualified domain names
• Override of domain name appended to unqualified names (instead of zone name)
  • useful for PTR records in reverse zones
• Multi-level defaults and options, overridable
• Upgrade data structure (if needed for new program version) without interrupting service
• Run standalone for usage as a Unix connector
  • This could be needed for big data sets, because the initialization from PowerDNS is done lazily (at least in v4) on first request (which possibly could time out on "big data"…) :-(

Planned

• Reduce redundancy in the data by automatically deriving corresponding data
  • A ⇒ PTR (in-addr.arpa)
  • AAAA ⇒ PTR (ip6.arpa)
  • …
• Support for defaults and zone appending (and possibly more) in plain-string records (those which are also object-supported)
• "Collect record", automatically combining A and/or AAAA records from "server records"
  • e.g. etcd.example.com based on etcd-1.example.com, etcd-2.example.com, …
• "Labels" for selectively applying defaults and/or options to record entries
  • sth. like com/example/-options-ptr → {"auto-ptr": true} and com/example/www/-options-collect → {"collect": …} for com/example/www-1/A+ptr+collect without global options
  • precedence betweeen QTYPE and id (id > label > QTYPE)
• Support JSON5 by flynn/json5 (replace default JSON, because JSON5 is a superset of JSON)
• Support YAML by go-yaml
• DNSSEC support (PowerDNS DNSSEC-specific calls)
• Implement getAllDomains backend call for enabling PowerDNS caching (for performance)
  • setting zone-cache-refresh-interval

Optional

• Support more encodings for values
  • EDN by go-edn
  • TOML by pelletier/go-toml or BurntSushi/toml
  • …
• DNS update support
• Prometheus exporter

Installation

git clone https://github.com/nixn/pdns-etcd3.git
cd pdns-etcd3
make

NOTE: go build will also work, but you will get a dynamically linked executable and incomplete version information in the binary. The build command in Makefile produces a static build with setting the version string properly.

Usage

Of course, you need an up and running ETCD v3 cluster and a PowerDNS installation.

You have to decide in which mode you want to use the backend: either the pipe mode or the unix mode.

Pipe mode

In pipe mode the backend is launched by PowerDNS dynamically and communicates with it via standard input and output. All the configuration options must be given in the PowerDNS configuration file. But since PowerDNS (at least as of v4) initiates the backend lazily, the 'initialize' call occurs with the first (client) request and the backend has to be fast enough to connect to ETCD, read all data, and reply to this first request. This can be too long, if there is much data to read.

As of PowerDNS v4.5 there is a setting to cache zone data, so the backend would be started and initialized before the first client request, but this call is currently not implemented (will be implemented later). Due to this the setting zone-cache-refresh-interval currently must be set to 0 (v4.5+).

Example PowerDNS configuration file:

launch=remote
remote-connection-string=pipe:command=/path/to/pdns-etcd3[,pdns-version=3|4|5][,<config>][,prefix=<string>][,timeout=<integer>][,log-<level>=<components>]
zone-cache-refresh-interval=0
# since in pipe mode every instance connects to ETCD and loads the data for itself (uses memory), possibly do this:
distributor-threads=1

<config> is one of config-file=... or endpoints=... (see "Parameters" below for details on the value). config-file overrides endpoints.

Unix mode

In unix mode the backend must be launched outside PowerDNS (manually, e.g. as a system service). It then creates a unix domain socket and listens for connections (from PowerDNS). It takes the ETCD related parameters from the command line and connects to it right after starting up. Then it accepts connections on the socket and serves them.

Each connection still begins with an 'initialize' call, but only the non-ETCD parameters are available to it. In this mode the data is loaded only once (uses memory only once).

The current restriction on the setting zone-cache-refresh-interval (see above) is here valid too, so set it to 0 for now.

Example PowerDNS configuration file:

launch=remote
remote-connection-string=unix:path=/path/to/pdns-etcd3-socket[,pdns-version=3|4|5][,log-<level>=<components>]
zone-cache-refresh-interval=0
# in unix mode it is ok to launch multiple access threads, the data is protected by mutexes for concurrent access (including updates)
distributor-threads=3

The backend is started in unix mode by passing the -unix argument to the executable (see below for details). It accepts further arguments to configure access to ETCD, one can execute ./pdns-etcd3 -help for usage information.

Parameters

All parameter keys must be given exactly as denoted here (no case modifications). The ETCD related parameters in unix mode are given as command line "options", starting with a -: e.g. -config-file=....

The parameters in detail (the ETCD related parameters, which have to be passed as command line argument in unix mode, are tagged by #UNIX):

• config-file=/path/to/etcd.conf #UNIX
  The path to an ETCD (client) configuration file, as accepted by the official client (see etcd/clientv3/config.go, TODO find documentation)
  TLS and authentication is only possible when using such a configuration file.
  Overrides endpoints parameter. Defaults to not set.
• endpoints=<IP:Port>[|<IP:Port>|...] #UNIX
  For a simple connection use the endpoints given here. endpoints accepts hostnames too (instead of IP), but be sure they are resolvable before PowerDNS has started.
  Defaults to [::1]:2379|127.0.0.1:2379.
• prefix=<string> #UNIX
  Every entry in ETCD will be prefixed with that. It is not interpreted or changed in any way, also the data watcher uses it, so any other keys under another prefix do not affect DNS data.
  Currently there seems to be a bug(?) in the ETCD client (not pdns-etcd3), which causes an empty prefix not to work. Just use one. Tip: Let the prefix start and end with /, so you can use etcdkeeper for easier web-based data management.
  There is no default (= empty).
• timeout=<duration> #UNIX or
  timeout=<integer> config file (in milliseconds, e.g. 1500 for 1.5 seconds)
  An optional parameter which sets the dial timeout to ETCD. Must be a positive value (>= 1ms).
  Defaults to 2 seconds.
• pdns-version=3|4|5
  The (major) PowerDNS version. Version 3 and 4 have incompatible protocols with the backend, so one must use the proper one. Version 5 is accepted, but works currently the same as 4 (no relevant API changes yet).
  Defaults to 4.
• log-<level>=<components> #UNIX and config file
  Sets the logging level of <components> to <level> (see below for values). <components> is one or more of the component names, separated by +. This parameter can be "repeated" for different logging levels. In unix mode, the levels are set separately for the program and the clients (PowerDNS connections).
  Example: log-debug=main+pdns,log-trace=etcd+data
  Defaults to info for all components.

ETCD structure

See ETCD structure. The structure lies beneath the prefix parameter (see above).

Compatibility

pdns-etcd3 is tested on PowerDNS versions 3.x.y and different 4.x.y, and uses an ETCD v3 cluster (API 3.0 or higher). It's currently only one version of each minor (.x), but most likely all (later) "patch" versions (.y) are compatible. Therefore, each release shall state which exact versions were used for testing, so one can be sure to have a working combination for deploying, when using those (tested) versions.

Testing / Debugging

There is much logging in the program for being able to test and debug it properly. It is structured and leveled, utilizing logrus. The structure consists of different components, namely main, pdns, etcd and data; the (seven) logging levels are taken from logrus. For each component an own logging level can be set, so that one can debug only the component(s) of interest. In the unix mode the components are "doubled", there is the program side with its components (main, etcd, data) and the (PDNS) client side (main, pdns), which can be configured separately. In pipe mode there is only one of each component.

The components in detail:

• main - The main thread / loop of the program, e.g. setting up logging, creating data objects, processing signals and events, etc.
• pdns - The communication with PowerDNS, e.g. incoming requests and sending results.
• etcd - The communication with ETCD, e.g. real queries against it, connection issues, watchers, etc.
• data - Everything concerning the values (records, ...), parsing data from ETCD, searching records for requests etc.

The levels in detail:

• panic - Something like the world's end. Actually not used.
• fatal - Errors which prevent the program to continue service. After a fatal error the program exits. (Mostly in main component.)
• error - Errors which don't prevent the program to continue service. Different meanings for different components.
• warning (or warn) - Not errors, but situations where it could be done better. An admin should take care of those.
• info - Useful information on the program, something like "initialized, ready for service". This is the default level for each component.
• debug - "Big steps", like "sending request to ETCD", "Handling event" or "default value not found for X"
• trace - Small steps and all values, e.g. "found default value for X in Y" or "record: www.example.com./A#some-id = 192.0.2.12"

License

Copyright © 2016-2024 nix https://keybase.io/nixn

Distributed under the Apache 2.0 license, available in the file LICENSE.

Donations

If you like pdns-etcd3, please consider donating to support the further development. Thank you!

Bitcoin (BTC): 1pdns4U2r4JqkzsJRpTEYNirTFLtuWee9
Monero (XMR): 4CjXUfpdcba5G5z1LXAx3ngoDtAHoFGdpJWvCayULXeaEhA4QvJEHdR7Xi3ptsbhSfGcSpdBHbK4CgyC6Qcwy5Rt2GGDfQCM7PcTgfEQ5Q
Ethereum (ETH): 0x003D87efb7069e875a8a1226c9DadaC03dE1f779

These addresses are dedicated to pdns-etcd3 development. For my general development, other projects and personal donation addresses see my profile or my web page.