Writing a Secure Systemd Service with Sandboxing and Dynamic Users

This post will walk one through a real world migration on how to apply the principle of minimal privilege to a systemd service. This is accomplished by extracting sensitive configuration fields into an environment file, templating the config, running the service as a dynamic user, and sandboxing the application with systemd primitives.

I’ve authored dness, the dynamic DNS client, which should be a conceptually easy to understand service for us to migrate. The program works like so:

• Read a TOML config file with DNS provider information (cloudflare, namecheap, godaddy)
• Query for the current WAN address using DNS or a service like ipify
• Query records stored in DNS provider, find any mismatches, and update
• Exit

Repeat this process every 5 minutes or so.

Baseline

I run dness on servers running systemd, so it’s natural to write a systemd service file to describe to the system how the application is executed. The dness process can be succinctly represented with the following systemd service file:

[Unit]
Description=A dynamic DNS client
Wants=network-online.target
After=network.target network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/dness -c /etc/dness/dness.conf

This is ok if one has absolute trust in themselves (which one shouldn’t), but this service file is running dness as root with unfettered access.

The original impetus for writing the service file this way is that dness.conf contains sensitive information about DNS provider accounts, as dness needs credentials to query and update DNS records. The proper way would be to create a dness user which dness is executed as and restrict dness.conf to this user. However, just as I’m a lazy developer, I’m a lazy sysadmin, and I took the easy approach of just running it as root and removing world read access to the config.

Systemd isn’t blind to this security hole. Recent versions of systemd ship with a utility to measure how exposed a system is from a given service:

systemd-analyze security dness

Gave a report that ended with:

→ Overall exposure level for dness.service: 9.6 UNSAFE 😨

For context, the score is out of 10 with 10 being the most unsafe. This doesn’t mean that dness is insecure, it’s just not security conscious and using systemd’s security features. If there was an arbitrary code execution vulnerability in dness then there is nothing mitigating the potential damages. Our goal is to shore defenses so that users (including myself) can be more confident in running dness. We should strive to follow the principle of minimal privilege where dness only has access to required resources.

Unprivileged Access to Sensitive Values

To fix the glaring issue of running as root we will first need to address dness.conf and solve how to pass sensitive fields (or allow access to sensitive files) for unprivileged accounts.

The method I applied is to split the sensitive fields from the configuration file into a separate file that declares them as environment variables. As long as this separate file is not world readable, the configuration file can be world readable without exposure of sensitive fields. And if the configuration file is world readable, then a non-privileged user with the correct environment can execute the service.

Example time. Let’s say our original dness.conf looked like:

[[domains]]
type = "cloudflare"
token = "dec0de"
zone = "example.com"
records = [
    "n.example.com"
]

The dec0de is a sensitive value, so the file shouldn’t be world readable. But if the config looked like:

[[domains]]
type = "cloudflare"
token = "{{MY_CLOUDFLARE_TOKEN}}"
zone = "example.com"
records = [
    "n.example.com"
]

Then there is nothing sensitive about dness.conf, so it can be made world readable. The dec0de has been extracted to a sibling dness.env, a file that has restricted permissions (root read/write only) and is sourced by a privileged account before startup:

MY_CLOUDFLARE_TOKEN=dec0de

Then dness.conf is rendered as a handlebar template with variables set from the environment before being parsed as regular TOML input. Handlebar was chosen due to its simplicity and ubiquity.

By using a templated config constructed from the execution environment, we can leverage systemd’s DynamicUser to allocate a new unprivileged user with every invocation. Read Lennart’s post for more info on DynamicUser. Our service file now looks like

[Unit]
Description=A dynamic DNS client
Wants=network-online.target
After=network.target network-online.target

[Service]
Type=oneshot
DynamicUser=yes
ExecStart=/usr/bin/dness -c /etc/dness/dness.conf
EnvironmentFile=-/etc/dness/dness.env

The EnvironmentFile is prefixed to - to denote that this file is not required for dness to run.

Let’s see where we stand now with the systemd security analysis:

systemd-analyze security dness

Outputs:

Overall exposure level for dness.service: 8.2 EXPOSED 🙁

A marked improvement to be sure, as we now have the convenience of not needing to create a system user to manage dness without compromising the security of sensitive fields. From the reported score of 8.2 there is plenty of low hanging fruit remaining.

Before moving on, an alternative to templated configs is for distros with systemd 247+ (at the moment, I believe practically none) can use LoadCredential to achieve the same thing. To me, using the handlebar config file is nice enough that I don’t plan on utilizing LoadCredential even when it reaches widespread availability.

Sandboxing

Now is the part where we describe to systemd what the application requires to function. Every app is different, and my recommendation is to start with a blank slate (which we did) and add restrictions until the app fails to function. It can be tedious to narrow in on the correct set, but the vast majority of applications will be able to enable most of the restrictions. For dness, I ended up with a service file that looks like:

[Unit]
Description=A dynamic DNS client
Wants=network-online.target
After=network.target network-online.target

[Service]
Type=oneshot
DynamicUser=yes
ExecStart=/usr/bin/dness -c /etc/dness/dness.conf
EnvironmentFile=-/etc/dness/dness.env

CapabilityBoundingSet=
RestrictAddressFamilies=AF_INET AF_INET6
SystemCallArchitectures=native
LockPersonality=yes
MemoryDenyWriteExecute=yes
PrivateDevices=yes
PrivateUsers=yes
ProtectControlGroups=yes
ProtectHome=yes
ProtectHostname=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
RestrictNamespaces=yes
RestrictRealtime=yes
SystemCallFilter=~@privileged @resources
SystemCallFilter=@system-service

Systemd is able to parse this file and determine that this application doesn’t do much other than talk to the internet via IPv4 (AF_INET) and IPv6 (AF_INET6).

One nice thing about systemd sandboxing is that it gracefully ignores unknown options. For instance, on my Ubuntu 18.04 machine (systemd 237) there are several unknown fields like (ProtectHostname) or values (@system-service) but systemd is still able to start the service fine and sandbox what it can.

Deducing the allowed system calls (snippet shown below) was the most aggravating part of writing this service file.

SystemCallFilter=~@privileged @resources
SystemCallFilter=@system-service

The proper way would be to strace known good dness executions, record all the syscalls, and then whitelist them. My laziness and preference for minimal configuration struck and I figured I’d use systemd’s predefined system call sets like @system-service. Unfortunately, @system-service is not available on Ubuntu 18.04 and it expands to syscalls (eg: ioctl, getrandom) that are not categorized under another set. So I settled on whitelisting system service syscalls (if available) and denying privileged and resources calls.

Let’s see where we stand now with the systemd security analysis:

systemd-analyze security dness

Outputs:

→ Overall exposure level for dness.service: 1.2 OK 🙂

1.2 is a great result and one should feel confident that the application has been sandboxed enough. There is no way to reach zero as dness needs to reach out to the internet to function.

Docker

I want to quickly touch on an argument to this approach of tightening down our service via systemd primitives. I know running everything (and I do mean everything) under docker is gaining popularity, and some might question why not run dness as a docker container. I’m a big user of docker. Love it – never going to install bare-metal redis or the JVM ever again. But it’s not a panacea. There are tradeoffs:

• If there is not a semi-official docker image then one must comb through third party docker images
• Disk and startup penalties
• Overkill for static binaries

Conclusion

The migration we did in this article was simplistic. There are plenty of other knobs that are available like StateDirectory for those that persist state to disk, socket activation, and chroot jails. These will become useful for more involved services.

In the end, it’s the sysadmin’s choice on how they want to run applications, but hopefully this article showed a great way for one to migrate a service to a more secure model without always feeling the need to resort to docker or virtual machines.