The footgun with Docker Compose shared configurations

Banner octopus graphic from the Compose repo

Docker Compose has a nice feature where several Compose files can be seamlessly fashioned together and allow for configuration reuse across environments. There is, however, counterintuitive behavior that can lead one to accidentally overwrite remote container images. Since I lost several hours to this, I figured I’d write about it to sear the behavior into my memory.

Create a folder structure like the one below:

.
├── docker-compose.yml
├── docker-compose.override.yml
├── Dockerfile
└── hello

With the following contents:

File: docker-compose.yml
version: '3'
services:
  mytest:
    image: alpine:3.14
    command: ls -1 /root/hello

File: docker-compose.override.yml
version: '3'
services:
  mytest:
    build:
      context: .

File: Dockerfile
FROM alpine:3.14
COPY hello /root/.

File: hello   <EMPTY>

In short, we’re creating a container that looks to see if a file is present and then we create an override configuration that adds the file to the container.

Running the container shouldn’t give any surprises:

$ docker-compose up --build
mytest-1  | /root/hello

But now it’s too late. The footgun has fired and we’ve overwritten the alpine:3.14 image:

$ docker run --rm alpine:3.14 ls -1 /root/hello
/root/hello

How did this happen

In the demonstration, the two Compose files combine to essentially form a single config that defines both a build and image attributes:

version: '3'
services:
  mytest:
    image: alpine:3.14
    build:
      context: .
    command: ls -1 /root/hello

The official docs include a section on Compose files with an image and a build section, but is vague (at least to me) about what the behavior should be.

It’s only when I looked at the logs did the behavior become clear that Compose overwrites the remote image tag.

 => [2/2] COPY hello /root/.
 => exporting to image
 => => exporting layers
 => => writing image sha256:420d8b7bfcf56d40706ffa2608310d47737df326ee789ffe3c788547d1c965f5
 => => naming to docker.io/library/alpine:3.14

The fix

First we’ll want to remove our accidental image:

docker-compose down
docker rmi alpine:3.14

Then give any overrides that have a build section a custom image name:

version: '3'
services:
  mytest:
    image: mytest/dev
    build:
      context: .

Now our resolved config looks like:

version: '3'
services:
  mytest:
    image: mytest/dev
    build:
      context: .
    command: ls -1 /root/hello

Our config now tags the built image with mytest/dev instead of alpine:3.14. Crisis averted.

We can take this a step further and protect our base Compose config by using the image’s sha256 hash.

version: '3'
services:
  mytest:
    image: alpine@sha256:4ff3ca91275773af45cb4b0834e12b7eb47d1c18f770a0b151381cd227f4c253
    command: ls -1 /root/hello

Now, if a downstream override defines its own build section but forgets to name the image appropriately, Compose will fail as one can’t overwrite an image referenced by a hash. Just make sure to propagate the hash used to the Dockerfile so that both the base and the override are in sync.

Real life occurrence

The example shown may be contrived, but how this problem reared its head in real life should make a more compelling argument.

I heavily rely on multiple Compose files in a project to share configuration between test, dev, and production environments.

The production environment uses an S3 endpoint (backblaze b2, specifically), the dev environment builds a minio container with TLS support, while the test environment uses minio without TLS support.

The only thing to know about minio is that it will automatically enable TLS if the relevant files are found in the certs directory.

It may be obvious where we’re going with this but here’s the setup, we use the test environment to store all the common configurations. Then the dev environment copies in the certs.

File: docker-compose.test.yml
version: '3'
services:
  s3:
    image: minio/minio:RELEASE.2022-07-08T00-05-23Z
    command: server /data
    # ... bunch more common configuration

File: docker-compose.dev.yml
version: '3'
services:
  s3:
    build:
      context: .
    command: server /data

File: Dockerfile
FROM minio/minio:RELEASE.2022-07-08T00-05-23Z
COPY ./certs /root/.minio/certs

File: certs/private.key   <EMPTY>
File: certs/public.crt   <EMPTY>

And they are invoked like so:

# spin up test environment
docker-compose -f ./docker-compose.test.yml # ...

# spin up dev environment
docker-compose -f ./docker-compose.test.yml \
               -f ./docker-compose.dev.yml # ...

Tests would initially work, but once I spun up the dev environment, all the tests would fail and I couldn’t figure out why.

The AWS node.js SDK returned some error about parsing unexpected input “C”, which didn’t give me much to start with, so the first hour was spent locating the issue and realizing that minio had been initialized with certs. The second hour was figuring out why minio was starting with certs. Definitely a frustrating experience, as I had assumed that I was starting from a known good state when using a specific image version.

I’ve half-heartedly pondered if there is a better way for Docker to disallow this very scenario, but there’s probably someone somewhere relying on the current behavior, so the rest of us need to step up our game and program and configure defensively.