The footgun with Docker Compose shared configurationsPublished on:
Banner octopus graphic from the Compose repo
Docker Compose has a nice feature where several Compose files can be seamlessly fashioned together and allow for configuration reuse across environments. There is, however, counterintuitive behavior that can lead one to accidentally overwrite remote container images. Since I lost several hours to this, I figured I’d write about it to sear the behavior into my memory.
Create a folder structure like the one below:
. ├── docker-compose.yml ├── docker-compose.override.yml ├── Dockerfile └── hello
With the following contents:
File: docker-compose.yml version: '3' services: mytest: image: alpine:3.14 command: ls -1 /root/hello File: docker-compose.override.yml version: '3' services: mytest: build: context: . File: Dockerfile FROM alpine:3.14 COPY hello /root/. File: hello <EMPTY>
In short, we’re creating a container that looks to see if a file is present and then we create an override configuration that adds the file to the container.
Running the container shouldn’t give any surprises:
$ docker-compose up --build mytest-1 | /root/hello
But now it’s too late. The footgun has fired and we’ve overwritten the alpine:3.14 image:
$ docker run --rm alpine:3.14 ls -1 /root/hello /root/hello
How did this happen
In the demonstration, the two Compose files combine to essentially form a single config that defines both a
version: '3' services: mytest: image: alpine:3.14 build: context: . command: ls -1 /root/hello
The official docs include a section on Compose files with an
image and a
build section, but is vague (at least to me) about what the behavior should be.
It’s only when I looked at the logs did the behavior become clear that Compose overwrites the remote image tag.
=> [2/2] COPY hello /root/. => exporting to image => => exporting layers => => writing image sha256:420d8b7bfcf56d40706ffa2608310d47737df326ee789ffe3c788547d1c965f5 => => naming to docker.io/library/alpine:3.14
First we’ll want to remove our accidental image:
docker-compose down docker rmi alpine:3.14
Then give any overrides that have a
build section a custom image name:
version: '3' services: mytest: image: mytest/dev build: context: .
Now our resolved config looks like:
version: '3' services: mytest: image: mytest/dev build: context: . command: ls -1 /root/hello
Our config now tags the built image with
mytest/dev instead of
alpine:3.14. Crisis averted.
We can take this a step further and protect our base Compose config by using the image’s sha256 hash.
version: '3' services: mytest: image: [email protected]:4ff3ca91275773af45cb4b0834e12b7eb47d1c18f770a0b151381cd227f4c253 command: ls -1 /root/hello
Now, if a downstream override defines its own
build section but forgets to name the
image appropriately, Compose will fail as one can’t overwrite an image referenced by a hash. Just make sure to propagate the hash used to the
Dockerfile so that both the base and the override are in sync.
Real life occurrence
The example shown may be contrived, but how this problem reared its head in real life should make a more compelling argument.
I heavily rely on multiple Compose files in a project to share configuration between test, dev, and production environments.
The production environment uses an S3 endpoint (backblaze b2, specifically), the dev environment builds a minio container with TLS support, while the test environment uses minio without TLS support.
The only thing to know about
minio is that it will automatically enable TLS if the relevant files are found in the
It may be obvious where we’re going with this but here’s the setup, we use the test environment to store all the common configurations. Then the dev environment copies in the certs.
File: docker-compose.test.yml version: '3' services: s3: image: minio/minio:RELEASE.2022-07-08T00-05-23Z command: server /data # ... bunch more common configuration File: docker-compose.dev.yml version: '3' services: s3: build: context: . command: server /data File: Dockerfile FROM minio/minio:RELEASE.2022-07-08T00-05-23Z COPY ./certs /root/.minio/certs File: certs/private.key <EMPTY> File: certs/public.crt <EMPTY>
And they are invoked like so:
# spin up test environment docker-compose -f ./docker-compose.test.yml # ... # spin up dev environment docker-compose -f ./docker-compose.test.yml \ -f ./docker-compose.dev.yml # ...
Tests would initially work, but once I spun up the dev environment, all the tests would fail and I couldn’t figure out why.
The AWS node.js SDK returned some error about parsing unexpected input “C”, which didn’t give me much to start with, so the first hour was spent locating the issue and realizing that minio had been initialized with certs. The second hour was figuring out why minio was starting with certs. Definitely a frustrating experience, as I had assumed that I was starting from a known good state when using a specific image version.
I’ve half-heartedly pondered if there is a better way for Docker to disallow this very scenario, but there’s probably someone somewhere relying on the current behavior, so the rest of us need to step up our game and program and configure defensively.
If you'd like to leave a comment, please email [email protected]