pungrumpy.com

machine view

Compose from first principles

TL;DR: Compose never sends your YAML to the daemon. It parses, interpolates, merges, expands, and resolves the file into a canonical model first, and only that model reaches the Engine API. This post rebuilds that pipeline step by step, checks each step against docker compose config, and shows why a compose linter that skips it ends up judging strings Docker never sees.

Published ....
Authors ......
Noppakorn Kaewsalabnil
Categories ...
DevOpsDeveloper Tools
Markdown .....
/writing/compose-from-first-principles.md
Human ........
/writing/compose-from-first-principles

Post

[image: Compose from first principles.png]

Run `docker compose config` in a project you know well and read the output next to the file you wrote. Ports have become objects. The environment list is a map. A `~` in a volume path is now your home directory, and a `${VAR:-default}` you never set has been replaced by its default. That output is not a pretty-printer. It is the data Compose sends to the Engine, and it is the first time most people see how much happens between the two.

I maintain a linter for these files. docker-doctor reads a compose file, runs rules over it, and prints a score, and every rule is a claim about what Docker will do with the file. A claim like that can only be right if the rule looks at the same thing Docker looks at. I learned how wide that gap is by shipping a rule that matched one path and watching it miss the same path spelled four other ways. This post is the model I wish I’d had before writing it: what Compose does between the YAML and the API, rebuilt one step at a time and checked against real `docker compose config` runs.

## What the daemon receives

The Docker daemon has never seen a compose file. Its API takes JSON. A container is created with `POST /containers/create`, and the mounts live in `HostConfig`: `Binds` is a list of `host-src:container-dest[:options]` strings, and `Mounts` is a list of objects with a `Type`, a `Source`, and a `Target`. There is no `services:` key in that payload, no `${VAR}`, no `~`, and no override file.

The daemon also has no notion of partial trust. Docker’s security page says only trusted users should be allowed to control the daemon. The reason it gives is that a client can start a container whose `/host` directory is `/` on the host, and alter the host filesystem without restriction. Every argument in this post about a linter rule rests on that fact. Anything that can reach the API has root on the machine.

Compose is the client that turns your file into those calls. Compose v2 is a Go program invoked as `docker compose`, and its loader is a library, compose-go, that the CLI imports. `docker compose config` exposes the loader’s output. The docs describe it as rendering the actual data model to be applied on the Docker Engine: it merges the files set by `-f`, resolves variables, and expands short notation into the canonical format. Here is a file most people would call finished:

```yaml compose.yaml
services:
  web:
    image: nginx
    ports:
      - "8080:80"
    environment:
      - FOO
      - BAR=baz
    volumes:
      - data:/var/lib/data:ro
volumes:
  data:
```

And here is the service after Compose is done with it:

```sh
$ docker compose config
services:
  web:
    environment:
      BAR: baz
      FOO: null
    image: nginx
    networks:
      default: null
    ports:
      - mode: ingress
        target: 80
        published: "8080"
        protocol: tcp
    volumes:
      - type: volume
        source: data
        target: /var/lib/data
        read_only: true
        volume: {}
```

Three of the four values changed shape. `"8080:80"` became a mapping with a mode and a protocol. `FOO` with no value became `FOO: null`, which Compose reads as “pass this through from the host”. `data:/var/lib/data:ro` became a mount with `type: volume` and `read_only: true`. Every rule in a compose linter is written against one of these two shapes, and picking the wrong one is the whole bug.

## Rebuilding the loader in six steps

compose-go’s loader runs thirteen steps in a fixed order, from parsing each file to filtering by profile. The diagram shows the ones a linter has to care about. The rest of this section rebuilds them in TypeScript and checks each one against Compose.

What runs between the file and the API

```trazo
file[compose.yaml] --> parse[Parse YAML]
parse --> interp[Interpolate]
interp --> extend[Resolve include and extends]
extend --> merge[Merge -f files]
merge --> validate[Validate schema]
validate --> canon[Expand short syntax]
canon --> paths[Resolve paths]
paths --> profiles[Filter profiles]
profiles --> api[Engine API]
```

### 1. Parse the YAML with merge keys on

Compose files lean on YAML anchors. The documented pattern is an `x-` extension field holding an anchor, with services pulling it in through the merge key `<<: *name`. Compose ignores any top-level field starting with `x-`, which the docs call the sole exception where it silently ignores unrecognized fields, and the merge key only applies to mappings, never sequences.

That merge key is YAML 1.1. The `yaml` package on npm defaults to the 1.2 schema, which leaves `<<` as a literal key, so a parser that forgets the option sees a service with a `<<` property and no volumes:

```typescript parse.ts
import { parse } from "yaml";

export const parseCompose = (source: string): unknown =>
  parse(source, { merge: true });
```

This is the one step docker-doctor runs for every compose rule. Everything below is what a rule has to do for itself.

### 2. Interpolate before anything else

Interpolation is the second step, and it runs on the raw parsed tree: before validation, before short syntax is expanded, before paths resolve. The syntax is the shell’s. `$VAR` and `${VAR}` substitute. `${VAR:-default}` uses the default when the variable is unset or empty, and `${VAR-default}` only when it is unset. `${VAR:?error}` fails the load, `${VAR:+alt}` substitutes the alternative when the variable is set, and `$$` is a literal dollar sign. Expressions nest, so `${VARIABLE:-${FOO:-default}}` is legal. Interpolation applies to values only, never keys.

Values come from three places, in order: the shell environment, then a file passed with `--env-file`, then a `.env` next to the compose file. A variable found in none of them is replaced with an empty string and a warning. `docker compose config --variables` lists every variable the file references with its default:

```sh
$ docker compose config --variables
NAME          REQUIRED   DEFAULT VALUE          ALTERNATE VALUE
DOCKER_SOCK   false      /var/run/docker.sock
```

A linter runs on a machine that is not the one running `up`, so it has no shell environment worth trusting. The value the file ships with is the default, and that is the value to judge. Here is a resolver that does only that, one level deep, where compose-go handles nesting:

```typescript interpolate.ts
const BRACED =
  /\$\{(?<name>[A-Za-z_][A-Za-z0-9_]*)(?:(?<op>:?[-?+])(?<arg>[^}]*))?\}/gu;

export const interpolate = (value: string): string =>
  value.replace(BRACED, (...args) => {
    const { op, arg = "" } = args.at(-1) as {
      op?: string;
      arg?: string;
    };
    if (op === ":-" || op === "-") {
      return arg;
    }
    return "";
  });
```

A variable with no default is not clean. It is unknown, and unknown has a sharp edge for a bind mount source. With `DOCKER_SOCK` unset, `${DOCKER_SOCK}:/var/run/docker.sock` interpolates to `:/var/run/docker.sock`, and Compose refuses the whole project with `invalid spec: :/var/run/docker.sock: empty section between colons`. The file the linter approved does not load.

### 3. Resolve extends, then merge the files

`extends` copies a service definition from another service or file into this one. The `file` path is relative to the main compose file, circular references are an error, and the docs are explicit that referenced resources such as volumes and networks are not imported for you. Then the files merge. Compose reads `compose.yaml` and an optional `compose.override.yaml` by default, or the `-f` files in the order given, and later files win.

The merge rules are what make override files useful and linters wrong. Mappings merge by adding missing entries and merging conflicting ones. Sequences append, except `command`, `entrypoint`, and `healthcheck.test`, which the last file replaces outright. Some sequences carry a unique key: `volumes` are unique by `target`, `ports` by the tuple of ip, target, published, and protocol, and `secrets` and `configs` by target. An entry that shares the key merges into the existing one instead of appending. `!reset` empties an attribute and `!override` replaces it wholesale.

Two files show the order and the uniqueness rule at once:

```yaml compose.yaml
x-sock: &sock
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
services:
  base:
    image: alpine:3.20
    volumes:
      - ./data:/data
  gw:
    <<: *sock
    extends:
      service: base
    profiles: [agents]
```
```yaml compose.override.yaml
services:
  base:
    volumes:
      - ./other:/data
      - ./logs:/logs
```

The merged `base` has `./logs`, appended, and `./other` at `/data`, because the override shares that target and replaced the source. `gw` is the interesting one: it got `./data`, not `./other`. Compose applied `extends` inside `compose.yaml` before it merged the override, so the extended service never saw the override at all.

```sh
$ docker compose --profile agents config --no-path-resolution
services:
  base:
    volumes:
      - type: bind
        source: ./other
        target: /data
      - type: bind
        source: ./logs
        target: /logs
  gw:
    profiles:
      - agents
    volumes:
      - type: bind
        source: ./data
        target: /data
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
```

For a linter, the merge step is a question of scope. docker-doctor discovers every `compose.*.yaml` and scores each on its own. An override that adds a socket mount is caught, since the override is a file too. An override that uses `!reset` to drop a mount from the base is not caught, and the base file keeps its finding. Per-file linting is a defensible choice. It is also a different claim from what `up` will do.

### 4. Validate, then expand the short syntax

Validation runs on the merged tree, so the schema sees `!override` already applied. Then the canonical transform rewrites every short form into its long form. The volumes case is the one that bit me, and compose-go’s `ParseVolume` is short enough to read in full. It walks the string and splits on `:`, but only when it is not inside a `${…}` substitution. A single section is an anonymous volume at that target. With two or more sections, the first character of the source decides. `.`, `/`, or `~` means a bind mount, a leading `\\` or a drive letter means a Windows path and also a bind, and anything else is a named volume.

```typescript volume.ts
const HOST_PATH = /^(?:[./~]|\\\\|[A-Za-z]:)/u;

const splitSpec = (spec: string): string[] => {
  const parts: string[] = [];
  let depth = 0;
  let current = "";
  for (const char of spec) {
    if (char === "{") depth += 1;
    if (char === "}") depth -= 1;
    if (char === ":" && depth === 0) {
      parts.push(current);
      current = "";
      continue;
    }
    current += char;
  }
  return [...parts, current];
};

export const parseVolume = (spec: string) => {
  const [first, target, mode] = splitSpec(spec);
  if (target === undefined) {
    return { type: "volume", target: first };
  }
  const type = HOST_PATH.test(first) ? "bind" : "volume";
  return { type, source: first, target, readOnly: mode === "ro" };
};
```

Two outputs from Compose confirm the edges. A bare `- /var/run/docker.sock` is not a bind mount of the socket. It renders as `type: volume` with a target and no source. And `\\.\pipe\docker_engine:\\.\pipe\docker_engine`, the named pipe the daemon listens on under Windows as `npipe:////./pipe/docker_engine`, renders as a bind mount even on Linux, because the `\\` prefix is enough.

### 5. Resolve paths against the project directory

Path resolution comes after the canonical transform, so it operates on `source` fields, not on strings. `~` expands to the home directory, and relative paths become absolute against the project directory, which is the directory of the first `-f` file even when an override elsewhere supplied the path. Two things do not change. `//var/run/docker.sock` keeps both slashes, and `/run/docker.sock` is not rewritten to `/var/run/docker.sock` or the other way, because Compose does not know that `/var/run` is a symlink to `/run` on every systemd distribution. `--no-path-resolution` skips the step.

### 6. Filter services by profile

Last, profiles. Services without a `profiles` attribute are always enabled. A service with one only exists in the model when that profile is active. In the two-file example, `gw` carries `profiles: [agents]`, and `docker compose config` without `--profile agents` omits it entirely. A linter should not. The service is one flag away from running, and its socket mount is exactly as dangerous then.

## Where the socket rule sits in the pipeline

Here is the detection that started this, as it shipped in docker-doctor 0.5.0:

```typescript compose-security.ts
const DOCKER_SOCKET = "/var/run/docker.sock";

const mountsDockerSocket = (volume: unknown): boolean => {
  if (typeof volume === "string") {
    return volume.split(":")[0] === DOCKER_SOCKET;
  }
  if (volume && typeof volume === "object") {
    return (volume as Record<string, unknown>).source === DOCKER_SOCKET;
  }
  return false;
};
```

It runs on the output of step one. Take the socket spellings I collected from Docker Desktop, rootless Docker, Git Bash, and a few compose files on my own machines, and trace each through the pipeline. The mcp-gateway examples use the first row:

| Spelling in the file | The rule compares | Compose sends | 0.5.0 |
| --- | --- | --- | --- |
| /var/run/docker.sock:/var/run/docker.sock | /var/run/docker.sock | /var/run/docker.sock | Reported |
| ${DOCKER_SOCK:-/var/run/docker.sock}:/var/run/docker.sock | ${DOCKER_SOCK | /var/run/docker.sock | Missed |
| /run/docker.sock:/var/run/docker.sock | /run/docker.sock | /run/docker.sock | Missed |
| ~/.docker/run/docker.sock:/var/run/docker.sock | ~/.docker/run/docker.sock | ~ expanded to the home directory | Missed |
| //var/run/docker.sock:/var/run/docker.sock | //var/run/docker.sock | //var/run/docker.sock | Missed |
| $XDG_RUNTIME_DIR/docker.sock:/var/run/docker.sock | $XDG_RUNTIME_DIR/docker.sock | /run/user/1000/docker.sock | Missed |
| /var/run/docker.sock | /var/run/docker.sock | An anonymous volume, no source | False positive |

Row two fails twice. The rule compared `${DOCKER_SOCK`, because `split(":")` cut at the colon inside the braces, and even a correct split would have compared an expression rather than the default it resolves to. Rows two and six need step two. Row seven needs step four: `split(":")[0]` of a string with no colon is the whole string, so an anonymous volume that happens to sit at the socket’s path is reported as a bind mount of it. Row four needs step five.

Rows three and five need something the pipeline never provides. Compose sends `/run/docker.sock` and `//var/run/docker.sock` to the daemon as written, and the kernel opens the same file as `/var/run/docker.sock`: one through the symlink, one because Linux treats a leading `//` as `/`. That second fact is why Git Bash users type the double slash at all, since MSYS stops rewriting an argument that starts with `//`. Rootless Docker listens at `$XDG_RUNTIME_DIR/docker.sock`. Docker Desktop on macOS listens at `~/.docker/run/docker.sock`, and its `/var/run/docker.sock` symlink is an optional setting that asks for your password. Both are the same daemon at paths that share nothing but their last segment. So even after the loader, the rule cannot be an equality check. It has to be a shape: a source that is a host path whose last segment is `docker.sock`, or the Windows pipe.

That is what the fix merged on 5 September does, and it ships in 0.5.1. Inside the one rule, it resolves `${VAR:-default}` to the default, splits the short syntax only at brace depth zero, treats a lone path as an anonymous volume, and matches any path-shaped source whose last segment is `docker.sock` or `pipe/docker_engine`. Every row in the table now reports except the last, which is now clean. A `${VAR}` source with no default is flagged only when the target is `/var/run/docker.sock`, since the target still says what the container expects to find there. That last call is a judgment, and it is the one place the rule reads the target instead of the source.

Then put the first five rows in one service and run `config`. Compose keeps one. Volumes are unique by target, so the merge step collapses the list to whichever entry came last, and the daemon receives a single bind. A linter that reports five findings is right about the file, Docker is right about the container, and they disagree because they read different stages of the same pipeline.

The examples in docker/compose-for-agents no longer mount the socket at all. Their gateway service bind-mounted `/var/run/docker.sock` in the first revisions and now says `use_api_socket: true`, and Compose passes that key through `config` untouched. There is no path in the file, so a path-shaped rule sees nothing, while the container still reaches the engine through the API socket. The rule’s help text already points people there. The uncomfortable part is that the thing to detect was never a string. It is whether the container can reach the daemon, and that arrives under more than one key.

The model rules have the same shape. In 0.5.0, `pin-model-version` read only the top-level `models:` element, which needs Compose v2.38 or later, and flagged an artifact with no tag or with `latest`. The provider form, `provider: { type: model, options: { model: ai/example-model } }`, names the same artifact under a different key, and the rule did not read it. That mattered because the compose-for-agents examples started on the provider form in June 2025, with unpinned artifacts like `ai/qwen3`, before moving to the top-level element with tags like `ai/gemma3:4B-Q4_0`. The same fix collects bindings from both places.

## When matching the string is fine

Not every rule needs the pipeline, and running it everywhere would be its own kind of wrong. The test is whether the value reaches Docker as written:

| Value | Reaches Docker as written | The rule needs |
| --- | --- | --- |
| privileged: true | Yes, a boolean under a schema key | Step 1 |
| version: at the top level | Compose ignores it entirely | Step 1 |
| image: docker/mcp-gateway | Yes, unless it contains ${ | Step 1, and skip variables |
| A volume source | No: interpolated, split, expanded, resolved | Steps 2, 4, 5, and a fact about the host |
| An environment value | Two spellings, list and map | Step 4 |

docker-doctor’s image rules already follow the middle row. `parseImageRef` marks anything containing `${` as a variable, and the rule skips it rather than guessing. That is the right shape for every step-two problem: judge the default when there is one, and say unknown when there is not.

## What the loader still cannot see

Rebuilding the loader closes the gap between the file and the model. It does not close the gap between the model and the host. `$XDG_RUNTIME_DIR` resolves on your laptop and not in CI. Symlinks are a fact about the machine that runs `up`. Profiles hide services from `config` that a linter should keep. And the spec moves: a file using `models:` is invalid on Compose v2.37, so a linter has to pick a version to believe, and docker-doctor believes the current one.

docker-doctor still runs step one and then the rules. The 0.5.1 socket rule rebuilds steps two and four for itself, inside one rule, which is the right fix for one rule and the wrong place for it to live. The next change is to the engine: interpolation and the canonical transform in front of every compose rule, so that rules read the model and not the text. It is the difference between a linter that agrees with `docker compose config` and one that agrees with your editor.

---

If you have a compose file with an MCP (Model Context Protocol) gateway in it, run it:

```sh
$ npx @docker-doctor/cli@latest .
```

And if your file spells the socket a way that is not in the table, I want to know.

# full archive: /ai/writing