I moved a 159-image OpenStack build to GitLab CI. Layer caching is not what made it faster.

Every post on Docker layer caching in GitLab CI says the same thing. Add --cache-from and the build gets faster. I spent a week moving an OpenStack image build from a hand-run server to GitLab CI on Kubernetes, and the layer cache turned out to be the last thing that mattered. This is the week as it happened, with the numbers.

I am an SRE on a team that runs an OpenStack private cloud. Last week my manager asked for something short to say and slow to do: make the container image build follow GitOps. A commit on the default branch or a git tag should drive the build. Nobody should run kolla-build by hand on the build server any more.

The build server was the whole problem. It was one long-lived machine where the team had built the images by hand for years. It had a checkout of the repo, a Python virtualenv with the right kolla version, a local copy of a patched OpenStack service under /opt, and a Docker daemon with every layer from every build still on disk. It worked because one person knew it.

The repo had no .gitlab-ci.yml and no tags. I had read the usual articles on Docker layer caching in GitLab CI before I started, and they all tell the same story. Docker-in-Docker starts empty, so every layer rebuilds. Add --cache-from, maybe BUILDKIT_INLINE_CACHE=1 or a registry cache with mode=max, and your ten-minute build becomes a two-minute build.

I went in expecting the cache to be the main piece of work. It was not. Here is the week.

Day one: build it on my laptop first

I did not know kolla. Before writing a pipeline for it I wanted to see one build with my own eyes, so I ran it on my Mac. That took most of a day and showed me how kolla works. The Mac is arm64, so every image I built was an arm64 image, which is why the pipeline later refuses a daemon whose architecture does not match the config.

Kolla renders one Dockerfile per OpenStack service from templates, then builds them as a tree. base is the Ubuntu image with the apt sources. openstack-base adds Python and the shared libraries. keystone-base adds keystone, and keystone, keystone-fernet and keystone-ssh build on that. Every service repeats the pattern. For our config that is 159 Dockerfiles.

Two lines in kolla-build.conf caught my eye on the first read:

1
# Do not rebuild images present in the docker cache
2
skip_existing = false
3

4
# Use the Docker cache when building
5
cache = false

I assumed they were an oversight from the hand-run days and made a note to turn them on later. I was wrong about that, but it took until day four to see why.

The laptop build also found the first blocker. One service, neutron, is built from type=local location=/opt/neutron. That directory exists on the build server and nowhere else. No runner can build those 12 images until the source is in git. I put it on the list for my manager and moved on.

Day two: the pipeline, approved on paper

I wrote the pipeline the way the team's other repos do it and sent my manager a diagram. It has four jobs:

• lint on a merge request and on the default branch: render all 159 Dockerfiles, run the unit tests for the registry helper.
• build: run kolla. A merge request builds and never pushes. A commit on the default branch builds and pushes a dev tag. A release candidate tag like 2025.1-rc1.12 builds and pushes that tag. A manual run on any branch is a test build, and it pushes only if whoever runs it adds PUSH=true.
• promote on a production tag like 2025.1-1.12: do not build. Check that the production tag and its release candidate point at the same commit, then retag the images through the registry API.
• clean on demand: delete a tag from every repository. Release candidate and dev tags delete at once. Production tags need a second confirmation variable.

The manager's reply was one line: looks fine, try it this way and adjust to real use later. That sentence decided how the rest of the week went. Try first, measure, then argue.

One design choice matters for caching more than any build flag. A production tag never rebuilds. The digest that passed testing as a release candidate is the digest that ships. That is a 40-minute build replaced by a few hundred API calls, and it is better than any cache hit because there is no question of whether the hit was correct.

1
promote:
2
  resource_group: registry
3
  rules:
4
    - if: $CI_COMMIT_TAG =~ /^[0-9]{4}\.[0-9]+-[0-9]+\.[0-9]+$/
5
  before_script:
6
    - git fetch --quiet --tags origin # the rc tag has to be there to compare
7
    - RC_TAG=${CI_COMMIT_TAG%-*}-rc${CI_COMMIT_TAG#*-}
8
    - test "$(git rev-parse "$RC_TAG^{commit}")" = "$CI_COMMIT_SHA"
9
  script:
10
    - LIST_ONLY=true TAG=$CI_COMMIT_TAG tools/build.sh | tools/registry.py retag --from "$RC_TAG" --to "$CI_COMMIT_TAG"

The retag command reads the image names from kolla's own list, so it tags the same set the build job would have built. It tags nothing if any image lacks the release candidate tag or already has the production tag on a different digest.

The first pipeline failed at checkout, before any of that ran. The repo carries seven kolla submodules, one per OpenStack release, and only one is used. I had set GIT_SUBMODULE_PATHS to that one.

The runner still runs git submodule init over the whole .gitmodules, and one orphan gitlink without an entry failed every job with "No url found for submodule path". The fix was an entry in .gitmodules. The lint job passed on the second try.

Day three: the build VM, which I should not have built

lint runs anywhere. build needs a Docker daemon and a lot of disk. My first answer was a dedicated build VM with a shell runner, provisioned by terraform so the next person can rebuild it.

That took a day and a half, and it was the wrong day and a half. The cloud API returned 500 on every server create, so I made the VM in the portal and imported it into the terraform state. The runner installed fine. Then it could not register, because the firewall in front of the GitLab server dropped traffic from the VM's floating IP. That is a ticket to another team with no date on it.

I did get one thing out of the VM. With the runner unable to register, I copied the repo to the VM as a git bundle and ran the build job's exact script by hand, with the job's variables. That gave me the first real numbers.

The disk surprised me too. I had mounted a data volume and pointed Docker's data-root at it, and the boot disk filled anyway. Docker 29 defaults to the containerd image store, so layers live under /var/lib/containerd, which data-root does not move. The fix is root in /etc/containerd/config.toml, written before containerd.io is installed.

The caching articles tell you to enable the containerd image store for cache backends. They do not say it moves your layers to a different disk.

Day four: the pivot, and why the cache stays off

My manager looked at the firewall ticket and said what I should have asked on day two. Our CI jobs already run on Kubernetes. Use the privileged runner with a docker:27-dind service, the way the other pipelines do. If that works, the VM, the terraform module and the firewall ticket all go away.

I prototyped it in an afternoon. The job image is python:3.12, not docker:27-cli, because kolla is a Python program that drives Docker through the SDK and never calls the CLI. The wrapper script waits for the daemon with the SDK. It refuses a daemon with the vfs storage driver, which copies every layer and runs a full build out of disk, and it refuses a daemon whose architecture does not match the config.

1
N=0
2
until python3 -c "import docker; docker.from_env().info()" 2>/dev/null; do
3
  N=$((N + 1))
4
  [ "$N" -lt 30 ] || { echo "no Docker daemon at $DOCKER_HOST after $N tries"; exit 1; }
5
  sleep 2
6
done

Then the same full build on Kubernetes:

The per-image numbers are close on both hosts, yet the full build is faster on Kubernetes. The VM had 16 vCPUs and the pod had 8, so CPU does not explain it. Kolla builds with 8 threads, so wall time is set by the longest chain, not by total CPU.

My unmeasured guess is the disk. The VM wrote layers through Docker 29's containerd store onto a cloud volume, and the dind pod wrote through overlay2 on the node's local disk.

The longest chain has a name. skyline-console is a web dashboard whose Dockerfile runs an npm build, and it takes 37 minutes on its own.

The 159-image row came a day later, after neutron got a git source, and it changed the picture. The twelve neutron images add about 25 minutes of thread time, and the build grew by more than 20 minutes of wall time. The threads were the reason. Kolla hands its 8 threads out in tree order, so skyline-console waited behind 25 children of base, then behind the first wave of openstack-base children. In three green runs it started at minute 25, 26 and 34, and all 8 threads were busy for most of the time before that. The chain that sets the wall time is base, a wait for a thread, openstack-base, a 10 to 12-minute wait, then 38 minutes of npm.

The two config lines from day one made sense here, and they stay as they are.

The cache is on a disk that goes away with the job. The daemon, its layer store and every image live in the dind pod. When the job ends the pod is gone. --cache-from helps only if the cache source is pulled first.

Pulling the cache costs more than rebuilding it. The inline cache pattern is docker pull image || true before the build. For one image that is one pull. For a tree of 159 images, most of them around 2 GB (the cinder images are 2.2 GB each), the pulls would move more data than the build writes.

A registry cache with mode=max is worse. It stores every intermediate layer of every image, so the cache would be larger than the image set.

A change at the root invalidates everything. All 159 images descend from base and openstack-base. The reason to run a build is almost always a change there: a new upstream constraints file, a security update in Ubuntu, a patch to a shared library. When that happens the cache is cold for every image. It would hit only on runs where nothing important changed, and those are the runs we do not need.

The one image that dominates would rarely hit the cache. skyline-console pulls its source from an upstream branch, and its npm step sits on top of openstack-base. A change in base, in openstack-base or in that upstream branch rebuilds the 37-minute step, and one of those changes is the reason most builds run. If I had started the week with --cache-from, I would have made the 131-second base image faster and stopped there, while the 37-minute image kept the pipeline at an hour.

So every build is a clean build, and the pipeline prints how long it took.

Day five: the parts that made the pipeline usable

With the build running, the rest of the day went to things the caching articles do not mention.

The job log was too big to read. A full build with per-image output is a 3.95 MB job log. GitLab runners cut the log at 4 MiB by default, and the part that gets cut is kolla's summary at the end, the only part anyone reads. With --quiet and --logs-dir, the per-image logs go to an artifact and the job log is 48 KB.

The after_script runs a short script over kolla's summary JSON. It prints one line of the images that built and one of the images that failed, each marked as its own failure or its parent's, then the last 25 lines of each failed image's log. The same result goes to GitLab as a JUnit report, so the merge request widget and the Tests tab list all 159 images as passed or failed.

1
after_script:
2
  - echo "disk after the build $(df -h "$CI_PROJECT_DIR" | tail -1)"
3
  - tools/summary.py kolla-logs kolla-logs/junit.xml
4
artifacts:
5
  when: always
6
  paths: [kolla-logs/]
7
  reports:
8
    junit: kolla-logs/junit.xml
9
  expire_in: 7 days

Kolla's own summary is what the cutoff used to remove, and it is why the script exists:

1
INFO:kolla.common.utils:=========================
2
INFO:kolla.common.utils:Successfully built images
3
INFO:kolla.common.utils:=========================
4
INFO:kolla.common.utils:aodh-api
5
INFO:kolla.common.utils:aodh-base
6
...
7
INFO:kolla.common.utils:===========================
8
INFO:kolla.common.utils:Images that failed to build
9
INFO:kolla.common.utils:===========================
10
ERROR:kolla.common.utils:cinder-base Failed with status: error
11
ERROR:kolla.common.utils:cinder-api Failed with status: matched
12
ERROR:kolla.common.utils:neutron-base Failed with status: building
13
ERROR:kolla.common.utils:neutron-server Failed with status: matched

A matched failure is a child whose parent failed. Only the error and building lines name a real problem, so 22 failures were 3 causes. The script says that in one line instead of 22.

There is no `docker login` without a docker CLI. The SDK and kolla both read $DOCKER_CONFIG/config.json, so the script writes that file from the CI variables and proves the credentials through the daemon before any image builds. A push with wrong credentials now fails in 37 seconds at the login, not after a 40-minute build. I checked this with a deliberately wrong password, which is the only way to know a refusal works.

One build takes a third of a node. Each build writes 18 GB of layers, and the node had about 50 GB free. Three builds on the same node would run it out, and nothing in the runner stops that.

The job has resource_group: kolla-build, so builds queue one at a time. It is a blunt fix. The right fix is an ephemeral-storage request on the dind service, which our runner does not expose yet.

The first review comment on the merge request asked for a free-disk check before the build starts. The script now reads the daemon's root directory, measures the free space there, and refuses to build with less than 15 GB. A build that cannot finish should fail in the first second, not in minute 35.

A private fork needs the job token, and the token must not be written to disk. One image clones an internal fork of an OpenStack service. Git can read the URL rewrite from environment variables instead of a config file, so nothing with the token is written to the pod and the artifact cannot leak it:

1
before_script:
2
  - export GIT_CONFIG_COUNT=1
3
    GIT_CONFIG_KEY_0="url.https://gitlab-ci-token:${CI_JOB_TOKEN}@${CI_SERVER_HOST}/.insteadOf"
4
    GIT_CONFIG_VALUE_0="https://${CI_SERVER_HOST}/"

Privileged is the price of dind. Every merge request now gets a privileged pod with a daemon on an unauthenticated TCP port for the length of the job. The team had already accepted that for their other pipelines. I wrote down in the merge request what the pod exposes, with the option to restrict the runner to protected branches, and left the decision to my manager.

That is the first question a security review will ask, and no caching tutorial mentions it.

What the week left open

The pipeline built 137 images in under 45 minutes on infrastructure that already existed. At the end of the week 22 images still failed, all for reasons outside the pipeline: the neutron source that lived only on the old server, a fork that moved and renamed its branch, and two cinder patches written against an older release that no longer applied. Each became its own small merge request. All three have merged since, neutron builds from the upstream stable branch like every other service, and the first all-green run of 159 images came the day after the week ended. The VM and its terraform module are a draft nobody needs to merge.

The all-green run also moved the target. With 159 images the build takes 63 to 70 minutes, and the longest chain alone is about 46. The gap is thread order, not cache. The next change is a way to start skyline-console first, or a second job that builds it alone on another node, and either one can be measured against the baseline the week produced.

The layer cache is still an option. For merge request builds, where the root images rarely change, skip_existing per image with a pull of only base and openstack-base is a small change, and now there is a baseline to measure it against. That is the lesson of the week. Caching is an optimization, and I nearly spent the week on an optimization without a baseline.

If I could tell myself five things on Monday:

• Run the full build with no cache before anything else. Find the longest chain, the biggest image, and the minute at which it starts.
• A layer cache on an ephemeral runner is a pull, not a cache. Count the pull before you believe the speedup.
• The cache that matters for releases is at the registry. Promote a tested digest instead of rebuilding it.
• Ask what runners the team already has before provisioning one.
• Fix the log size before the first green run. A 4 MiB cutoff hides exactly the lines you need.