Skip to content

CI is a service nobody is on call for

A dark wooded hill at dusk with a fire-lookout tower on its top, a single white lamp on its roof and the cabin unlit. Cool white house lights dot the valley on the left, and a tiny figure carrying an orange lantern climbs a dirt path toward the tower.

At work I'm an SRE. Every service I run has a latency target, a dashboard, and someone who gets paged when it drifts. CI has none of that, even though every change waits on it. Coding agents now open pull requests on my repos most days, so I went over nine of my own pipelines the way I'd go over a slow service. This post is the set of rules I ended up with. Each one comes with the case that taught it, and the post ends with the prompts I give agents so you can run the same audit on your repos.

Why CI drifts

A service gets slower and someone notices, because a graph turns red and a person owns that graph. A pipeline gets slower and nothing happens. The job is still green, and green is the only signal most of us look at.

So pipelines drift one reasonable line at a time. Someone adds a step for a real need that day. Nobody prices it again. Months later a job takes four minutes, and every line in it still has a good reason in its commit message.

Agents speed this up from both sides. They open more pull requests, so every slow step runs more often. And they add steps the way people do, at the end of the job, for a reason that's fair that day.

The rules below are how I stop that. None of them is specific to GitHub Actions or to my repos.

1. Find where the time goes before you fix anything

When an API gets slow, the first question is not how to make it faster. It is where the time goes. A CI job splits the same way a request trace does:

  • Wait: the job is queued and no runner has picked it up
  • Setup: checkout, toolchains, installs, cache restores
  • Run: the command the job exists for

Each part has a different fix. Wait is capacity, so it needs more or bigger runners. Setup is overhead every run pays again, so it needs deletion or caching. Run is the work itself, so it needs parallelism or a faster test.

On all nine of my repos, wait was 2 to 3 s at the median. A bigger runner would have changed nothing. Every slow job I found was slow in setup or in the order of its steps.

You don't need a tool for this. This script reads the last successful runs of a workflow from the GitHub API and ranks its steps by average duration:

steps.sh
#!/usr/bin/env bash
# usage: steps.sh owner/repo workflow.yml [runs]
set -euo pipefail
repo=$1 workflow=$2 runs=${3:-20}
runs_url="repos/$repo/actions/workflows/$workflow/runs"
gh api "$runs_url?status=success&per_page=$runs" --jq '.workflow_runs[].id' |
while read -r id; do
gh api "repos/$repo/actions/runs/$id/jobs" --jq '
.jobs[] | .name as $job | .steps[]
| select(.started_at and .completed_at)
| [$job, .name,
((.completed_at | fromdate) - (.started_at | fromdate))]
| @tsv'
done |
awk -F'\t' '{ key = $1 " / " $2; sum[key] += $3; n[key]++ }
END { for (k in sum) printf "%6.1f s %s\n", sum[k] / n[k], k }' |
sort -rn | head -10

The top line is where to look first.

2. Every step needs a reason that is still true

Once you know which step is slow, ask the question from an incident review. Why is this step here, and is that reason still true?

On logixlysia, a logging plugin of mine, the job that renders a terminal recording for the docs took about 257 s. Three job logs put 185 to 195 s of that inside one option, install-fonts: true. It was in the workflow's first version, so the recording would have a font. What it did was clone the whole Google Fonts repository on every run. The tool already installed the one font the recording used, by default. Four later edits to the same file left that line alone. Deleting it brought the step to 80 s.

The fix was one line. Finding it took asking why the line was there.

3. A cache is a bet, so price it

A cache bets that restoring is cheaper than redoing the work. Most workflows add one by habit and never check whether the bet pays.

On docker-doctor, a Dockerfile linter of mine, every job restored a 1.2 GB dependency cache. Over 682 jobs, the restore took 29 s at the median and the install after it took 2 s. With the cache gone, a cold install took 4 to 8 s, and the Code Quality workflow went from 72 s to 35 s.

A cache can also lose the other way. On sshpick, a Rust CLI, I changed how dependencies compile in tests. The cache key didn't include that setting, so every run restored the old build and then rebuilt everything. Lint went from 2 s to 41 s, with CI green the whole time.

Measure both sides, with the cache and without it, before you keep one. Then check what its key covers.

4. Order comes from dependencies, not from when you wrote it

Steps in a job run in the order someone wrote them. That's usually the order they were added, not the order they need.

On afterhours, a macOS app of mine, CI started with only the build. Two days later the tests arrived and went into the same job, because that's where new steps go. The tests took 44 s and the build 126 s, one after the other, and neither needed the other. As two jobs that start together, the time to a result went from 195 s to 109 s on the first run.

At work I'd call that two independent calls made in sequence. The fix is the same in both places.

5. Green has to mean the change ran

A passing job means the script reached its end. It doesn't mean the thing you care about happened.

When I split the afterhours job, the first push came back green, and neither new job had run. CI decides which apps a change affects, and a change to the workflow file didn't count as affecting anything. So CI skipped the jobs it was changing and reported a pass.

At work I've seen the same thing three more ways. An exit code meant to show a warning arrived as a plain failure on one runner type. A step that posted results got a 401, and curl without -f exited 0. A merge request gate built pipelines with zero jobs for about three months, because of a rule about which jobs a child pipeline keeps.

So for any change to CI itself, check that the job you changed ran, and read what it produced. Don't stop at the colour.

6. Red that means nothing teaches people to ignore red

The opposite failure is just as bad. On five of my repos, the release bot's pull request shows a red or blocked mark with no job behind it, 20 to 26 of every 100 runs on three of them. On another, a summary job that gates merges failed 54 times, and in 50 of them nothing had failed. A newer push had cancelled a sibling job.

A reader, human or agent, who learns that red usually means nothing will miss the red that matters. Every red mark should point at a real failure, or it should go.

7. Bound the worst case before you speed up the average

On one repo, cancelling superseded runs saved about 21 job-minutes in six weeks. In the same six weeks, one job hung for 6 hours until I cancelled it by hand. That one job was 41% of all job-minutes in the window.

At work we put a timeout on every network call. My CI jobs had none. A timeout of about twice the slowest good run would have saved 17 times what the cancelling did. Set the cap first, then tune the average.

8. A retry should never turn a real bug green

At work, two tests failed in ways that looked flaky. They turned out to be one bug. A proxy closed idle connections at exactly 10.0 s, and the client polled every 10 s, so some connections died and some didn't.

A change that retried the whole test on any error would have made the nightly run green. We closed it, because the retry would have hidden this bug and every future one behind the same green. Retry a step only when you can name the transient failure it covers.

9. The agent that wrote the change doesn't get to approve it

Agents are good at the fast part of this work. They can measure a hundred runs, find the slow step, and write the fix in minutes. They're bad at the same thing people are bad at, which is believing their own change worked because CI is green.

So I split the roles. One agent audits, a second writes the change, and a third, which saw none of the reasoning, reads the diff and the real CI run and gives a verdict. On the five pull requests from this audit, the verifier caught two mistakes that green CI didn't. One was a redundant install step I had added to the font fix. The other was the sshpick cache miss above, which would have made every run on main slower.

Prompts you can give your agent

These are the three prompts I use, cut down to what works with any coding agent. Run them in separate sessions, so the verifier doesn't share the writer's context.

The audit prompt is read-only, and "change nothing" is an allowed answer:

audit.md
Audit the CI of this repository. Do not change any files.
1. Pull the last 100 runs of each workflow from the CI provider's API.
Split each job's time into wait (queued), setup (checkout, installs,
caches) and run (the job's real command). Report p50 and p95 for each,
with the run count.
2. For the slowest steps, find the commit that added each one and say why
it was added. Say whether that reason is still true, with evidence.
3. For every cache, compare restore time with a cold install.
4. Count red or cancelled runs that had no real failure behind them.
5. List jobs with no timeout.
Propose at most four changes. For each one, give the baseline number, the
expected effect labelled as a guess, and the result that would make us
revert it. If nothing beats run-to-run noise, say so and propose nothing.

The change prompt keeps one change per pull request:

change.md
Make change N from the audit, and only that change, in its own pull request.
In the description, state the baseline with its run count, the expected
effect labelled as a guess, and the result that would make us revert it.
If the change touches a workflow file, confirm in the run logs that every
job it changes actually ran.

The verify prompt goes to a fresh session that didn't write the change:

verify.md
You did not write this pull request. Verify it.
Read the diff, then read the pull request's actual CI run: the job list,
the logs and any artifacts. Check that every changed job ran, that its
output matches the last good run on main, and that no cache or skip rule
hides work. Read the source of any action or tool the change relies on,
not the pull request description.
Reply with one line, "Verdict: PASS", "PASS+NOTES" or "FAIL", then the
evidence for it.

What changed on my repos

RepoRuleBefore, p50First run after
logixlysiaA reason that's still trueRender step about 257 s80 s
docker-doctorPrice the cacheCode Quality 72 s35 s
afterhoursOrder from dependencies195 s to a result109 s

The pull requests are logixlysia #638, docker-doctor #239 and afterhours #100. The "after" column is one run each, and a single fast run proves little. Each pull request names the number it has to reach over its next 10 to 15 runs, and I'll update this table when those runs exist. The other six repos were already fast, with 22 to 122 s to a result, and four of them got no change at all.

Start with one owner

You don't need a platform team for any of this. Pick one person, or one scheduled agent, to own CI the way someone owns a service. Once a month, they look at where the time goes, ask why the slowest step is there, and check what green means after every change to CI itself. A pipeline drifts when nobody is looking, and catching it takes one regular look.