React diffs the DOM. Kubernetes diffs reality. A mental model for the loop that lets systems repair themselves.
You push to main. The webhook fires, a build kicks off, and somewhere between two retries and a load balancer the event evaporates. Nothing is technically broken. The pipeline is waiting for a message that will never come, and nothing in the system will ever check again.
I've been studying how modern control planes avoid this failure mode, mostly by digging into Atlasflow and the Kubernetes controller model it builds on. This post is the mental model I've assembled along the way: what reconciliation actually is, why it works, and the sharp edges I found once I looked closely. I haven't operated one of these systems in production yet, so treat this as a well-researched map rather than a war story.
Most systems react to events
Most backend systems are event-driven in the plainest sense: something happens, a message fires, a handler runs. Git push → webhook → deploy. Payment received → webhook → subscription updated. It maps onto how we naturally think about cause and effect, which is most of why we keep building things this way.
The trouble is that every arrow in that chain is a place where reality can quietly diverge from what the user expects:
- The webhook never arrives.
- The handler crashes halfway through.
- The event is delivered twice.
- A timeout fires after the database commit but before the acknowledgment.
Each of these leaves the system in a state nobody is responsible for noticing. The event already happened. Nobody checks again.
What should exist right now?
Reconciliation flips the question. Instead of asking what just happened?, it asks what should exist right now?
Every resource gets two representations: the desired state (3 replicas, version v2) and the actual state (whatever is really running). A controller's entire job is to compare the two and, when they differ, take one small step to close the gap. Then it looks again.
That's the whole idea.
The most boring loop in software
Here's the entire pattern:
for {
desired := loadDesiredState()
actual := observeActualState()
if !matches(desired, actual) {
takeOneStep(desired, actual)
}
sleep(interval)
}No deployment state machine. No BUILDING → UPLOADING → STARTING → READY. No resume logic. If the process crashes right after creating an instance, the next iteration simply observes that reality is still one instance short and creates another.
Recovery here isn't a separate feature; it's just what the loop does on every pass. Taking one step per iteration, instead of executing a whole plan, also matters more than it looks. A plan can be interrupted halfway and leave you guessing where it stopped. A loop that re-derives its next action from current state has nothing to resume.
The same property handles rollouts for free. Change the desired version and the loop sees a different kind of drift — instances running the wrong version — and replaces them one at a time.
You already know this pattern
If you write frontend code, you've been using a reconciler for years. A React component tree declares the desired UI, the DOM is the actual state, and React diffs the two and applies the smallest set of mutations that closes the gap. It's even in the name: the React docs call this phase reconciliation.
Infrastructure as code is the same shape. terraform plan is the diff, apply is the corrective step, and the state file is a cache of “actual”. That cache is also why things drift the moment someone clicks around the console. Security tooling converges on the pattern too: a policy engine like OPA Gatekeeper doesn't just reject bad resources at admission, it periodically re-audits everything already running, because an audit that fires once and never looks again has the same failure mode as a lost webhook.
Kubernetes didn't invent the idea, and there's nothing container-shaped about it. The requirement is only that desired state is declarable and actual state is observable. Anywhere that holds (DNS records against a registrar, cron fleets, entitlements in a billing system) a reconciler can keep the two together.
Events become an optimization
This is my favorite property of the pattern. In an event-driven system, the event is both the trigger and the truth: lose it and nothing happens, ever. In a reconciliation system, events are only a latency optimization.
A notification wakes the controller so it reconciles now instead of at the next periodic sweep. If the notification is lost, the sweep still observes that desired and actual disagree, and fixes it. Correctness no longer depends on delivery. Only latency does.
The parts that aren't boring
Everything above is the brochure. Once I started reading actual controller code, I kept running into the same three sharp edges, and the pattern only works if you respect them.
Every step must be idempotent
The loop retries by design, which means every corrective action will eventually run twice. “Create one instance” is only safe because an extra instance is cheap to detect and undo on the next iteration. If the corrective action is “charge the customer”, running it twice is an incident.
That's the honest version of the billing example everyone reaches for. Reconciliation is a great fit for subscription state — should this customer have access right now? — because re-applying an entitlement is harmless. It's a terrible fit for retrying money movement unless every attempt carries an idempotency key that the payment provider actually enforces.
Your view of “actual” is stale
The loop's one structural weakness is the observeActualState() line. A freshly created instance takes time to boot, and if the reconciler can't see in-flight work, it will happily create another one every tick until the first ones finally land, then spend the next several ticks walking the excess back down. This is why the Kubernetes ReplicaSet controller tracks “expectations”: a note-to-self that says I already created one; don't call it drift until it lands.
// naive: only counts what it can already see
actual := len(running)
// expectation-aware: counts work still in flight
actual := len(running) + len(pendingCreates)The sweep interval is a real trade-off
The periodic sweep is what guarantees correctness, and it isn't free. Sweep every second and you hammer whatever API serves actual state; sweep every ten minutes and a lost event means up to ten minutes of drift. That window is a product decision, not an implementation detail. And at scale, thousands of resources sweeping on the same schedule become a thundering herd — real systems jitter their sweeps for a reason.
If I were building a control plane
This is the design I'd start from today, assembled from the systems I've been studying rather than from my own scars:
Desired state lives in Postgres, and LISTEN/NOTIFY wakes workers the moment it changes. That's the fast path. The slow path is a periodic sweep that re-enqueues anything that drifted, which is what makes lost notifications irrelevant. Workers take one idempotent step each and append what they did to an event log for replay and auditing.
One thing I had to untangle: earlier I claimed reconciliation removes the deployment state machine, and now I've drawn a queue and a log. The difference is what the machinery is for. The queue exists so work survives a crash; the log exists so humans can see what happened. Neither holds workflow state that correctness depends on — you could lose both, and the next sweep would still converge. The state machine worth avoiding is the one where being correct requires remembering which step you were on.
Reality should eventually match what was declared.
Boring survives production
The more I study distributed systems, the more reconciliation's appeal looks like a deliberate aesthetic. There's no clever orchestration and no attempt to enumerate every failure mode, just one loop, applied uniformly, that treats every failure the same way: as drift.
I haven't earned the production war stories yet. But I understand now why the people who have keep reaching for this pattern — not because it's clever, but because it's boring. Boring survives contact with production.