OpenStack core, from a lab that broke on day two

I started as a junior SRE at a cloud provider on a Monday. Day one was accounts, names and a DevStack all-in-one to work through the Certified OpenStack Administrator prep lab. Day two, the lab broke in four places, and each break exposed one seam between two core services. Working software hides its structure. This one was broken in enough places that the structure came out.

Some context on where I was standing. I came in from the software side. My interviewer said so, kindly, and the mental models I brought were Kubernetes and Terraform, not Nova and Neutron. I had read the OpenStack service map the weekend before and could recite the boot path from a diagram. I had never watched it fail.

The environment was DevStack on stable/2025.1 (Epoxy), with Ceph behind Glance, Nova and Cinder instead of the default local disk and LVM, and Ceph RadosGW standing in for Swift. On paper, everything was there. openstack endpoint list showed every service. ceph -s said HEALTH_OK for two days straight. It was also the one machine I was allowed to break, which on day two is the most valuable thing a new hire can be given.

Then I tried to boot a server, create a volume, upload an object and launch a stack. Each one failed differently, and each failure pointed at a boundary between two services that I had only seen as an arrow.

What is supposed to happen

The lab manual draws the request path once and expects you to be able to retell it in ninety days. This is the version I can retell after two, because I have seen each hop refuse.

Keystone issues a token scoped to a project and hands back the service catalog, the list of every endpoint in the cloud. Nova asks Placement which hosts still have room, picks one, and tells the compute agent on that host to build. The agent pulls the image from Glance, asks Neutron for a port, asks Cinder for a volume if the request had one, and hands the result to libvirt.

Every arrow in that diagram is a place where one service trusts what another one told it. The rest of this post is about four of those arrows.

Nova does not ask the host. The host reports.

The first failure was NoValidHost on a machine with CPU and RAM to spare. The obvious reading is that the flavor was too big. It was not. The number that mattered came from a different command:

1
openstack hypervisor stats show
2
# count: 0

Zero hypervisors. The compute process was running, and the scheduler still could not see a host, because the scheduler never talks to hosts. nova-compute reports its inventory to Placement about once a minute, and the scheduler reads Placement. A compute node that cannot report does not exist as far as scheduling is concerned, no matter how healthy the process looks.

1
nova-compute → (reports every ~60s) → Placement ← (asks) ← nova-scheduler

It is the same shape as a kubelet reporting to the API server. A node goes NotReady while you are still logged into it over SSH. So when the scheduler says there is no room, the first question is not whether there is enough. It is what the system knows about at all.

Reporting CPU required a working storage backend

Why could the compute node not report? The resource tracker crashed every cycle. The call chain in the traceback was the most useful thing I read all week:

1
update_available_resource()
2
  └─ _get_local_gb_info()
3
       └─ RBDDriver().get_pool_info()   ← needs to reach Ceph

The inventory report includes local disk. With images_type = rbd, local disk comes from the Ceph pool, so the report cannot be assembled until the RBD driver has talked to the cluster. Storage being unreachable took compute off the map, with every vCPU and gigabyte of RAM on the host sitting idle.

Ceph was unreachable for a reason that had nothing to do with Ceph being down. The client library on the host was 17.2.9 and the cluster was 20.2.4. Every keyring on the machine, including the monitor's own, failed with Malformed input. The keys were fine. The old client could not read them.

Nobody designed the coupling between a CPU report and a storage client version. It fell out of one number in the report depending on something outside the process. That is worth remembering the next time I write a health check: if this dependency goes away, does my check say unhealthy, or does it say nothing at all?

The workaround was to set images_type = qcow2 in the compute config so the tracker could finish a cycle. It is not a fix. It is a way to keep learning while the real fix waits on a re-provision, and on day two, knowing which of those two things you are doing is most of the job.

There are two layers of up, and they disagree

Volumes went straight to error. The process was fine. The service was not.

1
systemctl status devstack@c-vol      # active (running), 23 hours
2
openstack volume service list        # down, last heartbeat 23 hours ago

Cinder did this on purpose. Its log said so: Manager ... is reporting problems, not sending heartbeat. Service will appear "down". The volume manager could not reach its backend, so it stopped claiming to be ready. That is the right design. It also opens a gap between OS-level liveness and application-level readiness that a process monitor will never see.

Nova, hit by the same Ceph problem, crashed and restarted in a loop. That was easier to notice. A service that keeps running and quietly does nothing is worse than one that dies, and the only way to catch it is to watch the heartbeat the service itself publishes, not the process the OS reports.

The catalog is a registry, not a health check

Every object storage command returned 405. Not 404, not 503, but method not allowed, which reads like a wrong command. The command was fine. The endpoint in the catalog was missing its account path. That is how the Ceph plugin registers RadosGW, as a bare /swift/v1, and the step that adds the account path was meant to come later.

The Swift endpoint is stored as a template, not a URL: http://<rgw>:8080/swift/v1/AUTH_$(project_id)s. Keystone substitutes the project id when it issues a token, so one endpoint serves every tenant. That is how OpenStack does multi-tenancy at the catalog layer, and I would never have read that line if the endpoint had been correct.

Two requests that differ only in that path made the case:

1
TOKEN=$(openstack token issue -f value -c id)
2
PROJECT=$(openstack token issue -f value -c project_id)
3

4
curl -s -o /dev/null -w "%{http_code}\n" -H "X-Auth-Token: $TOKEN" \
5
  "http://<rgw>:8080/swift/v1/AUTH_$PROJECT"   # 204
6
curl -s -o /dev/null -w "%{http_code}\n" -H "X-Auth-Token: $TOKEN" \
7
  "http://<rgw>:8080/swift/v1"                 # 405

The fix is an endpoint set, not a delete and create, so there is no window where the endpoint is missing and the change can be reverted:

1
URL='http://<rgw>:8080/swift/v1/AUTH_$(project_id)s'
2

3
for IFACE in public admin internal; do
4
  openstack endpoint set --url "$URL" \
5
    $(openstack endpoint list --service swift --interface $IFACE -f value -c ID)
6
done
7

8
openstack container create test-fix && openstack container list

The single quotes matter. Without them the shell expands $(project_id) as a command substitution. Keystone is the one that fills it in, at runtime.

Heat was the same lesson from the other side. The catalog advertised it. The source, the config, the Apache vhost and the endpoint all existed. There was no database and no process, so every request bounced between 503 and 500. Registration is not readiness. A Kubernetes Service with an empty Endpoints list is the same object.

And ceph -s reporting HEALTH_OK all along was the same thing again. HEALTH_OK means the OSDs are in, the placement groups are clean and the monitors have quorum. It is the cluster checking itself. It says nothing about whether any client on this host can connect. A health signal measured from the wrong side measures nothing.

One line in a provisioning script

All of it had a single root. The cloud-init script that built the machine ran stack.sh, and stack.sh exited early inside the Ceph plugin:

1
Inferring config /etc/ceph/ceph.conf
2
ERROR: fsid does not match ceph.conf
3
/opt/stack/devstack-plugin-ceph/devstack/lib/cephadm:set_memory_config:1  exit_trap
4
+./stack.sh:exit_trap:578    exit 1

The fsid in ceph.conf matched the running cluster, so the message looked wrong. It was not. The plugin generates a fresh fsid with uuidgen on every run and skips the bootstrap when a monitor already exists, so every cephadm shell --fsid call after that compares a new id against the real one. The plugin does not cope with a host that already has a Ceph cluster, and this host had one before the script reached that line. That also means the OpenStack I was using did not come from this run: the plugin's install phase sits before DevStack creates a service database or starts an API. Where the cluster and the services came from, the log does not say.

What the log does say is that every line after ./stack.sh in the provisioning script never ran: the Ubuntu image upload, the RadosGW Keystone settings, the Swift endpoint correction, the extra CLI tools. The script contained the exact endpoint fix I ended up applying by hand. It just never got there.

If the script had died on its first line, someone would have noticed. It died in the middle, so it produced a machine that looked finished. Horizon loaded. The catalog was full. Nothing was verified after deploy, and nothing had to be, because nothing complained until it was used. A partial failure is more dangerous than a total one.

The twin lesson is about retries. stack.sh is a shell script with no transaction boundary and no notion of converging on a desired state. Running it again to repair the machine made things worse. Before it even reaches the plugin it restarts RabbitMQ and rewrites the service configs, and past the plugin it drops and recreates the service databases. The Glance API on this machine did not survive the attempt. A retry is only safe when the operation is idempotent. This one was not. I learned that by doing it, which is the expensive way.

What a new hire is allowed to touch

The hardest question on day two was not technical. It was which of these I should fix myself, which I should work around, and which I should hand to someone who has been here longer than 48 hours. I ended up with three buckets, and I wrote the report to the team in those terms.

The proposals I attached were the ones the table points at: re-provision the machine so the script runs to the end, add a smoke test after deploy so a half-built lab never gets handed to the next person, put a warning in the manual not to rerun stack.sh, and ship a second Cinder backend in the template so the volume labs survive a Ceph problem. None of that is mine to decide. All of it is mine to write down clearly.

The one thing that was not OpenStack

Once compute was back, instances hung during boot and never reached SSH. This looked like more of the same. It was not, and the way to prove it was to take OpenStack out of the experiment entirely: same image, same QEMU binary, same host, one flag changed.

While hung, the guest was at 100% CPU with around 107,000 interrupts per second against a normal 1,000 to 5,000, the instruction pointer parked at one address, interrupts enabled, and zero steal time. An interrupt storm inside the guest kernel, under nested virtualization on the outer hypervisor. A different team's problem, with a different fix.

There was a decoy. The compute config had cpu_model = Nehalem on an AMD host, which is wrong on its face and sits right next to the symptom. I was sure that was it. The A/B test said no. A wrong config next to the failure is the classic trap, and being willing to drop my own hypothesis when the evidence did not fit mattered more than guessing right the first time.

The workaround for the lab was virt_type = qemu and cpu_mode = none. Slow, but it boots.

What I check first now

What day two taught

The manual's goal for this lab was to be able to retell the request path without opening the docs. I can, and the reason is not the diagram. It is that I watched Placement lose a host, Cinder stop its heartbeat, Keystone hand out an endpoint that did not work, and Heat answer from behind a catalog entry with nothing behind it.

Each core service has its own idea of ready, and none of them is the process being alive. Nova's is an inventory report that landed in Placement. Cinder's is a heartbeat it will withhold on purpose. Keystone's catalog has no idea at all. For someone whose job title has reliability in it, that is the whole lesson: the SLI is never the backend saying it is healthy. It is the volume being created, from the customer's side, within the time we promised.

I had a list of first-week questions for the team before I started. After day two, three of them got sharper, because they came from something I had touched: how do we know a compute node in production is still reporting inventory, do our alerts watch processes or heartbeats, and is a new node verified before it is trusted, or deployed and believed. I would rather ask those on day three than discover them on call.