My read-only role passed Cinder's policy and saw an empty cloud

I was asked to build a role that can read every project in an OpenStack cloud and change nothing. Cinder answered it with 200 OK and an empty list. Cross-project reads in Cinder are decided by a single is_admin flag in three places, and the usual workaround gives the reader delete rights on every project. This is the mechanism, the patch I sent upstream to move the decision into policy, and the smaller copy of the same bug I found in my own patch.

The request was one sentence. Make a role that can read every project in the cloud and change nothing, so the people who audit and debug do not need admin to look around. Keystone created the role in a minute, and Cinder's policy file accepted it for the list APIs. Then Cinder answered 200 OK and returned the reader's own empty project.

This post follows that empty list down through Cinder, from the API to the database. There are three gates. Policy owns none of them. The obvious workaround hands the reader delete rights on every project. The patch I sent upstream moves the decision into policy, and on the way it turned out to have a smaller version of the same bug.

200 OK and nobody else's volumes

The cleanest reproduction is a fresh DevStack on Cinder master. One project admin with a volume admin-vol, one project repro with reader-vol, and a user reader whose only role is cloud_reader on repro. DevStack turns on the new policy defaults, so a custom role first needs one override before it can list at all. After that, this is what the two users see:

1
# as admin
2
openstack volume list --all-projects
3
# admin-vol, reader-vol
4

5
# as reader, cloud_reader only
6
openstack volume list --all-projects
7
# reader-vol
8
openstack volume show admin-vol
9
# 404

No 403 and no warning. The reader asked for every project and got a correct, complete-looking answer for one of them. A staging cloud with real tenants behaved the same way: the same request returned 200 and zero rows while admin saw every project. If you build dashboards or audit jobs on a role like this, they will report an empty cloud and look healthy doing it.

Policy said yes

The first guess is that policy denied the request. It did not. volume:get_all passed for cloud_reader, which is why the answer was 200 and not 403. In Cinder, that rule decides whether you may call the list API at all. It says nothing about how wide the list is allowed to be.

To find out what does decide the width, I ran Cinder 26.3.0 (2025.1) inside its own unit-test base, with a real sqlite schema, the real policy engine and the staging cloud's policy file, then followed one request through every layer. The width is decided three times, and never by a policy rule.

Three gates, and policy owns none of them

The first gate is in the filter validation. reject_invalid_filters in cinder/api/common.py lets an admin through with any filter. For everyone else it checks the query against the configured filters, and all_tenants is never one of them. It would normally be a 400, except for one special case:

1
if 'all_tenants' in invalid_filters:
2
    # NOTE: this is a special case: the cinderclient always adds
3
    # 'all_tenants', so we don't want to hold that against a non-admin
4
    # user and we silently ignore it.  See Bug #1917574.
5
    invalid_filters.remove('all_tenants')
6
    filters.pop('all_tenants')

That special case is not careless. python-cinderclient looks a resource up by name with all_tenants=1 on every call (bug 1917574), so refusing the parameter would break cinder show <name> for every non-admin user. Dropping it quietly is the reason the reader got an answer at all. It is also the reason the answer looked complete.

The second gate is the one that actually picks the branch. In volume/api.py, get_all has a single condition:

1
if context.is_admin and allTenants:
2
    # Need to remove all_tenants to pass the filtering below.
3
    del filters['all_tenants']
4
    volumes = objects.VolumeList.get_all(context, marker, limit, ...)
5
else:
6
    volumes = objects.VolumeList.get_all_by_project(
7
        context, context.project_id, marker, limit, ...)

The same context.is_admin and all_tenants shape repeats in the volume summary, the resource count, the snapshot list, the backup list, and in transfers, groups, group snapshots and attachments. I counted nine sites on master.

The third gate sits under both. db.volume_get_all and the other cross-project listers carry @require_admin_context. A non-admin context that somehow reached them would get AdminRequired. A patch that only fixes the branch in the API layer gets past the second gate and fails at this one.

All three read the same boolean. RequestContext computes is_admin once, at construction, from the policy rule context_is_admin, whose default is role:admin. So the width of every list in Cinder comes down to one question asked once per request: is this caller an administrator?

The one-line fix makes the reader an admin

Once you know that, the workaround is obvious. Add the reader role to context_is_admin:

1
"context_is_admin": "role:admin or role:cloud_reader"

In the test harness this works on the first try. The reader lists all three volumes. Then I asked the policy engine whether the same reader may delete a volume in a project it does not belong to, and the answer was True. Every rule written as is_admin:True or rule:admin_api now passes for it. The role I was asked to build was read-only. The workaround builds an administrator with a different name.

This has been known for a long time. Bug 1522264, filed in 2015, describes the same symptom and was still unassigned. A 2022 blueprint proposed per-resource policies for exactly this and never started. Nova solved its version years ago with rules like os_compute_api:servers:detail:get_all_tenants, which let an operator grant the wide read without granting anything else.

Moving the decision into policy

The change on Gerrit takes the Nova shape and keeps everything else in Cinder where it is. It adds three rules, each defaulting to rule:admin_api, so a deployment that does nothing sees no change:

1
"volume:get_all:all_tenants": "rule:admin_api or role:cloud_reader"
2
"volume:get_all_snapshots:all_tenants": "rule:admin_api or role:cloud_reader"
3
"backup:get_all:all_tenants": "rule:admin_api or role:cloud_reader"

A new RequestContext.read_scope() replaces the is_admin branch at the list, count and summary sites. It consumes all_tenants, checks the rule, and only when the rule passes returns an elevated copy of the context for the database read. The @require_admin_context gate stays exactly as it was. The caller's own is_admin is never touched, so nothing downstream of the read starts treating the reader as an admin.

Three decisions in the patch are about what it leaves alone.

• A caller the rule denies still gets its own project, not a 403. The cinderclient name lookup sends all_tenants=1 on every call, and a 403 would break it for every non-admin user.
• Showing another project's volume, snapshot or backup goes through the same rule as a one-row list. The shared getters volume_api.get(), get_snapshot() and backup_api.get() are not widened, because write paths fetch their target through them. Some of those actions, backup:create among them, still carry deprecated rules with an empty check string, so widening the getter would let a granted reader copy another project's data into its own.
• Transfers, groups, group snapshots and attachments keep the is_admin check for now. They are a second, smaller change, so the first one reads as one idea.

On the same DevStack, as the same reader, with nothing else changed:

The second row matters as much as the third. The patch does nothing until an operator grants the rule. The last column is the property the workaround breaks: the reader can now see admin-vol, and a delete still fails because volume:delete never heard of the new rule.

My patch had a smaller version of the same bug

While writing the second change, for groups and the other three resources, I found a problem in the first one. The detailed volume view decides whether to show a volume as part of a consistency group, and to do that it looks the group up. It looked it up with the caller's context. The group lookup is project-scoped, so as soon as the granted reader listed every project and one of the other project's volumes sat in a group, the whole list turned into a 404.

It is the same mistake as the original bug, one layer up. The list was widened by policy, and one helper underneath still answered the old question about whose project this is. I wrote the test first, test_granted_role_lists_another_projects_volume_in_a_group, watched it fail on the first patch set, then changed the view to skip the consistency-group alias when the group is not readable. That went up as patch set 2 of the same change.

Sending it upstream

I commented on the 2015 bug with a reproduction against 2025.1, put Closes-Bug: #1522264 in the commit, pushed it to Gerrit, and added four Cinder reviewers. Zuul ran 34 jobs and voted Verified +1. A few vendor CI systems failed, which on Cinder is common and has nothing to do with the change. The gate that matters is a core reviewer, and a week later nobody had voted on it.

The change came up at the Cinder team meeting on 4 November, on a topic the chair, a core reviewer, opened for it. The answer was neither yes nor no. Secure RBAC coverage in cinder-tempest-plugin is still incomplete, and several of the patches that would add it pass the gate only because they skip the tests that fail. The chair's first reading was that my change cleans out is_admin checks in the database layer, and nobody wants to remove those before tests can show they were unnecessary.

That fear is the right one, and it is the reason the third gate stayed where it was. The change touches nothing in cinder/db, and a unit test asserts that the database listers still raise AdminRequired for the reader. Once that was clear the ask got smaller, but it did not go away. The chair still wants to hold the change until the test debt moves, and said openly that this is unfair to me. They have yet to read the patch closely, but if the database layer holds up, good progress on the most closely related Tempest tests would be enough.

So the next work is not in the patch. Adding finer-grained policies is close enough to an API change that it needs a spec, a full one since Cinder has no spec-lite, though it can be short, and specs have to be approved by the freeze on 20 November. The tests have a softer target, milestone 2 in mid-January, and the chair said that can flex. I said I would work on the related tests, and I was told more than once that I do not have to fix all of it alone. At the time of writing the change still has no code review vote.

The second change, for transfers, groups, group snapshots and attachments, is ready and passes on DevStack. It waits for the first one, because a reviewer who disagrees with read_scope should only have to say so once.

Sharp edges

• A denied caller still gets a quiet, partial answer. That is deliberate, and it means you cannot tell from the response alone whether a role was granted the wide read. Test it with a project you know is empty.
• DevStack enables the new policy defaults. A custom role with no override fails volume:get_all outright with a 403, which looks like a different bug from the one in production.
• Keep the DB gate. I compared three designs for this patch, and all three kept @require_admin_context and satisfied it with an elevated copy of the context after the rule passed. Removing the gate would leave the next caller one forgotten policy check away from a cross-project read.
• The patch covers volumes, snapshots and backups. Until the second change lands, a reader still sees only its own transfers, groups, group snapshots and attachments.
• Ask about the spec before you write the patch. I asked after the change was up, and new policy rules turned out to need one. Cinder specs have to be approved by the freeze, so a late question can cost you the release.

Why this is bigger than one role

A read-only persona is only as real as the narrowest service in the path. Keystone will mint any role you ask for, and a policy file will happily grant it the read APIs, but a single is_admin check in code decides for every service that still has one. Before building a reader role across a cloud, it is worth a grep for is_admin next to all_tenants in each service you run. Every hit is a place where read and write are still the same permission.

For me the useful part was not the patch. It was learning that 200 OK with fewer rows than you expected is a failure mode of its own, one that no alert watching status codes will ever catch.