conndeck Download

conndeck blog

Forbidden: Debugging RBAC Without Handing Out cluster-admin

#kubernetes#rbac#security#debugging

It was 10 AM on a Thursday when the Slack DM arrived, and I could have written it myself: "the deploy job is getting Forbidden again, can you just give the CI account cluster-admin for an hour?"

The error was this, verbatim, in the pipeline log:

Error from server (Forbidden): deployments.apps "api" is forbidden:
User "system:serviceaccount:ci:deployer" cannot patch resource "deployments"
in API group "apps" in the namespace "staging"

No, you cannot have cluster-admin for an hour, because an hour is how credentials live forever. But the good news -- the news this post is about -- is that this error message is a complete specification of the fix. It took twenty minutes to resolve properly, and here's the whole method.

Quick answer: A Forbidden error names everything you need: the identity, the verb, the resource, the API group, and the namespace. Confirm the gap with kubectl auth can-i patch deployments -n staging --as=system:serviceaccount:ci:deployer, find which RoleBindings and ClusterRoleBindings reference that identity, and add the missing verb or resource to the right Role. If can-i still says no after your edit, check for the two classic bugs: a missing apiGroups entry in the rule, or a Role bound in the wrong namespace.

The error is a complete sentence

People skim Forbidden errors and miss that they're structured. Take it apart:

  • User system:serviceaccount:ci:deployer -- the identity. Every service account is system:serviceaccount:<namespace>:<name>.
  • verb patch -- what it tried to do. This is what your Role rule's verbs list must contain.
  • resource deployments -- what it tried to do it to.
  • API group apps -- which API family the resource lives in. "" means the core group (pods, services, secrets). This field is the silent killer; more below.
  • namespace staging -- where it tried.

Each part maps to a field in an RBAC rule. The fix for this exact error is a Role in staging with apiGroups: ["apps"], resources: ["deployments"], and patch in verbs. Nothing else. Not cluster-admin, not *, not "admin for a bit."

One caution: can-i evaluates the rules as they exist now. If the CI job's error is twenty minutes old, make sure someone hasn't already half-fixed it before you start editing.

kubectl auth can-i, your new favorite command

kubectl auth can-i asks the API server a yes/no question about authorization, and with --as it answers for any identity you're allowed to impersonate:

$ kubectl auth can-i patch deployments -n staging --as=system:serviceaccount:ci:deployer
no

no, confirmed -- the error is real and current. Now the two power moves. First, --list dumps everything an identity can do:

$ kubectl auth can-i --list -n staging --as=system:serviceaccount:ci:deployer
Resources                                       Non-Resource URLs   Resource Names   Verbs
deployments.apps                                []                  []               [get list watch create update]
configmaps                                      []                  []               [get list watch]
...

There it is in black and white: create and update, no patch. Second, test your fix *before* applying it by applying the Role and re-running can-i -- no waiting for the CI job to retry and fail again. Two flags worth adding to your kit: --as=<user> for human accounts, and --as-group=system:authenticated (or any group) when permissions come from group membership. Also note can-i answers about subresources too: kubectl auth can-i get pods/log is a different question than get pods.

Impersonation needs its own RBAC right (impersonate on users/serviceaccounts/groups), which cluster admins have. If --as itself returns Forbidden, that's the message telling you you're not an admin, which is a different afternoon.

Reading the bindings

can-i tells you what's missing; the bindings tell you where to add it. Find every binding that mentions the identity:

$ kubectl get rolebindings,clusterrolebindings -A -o wide | grep deployer
ci             deployer-staging       Role/deployer            45d
               deployer-read-cluster  ClusterRole/read-only    45d

Then read the Role itself, not from memory:

$ kubectl describe role deployer -n staging
Name:         deployer
Labels:       <none>
PolicyRule:
  Resources         Non-Resource URLs  Resource Names  Verbs
  ---------         -----------------  --------------  -----
  deployments.apps  []                 []              [get list watch create update]
  configmaps        []                 []              [get list watch]

The mental model that keeps this straight: Roles and ClusterRoles are sets of rules; RoleBindings and ClusterRoleBindings attach those rules to identities. The binding decides the scope. A ClusterRole attached with a *RoleBinding* only applies in that namespace -- this is the standard pattern for reusing a shared ClusterRole like view per-namespace. A Role referenced by a ClusterRoleBinding grants nothing at all, because a Role can't escape its namespace. And the built-in view, edit, and admin ClusterRoles carry aggregationRule labels, so if you've added rbac.authorization.k8s.io/aggregate-to-view: "true" to a custom ClusterRole, its rules silently merge into view -- check for that before wondering where permissions came from.

The fix that Thursday was one verb on the existing Role:

rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]

Re-ran can-i, got yes, told the deploy job to retry, denied the cluster-admin request with love.

The traps that make "it looks right" lie

Missing apiGroups. A rule with resources: ["deployments"] and no apiGroups only matches the core group, where deployments don't exist. It grants nothing and no one gets an error. Always write apiGroups: ["apps"] for deployments, ["batch"] for jobs, [""] for pods and secrets. kubectl api-resources shows the group for anything if you're unsure.

Wrong namespace. Roles and RoleBindings are namespaced; a RoleBinding in ci grants rights in ci, full stop. Binding in the wrong namespace is the number one reason a "correct" Role does nothing.

resourceNames and verbs you didn't expect. Rules scoped with resourceNames only apply to those named objects. And controllers need verbs humans forget: watch and list for anything that reconciles, create on pods/eviction for draining, update on leases for leader election.

I keep an eye on which service account each workload actually runs as -- full disclosure, conndeck surfaces that alongside the pod, which has saved me the "wait, which identity is this even using" detour more than once.

Frequently asked questions

How do I debug an RBAC Forbidden error in Kubernetes?

Read the error as a complete sentence: it names the user or service account, the verb, the resource, the API group, and the namespace. Then run kubectl auth can-i with --as set to that identity to confirm what it can and cannot do, and inspect the RoleBindings and ClusterRoleBindings that reference it. The fix is adding the missing verb or resource to a Role, not granting cluster-admin.

What does kubectl auth can-i do?

It asks the API server whether a given action is allowed, and it works for yourself or, with the --as flag, for any identity you are allowed to impersonate. For example, kubectl auth can-i list pods --as system:serviceaccount:ci:deployer -n staging tells you whether that service account can list pods in staging. It evaluates the real RBAC rules, so it is the fastest way to test a fix before applying it.

How do I check what permissions a service account has?

Use kubectl auth can-i --list --as system:serviceaccount:namespace:name to dump everything the account may do, optionally scoped to a namespace. For the source of those permissions, search bindings with kubectl get rolebindings,clusterrolebindings -A and grep for the service account name, then read the referenced Roles and ClusterRoles.

What is the difference between a Role and a ClusterRole?

A Role grants permissions inside one namespace only, while a ClusterRole defines cluster-wide or cross-namespace rules. The binding decides the scope: a ClusterRole bound with a RoleBinding applies only in that one namespace, while the same ClusterRole bound with a ClusterRoleBinding applies everywhere. A Role bound with a ClusterRoleBinding grants nothing at all, which surprises everyone once.

Why does my RBAC rule not work even though the Role looks right?

The two classic bugs are a missing or wrong apiGroups field and a scope mismatch. Resources like deployments live in the apps API group, so a rule that omits apiGroups only covers the core group and silently grants nothing for deployments. The other is binding a Role in the wrong namespace, because Roles and RoleBindings are namespaced and only apply where they live.

What actually matters

Forbidden is the most informative error in Kubernetes: identity, verb, resource, group, namespace, all in one line. Decode it, confirm with can-i --as, find the binding, add exactly the missing rule.

The cluster-admin request will come again next week. The answer stays no -- and with this loop taking twenty minutes, "no" stops being a bottleneck and starts being what it should be: the default.