conndeck Download

conndeck blog

Service Account Tokens: The Quiet Security Hole in Every Cluster

#kubernetes#security#service-accounts#workload-identity

At 9:15 on a Sunday morning, a junior engineer on our team ran a demo pod from a tutorial, exec'd into it, and ran cat /var/run/secrets/kubernetes.io/serviceaccount/token. Then, following the same tutorial, he used that token to list every pod in the cluster. From inside a container that had no business knowing the cluster existed.

Nobody did anything wrong. That's the point. He followed a popular tutorial, the cluster behaved exactly as designed, and the design is that every pod gets a credential to talk to the Kubernetes API unless you tell it not to. Multiply that default by every container you'll ever run, including the one that gets popped by a dependency vulnerability next year, and you have the quietest attack surface in your cluster. Not exotic, not zero-day -- just a token sitting in a well-known path, waiting.

Quick answer: Every pod gets a service account token mounted at /var/run/secrets/kubernetes.io/serviceaccount/token by default, and any code execution in that pod becomes Kubernetes API access with that token's RBAC permissions. Fix it in layers: set automountServiceAccountToken: false on the default ServiceAccount (or pod spec) for everything that doesn't call the API, rely on bound service account tokens -- the short-lived, audience-scoped projected tokens that are default since 1.21 -- for the ones that do, and never put a Kubernetes token in front of a cloud API where EKS IRSA, GKE Workload Identity, or Azure Workload Identity belongs. Then audit what's still mounting tokens it doesn't need.

The default is the problem

Here's what every pod carries unless you say otherwise:

$ kubectl exec deploy/demo -- ls /var/run/secrets/kubernetes.io/serviceaccount/
ca.crt
namespace
token
$ kubectl exec deploy/demo -- sh -c 'curl -s -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt https://kubernetes.default.svc/api/v1/namespaces/default/pods | head -c 200'
{"kind":"PodList","apiVersion":"v1","metadata":{"resourceVersion":"18443392"}...

That token is a JWT the API server trusts. What it can *do* is determined by the RBAC bound to the pod's service account -- and the default service account has no permissions, which is why everyone sleeps at night. The trouble starts with the service accounts that do have permissions: the CI deployer that can patch deployments, the controller that can read secrets, the Helm chart that helpfully binds its app to edit. Each of those tokens is mounted into every pod using that account, readable by any process in the container, including the one an attacker just got.

Turn off the mount

The fix for the vast majority of workloads is one line:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  namespace: payments
automountServiceAccountToken: false

Set it on the default ServiceAccount in each namespace and every pod that doesn't opt in stops getting a token. The workloads that genuinely call the Kubernetes API -- controllers, CI jobs, cert-manager-style integrations -- opt back in on their pod spec with automountServiceAccountToken: true on a dedicated service account with scoped RBAC. Pod spec wins over ServiceAccount if both are set, which is a footgun worth remembering when you're debugging why a workload still has a token.

The honest cost: something will break. A monitoring agent that talks to the API, a sidecar you forgot about. Roll this out per namespace, watch for CrashLoopBackOffs with 401 or Forbidden in the logs, and opt in what legitimately needs it. I keep a live pod view open during these rollouts -- conndeck, in my case -- because the failures show up as a visible wave of restarts and you want to catch the pattern in minutes, not from a ticket tomorrow.

Bound tokens changed the game

If your mental model of service account tokens is "a Secret containing a JWT that never expires," it's a decade out of date. The legacy tokens were exactly that: auto-generated Secrets, non-expiring, valid forever, still working after the pod and even the service account were deleted unless someone cleaned up. Kubernetes stopped auto-creating them in 1.24 for good reason.

Modern pods get bound service account tokens via the TokenRequest API, projected as a volume:

  • Short-lived -- default one hour, rotated automatically by the kubelet.
  • Audience-bound -- the token is only valid for the audience it was issued for, so a token minted for a cloud OIDC exchange won't authenticate to the Kubernetes API.
  • Pod-bound -- it's invalidated when the pod is deleted. Stolen token, dead pod, dead token.

You can see the projection in any modern pod spec's status, and request one explicitly when you need a custom audience:

volumes:
  - name: cloud-identity
    projected:
      sources:
        - serviceAccountToken:
            path: token
            audience: sts.amazonaws.com
            expirationSeconds: 3600

Bound tokens don't eliminate token theft -- an attacker inside a running pod still has a working credential for up to an hour -- but they turn a permanent compromise into a one-hour, single-audience, pod-scoped one. That's the difference between an incident and a bullet point.

Cloud APIs are a different problem

The most common misuse I see: a pod needs S3 access, so someone creates an IAM user, dumps the keys into a Kubernetes Secret, and mounts it. Now you have static cloud credentials in etcd, in backups, in every kubectl get secret -o yaml someone runs while screen-sharing. Or worse, the node instance profile gets broad IAM so pods "just work," and every pod on the node inherits it.

Workload identity exists precisely to kill both patterns. EKS IRSA, GKE Workload Identity, and Azure Workload Identity all do the same trick: the cluster is an OIDC issuer, a Kubernetes service account is annotated to map to a cloud IAM role, and the pod exchanges its bound token for short-lived cloud credentials via the cloud STS. On EKS:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: ledger-api
  namespace: payments
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/ledger-api-s3-read

No keys, no Secrets, credentials that expire in an hour and are scoped to exactly what that one service needs. The setup is an afternoon per cloud provider the first time. The audit finding it prevents is forever.

Frequently asked questions

Why are service account tokens a security risk in Kubernetes?

Every pod gets a service account token mounted by default, whether the workload needs one or not, so any code execution in any container becomes API access. The token's permissions are whatever the service account's RBAC grants, which is often more than people realize, and a stolen token works from anywhere the API server is reachable. The fix is disabling the default automount for pods that don't need API access, which is most of them.

How do I disable automatic service account token mounting?

Set automountServiceAccountToken to false on either the ServiceAccount or the pod spec, with the pod spec winning if both are set. Most teams set it on the default ServiceAccount in each namespace, or enforce it with an admission policy, and then explicitly opt in the workloads that actually call the Kubernetes API. Expect a few controllers and CI jobs to need the opt-in; normal web apps almost never do.

What are bound service account tokens and why are they better?

Bound tokens come from the TokenRequest API and are projected into the pod as a volume with a configurable audience and a short lifetime, usually an hour, and they're invalidated when the pod is deleted. They replaced the old legacy tokens, which were non-expiring Secrets that lived forever and kept working even after the pod was gone. Kubernetes has projected bound tokens by default since 1.21 and stopped auto-creating legacy token Secrets entirely in 1.24.

Should my application use a Kubernetes service account token to call cloud APIs?

No, use your cloud's workload identity instead: EKS IRSA, GKE Workload Identity, or Azure AD Workload Identity. These map a Kubernetes service account to a cloud IAM role through OIDC, so the pod exchanges its bound token for short-lived cloud credentials with no static secrets anywhere. Putting cloud keys in Secrets, or giving the node instance role broad permissions so pods inherit them, are both things you will eventually explain to an auditor.

How do I find pods still mounting service account tokens they don't need?

Query for pods where automountServiceAccountToken is not explicitly false and the mounted service account has non-trivial RBAC, or simply run kubectl get pods across namespaces and check the volumes for a kube-api-access projected token. In practice, audit the ServiceAccounts first: list their RoleBindings and ClusterRoleBindings, because a token bound to a service account with no permissions is low-risk, while a token bound to one that can read secrets cluster-wide is your incident report waiting for a timestamp.

The parting advice

Service account tokens are a default that made sense when clusters were small and trusting, and they're a liability at any real scale. Disable the automount, keep bound tokens for the workloads that need them, and let workload identity handle the cloud.

None of this is glamorous. That's rather the point: the quiet holes are the ones that stay open.