conndeck blog

Pod Security Standards: Getting to restricted Without a Revolt

On a Monday morning in March, I opened a ticket titled "Enable restricted PSS cluster-wide by Friday" and laughed out loud. Alone. In my kitchen.

The ticket came from our security team, who had just read a benchmark report and quite reasonably decided that our clusters should enforce the restricted Pod Security Standard. What they had not done was look at what was actually running. I knew, before I ran a single command, that roughly a third of our namespaces had at least one workload running as root, and that our observability vendor's agent ran privileged on every node. A Friday deadline for that is not a plan. It's an incident with a calendar invite.

We got to restricted eventually. It took nine weeks, zero Sev-1s, and one genuinely tense meeting with the observability vendor's account team. This is the playbook that came out of it.

Quick answer: Pod Security Standards are three built-in policy levels -- privileged, baseline, restricted -- enforced by the Pod Security Admission controller via namespace labels. To roll out restricted without breaking anything, label namespaces with pod-security.kubernetes.io/enforce=baseline but audit=restricted and warn=restricted first, mine the audit log for violations, fix or exempt the offenders, and only then flip enforce to restricted one namespace at a time. Never enforce a level you haven't audited first, and put an owner and an expiry date on every exemption.

What the three levels actually mean

The standards are short documents and worth reading once, but here's the working summary.

Privileged is a no-op. Everything is allowed. It exists so system components -- your CNI DaemonSet, your storage driver, your monitoring agent that insists on hostPID -- have somewhere legal to live.

Baseline blocks the known privilege-escalation paths: no hostPID, hostIPC, or hostNetwork, no privileged containers, no hostPath volumes, no adding capabilities beyond a small default set, no host ports. Most normal web apps and workers pass baseline unchanged. This is why baseline is the sensible default enforcement level for an existing cluster.

Restricted is the hardened target. On top of baseline it requires, per container: allowPrivilegeEscalation: false, runAsNonRoot: true (at pod or container level), capabilities.drop: ["ALL"] with at most NET_BIND_SERVICE added back, and seccompProfile set to RuntimeDefault or Localhost. Volume types narrow to configMap, secret, projected, downwardAPI, ephemeral, emptyDir, and PVCs. This is where the friction lives, because an enormous amount of software in the world assumes it can be root.

Audit mode is the whole trick

The single most important thing about Pod Security Admission is that enforcement is per-namespace, per-mode, and independent. Three labels, three behaviors:

kubectl label --overwrite ns payments \
  pod-security.kubernetes.io/enforce=baseline \
  pod-security.kubernetes.io/audit=restricted \
  pod-security.kubernetes.io/warn=restricted

enforce rejects violating pods at admission. audit records violations in the API server audit log but lets them through. warn prints a warning to whoever runs kubectl but lets them through. Set enforce to the level you can sustain today and audit/warn to the level you want tomorrow. Now every kubectl apply against a non-compliant namespace tells the developer exactly what's wrong:

Warning: would violate PodSecurity "restricted:latest":
  allowPrivilegeEscalation != false (container "api" must set securityContext.allowPrivilegeEscalation=false),
  runAsNonRoot != true (pod or container "api" must set securityContext.runAsNonRoot=true)

That warning is your migration tool. Developers see it in their own terminals, in their own words, weeks before anything breaks. The audit log, meanwhile, is your project tracker: grep it for pod-security.kubernetes.io/audit-violations annotations and count by namespace. When a namespace goes two weeks without a violation, it's ready for enforce=restricted.

One flag worth knowing: kubectl label ns foo pod-security.kubernetes.io/enforce=restricted --dry-run=server -o yaml runs the label change through admission dry-run and tells you which existing pods in the namespace violate the policy, before you commit to anything.

Fixing workloads once, not forty times

Most violations are the same four missing fields. A compliant pod looks like this:

spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: api
      image: registry.example.com/api:v2.14.3
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop: ["ALL"]

If your organization has forty teams with forty deployment templates, do not file forty tickets asking each team to paste this in. Fix it in the shared base template if you have one, or mutate it in at admission with Kyverno or your policy engine of choice, and reserve human tickets for the genuinely hard cases: apps that actually need a capability, images that can't run as an arbitrary UID, and vendors who ship YAML that belongs in a museum.

The runAsNonRoot failures deserve a special mention. The field only *asserts* that the container won't run as root -- it doesn't choose a UID. If the image's Dockerfile says USER root, the pod will now fail to start with container has runAsNonRoot and image will run as root, which is the policy working as designed but reads like a new bug. The fix is a non-root user in the image, or an explicit runAsUser in the pod spec for images you don't control. This is the class of thing where being able to see failing pods and their events across namespaces in one view saves real hours; I keep conndeck open on exactly this screen during PSS migrations, because the violations cluster and you want to spot the pattern, not triage pods one by one.

Exemptions: necessary, dangerous, temporary

Some workloads will never comply. The observability agent needs hostPID. The storage driver needs hostPath. You have three options, in increasing order of bluntness:

  1. Exempt the workload's *namespace* from the check entirely via the admission configuration file's exemptions block, which supports usernames, runtimeClassNames, and namespaces. This requires API server flags (--admission-control-config-file), so on managed clusters you may not have it.
  2. Label the workload's namespace pod-security.kubernetes.io/enforce=privileged. Simple, works everywhere, and the standard escape hatch for kube-system-adjacent tooling.
  3. Move the workload somewhere it belongs, like the vendor's own dedicated namespace, and stop letting it share a namespace with your apps.

Whatever you pick: write the exemption down with an owner and a review date. An exemption without an expiry is a permanent hole that someone will discover during an audit in 2028 and ask you about in front of your VP.

Frequently asked questions

What are the three Pod Security Standards in Kubernetes?

Privileged, baseline, and restricted. Privileged allows basically everything and exists for system-level workloads like CNI plugins and storage drivers. Baseline blocks the known privilege-escalation paths like hostPID, hostNetwork, hostPath volumes, and privileged containers while still letting most normal apps run. Restricted is the hardened target: it additionally requires runAsNonRoot, a seccomp profile, allowPrivilegeEscalation set to false, and dropping all Linux capabilities.

How do I roll out the restricted Pod Security Standard without breaking workloads?

Label every namespace with enforce at baseline but audit and warn at restricted, then let it bake for a couple of weeks. The audit annotations land in the API server audit log and the warnings show up directly in kubectl output, so you can build a list of non-compliant namespaces and fix them before anyone feels pain. Only flip enforce to restricted once a namespace has gone quiet in the audit log.

What is the difference between baseline and restricted Pod Security Standards?

Baseline prevents known privilege escalations but tolerates typical app defaults, so most workloads pass unchanged. Restricted enforces current pod-hardening best practice: containers must not run as root, must drop all capabilities except optionally NET_BIND_SERVICE, must set allowPrivilegeEscalation to false, and must have a seccomp profile set. Restricted also narrows allowed volume types to things like configMap, secret, projected, and ephemeral.

How do I exempt a namespace or workload from Pod Security Standards?

The blunt option is labeling the namespace with enforce set to privileged, which opts it out entirely. The better option is the exemptions section of the Pod Security admission configuration file on the API server, which can exempt specific usernames, runtime class names, or namespaces from checks altogether. Whichever you use, keep an owner and an expiry date on every exemption or it will outlive the person who requested it.

Why do my pods fail with allowPrivilegeEscalation errors after enabling restricted?

Because restricted requires every container to explicitly set allowPrivilegeEscalation to false, and omitting the field counts as a violation rather than defaulting safely. The same applies to runAsNonRoot, dropping ALL capabilities, and setting a seccompProfile. Most teams fix this once in their base manifest template or with a mutating policy in Kyverno rather than editing every deployment by hand.

The short version

Pod Security Standards are not hard technically. They're hard socially, because enforce-first turns a security improvement into a breaking change with your name on it. Audit-first flips that: the warnings do your advocacy for you, the audit log does your project management for you, and by the time you enforce anything, nothing happens.

Nine weeks, in our case. The Friday deadline was never mentioned again.