conndeck Download

conndeck blog

PodDisruptionBudgets: The Availability Feature Everyone Skips

#kubernetes#reliability#pdb#operations

The night our cluster upgrade turned a three-replica service into a zero-replica outage, we had 99.95% uptime targets, an on-call rotation, a status page, and not a single PodDisruptionBudget in the entire fleet.

It was a Tuesday, around 1 AM, which is when managed node upgrades happen if you're brave or haven't read the docs. Three nodes drained in sequence, as designed. Our inventory service ran three replicas -- one per node, thanks to a spread constraint someone had added and everyone had forgotten. The drain controller asked the eviction API politely each time, the eviction API checked for PDBs, found none, shrugged, and evicted. For nine glorious minutes, inventory had zero running pods while new ones waited for nodes that were still being replaced. Every order placed in that window got a 500. The upgrade worked perfectly. The application did not.

The fix was eleven lines of YAML that should have existed on day one. Let's talk about the feature everyone skips.

Quick answer: A PodDisruptionBudget limits how many pods of an app can be down during *voluntary* disruptions -- drains, node upgrades, autoscaler scale-downs. Set maxUnavailable: 1 (or a minAvailable floor) on every production workload, and the eviction API will refuse to remove pods faster than your budget allows. PDBs don't protect against node crashes, only API-driven removals. The failure modes to know: no PDB at all (drains take you to zero), and a PDB with minAvailable equal to your replica count (drains hang forever and block upgrades).

Voluntary vs involuntary: the line that matters

Pods disappear for two families of reasons:

  • Involuntary: hardware failure, kernel panic, OOM, the cloud reclaiming a spot instance. No API call, no warning, no budget. Kubernetes finds out after the fact.
  • Voluntary: kubectl drain, a managed control plane or node pool upgrade, cluster autoscaler scale-down, a VPA eviction. These all go through the eviction API, and the eviction API checks PDBs.

That distinction is the entire mental model. A PDB is a contract between your workload and whoever drains nodes. It cannot save you from physics; it can only make automation behave like a considerate colleague.

Here's the minimal version that would have saved our Tuesday:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: inventory-pdb
  namespace: shop
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: inventory

With this in place, that upgrade night goes differently: node one drains (two pods left, budget allows one down), node two's drain *waits* until the replacement pod is Running and Ready, and at no point does inventory drop below two replicas. Slower upgrade, zero 500s. Correct trade.

minAvailable vs maxUnavailable, and the status that matters

You pick exactly one knob:

  • minAvailable: 2 -- never go below two healthy pods. An absolute floor; you must remember to bump it when you scale up.
  • maxUnavailable: 1 -- never have more than one down. Scales naturally as replicas grow. Percentages work too: maxUnavailable: 25%.

For most services I default to maxUnavailable: 1: simple, safe, grows with the deployment. For big fleets (20+ replicas), a percentage avoids serializing drains into molasses.

The part everyone forgets to read is the PDB's status:

$ kubectl get pdb -n shop
NAME            MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
inventory-pdb   N/A             1                 1                     40d

ALLOWED DISRUPTIONS is the live answer to "can anything be evicted right now?" It's computed as healthy pods minus the floor. If it says 0 -- because a pod has been NotReady for an hour, or because you scaled down -- *every* drain touching this app will stall, even though the configuration looks fine. Check the status, not just the spec:

$ kubectl describe pdb inventory-pdb -n shop
...
Status:
  Current Healthy:    2
  Desired Healthy:    2
  Disruptions Allowed: 0
  Expected Pods:       3

Three expected, two healthy, zero disruptions allowed. That's a deployment with one sick pod quietly turning its own PDB into a wall. The drain isn't broken; the app is.

The blocking problem: when PDBs hold upgrades hostage

The same mechanism that saves you from drains can freeze them. The two classic configurations:

minAvailable equal to replicas. Three replicas, minAvailable: 3 means allowed disruptions is permanently zero. Every eviction is refused, kubectl drain retries forever, managed upgrades time out, and the autoscaler can't remove nodes. You've built an upgrade-immovable object. I see this in the wild constantly, usually written by someone who thought "minAvailable should obviously be all of them."

Sick pods eating the budget. As above: one pod crash-looping means the remaining pods are unevictable. Fixing the drain means fixing the app, which is correct behavior, but maddening if you don't know to look at Disruptions Allowed.

The drain output tells on itself:

$ kubectl drain worker-7 --ignore-daemonsets --delete-emptydir-data
...
error when evicting pods/"inventory-6d8f7b9c4-h2kxm" -n "shop" (will retry after 5s):
Cannot evict pod as it would violate the pod's disruption budget.

When you see that, the debugging order is: kubectl get pdb -A for anything with zero allowed disruptions, then figure out whether the zero is a bad config (floor too high) or a sick workload (healthy count too low). One is a YAML fix, the other is a real incident wearing a drain costume.

Worth knowing: hard kubelet evictions under node memory pressure don't consult PDBs at all. PDBs govern API-driven removals, full stop.

A production checklist that actually holds up

After enough upgrades, my PDB opinions have calcified:

  • Every production workload with 2+ replicas gets a PDB. One replica gets none -- a PDB on a single-replica deployment is either useless or a drain blocker, depending on the numbers.
  • Default to maxUnavailable: 1. Percentages for large fleets, minAvailable only when the floor is genuinely fixed.
  • The selector must match the pod labels exactly. A PDB whose selector matches nothing shows Expected Pods: 0 and protects nobody while looking reassuring in Git. Verify with kubectl get pdb after applying, every time.
  • Alert on Disruptions Allowed: 0 for more than a few minutes. It catches both misconfiguration and sick apps before the upgrade window does.
  • Audit before upgrade windows. Ten minutes with kubectl get pdb -A beats a stalled node pool at 2 AM.
  • Remember the consumers. Cluster autoscaler scale-down, VPA's updater, and your managed provider's upgrade pipeline all go through the same eviction API. A PDB that blocks drains blocks all of them.

When an upgrade stalls, being able to see pod health, drain progress, and which pods are pinned in one live view -- the sort of thing we built conndeck for -- turns a mystery into a list. But kubectl get pdb -A and the discipline to read the ALLOWED DISRUPTIONS column is most of the battle.

Frequently asked questions

What does a PodDisruptionBudget actually do?

A PDB tells Kubernetes the minimum number of pods for an app that must stay available during voluntary disruptions like drains, upgrades, and cluster autoscaler scale-downs. The eviction API checks the PDB before removing a pod and refuses if eviction would drop you below the budget. It does nothing about involuntary failures like node crashes, which don't ask permission.

What is the difference between minAvailable and maxUnavailable?

minAvailable is an absolute floor: keep at least this many pods running. maxUnavailable is a ceiling on concurrent disruption: at most this many can be down. You set one or the other, never both. For a 3-replica service, maxUnavailable: 1 and minAvailable: 2 are the same statement, but maxUnavailable: 1 scales as replicas grow while minAvailable: 2 does not.

Why is kubectl drain stuck on my node?

Almost always a PodDisruptionBudget. Drain evicts pods through the eviction API, and if every remaining eviction would violate a PDB, the drain hangs retrying. Run the drain and look for cannot evict pod errors naming the PDB. The common causes are a PDB with minAvailable equal to the replica count, or unhealthy pods elsewhere already consuming the entire disruption budget.

Can a PodDisruptionBudget block a cluster upgrade?

Yes. Managed Kubernetes upgrades and cluster autoscaler scale-downs drain nodes through the eviction API, which respects PDBs, so a misconfigured PDB can stall an upgrade indefinitely. The upgrade tooling typically times out and reports eviction failures. Audit PDBs before upgrade windows, especially any with minAvailable equal to replicas or pods that have been NotReady for a long time.

Do PodDisruptionBudgets protect against node failures?

No. PDBs only gate voluntary disruptions that go through the eviction API: drains, upgrades, autoscaler removals. A node that dies takes its pods with it regardless of any PDB, because hardware doesn't call the API server first. PDBs complement replica counts; they don't replace the need for enough replicas to absorb a node loss.

Where this leaves you

PDBs are the contract that turns node drains from a coin flip into a negotiated process. No PDB means automation can take you to zero; a floor equal to your replica count means automation can never move at all. maxUnavailable: 1 is the boring, correct default for nearly everything.

Write the eleven lines now, verify the selector matches, alert on zero allowed disruptions, and audit before upgrades. The next 1 AM node pool rotation will be a non-event, which is exactly what good infrastructure looks like.