conndeck Download

conndeck blog

Your Pod Is Stuck in Pending: The Actual Checklist

#kubernetes#debugging#scheduling#troubleshooting

It was 7:50 on a Monday morning, coffee still too hot to drink, when the pipeline declared victory and nothing came up.

The new batch service had merged Friday evening. CI was green, the GitOps controller said synced and healthy. But kubectl get pods -n etl showed three replicas, all Pending, zero restarts, sitting there like they had nowhere to be. Which, in a sense, they didn't. The cluster had twelve healthy nodes, most of them barely busy. And the scheduler wanted nothing to do with this pod.

Quick answer: A Pending pod means the scheduler could not find a node for it, and the reason is always in the pod's events. Run kubectl describe pod <name> and read the FailedScheduling line: it tells you how many nodes were rejected and why -- insufficient CPU or memory, untolerated taints, failed node affinity, or unbound PersistentVolumeClaims. Every fix starts from that line. If there are no events at all and no node assigned, check whether the pod is waiting on a PVC to bind or on a custom scheduler that isn't running.

describe first, everything else second

The single most important command for a Pending pod:

$ kubectl describe pod batch-runner-6f8c9d7b5-q2w9k -n etl
...
Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  FailedScheduling  4m12s (x5 over 9m)    default-scheduler  0/12 nodes are available: 12 Insufficient cpu. preemption: 0/12 nodes are available: 12 No preemption victims found for incoming pod.

That one line is the entire case file. Twelve nodes, twelve rejected for insufficient CPU. Not "the cluster is busy" in some vague sense -- the pod's CPU request does not fit anywhere, full stop. And note what the scheduler is measuring: requests, not usage. Your dashboard showing the fleet at 15 percent utilization is irrelevant. Scheduling happens against what pods have reserved, not what they're doing.

The preemption bit matters too. The scheduler checked whether evicting lower-priority pods would make room and found nothing worth evicting. If your pod has no priorityClassName, that line is normal and you can ignore it.

Decoding "0/N nodes are available"

The sentence after the colon is a per-reason tally across your nodes. The shapes I see most often:

  • 12 Insufficient cpu -- the request is bigger than any node's free allocatable CPU. Shrink the request or add capacity.
  • 3 Insufficient memory, 9 untolerated taint node-role.kubernetes.io/control-plane:NoSchedule -- read tallies together. Nine of your nodes are control plane and the three real workers are out of memory.
  • 12 didn't match Pod's node affinity/selector -- your nodeSelector or required affinity excludes the entire fleet.
  • 12 node(s) had untolerated taint {dedicated: gpu} -- a tainted pool you don't have a toleration for.
  • pod has unbound immediate PersistentVolumeClaims -- a storage problem, not a scheduling problem. See below.

One thing people misread: the reasons are tallies, and they don't need to sum to N. "8 Insufficient memory, 4 untolerated taint" doesn't mean four tainted nodes are the real issue. It means every node failed for at least one of those reasons. Fix the one that applies to the nodes you actually intend to run on.

Requests vs allocatable: the arithmetic that bites

Back to that Monday. The pod asked for cpu: 2 and memory: 4Gi. Reasonable-sounding. Here's what the nodes looked like:

$ kubectl describe node worker-04 | grep -A8 "Allocated resources"
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                7150m (89%)   12200m (152%)
  memory             11Gi (70%)    18Gi (115%)

Eight cores allocatable, 7150 millicores already reserved by other pods and the DaemonSet tax. Free headroom: 850m. My pod wanted 2000m. It was never going to fit, on this node or any of its eleven twins.

The author had picked cpu: 2 by vibes -- "it's a batch job, give it some room." Nobody had ever checked what the job actually used (about 400m at peak, it turned out). The fix was a request of 500m with a limit of 2, which scheduled instantly and ran fine. Right-sizing requests isn't a cost-optimization hobby; it's the difference between scheduling and Pending.

Taints, tolerations, and the affinity trap

Taints are how nodes say no. The control-plane taint (node-role.kubernetes.io/control-plane:NoSchedule) is the famous one, but the ones that bite people are self-inflicted: dedicated=gpu:NoSchedule, team=platform:NoSchedule, taints left over from a maintenance window nobody removed. The fix on the pod side is a toleration:

tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "etl"
    effect: "NoSchedule"

Node affinity is the mirror image: the pod saying where it wants to go. requiredDuringSchedulingIgnoredDuringExecution is a hard gate -- if nothing matches, the pod pends forever, and the scheduler will not soften the rule for you. I've seen a nodeSelector: {disktype: ssd} survive three cluster migrations into a fleet where the label was now disk: nvme. Twelve nodes, zero matches, one very patient pod.

The silent cases: PVCs and custom schedulers

Two causes produce Pending pods with unhelpful or missing scheduling events.

First, storage. If the pod mounts a PVC that can't bind, you'll see pod has unbound immediate PersistentVolumeClaims. Check the claim itself:

$ kubectl get pvc -n etl
NAME           STATUS    VOLUME   CAPACITY   STORAGECLASS   AGE
work-scratch   Pending                         local-ssd      42m

The nastiest variant is volumeBindingMode: WaitForFirstConsumer with topology-aware provisioning. The volume can only be created in the zone your pod lands in, the scheduler knows this, and if no node in the right zone has room, everything waits on everything. The PVC's own events (kubectl describe pvc) usually name the stuck topology.

Second, the pod spec's schedulerName. If someone set it to a batch scheduler that isn't running, the default scheduler ignores the pod entirely and you get no FailedScheduling events at all. Silence is itself a clue.

This is the class of problem where a live view of Pending pods across namespaces, with their events attached, saves real time -- which, full disclosure, is one of the reasons we built conndeck. But kubectl describe plus the discipline to read the PVC too would have caught every case above.

The checklist, condensed

  1. kubectl describe pod -- read the FailedScheduling event to the end.
  2. If it's Insufficient cpu/memory: check kubectl describe node Allocated vs Allocatable, then fix requests or add nodes.
  3. If it's taints: add a toleration or remove the taint, deliberately.
  4. If it's affinity: kubectl get nodes --show-labels and find the label that doesn't exist.
  5. If it's a PVC: kubectl describe pvc and follow the storage class.
  6. If there are no events: check schedulerName in the pod spec and whether that scheduler exists.

Frequently asked questions

Why is my Kubernetes pod stuck in Pending?

A Pending pod is one the scheduler could not place on any node, and the reason is always in the events. Run kubectl describe pod and read the FailedScheduling line: it tells you exactly how many nodes failed and why, whether that is insufficient CPU or memory, untolerated taints, failed node affinity, or an unbound PersistentVolumeClaim. Guessing before reading that line is how people lose an hour.

How do I fix 0/5 nodes are available?

The text after the colon is the diagnosis. Insufficient cpu or memory means the pod's requests do not fit on any node's allocatable capacity, so lower the requests or add bigger nodes. Untolerated taint means you need a matching toleration or the taint removed. Didn't match node affinity means your nodeSelector or affinity rules exclude everything. Each reason has a different fix, which is why the full event text matters.

What is the difference between taints and node affinity?

Taints live on nodes and repel pods that lack a matching toleration, pushing workloads away. Node affinity lives on the pod and pulls it toward nodes with matching labels. Taints are how you protect nodes, like the control plane or a dedicated GPU pool, and affinity is how a workload asks for specific hardware or topology. A Pending pod can be caused by either, and the FailedScheduling event names which one.

Why does my pod stay Pending when the nodes look underutilized?

Because the scheduler looks at requested resources, not actual usage. A node at 10 percent real utilization can still be full for scheduling purposes if existing pods have reserved most of the allocatable CPU and memory via requests. Check kubectl describe node and compare the Allocated resources table against Allocatable, not against what kubectl top shows.

Can a PVC keep a pod in Pending?

Yes. If the pod mounts a PersistentVolumeClaim that cannot bind, the scheduler refuses to place it and you will see pod has unbound immediate PersistentVolumeClaims in the events. With topology-aware storage it gets subtler: the volume can only provision in one zone, and if no node in that zone fits the pod, it waits forever. Check the claim with kubectl get pvc and read its events too.

What I'd tell a teammate

Pending is the politest failure mode Kubernetes has: nothing is crashing, nothing is on fire, and the scheduler is telling you exactly what it wants in the events. Read the FailedScheduling line to the end, remember that requests are not usage, and check the PVC before you start resizing nodes.

And if the events are completely silent, the pod isn't being rejected. It's being ignored. Go find out who was supposed to be listening.