conndeck Download

conndeck blog

PVC Stuck in Pending: Every Reason and the Fix

#kubernetes#storage#troubleshooting#pvc

It was a Friday, 4:40 PM, twenty minutes before a demo, when the new environment refused to come up. Every pod in the namespace sat in Pending, every PVC sat in Pending, and kubectl describe on the claim said, with total serenity: waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually. The provisioner was installed. It was running. It was doing absolutely nothing.

The fix took ninety seconds once I understood what the claim was actually telling me. The understanding took longer. This post is the understanding.

Quick answer: A Pending PVC means the claim hasn't been bound to a volume, and the Events section of kubectl describe pvc names the reason in almost every case. The five usual causes: WaitForFirstConsumer delaying provisioning until a pod schedules (normal, not a bug), the CSI provisioner missing or broken, a StorageClass with bad parameters or missing cloud permissions, topology conflicts where the volume and pod want different zones, or quota and capacity exhaustion. Read the events first, fix the named cause, and provisioning retries on its own.

The one command that answers it

Everything starts here:

$ kubectl describe pvc data -n demo
Name:          data
Namespace:     demo
StorageClass:  fast-ssd
Status:        Pending
Volume:
Events:
  Type    Reason                Age   From                         Message
  ----    ------                ----  ----                         -------
  Normal  WaitForFirstConsumer  2m    persistentvolume-controller  waiting for first consumer to be created before binding

Two fields do the diagnostic work: the event's Reason and its Message. Every Pending PVC in the wild is one of the cases below, and the event tells you which before you touch anything. People skip this and start reinstalling CSI drivers. Read the event.

Cause 1: WaitForFirstConsumer doing its job

The Friday demo failure, and the most misunderstood Pending state in Kubernetes. A StorageClass with volumeBindingMode: WaitForFirstConsumer *intentionally* does not provision until a pod that uses the claim is scheduled. The claim Pending by itself means nothing -- the question is whether a pod is coming.

In my case, the pods were Pending too, for an unrelated reason (a missing image pull secret), so no consumer ever materialized, so no volume was ever requested, so everyone waited on everyone. The tell:

$ kubectl get pods -n demo
NAME                     READY   STATUS              RESTARTS   AGE
app-6f8b7d9c4-x1a2b      0/1     ImagePullBackOff    0          4m

A PVC Pending *and* its pod not scheduled (for any non-volume reason) with a WaitForFirstConsumer class is a chicken-and-egg standoff. Fix whatever is blocking the pod; the claim binds the moment the pod gets a node. If the pod is Pending *because of* the volume, keep reading -- that's a different cause.

Cause 2: the provisioner is missing, dead, or deaf

If the event says waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually by the system administrator and stays that way for minutes with a pod already scheduled, the provisioner isn't doing its job. Check, in order:

$ kubectl get pods -n kube-system | grep csi
ebs-csi-controller-7b9c5d4f6-abcde    5/5   Running   0   12d
ebs-csi-node-xxxxx                    3/3   Running   0   12d
  • Controller not Running? Crashloops here are usually IAM or credential problems; kubectl logs on the controller's csi-provisioner container will say so.
  • Provisioner absent entirely? Fresh clusters and some managed add-ons ship without CSI drivers installed, or with the in-tree driver disabled. The PVC just waits forever.
  • Wrong provisioner name? A StorageClass that says provisioner: kubernetes.io/aws-ebs on a modern cluster where the in-tree driver is removed is silently dead. Nothing errors; nothing happens. Check that the provisioner string matches a controller that actually exists.

Cause 3: provisioning is happening -- and failing

This one's the loudest and the easiest. The events contain real errors from the driver:

Warning  ProvisioningFailed  12s (x5 over 2m)  ebs.csi.aws.com_ebs-csi-controller-xxx
  failed to provision volume with StorageClass "fast-ssd":
  rpc error: code = InvalidArgument desc = Volume capabilities MULTI_NODE_MULTI_WRITER not supported

Read the message literally; it's usually specific. The greatest hits:

  • Bad StorageClass parameters. type: gp4 (doesn't exist), an IOPS value invalid for the disk type, a KMS key ARN from another account. The driver rejects the request and the claim retries forever.
  • IAM/permissions. The driver's service account lacks ec2:CreateVolume or the equivalent. The error says UnauthorizedOperation or AccessDenied plainly.
  • Cloud quota. Volume count or total storage quota exhausted in that region. The fix is a quota ticket, not YAML.
  • Access mode mismatch. Asking for ReadWriteMany from a block-storage driver that only does RWO.

The good news: provisioning retries automatically, so fixing the StorageClass or the quota resolves the Pending claim without recreating anything. The bad news: if the StorageClass parameters were wrong, editing the claim won't help -- you fix the class (or point the claim at a corrected one) and let the retry happen.

Cause 4: topology conflicts -- the zone trap

This one shows up on the *pod*, not the PVC. Volume already provisioned and Bound, pod stuck Pending:

Warning  FailedScheduling  20s (x4 over 1m)  default-scheduler
  0/8 nodes are available: 3 node(s) had volume node affinity conflict,
  2 node(s) had taint ..., 3 Insufficient cpu.

A volume in us-east-1a was provisioned (Immediate binding mode, probably) and the pod wants to land anywhere else. The PV now carries node affinity pinning it to one zone, and no eligible node in that zone has room. Fixes, in order of preference: switch the StorageClass to WaitForFirstConsumer so this never happens again; add capacity in the volume's zone; or, if the data is disposable, delete the PV and claim and let them reprovision where the pod can run. Watching pods and claims sit in mutual Pending limbo is much easier to untangle when you can see the whole namespace at once -- the kind of live view we built conndeck to give you.

Cause 5: quota, capacity, and the quiet ones

The last grab bag, each with its own flavor of silence:

  • No default StorageClass. Claim doesn't set storageClassName, cluster has no class annotated storageclass.kubernetes.io/is-default-class: "true". Result: Pending, *zero events*, forever. kubectl get sc and look for (default) in the output.
  • ResourceQuota. A namespace quota on requests.storage or PVC count that's full. The API rejects the PVC creation or resize, which is at least loud, but I've seen it surface as a stuck resize instead.
  • Static provisioning mismatch. storageClassName: "" with no pre-created PV matching size and access mode. The claim waits for a PV that will never exist.
  • Storage backend full. Ceph pools and NFS servers don't page you; they just make provisioning fail with creative errors.

Frequently asked questions

Why is my PVC stuck in Pending?

A Pending PVC means Kubernetes hasn't bound it to a volume yet, and the cause is almost always visible in the PVC's events. Run kubectl describe pvc and read the Events section: either no provisioner responded, provisioning failed with an error, or the claim is deliberately waiting for a pod under WaitForFirstConsumer. The event text names the cause in nearly every case.

What does waiting for a volume to be created mean?

It means the StorageClass uses volumeBindingMode WaitForFirstConsumer, so provisioning is intentionally delayed until a pod that uses the claim gets scheduled. A Pending PVC with this message is not broken; it's waiting. Create or fix the pod that mounts the claim and the volume will be provisioned in whatever zone the pod lands on. If no pod ever comes, the claim sits Pending forever by design.

How do I fix failed to provision volume errors?

Read the full event message first, because the provisioner usually tells you what's wrong. Common causes are the CSI driver being down or missing, invalid parameters in the StorageClass like a disk type that doesn't exist in that region, exhausted cloud quota for volumes, or IAM permissions that don't let the driver create disks. Fixing the underlying issue retriggers provisioning automatically; you rarely need to recreate the claim.

What is a volume node affinity conflict?

It happens when a volume was provisioned in one availability zone but the pod can only be scheduled in another, so the scheduler reports that nodes had a volume node affinity conflict. This is the classic symptom of a StorageClass with Immediate binding on a multi-zone cluster. The durable fix is switching the StorageClass to WaitForFirstConsumer so the volume is created wherever the pod actually lands.

Can a missing default StorageClass cause Pending PVCs?

Yes. If your PVC doesn't name a storageClassName and the cluster has no StorageClass annotated as default, nothing will ever provision it and the claim stays Pending with no useful events. Either set storageClassName explicitly on the claim or mark one StorageClass as default with the storageclass.kubernetes.io/is-default-class annotation. Check kubectl get sc to see which class, if any, is the default.

The practical version

A Pending PVC is not a mystery; it's a message you haven't read yet. kubectl describe pvc, read the Events, and the claim will tell you whether it's waiting on a pod, waiting on a provisioner, or failing with an error that names the fix.

And before you reinstall a CSI driver at 4:40 on a Friday, check whether the pod is in ImagePullBackOff. Some of us learned that the hard way, twenty minutes before a demo.