conndeck Download

conndeck blog

PVs, PVCs, and StorageClasses: The Mental Model That Finally Clicked

#kubernetes#storage#persistent-volumes#storageclass

It was a Saturday morning, 9:15, and I was staring at a kubectl delete pvc command with my finger hovering over Enter. The cluster was being decommissioned, the namespace was going away, and the PVC was attached to a Postgres database holding six months of a customer's billing data. I knew the reclaim policy mattered. I could not, in that moment, remember which value meant "delete the disk" and which meant "keep it." I had been running Kubernetes for two years.

I looked it up, deleted nothing, and spent the rest of that weekend actually learning how the storage pieces fit together instead of cargo-culting YAML. This post is the mental model I should have had.

Quick answer: A PersistentVolume is a real piece of storage in the cluster, a PersistentVolumeClaim is a pod's request for storage that gets bound to a PV, and a StorageClass is a template that lets a provisioner create PVs automatically when claims appear. The reclaimPolicy on the PV decides whether the underlying disk is deleted or kept when the claim goes away, and volume expansion works by editing the PVC's requested size upward, if the StorageClass allows it. Bind those four ideas together and ninety percent of Kubernetes storage stops being mysterious.

The three objects, one sentence each

The confusion comes from the fact that three objects with similar names do three very different jobs. So:

  • PersistentVolume (PV): the storage itself. A cluster-scoped object representing a real disk -- an EBS volume, a Ceph RBD image, an NFS export. It has a capacity, access modes, and a reclaim policy.
  • PersistentVolumeClaim (PVC): the request. Namespaced, owned by whoever runs the workload, specifies a size, an access mode, and usually a StorageClass. Pods mount claims, never PVs directly.
  • StorageClass: the factory template. Names a provisioner and its parameters (disk type, filesystem, replication), plus defaults like the reclaim policy and binding mode.

The relationship that took me too long: the PVC and the pod live in your namespace and are yours. The PV is cluster infrastructure and is created either by you (static) or by the provisioner (dynamic). The claim and the volume get bound one-to-one, and the binding is sticky -- once bound, a PVC won't jump to a different PV even if a better match appears later.

Dynamic provisioning is the whole point

Static provisioning -- writing PV manifests by hand, matching them to claims by size and labels -- is how you learn the mechanics, and it's how nobody should run production. Dynamic provisioning is the normal path:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "4000"
  encrypted: "true"
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

Now a claim that references it:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 100Gi

When this PVC is created, the external-provisioner sidecar watching claims sees it, calls the EBS CSI driver, creates a real gp3 volume, and creates a PV pointing at it. The claim goes from Pending to Bound:

$ kubectl get pvc postgres-data
NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
postgres-data   Bound    pvc-7f3a9c21-4b2e-4d1a-9c5f-2e8a1b6d3f90   100Gi      RWO            fast-ssd       42s

That pvc-7f3a... name is the PV. It's generated, you never write it, and its lifecycle follows the claim. One detail worth knowing: storageClassName: "" (empty string) means "no dynamic provisioning, bind me to a static PV only," while omitting the field entirely means "use the default StorageClass." That difference has bitten everyone at least once.

Reclaim policies: the Saturday morning lesson

The reclaimPolicy on a PV decides what happens to the underlying storage when its claim is deleted. There are two values that matter:

  • Delete: the PV object and the actual disk are destroyed. This is the default for dynamically provisioned volumes on most cloud StorageClasses. Read that again before you delete anything.
  • Retain: the PV stays, moves to Released status, and the data on the disk is untouched. Nothing rebinds to it automatically -- a Released PV is inert until you intervene.

The Saturday scenario: the PVC deletion would have cascaded to the PV, the PV's reclaimPolicy was Delete (the EBS default), and the disk with the billing data would have been vaporized along with the namespace. The fix when you inherit a claim you can't afford to lose:

$ kubectl patch pv pvc-7f3a9c21-4b2e-4d1a-9c5f-2e8a1b6d3f90 \
    -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
persistentvolume/pvc-7f3a9c21-4b2e-4d1a-9c5f-2e8a1b6d3f90 patched

Do it on the PV, not the PVC, and do it before anything gets deleted. If you ever need to recover a Retained PV for a new claim, the dance is: delete the old PV object after clearing its claimRef, edit the new PV's claimRef to point at the new PVC, and the binding snaps into place. It's fiddly by design -- Kubernetes really doesn't want to hand old data to a new workload by accident.

My rule: production data classes get reclaimPolicy: Retain, always. Deleting a disk should be a deliberate act, not a side effect of tidying a namespace.

Expansion: the one-way door

Volume expansion is genuinely pleasant these days. Three requirements: the StorageClass has allowVolumeExpansion: true, the CSI driver supports resizing, and you're growing -- never shrinking. Then it's one edit:

$ kubectl patch pvc postgres-data -p '{"spec":{"resources":{"requests":{"storage":"200Gi"}}}}'
persistentvolumeclaim/postgres-data patched

$ kubectl get pvc postgres-data
NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
postgres-data   Bound    pvc-7f3a9c21-4b2e-4d1a-9c5f-2e8a1b6d3f90   200Gi      RWO            fast-ssd       3d

Watch the events while it happens. You'll typically see ExternalExpanding while the driver resizes the cloud disk, and on most modern drivers the filesystem is grown online with the pod still running. Older drivers set the FileSystemResizePending condition on the PVC and finish the job the next time the pod restarts -- check kubectl describe pvc if the capacity field updates but df inside the container disagrees.

Two traps. First, shrinking is impossible; if you over-provisioned, your path is a new smaller PVC and a data migration. Second, StatefulSet volumeClaimTemplates historically couldn't be edited in place, so growing a StatefulSet meant patching each PVC and then recreating the StatefulSet with --cascade=orphan. Recent Kubernetes versions allow updating volumeClaimTemplates resources fields, but verify against your cluster version before trusting it in anger.

WaitForFirstConsumer, or why your pod can't mount its disk

One StorageClass field deserves its own section because the failure mode is so common. With volumeBindingMode: Immediate (the default), the volume is provisioned the moment the PVC exists -- in whichever zone the provisioner picks. With WaitForFirstConsumer, provisioning waits until a pod using the claim is scheduled, and the volume is created in that pod's zone.

On multi-zone clusters, Immediate mode is a trap: the disk lands in us-east-1a, the scheduler places the pod in us-east-1b, and you get a pod stuck in Pending with events like:

Warning  FailedScheduling  12s (x3 over 30s)  default-scheduler
  0/6 nodes are available: 3 node(s) had volume node affinity conflict,
  3 Insufficient cpu.

The volume's node affinity now pins the pod to a zone with no room. The fix is the StorageClass field shown earlier, and it's why nearly every cloud StorageClass I ship uses WaitForFirstConsumer. Watching claims flip from Pending to Bound while pods schedule is one of those things that's much clearer in a live view of the namespace -- which, yes, is the kind of thing we built conndeck to make obvious.

Frequently asked questions

What's the difference between a PV and a PVC?

A PersistentVolume is the actual chunk of storage in the cluster, and a PersistentVolumeClaim is a workload's request for storage with a size and access mode. The claim is what your pod references, and Kubernetes binds it to a volume that satisfies the request. Think of the PV as the disk and the PVC as the ticket that says who gets to use it.

What does a StorageClass actually do?

A StorageClass describes a type of storage and names the provisioner that creates it, like the EBS CSI driver or a Ceph provisioner, along with parameters such as disk type or filesystem. When you create a PVC that references a StorageClass, the provisioner creates a real volume and a matching PV automatically. Without a StorageClass you have to create PVs by hand, which is called static provisioning and scales poorly.

What happens to my data when I delete a PVC?

It depends on the reclaimPolicy on the PersistentVolume. With Delete, which is the default for dynamically provisioned volumes on most clouds, the PV and the underlying disk are deleted and your data is gone. With Retain, the PV stays in a Released state and the underlying data survives until you clean it up manually. Always check the reclaim policy before deleting a claim attached to anything you care about.

Can I resize a PersistentVolumeClaim?

Yes, if the StorageClass has allowVolumeExpansion set to true and the CSI driver supports it. You edit the PVC and increase spec.resources.requests.storage, and the control plane expands the volume; most drivers also grow the filesystem online, though some require a pod restart. Shrinking a volume is not supported at all, so expansion is a one-way door.

What does WaitForFirstConsumer mean?

It's a volumeBindingMode on a StorageClass that delays provisioning until a pod that uses the PVC is actually scheduled. The scheduler picks a node first, then the volume is created in that node's zone, which prevents the classic failure where a disk gets provisioned in zone A and the pod lands in zone B and can never mount it. Most cloud StorageClasses should use it, and many don't by default.

Lessons for your cluster

PV is the disk, PVC is the request, StorageClass is the factory. Provisioning is dynamic by default, reclaim policies decide whether deletion means deletion, and expansion only goes up.

Learn those four sentences cold and you'll never hover over Enter on a Saturday morning wondering whether you're about to delete someone's billing database. Ask me how I know.