conndeck Download

conndeck blog

StatefulSets: When You Actually Need One

#kubernetes#statefulset#storage#databases

It was a Wednesday, around 3 PM, when I proudly "fixed" our Redis setup by converting the StatefulSet to a Deployment. Fewer moving parts, standard tooling, one less weird YAML kind to explain to new hires. I applied it, watched three shiny new pods come up, and then watched our cache hit rate fall off a cliff because every pod had mounted the same PVC and two of them were now serving an empty filesystem while the third held all the data.

That afternoon taught me more about StatefulSets than two years of reading docs had. Here's the version of that lesson that doesn't cost you an incident.

Quick answer: Use a StatefulSet when your pods need a stable identity -- a predictable name and DNS record, their own persistent storage that survives rescheduling, or ordered startup and shutdown. Databases with replication, queues, and consensus systems are the classic cases. If your pods are interchangeable and any replica can serve any request, a Deployment is the right answer and a StatefulSet is just extra machinery. The tell: if it matters *which* pod handles something, you want a StatefulSet.

What a StatefulSet actually guarantees

Strip away the YAML and a StatefulSet makes exactly three promises:

  • Stable network identity. Pods are named <statefulset>-<ordinal>: postgres-0, postgres-1, postgres-2. Delete postgres-1 and the replacement is also postgres-1. A Deployment names pods like web-7d9f4b8c6-x2k4p, and the replacement gets a new random suffix.
  • Stable storage per pod. Each ordinal gets its own PVC, created from volumeClaimTemplates, named like data-postgres-0. Pod dies, pod moves nodes, PVC follows. The claim outlives the pod entirely.
  • Ordered operations. Scale-up creates ordinal N+1 only after ordinal N is Running and Ready. Rollouts and scale-down proceed one pod at a time, highest ordinal first. No stampedes.

That's it. Everything else -- the headless service, podManagementPolicy, update strategies -- is machinery in service of those three guarantees.

The headless service, finally explained

The StatefulSet spec requires a serviceName, and the service it points at is one you create yourself:

apiVersion: v1
kind: Service
metadata:
  name: postgres-headless
spec:
  clusterIP: None
  selector:
    app: postgres
  ports:
    - port: 5432

clusterIP: None is the trick. A normal service gets one virtual IP and load-balances across pods. A headless service gets no IP at all -- instead, DNS returns a record for each pod: postgres-0.postgres-headless.default.svc.cluster.local resolves to that specific pod's IP, wherever it runs.

Why this matters: stateful systems talk to *members*, not to "the service." A Postgres replica needs to stream from postgres-0, specifically. A Kafka broker needs to reach broker 2, specifically. postgres-0 is a name that means the same thing forever, and the DNS record is how everything finds it. This is also how client connections that must hit the primary get pinned: point the write connection at postgres-0.<svc> and it follows the pod, not a load balancer.

A minimal working example

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres-headless
  replicas: 2
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:16
          ports:
            - containerPort: 5432
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: fast-ssd
        resources:
          requests:
            storage: 50Gi

Apply it and watch the ordering happen:

$ kubectl get pods -w
NAME         READY   STATUS    RESTARTS   AGE
postgres-0   1/1     Running   0          14s
postgres-1   0/1     Pending   0          0s
postgres-1   1/1     Running   0          12s

$ kubectl get pvc
NAME               STATUS   VOLUME       CAPACITY   STORAGECLASS   AGE
data-postgres-0    Bound    pvc-1a2b...  50Gi       fast-ssd       2m
data-postgres-1    Bound    pvc-3c4d...  50Gi       fast-ssd       1m

postgres-1 didn't exist until postgres-0 was Ready. Each pod got its own claim. Delete postgres-0 and it comes back as postgres-0, reattached to data-postgres-0, with all its data intact. That sequence -- stable name, stable disk, ordered creation -- is the entire product.

When a Deployment is actually fine

Here's the part the tutorial authors skip: most things people put in StatefulSets don't need one. My rules of thumb:

  • Single replica, one PVC, no clustering? Deployment with strategy: Recreate and a PVC. Yes, really. With one replica there's nothing to order and no identity to preserve; Recreate just prevents the old and new pod from mounting the disk simultaneously. Plenty of small internal Postgres instances run exactly like this, happily, for years.
  • Interchangeable workers with local scratch? Deployment. If any pod can handle any request, identity is overhead.
  • Clustered database with real replication? Now StatefulSet -- or, honestly, an operator like CloudNativePG that manages StatefulSets for you, plus failover, backups, and the hundred other things you'll otherwise build badly by hand.
  • Consensus systems (etcd, ZooKeeper, Kafka)? StatefulSet, no debate. Membership is identity.

The Redis incident, decoded: I'd given three interchangeable pods one shared volume. What the workload actually wanted was three independent pods with their own storage -- which is either a StatefulSet, or a Deployment where the pods don't persist anything at all. I picked the worst of both worlds.

The sharp edges worth knowing

PVCs don't get deleted on scale-down. Scale from 3 to 1 and data-postgres-2 stays behind, Bound and billing you. That's deliberate -- scale-down is not permission to destroy data -- but it means cleanup is your job, and leftover claims can reattach stale data if you scale back up later. (Recent Kubernetes versions added persistentVolumeClaimRetentionPolicy on the StatefulSet to automate this; check your version.)

podManagementPolicy: Parallel relaxes ordering so all pods start and stop together. Great for workloads like Elasticsearch data nodes where ordering buys you nothing and slow starts cost you minutes.

Rolling updates are ordered too, highest ordinal first, and partition in the update strategy lets you canary a new image on just the top ordinals. For a database, that's the difference between "hope" and "tested."

Watching this -- which ordinal is restarting, which claim is reattaching, whether the rollout is stuck on a not-Ready member -- is exactly the kind of stateful detail that's painful to reconstruct from kubectl get output. It's a big part of why we built conndeck to show pods, claims, and rollout order in one live view.

Frequently asked questions

What's the difference between a StatefulSet and a Deployment?

A Deployment manages interchangeable pods with random names and shared or no storage, while a StatefulSet gives each pod a stable ordinal name like postgres-0, its own PersistentVolumeClaim that survives rescheduling, and ordered create and delete. Use a StatefulSet when pod identity or per-pod storage matters, and a Deployment for everything else.

Why does a StatefulSet need a headless service?

The headless service is what gives each pod its own stable DNS name, like postgres-0.postgres-headless.default.svc.cluster.local, instead of one virtual IP load-balancing across all pods. Stateful applications usually need to address a specific member, such as a database primary, and DNS records for individual pods are how they find each other. The StatefulSet spec requires the serviceName field to point at a service you create yourself, typically with clusterIP set to None.

Can I run a database in a regular Deployment instead?

For a single-replica database backed by one PVC, a Deployment with Recreate strategy works fine and plenty of people do it. You lose per-replica identity and ordered operations, but with one replica there is nothing to order. The moment you need replication, failover, or per-member storage, a StatefulSet or a proper database operator becomes the right tool.

What happens when a StatefulSet pod is deleted?

The pod is recreated with the exact same name, the same DNS record, and the same PVC reattached, even if it lands on a different node. The data survives because the claim lives independently of the pod. This is the core guarantee: identity and storage persist across rescheduling, which is exactly what a Deployment does not give you.

How do I scale a StatefulSet safely?

Scaling up adds new ordinals one at a time, so postgres-2 is created only after postgres-1 is Running and Ready, and each new pod gets fresh PVCs from the volumeClaimTemplates. Scaling down removes pods in reverse ordinal order and terminates each gracefully, but the PVCs of removed pods are kept by default, so you delete them yourself if you want the storage back. Never scale a database StatefulSet without knowing how your application joins and removes members.

What I do differently now

StatefulSets are three guarantees -- stable name, stable disk, ordered operations -- wrapped in YAML. If your workload needs even one of them, reach for it. If it doesn't, a Deployment is simpler and you're not paying for machinery you don't use.

And if you ever find yourself converting a StatefulSet to a Deployment to "simplify" things, ask first whether the workload cares which pod is which. My cache hit rate wishes someone had asked me.