conndeck Download

conndeck blog

Running Postgres on Kubernetes: Operators Make It Sane

#kubernetes#postgres#operators#databases#storage

It was a Sunday, 6:50 AM, and our primary Postgres pod was gone. Not slow -- gone. The node it lived on had been reclaimed by the autoscaler overnight, and the hand-rolled StatefulSet I'd been so proud of was doing exactly what a StatefulSet does: recreating postgres-0 on a new node, mounting its volume, starting the database. Faithfully. Correctly. Into split-brain, because the old primary was still answering writes on a network partition nobody knew about.

Forty minutes to resolve, two hours of forensics, one stern conversation about why we were running a database with no failover logic. The following quarter we moved to an operator, and I have never once missed the YAML.

Quick answer: Running Postgres on Kubernetes is fine -- running it *without an operator* is the part that gets people hurt. Operators like CloudNativePG and Zalando's postgres-operator manage the StatefulSets for you and add the things raw Kubernetes can't: automated failover, replica bootstrap, continuous backups with point-in-time recovery, and safe rolling upgrades. Declare a Cluster resource with a size and a storage class, and the controller does the rest. The old objections -- performance, storage, stability -- are mostly a decade out of date.

Why the objections are stale

Every "don't run databases on Kubernetes" take boils down to three claims, and each has aged badly.

"Storage is too flaky." That was true when local storage meant hostPath and cloud volumes detached randomly. Modern CSI drivers, topology-aware provisioning with WaitForFirstConsumer, and ordered StatefulSet recovery have made pod-reschedule-with-disk-reattach a boring, reliable operation. A database pod moving nodes is now indistinguishable from a VM reboot, except faster.

"Kubernetes doesn't understand failover." Correct -- and irrelevant. Kubernetes doesn't understand your HTTP service either; your readiness gates and load balancers do. The operator is the layer that understands Postgres: replication lag, WAL, promotion, fencing. Kubernetes provides the primitives, the operator provides the database brain.

"Performance suffers." Networked storage has latency, sure. So does every cloud VM's disk. Provision gp3 or pd-ssd with sane IOPS and the difference versus RDS-style managed Postgres is usually in the noise -- and you're trading it for portability and no per-instance cloud tax.

What actually gets people is running Postgres with *none* of the machinery: a StatefulSet, a service, and a prayer. That's the setup that gives you my Sunday morning.

What an operator actually does

Install CloudNativePG and running a three-node HA cluster becomes this:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: orders-db
spec:
  instances: 3
  imageName: ghcr.io/cloudnative-pg/postgresql:16.4
  storage:
    size: 100Gi
    storageClass: fast-ssd
  bootstrap:
    initdb:
      database: orders
      owner: orders

The controller creates the pods, initializes the primary, clones the replicas from it, wires up streaming replication, creates three services (orders-db-rw for the primary, -ro for replicas, -r for anything), and starts monitoring everything:

$ kubectl get cluster orders-db
NAME        AGE   INSTANCES   READY   STATUS                     PRIMARY
orders-db   3d    3           3       Cluster in healthy state   orders-db-1

Kill the primary and watch:

$ kubectl delete pod orders-db-1
pod "orders-db-1" deleted

$ kubectl get cluster orders-db
NAME        AGE   INSTANCES   READY   STATUS                              PRIMARY
orders-db   3d    3           2       Failing over to orders-db-2         orders-db-2

Seconds later, orders-db-2 is primary, the orders-db-rw service points at it, and clients reconnect without a config change. When the old pod returns, it's demoted and rebuilt as a replica. The Zalando operator achieves the same via Patroni and a postgresql custom resource; the mechanics differ, the outcome is the same. The part I want to underline: at no point did a human run pg_ctl promote over SSH, which is how this used to go.

Backups that actually restore

The operator story is only complete with backups, and this is where CloudNativePG earns its keep. Point it at object storage:

spec:
  backup:
    barmanObjectStore:
      destinationPath: s3://my-backups/orders-db
      s3Credentials:
        accessKeyId:
          name: backup-creds
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: backup-creds
          key: SECRET_ACCESS_KEY
      wal:
        compression: gzip
  # plus a ScheduledBackup resource for nightly base backups

From then on, WAL segments stream to S3 continuously and base backups run on schedule -- which means point-in-time recovery to any moment in your retention window. And restore is declarative: a new Cluster whose bootstrap.recovery section references the old cluster's backup materializes a recovered database. No pg_basebackup incantations, no manual WAL replay.

Test the restore. I mean it -- schedule a quarterly drill where you actually recover into a scratch namespace and run a row count. Backups that have never been restored are a hypothesis, not a backup.

The honest tradeoffs

I'm an advocate, not a zealot, so:

  • You now operate a database platform. The operator removes toil, not understanding. When replication lag spikes at 2 AM, someone still needs to know what WAL is. The bar is lower than DIY, not zero.
  • Cross-AZ storage and pod scheduling interact. Pin instances across zones with proper affinity and use WaitForFirstConsumer storage classes, or failover will hand you a volume in the wrong zone at the worst moment.
  • Managed Postgres is still a fine choice. If your team is three people and the database is not your differentiator, RDS or Cloud SQL is a rational buy. The operator path pays off when you have many clusters, on-prem or multi-cloud requirements, or licensing costs that make managed pricing sting.
  • Upgrades are better, not automatic. Major version bumps still need planning; the operator handles minor rolling restarts cleanly.

Having the cluster's failover, instance roles, and PVC health visible in one live view matters more than you'd think during an actual incident -- seeing orders-db-2 promoted while its volume reattached is exactly the kind of thing we built conndeck to surface without ten kubectl commands.

Frequently asked questions

Is it safe to run Postgres on Kubernetes?

Yes, with an operator managing it. The horror stories come from hand-rolled StatefulSets with no failover logic, not from the platform itself. Operators like CloudNativePG handle automated failover, point-in-time recovery, connection pooling, and rolling upgrades, which covers most of what people mean when they say databases don't belong on Kubernetes. You still need to understand storage and backups, but that was true on VMs too.

What's the difference between CloudNativePG and the Zalando operator?

Both manage HA Postgres clusters on Kubernetes, but they differ in style. CloudNativePG uses its own purpose-built instance images, declarative cluster resources, and leans on Kubernetes-native patterns, with backup and recovery built around continuous WAL archiving to object storage. The Zalando postgres-operator is older, uses Patroni for failover and the Spilo image, and has a longer production track record at very large scale. CloudNativePG is the more modern choice for most new deployments; Zalando is proven and fine if it's already working for you.

How does failover work with a Postgres operator?

The operator continuously monitors the primary and the replicas, and when the primary becomes unreachable it promotes the healthiest replica and repoints the primary service at it, usually within seconds to a minute. Clients connecting through the service follow automatically. The old primary, when it returns, is rejoined as a replica. You don't write any of this logic yourself, which is precisely the point.

How do backups work with CloudNativePG?

You configure an object storage destination, typically S3 or a compatible store, and the operator continuously archives WAL segments there while taking scheduled base backups. That combination gives you point-in-time recovery to any moment since the oldest retained base backup. Restoring is declarative too: you create a new Cluster that references the backup in its bootstrap section.

Should I use a StatefulSet or an operator for Postgres?

Use an operator. A raw StatefulSet gives you stable names and storage, but failover, replica bootstrap, backup scheduling, certificate management, and safe upgrades are all left as exercises for the reader, and getting any of them wrong at 3 AM is brutal. The operator encodes years of Postgres operational knowledge into the controller, and it's free.

If you remember one thing

The question was never "Kubernetes or not." It's "with an operator or without." Without one, you're rebuilding failover, backups, and promotion by hand, badly, at 6:50 on a Sunday. With one, a three-node HA Postgres cluster is twenty lines of YAML and a controller that knows more about Postgres operations than most of us ever will.

Pick CloudNativePG for anything new, keep Zalando if it's working, wire up WAL archiving to S3 on day one, and rehearse a restore before you need one. Then go back to bed.