etcd Backup and Restore: The Drill Nobody Practices

It was a Tuesday, 2:20 AM, and the cluster had lost its mind. A botched control plane upgrade on node two of three had corrupted etcd's data directory, node three had been down for hardware reasons since the previous Friday (a ticket nobody had escalated), and node one was now alone, refusing to serve writes because a one-member cluster that *thinks* it should have three members has no quorum. The API server was returning errors on everything. Every GitOps pipeline, every autoscaler, every webhook: dead.

We had snapshots. Hourly, automated, pristine. What we did not have was a single human who had ever restored one. The next three hours were a manpage speedrun I wouldn't wish on anyone. Here's the drill we should have practiced.

Quick answer: Back up etcd with etcdctl snapshot save on a schedule, store snapshots off the control plane nodes, and -- the part everyone skips -- actually rehearse the restore: stop the apiserver and etcd, etcdctl snapshot restore into fresh data directories with correct initial-cluster flags, restart, verify. Quorum loss (a majority of members down) is the disaster case, and restoring from snapshot is the way back. A backup you've never restored is a hypothesis. Drill it quarterly.

What you're actually backing up

etcd is the cluster's memory. Every Deployment, Secret, ConfigMap, Node registration, lease, and CRD instance lives there and nowhere else. Lose etcd with no backup and you haven't lost configuration -- you've lost the *cluster*. The workloads still running on the nodes are orphaned zombies that nothing can manage.

The tool is etcdctl, and the backup is a point-in-time snapshot of the database:

$ ETCDCTL_API=3 etcdctl snapshot save /backups/etcd-2026-08-26-0220.db \
    --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key
{"level":"info","ts":...,"msg":"saved snapshot","path":"/backups/etcd-2026-08-26-0220.db"}
Snapshot saved at /backups/etcd-2026-08-26-0220.db

Two non-negotiable details. First, verify every snapshot before trusting it -- etcdctl snapshot status /backups/etcd-2026-08-26-0220.db -w table confirms the file is intact and tells you the revision it contains. Second, get the file off the node. A snapshot on the control plane node's local disk is a backup of the node for the node, and when the node dies, it dies together. Object storage, another host, anywhere else.

On kubeadm clusters the certs live at the paths above; managed Kubernetes (EKS, GKE, AKS) runs etcd for you and doesn't let you near it, which is a different situation I'll come back to.

Quorum: the actual disaster

etcd is a Raft consensus system, and Raft needs a majority. Three members tolerate one failure; two down means quorum is gone, and the whole API goes with it. That's the Tuesday scenario: three-member cluster, one member on hardware leave, one member corrupted, one member standing around unable to elect itself leader because one is not a majority of three.

The symptoms, so you recognize them:

$ kubectl get nodes
Error from server: rpc error: code = Unavailable desc = transport is closing

$ ETCDCTL_API=3 etcdctl endpoint status --cluster -w table
https://10.0.1.11:2379 ... is healthy: false ...

Running pods keep running -- the kubelet doesn't need the API to keep containers alive -- but nothing can change, schedule, or heal. A node dying mid-outage means those pods never get rescheduled. Every minute of quorum loss is a minute of frozen infrastructure, which is why the restore has to be muscle memory, not a research project.

The restore procedure, practiced

This is the kubeadm static-pod path, the most common self-managed shape. Read it once now, but know that reading is not drilling.

1. Stop everything. On all control plane nodes, move the static pod manifests out of /etc/kubernetes/manifests so the kubelet stops etcd and the apiserver:

$ mkdir -p /root/manifest-backup
$ mv /etc/kubernetes/manifests/etcd.yaml /root/manifest-backup/
$ mv /etc/kubernetes/manifests/kube-apiserver.yaml /root/manifest-backup/

2. Restore the snapshot on each node, into a fresh data directory, with member-specific identity flags:

$ ETCDCTL_API=3 etcdctl snapshot restore /backups/etcd-2026-08-26-0220.db \
    --data-dir=/var/lib/etcd-restore \
    --name=ip-10-0-1-11 \
    --initial-cluster=ip-10-0-1-11=https://10.0.1.11:2380,ip-10-0-1-12=https://10.0.1.12:2380,ip-10-0-1-13=https://10.0.1.13:2380 \
    --initial-cluster-token=etcd-cluster-restore-20260826 \
    --initial-advertise-peer-urls=https://10.0.1.11:2380

The flags that bite people: --name and --initial-advertise-peer-urls must differ per node and match what the etcd static pod manifest passes; --initial-cluster must list all members identically on every node; and --initial-cluster-token should be new, because it marks this as a fresh cluster generation. Then move the old data dir aside (mv /var/lib/etcd /var/lib/etcd.corrupt) and put the restored one in place, or update the manifest's volume path.

3. Bring it back. Restore the manifests, let the kubelet start etcd and the apiserver, then verify:

$ kubectl get nodes
NAME           STATUS   ROLES           AGE   VERSION
ip-10-0-1-11   Ready    control-plane   312d  v1.31.4

4. Confirm the delta. The cluster is now exactly as it was at snapshot time. Everything created or changed since -- deployments, secrets, scaled replica counts -- is gone. If your GitOps is in order, this is a non-event: ArgoCD or Flux reconciles the missing state back within minutes. If it isn't, you just learned why it should be.

One shortcut worth knowing: if only one member of three died, you don't need a snapshot restore at all. Remove the dead member (etcdctl member remove), wipe its data dir, and etcdctl member add it back so it rejoins and catches up from the healthy majority. Snapshot restores are for losing the quorum itself.

The drill, for real

We now do this quarterly, in a scratch cluster: take a snapshot, make some recognizable change (a ConfigMap with the date in it), nuke etcd's data directory on all three members, restore, and confirm the cluster comes back without the change. The first drill took half a day and surfaced two broken assumptions in our runbook. The most recent took forty minutes. That's the entire argument for practicing.

Keeping an eye on control plane health between drills matters too -- watching apiserver and etcd pod status, restart counts, and node conditions in a live cluster view is the kind of early warning that turns a 2 AM disaster into a 2 PM maintenance window, and it's part of why we built conndeck to treat the control plane as a first-class citizen.

Frequently asked questions

How do I back up etcd in Kubernetes?

Take a snapshot with etcdctl snapshot save against a live etcd member, using the API version 3 endpoint and the cluster's CA, certificate, and key for authentication. A snapshot is a consistent point-in-time copy of the entire etcd database, and you can take one without stopping anything. Automate it on a schedule and copy the snapshot file off the node, because a snapshot sitting on the failed node is not a backup.

How do I restore etcd from a snapshot?

Stop the kube-apiserver and etcd on all control plane nodes, run etcdctl snapshot restore on each node with the correct name, peer URLs, and initial cluster settings so each member gets its own data directory, then start etcd and the apiserver again. The key detail is that restore regenerates the data directories but does not rewrite your static pod manifests, so the new peer configuration must match what those manifests expect. Practice this before you need it.

What happens when etcd loses quorum?

The Kubernetes API becomes read-only at best and dead at worst: no new pods, no updates, no leader election, while already-running workloads mostly keep running. Quorum is lost when a majority of members are gone, so in a three-node cluster that's two failures. Recovery usually means restoring from snapshot or, in some partial-failure cases, removing dead members and re-adding them one at a time.

How often should I take etcd snapshots?

Hourly is a common baseline for production clusters, with snapshots kept for days and copied to remote storage. The right answer is bounded by how much cluster state change you can afford to lose: deployments, secrets, and config changes since the last snapshot are gone after a restore. Snapshot before any cluster upgrade or etcd version change too, no exceptions.

Does a managed Kubernetes service still need etcd backups?

With EKS, GKE, or AKS the provider runs etcd and handles its durability, and you cannot run etcdctl against it anyway. What you still need is workload-level backup, meaning your resources and persistent volumes, which is what tools like Velero cover. Accidentally deleting a namespace or a bad GitOps sync is your disaster even on managed Kubernetes, and the provider's etcd durability won't save you from it.

Where this leaves you

Snapshots are the easy half: etcdctl snapshot save, verify with snapshot status, ship the file off the node, hourly. The hard half is the restore, and it is only hard if the first time you do it is during an outage.

Schedule the drill. Corrupt a scratch cluster on a calm afternoon and bring it back. The 2 AM version of you -- the one doing a manpage speedrun while the API is down -- will be very grateful.