conndeck Download

conndeck blog

Velero: Backups That Survive a Real Disaster

#kubernetes#velero#backup#disaster-recovery

It was a Monday, 10:15 AM, when I discovered our Velero backups had been succeeding for four months and protecting approximately nothing. The schedule ran nightly, the backups reported Completed, the dashboard was green. Then a kubectl delete namespace with a typo'd name took out payments instead of payments-test, and I learned in front of my entire team that every PersistentVolumeClaim in that namespace had been excluded by a label selector nobody remembered writing.

We got the data back from a database dump someone had the foresight to keep. The Velero setup got rebuilt that week, correctly, with a restore test that now runs monthly. Here's the version I should have built the first time.

Quick answer: Velero backs up your cluster's resources (deployments, services, secrets, PVCs -- the YAML) to object storage, and backs up persistent volume *data* either via cloud snapshots or file-system backup with kopia (formerly restic). Install the server with a schedule (velero schedule create), and your cluster state lands in a bucket nightly. But the backup is the easy half: Velero setups fail silently through bad selectors, missing volume backup opt-ins, and untested restores. A backup that has never been restored to a second cluster is a guess.

Install and the anatomy of a backup

Velero is a server component in the cluster plus a CLI. Installing with the AWS plugin looks like:

$ velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.10.0 \
    --bucket conndeck-velero-backups \
    --backup-location-config region=us-east-1 \
    --snapshot-location-config region=us-east-1 \
    --secret-file ./credentials-velero \
    --use-node-agent \
    --default-volumes-to-fs-backup

Two flags there matter more than all the others, and I'll come back to both: --use-node-agent (the DaemonSet that does file-system volume backups) and --default-volumes-to-fs-backup (opt *all* volumes into fs-backup by default, so nobody has to remember an annotation).

A one-off backup:

$ velero backup create payments-manual-0826 --include-namespaces payments

$ velero backup get
NAME                     STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION
payments-manual-0826     Completed   0        0          2026-08-26 10:32:11 +0200 CEST    29d       default

That backup contains every API object in the namespace -- serialized YAML in the bucket -- plus the volume data if fs-backup or snapshots are configured. Restoring is the inverse:

$ velero restore create --from-backup payments-manual-0826

Schedules: set it, but verify it

Nobody runs backups by hand in production; you define a Schedule:

$ velero schedule create nightly \
    --schedule="0 2 * * *" \
    --include-namespaces payments,orders,inventory \
    --ttl 720h

TTL of 720 hours means 30 days of retention, and Velero garbage-collects expired backups on its own. This is also exactly where my Monday disaster was born, so let me be explicit about the traps:

  • Selectors are filters, and filters forget things. Our schedule used --selector environment=production, and the payments PVCs had been created by a Helm chart that didn't set that label. Nightly backups, four months, zero volumes matching. Prefer namespace-scoped schedules over label selectors for anything important, and audit what a selector actually matches: kubectl get all,pvc,secret -n payments -l environment=production.
  • A Completed backup isn't a complete backup. Completed means "everything I was asked to do succeeded." It says nothing about whether you asked for the right things. velero backup describe <name> --details shows what was actually captured, including which volumes went to fs-backup and which were skipped. Read it once for every schedule you create.
  • Back up cluster-scoped resources deliberately. Schedules that include namespaces don't capture cluster-scoped objects (ClusterRoles, CRDs, StorageClasses) unless you ask. Decide whether you need them and say so explicitly.

restic, kopia, and what backs up the actual data

Velero has two ways to capture volume contents, and picking between them is a real decision:

Cloud snapshots (--snapshot-volumes, via the provider plugin) snap the EBS/GCE/Azure disk at the storage layer. Fast, cheap, crash-consistent. The catch: snapshots live in the same cloud account and region, they're useless for cross-provider restores, and a snapshot of a running database is only as good as the database's crash recovery. Fine for a lot of state, wrong for everything.

File-system backup (the node agent, using kopia today, restic before it) copies volume contents file-by-file into your object storage bucket. Provider-agnostic -- an EBS volume can restore onto Ceph or a kind cluster -- deduplicated, incremental. Slower and more CPU-hungry than snapshots, and it works on most volume types (hostPath, EBS, NFS; not all, check the docs for your CSI driver).

My default posture: fs-backup everything with --default-volumes-to-fs-backup, and let the handful of workloads that need application-consistent handling (big databases) get their own purpose-built backup path -- an operator, WAL archiving, dumps -- with Velero as the secondary copy. Kopia is the right engine for new setups; restic in Velero is maintenance-mode, so don't build fresh on it.

The restore test that actually means something

Restoring into the same cluster tests Velero. Restoring into a *different* cluster tests your disaster recovery. Monthly, we do this:

# on a scratch cluster, velero pointed at the same bucket:
$ velero restore create dr-test-$(date +%m%d) \
    --from-schedule nightly \
    --include-namespaces payments \
    --namespace-mappings payments:payments-drtest

Then the checklist: are the pods Running? Did the PVCs bind -- and to what storage class? Does the app connect? Can you read a known row out of the restored data? The --namespace-mappings flag is the trick that makes same-cluster rehearsal possible without collisions, and it's also the feature that makes real migrations pleasant.

The failure you'll hit first, guaranteed: storage class mismatch. The backup references fast-ssd, the target cluster calls it gp3-csi, and every PVC comes back unbound. The fix is a ConfigMap in the velero namespace mapping old names to new:

apiVersion: v1
kind: ConfigMap
metadata:
  name: change-storage-class-config
  namespace: velero
  labels:
    velero.io/plugin-config: ""
    velero.io/change-storage-class: RestoreItemAction
data:
  fast-ssd: gp3-csi

Second-most-common: restored secrets that reference service account tokens or webhook certificates that don't exist in the new cluster. Test restores surface these on a quiet Tuesday instead of during an incident, which is the whole point. During a real restore drill, having the target namespace's pods, claims, and events visible in one live view -- the thing we built conndeck for -- is the difference between watching a recovery and guessing at one.

Frequently asked questions

What does Velero actually back up?

Velero backs up your Kubernetes resources, the YAML objects like deployments, services, secrets, and PVCs, into object storage, and it can also capture the persistent volume data itself, either through cloud disk snapshots or through file-system level backups using kopia or restic. Restoring recreates the resources and their data in the same cluster or a different one. It's a disaster recovery and migration tool, not a replacement for application-level database dumps.

Should I use restic or kopia with Velero?

Use kopia for anything new. It's the newer of Velero's two file-system backup integrations, it's faster, handles large volumes better, and restic support in Velero is in maintenance mode. Both work by backing up volume data at the filesystem level to object storage, which makes them provider-agnostic, unlike cloud disk snapshots that only work on one platform.

How do I schedule Velero backups?

Create a Schedule resource with a cron expression, and Velero creates a Backup object every time it fires. A daily schedule with a TTL like 720h keeps thirty days of backups and garbage-collects the older ones automatically. You can scope schedules with label selectors or included namespaces so critical namespaces back up more often than everything else.

Can Velero restore to a different cluster?

Yes, and it's one of Velero's best features. Point the second cluster's Velero install at the same object storage bucket in read-only mode, and its backups become available as restore sources. This is how you do real disaster recovery and cluster migrations. Watch out for storage class differences between clusters, which you handle with config map-based name mappings during restore.

Does Velero replace database backups?

No. Volume snapshots and file-system copies of a running database can be crash-consistent at best, and restoring one can mean replaying logs or recovering corrupted state. For Postgres, MySQL, and friends, use the database's own tooling or an operator's backup feature for your primary recovery path, and treat Velero as the belt-and-suspenders layer for everything around the database.

Lessons for your cluster

Velero is genuinely good software that fails in genuinely quiet ways: selectors that match nothing, volumes nobody opted in, restores nobody's tried. Install it with the node agent, default everything to fs-backup, schedule by namespace, and describe your backups once to see what they actually contain.

Then do the only step that matters: restore to a second cluster, monthly, on a calendar invite. The green Completed status is a claim. A verified restore is proof.