Upgrading Kubernetes Without Downtime or Drama
The first Kubernetes upgrade I ever ran solo was 1.19 to 1.20, self-managed, kubeadm, on a Friday. I know. I was younger then.
The control plane upgrade went fine. Then I started terminating worker nodes so the new ones would come up on the new AMI -- because draining them one by one felt slow, and I had dinner plans. Within four minutes, two single-replica internal services were hard down, a queue consumer with a 40-minute job in flight had been shot mid-job, and the on-call Slack channel was doing that thing where three people type at once. No customer-facing outage, by pure luck. Dinner was cancelled.
Eight years later, upgrades are the most boring thing I do, and they should be boring for you too. Here's the whole playbook.
Quick answer: A zero-drama Kubernetes upgrade is control plane first, nodes second, one minor version at a time, with every node going through kubectl drain so pod disruption budgets are honored. The version skew policy lets old kubelets talk to a new apiserver, so your workloads never notice. Downtime during upgrades comes from your workloads (single replicas, no PDBs, ignoring SIGTERM), not from Kubernetes. Upgrade every quarter so you never have to multi-hop, and hunt removed APIs before you touch anything.
The version skew rules that make this safe
Kubernetes is explicitly designed for mixed-version operation during upgrades, and the rules are worth internalizing because they dictate your order of operations:
- The kubelet must never be newer than the kube-apiserver, and may lag it by up to three minor versions (this was two, until 1.28 relaxed it to three).
- kubectl is supported within one minor version of the apiserver, either direction.
- Control plane components (scheduler, controller-manager) upgrade with the apiserver.
So the sequence writes itself: control plane first (it can talk to old kubelets), then nodes at their leisure. Your workloads keep running the whole time because a 1.33 kubelet happily reports to a 1.34 apiserver. What you must never do is the reverse -- a shiny new kubelet against an old apiserver is unsupported and bites in creative ways.
Before you touch anything: deprecation hunting and backups
Every minor release removes some API or behavior. The upgrade that goes sideways is rarely the one where a node fails; it's the one where your Ingress manifests reference networking.k8s.io/v1beta1 and the new apiserver refuses to admit them, or your PSPs vanish because PodSecurityPolicies were removed back in 1.25 and your admission setup assumed otherwise.
Two tools find this before the upgrade does. kubectl itself warns at apply time, but for a full sweep, run Pluto or kubent against the cluster:
$ kubent
11:04AM INF >>> Kube No Trouble `kubent` <<<
11:04AM INF Retrieved 142 resources from collector name=Cluster
______________________________________________________________________
>>> 1.32 Removed APIs <<<
----------------------------------------------------------------------
KIND NAMESPACE NAME API_VERSION
FlowSchema <undefined> my-old-flowschema flowcontrol.apiserver.k8s.io/v1beta3Fix everything it finds, then read the release notes' "Urgent Upgrade Notes" section -- actually read it, not skim -- because that's where "the flag you rely on is gone" lives. And take an etcd snapshot before a self-managed control plane upgrade. ETCDCTL_API=3 etcdctl snapshot save takes ten seconds and converts a catastrophic upgrade into a rollback. On EKS/GKE/AKS the provider owns etcd, but I still snapshot any cluster-state that lives outside Git.
Draining like you mean it
This is where my Friday went wrong. Node replacement should always go through the eviction API:
$ kubectl drain ip-10-0-3-87 --ignore-daemonsets --delete-emptydir-data
node/ip-10-0-3-87 cordoned
evicting pod payments/payments-api-6f8b9c5d7-qx2mt
evicting pod default/netshoot-4f2k9
...
node/ip-10-0-3-87 draineddrain cordons the node (no new pods), then evicts each pod *politely*: SIGTERM, grace period, and -- the critical part -- it respects PodDisruptionBudgets. If evicting this pod would drop payments-api below minAvailable: 2, the eviction blocks until capacity exists elsewhere. That's the difference between a rolling upgrade and my dinner-cancelling instance massacre.
Of course, PDBs only protect you if they exist and aren't themselves broken. Before any upgrade I sweep for the two failure modes: workloads with one replica and no PDB (they *will* blip during drain -- decide if that's acceptable), and PDBs that are too strict to ever allow eviction (minAvailable: 3 on a 3-replica deployment means drain hangs forever):
$ kubectl get pdb -A
NAMESPACE NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS
payments payments-api 2 N/A 1
search search-api N/A 50% 2
batch importer 3 N/A 0That last row is a drain-blocker. ALLOWED DISRUPTIONS: 0 on every PDB is the classic "why is my node upgrade stuck" answer. And the application side matters as much: a pod that takes 45 seconds to finish in-flight requests after SIGTERM needs terminationGracePeriodSeconds to match, or the drain's politeness is wasted on a process that ignores it.
Managed vs self-managed: same dance, different partner
On EKS, GKE, or AKS, the control plane upgrade is a button (or an API call), and it's in-place and HA, so workloads never notice. Your job shrinks to: click the button, wait, then roll the node groups. Managed node groups and Karpenter can even do the cordon-drain-replace dance for you, and they do respect PDBs these days -- though I'd still verify ALLOWED DISRUPTIONS first, because a stuck PDB turns a managed node upgrade into a 30-minute timeout and a support-flavored afternoon.
What managed services *don't* do for you: the add-ons. EKS's VPC CNI, kube-proxy, and CoreDNS have their own version compatibility matrix against the control plane, and upgrading the cluster without bumping the CNI is a classic way to inherit weird networking bugs. Check the add-on versions as part of the same change, not as an afterthought.
Self-managed with kubeadm, you run the control plane yourself: kubeadm upgrade plan, then kubeadm upgrade apply v1.34.x on the first control plane node, kubeadm upgrade node on the rest, then upgrade kubelets node by node with drains in between. It's genuinely well-documented and reliable -- the docs' per-version pages are excellent -- it's just more ceremony, and when something stalls, there's no support plan, there's you and the etcd snapshot you hopefully took. During all of this I keep a live view of the cluster open (conndeck, in my case) mostly to watch pods reschedule as nodes drain -- "did the PDB block" is a question your eyes answer faster than your terminal does.
The part people skip: practicing the rollback
Rollback for a node pool is easy -- keep the old node group around until the new one is proven, then scale down. Rollback for a self-managed control plane is "restore etcd and downgrade binaries," which is exactly as fun as it sounds, which is why you snapshot. On managed services, control plane rollback isn't a thing at all; your mitigation is that you tested the target version in staging first.
Which is the real takeaway about process: the production upgrade should be the *second* time you run it. Staging on the new version for a week flushes out the deprecated APIs you missed, the admission webhook that rejects the new pod spec fields, the ingress controller that needs a bump. Production then becomes a rerun of a thing you've already seen succeed.
Frequently asked questions
How often should I upgrade Kubernetes?
Often enough that you never jump more than one or two minor versions at once, which in practice means every two to four months. Kubernetes supports three minor releases at a time, so if you fall more than two versions behind you're forced into a multi-hop upgrade, which is where the real risk lives. Teams that upgrade every quarter treat it as routine maintenance; teams that upgrade once a year treat it as a project, and the project is what hurts.
What is the Kubernetes version skew policy?
It's the rule for which component versions can coexist during an upgrade. The kubelet may not be newer than the kube-apiserver and can lag it by up to three minor versions, kubectl is supported within one minor version of the apiserver either way, and the control plane components like the scheduler must match the apiserver's minor version. In practice this means you upgrade the control plane first, then the nodes, and you never let a worker run a kubelet newer than the API it talks to.
Will upgrading my cluster cause downtime?
Not if your workloads are built for it. The control plane upgrade on a highly available or managed cluster is invisible to workloads, and node upgrades are just rolling drains. Downtime during upgrades almost always comes from single-replica deployments, missing pod disruption budgets, or apps that don't handle SIGTERM gracefully, not from Kubernetes itself. Fix those three things and upgrades become boring, which is the goal.
Should I drain nodes or just let them get replaced?
Drain them, always. kubectl drain cordons the node, evicts pods politely through the eviction API, and crucially respects pod disruption budgets, so you can't accidentally take down the only replica of something. Just terminating instances -- which some autoscaler and node-pool flows do by default -- bypasses all of that and yanks pods out from under live traffic. Set --ignore-daemonsets and --delete-emptydir-data and drain is safe as well as polite.
Is upgrading managed Kubernetes like EKS easier than self-managed?
Yes, but less than the marketing suggests. The managed service upgrades the control plane for you, which removes the scariest third of the job, but you still own the node groups, the add-ons like the VPC CNI and CoreDNS whose versions must stay compatible, and all the workload-side readiness. Self-managed with kubeadm means you also run the control plane upgrade yourself, which is well-documented and reliable, just more ceremony and more your problem when it stalls.
The field-notes version
Upgrades are safe by design -- version skew exists precisely so old kubelets survive a new control plane. The drama always comes from skipping the preparation: unhunted deprecations, PDBs that don't exist or don't allow disruption, nodes terminated instead of drained, no etcd snapshot.
Do it quarterly, do staging first, drain everything, and read the urgent upgrade notes. The best upgrade is the one nobody in the company notices happened -- including, ideally, you.