conndeck Download

conndeck blog

Node NotReady at 2AM: A Recovery Walkthrough

#kubernetes#nodes#troubleshooting#incident-response

The page came in at 2:13 AM: node prod-worker-17 NotReady. Then, twelve seconds later, a second one. Then silence.

By 2:20 I was at the kitchen table with a laptop, watching kubectl get nodes redraw. Two nodes NotReady, forty pods in various states of Unknown, and -- this is the part that mattered -- exactly zero customer-facing alerts. Every service on those nodes had replicas elsewhere. Which bought me the one thing you want at 2 AM: time to think instead of time to panic.

What follows is the walkthrough I ran that night, in the order I ran it.

Quick answer: NotReady means the kubelet stopped reporting health to the API server -- it does not automatically mean the node or its pods are dead. Start with kubectl describe node and read the Conditions and Events to see whether it's disk pressure, memory pressure, or a plain heartbeat loss. Then triage the cause: kubelet or runtime down (restart it), disk full (free space), network partition (pods are probably still serving -- consider waiting), or a dead VM (drain and replace). Cordon first if you're unsure; drain only when you've decided the node isn't coming back.

What NotReady actually means

Every kubelet posts a status heartbeat to the API server roughly every ten seconds. When the node controller stops hearing from one, it marks the node's Ready condition Unknown after about forty seconds, and kubectl get nodes shows NotReady. That's it. NotReady is a statement about the *heartbeat*, not about the node, the pods, or your workloads.

This distinction matters enormously, because the common causes split into two very different families:

  • The node is actually broken. Kubelet crashed, container runtime hung, disk full, kernel panicked, VM preempted. Pods on it are dead or dying.
  • The node is fine but unreachable. A network partition between node and API server, an API server overload dropping updates, expired kubelet certificates. Pods on it may be serving traffic perfectly while the control plane insists they're gone.

Meanwhile, the eviction clock starts: the node controller taints the node node.kubernetes.io/not-ready:NoExecute, and pods ride it out for their tolerationSeconds -- 300 by default -- before eviction. Five minutes of grace you should use to diagnose, not to start deleting things.

Triage from the API side

Before touching the node itself, everything kubectl can tell you:

$ kubectl describe node prod-worker-17
...
Conditions:
  Type             Status    Reason                  Message
  ----             ------    ------                  -------
  Ready            Unknown   NodeStatusUnknown       Kubelet stopped posting node status.
  MemoryPressure   Unknown   NodeStatusUnknown       Kubelet stopped posting node status.
  DiskPressure     Unknown   NodeStatusUnknown       Kubelet stopped posting node status.
...
Events:
  Type     Reason                   Age    From             Message
  ----     ------                   ----   ----             -------
  Normal   NodeNotReady             9m     node-controller  Node prod-worker-17 status is now: NodeNotReady

Read Conditions first. If DiskPressure or MemoryPressure had been True in the last reported status, you'd see it in the node's recent history and events, and you'd have your answer: something on the box filled the disk or ate the memory. All-Unknown with "stopped posting node status" is a heartbeat loss -- kubelet down or network gone.

Then check the blast radius and the stranded workloads:

$ kubectl get pods -A -o wide --field-selector spec.nodeName=prod-worker-17

And one critical cloud-era check: open the cloud console. Spot and preemptible instances vanish without saying goodbye, and I've watched people SSH-debug a VM that no longer existed. Two nodes going NotReady within seconds of each other, like mine did, smells like shared infrastructure -- same rack, same zone, same underlying event.

On the box

If the node is reachable, get on it and work this order:

$ systemctl status kubelet
$ journalctl -u kubelet --since "1 hour ago" | tail -50
$ crictl ps
$ df -h / /var/lib/kubelet /var/log

The hits I've taken most often, in order. Disk full: /var/lib/kubelet or /var/log at 100 percent -- usually one container logging gigabytes to its writable layer. The kubelet detects the pressure, taints the node, starts evicting, and can wedge entirely. Free space, restart the kubelet, and it recovers. Runtime hung: crictl ps times out, the kubelet's PLEG errors fill the journal ("PLEG is not healthy"), restart the container runtime and then the kubelet. Expired certificates: the kubelet's client cert lapsed and the API server rejects its heartbeats -- the journal shows 401s. Fix the cert rotation, which is easier to say than to do at 2 AM, and is an argument for not disabling auto-rotation.

If the node is unreachable entirely, your diagnosis is the cloud console plus whatever out-of-band access you have. Don't improvise heroics.

Cordon, drain, or wait

This is the actual decision, and 2 AM brains are bad at it, so decide by rule:

Wait when the cause looks transient, the pods might still be serving, and replicas elsewhere are carrying the load. My two nodes turned out to be a network partition in one zone -- the boxes were fine, their pods were answering requests, and only the control-plane path was cut. Draining would have evicted healthy pods for nothing. I cordoned both nodes (kubectl cordon prod-worker-17) so nothing new would schedule onto a node I couldn't observe, then waited. The partition cleared at 2:41, the kubelets re-registered, the Unknowns flipped back to Running, and I uncordoned. Total user impact: zero.

Drain when the node is confirmed dead or needs intervention: kubectl drain prod-worker-17 --ignore-daemonsets --delete-emptydir-data. Drain respects PodDisruptionBudgets, which is exactly why you use it instead of deleting pods by hand. If drain hangs, read the PDB -- it's telling you evicting would drop you below minimum availability, and that's worth listening to.

Cordon alone is the underrated middle move: instantly reversible, zero disruption, stops the scheduler from feeding a sick node. When in doubt, cordon and observe. Watching node conditions and pod placement shift in real time -- full disclosure, the view we built into conndeck -- makes the "is it recovering?" question answerable at a glance instead of by re-running describe in a loop.

Frequently asked questions

What causes a Kubernetes node to go NotReady?

NotReady means the kubelet has stopped reporting a healthy status to the API server. The common causes are the kubelet or container runtime being down, disk pressure from a full filesystem, a network partition between the node and the API server, or the underlying VM being stopped or preempted by the cloud provider. Run kubectl describe node and read the Conditions and Events before doing anything else.

What happens to pods when a node goes NotReady?

Nothing immediately. After the kubelet misses its status updates, the node controller marks the node condition Unknown and adds the node.kubernetes.io/not-ready or unreachable taint. Pods tolerate that taint for 300 seconds by default, and only then are they evicted and rescheduled elsewhere. Stateful workloads and pods with custom tolerations may stay put much longer.

How do I recover a NotReady node?

First check from the API side with kubectl describe node for pressure conditions and events. Then get on the box if you can: systemctl status kubelet, journalctl -u kubelet, crictl ps to test the runtime, and df -h to check disk. Most recoveries are restarting the kubelet or runtime, or freeing disk space. If the box is unreachable, check the cloud console for a stopped or preempted VM.

When should I drain a node instead of waiting for it to come back?

Wait if the cause looks transient, the node's pods are still serving, and your workloads have enough replicas elsewhere, because a network-partitioned node often recovers on its own. Drain when the node is confirmed dead or needs maintenance, since draining evicts pods cleanly and respects PodDisruptionBudgets. If you are unsure, cordon first: it stops new scheduling immediately without evicting anything.

What is the difference between kubectl cordon and kubectl drain?

Cordon marks the node unschedulable so no new pods land on it, but everything already running stays put. Drain does a cordon and then evicts the existing pods, respecting PodDisruptionBudgets and needing flags like --ignore-daemonsets for DaemonSet pods. Cordon is the reversible, low-risk move; drain is the one that moves workloads, so use it when you actually intend to empty the node.

What I do differently now

NotReady is a heartbeat problem before it's anything else, and half the time the node -- and everything on it -- is alive and serving. Read the Conditions, check whether the VM still exists, and let the five-minute eviction grace work for you instead of against you.

Cordon early, drain deliberately, wait when the evidence says wait. The best 2 AM outcome is the one where you did almost nothing, on purpose, and were right.