Node Pressure Evictions: When the Kubelet Starts Killing

Q: What is the difference between Evicted and OOMKilled in Kubernetes?

An eviction is the kubelet proactively terminating pods when node resources (memory, disk, PIDs) approach exhaustion, before the kernel has to act. OOMKilled is the kernel's cgroup OOM killer terminating a single container that exceeded its memory limit. Eviction is a node-level pressure response; OOMKill is a container-level limit breach.

Q: Why do evicted pods stay in the pod list forever?

Evicted pods end in phase Failed with reason Evicted, and nothing deletes them automatically. They are tombstones, not workloads, and they accumulate until something cleans them. Most teams run a periodic job that deletes Failed pods older than an hour, or handle them from their tooling, so the pod list stays readable during real incidents.

Q: How does the kubelet choose which pods to evict?

It orders candidates by quality of service and over-commit: BestEffort pods first, then Burstable pods using more than their requests, and Guaranteed pods last, with pod priority breaking ties. Within the same class, the pod whose usage exceeds its request by the most goes first. This is why requests and limits are an availability feature, not a cost setting.

Q: What does 'The node was low on resource: memory' actually mean?

It is the event the kubelet emits when it starts evicting because available memory crossed a threshold, either the configured soft threshold held past its grace period or a hard threshold. Read it as: the node was about to run out, and pods were killed so the node itself would survive. The interesting question is always which workload drove usage over the line.

The first eviction wave I ever debugged took out our monitoring stack before it took out anything we could measure with. A log-hungry service had been writing to container stdout faster than rotation could keep up, node disk crossed the kubelet's eviction threshold, and the kubelet started choosing victims in strict order of least importance. BestEffort pods went first. Our metrics and log shippers were, embarrassingly, the least-requested pods in the cluster. The nodes survived. Our visibility into the surviving nodes did not.

Evictions are Kubernetes working as designed -- the kubelet killing workloads in a considered order so the node doesn't die chaos-style -- but "considered order" only matches "the order you'd choose" if you've configured requests, limits, and priorities with eviction in mind. Most clusters haven't, until the night they learn.

Quick answer: Eviction is the kubelet's proactive response to node pressure (memory, nodefs, imagefs, PIDs); OOMKill is the kernel enforcing a container's memory limit. The kubelet evicts BestEffort pods first, then Burstable pods over their requests, and touches Guaranteed pods last. Fix the pod list clutter with a Failed-pod reaper, fix the *cause* with real requests, limits, and log rotation, and alert on eviction rate -- it's your earliest signal of a node heading for trouble.

Eviction vs OOMKill: two different killers

These get conflated constantly, and the confusion costs debugging time because the remedies differ:

	Eviction	OOMKill
Who kills	kubelet (userspace)	kernel cgroup OOM killer
Trigger	node resource thresholds	container exceeds its memory limit
Scope	any pod on the node, by ranking	the specific container that overflowed
Pod status	`Evicted` (phase Failed)	`OOMKilled` (last state, restarts per policy)
Remedy	node capacity, requests/limits, QoS, cleanup	raise the limit or fix the leak

The deep detail people miss: a Guaranteed pod is not safe from eviction. If node pressure is severe enough, the kubelet will evict even pods using exactly their requests, in least-harm order. What Guaranteed status (requests equal limits on everything) buys is *last place* in the ranking, not immunity. And conversely, a pod using under its request can still be OOMKilled if its own limit is set below its real working set -- that's the container-level killer, unrelated to eviction. The OOMKilled debugging and requests/limits sizing posts cover that side of the fence.

The four signals and their thresholds

The kubelet watches several resources, and each maps to a different operational smell:

memory.available -- the headline signal. Hard defaults around 100Mi free; soft thresholds (like memory.available<1Gi with a grace period) let you act early.
nodefs -- the filesystem holding container logs and the kubelet's own scratch. This is the one log-heavy apps kill.
imagefs -- where container images live; when this runs low the kubelet garbage-collects unused images before evicting anything.
pid.available -- process count pressure, the quiet one that hits hosts with low PID limits and runaway apps.

Soft versus hard matters operationally. A soft threshold (eviction-soft, with eviction-soft-grace-period) is an early-warning tripwire: cross it, stay crossed for the grace period, and evictions begin. Hard thresholds (eviction-hard) fire immediately. Managed offerings set their own defaults here, so read yours rather than assuming -- and note the node surfaces this state publicly as conditions like MemoryPressure and DiskPressure, which is what well-behaved schedulers check before placing new pods on the node.

The victim ranking, and why requests are an availability feature

When pressure hits, the kubelet ranks evictable pods roughly like this:

BestEffort (no requests, no limits) -- first to go, always.
Burstable above request (requests set, usage exceeding them) -- next, ordered by how far over request they are.
Burstable under request and Guaranteed -- last, tie-broken by pod priority.

Read that list again as an incentive system. Pods without requests are volunteering to die first. Pods routinely exceeding requests are second volunteers. The ranking is why "requests are for the scheduler, limits are for safety" is a dangerously incomplete mental model: requests are your eviction insurance, and they feed the QoS classes that decide the night everything goes wrong.

Pod priority slots into this as a tie-breaker within the same class, which is exactly how our monitoring stack should have protected itself: same Burstable class as everyone else, but a high PriorityClass would have pushed it behind the app pods in the ranking. We fixed that the following week, with priority classes for observability and ingress-critical pods.

The log-disk incident, post-mortem in brief

Back to the original sin. The chain was: app writes verbose logs to stdout, container runtime writes them to nodefs with modest rotation limits, nodefs crosses the soft threshold on forty-odd nodes at roughly the same time (same app, same traffic pattern), evictions start cluster-wide by ranking. Monitoring died first (BestEffort), then stateless apps churned, and the nodes themselves stayed up -- so every dashboard we had said "nodes: fine."

The fixes, in the order I'd apply them today:

Log hygiene first: container log rotation limits (containerLogMaxSize on the kubelet) and an app-side verbosity setting that isn't "everything, forever." Disk pressure from logs is the most preventable eviction cause there is.
Requests on everything, so BestEffort stops being our default pod shape. This one change alone would have saved the dashboards.
PriorityClass for observability and edge pods, so the ranking's last-resort tier contains exactly the pods we'd choose last.
Alert on eviction rate, not just memory: kube_evictions_total climbing on any node is an earlier and more specific signal than node memory graphs, and it captures the disk-pressure path that memory alerts miss.

Cleanup of the tombstones was the mundane half: evicted pods sit in Failed phase forever unless deleted, and after a wave you're staring at hundreds of Evicted entries that make real pod lists unreadable during the next incident. A nightly kubectl delete pods --field-selector=status.phase=Failed -A job (or equivalent in your tooling) keeps the list honest. When I'm triaging an active incident, I want the pod view showing me workloads, not archaeology -- and being able to sort by restart and eviction signals in one place (conndeck does this, which is partly why the fix list above is so specific) turns eviction waves from mysteries into follow-the-signal work.

Frequently asked questions

What is the difference between Evicted and OOMKilled in Kubernetes?

An eviction is the kubelet proactively terminating pods when node resources (memory, disk, PIDs) approach exhaustion, before the kernel has to act. OOMKilled is the kernel's cgroup OOM killer terminating a single container that exceeded its memory limit. Eviction is a node-level pressure response; OOMKill is a container-level limit breach.

Why do evicted pods stay in the pod list forever?

Evicted pods end in phase Failed with reason Evicted, and nothing deletes them automatically. They are tombstones, not workloads, and they accumulate until something cleans them. Most teams run a periodic job that deletes Failed pods older than an hour, or handle them from their tooling, so the pod list stays readable during real incidents.

How does the kubelet choose which pods to evict?

It orders candidates by quality of service and over-commit: BestEffort pods first, then Burstable pods using more than their requests, and Guaranteed pods last, with pod priority breaking ties. Within the same class, the pod whose usage exceeds its request by the most goes first. This is why requests and limits are an availability feature, not a cost setting.

What does 'The node was low on resource: memory' actually mean?

It is the event the kubelet emits when it starts evicting because available memory crossed a threshold, either the configured soft threshold held past its grace period or a hard threshold. Read it as: the node was about to run out, and pods were killed so the node itself would survive. The interesting question is always which workload drove usage over the line.

What I watch now

Eviction rate per node, node conditions (MemoryPressure, DiskPressure), and the QoS distribution of every namespace -- because the ranking that will decide your next incident is determined by YAML that was merged months earlier, on a quiet afternoon, by someone thinking about cost rather than casualties.

The consolation is that eviction math is fully legible. Unlike so many failure modes in distributed systems, this one publishes its rules in advance and follows them exactly. Clusters that take the rules seriously -- requests everywhere, priorities for the precious few, rotation for the loud -- turn eviction from a 3am surprise into a boring, working safety valve. That's the standard I hold now, and it took exactly one blind monitoring outage to adopt it.