OOMKilled: Debugging Memory Limits Without Guessing

It was 2:20 on a Tuesday afternoon when the search-api pods started dying, and every graph said they shouldn't have.

Grafana showed the containers cruising at about 60 percent of their memory limit. Then, every twenty minutes or so, one would vanish and restart: exit code 137, Reason: OOMKilled. The graph line would drop to zero, a new pod would draw a fresh line at 60 percent, and the cycle repeated. Healthy-looking metrics, dead containers. I've been doing this long enough to know what that combination means: the metric and the kernel are measuring different things, and the kernel is right.

Quick answer: OOMKilled with exit code 137 means the kernel killed your container for exceeding its cgroup memory limit. Confirm it in kubectl describe pod under Last State, and make sure it says OOMKilled and not Evicted or Error -- those have different causes. If your metrics show memory below the limit, suspect sampling gaps (a spike between scrapes) or memory the kernel counts but your dashboard doesn't, like page cache. Then either raise the limit or profile the app's memory growth and fix the leak.

Exit 137, and the killers that aren't OOM

Exit code 137 is 128 + 9: the process received SIGKILL. In Kubernetes there are three common ways to get one, and people constantly conflate them.

The first is the cgroup OOM killer: the container's memory crossed its limit, the kernel picked a process inside that cgroup, and shot it. This is the real OOMKilled, and it looks like this:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Started:      Tue, 25 Aug 2026 14:02:11 +0200
  Finished:     Tue, 25 Aug 2026 14:22:47 +0200

The second is eviction: the *node* ran out of memory, and the kubelet evicted pods to protect itself. Those pods show Reason: Evicted and hang around as corpses until you clean them up. Different cause, different fix -- eviction is about node capacity and requests, not your container's limit.

The third is a plain SIGKILL from something else: kubectl delete pod, a node shutdown, a human. Those show Reason: Killed or no OOMKilled reason at all. If you're chasing exit 137 and the Last State doesn't say OOMKilled, stop tuning memory limits, because that's not your problem.

Requests, limits, and the QoS trap

Worth being precise, because the fuzzy mental model causes half the confusion:

resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "1Gi"

The request is a scheduling number. It reserves space on a node, and the scheduler treats it as spoken for forever. The limit is an enforcement number. It becomes the cgroup's memory ceiling, and crossing it by even a kilobyte gets you SIGKILLed. There is no grace period, no throttle, no warning. CPU limits throttle; memory limits kill.

Set a request with no limit and you get a pod that can balloon to the node's entire memory -- fine until the node runs dry, at which point the kubelet evicts, and it picks victims by QoS class and by how far each pod has exceeded its request. BestEffort pods (no requests at all) go first. Burstable pods running way over their request go next. This is why "we don't set limits, limits are dangerous" works great until one noisy neighbor eats a node.

"It died under the limit": the four usual explanations

Back to Tuesday. When the graph says 60 percent and the kernel says kill, it's one of these.

Your metrics missed the spike. kubectl top and most dashboards sample every 15 to 60 seconds. A container can go from 600Mi to over 1Gi and die in under a second -- one giant JSON payload parsed into memory, one unbounded query result. The graph interpolates the gap and looks serene. The tell: restarts correlate with specific requests or batch jobs, and the Started/Finished window in Last State is short.

The kernel counts more than you think. The cgroup limit applies to everything charged to that cgroup, including page cache the container's file writes generate. Apps that write logs or temp files to the container filesystem can get killed while their RSS looks modest, because the cache charged to them pushed the cgroup total over.

The JVM sized itself against the wrong number. Older or misconfigured JVMs either don't see the cgroup limit or see it and still blow past it, because the heap is only part of JVM memory. Metaspace, thread stacks, direct buffers, and JIT code cache all live outside Xmx. The fix on any remotely modern JDK:

-XX:MaxRAMPercentage=75.0

That caps the heap at 75 percent of the container limit and leaves the rest for non-heap. Setting Xmx1g on a 1Gi limit is asking the kernel to arbitrate, and the kernel always wins.

You're watching the wrong container. Multi-container pods die one container at a time. kubectl top pod --containers shows the sidecar quietly eating 900Mi while you stared at the app.

Hunting the actual leak

If restarts happen minutes to hours after start, under load, you have growth. Now find out what kind.

Watch the working set over a few hours -- container_memory_working_set_bytes in Prometheus is the metric the kubelet itself uses. A sawtooth that climbs and dies is a leak or an unbounded workload; a climb that plateaus means the limit is just too low, and raising it is the honest fix. Correlate the slope with request rate: memory growing proportionally with traffic smells like per-request allocation that isn't freed, not a slow background leak.

Then profile a live pod. Go services: expose pprof and grab kubectl port-forward + /debug/pprof/heap. JVM: a heap dump, or -XX:NativeMemoryTracking=summary if it's off-heap. Node: --inspect and a heap snapshot. Yes, this is actual work. It's also the difference between fixing the leak and playing limit-roulette every deploy.

My search-api turned out to be the first kind: a partner integration sent multi-hundred-megabyte payloads a few times an hour, the parser buffered the whole thing, and the spike lived and died between Prometheus scrapes. The fix was streaming the parse, plus a modest limit bump. Total memory tuning time once we stopped staring at averages: about an hour. Watching restart reasons and per-container memory side by side -- which, full disclosure, conndeck puts in one view -- is what made the pattern jump out.

Frequently asked questions

What does OOMKilled mean in Kubernetes?

OOMKilled means the Linux kernel's out-of-memory killer terminated your container because its cgroup exceeded its memory limit. You see it in the Last State block of kubectl describe pod alongside exit code 137, which is 128 plus 9, the SIGKILL signal. It is a hard kernel-enforced kill, not a Kubernetes decision, and it cannot be trapped or handled gracefully by your application.

Why did my pod get OOMKilled when memory was below the limit?

Almost always because your metrics are lying about timing. Tools like kubectl top sample periodically, and a container can spike from 60 percent to over the limit and die between samples, leaving the graph looking innocent. The kernel also counts more than RSS, including page cache charged to the container's cgroup, so the number you are watching may not be the number the kernel enforces.

What is the difference between memory requests and limits in Kubernetes?

The request is what the scheduler reserves on a node and what determines where the pod lands. The limit becomes the cgroup memory ceiling, and crossing it gets the container OOM-killed. A pod with no limit can use far more than its request, right up until the node itself runs out and the kubelet starts evicting pods, starting with the ones using the most memory relative to their requests.

How do I find a memory leak in a Kubernetes pod?

Watch container_memory_working_set_bytes in Prometheus or kubectl top over hours and look for sawtooth growth that ends in a restart. Correlate the growth rate with traffic or specific endpoints, then take a heap profile from a live pod: pprof for Go, a heap dump or Native Memory Tracking for the JVM, node --inspect for Node.js. Growth that never plateaus is a leak; growth that plateaus under load is just undersizing.

How do I stop a JVM from being OOMKilled in a container?

Modern JDKs read the cgroup limit, but they size the heap as a fraction of it and ignore everything else the JVM needs: metaspace, thread stacks, code cache, direct buffers. Set the heap explicitly with MaxRAMPercentage around 75 percent or a fixed Xmx well under the pod limit, and leave headroom for the non-heap memory. Never set Xmx equal to the container limit.

The rules I run with

OOMKilled is the kernel keeping its books, and its books don't match your dashboard. Confirm the Reason in Last State before anything else, because Evicted and plain SIGKILL are different incidents wearing the same exit code.

Then remember: memory limits don't throttle, they kill, and averages hide spikes. Profile the growth, fix the leak or right-size the limit, and let the JVM know what century it's running in.