CrashLoopBackOff: A Field Guide From Someone Who's Been There

It was 11:40 on a Tuesday night when I watched a pod die for the forty-seventh time.

The deployment had rolled out fine at 4pm. Green across the board, everyone went home. Then the liveness probe started failing, the kubelet started restarting the container, and by the time my phone buzzed, the restart counter looked like an odometer. kubectl get pods showed the familiar, taunting status: CrashLoopBackOff. Forty-seven restarts. I knew, with the particular certainty that only comes from having been wrong about this before, that the fix was going to be a single wrong number somewhere.

It was. I'll tell you which number at the end.

CrashLoopBackOff is the status Kubernetes reports when a container keeps exiting and the kubelet keeps restarting it with an exponential backoff, up to five minutes between attempts. That's all it is. It's not a diagnosis. It's a symptom that means "something is wrong and I don't know what." The good news: it's almost always one of five things. This is the field guide I wish someone had handed me years ago.

Quick answer: CrashLoopBackOff means a container keeps exiting and the kubelet keeps restarting it with a growing backoff, up to five minutes between attempts. It's almost always one of five causes: bad config, a missing Secret or ConfigMap, an OOM kill, a failing liveness probe, or a broken entrypoint. Run kubectl logs <pod> --previous to see the last dead container's output and kubectl describe pod to check the Last State reason, exit code, and Events — those two commands tell you which of the five you're dealing with.

Read the restart count first

Before you run anything else, look at the pod list:

$ kubectl get pods -n payments
NAME                          READY   STATUS             RESTARTS        AGE
ledger-api-7d9f4b8c6-x2k4p    0/1     CrashLoopBackOff   47 (4m ago)     7h

Two things matter here. The total restart count, and the number in parentheses, which tells you how long ago the last restart happened. A pod with 47 restarts and a last restart 4 minutes ago is still actively dying. A pod with 47 restarts and nothing for 40 minutes has recovered on its own or someone fixed it. People skip this step and go straight to logs, then waste ten minutes debugging a problem that stopped happening an hour ago.

The restart count also tells you the shape of the failure. A pod that dies every 10 seconds is failing fast: bad config, missing file, command not found. A pod that lives for 90 seconds and then dies is usually getting killed by something, and that something is often a probe.

describe vs logs --previous

These are your two real tools, and they answer different questions.

kubectl logs <pod> --previous shows you stdout and stderr from the last dead container. That "previous" flag is the one everyone forgets, and without it you're reading the logs of a container that started three seconds ago, which is useless. If the app printed a stack trace before it died, it's in there. One caveat: if the container is currently in a backoff wait and has no running instance, plain kubectl logs will fail outright. --previous is almost always what you want.

kubectl describe pod <pod> is what you read when the logs are empty or unhelpful. Scroll past the spec to the Last State block:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Started:      Tue, 18 Aug 2026 23:31:12 +0200
  Finished:     Tue, 18 Aug 2026 23:32:48 +0200

The Reason and Exit Code here are gold. OOMKilled with exit code 137 means the kernel's OOM killer got it. Error with exit code 1 means the process exited on its own and probably told you why in its logs. Exit code 127 means command not found, which is almost always an entrypoint problem. Then keep scrolling to Events at the bottom. That's where Liveness probe failed: Get "http://:8081/healthz": dial tcp :8081: connect: connection refused lives, and it often tells you the whole story in one line.

My rule of thumb: logs --previous for application failures, describe for infrastructure failures. When in doubt, run both.

The five usual causes

After enough of these, you stop guessing and start pattern-matching. Here are the five I actually see, roughly in order of frequency.

Bad config. The app starts, tries to read its config, and exits with code 1. A malformed YAML in a ConfigMap, an env var pointing at a database host that doesn't resolve, a feature flag value the binary doesn't recognize. The logs --previous output usually screams at you: panic: unable to parse config: yaml: line 14: did not find expected key. Easy to spot, embarrassing to ship.

Missing secrets or configmaps. The pod spec references a Secret that doesn't exist in that namespace, or does exist but is missing a key the container mounts with optional: false. Sometimes the pod won't even start the container and sits in CreateContainerConfigError instead, but I've seen plenty of apps that start fine, then crash on first use of the missing value. Check kubectl get secrets -n <ns> and compare against what the deployment expects. Cross-namespace copies are the classic trap: the secret exists in staging, nobody applied it to prod.

OOM kills. Exit code 137, Reason: OOMKilled. Either the container's memory limit is too low for what the app actually needs, or there's a leak. To tell the difference, look at the Started and Finished timestamps in Last State. Dying within seconds of startup suggests the limit is just wrong (JVM apps are notorious: the heap doesn't respect cgroup limits on older JDKs without -XX:MaxRAMPercentage). Dying after minutes or hours under load suggests a leak. Fix the limit first to stop the bleeding, then figure out why it's growing.

Failing liveness probes. The app is actually fine, or would be if you left it alone, but the probe keeps failing so the kubelet keeps killing it. This is the nastiest category because the app logs look completely healthy. The tell is the event stream full of probe failures and a lifetime-per-restart that matches your probe timing suspiciously well. Check the probe's port, path, and initialDelaySeconds. And please, if your app takes 45 seconds to warm up, give it a startupProbe instead of cranking the liveness delay and praying.

Entrypoint mistakes. Exit code 127, or a log line like exec: "gunicorrn": executable file not found in $PATH. Someone fat-fingered the command in the Dockerfile or the command: array in the deployment, or built an image where the binary landed somewhere off the PATH. Also common: command and args confusion. Remember that in Kubernetes, command overrides the Docker ENTRYPOINT and args overrides CMD. I've seen people put flags in command and wonder why nothing runs.

The war story: the probe on the wrong port

Back to that Tuesday night. The ledger-api service had just been split into two containers by a well-meaning refactor: the API on 8080, and a new internal admin server on 8081 serving metrics and a /healthz. The deployment YAML got updated. Mostly.

The liveness probe still pointed at port 8080, at path /healthz. But /healthz had moved to the admin server on 8081. The API on 8080 returned a 404 for it. The probe had failureThreshold: 3, the app took about 80 seconds to come up under its normal load, and the probe started hammering it after a 10 second delay. Three 404s, kill, restart, repeat. Forty-seven times.

The fix was one line:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8081
  initialDelaySeconds: 15
  periodSeconds: 10

What made it painful was that the app logs were spotless. Every container logged a clean startup, served requests for a bit, then vanished. No error, no panic, nothing. The only evidence was in kubectl describe, in the Events section, one line of Liveness probe failed: ... 404. Nobody had scrolled that far. Two engineers, ninety minutes, one wrong number.

This is the class of problem where having restart counts, exit codes, and probe events visible in one place actually changes the outcome, which, full disclosure, is part of why we built conndeck. But honestly, kubectl describe and the discipline to read to the bottom of it would have solved this one too.

Frequently asked questions

What causes CrashLoopBackOff in Kubernetes?

Almost every CrashLoopBackOff is one of five things: bad config that makes the app exit on startup, a missing Secret or ConfigMap, an OOM kill because the memory limit is too low, a failing liveness probe killing a healthy app, or a broken entrypoint. Check the restart count and timing pattern first, then match it against those five before you start changing anything.

How do I check why a pod is in CrashLoopBackOff?

Run kubectl logs on the pod with the --previous flag to see output from the last dead container, and kubectl describe pod to read the Last State block and Events. The Reason and Exit Code in Last State usually name the cause directly, and the Events section at the bottom is where probe failures show up. Reading current logs without --previous is the most common mistake, because you end up staring at a container that started three seconds ago.

What does exit code 137 mean in Kubernetes?

Exit code 137 means the container was killed by SIGKILL, which in Kubernetes almost always means the kernel's OOM killer terminated it for exceeding its memory limit. You'll see Reason: OOMKilled in the Last State block of kubectl describe. Either raise the memory limit or go hunting for a leak, and the Started and Finished timestamps tell you which one it probably is.

Why does my pod keep restarting when the logs look fine?

A failing liveness probe is the usual culprit. The app itself is healthy, but the probe is pointing at the wrong port or path, or it starts before the app has warmed up, so the kubelet keeps killing and restarting a perfectly good container. Check the Events section of kubectl describe for probe failure lines and compare the pod's lifetime per restart against your probe timing.

What's the difference between command and args in Kubernetes?

In a pod spec, command overrides the Docker image's ENTRYPOINT and args overrides CMD. That's the reverse of what a lot of people assume, and mixing it up is a classic source of exit code 127, the command not found error. If you put flags in the command field, the container tries to execute your flags as a binary and fails immediately.

The takeaway

CrashLoopBackOff feels chaotic the first few dozen times you meet it. It isn't. It's five causes wearing a trench coat, and two commands tell you which one. Read the restart count, check Last State, pull the previous logs, and match the pattern before you touch anything.

And when the app logs are perfectly clean but the pod keeps dying, stop looking at the app. Look at the probe.