Liveness, Readiness, Startup Probes: What I Actually Set

I turned off a client's liveness probe during an incident last spring, and it was the single most effective fix we shipped that week. Their app pods had been restarting in a loop for forty minutes: every restart dropped in-flight requests, every cold start made the next health check slower to answer, and the loop fed itself. The probe was "working." The restarts were the outage.

That incident rewired how I think about the three probe types. They look interchangeable in the docs -- all HTTP GETs to a /healthz-style endpoint -- but they gate completely different machinery, and picking the wrong one turns a blip into a self-inflicted restart storm.

Quick answer: Readiness controls traffic (failing removes the pod from Service endpoints, nothing restarts). Liveness controls lifecycle (failing makes the kubelet restart the container). Startup gates the other two during boot. My default setup: readiness on almost everything, startupProbe for anything with a slow or variable boot, and liveness only where a real deadlock story exists. Liveness checking your database is how restart storms are born.

Three probes, three different jobs

The clearest way I've found to explain this to teams is by what each probe *pulls*:

Probe	What failing it triggers	Use it for
readinessProbe	Pod removed from Service endpoints; container keeps running	Anything that recovers on its own: dependency blips, cache warmup, overload shedding
livenessProbe	Kubelet restarts the container	Deadlocks, hung event loops, states the process cannot exit by itself
startupProbe	Suppresses liveness/readiness until it succeeds once	Variable boot times; replaces initialDelaySeconds guessing

The asymmetry that matters: readiness failure is cheap, liveness failure is a restart. So the direction of my advice is fixed -- when in doubt, move a check from liveness to readiness. A pod that is alive but serving errors is a readiness problem. The Service routes around it, and humans get paged with real information instead of watching CrashLoopBackOff counts climb.

If you want a litmus test for liveness: name the deadlock it catches. If you cannot describe the stuck state the process can't escape, you don't have a liveness use case, you have a restart ritual.

The restart storm, dissected

Here's the failure mode that keeps biting teams, drawn from that spring incident. The app had a liveness probe hitting /health with all the defaults:

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  # periodSeconds: 10, timeoutSeconds: 1, failureThreshold: 3 (defaults)

The /health endpoint checked database connectivity, because "health" felt like the right place for it. Then the database had a thirty-second failover. Watch the cascade:

1. DB is unreachable; /health starts returning 500s.
2. Three failed checks at ten-second intervals, kubelet restarts the pod.
3. Every pod in the Deployment fails on roughly the same schedule, because they share the database.
4. Restarted pods reopen connection pools at once, hammering the recovering database.
5. Cold-started pods miss their first health checks too, so some loop a second time.

One database blip produced a rolling restart of the entire fleet, during the exact window when steady connections would have let it recover fastest. The fix wasn't tuning. It was deleting the DB check from /health, keeping it on /ready, and pointing liveness at a trivial "is the event loop alive" endpoint that only fails if the process is genuinely wedged.

The settings I actually change

Defaults are tuned for "never kill anything by accident," which makes them weak for busy production pods. The two I adjust most:

readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2
startupProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5
  failureThreshold: 60   # up to 5 minutes of boot time

• timeoutSeconds is the most under-appreciated knob. The default of one second is brutal for JVM warmups, Python apps importing heavyweight modules, or anything answering during a GC pause. Most "flapping probe" mysteries dissolve when the timeout becomes realistic.
• startupProbe kills the initialDelaySeconds ritual entirely. Instead of guessing "boots in about 45 seconds, delay 60 to be safe," you let the app report readiness when it's actually ready, and the failureThreshold becomes your hard boot deadline. If a pod truly cannot boot in five minutes, restarting it is the correct outcome anyway.
• failureThreshold on readiness controls how quickly a pod drops out of rotation. Two failures at five-second periods means a pod leaves the endpoints within about ten seconds of going bad, which is usually the right sensitivity for user-facing traffic.

One subtlety people trip on: readiness does not gate the *lifecycle*, only the *routing*. A pod that never turns ready still holds its resources, still counts toward your HPA replica math, and still blocks a rollout if it's the new pod. That's why a permanently-unready state needs alerting (see my notes on alerting on symptoms over causes), not just probe config.

Probe endpoints that respect their job

The endpoint design matters more than the YAML. Rules I hold teams to:

• /live (liveness): no dependencies, no locks, no disk. Return 200 if the event loop or main thread can execute code at all. This endpoint should be impossible to fail during someone else's incident.
• /ready (readiness): dependency checks allowed, plus "am I absorbing traffic I shouldn't." This is also where you can implement self-preservation: return 503 when your own queue depth or memory crosses a line, and load balancers stop sending you work you'd drop anyway.
• Cache the result if the check is expensive. A readiness endpoint that queries the database on every probe at five-second periods, times two hundred pods, is its own little DoS against your database.

For gRPC services, use the native gRPC probe (grpc: probe type) instead of an HTTP shim; for anything with heavy boot work, make readiness flip false *before* you start tearing down state on shutdown, so endpoints update while the pod can still finish in-flight requests. The shutdown sequence deserves its own post, but the probe half is: go unready first, then drain.

Debugging a probe that's "lying"

When a pod is restarting and the probe is blamed, I check three things in order, usually straight from the pod view in conndeck because scanning events across a fleet in a terminal gets old fast:

1. What did the probe actually receive? kubectl describe pod shows Readiness probe failed: HTTP probe failed with statuscode: 500 or a timeout distinction. A 500 is the app speaking; a timeout is the app not speaking at all. Those are different bugs.
2. Does the failure correlate with load or with time? Failures clustering at deploys point to cold-start misses; clustering at traffic peaks point to timeout math; perfectly periodic failures point to a dependency's own probe cadence.
3. Is the endpoint checking anything besides the process? If yes, that's your answer, move it to readiness.

The pod-stuck-pending sibling problem is scheduling refusing to place your pod; probe failures are the runtime refusing to trust it. Different layers, same debugging discipline: read the exact failure reason, don't pattern-match on the symptom name.

Frequently asked questions

What is the difference between liveness and readiness probes?

A failing liveness probe makes the kubelet restart the container. A failing readiness probe just removes the pod from Service endpoints while the container keeps running. Use readiness for anything that can recover on its own, and reserve liveness for states the process cannot escape without a restart.

Should a liveness probe check database connectivity?

No. If the database blips for thirty seconds and your liveness endpoint reports unhealthy, the kubelet restarts every app pod at once, turning one incident into two. Put dependency checks in readiness if you want traffic routed away, and keep liveness about the process itself.

When should I use a startupProbe instead of initialDelaySeconds?

Whenever your app has a variable boot time. A startupProbe delays liveness and readiness evaluations until the app reports ready once, so a slow boot never triggers a restart loop the way a fixed initialDelaySeconds guess does. Set failureThreshold high enough to cover your worst boot, not your average one.

Why did my liveness probe fail during a deploy even though the app was fine?

Mostly timeout math: the default probe timeout is one second, and a pod that is busy warming caches, JIT-compiling, or absorbing a traffic shift can easily miss it three times in a row, which is the default failureThreshold. Raise timeoutSeconds before you raise thresholds blindly, and check whether the failure correlates with load spikes.

The probe setup I actually ship

Readiness everywhere, startupProbe on anything with a real boot, liveness only with a named deadlock attached. Dependency checks live in /ready, never /live, and probe timeouts reflect what the endpoint can honestly answer under load.

Probe config is also one of those places where seeing the whole fleet at once pays for itself: the one Deployment someone configured in 2023 with a one-second timeout and a dependency-checking liveness endpoint is invisible until you sort pods by restart count. Whatever tool you use for that, make restart-count anomalies a first-class view, not a kubectl incantation.