conndeck Download

conndeck blog

Requests and Limits: Stop Copying Numbers From Stack Overflow

#kubernetes#resources#performance#rightsizing

At 2:20 on a Wednesday afternoon, our payment gateway started dropping transactions. Not all of them. Just enough that the error budget graph looked like it had been nibbled by mice.

CPU on the nodes: 30%. Memory: fine. Error logs: clean. It took me two hours and one very patient colleague to find it: container_cpu_cfs_throttled_seconds_total was climbing like a rocket on exactly one deployment. Somebody -- and I say this with love, because the somebody was me, six months earlier -- had copied limits: cpu: 500m from a Stack Overflow answer into a service that needed about two cores during burst windows. The nodes were idle. The pods were strangled. Kubernetes was doing precisely what the YAML said.

That day I stopped treating requests and limits as boilerplate and started treating them as the single highest-leverage performance knob in the whole system.

Quick answer: Requests tell the scheduler how much room to reserve and determine what you're guaranteed; limits cap what the container can use at runtime. Exceed the CPU limit and you get throttled (slow, sneaky); exceed the memory limit and you get OOM killed (fast, obvious). Set them from measurement, not folklore: observe real usage under load, set requests near typical usage and memory limits above peak, and think twice before setting tight CPU limits, because per-container throttling ignores idle capacity elsewhere on the node.

What requests and limits actually do

Strip away the YAML and there are two separate mechanisms:

Requests exist for the scheduler. When the scheduler places a pod, it sums container requests and looks for a node with that much *unallocated* capacity. Requests never throttle or kill anything by themselves. They are a reservation, and on an overcommitted node they're also your claim during contention: CPU time is divided proportionally to requests via cgroup weights.

Limits exist for the kubelet and the kernel. CPU limits translate to a CFS quota: 500m means 50ms of CPU per 100ms period, enforced per container. Memory limits translate to a cgroup memory ceiling: cross it and the OOM killer terminates the container with exit code 137.

resources:
  requests:
    cpu: 250m
    memory: 512Mi
  limits:
    cpu: "1"
    memory: 1Gi

Two consequences people miss. First, CPU limit enforcement is *per period*: an app that needs 2 cores for 30ms out of every 100ms gets throttled hard even while the node idles. Second, a pod using more memory than its *request* can be evicted under node pressure even below its *limit*. The request is the line that matters when the node gets tight.

Throttling vs OOM: the two failure modes

The failure asymmetry is the thing to internalize:

  • CPU overage = throttle. Silent-ish. Latency climbs, throughput sags, nothing crashes, no events fire. You find it in metrics or not at all.
  • Memory overage = kill. Loud. OOMKilled, exit code 137, restart counter increments, everyone gets paged.

This is why the "should I set CPU limits" debate exists. Memory limits are nearly always right -- an unbounded memory leak will eat a node, and the kill is the correct response. CPU limits are a tax: they protect against noisy neighbors, but they convert burstiness into latency, and the throttling metric counts throttling within each 100ms window in ways that surprise people ("we only use 60% of the limit on average!" -- yes, and you're throttled 40% of the periods).

My position, after the payment gateway incident: set CPU requests set memory limits always, and set CPU limits only where multi-tenant fairness genuinely demands it, sized well above burst needs. Plenty of good engineers disagree. Everyone agrees on measuring.

Measuring instead of guessing

The quick snapshot:

$ kubectl top pod -n payments --containers
POD                                  NAME       CPU(cores)   MEMORY(bytes)
gateway-7f9c5b4d6-h2kxm              gateway    1834m        812Mi
gateway-7f9c5b4d6-q9zvt              gateway    1712m        790Mi

There's your 500m limit being violated by a factor of four, in one command. But a snapshot lies about peaks, so the real workflow is:

  1. Run the workload under realistic load -- production traffic, a replay, or at minimum a load test that doesn't insult you.
  2. Pull percentiles over days, not minutes. max_over_time(container_memory_working_set_bytes{pod=~"gateway-.*"}[1d]) and the P95 of CPU usage rate.
  3. Set requests around P50-P75 of CPU usage and typical memory working set. Requests too high waste capacity; too low and you get scheduled onto crowded nodes and starve under contention.
  4. Set memory limits 10-25% above the P99 peak. Too tight is an OOM kill waiting for a traffic spike; too loose just delays the inevitable leak diagnosis.
  5. Set CPU limits, if you use them, above the burst profile, then watch container_cpu_cfs_throttled_seconds_total to verify you got it right.

Or let VPA in updateMode: "Off" do steps 1-4 for you and hand you target, lowerBound, and upperBound per container. It's the cheapest right-sizing audit in the ecosystem, and it's how I now sanity-check every number before it goes into Git.

The traps that get everyone

JVM and other runtimes vs cgroup limits. Older JDKs sized heap off node memory, not the container limit, which is how you get a pod with a 1Gi limit allocating a 4Gi heap and OOMing in minutes. Modern JDKs (8u191+, all of 11+) read cgroups correctly, but verify --max-ram-percentage behavior anyway, and remember the JVM needs headroom *above* heap for metaspace, threads, and JIT.

Guaranteed QoS by accident. Set requests exactly equal to limits and your pods become Guaranteed class -- last to be evicted, but also unable to burst a millicore past request. Some teams do this deliberately for latency-sensitive services. Fine. Just don't drift into it because someone copied a template where both fields said 500m.

initContainers have resources too. The scheduler computes effective requests as the max of init container requests versus the sum of app containers, whichever is higher. A data-loader init container asking for 4Gi makes your whole pod need a 4Gi hole to schedule, permanently.

Limits without requests, or vice versa. If a LimitRange in the namespace injects defaults, your "no resources set" pod may silently get a 100m CPU limit that throttles it into the ground. Check what's actually running: kubectl get pod -o jsonpath='{.spec.containers[*].resources}' doesn't lie, and seeing live usage against limits in one view -- the thing we built conndeck to make glanceable -- turns this from archaeology into a ten-second check.

Frequently asked questions

What is the difference between requests and limits in Kubernetes?

Requests are what the scheduler uses to place the pod and what you're guaranteed; limits are the ceiling the kubelet enforces at runtime. Exceed your CPU limit and you get throttled. Exceed your memory limit and the container gets OOM killed. A pod can also be evicted for using more than its request when the node runs out of memory, even if it's under its limit.

Should I set CPU limits on my pods?

It's a genuine debate. Limits protect nodes from noisy neighbors but cause throttling that shows up as weird latency spikes, especially for bursty or multi-threaded apps that exceed their limit in short bursts within a 100ms CFS period. A common middle ground is to set CPU requests for scheduling and either omit CPU limits or set them generously, while always setting memory limits since memory overuse is fatal rather than just slow.

How do I find the right CPU and memory values for my pods?

Measure, don't guess. Run the workload under realistic load, then look at kubectl top pod for a quick snapshot or your metrics backend for percentiles over days. Set requests near the P50 to P75 of real usage and memory limits a bit above the P99 peak. VPA in recommendation mode automates exactly this and gives you lower, target, and upper bounds per container.

Why is my pod being CPU throttled when the node has free CPU?

Because CPU limits are enforced per container via the kernel's CFS quota, regardless of idle capacity elsewhere on the node. A container limited to 500m gets at most half a core per 100ms period even if fifteen other cores are asleep. Check the container_cpu_cfs_throttled_seconds_total metric, and if throttling correlates with latency, raise or remove the CPU limit.

What happens if I don't set requests or limits at all?

The pod lands in the BestEffort QoS class, which makes it the first candidate for eviction under node memory pressure, and it can consume everything on the node with no ceiling. It also breaks CPU-based HPA, which needs requests to compute utilization. For anything that matters, set both requests and memory limits at minimum.

Where this bites in production

Requests are a reservation, limits are an enforcement mechanism, and neither one means what the average blog comment says it means. CPU overage makes you slow, memory overage makes you dead, and both failure modes are fully visible in metrics if you bother to look.

Measure real usage, set numbers from percentiles, keep memory limits tight-ish and CPU limits loose or absent, and never paste resource blocks you haven't validated against your own workload. The Stack Overflow answer doesn't run your traffic.