The Day Our Staging Cluster Ran Out of IP Addresses

Q: How do I check how many IP addresses are left in my subnet?

Run aws ec2 describe-subnets with the subnet ID and query Subnets[].AvailableIpAddressCount. Zero means you're out. Worth graphing as a metric too — it's a five-line Prometheus recording rule and an alert at 20% free, and it would've saved us two hours of debugging.

Q: How do I stop EKS nodes from using so many IP addresses?

Two knobs: cap the kubelet's maxPods (the default is 110, which is almost always wrong) and set WARM_IP_TARGET on the aws-node DaemonSet so the CNI stops pre-allocating whole ENIs' worth of IPs it doesn't need. We landed on maxPods: 40 and WARM_IP_TARGET=5, sized from actual p95 pod counts plus headroom.

It was a Tuesday, 9:40 in the morning, and the first message in our team's Slack channel was a screenshot of kubectl get pods. Forty-some pods in ContainerCreating. Somebody had merged a big feature branch the night before, Argo CD had synced it into staging like it was supposed to, and now half the environment wouldn't start.

My first guess was the registry. It's always the registry, except when it isn't.

Quick answer: When EKS pods get stuck in ContainerCreating with a failed to assign an IP address to container error, your VPC subnet is almost certainly out of free IPs — check AvailableIpAddressCount on the subnet before anything else. The usual root causes are an untouched kubelet --max-pods default of 110 and namespace sprawl nobody prunes. The fix is capping maxPods, setting WARM_IP_TARGET on aws-node, and putting an expiry date on temporary environments.

The events told a different story:

$ kubectl describe pod checkout-api-7d9f4c8b6-x2kpt -n checkout
...
Warning  FailedCreatePodSandBox  3m12s  kubelet
  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to
  set up sandbox container "a1b2c3..." network for pod "checkout-api-...":
  networkPlugin cni failed to set up pod "checkout-api-..._checkout" network:
  add cmd: failed to assign an IP address to container

failed to assign an IP address to container. If you've run EKS for a while, you know this sentence by heart, the way you know the smell of your own house burning down. We run the AWS VPC CNI (aws-node), version 1.18.x at the time, which means every pod gets a real, routable IP out of the VPC subnet. No overlay, no NAT magic. And real IPs come from a finite pool.

Ours was empty.

The five-minute diagnosis that took two hours

The actual check is embarrassingly simple:

$ aws ec2 describe-subnets --subnet-ids subnet-0a1b2c3d4e5f \
    --query 'Subnets[].AvailableIpAddressCount'
[ 0 ]

Zero. A /24 gives you 256 addresses, AWS keeps five of them, and all 251 usable ones were spoken for. But getting to that one command took the better part of two hours, because everything upstream of it looks like something else. Pods in ContainerCreating look like image pull problems. CNI errors look like a node problem, so we cordoned and drained a node, which helped for about ninety seconds until the replacement node hit the same wall. One engineer was convinced the CNI plugin itself was broken and started downgrading aws-node on a canary node. That was a fun twenty minutes.

The thing that finally tipped me off was boring: kubectl get nodes showed we only had six nodes in the cluster, and yet the subnet was full. Six nodes should not eat 251 IPs. Unless every node is packed to the gills with pods. Which, it turned out, ours were.

How a staging cluster eats 250 IPs

Two compounding mistakes, both completely standard.

Mistake one: nobody ever set max-pods. Our nodes were m5.xlarge instances, launched years ago with a node group config that predated most of the current team. The kubelet's --max-pods was sitting at the default of 110. The VPC CNI on an m5.xlarge can attach 4 ENIs with 15 IPs each, so each node could in principle hand out around 58 pod IPs. Nobody had ever reconciled those two numbers, so nodes just kept accepting pods and burning through secondary ENI IPs as fast as the scheduler could place them.

Mistake two: namespace sprawl. Staging had accumulated 30+ namespaces. Every feature branch got one, courtesy of an ApplicationSet with a Git directory generator that somebody set up in 2024 and never revisited. Old namespaces never got pruned because, well, it's staging, and deleting things is scary and not deleting things is free. Except it isn't free. Each of those namespaces held a handful of Deployments with 2–3 replicas, plus the jobs, plus the Redis and Postgres sidecars people kept adding for "realistic" testing.

Do the math with me. Six nodes, roughly 40 pods per node on average, a chunk of DaemonSets on top. Around 250 pod IPs. On a /24. We were one medium-sized deploy away from the wall at all times, and Monday night's merge was that deploy.

The fix, in the order we did it

First, stop the bleeding. We deleted a dozen dead namespaces — branches that had merged months ago, a proof-of-concept from someone who'd left the company, an environment literally named test2. That freed up around 60 IPs in ten minutes and pods started scheduling again. Total time from "what is happening" to "staging works": about three hours. Not our finest incident, not our worst.

Then the real fixes, over the next week:

# KubeletConfiguration, via the node group's bootstrap config
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
maxPods: 40

We capped max-pods at 40 per node, sized from actual p95 pod counts plus headroom, not vibes. We also set WARM_IP_TARGET=5 on the aws-node DaemonSet so the CNI stopped greedily pre-allocating whole ENIs' worth of IPs it didn't need:

kubectl set env daemonset/aws-node -n kube-system WARM_IP_TARGET=5

And the less technical fix, the one that actually mattered: a TTL. Feature-branch namespaces now get an expiry label, and a little CronJob deletes anything older than 14 days unless someone renews it. Nobody has complained once, which tells you how alive those environments really were.

Frequently asked questions

What causes the "failed to assign an IP address to container" error in EKS?

It means the AWS VPC CNI couldn't hand the pod an IP because the subnet's pool of usable addresses is empty. The VPC CNI gives every pod a real routable VPC IP, so there's no overlay to hide behind — when the subnet is out, new pods sit in ContainerCreating. In our case a /24 had all 251 usable addresses spoken for.

How do I check how many IP addresses are left in my subnet?

Run aws ec2 describe-subnets with the subnet ID and query Subnets[].AvailableIpAddressCount. Zero means you're out. Worth graphing as a metric too — it's a five-line Prometheus recording rule and an alert at 20% free, and it would've saved us two hours of debugging.

How do I stop EKS nodes from using so many IP addresses?

Two knobs: cap the kubelet's maxPods (the default is 110, which is almost always wrong) and set WARM_IP_TARGET on the aws-node DaemonSet so the CNI stops pre-allocating whole ENIs' worth of IPs it doesn't need. We landed on maxPods: 40 and WARM_IP_TARGET=5, sized from actual p95 pod counts plus headroom.

Why would a staging cluster run out of IP addresses?

Usually the same two mistakes we made: nobody sets max-pods, so nodes pack in far more pods than anyone planned, and namespaces multiply forever because nobody prunes old environments. Staging clusters accumulate feature-branch namespaces, sidecars, and forgotten test deployments until one medium-sized deploy tips the subnet over.

The lessons, which are not exciting

I've turned this incident over in my head a few times since, and I keep landing on the same unglamorous conclusions.

Staging is production for your developers. When staging is down, your whole team is down. We had seven engineers blocked for half a day because we treated the cluster they work in all day as a place where hygiene is optional. The blast radius was smaller than a prod outage, but the cost per hour was the same salaries.

IP addresses are a capacity metric, and almost nobody graphs them. We had dashboards for CPU, memory, disk, request latency, the works. Nothing tracked AvailableIpAddressCount per subnet or pod IP allocation per node. That's a five-line Prometheus recording rule and an alert at 20% free. We have it now. This is the kind of failure that only shows up when you look across the whole cluster, which is what we built conndeck's fleet view for — cluster-level resources that only become visible when they're gone.

Defaults are decisions. --max-pods: 110 is a decision someone at Kubernetes made a decade ago for a very different world, and by not overriding it we made it ours. Same with the CNI's warm pool defaults. You don't have to tune everything on day one, but you should at least know which knobs are set to what, and why.

And the last one: cleanup is a feature, not a chore. Every "temporary" namespace, preview environment, and test deployment needs a death date at birth. If your process for creating environments is automated and your process for destroying them is "someone remembers," you don't have a process, you have a countdown timer.

The subnet's fine now, by the way. 140 free addresses, an alert that will page us long before it matters, and a Slack bot that announces namespace reaping every Friday like a tiny grim reaper. Boring infrastructure. Exactly how I like it.