EKS Networking: VPC CNI, Prefix Delegation, and the IP Math
Cluster autoscaler had just added 38 nodes to absorb a Saturday traffic spike, and eleven minutes later half the fleet was refusing pods. The event stream was a wall of FailedCreatePodSandBox, and the aws-node logs repeated one line:
ERROR ipamd/ipamd.go failed to assign IPs: AssignPrivateIpAddresses:
api error Throttling: Rate exceededEvery node's ipamD was hammering the EC2 API for secondary IPs at once, EC2 rate-limited the whole account, and pods sat in ContainerCreating until the storm passed. The IPs were there, we just couldn't ask for them fast enough.
Quick answer: EKS pod density is governed by ENI slot math, (ENIs × (IPs per ENI − 1)) + 2, and the AWS VPC CNI keeps a warm pool of IPs on every node via ipamD. Prefix delegation replaces secondary IPs with /28 prefixes, multiplying per-node pod limits and cutting EC2 API calls, at the cost of coarser allocation granularity. Tune MINIMUM_IP_TARGET and WARM_IP_TARGET from real pod-churn data, or meet an empty subnet or a throttling error at 2 AM.
How the VPC CNI actually assigns IPs
There's no overlay. The VPC CNI attaches Elastic Network Interfaces to each node (how many depends on instance type); each ENI gets a primary IP plus a fixed number of secondary-IP slots, and pods get real, routable VPC addresses from those slots.
The agent doing this is ipamD, inside the aws-node DaemonSet. It maintains a warm pool, IPs (or ENIs) allocated but not yet handed to pods, so a new sandbox gets its address without an EC2 round-trip. Three env vars steer it: WARM_ENI_TARGET (default 1; full spare ENIs, expensive in IP space), WARM_IP_TARGET (extra *IPs* kept warm above demand), and MINIMUM_IP_TARGET (allocate at least this many, even idle).
When ipamD finds the pool short, it calls AssignPrivateIpAddresses (or attaches a new ENI when no slots are free). That call is the unit of throttling. My Saturday was 38 nodes × a dozen calls in the same minute, against an account API budget shared with everything else we run.
The per-node pod-density math
A node's pod limit is set by its ENI topology, not its vCPUs:
max pods = ENIs × (IPv4 addresses per ENI − 1) + 2The −1 is each ENI's primary address; the +2 covers host-network pods. In practice:
| Instance | ENIs | IPs/ENI | Max pods | With /28 prefixes |
|---|---|---|---|---|
| m5.large | 3 | 10 | 29 | 110 |
| m5.xlarge | 4 | 15 | 58 | 110 |
| c5.4xlarge | 8 | 30 | 234 | 110 |
I've watched teams burn money here in both directions: m5.larges stalling rollouts at 29 pods while CPU sits at 20% are the classic; c5.4xlarges idling at 12 pods are the reverse. Set the limit yourself with /etc/eks/bootstrap.sh --use-max-pods false --kubelet-extra-args '--max-pods=N', and N is a lie if it exceeds the ENI math.
Prefix delegation: when /28 helps and when it hurts
Set ENABLE_PREFIX_DELEGATION=true on aws-node (v1.9.0+, Nitro instances only) and each secondary-IP slot becomes a /28 prefix, 16 addresses. The m5.large's 3 × 9 slots now hold 432 theoretical IPs; AWS's published guidance lifts its cap to 110, where the kubelet becomes the binding constraint anyway.
Two second-order effects matter more than the headline:
It cuts API calls. Allocating 16 IPs is one AssignPrivateIpAddresses call, not sixteen. In that storm, prefix delegation alone would have cut our call volume ~10× and we'd likely never have hit the throttle. That's the real reason I enable it everywhere now.
The granularity bites. A node running 3 pods still burns at least one full /28. Worse, a prefix is only released when *every* IP in it is free, so rolling churn fragments prefixes, one straggler pod pins 14 free addresses. WARM_PREFIX_TARGET (default 1) sets how many spare prefixes sit idle per node; on IP-starved subnets, that's real money. Existing nodes don't pick it up, you drain and replace them after flipping the flag, which surprises exactly everyone who flips it live.
My rule: prefix delegation on nodes hosting ~30+ pods or bursty workloads; skip it on small steady nodes in tight subnets, where plain WARM_IP_TARGET tuning wastes less.
Tuning the warm pool under churn
The defaults (a whole spare ENI per node) are a bad compromise: wasteful in IP space, yet slow in a burst, since attaching an ENI takes 10+ seconds. What I run on worker groups with real churn:
env:
- name: ENABLE_PREFIX_DELEGATION
value: "true"
- name: MINIMUM_IP_TARGET
value: "40"
- name: WARM_IP_TARGET
value: "15"Numbers come from metrics, not vibes. aws-node exposes Prometheus metrics on :61678/metrics; graph awscni_assigned_ip_addresses per node, set MINIMUM_IP_TARGET near p50 and WARM_IP_TARGET at p95-minus-p50. Watch awscni_aws_api_error_count (throttling) and awscni_total_ipv4_prefixes (fragmentation pinning addresses). I keep these in conndeck next to pod counts; the gap between the curves is the waste.
One footgun: with custom networking (AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true plus ENIConfig objects from crd.k8s.amazonaws.com/v1alpha1), pods draw from dedicated pod subnets while the node sits elsewhere. It's the right fix for exhausted VPCs, but every warm-pool knob then applies to those pod subnets.
Security groups for pods
Different mechanism entirely. With ENABLE_POD_ENI=true, the VPC resource controller attaches a trunk ENI (aws-k8s-trunk-eni) to each eligible node, and pods matched by a SecurityGroupPolicy (vpcresources.k8s.aws/v1beta1) get their own *branch* ENI with its own security groups. No prefixes; each SGP pod burns one subnet IP and one slot from the per-instance branch-ENI limit. I use it for one thing: pods needing a security group distinct from the node's, say, reaching RDS with a tight ingress rule. Apply it broadly and you'll discover branch-ENI limits mid-rollout.
The failure modes I keep seeing
- IP exhaustion:
add cmd: failed to assign an IP address to containerin events,AvailableIpAddressCountat 0 on the subnet. Fix: smaller warm pool, prefix delegation, or custom networking into bigger subnets. - API throttling:
Throttling: Rate exceededin ipamD logs during mass scale-ups. Fix: prefix delegation,MINIMUM_IP_TARGETso nodes boot pre-armed, staged autoscaler scale-up. - Stuck sandboxes with
cni plugin not initialized: aws-node isn't healthy on that node, a crashed DaemonSet pod, or a node that joined before the CNI config was written. Restart aws-node, cordon if it recurs.
Frequently asked questions
How many pods can run on one EKS node?
It depends on the instance type, not the CPU. The formula is (ENIs x (IPv4 addresses per ENI minus 1)) + 2, because every pod needs a real VPC IP from an ENI. An m5.large has 3 ENIs with 10 addresses each, so 3 x 9 + 2 = 29 pods. Enable prefix delegation and that same m5.large goes to 110.
Does prefix delegation increase pod density in EKS?
Yes, dramatically. With ENABLE_PREFIX_DELEGATION=true, each secondary-IP slot on an ENI carries a /28 prefix (16 IPs) instead of one address, so an m5.large jumps from 29 to 110 pods. It needs Nitro-based instances and VPC CNI 1.9.0 or later, and /28 granularity wastes addresses on nodes with few pods.
How should I set WARM_IP_TARGET and MINIMUM_IP_TARGET on the aws-node DaemonSet?
Set MINIMUM_IP_TARGET to your expected steady-state pod count per node so ipamD pre-allocates that many IPs at boot, and WARM_IP_TARGET to the burst headroom you need for rapid scale-ups, like p50 and p95 of actual churn. Leaving both unset makes the CNI keep a whole spare ENI per node, which is expensive on IP space.
Why do EKS pods get stuck in ContainerCreating with FailedCreatePodSandBox?
Almost always the VPC CNI failed to hand the sandbox an IP. Run kubectl describe pod and look for 'failed to assign an IP address to container' (subnet or warm pool empty, check AvailableIpAddressCount) or 'cni plugin not initialized' (aws-node isn't healthy on that node). The first is capacity, the second is a node-agent problem.
Do security groups for pods work with prefix delegation?
They solve different problems and coexist fine, but security groups for pods does not use prefixes, matched pods get a dedicated branch ENI off the node's trunk ENI, each with its own security group and one subnet IP. Prefix delegation only applies to pods on the node's standard ENIs. Branch ENIs have their own per-node limits, which vary by instance type.
---
That 2 AM page rewired how I think about EKS networking: it's a budget, ENI slots, /28 blocks, API calls, and you either do the arithmetic up front or the cluster does it for you at the worst possible moment. Do the math now, while it's quiet.