Why Your Cluster Autoscaler Isn't Scaling

Black Friday, 2025, 6:04 AM. Traffic tripled overnight, the HPA did its job and asked for 60 replicas, and 42 of them sat Pending while the cluster autoscaler -- the thing we were paying a cloud provider real money to run -- did absolutely nothing.

No errors. No crashes. The autoscaler logs were serene. The scheduler events said Insufficient cpu, which was true and also completely unhelpful, because we had three node groups with headroom in their max sizes. It took me until 7:30 to find the cause, and it was one line: a nodeSelector somebody had added to the deployment six weeks earlier pointed at a label that existed on exactly zero node groups. The autoscaler wasn't broken. It had correctly determined that adding a node wouldn't help, and gone back to sleep.

That's the thing nobody tells you about the cluster autoscaler: its failure mode is silence. It doesn't error when it can't help you. It just declines, politely, in an event you have to know to look for.

Quick answer: When pods are Pending and the cluster autoscaler won't scale up, it means the autoscaler simulated adding a node from every available group and none of them would fit the pod. The usual causes: the pod's requests exceed any node group's instance size, a nodeSelector or affinity matches no group's labels, missing tolerations for the group's taints, the group is already at its max size, or a cloud quota is exhausted. Run kubectl describe pod on a pending pod and read the didn't trigger scale-up events -- the autoscaler tells you exactly why, per node group.

How the autoscaler actually decides

The cluster autoscaler doesn't watch node utilization. It watches pending pods. Every scan loop (default 10 seconds), it asks one question: is there a pod that has been unschedulable for a while, and would adding a node from some group let the scheduler place it?

That second clause is everything. The autoscaler builds a template node from each node group -- instance type, labels, taints, capacity -- and runs the scheduler's predicates against it in simulation. If no group's template can host the pod, the answer is no, forever, and the only record is an event on the pod.

So "why won't it scale" decomposes into a short list of ways the simulation can fail. Here's each one and how to spot it.

The event that tells you everything

Start here, always:

$ kubectl describe pod checkout-6d8f7b9c4-x2kqm -n shop
...
Events:
  Type     Reason             Age   Message
  ----     ------             ----  -------
  Warning  FailedScheduling   12m   0/9 nodes are available: 9 Insufficient cpu.
  Normal   NotTriggerScaleUp  11m   pod didn't trigger scale-up: 1 max node group size reached, 2 node(s) didn't match Pod's node affinity/selector

NotTriggerScaleUp is the autoscaler talking to you. The message lists every node group and its specific objection. In this output: one group is maxed out, and two more don't match the pod's affinity. That single line would have saved me ninety minutes on Black Friday.

If the event is missing entirely, check that the autoscaler is actually running and looking at the right cluster -- yes, I've seen a team debug a staging cluster's autoscaler while pods pended in prod. If you need more detail, the cluster-autoscaler-status ConfigMap in kube-system has a human-readable per-group health and decision summary, and the autoscaler's own logs at -v=4 narrate the whole simulation.

Requests that no node can satisfy

The dumbest and most common cause: the pod asks for more than any node offers.

$ kubectl get pod checkout-6d8f7b9c4-x2kqm -n shop -o jsonpath='{.spec.containers[*].resources.requests}'
{"cpu":"3500m","memory":"6Gi"}

Node group runs m5.xlarge -- 4 vCPU, 16GiB, minus daemonsets and system reserved. A 3.5-core request fits, barely, exactly once per node. That's not a bug; it just means every scale-up buys you one pod. But bump that request to 4.5 cores -- say, because a VPA recommendation drifted upward, or someone fat-fingered a template -- and *no* node in the group can ever fit it. The autoscaler will decline every scan, correctly, forever.

The fix is either capping requests to something your instance types can host, or adding a bigger node group. Keep a rough invariant in your head: max pod request in a workload should be comfortably under the allocatable of the smallest node you want it to land on.

Labels, taints, and affinity mismatches

This was my Black Friday. The pod spec said:

nodeSelector:
  workload: compute-optimized

And the node groups were labeled workload: general and workload: memory. The scheduler found no matching node, and the autoscaler found no matching *template*, so it correctly reported "would not help." The autoscaler does not create nodes with labels the pod wants; it only expands groups that already advertise them.

The taint version is identical in spirit: a node group tainted dedicated=ml:NoSchedule will never host a pod without a matching toleration, and the autoscaler won't burn money trying. When a pending pod's events mention taints or affinity, compare three things side by side: the pod's nodeSelector/affinity/tolerations, the node group's labels and taints as the autoscaler sees them (check an existing node from that group with kubectl get node -o yaml), and the group's actual max size. One of the three is lying.

Max sizes, quotas, and the walls you can't see

Each node group has a min and max, and the autoscaler respects them absolutely. max node group size reached means what it says. Check before assuming:

$ kubectl get configmap cluster-autoscaler-status -n kube-system -o jsonpath='{.data.status}' | head -30

Less obvious: the group can be under its Kubernetes max and still fail because the *cloud* said no. Instance quota exhausted in the region, no capacity for that instance type in the AZ (spot pools do this routinely), or a launch template referencing an AMI that no longer exists. In those cases the autoscaler *does* trigger a scale-up, the node never joins, and after a timeout it backs off the group. The events say FailedToScaleUpGroup or the cloud provider logs say InsufficientInstanceCapacity. If nodes are being requested but never appearing, stop looking at Kubernetes entirely and go read your cloud console.

Scale down: the other direction of stuck

While you're here, the inverse failure is just as common: nodes that refuse to leave. The autoscaler removes underutilized nodes by draining them, and draining means evicting pods, which means respecting PodDisruptionBudgets. A PDB with minAvailable equal to the replica count makes every pod unevictable, which pins every node those pods touch.

Other classic scale-down blockers: pods with emptyDir or local storage (unless annotated cluster-autoscaler.kubernetes.io/safe-to-evict: "true"), pods not owned by a controller, and pods using hostPath. When a node won't die, the autoscaler's logs name the pod and the reason per node. Watching Pending pods and node churn in a live view -- which, full disclosure, is part of why we built conndeck -- makes the difference between "the autoscaler is broken" and "oh, that PDB again" about a two-second diagnosis.

Frequently asked questions

Why are my pods Pending but the cluster autoscaler won't add nodes?

The autoscaler only scales up if some node group could actually fit the pod. Check the pod's events for the scheduler's reason, then verify the request fits the node group's instance type, the pod's nodeSelector and tolerations match the group's labels and taints, and the group is below its max size. If any of those fail, the autoscaler correctly concludes that adding a node wouldn't help and does nothing.

How do I see why the cluster autoscaler skipped a scale-up?

Run kubectl describe on the pending pod and read the events: the autoscaler posts events like didn't trigger scale-up with a reason per node group, such as max node group size reached or no node group could fit the pod. The autoscaler's own logs and the cluster-autoscaler-status ConfigMap give the full per-group decision breakdown.

Can a PodDisruptionBudget stop the cluster autoscaler?

Yes, on scale down. The autoscaler drains nodes by evicting pods, and evictions respect PDBs, so a PDB with minAvailable equal to the replica count can block node removal indefinitely. On scale up, PDBs don't block, but a pending pod that would violate nothing still needs a node group that can fit it.

Why won't the cluster autoscaler scale down my idle nodes?

The usual blockers are pods with local storage, pods not managed by a controller, pods using hostPath, restrictive PDBs, or nodes whose utilization is above the scale-down threshold because of one large pod with nowhere to go. System pods without a PDB and pods annotated with safe-to-evict false also pin nodes. The autoscaler logs name the exact blocker per node.

Does the cluster autoscaler look at pod limits or requests?

Requests only. The autoscaler simulates scheduling using pod requests against a template node from each group, so a pod requesting more CPU or memory than any node in the group offers can never trigger a scale-up. Limits are irrelevant to its math, which is another reason runaway VPA recommendations can silently strand pods.

The takeaway

The cluster autoscaler's failure mode is silence, so learn to interrogate it: describe the pending pod, read the NotTriggerScaleUp event, and let it enumerate its own objections per node group.

Almost every "autoscaler is broken" incident is one of five things: requests too big for the instance type, labels or taints that match no group, a maxed-out group, an exhausted cloud quota, or a PDB pinning nodes on the way down. Check them in that order and you'll skip the 90 minutes I didn't.