AKS in Production: CNI Overlay, Workload Identity, Hidden Limits

The page came in at 02:14 on a Saturday, which is how I learned our AKS cluster had started upgrading itself. The control plane had gone 1.28 to 1.29 without drama. Then the biggest node pool — 24 Standard_D8s_v5 nodes, all the stateful stuff — stalled mid-surge, and enough pods went Pending that PagerDuty got involved.

The AKS upgrade status said Running. It said that for four more hours.

Quick answer: AKS hides its control plane completely, so upgrades, quota failures, and API throttling surface as symptoms on your nodes, never as root causes you can inspect. The three decisions that matter most are CNI mode (Overlay, unless pods genuinely need VNet IPs), identity (Workload Identity, AAD pod identity is dead), and surge math against your regional vCPU quota, because surge VMs are provisioned *before* anything drains and a failed surge looks exactly like a hung upgrade.

The control plane you can't see

The actual error, when we finally found it, was in the subscription activity log, nowhere near anything kubectl could show:

OperationNotAllowed: Operation could not be completed as it results in
exceeding approved standardDASv5Family Cores quota. Current Usage: 344,
Additional Requested: 16, Quota Limit: 350.

Sixteen vCPUs. Two surge nodes. That's what hung a production upgrade at 2 AM, and nothing in Kubernetes said so, the VMSS scale-out just failed on the Azure side. This is the first thing to internalize about AKS: the control plane is a black box. No API server access, no etcd, nothing to kubectl. You get diagnostic settings streaming control plane logs to Log Analytics (enable kube-apiserver and kube-audit-admin; it's the only way you'll ever see a 429 from the API server itself), the activity log, and vibes.

So pick an upgrade channel deliberately, none, patch, stable, rapid, or node-image; stable tracks N-1 and is where I'd leave anything customer-facing, and give every auto-upgrading cluster a planned maintenance window, or Azure picks the time:

az aks maintenanceconfiguration add \
  -g prod-rg --cluster-name prod-aks \
  --name weekly --day-of-week Saturday \
  --start-hour 2 --duration 6

Windows are weekly at most frequent, minimum four hours. If your SLO can't survive a Saturday control plane bump, your problem is the SLO, not the schedule.

CNI: the decision you can't easily undo

Network plugin changes mean a cluster rebuild in practice, so get this right on day one.

	kubenet	Azure CNI	CNI Overlay
Pod IPs routable in VNet	no (NAT + UDRs)	yes	no (VXLAN)
Default max pods/node	110	30	250
Practical node ceiling	~400 (route table limit)	VNet IP space	5,000

Kubenet is deprecated and retires March 31, 2028. Good riddance, every off-node pod hop rides a user-defined route, and route tables cap at 400 routes, a hard ceiling people discover during a scale event. Classic Azure CNI gives every pod a real VNet IP: wonderful for NSG flow logs, terrible for address arithmetic. At the default 30 pods per node, a 200-node cluster wants a /16's worth of headroom planning.

Overlay is why the IP conversation changed. Pods draw from a private CIDR, 10.244.0.0/16 by default, carved into a /24 per node, encapsulated in VXLAN. Only nodes consume VNet addresses, so a 5,000-node cluster needs 5,000 VNet IPs, not 1.25 million:

az aks create -n prod-aks -g prod-rg \
  --network-plugin azure --network-plugin-mode overlay \
  --pod-cidr 100.64.0.0/10

Use Overlay for almost everything new. Exceptions are real but rare: pod IPs reachable from on-prem over ExpressRoute without NAT, or tooling that inspects pod IPs at the VNet layer. If you don't know you need that, you don't.

Workload Identity: the only identity story left

AAD pod identity is deprecated, the repo is archived, and the MIC/NMI DaemonSets with their AzureIdentity CRDs are living on borrowed time. If you're still on it, migration is a this-quarter task.

Workload Identity is OIDC federation, full stop: the service account token becomes the credential, exchanged with Entra for an Azure AD token. Three moving parts:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: payments-api
  annotations:
    azure.workload.identity/client-id: "7f3a9c2e-...."
---
# on the pod spec:
spec:
  serviceAccountName: payments-api
  template:
    metadata:
      labels:
        azure.workload.identity/use: "true"

Plus a federated identity credential on the user-assigned managed identity, scoped to your cluster's OIDC issuer and the subject system:serviceaccount:payments:payments-api. The mutating webhook injects the projected token and AZURE_CLIENT_ID, AZURE_TENANT_ID, and AZURE_FEDERATED_TOKEN_FILE, which DefaultAzureCredential picks up with no code changes.

One edge case that bites: the federated credential subject must match namespace *and* service account name exactly. Rename the service account and every token exchange fails with AADSTS700213, with nothing in the pod to tell you why. Ask me how I know.

Surge math and the quotas that only exist under load

Back to my Saturday. AKS upgrades surge one extra node by default, maxSurge: 1, and surge nodes, which count against your regional vCPU quota immediately, are created *before* the first old node cordons. We were at 344 of 350 approved cores in that family because someone had scaled the pool for a March load test and never scaled back. Two surge nodes, sixteen cores, OperationNotAllowed, and the upgrade retried into a wall for hours while pods piled up Pending.

The math is simple and nobody does it: surge capacity = maxSurge × node size must fit inside remaining regional quota *for that VM family*, plus whatever else might scale during the window. Request quota increases days before maintenance windows, they are not instant, and definitely not at 2 AM.

Other limits that only show up under load, from my scar tissue:

1,000 nodes per cluster by default, 5,000 as the documented max. 100 node pools per cluster.
API server throttling. No apiserver metrics endpoint to watch, but a controller storm, a CronJob fan-out creating 400 Jobs a minute, say, earns you client-side 429s and retry storms. The kube-apiserver logs in Log Analytics show Throttling request entries if you enabled diagnostics.
SNAT ports on the Standard Load Balancer: 64,000 per backend instance, shared across all that node's outbound flows. Connection-churny workloads exhaust it before CPU gets interesting; watch snatConnectionCount and consider NAT Gateway for egress-heavy pools.

The first thing I check now when an AKS cluster misbehaves is the activity log, before kubectl, before anything. It's also why we run conndeck against our clusters, the "is this a Kubernetes problem or an Azure problem" question wastes the first twenty minutes of every incident.

Frequently asked questions

Is kubenet still OK for new AKS clusters?

No. Kubenet is deprecated and retires on March 31, 2028, and route tables cap at 400 routes, which quietly limits kubenet clusters to about 400 nodes. New clusters should use Azure CNI Overlay unless pods genuinely need routable VNet IPs.

How does Azure CNI Overlay assign pod IPs?

Pods get IPs from a private overlay CIDR, 10.244.0.0/16 by default, carved into a /24 per node, and traffic between nodes rides VXLAN. Only nodes consume real VNet addresses, so a 5,000-node cluster needs about 5,000 VNet IPs instead of 5,000 times the 250-pod default per node.

How do I migrate from AAD pod identity to workload identity?

Create a federated identity credential on a user-assigned managed identity trusting your cluster's OIDC issuer and the service account's subject, annotate the service account with azure.workload.identity/client-id, and label the pod azure.workload.identity/use: "true". The mutating webhook injects the projected token and AZURE_* environment variables. AAD pod identity is deprecated and archived, so the MIC/NMI DaemonSets are living on borrowed time.

Why is my AKS node pool upgrade stuck with 429 or quota errors?

Surge nodes count against your regional vCPU quota before any old node cordons, so an upgrade can fail to create VMs with OperationNotAllowed. The activity log shows it clearly while the AKS upgrade status just says Running. Raise the quota for that VM family ahead of the maintenance window.

Can I access the AKS control plane or API server logs?

No. There is no SSH, no etcd snapshot, and no API server metrics endpoint. Your only windows are diagnostic settings streaming control plane logs to Log Analytics, the activity log, and whatever your clients observe. Budget a few dollars a month for log ingestion on any cluster you'd page on.

The takeaway I keep relearning

That Saturday ended anticlimactically: we raised the quota at 06:40, the surge nodes appeared, and the upgrade finished in twenty minutes. User impact: zero, thanks to PodDisruptionBudgets. Team impact: one ruined weekend and a runbook that now starts with "check the activity log and regional quota first."

Managed Kubernetes doesn't mean the platform's limits go away. It means they move somewhere you can't see, and they send their invoices in the middle of the night.