conndeck Download

conndeck blog

HPA, VPA, KEDA: Which Autoscaler Does What

#kubernetes#autoscaling#hpa#keda#scaling

It was 9:15 on a Monday morning when our checkout service fell over, thirty seconds after the marketing team pushed a flash sale email to two million people.

The HPA was configured. It was configured beautifully, in fact: target 70% CPU, min 4, max 40. What actually happened was the queue backed up to 400,000 messages while CPU lounged around at 35%, because the workers were blocked on a downstream rate limit, not compute. The autoscaler sat there, green and content, doing exactly what we told it to do. Scaling on the wrong metric is worse than not scaling at all, because it gives you the confidence to go back to sleep.

That incident sent me down the rabbit hole of Kubernetes autoscaling, and the first thing I learned is that "autoscaling" is three different tools solving three different problems, and picking the wrong one is the default outcome.

Quick answer: HPA adds or removes pod replicas based on metrics like CPU, memory, or custom metrics; use it for stateless workloads with load that tracks a measurable signal. VPA changes the CPU and memory requests of existing pods, which requires restarts; use it to right-size workloads or get recommendations. KEDA scales on external event sources like queue depth or Kafka lag and can scale to zero; use it for event-driven workers. Never run HPA and VPA together on the same metric, and prefer VPA recommendation mode when combining them.

The three tools, one sentence each

Before the details, pin the mental model:

  • HPA (Horizontal Pod Autoscaler): changes the *number of pods*. Built into Kubernetes. Reacts in a minute or two. No restarts.
  • VPA (Vertical Pod Autoscaler): changes the *size of each pod's requests*. An add-on (autoscaler/vertical-pod-autoscaler). Requires pod recreation to apply. Slow, disruptive, honest.
  • KEDA (Kubernetes Event-Driven Autoscaling): changes the *number of pods* based on *external signals* like queue depth, and can go all the way to zero. An add-on that drives an HPA it generates for you.

If your workload is a web API whose load tracks CPU, you need HPA and only HPA. If your workloads are chronically over- or under-provisioned, you need VPA's recommender. If your workers drain a queue, you need KEDA. Most mature platforms end up running all three, on different workloads, for different reasons.

HPA: the one you already have

HPA is a control loop in kube-controller-manager. Every --horizontal-pod-autoscaler-sync-period (default 15 seconds), it reads metrics, computes desiredReplicas = ceil(currentReplicas * currentMetric / targetMetric), and patches the scale subresource.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-api
  minReplicas: 4
  maxReplicas: 40
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

The autoscaling/v2 API is the one you want; v1 only does CPU and is ancient history. The detail that bites everyone: utilization is computed against requests, not limits, and pods without CPU requests are invisible to CPU-based HPA. No requests, no scaling, no warning.

HPA can also scale on custom and external metrics, like requests-per-second from Prometheus via an adapter:

  metrics:
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "500"

That was the actual fix for our Monday incident, eventually: scale on queue depth, not CPU. Which brings me to the tool that does that natively.

KEDA: scaling on things that aren't CPU

KEDA's trick is simple and great. You define a ScaledObject with one or more triggers pointing at external systems, and KEDA exposes those as external metrics to an HPA object it creates and manages behind the scenes:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: order-worker
spec:
  scaleTargetRef:
    name: order-worker
  minReplicaCount: 0
  maxReplicaCount: 30
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka.infra:9092
        consumerGroup: order-workers
        topic: orders
        lagThreshold: "1000"

Two capabilities make KEDA worth the extra CRD. First, minReplicaCount: 0. Plain HPA bottoms out at one replica; KEDA can scale idle workers to nothing and wake them when work appears, which for a fleet of nightly batch consumers is real money. Second, the trigger catalog is huge: Kafka, SQS, RabbitMQ, Redis streams, Postgres queries, cron schedules, Prometheus queries. If the signal exists, there's probably a scaler for it.

The gotcha I hit: KEDA-generated HPAs still obey HPA's stabilization and flapping rules, and a lag threshold that's too low relative to your consumption rate makes replicas yo-yo. Set advanced.horizontalPodAutoscalerConfig.behavior explicitly rather than inheriting defaults you haven't read.

VPA: the honest broker

VPA is three components: a recommender that watches actual usage and computes suggested requests, an updater that evicts pods whose requests drift too far from the recommendation, and an admission plugin that stamps the recommended requests onto new pods.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: search-indexer
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: search-indexer
  updatePolicy:
    updateMode: "Off"

Start with updateMode: "Off". In that mode VPA does nothing except maintain a live recommendation you can read:

$ kubectl get vpa search-indexer -o jsonpath='{.status.recommendation.containerRecommendations[0]}'
{"containerName":"indexer","lowerBound":{"cpu":"250m","memory":"512Mi"},"target":{"cpu":"780m","memory":"1.2Gi"},"upperBound":{"cpu":"1500m","memory":"2.1Gi"}}

Even if you never let VPA touch a single pod, those target values are the cheapest right-sizing audit you'll ever get. I run VPA in Off mode on every cluster I manage just for the recommendations, and it's how I caught a Java service running with a 4Gi request and a 600Mi actual working set for eight months.

When you do turn on Auto or Recreate, remember what it means: the updater evicts pods to resize them. On a single-replica deployment with no PDB, that's an outage every time the recommendation drifts.

Making them play nice (and the conflicts that hurt)

The classic trap is HPA and VPA on the same deployment, both keyed on CPU. VPA raises the CPU request, which lowers measured utilization, which makes HPA scale up, which changes the usage profile, which makes VPA recommend again. Congratulations, you've built a control-theory demonstration of instability.

The patterns that actually work:

  • HPA on a non-resource metric + VPA on requests. Scale horizontally on queue depth or RPS, let VPA keep pod sizing honest. No shared signal, no feedback loop.
  • HPA alone + VPA in Off mode. You read the recommendations weekly and bump requests in Git like an adult. This is my default.
  • KEDA for event-driven, HPA for request-driven, VPA recommender everywhere. Different workloads, different tools, one cluster.

One more conflict worth knowing: the cluster autoscaler watches for unschedulable pods, and VPA can make pods unschedulable by recommending requests larger than any node. If your VPA upper bounds aren't capped (resourcePolicy.containerPolicies[].maxAllowed), a single memory-hungry JVM can eventually request a node size your node groups don't offer, and it will sit Pending forever while everyone blames the autoscaler.

When I'm untangling which controller did what to a workload at 2am, being able to see replica counts, requests, and recommendation drift in one live view is genuinely useful -- it's part of why we built conndeck -- but the real fix is boring: decide upfront which metric owns scaling and which tool owns sizing, and write it down.

Frequently asked questions

What is the difference between HPA and VPA?

HPA scales horizontally by adding or removing pod replicas based on a metric, while VPA scales vertically by changing the CPU and memory requests of existing pods. HPA reacts in minutes without restarting anything. VPA changes require pod recreation to take effect, so it moves slower and causes restarts, but it fixes the underlying sizing rather than spreading load across more copies.

Can you use HPA and VPA on the same deployment?

Yes, but not on the same metric. If HPA scales on CPU while VPA adjusts CPU requests, they fight each other because VPA changing requests changes the utilization HPA sees, causing feedback loops. The safe pattern is HPA on a custom or external metric like queue depth while VPA manages CPU and memory requests, or run VPA in recommendation-only mode alongside HPA.

When should I use KEDA instead of HPA?

Use KEDA when the thing you want to scale on lives outside the cluster, like a Kafka lag, an SQS queue depth, a Redis list length, or a cron schedule. KEDA also adds scale-to-zero, which plain HPA cannot do because one replica is its floor. If your metric is CPU or memory on the pods themselves, HPA alone is simpler and one less controller to run.

Does VPA restart my pods?

Yes. When VPA's updater decides a pod's requests should change, it evicts the pod so the replacement gets created with new requests. You can control this with updateMode: set it to Off or Initial to get recommendations without restarts, or use Recreate and Auto to let it evict. PodDisruptionBudgets are respected by the updater in recent versions.

Can KEDA and HPA be used together?

KEDA actually works by creating and managing an HPA object for you under the hood. The ScaledObject is the interface, and KEDA translates your triggers into external metrics that its generated HPA consumes. So you do not run them side by side on the same workload; you use KEDA instead of writing your own external metrics pipeline.

What actually matters

Autoscaling isn't one decision, it's three: how many pods, how big each pod is, and what signal should drive the change. HPA owns the first, VPA owns the second, and KEDA exists for the signals that don't live on the pods.

Pick the tool that matches your actual bottleneck, never let two controllers fight over the same metric, and run VPA in recommendation mode even if you never automate it. Your future self, staring at a green HPA during an outage, will thank you.