conndeck Download

conndeck blog

Kubernetes Cost Optimization: Finding the Idle Half of Your Cluster

#kubernetes#cost#finops#opencost#capacity

In March, our CFO forwarded me the AWS bill with a one-line comment: "Is this number right?"

It was right. That was the problem. Our main production cluster -- a modest thing, maybe 40 workloads -- was costing us roughly $14k a month in compute. So I did what any self-respecting platform engineer does when finance comes calling: I pulled the actual utilization numbers to prove the spend was justified.

The numbers did not prove the spend was justified. Cluster-wide, we were using about 18% of the CPU and 31% of the memory we were paying for. We weren't running a production platform. We were running a very expensive reservation system for capacity nobody used.

Quick answer: Most Kubernetes cost problems come from one gap: container requests don't match actual usage, so nodes are sized for reservations, not reality. Measure the gap with kubectl top and a cost tool like OpenCost, right-size requests toward the p95 of real usage (VPA recommendation mode does this for you), move tolerant workloads to spot nodes, and then build a monthly habit of re-checking, because the drift comes back. Expect to find a third to half your spend sitting idle on the first pass.

The gap: requests vs reality

Kubernetes schedules on requests, not usage. When a developer writes requests: { cpu: "2", memory: 4Gi } -- usually copied from a template, usually a guess, usually rounded up "to be safe" -- the scheduler reserves that much of a node for the pod whether it uses it or not. Enough generous guesses and your nodes are "full" at 20% actual utilization, and the autoscaler obediently adds more.

You can see the gap with two commands. What you're paying for:

$ kubectl top nodes
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node-a1        1842m        23%    6241Mi          37%
node-a2        1204m        15%    4802Mi          29%
node-b1        2109m        26%    7108Mi          41%

Versus what's reserved, which is what actually determines capacity:

$ kubectl describe node node-a1 | grep -A5 "Allocated resources"
Allocated resources:
  (Total limits may be over 100 percent...)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                7100m (88%)   12400m (155%)
  memory             12Gi (75%)    20Gi (125%)

There's your bill in miniature: 88% of the node's CPU reserved, 23% used. Multiply by every node, every hour, every month.

Get real numbers with OpenCost

Eyeballing kubectl top tells you waste exists; it doesn't tell you whose. For that you want cost allocation, and the open-source answer is OpenCost -- the CNCF project that Kubecost is built on. It combines your cloud provider's pricing with Prometheus metrics and attributes spend per namespace, deployment, and label:

$ helm repo add opencost https://opencost.github.io/opencost-helm-chart
$ helm install opencost opencost/opencost -n opencost --create-namespace

Once it's had a day to accumulate, the API gives you the league table:

$ curl -s http://localhost:9003/allocation/compute?window=7d&aggregate=namespace \
  | jq '.data[0] | to_entries | sort_by(-.value.totalCost)[:5]'
[
  {"key":"payments","value":{"totalCost": 912.44, ...}},
  {"key":"search","value":{"totalCost": 701.18, ...}},
  {"key":"data-pipelines","value":{"totalCost": 655.03, ...}}
]

Two things fell out of this immediately for us. First, the payments namespace was our most expensive -- and its p95 CPU usage was 8% of requests. Second, a loadtest namespace someone spun up in January was still burning $400 a month on a cluster nobody remembered. Cost tooling pays for itself the first time it surfaces a zombie.

Right-sizing without breaking production

The fear with lowering requests is that you'll throttle something at peak. Fair. The method that works: measure over at least two weeks including peaks, and set requests to roughly the p95 of actual usage, not the max and not the mean.

You can do the math in PromQL per container, or let the Vertical Pod Autoscaler do it. Installed in recommendation mode (updateMode: "Off"), it watches usage and publishes suggested requests without touching anything:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: payments-api
  namespace: payments
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payments-api
  updatePolicy:
    updateMode: "Off"
$ kubectl get vpa payments-api -n payments -o jsonpath='{.status.recommendation.containerRecommendations[0].target}' | jq
{
  "cpu": "180m",
  "memory": "640Mi"
}

The deployment asked for 2 cores and 4Gi. VPA, having watched two weeks of reality, suggested 180m and 640Mi. That's the "idle half" made concrete -- and payments-api was not unusual. Roll the changes gradually, namespace by namespace, watch your error rates, and keep limits where they were; you're shrinking the reservation, not the ceiling. One note: VPA recommendations assume steady-state behavior, so sanity-check cron-ish and bursty workloads by hand.

Spot nodes: the other half of the savings

Right-sizing fixes the reservation gap. Spot capacity fixes the price per hour, and it's worth 60-70% off on-demand if your workloads tolerate interruption. The pattern that survives production is a mixed fleet: an on-demand node group sized for your baseline, plus a spot group that absorbs everything tolerant.

Taint the spot pool so only opt-in workloads land there:

taints:
  - key: spot
    value: "true"
    effect: NoSchedule
# on the workload
tolerations:
  - key: spot
    operator: Equal
    value: "true"
    effect: NoSchedule

Good spot citizens: stateless services with 3+ replicas and a PodDisruptionBudget, queue consumers, CI runners, batch jobs. Bad spot citizens: StatefulSets with local data, single-replica anything, and your Prometheus server. On EKS and GKE the interruption handling is largely built in (Karpenter and the managed autoscalers drain nodes on rebalance notices); on self-managed clusters you want the AWS Node Termination Handler or equivalent so evicted nodes drain gracefully instead of vanishing mid-request.

Making it stick

Everything above decays. Teams ship, templates get copied, and next quarter the drift is back. Two habits keep it from compounding: a monthly 30-minute review of requests-vs-p95 by namespace (this is one of the views I keep open in conndeck during that review, since it shows requested versus used side by side without me writing PromQL at 9 AM), and a zombie sweep -- orphaned namespaces, detached load balancers, forgotten node groups. And report savings in dollars to whoever forwarded you the bill in the first place. Nothing buys a platform team goodwill like a CFO who stops asking whether the number is right.

Frequently asked questions

Why is my Kubernetes cluster so expensive?

In almost every case the money goes to resources that were requested but never used. Developers guess at CPU and memory requests, round up for safety, and never revisit them, so the scheduler reserves capacity that sits idle while you pay for the nodes underneath it. Industry data and my own audits consistently show 30 to 50 percent of allocated capacity going unused. The fix starts with measuring actual usage against requests, not with buying smaller nodes.

What's the difference between OpenCost and Kubecost?

OpenCost is the open source, CNCF-hosted core that does cost allocation: it reads your cloud provider's pricing data plus Prometheus metrics and attributes spend to namespaces, workloads, and labels. Kubecost is the commercial product built on top of OpenCost, adding a polished UI, savings recommendations, and governance features. If you want numbers you can query and pipe into your own dashboards, OpenCost alone is enough; if you want a product your finance team will open, that's Kubecost.

How do I figure out the right CPU and memory requests?

Measure actual usage over at least two weeks, covering traffic peaks, then set requests around the 95th percentile of real usage rather than the maximum or the average. The Vertical Pod Autoscaler in recommendation mode automates this by computing suggested requests from observed usage without changing anything itself. Expect requests to come down by half or more on most workloads, because humans round up and real usage has a long tail.

Can I use spot instances for production Kubernetes workloads?

Yes, with discipline. Spot works well for stateless services with multiple replicas, pod disruption budgets, and graceful shutdown handling, and for anything batch or queue-driven. Keep stateful workloads and anything that can't tolerate a two-minute eviction notice on on-demand capacity. A mixed node group setup -- an on-demand baseline plus a spot pool that absorbs bursts -- is the pattern that survives contact with production.

How often should I review Kubernetes costs?

Weekly for the trend line, monthly for action. Usage drifts as teams ship, requests get copied forward from template to template, and yesterday's right-sized workload becomes today's idle waste. A monthly pass of comparing requests to the 95th percentile of usage, plus cleaning up orphaned namespaces and abandoned load balancers, keeps the drift from compounding into a scary invoice.

What I'd tell a teammate

The expensive part of Kubernetes isn't Kubernetes -- it's the distance between what you reserved and what you use. Close that gap with measurement (OpenCost, VPA recommendations), shrink requests toward p95 reality, and let spot capacity carry the workloads that don't mind.

Then calendar the review, because waste grows back like a lawn. The CFO's follow-up email, three months later, was also one line: "Keep doing whatever this is."