kube-prometheus-stack: Monitoring That Won't Page You at 3AM

At 3:14 AM on a Sunday, my phone went off with the page that finally broke me: CPUThrottlingHigh on a namespace running a batch job that had been throttled, by design, for six months.

I did what you do. I got up, opened the laptop, checked the service's latency dashboards. Flat. Checked the job. Progressing happily at the speed it was configured to progress at. Nothing was wrong. Nothing had ever been wrong in the two hundred previous times this alert fired. I closed the laptop, lay awake for an hour doing math about how many nights like this the on-call rotation had burned, and decided on Monday morning we were going to fix our monitoring stack properly.

We ran kube-prometheus-stack, like everyone else. The chart is genuinely excellent -- one Helm install and you have Prometheus, Alertmanager, Grafana, kube-state-metrics, and node exporter with dashboards that would take months to build by hand. The problem is it ships with an alert rule set designed to be defensible for every cluster on Earth, which means it's tuned for none of them. Here's how I turned it into something that pages me when things are actually broken.

Quick answer: kube-prometheus-stack is the standard one-chart install for Prometheus, Alertmanager, Grafana, and friends on Kubernetes, and its defaults are a starting point, not a finished setup. The three jobs after helm install are: audit and tune the default alert rules (disable or re-threshold the noisy ones like CPUThrottlingHigh), add recording rules for the expensive queries your dashboards and alerts run constantly, and control cardinality before Prometheus eats all your RAM. None of this requires replacing the chart -- it's all values.yaml, PrometheusRules, and relabeling.

Audit the default alerts before they audit you

The chart ships a large default rule set via defaultRules, and some of those rules are famous for the wrong reasons. CPUThrottlingHigh fires when a container's CPU is throttled more than 25% of the time -- which happens constantly to any workload with tight limits, and almost never means a user noticed. KubeMemoryOvercommit fires on clusters that are intentionally overcommitted, which is most clusters. Watchdog fires forever, on purpose, to prove your alerting pipeline works.

Before you touch anything, look at the last 90 days of pages and sort by alert name. In our case the top five alerts produced 94% of pages and approximately 0% of actions. That's your tuning list.

The chart gives you two clean levers in values.yaml. First, disabling individual default alerts:

defaultRules:
  disabled:
    CPUThrottlingHigh: true
    KubeMemoryOvercommit: true

Second, overriding a rule that has the right idea but the wrong numbers. Don't fork the upstream rule set -- disable the default and ship your own version as a PrometheusRule:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: tuned-alerts
  namespace: monitoring
  labels:
    release: monitoring
spec:
  groups:
    - name: kubernetes-apps-tuned
      rules:
        - alert: KubePodCrashLooping
          expr: |
            increase(kube_pod_container_status_restarts_total{namespace!~"kube-system|monitoring"}[15m]) > 3
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} restarting frequently"

Note the two deliberate changes from the stock rule: a higher restart threshold, and a namespace filter so your monitoring stack's own growing pains don't page anyone. The release: monitoring label (matching your Helm release name) is what makes the operator pick the rule up -- forget it and your rule silently does nothing. Ask me how I know.

Recording rules: stop paying for the same query 400 times a day

Open your Grafana dashboards and count how many panels run some variation of sum by (namespace) (rate(container_cpu_usage_seconds_total{...}[5m])). Every refresh, on every open tab, for every engineer. Prometheus happily recomputes it, and it happily melts.

Recording rules evaluate the expression once, on an interval, and store the result as a new series:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: recording-rules
  namespace: monitoring
  labels:
    release: monitoring
spec:
  groups:
    - name: capacity.rules
      interval: 30s
      rules:
        - record: namespace:container_cpu_usage:rate5m
          expr: |
            sum by (namespace) (
              rate(container_cpu_usage_seconds_total{container!="", image!=""}[5m])
            )
        - record: namespace:container_memory_working_set:sum
          expr: |
            sum by (namespace) (
              container_memory_working_set_bytes{container!="", image!=""}
            )
        - record: namespace:cpu_requested:sum
          expr: |
            sum by (namespace) (
              kube_pod_container_resource_requests{resource="cpu"}
            )

That last trio is also your capacity story: usage vs requests per namespace, precomputed. Now dashboards query namespace:container_cpu_usage:rate5m -- one series per namespace instead of thousands of raw series -- and the alert namespace:container_cpu_usage:rate5m / namespace:cpu_requested:sum > 0.95 evaluates in milliseconds. Follow the level:metric:operation naming convention and future you will be able to find things.

Cardinality: the bill you pay in RAM

Prometheus doesn't charge you per cluster; it charges you per active time series. Every unique combination of metric name and label values is a series, and every one of them lives in RAM. When Prometheus falls over or starts eating 30 GB, cardinality is almost always why.

Find the offenders first:

topk(10, count by (__name__)({__name__=~".+"}))

On a typical cluster you'll get something like:

apiserver_request_duration_seconds_bucket{}     74218
container_cpu_usage_seconds_total{}             51930
etcd_request_duration_seconds_bucket{}          31400
kube_pod_labels{}                               22864

Three fixes cover most of this. First, drop the histogram bucket series you never query, at scrape time, with metricRelabelings in values.yaml:

kubeEtcd:
  serviceMonitor:
    metricRelabelings:
      - action: drop
        sourceLabels: [__name__]
        regex: etcd_request_duration_seconds_bucket

Dropping at scrape time beats dropping at ingest because the series never gets created. Second, tame kube_pod_labels, which copies every pod label into a metric -- and in clusters where CI stamps builds with commit SHAs and ticket IDs, that's a cardinality bomb:

kube-state-metrics:
  metricLabelsAllowlist:
    pods:
      - app_kubernetes_io/name
      - app_kubernetes_io/instance

Third, fix your own instrumentation. If your app exposes http_requests_total{path="/users/8f3a-..."}, no chart setting will save you. Route-level labels with bounded value sets, always.

The rest of the levers

A few values I change on every install, quickly: prometheus.prometheusSpec.retention (default is 10d; 15d to 30d is the sweet spot before you want Thanos or Mimir anyway), retentionSize so a cardinality spike fills disk rather than OOMing the pod, and a real storageClass so the data survives a node replacement. And give Alertmanager actual receivers -- the chart default is a null receiver, which is a charming way to discover, during an incident, that your alerts have been going nowhere.

The loop I run quarterly: export the firing-alert history, mark each alert "acted on" or "noise", and delete or re-tune the noise. Alert quality decays as systems change. Treat the rule set like code you own, because after the values.yaml tuning, it is.

Frequently asked questions

What is kube-prometheus-stack?

It's a Helm chart maintained by the prometheus-community org that installs Prometheus, Alertmanager, Grafana, kube-state-metrics, the node exporter, and the Prometheus Operator in one shot, wired together with a sane set of default dashboards and alert rules. It's the de facto standard way to monitor a Kubernetes cluster. The catch is that the defaults are designed to work everywhere, which means they're tuned for nobody in particular.

Which default alerts in kube-prometheus-stack should I disable?

CPUThrottlingHigh is the classic one -- it fires constantly on containers with low CPU limits and rarely correlates with user-visible harm, so most teams either disable it or raise its threshold dramatically. KubeMemoryOvercommit and some of the kubelet and scheduler rules also tend to misfire on small clusters. The right move is an audit: look at what actually paged you in the last 90 days, and disable or re-tune anything that never led to a human doing something.

How do I reduce Prometheus cardinality?

First find the offenders with topk on a count by metric name, then act. The usual fixes are dropping high-cardinality histogram buckets you never query with metricRelabelings, stripping unbounded labels from kube-state-metrics with metricLabelsAllowlist or a denylist, and removing labels that leak things like request IDs or user agents from your own instrumentation. Histogram bucket metrics and kube_pod_labels are the two biggest offenders I see in the wild.

What are recording rules and why should I use them?

Recording rules precompute a PromQL expression on a schedule and store the result as a new time series. You use them because the raw query behind a dashboard panel -- summing rates across thousands of series with regex matchers -- is expensive every time someone hits refresh, and the recorded version is nearly free. They also give your alerts a stable, precomputed signal so a slow query never delays a page that matters.

How much memory does Prometheus need per cluster?

A rough rule of thumb is a few bytes per sample in the head block, so the driver is your active series count times scrape frequency, not the cluster size as such. A mid-size cluster with the kube-prometheus-stack defaults lands around 500k to 1M active series and is comfortable in 4 to 8 GB. If you're well above that, cardinality is the problem, not hardware, and more RAM is treating the symptom.

The rules I run with

kube-prometheus-stack out of the box is a great monitoring system for a cluster that doesn't exist: average-sized, average-labeled, with an average on-call who loves 3 AM pages about CPU throttling. Yours isn't that cluster.

Audit the pages, tune the rules, record the expensive queries, drop the cardinality you don't need. An hour of values.yaml buys you a monitoring stack that earns trust -- and an on-call rotation that stops flinching at its own phone.