SLOs for Kubernetes Services: Alerting on What Actually Matters
I once ran on-call for a service that paged, on average, 23 times a week. I know the number because I counted, during a week where I had been paged four times before breakfast on Tuesday and was looking for someone to blame.
The blame, it turned out, belonged to the alerting design. Of those 23 weekly pages, our retro found that two -- two -- corresponded to anything a user could have noticed. The rest were CPU spikes on a service that autoscaled, disk warnings on nodes that self-healed, latency blips lasting ninety seconds, and one alert that fired every Sunday at 2 AM when the backup job ran, like clockwork, for months. Nobody had fixed the backup job alert because fixing alerts wasn't anyone's job. Everyone had just quietly learned that pages meant nothing, which is the actual incident: the day a real page arrived, it got the same groggy dismissal as the other twenty-two.
We fixed it by throwing out almost everything and rebuilding around SLOs. It took a month, and pages dropped to about two a week -- nearly all of them real. This is the playbook.
Quick answer: SLO-based alerting means you stop paging on infrastructure symptoms (high CPU, pod restarts) and start paging on whether users are being hurt, measured by how fast the service is burning its error budget. Define an SLI (usually request success ratio from Prometheus metrics), pick a target like 99.9%, and alert with multi-window, multi-burn-rate rules so fast severe burns page immediately and slow burns file tickets. Then audit every remaining alert quarterly: if it fired and nobody acted, delete or downgrade it. Alerts are for things that require a human now -- everything else is a dashboard.
Symptom alerts lie to you
The classic Kubernetes alert set -- high CPU, memory pressure, pod restarts, node NotReady -- answers the question "is the infrastructure busy?" Users do not care if the infrastructure is busy. They care if their request succeeded, reasonably fast. These sound like the same thing and aren't: a CPU-saturated, autoscaled service serving every request fine is a non-event, while a service at 12% CPU returning 500s because a dependency's TLS cert expired is a five-alarm fire. Symptom alerts fire on the first and sleep through the second.
The shift is from "is something unusual happening" to "is the service meeting its promise." That promise, made measurable, is the SLO.
Define the SLI and the SLO
For a typical HTTP service, the SLI that carries the most weight is the success ratio, straight from your ingress or app metrics:
sli = sum(rate(http_requests_total{job="payments-api", status!~"5.."}[5m]))
/ sum(rate(http_requests_total{job="payments-api"}[5m]))Pick the SLO from what users actually tolerate, not from what sounds impressive. For most services that's 99.9% over a rolling 30 days: about 43 minutes of budget for failure per month. Every additional nine costs roughly an order of magnitude more engineering and buys less user happiness than the last one did. Write the SLI, the target, and the window down somewhere a PM can read it -- the SLO is a product decision wearing a PromQL costume, and getting that agreement upfront is what later lets you say "no, we're not paging on that."
That gap between 99.9% and 100% is the error budget: 0.1% of requests are *allowed* to fail. The budget converts reliability from a moral position into a spendable resource, which is exactly how you should treat it.
Burn-rate alerts: the multi-window pattern
Alerting on "error ratio > 0.1% right now" pages you for every blip. Alerting on monthly budget exhaustion tells you in week three that week one was bad. The pattern from the Google SRE workbook that threads the needle is multi-window, multi-burn-rate alerting: fire when the budget is being consumed fast enough to matter, checked over both a long window (is this real?) and a short window (is it still happening?).
The recording rules and alerts, for a 99.9% SLO with a 30-day window:
groups:
- name: slo.rules
rules:
- record: job:error_ratio_1h
expr: |
sum by (job) (rate(http_requests_total{status=~"5.."}[1h]))
/ sum by (job) (rate(http_requests_total[1h]))
- record: job:error_ratio_6h
expr: |
sum by (job) (rate(http_requests_total{status=~"5.."}[6h]))
/ sum by (job) (rate(http_requests_total[6h]))
- record: job:error_ratio_5m
expr: |
sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
/ sum by (job) (rate(http_requests_total[5m]))
- record: job:error_ratio_30m
expr: |
sum by (job) (rate(http_requests_total{status=~"5.."}[30m]))
/ sum by (job) (rate(http_requests_total[30m]))
# Fast burn: 2% of budget in 1 hour (14.4x). Pages.
- alert: ErrorBudgetBurnFast
expr: |
job:error_ratio_1h > 14.4 * 0.001
and job:error_ratio_5m > 14.4 * 0.001
for: 2m
labels:
severity: page
annotations:
summary: "{{ $labels.job }} is burning error budget fast"
# Slow burn: 100% of budget in 3 days (1x). Tickets.
- alert: ErrorBudgetBurnSlow
expr: |
job:error_ratio_6h > 6 * 0.001
and job:error_ratio_30m > 6 * 0.001
for: 1h
labels:
severity: ticketRead the math once and it sticks: a burn rate of 14.4 means the service will consume the entire 30-day budget in 2 days, so the long window (1h) confirms it's a sustained problem and the short window (5m) confirms it's happening *right now* -- no paging on an incident that already resolved itself. The slow-burn rule catches the low-grade bleed that never spikes but quietly eats the budget, and it files a ticket instead of a page because nobody needs to be awake for it at 4 AM. Two alerts replace an entire wall of symptom rules, and every page now means "users are being hurt, and it's getting worse."
The error budget is a steering wheel
Once the budget exists as a number, it starts making decisions for you -- which is the point. Budget mostly intact at mid-month? Ship the risky migration, raise the deploy cadence, this is what the budget is *for*. Budget nearly exhausted? Freeze non-essential deploys and point the team at reliability work until the rolling window recovers. The SRE book calls this an error budget policy; in practice it's two sentences in your team handbook and a dashboard everyone can see. It's the first mechanism I've found that ends the dev-velocity-versus-ops-stability argument, because both sides finally share one number instead of competing anxieties.
Killing the old alerts
The final step is the one teams skip: the audit. Export everything that fired in the last 90 days, and for each alert answer one question -- when this fired, did a human do something useful that couldn't have waited? Three buckets result. Keep: drove immediate action (almost always your burn-rate alerts and a small set of truly-critical symptoms, like cert expiry inside 7 days). Downgrade to ticket or dashboard: real signal, no urgency -- the Sunday backup alert lived here, and the job got fixed the following sprint once it was a ticket instead of a page everyone ignored. Delete: never once drove action. We deleted about 60% of our rule set, including, with some ceremony, the CPU alert that had paged us 200 times.
One rule of thumb from the audit: an alert that fires and gets acknowledged with nothing but a sigh is worse than no alert, because it trains the whole rotation to treat pages as noise. I keep the error-rate dashboards on a second monitor during incidents now (conndeck earns its keep here, watching pod health while the burn-rate math runs), but the pager only speaks when the budget does.
Frequently asked questions
What is an SLO in Kubernetes terms?
An SLO is a measurable promise about a service's behavior, like 99.9 percent of requests completing successfully in under 300 milliseconds, measured over a rolling 30 days. The measurement itself is the SLI, usually derived from Prometheus metrics like the ratio of non-5xx requests or histogram quantiles. The SLO is the target, the error budget is the tolerated gap between the target and perfection, and together they replace vibes with a number you can alert and plan on.
What is a burn rate alert?
A burn rate alert measures how fast you're consuming your error budget and pages when the rate is high enough to exhaust the budget before the window ends. A burn rate of 1 means you'll use exactly the budget over the window; 14.4 means you'll burn a month of budget in two days, which deserves a page. Alerting on burn rate instead of raw error ratio means brief blips that users barely notice don't wake anyone up, while sustained or severe degradation always does.
How many nines should my SLO have?
As few as your users will accept, because each nine is exponentially more expensive to build and operate. For most internal and B2B services, 99.9 percent -- about 43 minutes of error budget per month -- is the sweet spot; 99.95 and 99.99 demand redundancy and engineering time that should be justified by actual revenue impact. An SLO tighter than your dependencies or your users' ability to notice is waste, and an SLO looser than user patience is a slow-motion incident.
How do I reduce alert fatigue without missing real incidents?
Replace symptom-based alerts like high CPU with SLO burn-rate alerts that only fire when users are actually affected, then audit everything that's left. The audit is simple: for every alert that fired in the last 90 days, record whether a human did something useful in response. Anything that never drove action gets deleted, downgraded to a ticket, or re-tuned, because an alert that people ignore teaches the team to ignore alerts.
What's the difference between an SLO and an SLA?
An SLA is the contractual promise to customers, with financial consequences for missing it, while an SLO is the internal target you engineer against. Your SLO should be stricter than your SLA so the internal alarms trip well before the contract is breached. SLI is the third term: the actual measurement, like success ratio, that both of them reference.
What actually matters
Alert fatigue isn't a people problem, it's a design problem: pages that don't require humans train humans to ignore pages. SLOs and burn-rate alerting flip the question from "is something weird" to "are users hurting," and the multi-window pattern makes the pager precise enough to trust again.
Then audit ruthlessly and delete without mercy. The rotation that went from 23 weekly pages to 2 didn't get a better on-call engineer. It got an alerting system that finally meant what it said.