Argo Rollouts: Canary Deploys Without a Platform Team
At 4:12 on a Friday, we shipped a change to the ledger-api that looked harmless: a connection-pool tweak. By 4:19 the p95 latency graph looked like a hockey stick, and by 4:25 I was doing the thing every SRE has done -- a full rolling revert while composing the incident channel message in my head. The part that stung wasn't the bad deploy. It was that for seven minutes, a hundred percent of traffic was on a version we'd have rejected after thirty seconds of looking at a graph.
The next sprint I put Argo Rollouts in front of that service. No platform team, no service mesh, one engineer. The following quarter, the same class of bug shipped again. It went to ten percent of traffic, the error-rate analysis failed, the rollout aborted itself, and I found out about it from a Slack notification while making coffee. That's the sales pitch, and it's free.
Quick answer: Argo Rollouts replaces your Deployment with a Rollout CRD that supports canary steps: shift a percentage of replicas or traffic, pause, run metric analysis against Prometheus (or Datadog, etc.), and promote or abort automatically. You don't need a service mesh -- without one, weight is a replica ratio, which is coarse but useful. Install the controller and the kubectl plugin, convert a Deployment to a Rollout, add an AnalysisTemplate with a success condition on your error rate, and a bad deploy now dies at 10% traffic instead of 100%.
What you actually replace
A Rollout is a drop-in stand-in for a Deployment. Same pod template, same selectors -- the spec is deliberately familiar, with strategy swapped out:
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: ledger-api
namespace: payments
spec:
replicas: 10
revisionHistoryLimit: 5
selector:
matchLabels:
app: ledger-api
template:
metadata:
labels:
app: ledger-api
spec:
containers:
- name: ledger-api
image: registry.acme.com/ledger-api:1.20.0
ports:
- containerPort: 8080
strategy:
canary:
steps:
- setWeight: 10
- pause: {duration: 2m}
- setWeight: 40
- analysis:
templates:
- templateName: error-rate
- pause: {duration: 3m}Install is one controller (kubectl create namespace argo-rollouts plus the install manifest) and the kubectl-argo-rollouts plugin. Converting an existing service means changing kind: Deployment to kind: Rollout and adding the strategy. Keep revisionHistoryLimit at 5 or more -- that limit is what makes instant rollback possible later.
The dumbest canary still beats a rolling update
Before metrics, appreciate what the steps alone buy you. setWeight: 10 with no traffic manager spins up roughly 10% of replicas on the new version -- one pod out of ten -- and your plain Service sends it its share of traffic. A pause holds it there while a human (or a tired engineer's glance at a dashboard) decides. Even with zero automation, you've converted "100% of users hit the bug" into "10% of users hit the bug, briefly."
The step list reads top to bottom, and analysis steps block until the referenced AnalysisTemplate concludes. pause without a duration waits forever for kubectl argo rollouts promote, which is a perfectly good manual gate for prod while you're building trust in the automation.
Letting metrics do the promoting
The AnalysisTemplate is where Rollouts earns its name. Mine queries Prometheus for the 5xx rate, four times at 30-second intervals, and every measurement must pass:
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: error-rate
namespace: payments
spec:
metrics:
- name: http-5xx-rate
interval: 30s
count: 4
successCondition: result[0] < 0.01
failureLimit: 1
provider:
prometheus:
address: http://prometheus-operated.monitoring.svc:9090
query: |
sum(rate(http_requests_total{namespace="payments",app="ledger-api",status=~"5.."}[2m]))
/
sum(rate(http_requests_total{namespace="payments",app="ledger-api"}[2m]))A few things that took me a deploy or two to learn. count: 4 with interval: 30s means two minutes of measurement -- size that to how fast your failure modes show up; the connection-pool bug took ninety seconds to smoke. failureLimit: 1 means a single bad sample aborts; raise it if your metrics are noisy, lower it if your error budget is precious. And the query needs enough traffic to be meaningful -- a canary at 10% weight on a service doing three requests a minute will pass a poisoned version through pure silence. For low-traffic services, weight the canary higher or add a synthetic load check as a second metric.
When the analysis fails, the rollout aborts on its own: canary pods scale to zero, the stable version keeps serving, status goes Degraded. Nobody's Friday gets ruined.
Operating it day to day
The kubectl plugin's live view is what I keep open during a deploy:
$ kubectl argo rollouts get rollout ledger-api -n payments --watch
Name: ledger-api
Namespace: payments
Status: ॥ Paused
Message: CanaryPauseStep
Strategy: Canary
Step: 3/5
SetWeight: 40
ActualWeight: 40
Images: registry.acme.com/ledger-api:1.19.3 (stable)
registry.acme.com/ledger-api:1.20.0 (canary)
Replicas:
Desired: 10
Current: 14
Updated: 4
Ready: 14
Available: 14Your three verbs: kubectl argo rollouts promote ledger-api to push past a pause, kubectl argo rollouts abort ledger-api to kill the canary and reweight to stable, and kubectl argo rollouts undo ledger-api to roll back a fully-promoted bad release to the previous ReplicaSet. Because the old ReplicaSet is still sitting there (that's the revisionHistoryLimit doing its job), undo is seconds, not a redeploy. For watching the canary pods themselves behave -- restarts, readiness flapping under the new weight -- I keep the namespace open in conndeck alongside this, because the plugin view won't tell you the canary pod is CrashLooping until the analysis notices.
The gotchas that bite
Weight is replicas, not traffic, without a mesh. One canary pod on a 10-replica service gets *roughly* a tenth of requests, with kube-proxy's usual randomness. If you need exact percentages or "only internal users hit the canary," wire Rollouts to NGINX Ingress or Istio and it will program real traffic splits. Don't pretend replica ratio is that.
Analysis on empty metrics passes. A query that returns nothing because the canary got zero requests evaluates as success. Add a traffic floor (count on total requests, or a clamp_min guard) so silence fails closed.
HPA fights. If a HorizontalPodAutoscaler scales the Rollout, make sure it points at the Rollout resource, not an underlying ReplicaSet, and think about what scaling during a canary means for your weights. Rollouts handles this correctly only when it owns the scale target.
ArgoCD interaction. ArgoCD understands Rollout health natively, but a paused canary shows as Progressing -- expected, not stuck. And resist enabling self-heal on top of an aborted rollout; you want a human to read the failed AnalysisRun before retrying.
Frequently asked questions
Do I need a service mesh or special ingress to use Argo Rollouts?
No. Without a traffic manager, canary weight is just a replica ratio: setWeight 10 on ten replicas runs one canary pod, which receives roughly a tenth of the traffic through your normal Service. That's coarse but honest, and for many internal services it's enough. When you need exact percentages or header-based routing, add a supported ingress like NGINX or a mesh like Istio, and Rollouts will program it for you.
What's the difference between Argo Rollouts and Flagger?
Both do progressive delivery, but they attach differently. Argo Rollouts replaces your Deployment with its own Rollout CRD and controller, plus a kubectl plugin and dashboard. Flagger keeps your Deployment and drives the mesh or ingress on top of it. In practice, teams in the Argo ecosystem tend toward Rollouts and Flux teams toward Flagger; the capability overlap is large, so follow your tooling, not the feature list.
What happens when a metric analysis fails during a rollout?
The rollout aborts: the canary ReplicaSet scales to zero, all traffic returns to the stable version, and the Rollout's status shows Degraded with the failed AnalysisRun attached. Nothing else changes and no humans are paged unless you alert on it. Fix the problem, re-trigger the rollout with a new version or a retry, and the analysis starts fresh.
How do I roll back after a canary was fully promoted?
Run kubectl argo rollouts undo against the rollout, which shifts it back to the previous ReplicaSet -- essentially an instant rollback since the old pods are usually still warm. For this to work well, keep revisionHistoryLimit high enough that the previous stable ReplicaSet still exists. If the bad version shipped a database migration, no rollback tool saves you; that was always true.
Does Argo Rollouts work with ArgoCD?
Yes, cleanly. ArgoCD ships a built-in health check for the Rollout kind, so a paused mid-canary rollout shows as Progressing rather than Healthy, and sync waves and hooks work as usual. One thing to know: don't use ArgoCD's self-healing to fight an aborted rollout back into existence; retry the rollout deliberately after you've looked at why the analysis failed.
The takeaway
A canary with a two-minute pause and one Prometheus query would have caught every bad deploy we've had in the last year, and it takes an afternoon to set up. You don't need a mesh, a platform team, or permission from anyone.
Start with replica-ratio weights and a manual promote. Add the AnalysisTemplate once you trust it. The first time a rollout aborts itself while you're making coffee, you'll wonder why you waited.