Managing a Fleet of Clusters Without Losing Track of Reality

The message in the incident channel read, in full: "why is the fraud service scaled to 0 in eu-west."

It was scaled to 0 because I had scaled it to 0. In my head, I was in our dev cluster, where I'd been load-testing scale-down behavior twenty minutes earlier. My terminal had other ideas: the context was still set to eks-eu-prod, one kubectx command ago, and kubectl scale deploy/fraud-scorer --replicas=0 executes instantly and without opinion in any cluster you point it at. Recovery took ninety seconds. The post-incident writeup, the new safeguards, and my lingering embarrassment took considerably longer.

We had eleven clusters by then -- a couple of dev, staging, and production per region, the way it always grows. Two clusters is a pair. Eleven is a fleet, and a fleet needs systems, not memory. Here's what I put in place after that afternoon, and what I'd do from day one if I had it to do again.

Quick answer: Multi-cluster pain has three roots: context-switching accidents, no fleet-wide visibility, and config drift between clusters. The fixes, in order: strict kubeconfig hygiene (one file per cluster, short context names, prompt integration), per-cluster Prometheus remote-writing to a central store so one Grafana sees everything, and GitOps that fans out to clusters automatically via ArgoCD ApplicationSets or Fleet so platform changes are one commit, not twenty. Below ~5 clusters you can survive on discipline; above that you need the tooling.

Context switching is where bodies are buried

The fraud-scorer incident was a context problem, and context problems are solved with hygiene, not willpower. The setup that has kept me honest since:

One kubeconfig file per cluster, in ~kube/clusters/, combined via the environment variable:

$ ls ~/.kube/clusters/
dev-us.yaml      staging-us.yaml  prod-us.yaml
dev-eu.yaml      staging-eu.yaml  prod-eu.yaml
$ export KUBECONFIG=$(ls ~/.kube/clusters/*.yaml | tr '\n' ':')
$ kubectl config get-contexts
CURRENT   NAME       CLUSTER                AUTHINFO
          dev-us     dev-us.cluster.local   dev-us-admin
*         prod-eu    prod-eu.cluster.local  prod-eu-admin

Short, unambiguous context names -- rename them with kubectl config rename-context if your cloud provider hands you an ARN novel. One file per cluster means credentials are scoped, rotatable, and obvious; a single hand-merged mega-kubeconfig is where stale tokens and mystery contexts breed. kubectx for switching, and -- this is the part that actually saved me -- the current context in my shell prompt via starship, in red when it contains the string prod. You will still eventually run something in the wrong place; the goal is to make the wrong place *loud*.

For the genuinely dangerous operations, add friction on purpose: separate AWS/GCP profiles per environment so prod creds require a distinct login, or an admission policy that requires an annotation for scale-to-zero. Feels bureaucratic until the first time it eats a command meant for dev.

Fleet-wide visibility: one place that knows everything

The second failure mode is quieter: each cluster has its own Prometheus, its own Grafana, its own alerts, and "is anything on fire" becomes an eleven-tab browser exercise. Someone misses the tab where a cluster has been NotReady for six hours. Ask me how I know. Actually, don't.

The architecture that works is federated by push, not pull. Every cluster runs its own local Prometheus (fast, independent, survives network partitions) and remote-writes to a central store -- Thanos, Mimir, or a managed offering:

# kube-prometheus-stack values, per cluster
prometheus:
  prometheusSpec:
    externalLabels:
      cluster: prod-eu
      region: eu-west-1
    remoteWrite:
      - url: https://mimir.infra.example.com/api/v1/push
        queueConfig:
          maxSamplesPerSend: 1000

The externalLabels block is the load-bearing detail: every series from this cluster arrives centrally stamped with cluster="prod-eu", which means one Grafana can answer both "error rate across the fleet" and "error rate in prod-eu" with the same query plus a label matcher. Alerts follow the same pattern -- rules evaluate locally in each cluster, Alertmanager routes centrally with the cluster label attached, so the page tells you *where* before you even open a laptop.

The rule I'd tattoo on the onboarding doc: no cluster joins the fleet without remote-write and alert routing configured. A cluster you can't see from the central dashboard is a cluster you're not really running.

GitOps per cluster, defined once

The third failure mode is drift. You install the ingress controller on cluster one in March. Cluster four gets it in June, newer version, slightly different values because you copied from a wiki page that had been edited. Eighteen months later you have eleven subtly different platforms and every bug report begins with "which cluster?"

The fix is defining platform components once and fanning out automatically. With ArgoCD, that's an ApplicationSet with the cluster generator:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-ingress
  namespace: argocd
spec:
  generators:
    - clusters:
        selector:
          matchLabels:
            environment: prod
  template:
    metadata:
      name: "ingress-{{ name }}"
    spec:
      project: platform
      source:
        repoURL: git@github.com:example/platform-addons.git
        path: charts/ingress-nginx
        targetRevision: v1.11.2
        helm:
          values: |
            controller:
              replicaCount: 3
      destination:
        server: "{{ server }}"
        namespace: ingress-nginx
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Every registered cluster labeled environment: prod gets the same ingress chart at the same pinned version, and adding cluster twelve to the fleet is "register it with ArgoCD," full stop. Cluster labels carry the legitimate differences -- region, size class, compliance tier -- so divergence is declared in metadata rather than discovered by diff at 2 AM. One revision bump, one PR, every cluster converges. If you're not an ArgoCD shop, Rancher Fleet and Flux with its Kustomization-per-cluster pattern do the same job; the tool matters less than the principle that per-cluster hand-edits are forbidden.

Watching twelve clusters converge on a new add-on version is also where I appreciate a good multi-cluster GUI -- I keep conndeck pointed at the fleet during those rollouts, because "did every cluster actually take the change" is a glance, not eleven terminal tabs.

Know when discipline stops scaling

An honest word on scope: below five clusters, all of the above can be a Makefile, a shell prompt, and a habit, and that's fine -- don't buy a fleet management platform to run a dev cluster and a prod cluster. The forcing functions to level up are: you can't answer "what version of X is running where" without checking, you've had a wrong-context incident, or onboarding a cluster takes a human more than a day. Any one of those means the fleet has outgrown memory and it's time for the ApplicationSet, the central metrics store, and the prompt that turns red.

Frequently asked questions

How do I manage kubectl across many clusters?

Keep one kubeconfig file per cluster in a directory and point KUBECONFIG at all of them, or use kubectx to switch contexts by short name. The critical discipline is never letting a context named something vague like prod stay current between sessions, because the classic multi-cluster incident is running a destructive command against the wrong cluster. Renaming contexts to short unambiguous names and making your shell prompt show the current context prevents most of these.

When do you need a fleet management tool like Rancher Fleet or ArgoCD?

The tipping point is usually somewhere between five and ten clusters, or whenever you catch yourself applying the same manifest change by hand to a third cluster. Below that, per-cluster GitOps with a shared repo of platform components works fine. Above it, you need a tool that treats clusters as a target set, so an add-on upgrade is one commit instead of twenty pull requests and a spreadsheet.

How do I get visibility across all my clusters at once?

Each cluster runs its own Prometheus and remote-writes to a central store like Thanos, Mimir, or a managed service, so one Grafana queries the whole fleet with a cluster label to split by. Alerts follow the same path: generated locally in each cluster, routed to one Alertmanager or incident tool with the cluster label attached. The alternative, scraping every cluster from a central Prometheus, becomes a firewall and reliability headache fast.

How do I keep add-on versions consistent across clusters?

Define each platform component once in Git with a pinned version, then use ArgoCD ApplicationSets with the cluster generator or a similar fan-out mechanism so every cluster gets the same set of Applications automatically. Cluster labels drive the exceptions, like region-specific values, so consistency is the default and divergence is explicit. Whatever you do, stop hand-editing per-cluster copies, because that is how you end up with eleven subtly different ingress controllers.

Should every cluster have its own kubeconfig file?

Yes, one file per cluster is the cleanest setup, combined with the KUBECONFIG environment variable listing them all so kubectl still sees every context. It keeps credentials scoped so a leaked config exposes one cluster, lets you check individual configs into vault or a password manager cleanly, and makes it obvious which cluster a file belongs to. One giant merged kubeconfig with copied snippets is where stale credentials and mystery contexts come from.

The parting advice

A fleet of clusters fails in three boring ways: wrong-context commands, invisible clusters, and drift. All three are process problems wearing a tooling costume, and all three have known fixes -- kubeconfig hygiene, central metrics with cluster labels, and GitOps that fans out instead of copy-pastes.

The fraud scorer survived. My confidence took longer to recover. Build the safeguards before the fleet builds them for you, mid-incident.