ArgoCD Sharding, Reconciliation Tuning, and the 1,000-App Wall
The hotfix itself was three lines of YAML. Getting it out took 31 minutes, and the alert that finally made us look (argocd-application-controller-0 CPU throttling, sync latency > 25 min) wasn't about a bad deploy at all. Three weeks after an aggressive onboarding push, we had quietly crossed 850 Applications on an ArgoCD instance sized for 150. Every sync that night, including an actual emergency, sat behind hundreds of routine refreshes. Nothing was technically "down," so nobody got paged for the part that mattered.
We were on ArgoCD v2.10.1, defaults almost everywhere: one controller replica, 20 status processors, 10 operation processors, 180-second reconciliation timeout. Those defaults are fine until the day they aren't.
Quick answer: Past a few hundred Applications, ArgoCD's bottleneck is the single application controller's reconciliation loop, and the fixes are sharding the controller, raising status and operation processor counts, and stretching timeout.reconciliation with jitter so refreshes stop stampeding. Sharding has a real cost: with the default round-robin assignment, changing the replica count reassigns nearly every cluster and triggers a full cache rebuild, so switch to consistent-hashing sharding first. The repo server is the second wall — monorepo manifest generation eats memory — and app-of-apps should become ApplicationSets well before four figures.
What actually breaks first: the reconciliation loop
Every Application gets reconciled on a timer (timeout.reconciliation in argocd-cm, default 180s), plus on events: git webhooks, syncs, health changes. Each run is worked by one of the status processors — a fixed pool of goroutines, 20 by default. Do the arithmetic: 850 Apps / 180s means ~4.7 refreshes per second of steady-state work alone. When a refresh starts taking longer than the slack in the system, a slow destination API server, an app with 400 resources, Redis latency creeping up, the queue grows and everything behind it waits.
The symptom users report is "my sync is stuck in Progressing." The actual state is workqueue_depth on the appOperationProcessing queue climbing while the controller passes every liveness probe.
Sharding, and the rebalance bill nobody mentions
The application controller is a StatefulSet; with more than one replica, each shard claims a subset of the *registered clusters* and reconciles only the Applications pointing at them. Sharding is per-cluster, not per-Application: if 700 of your 900 Apps target one management cluster, three replicas give that cluster's Apps exactly one shard. Zero improvement.
Configuration lives in argocd-cmd-params-cm:
data:
controller.replicas: "3"
controller.sharding.algorithm: "consistent-hashing"The default assignment (legacy round-robin over a hash of the cluster UID) has a nasty property: adding one shard reassigns roughly two-thirds of all clusters. Each gets its cache rebuilt on the new shard, informer resync plus manifest regeneration for every App on it, landing on the repo server all at once. I've watched a "routine" scale from 2 to 3 shards turn a 5-minute change into a 40-minute reconciliation brown-out.
Consistent-hashing sharding (available since v2.8, worth the upgrade by itself) limits movement to roughly 1/N of clusters when you add the Nth shard. Since v2.9 there's also controller.dynamic.cluster.distribution.enabled: "true", which balances by Apps per cluster rather than raw count, use it if your Apps are distributed as unevenly as everyone else's.
One honest warning: scale *up* before you're drowning, not during. A resharding event on a deep queue turns latency into an outage.
Processors, timeouts, and jitter
Once sharding is sane, the knobs are processor counts and the reconciliation timer:
# argocd-cmd-params-cm
data:
controller.status.processors: "50"
controller.operation.processors: "20"# argocd-cm
data:
timeout.reconciliation: "300s"
timeout.reconciliation.jitter: "60s"Status processors handle refresh work and should scale with App count; 50 per ~1,000 Apps is a starting point, not a law. Operation processors run actual syncs, writes against destination API servers, so keep them lower unless you enjoy etcd compaction alerts.
The jitter matters more than it looks. Without it, every App created during a bootstrap wave shares the same 180-second phase, and controller load arrives in tsunamis. While you're in argocd-cm, check kubectl.qps and kubectl.burst (defaults 50/100): a controller with raised processor counts can throttle against its own client-side rate limit and look, confusingly, like the API server is slow.
The repo server is the second wall
The controller gets the attention, but at our scale the repo server OOMed first. Every uncached manifest generation shells out to kustomize build or helm template, and on a monorepo each invocation loads a lot of YAML. A reshard or Redis flush invalidates the manifest cache (24-hour TTL by default), and suddenly hundreds of generations run concurrently against the same repo. We watched argocd-repo-server climb past 6 Gi working set and get OOMKilled mid-regeneration, which invalidates more cache, which causes more regeneration. It loops.
Two knobs do most of the work:
# argocd-cm
data:
reposerver.parallelism.limit: "4"Cap concurrency, give the pod a real memory request, and watch argocd_repo_pending_request_total, a rising value means generations are waiting on the semaphore and sync latency is repo-bound, not controller-bound. Also: if your monorepo Apps don't set spec.source.path narrowly, every git webhook forces the repo server to prove nothing relevant changed, repo-wide. Tight paths are free performance.
App-of-apps stops scaling before Applications do
App-of-apps is lovely at 20 children. At 200, the parent's health becomes a roll-up of every child, one flapping child keeps the parent Degraded forever, and child syncs effectively serialize behind the parent's own sync. We replaced ours with an ApplicationSet per environment (the controller ships with ArgoCD since v2.3): children now reconcile as independent Applications, in parallel, across shards. Keep app-of-apps for small, tightly-coupled bundles; everything else belongs in a generator.
Metrics to watch before users notice
histogram_quantile(0.95, sum(rate(argocd_app_reconcile_bucket[10m])) by (le)), reconciliation duration; alert on sustained growth, not spikes.workqueue_depth{name=~"appOperationProcessing|appRefresh"}, the queue users feel.argocd_repo_pending_request_total, repo server saturation.argocd_git_request_total{request_type="ls-remote"}, monorepo polling cost.container_memory_working_set_byteson repo-server and controller pods vs. request.
That last one is where conndeck earns its keep for me, I keep the ArgoCD namespace's pods and the Applications in one view, so "syncs are slow" and "repo-server is at 92% of its limit" show up in the same glance.
Frequently asked questions
How do I shard the ArgoCD application controller?
Set controller.replicas in argocd-cmd-params-cm to the number of shards you want and set controller.sharding.algorithm to consistent-hashing. Each replica of the argocd-application-controller StatefulSet then owns a slice of the registered clusters rather than every cluster on one pod. Plan the rebalance before you change the replica count, because every cluster that moves shards triggers a full cache rebuild and manifest regeneration for its Applications.
What status and operation processor values should ArgoCD use for 1,000 Applications?
Start with controller.status.processors around 50 and controller.operation.processors around 20 in argocd-cmd-params-cm, up from the defaults of 20 and 10. Status processors do the steady-state refresh work and should scale with Application count, while operation processors handle syncs and should stay lower because every sync spawns real API writes against the destination cluster. Watch the appOperationProcessing and appRefresh queues and raise values only while the depth is still growing.
Why does the ArgoCD repo server run out of memory on monorepos?
Manifest generation for Kustomize and Helm apps is done by shelling out to tools that load the whole chart or overlay into memory, and a big monorepo makes every one of those calls expensive. Concurrent generations multiply it, so an unbounded repo server on a monorepo will OOM under a reshard or cache expiry storm. Cap concurrency with reposerver.parallelism.limit in argocd-cm and give the repo server a generous memory request so the Go runtime and the forked tools have headroom.
When should I replace app-of-apps with ApplicationSets?
Replace app-of-apps when the parent Application itself becomes the bottleneck: child syncs serialize behind the parent's health assessment, a failed child blocks the parent's status, and adding a child means syncing the parent's manifests first. ApplicationSets generate flat, independent Applications that reconcile in parallel and don't couple your blast radius to one parent. Past roughly a hundred generated Applications, the app-of-apps pattern spends more time reconciling its own bookkeeping than deploying your workloads.
The 1,000-App wall isn't a number ArgoCD publishes; it's the day your defaults stop being invisible. If I could hand my past self one thing, it wouldn't be a bigger controller pod, it would be those five metrics on a dashboard, with alerts, six months before the page. Scale shards early, tune queues deliberately, and keep the wall theoretical.