conndeck Download

conndeck blog

ArgoCD Sync Waves: Ordering Your Deploys Without Losing Your Mind

#argocd#kubernetes#gitops#deployment

It was a Thursday night, around 11:40 PM, and our staging cluster rebuild had been failing for forty minutes. The pipeline was simple: one ArgoCD Application, about sixty manifests, cert-manager CRDs included. And every single sync died on the same line:

error: resource mapping not found for name: "internal-ca"
from "issuer.yaml": no matches for kind "ClusterIssuer" in version "cert-manager.io/v1"

The ClusterIssuer manifest was fine. The cert-manager install was fine, given about ninety seconds to actually register its CRDs. The problem was that ArgoCD applied both in the same breath, and the API server shrugged. Kubernetes has no idea what a ClusterIssuer is until the CRD lands. ArgoCD doesn't either.

Quick answer: ArgoCD sync waves order resources within a single Application's sync. Every resource gets a wave number via the argocd.argoproj.io/sync-wave annotation (default is wave 0), ArgoCD applies waves lowest-first, and it waits for each wave to be synced *and* healthy before starting the next. Use them for hard ordering dependencies — CRDs before custom resources, database before app — and keep it to three or four waves, because each wave is a health gate that can stall.

That night I finally stopped treating sync ordering as a nice-to-have and learned how sync waves actually work. This post is what I wish someone had handed me then.

The problem nobody warns you about

GitOps sells a beautiful lie: declare everything, apply everything, convergence sorts it out. Mostly true. Except Kubernetes is full of hard ordering dependencies that convergence papers over on a good day and face-plants on a bad one:

  • CRDs must exist before any custom resource of that type.
  • A namespace must exist before you put things in it (ArgoCD handles this one for you, thankfully).
  • The database migration Job should finish before the app rolls out expecting the new schema.
  • Your webhooks and operators should be serving before workloads that rely on their mutating behavior.

Left alone, ArgoCD applies resources in a reasonable default order within a sync (namespaces first, then CRDs, then other stuff), but "reasonable" has limits. It won't wait for your CRDs to be established before applying the CRs that use them, and it definitely won't wait for your postgres StatefulSet to be Ready before starting the API that crash-loops without it.

Sync waves are the escape hatch.

How sync waves actually work

A sync wave is just an annotation on a resource:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"

The mechanics, in plain terms:

  1. Every resource in an Application belongs to a wave. Default is wave 0.
  2. ArgoCD syncs wave by wave, lowest number first.
  3. Before moving to the next wave, it waits for everything in the current wave to be both synced and healthy.

That third point is the whole ballgame. A wave isn't "apply this first." It's "apply this, and don't proceed until it's actually running and passing health checks." A Deployment in wave 1 blocks wave 2 until its pods are Ready. That's the behavior you want for a database before an app.

And yes, negative waves are allowed. -5 syncs before -1, which syncs before 0. This is genuinely useful: annotate your one picky resource with -1 instead of renumbering forty other files to make room. I use negative waves constantly.

One scoping rule that trips people up: waves order resources within a single sync operation of one Application. They do nothing across separate Applications. If your "platform" app and your "tenant" app need ordering, you either merge them, or you go the app-of-apps route where the parent syncs child Applications — and yes, sync-wave annotations work on Application resources too, which is how most app-of-apps setups stage platform layers.

A realistic example

Here's a trimmed-down version of what that broken staging app looks like now. Three waves, negative one through one:

# wave -1: cert-manager itself (Helm chart rendered into the app,
# or a separate install manifest)
apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
---
# wave 0 (default): the issuer. CRDs are established by now.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca
  # no annotation needed; wave 0 is the default
spec:
  ca:
    secretName: internal-ca-keypair
---
# wave 1: the app that mounts certs issued by that issuer
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  annotations:
    argocd.argoproj.io/sync-wave: "1"
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: registry.example.com/api:v2.14.3

Same app, same manifests, but now the sync proceeds in stages and each stage gates the next on health.

If you're keeping a CRD and its CRs in one Application (I generally don't, but people do), you also want this on the Application, because ArgoCD's dry-run will choke on the unknown CR type before waves even get a chance:

metadata:
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true

For the database-before-app case, the pattern is identical: StatefulSet (plus its migration Job) in wave 0, the app in wave 1. ArgoCD's built-in health assessment for StatefulSets waits for Ready replicas, so wave 1 literally cannot start until postgres is up. No init containers with nc -z hacks required.

Mistakes I've made so you don't have to

Waves that don't wait for anything. The health gate only works if the resource has a meaningful health status. Built-in kinds do. Random CRDs do not, by default — ArgoCD treats many custom resources as healthy the moment they exist. If your operator installs in wave 0 and reports its real readiness through a status field, you need a custom Lua health check in the ArgoCD ConfigMap (argocd-cm, resource.customizations.health.<group_kind>) or wave 1 will fire while the operator is still pulling its image. Ask me how I know. Actually, don't — it involved a Linkerd control plane and a very quiet incident channel.

Mixing waves and hooks carelessly. Hooks (argocd.argoproj.io/hook: PreSync|Sync|PostSync|SyncFail) and waves solve adjacent problems and interact in ways that surprise people. PreSync hooks run before *any* wave. PostSync hooks run after *all* of them. Sync-phase hooks run within the wave you assign them, alongside regular resources of that wave. So a migration Job annotated as a plain resource in wave 0 runs concurrently with everything else in wave 0 — if the database is also wave 0, the migration may run before postgres is Ready. Fix: database in wave 0, migration as a Sync hook in wave 1, app in wave 2. Draw it on a whiteboard once and it'll stick forever.

Over-waving. I've seen apps with eleven waves where the author clearly annotated every resource in the order they'd kubectl apply them by hand. Each wave is a health gate, which means each wave is latency and another place a sync can stall. Three or four waves covers nearly every real dependency graph. If you need eleven, the app probably wants splitting, not more annotations.

Expecting waves to fix cross-cluster or cross-app ordering. They won't. An ApplicationSet generating one Application per cluster gives you independent syncs; wave annotations inside the templates order resources within each cluster, not rollouts across clusters. That's what rollout strategies and progressive delivery tooling are for.

Watching waves actually work

The part that sold me on waves wasn't the annotation, it was watching a sync behave like a pipeline for the first time: namespace Healthy, cert-manager Healthy, issuer Healthy, then the app finally rolling. ArgoCD's UI shows wave numbers per resource if you look, but I'll admit I usually watch this from the cluster side — seeing pods come up in dependency order in a live view of the namespace is exactly the kind of thing we built conndeck to make obvious, because "is wave 1 stuck or just slow" is a question you answer with your eyes, not with argocd app get.

Frequently asked questions

What is the argocd.argoproj.io/sync-wave annotation?

It's an annotation you put on a resource to assign it to a numbered sync wave, and every resource without one sits in wave 0 by default. ArgoCD syncs waves from lowest number to highest, and it won't move to the next wave until everything in the current one is both synced and healthy. That's what turns a flat apply into ordered, health-gated stages.

Do ArgoCD sync waves work across multiple Applications?

No. Waves only order resources within a single sync of one Application; separate Applications sync independently and ignore each other's wave numbers. If you need ordering between apps, the usual move is app-of-apps, where sync-wave annotations on the child Application resources themselves stage the platform layers.

What's the difference between ArgoCD sync waves and hooks?

Waves order and health-gate regular resources during the Sync phase, while hooks (PreSync, Sync, PostSync, SyncFail) run jobs around or inside that process. PreSync hooks run before any wave, PostSync hooks run after all of them, and Sync hooks run inside the wave you assign them. A migration Job, for example, usually wants to be a Sync hook in a wave after the database, not a plain resource sharing the database's wave.

Can sync waves wait for a custom resource to be ready?

Only if ArgoCD has a meaningful health check for that kind. Built-in kinds report health properly, but many CRDs are treated as healthy the moment they exist, so the next wave fires too early. The fix is a custom Lua health check in the argocd-cm ConfigMap so the wave actually gates on real readiness.

Can I use negative numbers for sync waves?

Yes, negative waves are fully supported and sync before the default wave 0. Annotating one picky resource with -1 beats renumbering every other file in the app to make room at the front. I use them constantly.

The takeaway

Sync waves are a small feature with an outsized payoff: one annotation, real ordering, health-gated stages, and an end to no matches for kind at midnight. Keep the count low, respect the health check underneath each gate, and be deliberate about hooks versus waves. Do that, and ordering your deploys stops being the thing that keeps you up on Thursdays.