Promoting Between Environments in GitOps Without Losing Your Mind

I have a scar from the old way. It was March, and "promoting to prod" meant me, a terminal, and a folder called prod-manifests-final-v2 that I rsynced against staging and applied by hand. That night I promoted staging's ConfigMap along with the deployment. Staging's ConfigMap pointed at the staging database. We replayed four hours of transactions and I rewrote the release process the following week.

The rewrite stuck, and the core of it is one idea: promotion is a git operation, not a cluster operation. Once you internalize that, most of the "how do I do environments in GitOps" anxiety dissolves. Here's the layout and the workflow that have survived two years and one acquisition since.

Quick answer: Keep one config repo with a Kustomize base per app and an overlay per environment. Never copy manifests between environment folders -- the base is shared, the overlay holds only what differs. Promotion means a PR that moves the exact image tag or digest from the staging overlay to the prod overlay; CI opens it, a human merges it, the reconciler applies it. The artifact being promoted is immutable: same digest, never rebuilt, never retagged. Hand edits to clusters are forbidden because they either get reverted or become silent drift.

The repo layout decision first

Before overlays, settle where things live. The two-repo pattern has earned its keep: the app repo holds source code and the CI that builds images, and the config repo holds the manifests the reconcilers watch. CI in the app repo gets write access to exactly one thing -- opening PRs against the config repo -- which keeps your audit trail clean and your credentials scoped.

Inside the config repo, one directory per environment, not one repo per environment:

apps/
  ledger-api/
    base/
      deployment.yaml
      service.yaml
      kustomization.yaml
    overlays/
      staging/
        kustomization.yaml
        patch-resources.yaml
      prod/
        kustomization.yaml
        patch-resources.yaml
clusters/
  staging/
    ledger-api.yaml     # the Flux Kustomization or ArgoCD Application
  prod/
    ledger-api.yaml

Separate repos for staging and prod sound clean until promotion day, when "copy the tag from repo A to repo B" becomes a script someone wrote in 2024 that nobody owns. In one repo, promotion is a diff you can read in a single PR. Split only when a compliance boundary or genuinely different owning team forces it.

Overlays, not copies

The rule that prevents my March incident: environment folders contain *differences*, never full manifests. The base holds the deployment as the app actually is. The overlay holds replicas, resources, ingress hosts, and the image tag:

# apps/ledger-api/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: payments
resources:
  - ../../base
images:
  - name: registry.acme.com/ledger-api
    newTag: "1.19.3"  # updated by CI promotion PRs only
patches:
  - path: patch-resources.yaml
  - path: patch-replicas.yaml

Because the ConfigMap lives in the base and environment-specific values live in overlays, there is no prod-manifests-final-v2 folder and no way to drag a staging database URL into prod by copying a file. If you catch yourself duplicating a whole manifest between overlays, that's the smell -- pull it into the base and patch the difference. Copy-paste between environments is how drift is born.

The thing that actually moves: image tags

Day to day, 95% of what "promotion" means is bumping an image tag. Two rules make it safe.

Immutable tags only. No latest, no mutable v1.20 that gets re-pushed. Use full versions or digests. The tag pinned in git is simultaneously your deployment record, your audit trail, and your rollback button. With latest, git says nothing about what's running and rollback means hoping the registry still has the old image.

CI writes tags, humans merge PRs. When app CI builds 1.20.0, it opens a PR against the config repo updating the *staging* overlay -- kustomize edit set image in the overlay directory does this cleanly:

$ cd apps/ledger-api/overlays/staging
$ kustomize edit set image registry.acme.com/ledger-api=registry.acme.com/ledger-api:1.20.0
$ git diff
-    newTag: "1.19.2"
+    newTag: "1.20.0"

Staging's reconciler picks it up, soak tests run, and then the promotion PR -- which can also be CI-generated -- copies that exact tag into the prod overlay. The digest that ran in staging is the digest that runs in prod. Same artifact, no rebuild, no "it worked in staging but we rebuilt for prod." For extra rigor, promote by digest (newTag plus digest: in the images transformer) so a registry mishap can't swap bytes under a tag.

The promotion PR is your release process

This is the part teams under-appreciate: the PR diff *is* the release. One line changing in one file, from 1.19.3 to 1.20.0, with the app-repo changelog linked in the description. Reviewers can answer "what's going out" in five seconds because the diff is complete and honest -- nothing ships except through it.

A few habits that keep it honest. CODEOWNERS on overlays/prod/ so prod changes need a platform review, while staging merges freely. Batch promotions deliberately: if three services promote together every Tuesday, make that a visible convention rather than an accident. And when someone asks "what's deployed to prod right now," the answer is git show main:apps/ledger-api/overlays/prod/kustomization.yaml, not a Slack thread. When I want to confirm the cluster agrees with git, I check the live image on the running pods -- conndeck shows it in the workload view -- but git is the answer that matters.

Where this falls apart

Hotfixes bypassing the pipeline. A kubectl edit during an incident is either stomped by the reconciler (confusing) or silently persists (dangerous, and the next promotion reverts it mid-incident -- the worst timeline). Hotfixes go through the same PR path, expedited. Make the fast path fast instead of making a side door.

Environment-specific secrets. They don't belong in the config repo at all -- ExternalSecrets pointing at Vault or a cloud secret manager, with per-environment paths, so the overlay holds a reference and never a value.

Skipping staging "just this once." The whole scheme's integrity comes from the rule that prod only runs what staging ran. Break it and you're back to rsync with extra steps.

Overlay rot. Twice a year, diff the rendered environments (kustomize build overlays/staging > /tmp/s && kustomize build overlays/prod > /tmp/p && diff /tmp/s /tmp/p) and ask whether every difference is intentional. The answer will embarrass you at least once.

Frequently asked questions

Should all environments live in one GitOps repo or several?

One config repo with a directory per environment is right for most teams: promotion becomes a plain PR, diffs between environments are one command away, and there's exactly one place to audit. Split into multiple repos when you have a hard boundary -- compliance separation, different teams owning different environments, or prod access that must not overlap with staging access. Split repos make promotion a cross-repo dance, so don't pay that cost without a reason.

How do I promote an image from staging to production in GitOps?

Merge a pull request that copies the exact tag or digest from the staging overlay into the prod overlay. The critical rule is that the artifact never changes: you promote the same digest that staging ran, you never rebuild and never retag. CI can open this PR automatically after staging soak tests pass, but a human should still see the diff, because the diff is the release notes.

Why shouldn't I use the latest tag with a GitOps reconciler?

Because git stops being the source of truth for what's running. With latest, the cluster contents depend on when the registry was last pulled, rollbacks have nothing to roll back to, and two clusters can run different code while showing identical git state. Pin an immutable tag or digest in the overlay, and let CI update it via PR. The tag in git is your audit trail and your rollback button.

Where do environment-specific secrets go in a GitOps repo?

Nowhere in the repo, ideally. Use External Secrets Operator or a similar tool so the manifest in git holds a reference to the secret's location in Vault or a cloud secret manager, not the value. If you're still on SOPS or Sealed Secrets, at minimum keep per-environment ciphertext in the environment's overlay with per-environment keys, never shared ciphertext across environments.

How do I handle an emergency hotfix to production in GitOps?

Through the same pipeline, just faster: a small PR against the prod overlay, expedited review, merged in minutes. The reconciler applies it and git now matches reality. What you must not do is kubectl edit the cluster, because either your reconciler fights you by reverting it, or worse, it doesn't, and you've created silent drift that the next promotion will blow away at the worst possible moment.

What I'd tell a teammate

Promotion in GitOps is a one-line diff reviewed by a human and applied by a robot. Everything else -- the overlays, the immutable tags, the CI-opened PRs -- exists to keep that sentence true.

Set it up once, forbid the side doors, and "what's in prod right now" becomes a question git answers instead of a question you dread.