GitOps Is a Discipline, Not a Tool

Two years ago I watched a team of seven engineers run ArgoCD v2.5 and kubectl apply from their laptops at the same time, for eight months straight. ArgoCD was there. The Applications were green. Every service in the cluster had a little badge saying "Synced" in the UI, and every afternoon someone would still SSH to a bastion and hot-patch a deployment by hand because "it's just a one-line change."

The one-line change was a memory limit. It got overwritten by the next sync. Then it got re-applied by hand. Then someone added IgnoreDifferences for that field, and from that point on nobody actually knew what was running.

That's the story I think about whenever someone tells me they've "adopted GitOps."

Quick answer: GitOps isn't something you install — it's a rule the team agrees to: the desired state of the cluster lives in Git, and Git is the only way in. ArgoCD or Flux can reconcile that state, but no controller stops someone from running kubectl apply from their laptop; only revoked write access, a fast hotfix-through-Git path, and treating drift as a bug do. If a change doesn't come from a merged pull request, it doesn't exist — that's the whole discipline, and the tool is maybe ten percent of it.

The install is the easy part

ArgoCD is genuinely easy to stand up now. You apply the manifests, you point it at a repo, and within an hour you have a dashboard full of green hearts and a sync loop running every three minutes. It feels like progress. It looks great in a demo to your VP.

But here's the uncomfortable bit: nothing about installing a sync controller stops a human from running kubectl edit at 4:47 PM on a Friday. The controller will reconcile the drift back in a few minutes, sure. Or it won't, because someone set the app to manual sync "temporarily" in March and nobody remembers why. Either way, the cluster and the repo are now two sources of truth with a treaty that's held together by good intentions.

I've been on both sides of this. I was once the person saying "we have ArgoCD now, we're doing GitOps." What we were actually doing was running a very expensive dashboard that watched us deploy things the same way we always had.

GitOps is a promise the team makes, not software you install

The actual definition that matters is boring: the desired state of the cluster lives in Git, and Git is the only way in. Not the preferred way. The only way. If a change doesn't come from a merged pull request, it doesn't exist.

That's a discipline, and like most disciplines it's mildly annoying. It means the senior engineer who's been running this cluster since it was three nodes on kubeadm doesn't get to fix things by hand anymore, even when he's right. It means the platform team's repo permissions matter more than their cluster RBAC, because the repo is where the power actually lives now. And it means your incident process changes shape: at 2 AM, the fix isn't a command, it's a PR with a fast review path.

None of that comes in the Helm chart.

The team I mentioned earlier only got there after the memory-limit incident turned into a small outage. The postmortem action item wasn't "configure ArgoCD better." It was a rule, written down, agreed to by everyone including the two people who grumbled about it: nobody touches the cluster directly, period. Read-only kubeconfigs for production. If you need to see something, you look. If you need to change something, you open a PR.

The habits that make it real

A few specific things separated the teams I've seen do this well from the ones with a pretty dashboard and kubectl-shaped holes in their process.

First, they revoked write access instead of relying on willpower. Willpower loses every single time there's a paging alert. The good teams made the manual path technically hard: production contexts in everyone's kubeconfig pointed at a read-only service account, and break-glass write credentials lived somewhere that required a second person to sign off. That sounds heavy. It's about thirty lines of Terraform, and it removes the temptation entirely instead of hoping everyone resists it.

Second, they put the emergency path through Git too, and made it fast. The objection is always the same: "when prod is down I can't wait for CI." Fair. But a hotfix PR doesn't need your full pipeline. Here's the shape I've seen work, a stripped-down workflow that only runs on branches prefixed hotfix/:

# .github/workflows/hotfix.yaml
on:
  push:
    branches: ["hotfix/**"]

jobs:
  sync:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Validate manifests
        run: kubectl apply --dry-run=server --server-side -f manifests/prod/
      - name: Merge immediately
        run: gh pr create --fill --base main && gh pr merge --auto --squash

Thirty seconds from push to merged. ArgoCD picks it up on the next poll, or you click refresh because you're a human under stress and that's fine. The point is the paper trail exists. When the dust settles you can see exactly what changed, who approved it, and roll it back with a revert instead of archaeology.

Third, they treated drift as a bug, not a warning. ArgoCD will happily show you an app as OutOfSync forever if you let it. Teams that took GitOps seriously paged on drift that lasted more than a few minutes, because drift means someone bypassed the process, and the process is the whole thing. This is also where good visibility tooling earns its keep: drift is easy to miss in a wall of YAML diffs and much harder to miss when you can see the live object next to the declared one. That gap, between "what Git says" and "what's actually running," is exactly the kind of drift we built conndeck to put in front of your face.

Accepting the slower path is the price of admission

I want to be honest about the cost, because people gloss over it. GitOps done properly is slower at the moment you most want speed. The five-second kubectl scale becomes a PR, a review, a merge, a sync. Even with a hotfix path that's two minutes instead of five seconds.

Those two minutes buy you something concrete, though: a history. Every change to production has an author, a timestamp, a diff, and a revert button. Six months in, when you're staring at a latency regression and someone asks "what changed in March?", the answer is git log instead of a Slack search and three people trying to remember.

You also get onboarding that's almost unfair. New engineer joins, reads the repo, and knows the entire state of the platform. Not the documented state from a wiki page last updated in 2024. The actual state, because there's no other state.

Frequently asked questions

What is the difference between GitOps and just installing ArgoCD?

Installing ArgoCD gives you a sync controller and a dashboard; GitOps is the team agreement that Git is the only way changes reach the cluster. If engineers still run kubectl apply or kubectl edit from their laptops, you have an expensive dashboard, not GitOps. The tool reconciles state, but only a written, agreed rule that the repo is the single source of truth makes it real.

How do I stop engineers from making manual changes to production?

Remove write access instead of relying on willpower, because willpower loses every time there's a paging alert. Point production kubeconfig contexts at a read-only service account, and keep break-glass write credentials somewhere that requires a second person to sign off. It's roughly thirty lines of Terraform and it removes the temptation entirely instead of hoping everyone resists it.

What causes configuration drift in a Kubernetes cluster?

Drift comes from any change that doesn't go through Git: a hand-applied kubectl patch, a field someone excluded with IgnoreDifferences, or an app left on manual sync. Once drift exists, the cluster and the repo become two sources of truth and nobody actually knows what's running. Treat drift that lasts more than a few minutes as a bug worth paging on, because it means someone bypassed the process.

How do you handle hotfixes in GitOps without slowing down incident response?

Route the emergency path through Git too, but make it fast. A stripped-down CI workflow that only runs on branches prefixed hotfix/ can validate manifests with a server-side dry run and auto-merge a PR in about thirty seconds. The sync controller picks it up on the next poll, and you get a full paper trail you can roll back with a revert instead of archaeology.

Is GitOps slower than deploying directly with kubectl?

Yes, at the exact moment you most want speed: a five-second kubectl scale becomes a PR, a review, a merge, and a sync, roughly two minutes even with a hotfix path. What those two minutes buy is a history, so every change to production has an author, a timestamp, a diff, and a revert button. When someone asks what changed in March, the answer is git log instead of a Slack search.

The moment it clicks

There's a specific moment where you know a team has actually crossed over, and it's never the day ArgoCD goes in.

For that team of seven, it came during a Friday deploy about four months after the read-only rule. Someone merged a change with a typo in an image tag, api:v1.14.2-hotxif instead of hotfix. ArgoCD flipped to Degraded, the rollback was a one-click revert of the merge commit, and the whole thing took ninety seconds. Nobody opened a terminal. Nobody touched the cluster. Afterwards one of the engineers said, almost surprised, "that was easier than the old way."

That's the moment. Not the dashboard, not the green hearts. The moment the team realizes the discipline isn't a tax on their speed but the thing that makes speed safe. When your incident response is "revert the commit" and your audit trail is git log, the manual path doesn't feel like a shortcut anymore. It feels like what it is: punching a hole in the one thing keeping the system coherent.

Install the tool, sure. It's a good tool. But the tool is maybe ten percent of it. The other ninety is a team agreeing, out loud and in writing, that the repo is the truth and their keyboards aren't. Everything else is decoration.