kubectl apply Is Lying to You (and What to Do About It)

A few years back I was on a team of five running a mid-sized Kubernetes 1.26 cluster, and one Thursday night our payments service started throwing OOMKilled pods at 2 AM. The fix was obvious: bump the memory limit. Someone applied the new manifest, kubectl cheerfully printed deployment.apps/payments configured, and we went back to bed.

The pods kept dying. Same limit, 256Mi, staring at us from kubectl get pod -o yaml. The change had "applied successfully" and silently done nothing.

That night taught me something I now consider one of the most important facts about Kubernetes: kubectl apply does not tell you the state of the cluster. It tells you the API server accepted a patch. Those are wildly different things, and the gap between them is where outages live.

Quick answer: kubectl apply printing configured only means the API server accepted your patch. It says nothing about whether your field actually won or the workload converged, because client-side apply is a three-way merge against the last-applied-configuration annotation, and fields written by webhooks, controllers, or other tools are invisible to it. Diff against the server before you apply, watch kubectl rollout status after, and put something (ArgoCD, or even a CI cron running kubectl diff) between your repo and the cluster that never stops comparing the two.

What apply actually does

Client-side apply is a three-way merge. kubectl compares three documents: the YAML you just gave it, the YAML you gave it last time (stored in the kubectl.kubernetes.io/last-applied-configuration annotation), and the live object in etcd. From those three it computes a patch.

This design has a famous failure mode. Fields that were never in your last-applied annotation are invisible to the merge. If something else wrote a field, an admission webhook, a controller, a human running kubectl edit, apply doesn't know it exists and won't touch it. Your manifest becomes a subset of reality, and the extra bits accumulate forever.

That's exactly what bit us. A mutating webhook our security team had added weeks earlier was injecting a resource patch into the payments deployment. Our new memory limit landed in the annotation, the webhook's value landed in the live spec, and since our annotation said "the limit was always 256Mi," the merge saw nothing to change. configured. Sure.

And it goes the other way too. Delete a field from your YAML and apply will remove it, but only if it was in last-applied. The classic trap is spec.replicas on a deployment managed by an HPA. You delete the line from your manifest because "the HPA owns it now," apply computes a deletion patch, the deployment controller resets replicas to its default of 1, and your HPA spends the next minute clawing its way back up while traffic falls over. I've watched this exact sequence take down a checkout flow during a sale. kubectl apply --prune exists, but it's label-scoped, easy to misuse, and honestly not something I'd run against production without a very good reason.

"Applied" is not "running"

Even when the patch is perfect, apply's exit code tells you nothing about convergence. A deployment whose image doesn't exist applies cleanly. A pod stuck in CrashLoopBackOff because of a missing ConfigMap applies cleanly. A StatefulSet that will never schedule because the PVC's storage class is gone applies cleanly.

The API server is an admission point, not a status report. The controllers that turn your spec into running containers work asynchronously, and they fail in ways that only surface in object status, events, and logs, none of which apply reads.

So the bare minimum I now insist on after any apply that matters:

kubectl apply -f deploy.yaml
kubectl rollout status deploy/payments --timeout=90s
kubectl get events --sort-by=.lastTimestamp | tail -20

And before the apply, a diff against the server, not against my last-applied fiction:

kubectl diff -f deploy.yaml

kubectl diff does a server-side dry run, so it shows what would actually change, including fields other actors own. It's the closest thing apply has to honesty.

Server-side apply: better, and then the managers fight

Server-side apply (kubectl apply --server-side, GA since 1.22) moves the merge to the API server and tracks ownership per-field in metadata.managedFields. Each actor that writes a field becomes a field manager. This fixes real problems: no more giant last-applied annotation hitting the 256KB annotation limit, and explicit ownership means drift from other writers is visible instead of invisible.

The trade is that ownership is enforced. When two managers claim the same field, you get errors like:

Apply failed with 1 conflict: conflicts with "kubectl-client-side-apply"
using apps/v1: .spec.template.spec.containers[name="app"].resources.limits.memory

This is where teams lose an afternoon. The typical story: CI applies with --server-side, but some older pipeline or a Helm release still uses client-side apply, and the two managers fight over resources, replicas, or labels. You can stomp the conflict with --force-conflicts, which transfers ownership to you, and sometimes that's the right call during a migration. But as a permanent flag it's a code smell. You're telling the API server "I don't care who else thinks they own this," which is precisely the blindness that got us into trouble in the first place.

My rule: pick one apply strategy per resource, across every tool that touches it, and stick to it. Mixed managers are worse than either option alone. If you want to see who's fighting over a field, the answer is right there:

kubectl get deploy payments -o yaml --show-managed-fields

It's verbose. Pipe it through yq '.metadata.managedFields' and you'll see exactly which manager owns what, and who's been writing behind your back.

Closing the loop with drift detection

All of this points at the real fix: stop treating apply as a one-way command and start comparing desired state to live state continuously.

If you're ready for GitOps, ArgoCD is the obvious answer and for once the hype is deserved. An Application points at a Git repo, Argo diffs the rendered manifests against live objects every few minutes, and anything that diverges shows up as OutOfSync. With selfHeal: true in the sync policy it will even revert manual kubectl edit changes automatically. The argocd.argoproj.io/sync-wave annotations and ApplicationSet generators handle ordering and multi-cluster fan-out, so the model scales past the toy-demo stage. The important part isn't the tooling, it's that drift becomes a first-class, visible state instead of a rumor.

If you're not ready for Argo, a poor man's version still beats nothing: a cron job in CI that runs kubectl diff -f ./manifests/ against the cluster and pages someone on a non-zero exit. Ugly, effective, and it would have caught our 256Mi limit in minutes instead of at 2 AM.

This whole class of problem, the distance between what you declared and what's actually running, is the reason we built conndeck the way we did: live state front and center, not something you have to remember to query. But whatever you use, the principle holds.

Frequently asked questions

Why does kubectl apply say "configured" when nothing changed?

Because client-side apply is a three-way merge between your YAML, the last-applied-configuration annotation, and the live object, and fields that were never in that annotation are invisible to the merge. If a mutating webhook, controller, or someone running kubectl edit wrote the field, apply doesn't know it exists and reports success while leaving the other writer's value in place. Run kubectl diff before applying to see what would actually change.

What's the difference between client-side and server-side apply?

Client-side apply computes the patch locally using the last-applied-configuration annotation, while server-side apply (kubectl apply --server-side, GA since Kubernetes 1.22) moves the merge to the API server and tracks per-field ownership in metadata.managedFields. Server-side apply kills the 256KB annotation limit problem and makes drift from other writers visible, but it enforces ownership, so two tools fighting over the same field produce conflict errors.

How do I fix "Apply failed with conflict" errors in Kubernetes?

The error means another field manager owns the field you're trying to write, usually because CI uses server-side apply while an older pipeline or Helm release still uses client-side apply. The real fix is to pick one apply strategy per resource across every tool that touches it. You can pass --force-conflicts to take ownership during a migration, but as a permanent flag it just recreates the blindness that caused the problem.

How do I detect configuration drift in Kubernetes?

The reliable way is to continuously diff your declared manifests against live cluster state instead of trusting apply's success message. ArgoCD does this natively: it compares Git to live objects every few minutes and flags divergence as OutOfSync, with optional self-healing. A cheap alternative is a CI cron job running kubectl diff -f ./manifests/ that pages someone on a non-zero exit.

Does kubectl apply wait for the rollout to finish?

No. kubectl apply returns as soon as the API server accepts the patch, and controllers converge asynchronously, so a deployment with a missing image or a crashing pod still applies cleanly. Follow any apply that matters with kubectl rollout status and a check of recent events to confirm the change actually landed.

Lessons for your cluster

kubectl apply is a patch submission tool wearing a truth-teller's clothes. "Configured" means a merge succeeded. It doesn't mean your field won, it doesn't mean the controllers converged, and it definitely doesn't mean the cluster looks like your YAML.

So stop trusting the success message. Diff before you apply, watch the rollout after, keep one field manager per resource, and put something, ArgoCD or a cron job, between your Git repo and reality that never stops comparing the two. The cluster is the source of truth only if you're actually looking at it.