conndeck Download

conndeck blog

External Secrets Operator: Stop Committing Secrets

#kubernetes#secrets#security#vault#gitops

In April I got pulled into a security review that started with one question from the auditor: "How many people can read your production database password?" I opened our config repo to show her the Secret manifest -- base64, obviously -- and watched her scroll past two hundred names on the repository's access list. "Two hundred," she said. "Plus everyone who's ever cloned it."

We weren't even sloppy. We had SOPS on the roadmap. But sitting in that room, I realized SOPS would have changed the answer from "two hundred" to "whoever holds the age key, plus every ciphertext in git history forever," and that's a smaller upgrade than it sounds. Six weeks later we were on External Secrets Operator, and the answer became "nobody with repo access can read it at all." This post is the setup I wish we'd started with.

Quick answer: External Secrets Operator (ESO) syncs secrets from an external provider -- Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault -- into real Kubernetes Secrets, so git holds references instead of values. You define a ClusterSecretStore describing how to authenticate to the provider, then one ExternalSecret per application secret describing which remote keys map to which local keys. ESO polls on a refreshInterval and updates the Secret when the remote value rotates. Encrypted-in-git schemes like SOPS and Sealed Secrets are better than plaintext, but they still leave ciphertext in history and put rotation in your commit log.

Why encrypted secrets in git still lose

Base64 is encoding, not encryption, and everyone knows it -- yet half the clusters I've audited still have "temporary" base64 Secrets from 2023. The standard fix is SOPS or Sealed Secrets, and they're genuinely better. But the failure modes just move:

  • Every value you've ever used stays in git history, encrypted or not. Rotate because of a leak and the old ciphertext is still there, one key compromise away from exposure.
  • Rotation is a manual, repo-by-repo, environment-by-environment re-encryption dance. People skip it. I've seen "rotated quarterly" secrets with 2022 timestamps.
  • The decryption key (age private key, KMS grant, sealed-secrets controller key) becomes the one secret that guards all the others, and now you have a meta-secret problem.

The philosophical issue is that git remains the source of truth for the secret's *value*. It shouldn't be. Git should record *which* secret an app needs; the secret manager should be the source of truth for what it *is*. That's the inversion ESO gives you.

How ESO actually works

Two CRDs do the work. A SecretStore (namespaced) or ClusterSecretStore (cluster-wide) tells the operator how to reach and authenticate to the provider. An ExternalSecret lives in the app's namespace and maps remote keys to a local Secret. The operator polls, and on change, writes the Kubernetes Secret itself.

For Vault with Kubernetes auth, the store looks like this:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault
spec:
  provider:
    vault:
      server: "https://vault.internal.acme.com"
      path: "kv"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "eso-reader"
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets
  conditions:
    - namespaces: ["payments", "staging-*"]

The conditions block matters more than people realize: without it, any namespace can reference the store. Scope it, and on the Vault side scope the eso-reader role to kv/data/payments/* style paths so one team's service account can't read another team's secrets.

Then the app-side object is almost boring, which is the point:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: ledger-api-db
  namespace: payments
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: vault
  target:
    name: ledger-api-db
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: kv/payments/ledger-api
        property: password
    - secretKey: username
      remoteRef:
        key: kv/payments/ledger-api
        property: username

Commit that to git happily. It's a pointer. With creationPolicy: Owner, deleting the ExternalSecret garbage-collects the Secret, which keeps namespaces clean when apps get retired.

The same pattern works against AWS Secrets Manager (provider: aws with service: SecretsManager), GCP, or Azure -- the ExternalSecret barely changes, which is a nice hedge against cloud migrations. For day-to-day "did the sync land" checks, the status tells you everything:

$ kubectl get externalsecret -n payments
NAME            STORE   REFRESHINTERVAL   STATUS         READY
ledger-api-db   vault   1h                SecretSynced   True

$ kubectl get secret ledger-api-db -n payments
NAME            TYPE     DATA   AGE
ledger-api-db   Opaque   2      12d

Rotation and the refresh loop

Rotation is the whole reason to do this, so understand the mechanics cold. ESO polls the provider every refreshInterval (I use 1h for databases, 15m for anything that rotates often). When the remote value changes, ESO updates the Secret object. What happens next depends on how your app consumes it:

Volume-mounted secrets get updated by the kubelet on its sync loop (with caching, so up to a couple of minutes). Apps that re-read the file pick up the new value with no restart. Most don't re-read.

Environment variables never change in a running container, period. If your app does envFrom: secretRef, a rotated secret does nothing until the pod restarts. The standard fix is stakater/Reloader watching the Secret, or wiring rotation to trigger a rollout. Decide this *before* you rotate production credentials for the first time, not during.

One operational note: don't set refreshInterval: 10s on three hundred ExternalSecrets pointing at AWS Secrets Manager. You'll discover their API rate limits and per-call pricing in the same afternoon. Ask me how I know.

Failure modes worth planning for

Provider outage. ESO keeps the last synced Secret in place, flips the ExternalSecret to SecretSyncedError, and backs off. Apps keep running on the old value. This is the correct failure mode, but alert on the status condition -- a quiet three-day sync failure means your rotation silently isn't happening.

Missing key or typo in remoteRef. The sync fails with the error on the object's status and events, but *no Secret gets created*, so pods referencing it sit in CreateContainerConfigError and the whole thing looks like an app bug until someone runs kubectl describe externalsecret. I've trained teams to check READY on the ExternalSecret before debugging the pod. Watching the Secret and its owning ExternalSecret side by side -- something conndeck happens to make easy since both live in the same namespace view -- turns that from a twenty-minute hunt into a glance.

Deleted upstream secret. Default behavior leaves the orphaned value in place rather than vanishing a Secret out from under running pods. That's what you want, but it means decommissioning needs an explicit cleanup pass, not just deleting from Vault.

Frequently asked questions

Is it safe to commit SOPS-encrypted or sealed secrets to git?

It's far better than plaintext, but ciphertext in git still rots. Every old value stays in history forever, rotation means re-encrypting and committing across every environment, and the decryption key becomes the single secret that protects all the others. The deeper problem is that your repo still dictates what the secret is, when really the secret store should. External Secrets Operator flips that relationship, which is why I prefer it.

How does External Secrets Operator rotate secrets?

ESO polls the external provider on each ExternalSecret's refreshInterval, and when the remote value changes, it updates the Kubernetes Secret it owns. What it does not do is restart your pods, and environment variables never update inside a running container regardless. Mount the Secret as a volume so the kubelet refreshes the file, or use something like stakater's Reloader to roll deployments when a Secret changes.

What happens to my apps if Vault or the secret manager goes down?

Nothing, initially. ESO keeps the last successfully synced Kubernetes Secret in place, marks the ExternalSecret with a SecretSyncedError status, and retries with backoff. Your app keeps running on the previous value. The risk is a pod restart coinciding with a long provider outage if the Secret was never synced in the first place, so alert on the sync status, not just on Vault itself.

Should I use External Secrets Operator or the Secrets Store CSI Driver?

The CSI driver mounts secrets as in-memory volumes directly from the provider, with no Kubernetes Secret object, and updates propagate to the mount without a pod restart. ESO creates real Secret objects, which work with envFrom, Helm charts, and every tool that expects a Secret to exist. I default to ESO for compatibility and reach for the CSI driver when a compliance requirement forbids secrets existing as Kubernetes Secrets at all.

Can I restrict which namespaces use a ClusterSecretStore?

Yes. A ClusterSecretStore is cluster-wide by default, but its spec.conditions field accepts a namespaces list that limits where it can be referenced from. Combine that with Kubernetes auth roles in Vault that only allow the ESO service account to read specific paths, and a team in namespace A can't pull secrets from namespace B's path even if they guess the key name.

The short version

Git is a great source of truth for manifests and a terrible one for secret values -- encrypted or not, history is forever and rotation becomes a commit-driven chore. External Secrets Operator inverts the relationship: git holds pointers, the provider holds values, and rotation becomes a poll instead of a project.

Set up the ClusterSecretStore once, scope it properly, decide your restart story for env-var consumers, and you can finally answer the auditor's question with a number you're proud of.