Kubernetes Audit Logs: Finding the Needle Before the Incident

At 6:40 AM on a Tuesday, a teammate asked the question you never want to hear on no coffee: "Who deleted the cert-manager ClusterIssuer last night?" The issuer was back by 7:15 -- someone had reapplied it from git -- but the *who* mattered, because cert-manager had been happily renewing certs off it and a day-long silence would have been a sev.

We found the answer in the audit log in about four minutes. It was a GitOps service account, applying a stale branch that predated the issuer's rename. Not malicious, just confusing, and the audit log turned a multi-hour archaeology dig into a four-minute query. That's the entire case for audit logging: not compliance, not paranoia -- just being able to answer "who did what, when" while it still matters. The catch is that a default audit configuration on a busy cluster is a firehose that costs real money and answers nothing. The policy is where you fix that.

Quick answer: Kubernetes audit logging is an API server feature: you write an audit policy file that maps event categories to levels (None, Metadata, Request, RequestResponse) and ship the output to a file or webhook sink. Tune the policy to drop noise at the source -- kubelet status updates, lease churn, system-identity reads -- because system components generate most of the volume. Alert narrowly: exec sessions, secret reads by unusual identities, RBAC changes, token creation, and the audit stream itself going silent. Keep Metadata level for most events and RequestResponse only for secrets and RBAC. Managed clusters expose the same stream through EKS/GKE/AKS control-plane logging.

The policy file is the product

The API server evaluates rules top to bottom, first match wins, so the order is the design. Here's a policy I've run in production, annotated:

apiVersion: audit.k8s.io/v1
kind: Policy
# Don't log these at all: pure noise, enormous volume
rules:
  # kubelet and node heartbeats
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
      - group: ""
        resources: ["endpoints", "services"]
  - level: None
    userGroups: ["system:nodes"]
    verbs: ["get", "list", "watch", "update", "patch"]
    resources:
      - group: ""
        resources: ["nodes", "pods"]
  # leader election and endpointslice churn
  - level: None
    resources:
      - group: "coordination.k8s.io"
        resources: ["leases"]
      - group: "discovery.k8s.io"
        resources: ["endpointslices"]
    verbs: ["get", "list", "watch", "update"]
  # system components reading things constantly
  - level: None
    users: ["system:kube-scheduler", "system:kube-controller-manager"]
    verbs: ["get", "list", "watch"]
  # secrets and RBAC: full bodies, this is what investigators read
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets"]
      - group: "rbac.authorization.k8s.io"
  # exec and friends: request body is enough
  - level: Request
    verbs: ["create"]
    resources:
      - group: ""
        resources: ["pods/exec", "pods/portforward", "pods/attach"]
  # everything else: who, what, when, no bodies
  - level: Metadata
    omitStages: ["RequestReceived", "ResponseStarted"]

On a self-managed cluster, you enable it with API server flags:

--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit.log
--audit-log-maxage=7
--audit-log-maxbackup=10
--audit-log-maxsize=200

On managed clusters the control plane isn't yours to flag: EKS streams audit logs to CloudWatch when you enable the audit control-plane log type, GKE writes them to Cloud Logging (and lets you shave volume by disabling redundant data-access logs), AKS sends them via diagnostic settings. You get less policy control -- GKE and AKS let you tune somewhat, EKS mostly gives you the default policy -- which is worth knowing before you promise an auditor something specific.

Metadata vs RequestResponse is a cost decision

Every event gets a level, and the level is the bill. Metadata records the actor, verb, object, namespace, timestamp, and response code -- a few hundred bytes. RequestResponse adds the full request and response bodies, and a single kubectl get secrets -o yaml in a big namespace can produce a megabyte. Multiply by a fleet of controllers listing things all day and your log platform invoice becomes a topic of executive interest.

My rule: Metadata is the default for everything, RequestResponse only for the resources where the body *is* the evidence -- secrets access and RBAC changes. Those two categories are low-volume and high-value, which is exactly the profile that justifies the expensive level. One more subtlety: omitStages for RequestReceived (and ResponseStarted for long-running requests) drops duplicate events that add latency and volume without adding information.

And yes, RequestResponse on secrets means the audit log contains secret values. Restrict access to the log sink accordingly; the log backend is now itself a sensitive store.

What to actually alert on

Most audit events are boring. The ones that aren't fit a short list, and each one maps to a query on fields every event carries: user.username, verb, objectRef, sourceIPs, responseStatus.

1. Exec into anything. objectRef.subresource: exec. Every interactive shell in production is either a human debugging (fine, but know about it) or an attacker walking in (not fine).
2. Secret reads by non-system identities. A get or list on secrets where the username isn't a known controller or CI account is the single highest-signal rule I run.
3. RBAC mutations. Any create, update, patch, or delete on rbac.authorization.k8s.io resources, full stop. Especially ClusterRoleBindings referencing cluster-admin.
4. Token creation. serviceaccounts/token subresource creates -- someone minting bound tokens outside normal workload startup.
5. Deletes of namespaces and production workloads. The 6:40 AM question, answered by a filter instead of a panic.
6. The stream itself. An alert on audit volume dropping to zero. Anyone sophisticated enough to matter will try to blind you first; on managed clusters, also alert on someone *disabling* control-plane logging.

Everything else goes to dashboards. An alert that fires weekly and means nothing is a paging policy that gets ignored by month three.

Cost control at the source

The cheapest audit event is the one you never emit. Volume on a typical cluster is dominated by system identities -- kubelet heartbeats, controller-manager reads, endpoint and lease churn -- so dropping those categories with None rules removes the bulk of your bill while sacrificing nothing you'd ever investigate. When I first turned this up on a busy cluster, the None rules above cut volume by over 80% before we touched anything else.

Beyond the policy: prefer a webhook sink or a node agent shipping the file straight to cheap object storage, with a hot window in your log platform and the rest in cold storage with lifecycle rules. Retention is a separate decision from searchability -- you want 90 days queryable and a year-plus retrievable, and those should not cost the same. When I'm answering an ad-hoc "who touched this" question live, I usually check the object's events and owner references first -- conndeck surfaces those next to the resource -- and only drop into the audit archive when the question is about identity rather than state. Use the cheap tool first; keep the archive for when it counts.

Frequently asked questions

How do I enable audit logging in Kubernetes?

Point the API server at an audit policy file with --audit-policy-file and give it a sink, either a local file with --audit-log-path plus rotation flags, or a webhook backend with --audit-webhook-config-file that ships events to your logging stack. On managed Kubernetes it's a control-plane setting: EKS, GKE, and AKS all have a toggle to stream control-plane logs including audit to their cloud logging service. The policy file is the part you actually design; the sink is plumbing.

What are the Kubernetes audit log levels and which should I use?

None, Metadata, Request, and RequestResponse. Metadata logs who did what to which object and is cheap enough for most events. Request logs the request body too, and RequestResponse logs the body and the response, which is where secrets and full payloads land. The practical default is Metadata for almost everything, RequestResponse narrowly scoped to secrets access and RBAC changes, and None for high-volume noise like kubelet status updates.

What should I alert on in Kubernetes audit logs?

The short list: any exec or portforward into a pod, reads of secrets outside expected system identities, RBAC changes like ClusterRoleBindings to cluster-admin, service account token creation, and deletes of namespaces or production workloads. Also alert on audit log volume dropping to zero, because an attacker who can touch the control plane will try to blind you first. Everything else belongs in dashboards, not pagers.

How do I reduce the cost of Kubernetes audit logging?

Cut volume at the policy layer, not at the log sink, by dropping entire categories with None rules: kubelet and node status updates, endpoint and lease churn, and read-only traffic from system components like the scheduler and controller manager. On a busy cluster that alone removes the majority of events, since system identities generate most of the traffic. Then keep Metadata level for what's left and reserve the expensive body-logging levels for the few resources that justify them.

How long should I retain Kubernetes audit logs?

Long enough to answer questions about last quarter, which in practice means 90 days hot and a year or more in cheap cold storage, or whatever your compliance regime demands if it's longer. The incident you investigate in March often started in November, and a 7-day retention window means you get to say we don't know in your own postmortem. Cheap object storage with lifecycle rules makes long retention nearly free compared to your hot log platform.

The short version

Audit logging fails in two directions: off entirely, or on at full blast until the invoice arrives and someone turns it off. The policy file is how you land in the middle -- None for the noise, Metadata for the record, RequestResponse for the evidence.

Set it up on a quiet afternoon. The 6:40 AM version of you will be grateful.