Building Images Inside Kubernetes: Kaniko vs BuildKit

The security ticket was polite, which made it worse. "Please explain why 40 CI pods per day run privileged on production-adjacent nodes." It was May, our builds ran on docker:dind sidecars, and the real answer was "because that's how the blog post said to do it in 2021." Mounting the Docker socket would have been even worse. Either pattern means every pull request can execute as root on the node, and our compliance person had done the math.

So I spent two weeks replacing it, tested both serious options on real workloads, and kept notes. Kaniko and BuildKit are the two that matter for in-cluster builds, they make opposite trade-offs, and the right answer depends on how much CI throughput you actually need. Here's what the benchmarks and the battle scars say.

Quick answer: Both build OCI images without a Docker daemon and without privileged pods. Kaniko is a single-shot executor: you run it as a pod, it builds, it pushes, it exits -- operationally trivial, slower builds, caching via a registry repo only. BuildKit is a long-lived daemon you deploy once and drive with buildctl: rootless-capable, much faster, with a real local cache plus registry cache export/import. Small teams with modest build volume: Kaniko. Heavy CI where build time is a bottleneck: BuildKit.

Why docker:dind is a dead end

Worth being blunt, because I still see it in the wild. docker:dind needs securityContext.privileged: true, and privileged means the container can escape to the node with a syscall or two. Mounting /var/run/docker.sock skips the sidecar but hands builds the ability to run *any* container on the host Docker daemon -- including one that bind-mounts /. In both cases, "untrusted PR code runs in CI" becomes "untrusted PR code owns your node." On a cluster that also runs real workloads, that's disqualifying. Rootless in-cluster builders exist precisely so you don't have to choose between CI speed and node security.

Kaniko: the trusty mule

Kaniko's model is beautifully boring. The executor image contains no Docker daemon at all. It unpacks the base image into its own filesystem, executes each Dockerfile instruction in userspace, snapshots the result layer by layer, and pushes the final image straight to your registry. A build is a pod:

apiVersion: v1
kind: Pod
metadata:
  name: kaniko-ledger-api
  namespace: ci
spec:
  restartPolicy: Never
  containers:
    - name: kaniko
      image: gcr.io/kaniko-project/executor:v1.23.2-debug
      args:
        - --dockerfile=Dockerfile
        - --context=git://github.com/acme/ledger-api.git#refs/heads/main
        - --destination=registry.acme.com/ledger-api:1.20.0
        - --cache=true
        - --cache-repo=registry.acme.com/cache/ledger-api
        - --cache-ttl=72h
      volumeMounts:
        - name: docker-config
          mountPath: /kaniko/.docker
  volumes:
    - name: docker-config
      secret:
        secretName: registry-credentials
        items:
          - key: .dockerconfigjson
            path: config.json

Use the -debug or -debug-slim image (it has a shell, which matters when a build fails and you need to poke around). The push credential is a mounted docker config.json, so registry auth is just a Secret. No daemon, no privileged flag, no socket. The operational story is a single container that exits -- which is why Kaniko remains the fastest path from "we need in-cluster builds" to done.

The cost is speed. Kaniko snapshots the filesystem after every instruction, and snapshotting is I/O-bound grunt work. Registry caching (--cache=true) helps mostly with RUN layers; the big COPY . /app in most Dockerfiles re-uploads and re-executes constantly. It also can't parallelize build stages the way BuildKit does.

BuildKit: the fast one, with a daemon

BuildKit is the engine modern docker build uses, run standalone. You deploy buildkitd once -- the moby/buildkit:rootless image runs it without privileges -- and CI jobs connect with buildctl over TCP:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: buildkitd
  namespace: ci
spec:
  replicas: 1
  selector:
    matchLabels:
      app: buildkitd
  template:
    metadata:
      labels:
        app: buildkitd
    spec:
      containers:
        - name: buildkitd
          image: moby/buildkit:v0.13.2-rootless
          args:
            - --addr
            - tcp://0.0.0.0:1234
          securityContext:
            runAsUser: 1000
            runAsGroup: 1000
            seccompProfile:
              type: Unconfined
          ports:
            - containerPort: 1234
          volumeMounts:
            - name: cache
              mountPath: /home/user/.local/share/buildkit
      volumes:
        - name: cache
          persistentVolumeClaim:
            claimName: buildkitd-cache

Then builds are one client call away:

$ buildctl --addr tcp://buildkitd.ci.svc:1234 build \
  --frontend dockerfile.v0 \
  --opt context=https://github.com/acme/ledger-api.git \
  --opt filename=Dockerfile \
  --output type=image,name=registry.acme.com/ledger-api:1.20.0,push=true \
  --export-cache type=registry,ref=registry.acme.com/cache/ledger-api,mode=max \
  --import-cache type=registry,ref=registry.acme.com/cache/ledger-api
#2 [internal] load git source
#3 [1/6] FROM registry.acme.com/base/golang:1.22
#4 [2/6] COPY go.mod go.sum .   CACHED
#5 [3/6] RUN go mod download    CACHED
#6 [4/6] COPY . .
#7 [5/6] RUN go build -o /ledger-api ./cmd
#8 exporting to image
DONE  38.2s

Two things to note. The rootless image wants seccompProfile: Unconfined in stock Kubernetes, which is a real (if much smaller) relaxation -- sandbox it properly if you build untrusted code, or look at gVisor. And the PVC is what makes the local cache survive pod restarts; without it, every reschedule is a cold build.

Caching is where the speed actually lives

Raw benchmark numbers from my own cluster, same 6-stage Go Dockerfile, don't-quote-me-but: Kaniko cold was about 9m40s, warm with registry cache about 2m10s. BuildKit cold about 4m15s, warm off the local cache 40s. The gap is the cache architecture, full stop.

Kaniko's cache is a registry repo of intermediate layers: portable across runners, but only RUN instructions cache well, and every build pays push/pull overhead for cache artifacts. BuildKit's local cache is content-addressable on disk, near-instant on hits, and parallelizes independent stages. --export-cache ... mode=max gives you registry-side cache too, which is how you keep warm builds when the daemon pod reschedules to another node. In practice I run both: local cache for the common case, registry cache as the parachute.

If you take one thing from this section: whatever builder you pick, a CI setup without a cache strategy is a decision to pay full build time on every commit forever.

Which one I'd pick

Kaniko when build volume is modest, you want zero daemon babysitting, and builds are one-off pods spawned by your CI runner of choice. It also composes well with Tekton and GitLab runners that expect a container-per-step model.

BuildKit when build time is a tax your developers feel daily. The daemon is one more thing to run, but it's a quiet one, and the cache pays for it within a week on any repo with real churn.

Whichever you land on, run builds on a dedicated node pool with a taint, keep the push credential scoped to exactly the repositories CI owns, and when a build pod misbehaves you'll be staring at it in the cluster anyway -- I keep the ci namespace visible in conndeck because "why is today's build stuck" is answered faster by looking at the pod than by reading the runner logs. And retire the dind sidecars. The security ticket writes itself otherwise.

Frequently asked questions

Is it safe to run docker:dind or mount the Docker socket in Kubernetes?

No, not for CI on a shared cluster. Docker-in-Docker requires a privileged container, and privileged is effectively root on the node. Mounting /var/run/docker.sock is worse: any build that can talk to the socket can launch a container that mounts the host filesystem. Both patterns turn every pull request into node-level code execution, which is exactly why rootless builders like Kaniko and BuildKit exist.

Which is faster, Kaniko or BuildKit?

BuildKit, and usually not close. Kaniko snapshots the filesystem after every Dockerfile instruction, which is slow on cold builds and only partially saved by registry caching. BuildKit has a proper content-addressable local cache, parallel stage execution, and smarter layer reuse. In my tests on the same node, a warm BuildKit build ran several times faster than a warm Kaniko build on a medium Go service.

Does Kaniko need root or a privileged pod?

The Kaniko executor runs as root inside its container but does not require a privileged pod, because it never talks to a Docker daemon -- it unpacks base images and executes Dockerfile instructions in userspace with its own snapshotter. That said, running as root with broad filesystem access is not free of risk, so run builds on dedicated nodes, with gVisor or similar sandboxing, if you build untrusted code.

How do I enable caching with Kaniko?

Pass --cache=true and --cache-repo pointing at a registry repository, and Kaniko will store cache layers there and reuse them across builds. It works, but only for RUN instructions in a limited way, and it does nothing for COPY-heavy Dockerfiles where most of your time goes. Set --cache-ttl to bound how long stale layers live, and expect warm-build wins that are meaningful but nowhere near BuildKit's local cache.

Can I use Buildah instead of Kaniko or BuildKit?

Yes, and it's a legitimate third option. buildah bud runs daemonless and can run rootless with the right securityContext, and it handles OCI builds well. The trade-off is the ecosystem: Kaniko is simpler to drop into a pod spec, BuildKit has the best caching and Dockerfile compatibility, while Buildah sits between them with fewer turnkey CI examples. All three beat docker:dind.

What I do differently now

Stop running docker:dind on shared clusters -- that part isn't a judgment call. Between the replacements, Kaniko trades build speed for operational simplicity, and BuildKit trades one quiet daemon for builds that are several times faster with a real cache.

Start with Kaniko if you're unsure; it's a pod spec. When build time starts showing up in developer complaints, graduate to BuildKit and a proper cache. Your nodes, and your security team, will thank you either way.