kubectl debug: Ephemeral Containers Changed How I Troubleshoot

Last year we finally finished moving our services to distroless images. The security team was thrilled. The vulnerability scanner went quiet. Image sizes dropped by 90%.

Then, on a Wednesday afternoon, a payments pod started intermittently failing readiness checks, and I typed the command I'd typed ten thousand times before:

$ kubectl exec -it payments-api-6f8b9c5d7-qx2mt -n payments -- /bin/sh
error: Internal error occurred: error executing command in container:
failed to exec in container: failed to start exec "a1b2c3...":
OCI runtime exec failed: exec failed: unable to start container process:
exec: "/bin/sh": stat /bin/sh: no such file or directory: unknown

Right. Distroless. No shell, no coreutils, no ls, nothing. The image contained exactly one binary and the CA certificates, which is the entire point. I stood there mentally SSHing into the node like it was 2019 before remembering we didn't have node SSH access anymore either. Then a teammate leaned over and said the four words that changed how I troubleshoot: "Just use kubectl debug."

Quick answer: kubectl debug attaches an ephemeral container -- a temporary, tooling-filled container -- to a running pod, so you can troubleshoot images that have no shell, including distroless and scratch. Use kubectl debug -it <pod> --image=<toolkit> --target=<container> for live pods, --copy-to to experiment on a copy of the pod instead of the real one, and kubectl debug node/<node> to get root-level access to a node itself. It's been GA since Kubernetes 1.25 and has made exec-into-a-shell the plan B it always should have been.

Why the old way died

kubectl exec was always a deal with the devil: to make troubleshooting possible, every production image carried a shell, package manager, and a pile of utilities -- which also meant every compromised container carried a shell, package manager, and a pile of utilities. Distroless and scratch-based images broke that deal on purpose. Great for security posture. Terrible at 3 PM when a pod is misbehaving and your only window into it is its stdout.

Ephemeral containers split the difference: the production image stays minimal, and when you need to look inside, you attach a second container carrying your tools. It joins the pod and shares its world -- network namespace, and optionally process namespace -- then vanishes when you're done. No restart, no new pod, no SSH.

Debugging a live distroless pod

The bread-and-butter invocation:

$ kubectl debug -it payments-api-6f8b9c5d7-qx2mt -n payments \
    --image=busybox:1.36 --target=payments-api
Targeting container "payments-api". If you don't see processes from this container, retry with kubectl debug --share-processes.
If you don't see a command prompt, try pressing enter.
/ #

That shell is running inside the pod. wget the failing readiness endpoint from localhost, poke the DNS, check the listener -- everything works, because you're on the pod's network. / # wget -qO- localhost:8080/healthz answered my original question in about ten seconds (the endpoint was fine; the failure was DNS, and it was in the *other* direction).

Busybox is fine for network checks, but my default is a fat toolkit image -- nicolaka/netshoot is the community favorite, with tcpdump, curl, dig, ss, strace, and friends preinstalled:

$ kubectl debug -it payments-api-6f8b9c5d7-qx2mt -n payments \
    --image=nicolaka/netshoot:v0.13 --target=payments-api --share-processes

Two flags there matter. --target picks which container's filesystem and process view you share (skip it on single-container pods, but be explicit on multi-container ones -- the default picks the first, which is the istio-proxy half the time, ask me how I know). --share-processes turns on process namespace sharing so you can see and signal the app process itself: ps aux shows your actual app, and you can strace its syscalls or grab a thread dump with kill -QUIT.

One thing ephemeral containers *can't* do is mount the target container's image filesystem directly -- you see the processes and the network, but not its files, unless the pod shares a volume. For file-level work, reach for the copy trick below.

Copy-based debugging: change the pod without changing the pod

Sometimes you don't want to observe the process; you want to change how it starts. Add a debug flag, run it under a wrapper, replace the crashing binary's image with a debug build. Doing that to the live pod means a restart and a hope. Instead, make a copy:

$ kubectl debug payments-api-6f8b9c5d7-qx2mt -n payments -it \
    --copy-to=payments-api-debug \
    --container=payments-api \
    --image=registry.example.com/payments-api:v2.14.3-debug \
    -- sh

This creates a new pod, payments-api-debug, cloned from the original spec but with the container image swapped -- and with no labels, so no Service routes traffic to it. It's your own private replica to break. Variations I use constantly: --set-image=payments-api=...:debug (same thing, different flag), or keeping the image but overriding the command to sleep infinity so I can exec in and run the app by hand with extra logging, watching it fail in slow motion. Delete the copy when done; it won't clean up after itself.

Node debugging: root without SSH

When the pod is fine and the *node* is the suspect -- disk pressure, mysteriously stuck pods, CNI weirdness, a kubelet that's lying to you -- kubectl debug goes one level deeper:

$ kubectl debug node/ip-10-0-3-87.ec2.internal -it \
    --image=ubuntu:24.04 --profile=sysadmin
Creating debugging pod node-debugger-ip-10-0-3-87.ec2.internal-4f2k9 ...
root@ip-10-0-3-87:/# chroot /host
root@ip-10-0-3-87:/# journalctl -u kubelet --since "1 hour ago" | tail -20

The debugger pod runs on the node with the host's root filesystem mounted at /host, and chroot /host makes it effectively a root shell on the machine. crictl ps to see what containerd thinks is running, check dmesg for OOM kills the API never surfaced, inspect the CNI config in /host/etc/cni/net.d/. The --profile=sysadmin flag (Kubernetes 1.27+) is the modern way to get the privileges this needs; older clusters want --image plus a pod-security-compatible namespace.

This is root on a production node, delivered via the Kubernetes API. Treat it accordingly -- which brings us to access control.

Keeping it from becoming a backdoor

Ephemeral containers are governed by RBAC on the pods/ephemeralcontainers subresource, separately from normal pod access. That's the feature: you can grant the on-call group debug rights on production pods without granting them create pods or exec, and every attachment lands in the API audit log. A minimal role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-debugger
  namespace: payments
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["pods/ephemeralcontainers"]
    verbs: ["update", "patch"]

The debugging session itself -- the ephemeral container spec, the copy pod -- is visible in kubectl describe pod, so nothing happens in the dark. Which, is the part I like most: the tooling is finally good enough that "minimal production images" and "debuggable production" stopped being a trade-off. When I'm triaging, I still start in conndeck to spot which pod is sick, but the moment I need hands inside the pod, kubectl debug is the move.

Frequently asked questions

What is an ephemeral container in Kubernetes?

It's a container you attach to an already-running pod using kubectl debug, meant purely for troubleshooting. Unlike regular containers it can't have ports, probes, or resource guarantees, and it disappears when it exits or the pod goes away. The feature went GA in Kubernetes 1.25, and it's the supported answer to the question of how to get a shell into a pod whose image doesn't have one.

How do I debug a distroless container with no shell?

Run kubectl debug -it on the pod with a debug image like busybox or a full toolkit image, and point --target at the container you care about. The ephemeral container joins the pod's namespaces, so you get a shell that can see the target's processes, filesystem, and network even though the target image itself has no shell at all. Add --share-processes or use the process-sharing flag if you need to see the other containers' processes.

How do I debug a Kubernetes node directly?

kubectl debug node slash node-name spins up a privileged pod on that node with the host's root filesystem mounted at slash host. From there you can chroot into slash host and you effectively have root on the machine, which is how you inspect kubelet logs, disk pressure, iptables rules, and CNI state without SSH. Treat it like root, because it is: it bypasses most of the guardrails you'd normally have.

What's the difference between kubectl debug --copy-to and a normal debug session?

A plain ephemeral container debugs the live pod in place, which is what you want for inspecting a real failure. --copy-to instead creates a copy of the pod with your changes applied -- a different image, an added shell container, a modified command -- so you can experiment without touching the running workload. The copy gets a new name and won't receive Service traffic, which makes it the safe choice when you need to restart the process under a debugger or change its entrypoint.

Are ephemeral containers safe to use in production?

They're far safer than the alternatives they replace, like SSHing to nodes or baking shells into production images. Access is governed by RBAC through the pods slash ephemeralcontainers subresource, so you can grant debug rights to on-call engineers without granting them deploy rights, and every attachment shows up in the audit log. The main real risks are performance ones: don't attach a profiler that eats CPU to a pod that's already at its limit during peak traffic.

Where this leaves you

Distroless won, and kubectl exec -- /bin/sh lost. kubectl debug is the replacement: ephemeral containers for live pods, --copy-to for safe experiments, node/ for the machines underneath. Learn the three incantations and no image is too minimal to troubleshoot.

Just remember to grant pods/ephemeralcontainers to your on-call before the incident, not during it. Discovering RBAC gaps mid-page is a genre of fun I no longer recommend.