ImagePullBackOff: Every Cause I've Actually Hit
It was 4:45 on a Friday afternoon when the last deploy of the week hung.
The pipeline had passed, the rollout started, and two of the new pods came up clean. The third sat at ErrImagePull, then flipped to ImagePullBackOff. Same image, same manifest, same cluster. Two nodes could pull it and one couldn't, which is exactly the kind of detail that tells you this isn't a typo. It was, in the end, a rate limit, and I'll get to why only one node tripped it.
ImagePullBackOff is one of those statuses that feels like one problem and is actually four. Here's every cause I've hit in production and how to tell them apart in under two minutes.
Quick answer: ImagePullBackOff means the kubelet can't pull the container image and is retrying with a growing backoff. The cause is written in the pod's events: run kubectl describe pod <name> and read the Failed to pull image message. not found means the tag or repo is wrong, unauthorized means registry auth, toomanyrequests means a rate limit, and x509 or timeout errors mean network or certificate problems. Fix whichever one the event names -- don't guess.
ErrImagePull vs ImagePullBackOff
These are the same failure at different ages. ErrImagePull is the state immediately after a pull attempt fails. Once it has failed a few times, the kubelet starts backing off between retries -- ten seconds, then twenty, then forty, up to five minutes -- and the status flips to ImagePullBackOff. The BackOff variant doesn't mean anything new happened. It means the original error kept happening.
One wrinkle worth knowing: pulls happen on the node, not in the control plane. A pod scheduled to a healthy node can pull an image that fails everywhere else. That's why my Friday deploy had two running pods and one stuck one. Whenever pull behavior differs between replicas, think per-node: node identity, node network, node-level rate limit budget.
The error message is the diagnosis
Don't stop at the status column. The actual error from the container runtime is in the events:
$ kubectl describe pod api-7d9f4b8c6-x2k4p -n payments
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Failed 38s (x3 over 75s) kubelet Failed to pull image "registry.acme.io/payments/api:v2.31.0": rpc error: code = NotFound desc = failed to pull and unpack image "registry.acme.io/payments/api:v2.31.0": failed to resolve reference "registry.acme.io/payments/api:v2.31.0": registry.acme.io/payments/api:v2.31.0: not foundEverything after Failed to pull image is the registry talking to you, and it falls into four buckets:
not found/manifest unknown-- the registry answered, and it doesn't have that tag or repo.unauthorized/401/no basic auth credentials-- authentication problem.toomanyrequests-- rate limit.x509: certificate signed by unknown authority,dial tcp ... i/o timeout,no such host-- network, DNS, or PKI between the node and the registry.
Ninety percent of the job is reading that line carefully instead of skimming it. not found and unauthorized look identical in the pod list and have completely different fixes.
Tags, typos, and the "it exists" argument
For not found errors, work this order. First, verify the image yourself, from your laptop, with the exact full reference from the pod spec:
$ docker manifest inspect registry.acme.io/payments/api:v2.31.0
no such manifest: registry.acme.io/payments/api:v2.31.0If that fails, the tag genuinely isn't there. Now find out what CI actually pushed. The classic cases: the pipeline pushed v2.31 and the manifest says v2.31.0. The image was pushed to the dev registry but never promoted to the prod one the cluster pulls from (that was a Tuesday I'll remember). Someone renamed the repository and half the manifests still point at the old path. Or the CI job quietly failed at the push step after the tests passed, so everything looked green.
Mutable tags deserve their own sentence. If you're deploying :latest or :main, "the tag exists" tells you nothing about which image you'll get, and with imagePullPolicy: IfNotPresent a node that already has a stale :latest will happily run it forever. Pin to immutable tags or digests (image: app@sha256:...) and this whole category of confusion goes away.
Registry auth: secrets, namespaces, and node identity
For unauthorized, the question is who was supposed to authenticate. For classic private registries, that's an imagePullSecret:
$ kubectl create secret docker-registry regcred \
--docker-server=registry.acme.io \
--docker-username=robot-deploy \
--docker-password=$TOKEN -n paymentsThen reference it in the pod spec under imagePullSecrets, or -- better -- attach it once to the namespace's ServiceAccount (kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "regcred"}]}' -n payments) so every pod gets it without per-deployment edits.
The failure modes, in order of how often I've seen them: the secret exists in a different namespace (secrets don't cross namespaces, and nobody gets an error, just 401s); the credentials expired or the robot account was rotated and the secret wasn't; the registry hostname in the secret doesn't exactly match the one in the image reference, so the kubelet never tries those credentials.
On managed clusters there's a second auth path people forget. EKS nodes authenticate to ECR through the kubelet's credential provider and the node's IAM role; GKE does the same for Artifact Registry. No secrets involved. A 401 from ECR on EKS means the node role lacks ecr:BatchGetImage, or the repo policy denies the account -- secrets won't fix it, and adding them can mask the real problem until they expire.
Rate limits and the one-node mystery
The toomanyrequests bucket is almost always Docker Hub. Anonymous pulls are capped per IP, and your nodes usually egress through a shared NAT gateway, so the entire cluster shares one quota. A cluster that churns pods -- CI runners, autoscaling, cronjobs -- can burn through it by lunch.
My Friday one-node mystery was exactly this, with a twist: that node was running a monitoring DaemonSet that pulled a fresh agent image every hour, so it individually hit the limit first. The two healthy nodes still had budget left.
The fixes, best first: mirror the images you depend on into your own registry or a pull-through cache (this also protects you from upstream outages and deleted tags); authenticate your Hub pulls, which raises the quota and scopes it to the account instead of the NAT IP; or stop pulling from Hub for anything that matters. The emergency fix is docker login on the nodes via a registry secret, but treat that as a bridge, not the destination.
Frequently asked questions
What causes ImagePullBackOff in Kubernetes?
Four things account for nearly all of them: the image or tag does not exist in the registry, the node cannot authenticate to a private registry, the registry is rate limiting you, or there is a network or DNS problem between the node and the registry. Run kubectl describe pod and read the actual pull error in the Events section, because the message names which of the four you are dealing with.
What is the difference between ErrImagePull and ImagePullBackOff?
ErrImagePull is the state right after the first failed pull attempt, and ImagePullBackOff is what the kubelet reports once it has failed repeatedly and started waiting between retries with an exponential backoff. They have the same causes and the same fixes. The BackOff status just means it has been failing for a while, and the retry interval can stretch to five minutes.
How do I fix ImagePullBackOff on a private registry?
Create a docker-registry Secret in the same namespace as the pod, then reference it in the pod spec under imagePullSecrets, or attach it to the namespace's ServiceAccount so every pod gets it. The most common mistake is creating the secret in the wrong namespace, because secrets do not cross namespace boundaries. On managed clusters like EKS or GKE, the node identity usually handles registry auth, so a 401 there points at IAM, not secrets.
Why does my pod fail to pull an image that definitely exists?
Usually one of three things: the tag exists in a different registry or repository than the one in your manifest, the tag was overwritten or never actually pushed by CI, or the cluster is pulling through a mirror or proxy that does not have it. Verify with a manual pull from your own machine using the exact full image reference from the pod spec, including the registry hostname.
What does the Docker Hub rate limit error look like in Kubernetes?
The event reads toomanyrequests and mentions you have reached your pull rate limit. Docker Hub caps anonymous pulls per IP address, and since your nodes typically egress through a shared NAT, the whole cluster burns through one quota. The fixes are authenticated pulls with docker-registry secrets, a pull-through cache or mirror, or moving images to a registry without those limits.
Where this bites in production
ImagePullBackOff is not one error, it's four, and the kubelet wrote the answer in the pod's events. Read the full pull error, sort it into not found, unauthorized, rate limit, or network, and fix that one thing.
Then do the boring preventive work: immutable tags, a mirror for anything you depend on, and auth that lives where the nodes can actually use it. Friday-you will be grateful.