Taints and Tolerations in Production: Beyond NoSchedule

Our GPU pool sat idle for two weeks while the queue of training jobs grew, and the reason was one copied-from-Stack-Overflow toleration in the wrong direction. The jobs were supposed to land *on* the GPU nodes; instead a caching sidecar deployment with a blanket operator: Exists toleration was camping on the pool, the scheduler had no room for the actual training pods, and nobody had a view that connected "idle expensive pool" with "one deployment's toleration field."

Taints and tolerations look like the simplest feature in Kubernetes -- one repels, one permits -- and they are, right up until the interaction rules start mattering in production. Here's the mental model and the setup I use now, after that bill arrived.

Quick answer: Taints go on nodes and repel pods without a matching toleration; tolerations go on pods and permit them past a taint. NoSchedule affects new pods only, NoExecute also evicts running ones, PreferNoSchedule is a soft hint. A dedicated pool needs taint plus node affinity: the taint keeps others out, the affinity keeps your workload in. Never ship a blanket operator: Exists toleration without an expiry plan.

The three effects, in order of how much they can hurt you

NoSchedule -- only new pods are affected. Everything already running stays. This is the effect for permanent pool segregation: spot pools, GPU pools, compliance-scoped pools. It's predictable, which in scheduling land is a compliment.

PreferNoSchedule -- the scheduler tries to avoid placing intolerant pods but will, if the cluster is tight. It's the right effect for preference-shaped things like "keep workloads off bare-metal nodes unless we're desperate." Just remember that "prefer" means your segregation silently dissolves under pressure, which is either a feature or a bug depending on the pool.

NoExecute -- the sharp one. Pods on the node that don't tolerate the taint get evicted when the taint lands. Its legitimate home is lifecycle: the control plane itself taints failing nodes with node.kubernetes.io/not-ready and node.kubernetes.io/unreachable as NoExecute, draining workloads from nodes that have stopped being trustworthy. When you apply NoExecute yourself, you're performing an eviction event, so scope it and stage it like one.

The pairing with tolerationSeconds is the subtle part:

tolerations:
  - key: "node.kubernetes.io/not-ready"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 120

Read it as: "if this taint lands, I accept eviction, but only after 120 seconds." If the taint is removed within the window, the pod stays. That grace period is a design decision -- your decision, per workload -- about how quickly a degraded node should shake off this specific pod.

The tolerations you didn't know you had

Here's a detail that explains a lot of "wait, why didn't our pods get evicted?" moments: the admission layer injects default tolerations into every pod. Notably, not-ready and unreachable come in as NoExecute with tolerationSeconds: 300.

That five-minute grace is why a brief node network partition doesn't nuke every workload: the control plane taints the node immediately, workloads would be eligible for eviction, but the injected toleration buys five minutes for the blip to pass. It's a sensible default with sharp edges at both ends. Latency-critical workloads sometimes lower it (accept faster failover to a healthy node), while stateful systems sometimes raise it (a partition is not a reason to churn a database primary). Either way, change it per workload, deliberately, and write down why -- the node NotReady recovery path depends on this choreography more than most teams realize.

Dedicated pools: the two-key pattern

The single most common taint request is "keep X off our special nodes and keep Y on them." The mistake is doing only half:

• Taint only: your special workload still spreads across *all* nodes, because taints repel, they don't attract. You paid for a pool and got a suggestion.
• Affinity only: your workload lands on the pool, but so does everything else that fits. The pool isn't dedicated, it's just preferred.

The complete pattern is both:

# 1. Repel: taint the pool (once, per node or via your nodepool tooling)
kubectl taint nodes -l pool=spot workloads=spot:NoSchedule

# 2. Attract: pin the workload to the pool
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: pool
              operator: In
              values: ["spot"]
tolerations:
  - key: "workloads"
    operator: "Equal"
    value: "spot"
    effect: "NoSchedule"

The affinity pin is also what makes capacity math honest: if the pool is out of room, the pod goes Pending instead of silently polluting general nodes. Pending is a *signal* (see pod-stuck-pending debugging), and here it's telling you the truth: your dedicated capacity is full.

Spot pools and the eviction calculus

Spot/preemptible nodes deserve their own note because they combine two taint stories. First, taint them (NoSchedule) so only spot-aware workloads land there. Second, decide your tolerationSeconds posture for the pressure taints: spot nodes die and degrade more often, and workloads that tolerate not-ready/unreachable *forever* will sit as ghosts on a dead spot node instead of rescheduling.

The configuration I've settled on for spot-resident batch work: tolerate the spot taint indefinitely (that's the point of the pool), but keep the injected not-ready/unreachable grace short-ish, because when a spot node goes away, speed of rescheduling *is* the availability story. Stateful workloads get the opposite numbers, and neither pool shares a node pool spec with the other.

The blanket toleration anti-pattern

The GPU-pool incident from the intro came from this YAML, in a deployment whose author just wanted it to schedule "anywhere":

tolerations:
  - operator: "Exists"   # tolerates EVERY taint, all effects

With no key, that tolerates every taint in the cluster: not-ready, unreachable, disk-pressure, your compliance segregation, the GPU pool, everything. It works -- until the day it works too well, and your cache tier is occupying the training pool, or your cron jobs ride on nodes mid-NoExecute drain.

Rules I enforce now:

• Tolerations name a key. Always. operator: Exists with a key means "any value of this taint," which is occasionally right; with no key it means "any taint at all," which is never right for an app workload.
• Every blanket-looking toleration in a base chart gets reviewed like a security setting, because that's what it is: an authorization to ignore node state.
• The audit question for any scheduling mystery is "which tolerations and affinities select this pod?" -- a question that's only answerable at 3am if you can see pod spec and node taints side by side. This is a good moment to plug the scheduler internals post for how the filter phase actually consumes these fields.

Taints vs affinity vs topology: who wins?

A quick decision table, because these three get confused constantly:

You want...	Reach for
"Keep everything off these nodes except X"	Taint (NoSchedule) + affinity for X
"X should run on these nodes if possible"	nodeAffinity (preferred) or PreferNoSchedule framing
"X must not run on the same node/zone as Y"	pod anti-affinity or topology spread constraints
"Evict what's here, now"	Taint with NoExecute (cordon/drain does this under the hood)

Drain is worth one line of respect: kubectl drain applies an unschedulable taint and then evicts with NoExecute semantics -- your workload tolerations and PDBs are what stand between a drain and an outage, which is why upgrade runtimes are really toleration-and-budget exercises in disguise.

Frequently asked questions

What is the difference between NoSchedule, PreferNoSchedule, and NoExecute?

NoSchedule keeps new pods off the node but leaves running pods alone. PreferNoSchedule is a soft hint the scheduler tries to honor. NoExecute additionally evicts pods already running on the node if they lack a toleration, which makes it the one to handle with care.

How do I create a dedicated node pool for a workload?

You need both halves: a taint on the nodes to repel everything that does not tolerate the key, and a nodeAffinity rule on the workload to attract it onto that pool. Taints alone only stop others from coming in; affinity alone only pulls your workload in, without stopping anyone else.

Why do pods survive NotReady nodes for a few minutes by default?

The API server automatically injects tolerations for node.kubernetes.io/not-ready and node.kubernetes.io/unreachable with tolerationSeconds of 300 into every pod. That grace window absorbs brief node blips without mass evictions. Shorten it deliberately for latency-sensitive workloads, but know that lowering it fleet-wide turns short network partitions into restart storms.

Is operator: Exists in a toleration dangerous?

It tolerates every value of the given key, or every taint at all if no key is given, which is usually more than the author intended. The classic accident is copying a blanket toleration onto a workload so it can schedule onto one special pool, and later finding it camping on GPU nodes or spot nodes it was never meant for.

Where this bites in production

The recurring failure modes are all variations on one theme: tolerations outliving their purpose. The blanket Exists from a debugging session. The tolerationSeconds: 3600 someone raised during an incident and never lowered. The pool whose taint changed key and whose workload tolerations now match nothing.

Audit these the way you audit RBAC: which workloads can ignore which node states, and does that still match the intent? When the answer lives in a searchable view instead of forty YAML files, taints stop being a source of surprises and go back to being the simple feature they advertise themselves as.