Inside the Kubernetes Scheduler: Why Your Pods Land Where They Do

The page fired at 3:12 AM: p99 latency on checkout, up 4x. Grafana said one node, ip-10-0-3-47, was pinned at 96% CPU while the cluster averaged 31%. And kubectl delivered the punchline: 41 of 180 checkout pods were sitting on that single node. In a 120-node cluster. On a Deployment with a perfectly reasonable-looking topologySpreadConstraints block.

Nothing was broken. The scheduler had done exactly what it was told. That's the uncomfortable truth about kube-scheduler: it's not a placement oracle, it's a one-shot decision function that runs once per pod and never looks back.

Quick answer: kube-scheduler works in two phases. Filtering (the Filter extension point) throws out every node that fails a hard rule, taints, required affinity, insufficient resources, and scoring ranks the survivors, with each plugin contributing a weighted 0–100 score. Topology spread with whenUnsatisfiable: ScheduleAnyway is only a scoring preference, and all spread skew is computed at schedule time against the nodes eligible *at that moment*. The scheduler never reconsiders a running pod, which is exactly why the descheduler exists.

Filtering vs scoring: the two phases

Every pod the scheduler picks off the queue goes through a scheduling cycle, serialized one pod at a time: snapshot the cluster state, run PreFilter, run Filter plugins in parallel across nodes, and collect feasibleNodes. If that set is empty, the pod doesn't just fail, it drops into PostFilter, which is where preemption lives.

If nodes survive, scoring starts. Each scoring plugin (PreScore, Score, and for the topology-aware ones NormalizeScore) returns a value between 0 and 100 per node, multiplied by the plugin's configured weight, summed into a final score. Ties are broken at random in selectHost. Then the scheduler *assumes* the pod on the winning node (Reserve), and the binding cycle runs asynchronously in a goroutine, Permit, PreBind, Bind.

The distinction that bites people: filtering is boolean, scoring is negotiable. A taint without a toleration is a wall. A preferred affinity is a vote.

What your constraints actually do to the math

Knob	Filter or Score	Plugin (weight)
nodeSelector / required node affinity	Filter	NodeAffinity
preferred node affinity	Score	NodeAffinity (1)
Taints: NoSchedule, NoExecute	Filter	TaintToleration
Taints: PreferNoSchedule	Score	TaintToleration (3)
Topology spread DoNotSchedule	Filter	PodTopologySpread
Topology spread ScheduleAnyway	Score	PodTopologySpread (2)
Preferred pod (anti-)affinity	Score	InterPodAffinity (2)
Resource fit	Score	NodeResourcesFit (1)

Two things in that table deserve more attention than they get.

First, NodeResourcesFit defaults to LeastAllocated, which means an almost-empty node scores near 100. Every freshly joined node is a pod magnet until it fills up. You can flip it to MostAllocated (bin-packing) or RequestedToCapacityRatio with custom shape points, but that's a cluster-wide scheduler-config decision, not a per-workload one.

Second, the weights. TaintToleration's PreferNoSchedule scores at weight 3, triple the resource fit score. A soft taint is nearly a hard one. Meanwhile your lovingly crafted ScheduleAnyway spread at weight 2 can still lose when preferences stack against it.

Back to 3 AM: how "spread" piled 41 pods onto one node

The setup: EKS 1.29, about 120 nodes, Cilium as the CNI. That evening someone rolled a managed node group upgrade. That same evening a flash sale started and the HPA walked checkout from 60 to 180 replicas. The new nodes joining the cluster carried the node.cilium.io/agent-not-ready=true:NoSchedule taint until the Cilium agent came up, which, under the load of a surge, took a while on some of them.

The mechanism: PodTopologySpread computes skew over *eligible domains only*. With the default nodeAffinityPolicy: Honor and nodeTaintsPolicy: Honor, nodes the pod can't run on are excluded from the calculation entirely. At 3 AM, exactly four nodes passed all the filters. The scheduler spread 120 new pods beautifully, across four nodes. maxSkew: 1, honored to the letter. ip-10-0-3-47, being a large instance, absorbed 41 of them.

Twenty minutes later every node was Ready and taint-free, and it didn't matter one bit. The scheduler had nothing left to decide; the pods were already bound. There is no reconcile loop for placement. We ate a degraded sale night over a constraint that was, at every instant, perfectly satisfied.

The fixes: whenUnsatisfiable: DoNotSchedule for services where we'd rather see Pending pods than a hotspot, and the descheduler to clean up historical skew. We also started watching scheduler_pending_pods{queue="unschedulable"} during node rotations.

Preemption: PriorityClass and its blast radius

When a pod fails all filters, PostFilter runs DefaultPreemption: the scheduler simulates evicting lower-priority pods node by node, and picks the victim set that makes the pod fit with the least damage, fewest victims, then fewest PDB violations. PDBs are *preferred*, not guaranteed. Victims get their graceful termination period and go back through the queue, where, if the cluster is genuinely full, they may preempt something else. One fat critical pod can evict a dozen batch workers across four nodes in a single cycle.

Two opinions I've earned the hard way. Use preemptionPolicy: Never on any PriorityClass where you want queue priority without eviction rights, it does exactly what it says and saves you from yourself. And watch scheduler_preemption_attempts_total plus scheduler_preemption_victims; if those move during normal operation, your capacity planning has a bug, not your scheduler.

Scheduler profiles: when to write one

KubeSchedulerConfiguration lets you define multiple named profiles in one scheduler process, each with its own plugin config. Pods opt in with spec.schedulerName. The canonical use case is bin-packing batch jobs without penalizing latency-sensitive services:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: default-scheduler
  - schedulerName: batch-binpack
    pluginConfig:
      - name: NodeResourcesFit
        args:
          scoringStrategy:
            type: MostAllocated

Profiles share one binary, one cache, one queue machinery, cheap to add. If you want to reorder the queue itself or implement gang scheduling, you've outgrown profiles; that's what projects like Kueue are for.

Why the descheduler exists

Everything above has the same root cause: scheduling is a point-in-time decision, and clusters drift. The descheduler (sigs.k8s.io/descheduler, usually a CronJob) is the deliberate second pass: it evicts pods that violate today's rules, LowNodeUtilization, RemovePodsViolatingTopologySpread, RemovePodsViolatingNodeAffinity, through the eviction API, so PDBs are honored and the scheduler places replacements with current information. Cap the chaos with maxNoOfPodsToEvictPerNode.

If you take one thing from this post: read every scheduling knob as either "wall" or "vote," and remember that both are only counted once, at bind time. The pods landing in the wrong place are usually telling you when the decision happened, not what the rules say.

Frequently asked questions

What is the difference between filtering and scoring in kube-scheduler?

Filtering is binary: every node that fails any Filter plugin is out, and the pod stays Pending if zero nodes pass. Scoring ranks the surviving nodes, each score plugin returns 0 to 100 per node, multiplies by its configured weight, and the scheduler picks the highest total with random tie-breaking. Most placement surprises live in scoring, because preferences quietly lose to each other there.

Why do my topologySpreadConstraints not spread existing pods?

Topology spread is only evaluated while a pod is being scheduled, and skew is computed against the nodes eligible at that moment. kube-scheduler never re-evaluates a running pod, so skew created by node outages, cordons, or a scale-up during an upgrade stays in place forever. The descheduler's RemovePodsViolatingTopologySpread plugin is the standard way to fix it after the fact.

How does PriorityClass preemption work in Kubernetes?

When a higher-priority pod can't be scheduled, the DefaultPreemption plugin simulates evicting lower-priority pods on each node and picks the victim set that makes the pod fit with the least damage, preferring to respect PodDisruptionBudgets but not guaranteeing them. Victims get a graceful termination and go back through the queue, which can cascade into more evictions. Set preemptionPolicy: Never on a PriorityClass if you want queue priority without ever evicting anything.

When should I use a scheduler profile instead of a custom scheduler?

Use a profile when you want different plugin behavior, say MostAllocated bin-packing for batch jobs, without running a second scheduler binary; pods select it with spec.schedulerName. Profiles share one kube-scheduler process and its caches, so they're cheap. Write an out-of-tree scheduler only when you need logic the framework's extension points can't express.