14.1 The Kubernetes Scheduler — How It Picks a Node

⏱️ ~5 min read

TL;DR: The kube-scheduler watches for unscheduled pods and picks the best node for each one by running every candidate node through two phases: Filtering (eliminate nodes that can’t run the pod) and Scoring (rank the survivors; highest score wins). You influence both phases using the placement mechanisms covered in this chapter.

The Scheduling Pipeline

Every pod starts life as Pending — it exists in etcd but has no nodeName. The scheduler’s job is to assign one.

graph TD
    P["New Pod\n(Pending, no nodeName)"] --> WATCH["kube-scheduler\nwatching API server"]
    WATCH --> FILTER["Phase 1: Filtering\n(eliminate unfit nodes)"]
    FILTER --> SCORE["Phase 2: Scoring\n(rank remaining nodes)"]
    SCORE --> BIND["Bind: write nodeName\nto pod spec"]
    BIND --> KUBELET["kubelet on chosen node\npulls image + starts containers"]

    subgraph "Filter Plugins (examples)"
        F1["NodeResourcesFit\n(enough CPU/memory?)"]
        F2["NodeAffinity\n(label matches?)"]
        F3["TaintToleration\n(tolerates taints?)"]
        F4["PodTopologySpread\n(spread constraints met?)"]
    end

    subgraph "Score Plugins (examples)"
        S1["LeastAllocated\n(spread load)"]
        S2["NodeAffinity\n(preferred labels)"]
        S3["ImageLocality\n(image already pulled?)"]
    end

    FILTER -.-> F1 & F2 & F3 & F4
    SCORE -.-> S1 & S2 & S3

Phase 1 — Filtering (Hard Constraints)

A node is eliminated if any filter fails. Common filter checks:

If zero nodes survive filtering, the pod stays Pending. This is the most common cause of pods stuck in Pending — always check kubectl describe pod <name> for the scheduler’s message.

Phase 2 — Scoring (Soft Preferences)

Surviving nodes are each scored 0–100 by each scoring plugin, then weighted and summed. Highest total wins.

Diagnosing Scheduling Problems

# Pod stuck in Pending — check why:
kubectl describe pod my-pod | grep -A 20 "Events:"

Common messages and their meaning:

# See all pending pods across the cluster
kubectl get pods -A | grep Pending

# Check scheduler events
kubectl get events --field-selector reason=FailedScheduling -A

# See what resources are actually available on nodes
kubectl describe nodes | grep -A 5 "Allocated resources"

Manual Scheduling (Bypass the Scheduler)

You can hard-pin a pod to a specific node by setting nodeName directly — the scheduler is skipped entirely:

apiVersion: v1
kind: Pod
metadata:
  name: pinned-pod
spec:
  nodeName: minikube-m02   # ← Bypass scheduler; go directly to this node
  containers:
  - name: nginx
    image: nginx:alpine

⚠️ Avoid in production. If that node goes down, the pod is not rescheduled — it just stays Failed. Use node affinity rules instead so the scheduler can pick an equivalent node.

The NodeName Flow vs Scheduler Flow

Manual (nodeName):    Pod → API Server → kubelet on named node
                      (no scheduler involved, no rescheduling if node fails)

Scheduler Flow:       Pod → API Server → Scheduler → Filter → Score → Bind
                      (rescheduled on failure, respects resource availability)

✅ Quick Check

Q1: A pod has requests.memory: 512Mi but all nodes only have 400Mi free. What happens?

Q2: What’s the difference between a node surviving Filtering vs being chosen after Scoring?

Q3: Your pod is stuck in Pending. What’s the first command you run?