Introduction

You know Docker. You’ve written Compose files. You’ve built images, connected containers, and mounted volumes. Now your app needs to survive the real world — scaling, self-healing, zero-downtime deploys, secrets management, and multi-service coordination at scale. That’s what this book teaches.

What You’ll Build

By the end of this book, you’ll have deployed KubeShop — a five-service microservices e-commerce app — to both a local Minikube cluster and a production-grade Azure Kubernetes Service (AKS) cluster, complete with Helm packaging, autoscaling, RBAC, and a full CI/CD pipeline.

graph LR
    U([User]) --> W[web :3000\nAPI Gateway]
    W --> C[catalogue :3001]
    W --> CA[cart :3002]
    W --> O[orders :3003]
    W --> P[payment :5000]
    C --> MDB1[(MongoDB)]
    O --> MDB2[(MongoDB)]
    CA --> R[(Redis)]

The stack: Node.js · Python · MongoDB · Redis · Docker · Kubernetes · Helm · GitHub Actions · ArgoCD · AKS

Prerequisites

You need these before starting. Run the check commands — if they pass, you’re good to go.

💡 Tip: On Ubuntu/Debian, the quickest setup is:

# kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

# minikube
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube

# helm
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

How to Use This Book

Each chapter is designed to take 30–60 minutes and ends with a hands-on lab. The pattern is always:

Concept (short) → Example (immediate) → Try it → Break it

• Read linearly if you’re new to K8s.
• Jump around if you have experience — every section is self-contained.
• Do the labs. Reading about K8s is like reading about swimming. You have to get in the water.

🔥 The “Break It!” philosophy: The fastest way to understand a system is to watch it fail. Every chapter has intentional breakage exercises. Don’t skip them.

Chapter Roadmap

Ready? Let’s go to Chapter 1. →

Chapter 1: The Container Orchestration Problem

⏱️ Total chapter time: ~55 min (25 min reading + 30 min lab)

After this chapter, you will be able to: Explain why Kubernetes exists, describe every component of a K8s cluster, and navigate a live Minikube cluster using kubectl.

What’s Inside

Prerequisites

• Docker installed and working (docker ps succeeds)
• kubectl installed (kubectl version --client)
• minikube installed (minikube version)
• No prior Kubernetes experience needed

1.1 From Docker to Orchestration — Why Compose Isn’t Enough

⏱️ ~8 min read

TL;DR: Docker Compose is a great local tool, but it breaks down the moment you need resilience, scaling, or multi-host deployments. Kubernetes solves all of that — systematically.

You Already Know How to Do This

Here’s a dead-simple Docker Compose file that runs a web app and its database:

# docker-compose.yml
services:
  web:
    image: myapp:latest
    ports:
      - "3000:3000"
    environment:
      - DB_HOST=db
    depends_on:
      - db

  db:
    image: mongo:7
    volumes:
      - mongo_data:/data/db

volumes:
  mongo_data:

Run docker compose up and it works. Beautifully simple.

Now your app gets traffic. Real traffic. And things start to go wrong.

The Five Walls You’ll Hit

Wall 1: No Self-Healing

Your web container crashes at 3am. Compose does nothing. It’s dead until someone manually restarts it.

🔗 Docker Parallel: restart: always in Compose helps, but only on the same host. If the host dies, so does everything on it.

Wall 2: Manual Scaling

You need 10 instances of web to handle load:

docker compose up --scale web=10

OK — but now all 10 are on the same machine, competing for the same CPU and RAM. You’ve created 10x the problem.

Wall 3: Rolling Updates Are Your Problem

Deploying a new version means downtime unless you write complex scripts yourself. Compose has no concept of “replace containers one at a time while keeping traffic flowing.”

Wall 4: Single Host

Compose is a single-machine tool. Your app is bound to one server. If that server goes down, everything goes down. Scale beyond one machine? You’re on your own.

Wall 5: No Load Balancing

Who distributes traffic across your 10 web containers? Compose doesn’t. You’d need to set up Nginx or HAProxy yourself, manually update configs when containers come and go.

What Kubernetes Gives You

Kubernetes is what happens when you solve all five walls systematically.

graph LR
    subgraph "Docker Compose World"
        DC[Your Laptop\nor Single VM]
        DC --> C1[container 1]
        DC --> C2[container 2]
    end

    subgraph "Kubernetes World"
        CP[Control Plane\nK8s Brain]
        N1[Node 1]
        N2[Node 2]
        N3[Node 3]
        CP --> N1
        CP --> N2
        CP --> N3
        N1 --> P1[Pod] & P2[Pod]
        N2 --> P3[Pod] & P4[Pod]
        N3 --> P5[Pod]
    end

The Mental Shift: Desired State

The single most important concept in Kubernetes:

You declare what you want. Kubernetes makes it happen and keeps it that way.

In Docker Compose, you issue commands: “start this container”, “stop that container.” You’re in control.

In Kubernetes, you write a declaration: “I want 5 instances of this app running at all times.” Then you walk away. Kubernetes watches the cluster 24/7 and reconciles reality with your declaration.

graph LR
    Y[You write YAML:\n'I want 5 replicas'] --> K[Kubernetes API]
    K --> CM[Controller Manager\nconstantly watches]
    CM -->|"Reality: 5 pods running\n✅ Nothing to do"| OK[All good]
    CM -->|"Reality: 3 pods running\n❌ 2 missing"| FIX[Create 2 new pods]

This is called the reconciliation loop — or more precisely, the control loop. It runs constantly, forever, for every resource in your cluster.

🔗 Docker Parallel: There’s no equivalent in Compose. The closest thing is restart: always, but that only handles crashes, not desired-count enforcement.

Try It

Before we write any Kubernetes configs, let’s see Compose’s limit firsthand:

# Kill a container and watch Compose do nothing (without restart policy)
docker run -d --name test-no-restart nginx
docker kill test-no-restart
docker ps -a  # Status: Exited

Expected output:

CONTAINER ID   IMAGE   COMMAND   CREATED   STATUS                     PORTS   NAMES
abc123         nginx   ...       ...       Exited (137) 5 seconds ago         test-no-restart

Dead. Staying dead. Now you understand why you need Kubernetes.

# Cleanup
docker rm test-no-restart

Key Takeaways

✅ Quick Check

Q1: You have 3 replicas of your app running. You manually delete one. What does Kubernetes do?

Q2: Your team uses docker-compose up --scale web=5. Why is this insufficient for production?

Q3: What happens in Kubernetes if you don’t explicitly tell it to stop something?

1.2 Kubernetes Architecture — The 10,000ft View

⏱️ ~8 min read

TL;DR: A Kubernetes cluster has two kinds of machines: a Control Plane (the brain) and Worker Nodes (the muscle). Everything you do goes through the API Server.

The Two-Tier Architecture

Every Kubernetes cluster, whether it’s running on your laptop via Minikube or powering Netflix in production, follows the same fundamental architecture.

graph TB
    subgraph "Control Plane (The Brain)"
        API[🔵 API Server\nkube-apiserver]
        ETCD[🗃️ etcd\nCluster Store]
        SCHED[📅 Scheduler\nkube-scheduler]
        CM[⚙️ Controller Manager\nkube-controller-manager]

        API <-->|read/write| ETCD
        API --> SCHED
        API --> CM
    end

    subgraph "Worker Node 1"
        KB1[Kubelet]
        KP1[kube-proxy]
        CR1[Container Runtime\ncontainerd / CRI-O]
        P1[Pod A]
        P2[Pod B]
        KB1 --> CR1 --> P1 & P2
    end

    subgraph "Worker Node 2"
        KB2[Kubelet]
        KP2[kube-proxy]
        CR2[Container Runtime]
        P3[Pod C]
        KB2 --> CR2 --> P3
    end

    YOU([👤 You]) -->|kubectl| API
    API -->|instructions| KB1 & KB2

The Control Plane — The Brain

The control plane runs on dedicated machines (usually 1–3 for HA). It makes all the cluster-wide decisions. You never run application workloads on control plane nodes.

📝 Note: The control plane components communicate only through the API Server. Nothing talks to etcd directly except the API Server.

Worker Nodes — The Muscle

Worker nodes are where your application pods actually run. A cluster can have 1 to thousands of them.

🔗 Docker Parallel: The container runtime on a worker node is doing the same job as the Docker daemon on your machine — pulling images and running containers. The Kubelet is the layer above it telling it what to run.

The Flow: What Happens When You Run kubectl apply

Let’s trace a deployment request through the entire system:

sequenceDiagram
    participant You
    participant API as API Server
    participant ETCD as etcd
    participant SCHED as Scheduler
    participant CM as Controller Manager
    participant KL as Kubelet (Node)

    You->>API: kubectl apply -f deployment.yaml
    API->>ETCD: Store deployment spec
    API-->>You: "deployment created"
    CM->>API: Watch for new deployments
    API-->>CM: "new Deployment found"
    CM->>API: Create ReplicaSet → create Pods
    API->>ETCD: Store pending pods
    SCHED->>API: Watch for unscheduled pods
    API-->>SCHED: "3 unscheduled pods"
    SCHED->>API: Assign pods to nodes
    API->>ETCD: Update pod → node assignment
    KL->>API: Watch for pods assigned to my node
    API-->>KL: "here are your pods"
    KL->>KL: Pull image, start containers
    KL->>API: Report pod status: Running

The key insight: everything is mediated by the API Server and persisted in etcd. Components don’t talk to each other directly — they all watch the API Server for changes.

Minikube: One Machine, Full Cluster

For local development, Minikube runs the entire cluster (control plane + one worker node) inside a single VM or Docker container on your machine.

Your Laptop
└── Minikube VM / Docker Container
    ├── Control Plane (kube-apiserver, etcd, scheduler, controller-manager)
    └── Worker Node (kubelet, kube-proxy, containerd)
        └── Your pods run here

This is why Minikube is perfect for learning — you get a real K8s cluster with no cloud bills.

⚠️ Warning: Minikube’s single-node setup means control plane and workloads share resources. Never use Minikube in production.

Try It

# Start minikube (if not already running)
minikube start

# See all cluster components
kubectl get pods -n kube-system

Expected output:

NAME                               READY   STATUS    RESTARTS   AGE
coredns-5dd5756b68-xxxxx           1/1     Running   0          5m
etcd-minikube                      1/1     Running   0          5m
kube-apiserver-minikube            1/1     Running   0          5m
kube-controller-manager-minikube   1/1     Running   0          5m
kube-proxy-xxxxx                   1/1     Running   0          5m
kube-scheduler-minikube            1/1     Running   0          5m
storage-provisioner                1/1     Running   0          5m

You’re looking at the control plane components running as pods. Yes — K8s runs its own control plane inside itself (except etcd, which runs as a static pod directly managed by the kubelet).

# Inspect your single node
kubectl get nodes

Expected output:

NAME       STATUS   ROLES           AGE   VERSION
minikube   Ready    control-plane   5m    v1.30.x

Key Takeaways

✅ Quick Check

Q1: You submit a pod spec with kubectl apply. Which component decides which node the pod runs on?

Q2: etcd goes down in your production cluster. What happens?

Q3: A worker node loses network connectivity to the control plane. What happens to the pods already running on it?

1.3 Control Plane Deep Dive

⏱️ ~6 min read

TL;DR: Four components run the K8s control plane. Understanding what each one does — and what breaks when it’s gone — will save you hours of debugging.

The Four Components, Explained

🔵 API Server (kube-apiserver)

The API Server is the only component that talks to etcd. Everything else — the scheduler, controller manager, kubelet, kubectl — talks to the API Server, never directly to each other.

graph LR
    kubectl --> API
    Scheduler --> API
    CM[Controller Manager] --> API
    Kubelet --> API
    API <--> etcd

What it does:

• Validates and processes all API requests (authentication, authorization, admission control)
• Reads/writes cluster state to etcd
• Exposes the Kubernetes REST API on port 6443

What breaks without it: Everything. You can’t do anything. Existing pods keep running (kubelet is independent), but no changes are possible.

🗃️ etcd

etcd is a distributed key-value store used as Kubernetes’ backing store for all cluster data. Every object you create — every Pod, Service, ConfigMap, Secret — is stored here as JSON.

# What etcd stores (conceptually):
/registry/pods/default/my-nginx-pod         → { "kind": "Pod", "status": "Running", ... }
/registry/deployments/default/my-app        → { "kind": "Deployment", "replicas": 3, ... }
/registry/services/default/my-svc          → { "kind": "Service", "clusterIP": "10.96.0.1", ... }

Key facts:

• Uses the Raft consensus algorithm — needs a majority (quorum) of members to be healthy
• Always run 3 or 5 instances in production (never 2 or 4 — even numbers create split-brain risk)
• Back up etcd regularly — losing it means losing all cluster state

⚠️ Warning: etcd is not a general-purpose database. It’s purpose-built for small, critical configuration data. Don’t store application data in it.

📅 Scheduler (kube-scheduler)

The scheduler watches for Pending pods (pods with no node assigned) and decides where to run them.

graph TD
    A[New Pod: Pending\nno node assigned] --> B{Scheduler}
    B --> C[Filter: Which nodes\ncan run this pod?]
    C --> D[Score: Which node\nis the best fit?]
    D --> E[Bind pod to winning node]
    E --> F[Pod: Scheduled\nKubelet takes over]

The scheduling decision considers:

• Resource requests (CPU/memory) — does the node have enough?
• Node selectors and affinity rules — does the pod want a specific node?
• Taints and tolerations — is the pod “allowed” on this node?
• Pod spread constraints — for even distribution

📝 Note: The scheduler doesn’t start containers — it just writes a node assignment to etcd. The kubelet on that node then picks it up and does the actual work.

Try It

Watch the scheduler in action:

# Create a pod and watch its transition through Pending → Running
kubectl run watcher-test --image=nginx --restart=Never
kubectl get pod watcher-test -w

Expected output:

NAME           READY   STATUS    RESTARTS   AGE
watcher-test   0/1     Pending   0          0s
watcher-test   0/1     ContainerCreating   0          1s
watcher-test   1/1     Running   0          3s

# Cleanup
kubectl delete pod watcher-test

⚙️ Controller Manager (kube-controller-manager)

This is a single binary that runs dozens of controllers — each one a control loop watching for a specific resource type.

graph LR
    CM[Controller Manager]
    CM --> RC[ReplicaSet Controller]
    CM --> DC[Deployment Controller]
    CM --> NC[Node Controller]
    CM --> JC[Job Controller]
    CM --> EC[Endpoints Controller]
    CM --> SA[ServiceAccount Controller]

    RC -->|"Desired: 3, Got: 2\n→ Create 1 pod"| API[API Server]
    NC -->|"Node unreachable\n→ Evict pods"| API

A few important controllers:

💡 Tip: When you’re confused about why something happened automatically (a pod was deleted, a new one appeared), the answer is almost always “a controller did it.”

What Each Component’s Failure Looks Like

This table reveals something important: your applications keep running even if the control plane goes down. The kubelet on each worker node is self-sufficient enough to maintain existing pods. The control plane is only needed for changes.

Key Takeaways

✅ Quick Check

Q1: You scale a Deployment from 3 to 5 replicas. Which control plane component actually creates the 2 new pods?

Q2: Why does Kubernetes run etcd with 3 instances instead of 2?

Q3: Your Scheduler pod crashes on Minikube. You had 3 running nginx pods. What happens to them?

1.4 Worker Nodes and the Kubelet

⏱️ ~5 min read

TL;DR: Worker nodes are where your pods actually live. Three components do the work: Kubelet (the agent), kube-proxy (the network rules), and a container runtime (the actual container runner).

The Three Node Components

graph TB
    subgraph "Worker Node"
        KB[Kubelet\n'The Node Agent']
        KP[kube-proxy\n'Network Rules']
        CR[Container Runtime\ncontainerd]
        
        subgraph "Pod A"
            C1[Container 1]
            C2[Container 2]
        end
        
        subgraph "Pod B"
            C3[Container 3]
        end
        
        KB -->|manages| C1 & C2 & C3
        CR -->|runs| C1 & C2 & C3
        KP -->|routes traffic to| C1 & C3
    end

    API[API Server] -->|pod specs| KB
    KB -->|status updates| API

Kubelet — The Node Agent

The Kubelet is the most important component on each worker node. It’s the bridge between the control plane and the containers.

What the Kubelet does:

1. Watches the API Server for pods assigned to its node
2. Tells the container runtime to pull images and start containers
3. Runs health checks (liveness/readiness probes) on containers
4. Reports pod status back to the API Server continuously

# See kubelet logs on Minikube
minikube ssh "sudo journalctl -u kubelet -n 50 --no-pager"

📝 Note: The Kubelet is the only component that runs directly on the node as a systemd service — not as a pod. Everything else can be containerized, but not the thing that starts containers.

🔗 Docker Parallel: The Kubelet is like a smarter, cluster-aware docker run — but instead of you telling it what to run, the API Server does.

kube-proxy — Network Rules Manager

kube-proxy runs on every node and maintains iptables/IPVS rules that implement Kubernetes Services.

What it does: When you create a Service with a virtual IP (ClusterIP), kube-proxy programs the node’s networking layer to forward traffic destined for that IP to the correct pods.

graph LR
    Client[Client Pod] -->|"10.96.0.1:80\n(Service ClusterIP)"| KP[kube-proxy rules\non this node]
    KP -->|forwards to| P1[Pod 10.244.1.5:8080]
    KP -->|or forwards to| P2[Pod 10.244.2.3:8080]
    KP -->|or forwards to| P3[Pod 10.244.3.7:8080]

📝 Note: In modern clusters (K8s 1.9+), kube-proxy defaults to IPVS mode, which is more efficient than iptables for large clusters with thousands of services.

Container Runtime — The Actual Container Runner

The container runtime is what actually pulls images and runs containers. Kubernetes uses the Container Runtime Interface (CRI) to talk to it, making the runtime swappable.

⚠️ Warning: Docker was removed as a direct Kubernetes runtime in v1.24. Minikube still uses Docker as the node driver (to create the VM), but containerd is the runtime inside the cluster. Your Docker images still work — the image format is standardized (OCI).

# Check the runtime on your minikube node
kubectl get node minikube -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}'

Expected output:

containerd://1.7.x

How a Pod Gets Running: The Full Chain

Here’s the complete journey from your kubectl apply to a running container:

sequenceDiagram
    participant K as kubectl
    participant A as API Server
    participant S as Scheduler
    participant KB as Kubelet
    participant CR as Container Runtime

    K->>A: Apply pod spec
    A->>A: Validate, store in etcd
    S->>A: Poll: any unscheduled pods?
    A-->>S: Yes, pod XYZ
    S->>A: Assign pod XYZ to node-1
    KB->>A: Poll: any pods for me?
    A-->>KB: Yes, pod XYZ
    KB->>CR: Pull image nginx:latest
    CR-->>KB: Image ready
    KB->>CR: Start container
    CR-->>KB: Container running (PID 1234)
    KB->>A: Update pod status → Running

Try It

# Get detailed node info — see the kubelet version, container runtime, OS
kubectl describe node minikube | head -40

Look for:

• Container Runtime Version — shows containerd
• Kubelet Version — the K8s version running on this node
• Allocatable — how much CPU/memory is available for pods
• Conditions — should be Ready: True

Key Takeaways

✅ Quick Check

Q1: The Kubelet on node-2 crashes. What happens to pods on that node?

Q2: Why can’t Kubernetes use the Docker daemon directly as its container runtime after v1.24?

Q3: You create a Service with clusterIP: 10.96.5.50. Which component makes that IP actually work?

Lab: Your First Cluster — Minikube Setup & Exploration

⏱️ ~30 min hands-on

Objectives

• Start a Minikube cluster and verify all control plane components are running
• Inspect a node and understand what the output means
• Navigate the cluster using kubectl get, describe, and cluster-info
• Access the Kubernetes Dashboard
• Call the API Server directly with curl

Setup

# Verify your tools are installed
kubectl version --client
minikube version
docker version

# Start minikube with 2 CPUs and 2GB RAM
minikube start --cpus=2 --memory=2048

Expected output:

😄  minikube v1.33.x on Linux
✨  Using the docker driver based on existing profile
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.44 ...
🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.30.x on Docker 26.x.x ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Exercise 1: Verify the Cluster is Healthy

What we’re doing: Check that all control plane components are running and the node is ready.

kubectl cluster-info

Expected output:

Kubernetes control plane is running at https://192.168.49.2:8443
CoreDNS is running at https://192.168.49.2:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

💡 What just happened? cluster-info shows the API Server URL and any running cluster services. The IP 192.168.49.2 is the Minikube VM’s internal IP.

# Check node status
kubectl get nodes

Expected output:

NAME       STATUS   ROLES           AGE   VERSION
minikube   Ready    control-plane   5m    v1.30.x

# Get more detail
kubectl get nodes -o wide

Expected output:

NAME       STATUS   ROLES           AGE   VERSION    INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
minikube   Ready    control-plane   5m    v1.30.x    192.168.49.2   <none>        Ubuntu 22.04.4 LTS   5.15.x           containerd://1.7.x

Note the CONTAINER-RUNTIME column — you can see containerd is what’s actually running your containers.

Exercise 2: Inspect the Control Plane Pods

What we’re doing: Kubernetes runs its own control plane as pods in the kube-system namespace.

kubectl get pods -n kube-system

Expected output:

NAME                               READY   STATUS    RESTARTS   AGE
coredns-5dd5756b68-abcde           1/1     Running   0          10m
etcd-minikube                      1/1     Running   0          10m
kube-apiserver-minikube            1/1     Running   0          10m
kube-controller-manager-minikube   1/1     Running   0          10m
kube-proxy-xxxxx                   1/1     Running   0          10m
kube-scheduler-minikube            1/1     Running   0          10m
storage-provisioner                1/1     Running   0          10m

💡 What just happened? The control plane runs as pods! Each component (etcd, API server, scheduler, controller manager) is a pod managed by the kubelet. This is called a static pod — defined directly in /etc/kubernetes/manifests/ on the node, not via the API.

# Describe the API server pod
kubectl describe pod kube-apiserver-minikube -n kube-system | head -30

Look at the Image: line — you’ll see the exact version of the API server binary being used.

Exercise 3: Deeply Inspect the Node

kubectl describe node minikube

This is a long output. Here’s what to focus on:

# Pipe through grep to extract key sections
kubectl describe node minikube | grep -A5 "Conditions:"
kubectl describe node minikube | grep -A5 "Allocatable:"
kubectl describe node minikube | grep -A10 "Allocated resources:"

What to look for:

Exercise 4: Explore All Namespaces

What we’re doing: Everything in K8s lives in a namespace. Let’s see what’s already there.

kubectl get namespaces

Expected output:

NAME              STATUS   AGE
default           Active   15m
kube-node-lease   Active   15m
kube-public       Active   15m
kube-system       Active   15m

# See EVERYTHING in every namespace
kubectl get all --all-namespaces

Exercise 5: Access the Dashboard

minikube dashboard

💡 What just happened? Minikube opens a web browser with the Kubernetes Dashboard — a visual UI for the cluster. You can see pods, deployments, services, and more. It’s useful for exploration but not recommended for production management.

The dashboard opens automatically. Explore:

• Workloads → Pods → see the kube-system pods
• Cluster → Nodes → see node resource usage

Press Ctrl+C to stop when done.

Exercise 6: Talk to the API Server Directly

Every kubectl command is just an HTTP request. Let’s see the raw API.

# Start a proxy to the API Server (runs in background)
kubectl proxy --port=8001 &

# Now call the API directly with curl
curl http://localhost:8001/api/v1/namespaces

Expected output (truncated):

{
  "kind": "NamespaceList",
  "apiVersion": "v1",
  "items": [
    { "metadata": { "name": "default" } },
    { "metadata": { "name": "kube-system" } }
  ]
}

# List pods in kube-system via raw API
curl http://localhost:8001/api/v1/namespaces/kube-system/pods | python3 -m json.tool | grep '"name"' | head -10

# Stop the proxy
kill %1

💡 What just happened? You bypassed kubectl entirely and spoke directly to the Kubernetes API Server. This is exactly what kubectl does under the hood — it formats HTTP requests and pretty-prints the JSON responses.

🔥 Break It! Challenge

Understand what happens when a control plane pod “disappears.”

# On minikube, static pods are managed by the kubelet from manifests
minikube ssh "ls /etc/kubernetes/manifests/"

Expected output:

etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml

These files define the control plane pods. If you delete one, the kubelet immediately recreates it. Try it:

# Move the scheduler manifest (simulating a crash)
minikube ssh "sudo mv /etc/kubernetes/manifests/kube-scheduler.yaml /tmp/"

# Watch the scheduler disappear
kubectl get pods -n kube-system -w

You’ll see kube-scheduler-minikube go Terminating. Now create a new pod — it’ll stay Pending:

kubectl run test-pod --image=nginx
kubectl get pod test-pod  # Status: Pending ← no scheduler!

Now restore it:

minikube ssh "sudo mv /tmp/kube-scheduler.yaml /etc/kubernetes/manifests/"
sleep 10
kubectl get pods -n kube-system  # Scheduler is back
kubectl get pod test-pod          # Now Running!
kubectl delete pod test-pod

What you learned: The Scheduler is needed for new pod placement. Existing pods are unaffected by its absence.

Cleanup

# Everything is clean — minikube cluster keeps running for future chapters
minikube status

What We Learned

Chapter 2: kubectl — Your Swiss Army Knife

⏱️ Total chapter time: ~45 min (20 min reading + 25 min lab)

After this chapter, you will be able to: Query, inspect, create, and delete any Kubernetes resource from the command line — and do it efficiently with output formatting and filtering.

What’s Inside

Prerequisites

• Completed Chapter 1 (Minikube cluster running)
• minikube status shows Running

2.1 Anatomy of a kubectl Command

⏱️ ~5 min read

TL;DR: Every kubectl command follows one pattern: kubectl [verb] [resource] [name] [flags]. Master this pattern and you can figure out any command on the fly.

The Command Structure

kubectl  <verb>    <resource>  <name>      <flags>
kubectl  get       pods        my-nginx    --namespace=default
kubectl  describe  node        minikube
kubectl  delete    deployment  my-app      --grace-period=0
kubectl  logs      pod/my-pod  -c sidecar  -f

That’s it. Once this pattern is in muscle memory, you don’t need to memorize hundreds of commands.

The Verbs

The verbs you’ll use 90% of the time:

The Resources

Resources are what K8s manages. You refer to them by their kind name (or shorthand):

# These are all equivalent
kubectl get deployments
kubectl get deployment
kubectl get deploy

# See ALL resource types in your cluster
kubectl api-resources

Essential Flags

Getting Help — Built In

You never need to Google basic syntax. kubectl has it all:

# Get help on any verb
kubectl get --help

# Get help on any resource type
kubectl explain pod
kubectl explain pod.spec
kubectl explain pod.spec.containers
kubectl explain pod.spec.containers.resources.limits

kubectl explain is like having the K8s API reference built into your terminal. Use it constantly.

Expected output for kubectl explain pod.spec.containers.resources.limits:

KIND:     Pod
VERSION:  v1

FIELD:    limits <map[string]Quantity>

DESCRIPTION:
     Limits describes the maximum amount of compute resources allowed. More info:
     https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

Try It

# Confirm your cluster is running
kubectl get nodes

# List all pods across all namespaces
kubectl get pods -A

# Get help for the 'exec' verb
kubectl exec --help

Key Takeaways

✅ Quick Check

Q1: What does kubectl get deploy -A do?

Q2: You want to see the detailed spec of a ConfigMap named app-config in the staging namespace. Which command?

Q3: You run kubectl delete pod my-pod but the pod comes back immediately. Why?

2.2 Imperative vs Declarative — Two Ways to Talk to K8s

⏱️ ~5 min read

TL;DR: Imperative = “do this now.” Declarative = “this is what I want.” Use imperative for quick experiments; always use declarative (kubectl apply -f) in real workflows.

Imperative: Give Orders

Imperative commands tell Kubernetes exactly what action to take. Fast and convenient for one-off tasks.

# Imperative: "Create an nginx pod, right now"
kubectl run my-nginx --image=nginx --port=80

# Imperative: "Scale this deployment to 5"
kubectl scale deployment my-app --replicas=5

# Imperative: "Create a ConfigMap with this value"
kubectl create configmap app-config --from-literal=DB_HOST=localhost

🔗 Docker Parallel: This is like docker run — you issue a command, something happens.

The problem: How do you track what was created? How do teammates reproduce it? How do you put it in Git?

Declarative: Describe Desired State

Declarative means writing a YAML file that describes what you want to exist, then applying it.

# nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.25
        ports:
        - containerPort: 80

kubectl apply -f nginx-deployment.yaml

🔗 Docker Parallel: This is like docker-compose.yml — you describe the desired state, and the tool figures out what actions to take.

apply is idempotent. Run it 10 times — it only makes changes when the file differs from the live state. This is safe to automate.

The Key Difference: create vs apply

# This will fail on second run:
kubectl create -f nginx-deployment.yaml
kubectl create -f nginx-deployment.yaml  # ❌ Error: already exists

# This is always safe:
kubectl apply -f nginx-deployment.yaml
kubectl apply -f nginx-deployment.yaml  # ✅ "unchanged"

Generating YAML from Imperative Commands

Here’s a power move: use imperative commands to generate YAML files, then commit those.

# Generate a Pod YAML without creating it
kubectl run my-pod --image=nginx --dry-run=client -o yaml

# Generate a Deployment YAML
kubectl create deployment my-app --image=nginx --replicas=3 \
  --dry-run=client -o yaml > my-deployment.yaml

# Generate a Service YAML
kubectl expose deployment my-app --port=80 --type=ClusterIP \
  --dry-run=client -o yaml > my-service.yaml

Expected output for the pod command:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: my-pod
  name: my-pod
spec:
  containers:
  - image: nginx
    name: my-pod
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

💡 Tip: --dry-run=client -o yaml is one of the most useful kubectl tricks. It generates a valid YAML scaffold that you edit and commit — way faster than writing YAML from scratch.

When to Use Which

🏭 In Production: Imperative commands have no audit trail. In production, every change should go through kubectl apply backed by a Git commit. This is the foundation of GitOps.

Try It

# Create a deployment imperatively
kubectl create deployment demo --image=nginx --replicas=2

# Now export it as YAML to see what K8s actually stored
kubectl get deployment demo -o yaml

# Delete it
kubectl delete deployment demo

# Now do the same thing declaratively
kubectl create deployment demo --image=nginx --replicas=2 \
  --dry-run=client -o yaml | kubectl apply -f -

# Verify
kubectl get deploy demo
kubectl delete deployment demo

Key Takeaways

✅ Quick Check

Q1: You run kubectl apply -f app.yaml in your CI pipeline. The resource already exists with different settings. What happens?

Q2: What does kubectl run test --image=busybox --dry-run=client -o yaml actually do?

Q3: A colleague used imperative commands to configure the production cluster. Why is this a problem?

2.3 Context, Namespaces, and kubeconfig

⏱️ ~5 min read

TL;DR: kubeconfig stores your cluster credentials. A context is a named combo of cluster + user + namespace. Namespaces isolate resources within a cluster. Knowing how to switch between them is essential once you have more than one cluster.

The kubeconfig File

When you ran minikube start, it automatically configured ~/.kube/config. This file tells kubectl which cluster to talk to and how to authenticate.

cat ~/.kube/config

Structure (simplified):

apiVersion: v1
kind: Config

clusters:
- name: minikube
  cluster:
    server: https://192.168.49.2:8443          # API Server URL
    certificate-authority: ~/.minikube/ca.crt  # TLS cert to trust

users:
- name: minikube
  user:
    client-certificate: ~/.minikube/profiles/minikube/client.crt
    client-key: ~/.minikube/profiles/minikube/client.key

contexts:
- name: minikube
  context:
    cluster: minikube    # which cluster
    user: minikube       # which credentials
    namespace: default   # default namespace

current-context: minikube   # ← which context is active RIGHT NOW

Three sections: clusters (where), users (who), contexts (which cluster + which user + which namespace).

Working with Contexts

A context is a named shortcut: “cluster X, authenticated as user Y, defaulting to namespace Z.”

# See all contexts
kubectl config get-contexts

# See the current context
kubectl config current-context

# Switch to a different context
kubectl config use-context my-other-cluster

# View the full kubeconfig
kubectl config view

Expected output for get-contexts:

CURRENT   NAME       CLUSTER    AUTHINFO   NAMESPACE
*         minikube   minikube   minikube   default

The * marks your active context. If you had an AKS cluster configured, it would appear here too.

⚠️ Warning: The classic ops horror story: running kubectl delete against production when you meant staging. Always verify your current context before destructive commands. Some teams alias kubectl to print the context on every command.

# Quick safety check — add this to your .bashrc or .zshrc
alias kctx='kubectl config current-context'

Namespaces — Isolation Within a Cluster

A namespace is a virtual partition within a single cluster. Resources in different namespaces are isolated from each other (mostly — Nodes and PersistentVolumes are cluster-wide).

graph TB
    subgraph "Kubernetes Cluster"
        subgraph "namespace: default"
            P1[web-pod]
            S1[web-svc]
        end
        subgraph "namespace: staging"
            P2[web-pod]
            S2[web-svc]
        end
        subgraph "namespace: monitoring"
            P3[prometheus-pod]
            P4[grafana-pod]
        end
        N1[Node 1] & N2[Node 2]
    end

The pods named web-pod in default and staging are completely separate objects — same name, different namespaces.

# List namespaces
kubectl get namespaces

# Create a namespace
kubectl create namespace staging

# Run a command in a specific namespace
kubectl get pods -n staging

# Set your default namespace (so you don't type -n every time)
kubectl config set-context --current --namespace=staging

# Reset to default
kubectl config set-context --current --namespace=default

Namespace DNS

Services in different namespaces can reach each other via DNS:

# Format: SERVICE-NAME.NAMESPACE.svc.cluster.local
# From any pod in the cluster:
curl http://web-svc.staging.svc.cluster.local
curl http://web-svc.default.svc.cluster.local

# Short form works within the same namespace:
curl http://web-svc

Managing Multiple Clusters

In the real world you’ll have multiple clusters (local minikube, staging, production). The workflow:

# Add a new cluster config (e.g., after AKS provisioning)
az aks get-credentials --resource-group my-rg --name my-aks-cluster

# Now you have two contexts
kubectl config get-contexts

# Switch between them
kubectl config use-context minikube
kubectl config use-context my-aks-cluster

💡 Tip: Install kubectx + kubens — tiny tools that make switching contexts and namespaces much faster:

kubectx minikube          # switch cluster context
kubens staging            # switch namespace

They also highlight the current context in your shell prompt.

Try It

# Create two namespaces
kubectl create namespace dev
kubectl create namespace staging

# Deploy an nginx pod to each
kubectl run nginx-dev --image=nginx -n dev
kubectl run nginx-staging --image=nginx -n staging

# See both — notice same name, different namespace
kubectl get pods -A | grep nginx

# Clean up
kubectl delete namespace dev staging

Expected output:

dev         nginx-dev       1/1     Running   0   30s
staging     nginx-staging   1/1     Running   0   28s

Key Takeaways

✅ Quick Check

Q1: You accidentally ran a command against production instead of staging. How could you prevent this in the future?

Q2: A pod in namespace frontend needs to call a service named api in namespace backend. What URL should it use?

Q3: You set kubectl config set-context --current --namespace=staging. What changes in your kubeconfig?

2.4 Output Formatting, Filtering, and JSONPath

⏱️ ~6 min read

TL;DR: kubectl get is not just for listing things — with the right flags it’s a precision query tool. -o wide, -o yaml, -o jsonpath, and -l (label selectors) are how senior engineers extract exactly the information they need.

Output Formats

# Default tabular output
kubectl get pods

# More columns (node, IP)
kubectl get pods -o wide

# Full YAML — what's actually stored in etcd
kubectl get pod my-pod -o yaml

# JSON format
kubectl get pod my-pod -o json

# Just the names — great for piping
kubectl get pods -o name

The -o yaml trick is critical. When you want to understand what fields a resource has, or when you need to copy an existing resource and modify it:

# Export a live resource as editable YAML
kubectl get deployment my-app -o yaml > my-app-copy.yaml
# Edit the file, change the name, apply it
kubectl apply -f my-app-copy.yaml

Label Selectors — Filtering by Label

Labels are key-value pairs attached to resources. They’re how K8s knows which pods belong to which Service or Deployment.

# List pods with a specific label
kubectl get pods -l app=nginx

# Multiple labels (AND)
kubectl get pods -l app=nginx,env=production

# All resources with a label, across types
kubectl get all -l app=my-app

# Show labels in output
kubectl get pods --show-labels

graph LR
    subgraph "All Pods"
        P1["web-1\napp=nginx\nenv=prod"]
        P2["web-2\napp=nginx\nenv=prod"]
        P3["api-1\napp=api\nenv=prod"]
        P4["web-dev\napp=nginx\nenv=dev"]
    end

    Q["kubectl get pods\n-l app=nginx,env=prod"] -->|matches| P1 & P2

🔗 Docker Parallel: Docker Compose has labels: on services too, but they’re mostly cosmetic. In Kubernetes, labels are fundamental — Services and Deployments use label selectors to know which pods they own.

Field Selectors — Filtering by Status Fields

# Only Running pods
kubectl get pods --field-selector=status.phase=Running

# Pods on a specific node
kubectl get pods --field-selector=spec.nodeName=minikube

# Failed pods across all namespaces
kubectl get pods -A --field-selector=status.phase=Failed

JSONPath — Surgical Extraction

When you need one specific field from a resource, JSONPath is the tool.

# Get just the node's IP address
kubectl get node minikube -o jsonpath='{.status.addresses[0].address}'

# Get all pod names
kubectl get pods -o jsonpath='{.items[*].metadata.name}'

# Get image names for every container in every pod
kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}'

# Get the ClusterIP of a service
kubectl get svc my-svc -o jsonpath='{.spec.clusterIP}'

Expected output for pod names:

my-nginx-abc12   my-api-def34   redis-xyz56

💡 Tip: Not sure of the JSONPath for what you want? Start with -o json to see the full structure, then build your JSONPath from there.

Custom Columns — Table Output You Define

# Custom table: name, image, node
kubectl get pods -o custom-columns=\
'NAME:.metadata.name,IMAGE:.spec.containers[0].image,NODE:.spec.nodeName'

Expected output:

NAME             IMAGE          NODE
my-nginx-abc12   nginx:1.25     minikube
my-api-def34     node:20-slim   minikube

Sorting Output

# Sort pods by creation time (newest last)
kubectl get pods --sort-by=.metadata.creationTimestamp

# Sort by restart count (most restarted first — useful for debugging)
kubectl get pods --sort-by='.status.containerStatuses[0].restartCount'

# Sort nodes by CPU capacity
kubectl get nodes --sort-by='.status.capacity.cpu'

Combining It All — Real-World Recipes

# "Which pods are running on which nodes, sorted by node?"
kubectl get pods -o wide --sort-by='.spec.nodeName'

# "What images are my deployments using?"
kubectl get deployments -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.template.spec.containers[0].image}{"\n"}{end}'

# "Find all pods NOT in Running state"
kubectl get pods -A --field-selector='status.phase!=Running'

# "Get the NodePort for a service"
kubectl get svc my-svc -o jsonpath='{.spec.ports[0].nodePort}'

Try It

# Create a test deployment
kubectl create deployment fmt-demo --image=nginx:1.25 --replicas=2

# Try each format
kubectl get pods -l app=fmt-demo -o wide
kubectl get pods -l app=fmt-demo -o yaml | grep -A3 "image:"
kubectl get pods -l app=fmt-demo -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'

# Custom columns
kubectl get pods -l app=fmt-demo \
  -o custom-columns='POD:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName'

# Cleanup
kubectl delete deployment fmt-demo

Key Takeaways

✅ Quick Check

Q1: You want a list of all pod names and their container images in a single line per pod. Which output format do you use?

Q2: A Service’s label selector is app=backend. You have pods labeled app=backend-v2. Does the Service route to them?

Q3: You need to find all pods that are NOT in the Running phase across all namespaces. What command?

Lab: kubectl Power User Drills

⏱️ ~25 min hands-on

Objectives

• Use kubectl explain to look up API fields without Googling
• Generate YAML using --dry-run=client and edit it
• Filter pods by label, field, and namespace
• Extract specific data with JSONPath and custom columns
• Switch and manage namespaces confidently
• Debug a misconfigured deployment

Setup

# Verify cluster is running
kubectl get nodes

# Create a dedicated namespace for this lab
kubectl create namespace drill-lab

# Switch to it
kubectl config set-context --current --namespace=drill-lab

# Verify
kubectl config view --minify | grep namespace

Expected output:

    namespace: drill-lab

Exercise 1: Use kubectl explain as Your Dictionary

What we’re doing: Look up API field definitions without leaving the terminal.

# What fields does a Pod spec have?
kubectl explain pod.spec

# What are the container resource fields?
kubectl explain pod.spec.containers.resources

# What does restartPolicy accept?
kubectl explain pod.spec.restartPolicy

Challenge: Find out what pod.spec.containers.livenessProbe.httpGet fields are available — using only kubectl explain.

kubectl explain pod.spec.containers.livenessProbe.httpGet

Expected output (key fields):

FIELDS:
  host     <string>
  httpHeaders  <[]Object>
  path     <string>     -required-
  port     <IntOrString> -required-
  scheme   <string>

💡 What just happened? You just looked up the Kubernetes API reference without opening a browser. Make this a habit.

Exercise 2: Generate YAML Without Writing It From Scratch

What we’re doing: Use imperative commands + --dry-run=client -o yaml to scaffold manifests.

# Generate a Deployment manifest
kubectl create deployment webserver \
  --image=nginx:1.25 \
  --replicas=3 \
  --port=80 \
  --dry-run=client -o yaml > /tmp/webserver-deployment.yaml

cat /tmp/webserver-deployment.yaml

Now edit it — add a label to the pod template:

# Open the file and add env=lab under the existing labels in spec.template.metadata.labels
# Use any editor: nano, vim, or:
sed -i '/app: webserver/a\        env: lab' /tmp/webserver-deployment.yaml

Verify your edit:

grep -A5 "labels:" /tmp/webserver-deployment.yaml | head -20

Apply it:

kubectl apply -f /tmp/webserver-deployment.yaml
kubectl get deployment webserver
kubectl get pods --show-labels

Expected output:

NAME              READY   STATUS    RESTARTS   AGE   LABELS
webserver-xxxxx   1/1     Running   0          10s   app=webserver,env=lab,pod-template-hash=xxxxx
webserver-xxxxx   1/1     Running   0          10s   app=webserver,env=lab,pod-template-hash=xxxxx
webserver-xxxxx   1/1     Running   0          10s   app=webserver,env=lab,pod-template-hash=xxxxx

Exercise 3: Deploy Multiple Apps with Different Labels

What we’re doing: Set up a multi-app environment to practice filtering.

# Deploy a second app (simulating an API service)
kubectl create deployment api-server \
  --image=node:20-slim \
  --replicas=2 \
  -n drill-lab

# Label the api-server pods differently
kubectl label deployment api-server tier=backend env=lab

# Create a third deployment
kubectl create deployment redis-cache \
  --image=redis:7 \
  --replicas=1 \
  -n drill-lab

kubectl label deployment redis-cache tier=cache env=lab

Now practice filtering:

# All pods
kubectl get pods

# Only webserver pods
kubectl get pods -l app=webserver

# All pods with env=lab
kubectl get pods -l env=lab

# All pods with env=lab AND tier=backend
kubectl get pods -l env=lab,tier=backend

# Show labels in output
kubectl get pods --show-labels

# Non-running pods
kubectl get pods --field-selector=status.phase!=Running

Exercise 4: JSONPath Drills

What we’re doing: Extract specific fields like a surgeon.

# Get all pod names, one per line
kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'

# Get pod name + image, tab-separated
kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}'

# Get the IP of the first webserver pod
kubectl get pods -l app=webserver -o jsonpath='{.items[0].status.podIP}'

# Get restart counts for all pods
kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].restartCount}{"\n"}{end}'

Custom columns version — same info, prettier output:

kubectl get pods \
  -o custom-columns='NAME:.metadata.name,IMAGE:.spec.containers[0].image,IP:.status.podIP,RESTARTS:.status.containerStatuses[0].restartCount'

Expected output:

NAME                    IMAGE        IP            RESTARTS
webserver-abc-xxx       nginx:1.25   10.244.0.5    0
webserver-abc-yyy       nginx:1.25   10.244.0.6    0
api-server-def-xxx      node:20-slim 10.244.0.7    0
redis-cache-ghi-xxx     redis:7      10.244.0.8    0

Exercise 5: Namespace Operations

What we’re doing: Work across namespaces.

# Create another namespace with a pod
kubectl create namespace other-lab
kubectl run isolated-pod --image=nginx -n other-lab

# Prove the namespaces are isolated
kubectl get pods                    # drill-lab pods only
kubectl get pods -n other-lab       # other-lab pods only
kubectl get pods -A                 # ALL pods, all namespaces

# Cross-namespace view with labels
kubectl get pods -A --show-labels

# Clean up other-lab
kubectl delete namespace other-lab

Exercise 6: Watch Mode + Sorting

# In one terminal, watch pods:
kubectl get pods -w &

# In the same terminal, scale up/down and watch the changes:
kubectl scale deployment webserver --replicas=5
sleep 3
kubectl scale deployment webserver --replicas=2

# Stop the watch
kill %1

# Sort all pods by restart count
kubectl get pods --sort-by='.status.containerStatuses[0].restartCount'

# Sort by creation time
kubectl get pods --sort-by=.metadata.creationTimestamp

Exercise 7: Debug a Broken Deployment

What we’re doing: Apply a broken deployment and diagnose it using only kubectl.

# Apply this broken deployment (bad image name)
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: broken-app
  namespace: drill-lab
spec:
  replicas: 2
  selector:
    matchLabels:
      app: broken-app
  template:
    metadata:
      labels:
        app: broken-app
    spec:
      containers:
      - name: app
        image: nginx:doesnotexist999
        ports:
        - containerPort: 80
EOF

Now diagnose:

# 1. See the deployment status
kubectl get deployment broken-app

# 2. See the pods
kubectl get pods -l app=broken-app

# 3. Describe a failing pod — look at Events at the bottom
kubectl describe pod -l app=broken-app

# 4. Check the specific error
kubectl get pods -l app=broken-app -o jsonpath='{range .items[*]}{.status.containerStatuses[0].state.waiting.reason}{"\n"}{end}'

Expected output from describe (Events section):

Events:
  Type     Reason     Message
  ----     ------     -------
  Warning  Failed     Failed to pull image "nginx:doesnotexist999": ... not found
  Warning  Failed     Error: ErrImagePull
  Warning  BackOff    Back-off pulling image "nginx:doesnotexist999"

Fix it:

kubectl set image deployment/broken-app app=nginx:1.25
kubectl rollout status deployment/broken-app

💡 What just happened? kubectl set image is an imperative way to update a container image. The deployment controller immediately starts a rolling update. In production, you’d update the YAML and kubectl apply.

🔥 Break It! Challenge

What happens if you delete a namespace that has running workloads?

# Watch what happens to all resources in drill-lab if we delete it
# (Don't do this yet — just understand the outcome)

# First: list everything in the namespace
kubectl get all -n drill-lab

# The answer: deleting a namespace cascades to ALL resources in it
# kubectl delete namespace drill-lab  ← this would kill everything

# Instead, try deleting just the api-server deployment and watch
# the pods disappear but the other deployments stay:
kubectl delete deployment api-server
kubectl get pods -w   # watch pods go away, then Ctrl+C

The lesson: Namespace deletion is nuclear — it kills everything inside it with no confirmation. Never delete namespaces in production without knowing exactly what’s inside.

Cleanup

# Switch back to default namespace first
kubectl config set-context --current --namespace=default

# Delete the lab namespace and everything in it
kubectl delete namespace drill-lab

# Verify cleanup
kubectl get all -n drill-lab 2>&1  # Should say "No resources found"

What We Learned

Chapter 3: Pods — The Atomic Unit

⏱️ Total chapter time: ~55 min (25 min reading + 30 min lab)

After this chapter, you will be able to: Create pods declaratively, understand every phase of the pod lifecycle, build multi-container pods, set resource limits, and debug failures like CrashLoopBackOff and OOMKilled.

What’s Inside

Prerequisites

• Completed Chapter 2 (kubectl confident)
• minikube status shows Running

3.1 What Is a Pod, Really?

⏱️ ~5 min read

TL;DR: A Pod is NOT a container. It’s a wrapper around one or more containers that share the same network namespace and can share storage volumes. It’s the smallest deployable unit in Kubernetes.

The Key Insight: Pods Are Not Containers

🔗 Docker Parallel: In Docker, you deploy containers. In Kubernetes, you deploy Pods. A Pod can contain one container (usually) or multiple tightly-coupled containers that need to share a network stack.

Here’s the minimal pod definition:

# my-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: my-nginx
  labels:
    app: nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.25
    ports:
    - containerPort: 80

kubectl apply -f my-pod.yaml
kubectl get pod my-nginx

What Containers in a Pod Share

This is what makes pods special. Every container in a pod shares:

graph TB
    subgraph "Pod"
        NET[Shared Network Namespace\nSame IP address\nSame localhost]
        subgraph "Container A\nnginx"
            PA[:80]
        end
        subgraph "Container B\nlog-shipper"
            PB[:9000]
        end
        VOL[Shared Volume\n/var/log]
        PA --- NET
        PB --- NET
        PA --- VOL
        PB --- VOL
    end

⚠️ Warning: Because containers in a pod share a network namespace, two containers cannot both listen on port 80. Plan your container ports carefully in multi-container pods.

Why Not Just Run Containers Directly?

Valid question. The answer is: pods add a layer of abstraction that gives Kubernetes useful guarantees.

1. Atomic scheduling — all containers in a pod land on the same node together
2. Shared lifecycle — pod-level restart policies apply to all containers
3. Co-location guarantee — sidecar and main container are always on the same machine, eliminating network latency between them

📝 Note: In practice, 90% of pods contain exactly one container. Multi-container pods are for specific patterns (sidecar, init) covered in section 3.3.

The Pod YAML Anatomy

apiVersion: v1           # API group version — always v1 for Pods
kind: Pod                # Resource type
metadata:
  name: my-app           # Unique name within a namespace
  namespace: default     # Namespace (defaults to "default")
  labels:                # Key-value pairs for selecting/grouping
    app: my-app
    version: "1.0"
  annotations:           # Non-identifying metadata (not used for selection)
    description: "Demo pod"
spec:
  containers:            # List of containers (at least one required)
  - name: app            # Container name (unique within the pod)
    image: nginx:1.25    # Image:tag — always pin the tag!
    ports:
    - containerPort: 80  # Informational only — doesn't actually publish the port
    env:                 # Environment variables
    - name: ENV
      value: production
    resources:           # CPU/memory limits (covered in 3.4)
      requests:
        memory: "64Mi"
        cpu: "100m"
      limits:
        memory: "128Mi"
        cpu: "200m"
  restartPolicy: Always  # Always | OnFailure | Never

⚠️ Warning: containerPort in a pod spec is purely informational — it doesn’t actually open a firewall port or publish the container. Networking is handled by Services (Chapter 5).

Try It

# Apply the pod
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: my-nginx
  labels:
    app: nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.25
    ports:
    - containerPort: 80
EOF

# Watch it start
kubectl get pod my-nginx -w

# Once Running — get full details
kubectl describe pod my-nginx

# Access it directly (no Service needed for testing)
kubectl port-forward pod/my-nginx 8080:80 &
curl http://localhost:8080
kill %1

# Cleanup
kubectl delete pod my-nginx

Key Takeaways

✅ Quick Check

Q1: You have two containers in a pod. Container A listens on port 8080. Can Container B also listen on port 8080?

Q2: Container A in a pod writes a file to /tmp/data. Can Container B read it?

Q3: You delete a pod. Does Kubernetes immediately create a new one?

3.2 Pod Lifecycle and Phases

⏱️ ~6 min read

TL;DR: A pod moves through phases: Pending → Running → Succeeded/Failed. Each container inside has its own state too. Understanding both levels is how you debug 80% of pod problems.

Pod Phases

stateDiagram-v2
    [*] --> Pending : Pod accepted by API Server
    Pending --> Running : All containers started
    Running --> Succeeded : All containers exited 0
    Running --> Failed : At least one container exited non-zero
    Pending --> Failed : Image pull error / unschedulable
    Running --> Unknown : Node lost contact with control plane

# Check phase
kubectl get pod my-pod -o jsonpath='{.status.phase}'

Container States

Inside a running pod, each container has its own state — independent of the pod phase:

# Get container state details
kubectl get pod my-pod -o jsonpath='{.status.containerStatuses[0].state}'

Pod Conditions

Conditions give more granular status than phase. They’re how kubectl determines what to show in kubectl describe:

kubectl get pod my-pod -o jsonpath='{.status.conditions}' | python3 -m json.tool

The Critical Failure Modes

CrashLoopBackOff

The container keeps crashing. K8s restarts it, but adds an exponential back-off delay (10s, 20s, 40s… up to 5 min).

# Cause: check the logs from the PREVIOUS run
kubectl logs my-pod --previous

# Or describe to see restart count and last exit code
kubectl describe pod my-pod | grep -A5 "Last State:"

ImagePullBackOff / ErrImagePull

Kubernetes can’t pull the container image. Causes:

• Wrong image name / tag
• Private registry without credentials
• No internet connectivity on the node

# See the exact error
kubectl describe pod my-pod | grep -A5 "Events:"

OOMKilled

The container exceeded its memory limit and was killed by the kernel.

# Check exit code — OOMKilled shows exitCode: 137
kubectl get pod my-pod -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'

Pending (stuck)

Pod can’t be scheduled. Causes:

• Not enough CPU/memory on any node
• Node selector or affinity rules don’t match any node
• Taint not tolerated

# See why it's stuck
kubectl describe pod my-pod | grep -A10 "Events:"

Restart Policies

restartPolicy controls what happens when a container exits:

spec:
  restartPolicy: OnFailure   # for Jobs
  containers:
  - name: worker
    image: my-batch-job

🔗 Docker Parallel: This is equivalent to restart: unless-stopped (Always), restart: on-failure (OnFailure), and no restart policy (Never) in Docker Compose.

Try It

# Create a pod that immediately crashes
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: crasher
spec:
  containers:
  - name: app
    image: busybox
    command: ["sh", "-c", "echo 'Starting...' && exit 1"]
  restartPolicy: Always
EOF

# Watch it crash and enter CrashLoopBackOff
kubectl get pod crasher -w

# See the logs from the crashed run
kubectl logs crasher
kubectl logs crasher --previous   # After it's restarted at least once

# See restart count and exit code
kubectl describe pod crasher | grep -A8 "Containers:"

# Cleanup
kubectl delete pod crasher

Expected progression:

NAME      READY   STATUS              RESTARTS   AGE
crasher   0/1     ContainerCreating   0          2s
crasher   0/1     Error               0          3s
crasher   0/1     CrashLoopBackOff    1          8s
crasher   0/1     Error               2          28s
crasher   0/1     CrashLoopBackOff    3          42s

Key Takeaways

✅ Quick Check

Q1: A pod is in Running phase but 0/1 READY. What does this mean?

Q2: Your pod has RESTARTS: 47 and status CrashLoopBackOff. What’s your first debugging step?

Q3: What’s the difference between a pod in Failed phase vs CrashLoopBackOff status?

3.3 Multi-Container Pods: Sidecars, Init, and Ambassadors

⏱️ ~7 min read

TL;DR: Multi-container pods solve specific co-location problems. The three patterns are: sidecar (augments the main container), init container (runs setup before the main container), and ambassador (proxies external traffic). Show the YAML first, understand the pattern second.

Pattern 1: Sidecar

A sidecar runs alongside the main container, sharing its network and storage. The most common use case: log shipping.

# sidecar-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: app-with-logger
spec:
  volumes:
  - name: shared-logs      # Shared volume between containers
    emptyDir: {}

  containers:
  # Main application
  - name: app
    image: nginx:1.25
    volumeMounts:
    - name: shared-logs
      mountPath: /var/log/nginx

  # Sidecar: ships logs from shared volume
  - name: log-shipper
    image: busybox
    command: ["sh", "-c", "tail -f /logs/access.log"]
    volumeMounts:
    - name: shared-logs
      mountPath: /logs         # Same volume, different mount path

graph LR
    subgraph "Pod: app-with-logger"
        APP[nginx\nwrites to\n/var/log/nginx]
        VOL[(emptyDir\nshared-logs)]
        LOG[log-shipper\nreads from\n/logs]
        APP -->|writes| VOL
        VOL -->|reads| LOG
        LOG -->|ships to| EXT[External\nLog System]
    end

Key facts about emptyDir:

• Created when the pod starts, deleted when the pod dies
• Lives on the node’s disk (or RAM if medium: Memory)
• Perfect for sharing data between containers in a pod

🔗 Docker Parallel: No direct equivalent in Compose. You’d need to set up a log driver or bind mount. The sidecar pattern keeps the main app container simple and logging as a separate concern.

Pattern 2: Init Containers

Init containers run sequentially before any regular containers start. They must complete successfully. If they fail, the pod restarts.

# init-container-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: app-with-init
spec:
  initContainers:
  # Runs FIRST — waits for a dependency to be ready
  - name: wait-for-db
    image: busybox
    command:
    - sh
    - -c
    - |
      until nc -z postgres-svc 5432; do
        echo "Waiting for postgres..."
        sleep 2
      done
      echo "Postgres is up!"

  # Runs SECOND — database migration
  - name: run-migrations
    image: myapp:latest
    command: ["python", "manage.py", "migrate"]
    env:
    - name: DATABASE_URL
      value: "postgresql://postgres-svc:5432/mydb"

  # Only starts AFTER all init containers succeed
  containers:
  - name: app
    image: myapp:latest
    ports:
    - containerPort: 8000

sequenceDiagram
    participant K as Kubernetes
    participant I1 as init: wait-for-db
    participant I2 as init: run-migrations
    participant A as app container

    K->>I1: Start
    I1-->>K: ✅ Exit 0
    K->>I2: Start
    I2-->>K: ✅ Exit 0
    K->>A: Start (finally!)
    A-->>K: Running

Why use init containers instead of a startup script in the main container?

• Separation of concerns — the app image stays clean
• Different image per init step (wait with busybox, migrate with your app)
• Init containers can have different security contexts
• Failures are clearly visible: kubectl get pod shows Init:0/2 then Init:1/2 etc.

# See init container progress
kubectl get pod app-with-init
# Output: Init:1/2  ← first init done, waiting on second

Pattern 3: Ambassador

An ambassador container proxies connections between the main container and external services — useful for handling TLS, service discovery, or protocol translation locally.

# ambassador-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: app-with-ambassador
spec:
  containers:
  # Main app talks to localhost:6379 (simple, no auth)
  - name: app
    image: myapp:latest
    env:
    - name: REDIS_HOST
      value: localhost      # ← talks to ambassador, not Redis directly
    - name: REDIS_PORT
      value: "6379"

  # Ambassador: handles auth and TLS to real Redis
  - name: redis-proxy
    image: envoyproxy/envoy:v1.29
    # Envoy configured to proxy localhost:6379 → redis-cloud.example.com:6380 (TLS+auth)

📝 Note: The ambassador pattern is less common today — service meshes (Istio, Linkerd) have largely replaced it for production TLS/auth proxying. But it’s still useful for local development and specific edge cases.

Checking Multi-Container Pods

With multiple containers, you must specify which one you’re targeting:

# Logs from a specific container
kubectl logs app-with-logger -c log-shipper
kubectl logs app-with-logger -c app

# Exec into a specific container
kubectl exec -it app-with-logger -c app -- bash
kubectl exec -it app-with-logger -c log-shipper -- sh

# Describe shows all containers
kubectl describe pod app-with-logger

Try It

# Deploy the sidecar pod
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: sidecar-demo
spec:
  volumes:
  - name: shared-data
    emptyDir: {}
  containers:
  - name: writer
    image: busybox
    command: ["sh", "-c", "while true; do date >> /data/log.txt; sleep 2; done"]
    volumeMounts:
    - name: shared-data
      mountPath: /data
  - name: reader
    image: busybox
    command: ["sh", "-c", "tail -f /shared/log.txt"]
    volumeMounts:
    - name: shared-data
      mountPath: /shared
EOF

# Watch both containers start
kubectl get pod sidecar-demo -w

# See the writer's output via reader's logs
kubectl logs sidecar-demo -c reader -f &

# Let it run for 10 seconds, then stop
sleep 10 && kill %1

# Exec into the writer and check the file directly
kubectl exec -it sidecar-demo -c writer -- cat /data/log.txt

# Cleanup
kubectl delete pod sidecar-demo

Expected log output (reader):

Mon Jul 14 10:00:01 UTC 2026
Mon Jul 14 10:00:03 UTC 2026
Mon Jul 14 10:00:05 UTC 2026

Key Takeaways

✅ Quick Check

Q1: Your app container takes 60 seconds to initialize and fails readiness checks during that time. Should you use an init container or a startup probe?

Q2: An init container in a pod exits with code 1. What happens?

Q3: You have a sidecar container that ships logs to an external service. The main app container crashes and restarts. What happens to the sidecar?

3.4 Resource Requests and Limits

⏱️ ~6 min read

TL;DR: requests = “I need at least this much” (used for scheduling). limits = “I cannot exceed this much” (enforced at runtime). Getting these right is the difference between a stable cluster and one that OOMKills pods in the middle of the night.

The Two Numbers: Requests vs Limits

resources:
  requests:
    memory: "128Mi"   # Scheduler guarantee: only place me on a node with 128Mi free
    cpu: "250m"       # Scheduler guarantee: only place me on a node with 250m CPU free
  limits:
    memory: "256Mi"   # Hard limit: if I exceed this, I get OOMKilled
    cpu: "500m"       # Soft limit: if I exceed this, I get throttled (not killed)

The difference in behavior between CPU and memory limits is critical:

graph LR
    subgraph "CPU Exceeded"
        A[Container wants 600m CPU] --> B[Limit: 500m]
        B --> C[Throttled\nSlowed down\nStill running ✅]
    end
    subgraph "Memory Exceeded"
        D[Container uses 300Mi RAM] --> E[Limit: 256Mi]
        E --> F[OOMKilled\nTerminated immediately ❌\nExit code 137]
    end

CPU Units: Millicores

CPU is measured in millicores (m). 1000m = 1 CPU core.

250m  = ¼ of a CPU core
500m  = ½ of a CPU core
1000m = 1 full CPU core  (same as writing "1")
2000m = 2 CPU cores      (same as writing "2")

# These are equivalent:
cpu: "1"
cpu: "1000m"

💡 Tip: For most web services, start with requests: 100m and limits: 500m. Adjust based on profiling. Node.js and Python are rarely CPU-bound; JVM apps need more.

Memory Units

Memory uses standard binary suffixes:

64Mi   = 64 mebibytes (~67 MB)
128Mi  = 128 mebibytes (~134 MB)
256Mi  = 256 mebibytes
1Gi    = 1 gibibyte (~1.07 GB)

⚠️ Warning: M (megabytes) and Mi (mebibytes) are different. Use Mi to avoid confusion. K8s accepts both but they differ by ~5%.

QoS Classes: What Kubernetes Does Under Pressure

Based on how you set requests/limits, Kubernetes assigns each pod a Quality of Service class. This determines which pods get evicted first when a node runs low on memory.

# Guaranteed QoS — for critical workloads
resources:
  requests:
    memory: "256Mi"
    cpu: "500m"
  limits:
    memory: "256Mi"   # ← same as requests
    cpu: "500m"       # ← same as requests

# BestEffort QoS — avoid in production
# (no resources block at all)

🏭 In Production: Always set resource requests for every container. Without them, the scheduler makes uninformed decisions and nodes can get overloaded. BestEffort pods are the first to die when the node gets memory pressure.

LimitRange: Cluster-Wide Defaults

Admins can set default limits per namespace so developers don’t have to:

# limitrange.yaml
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: default
spec:
  limits:
  - type: Container
    default:            # Applied if no limits set
      memory: "256Mi"
      cpu: "500m"
    defaultRequest:     # Applied if no requests set
      memory: "128Mi"
      cpu: "100m"
    max:                # Nobody in this namespace can set higher than this
      memory: "2Gi"
      cpu: "2"

Try It

# Deploy a pod with resource limits
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: resource-demo
spec:
  containers:
  - name: app
    image: nginx:1.25
    resources:
      requests:
        memory: "64Mi"
        cpu: "100m"
      limits:
        memory: "128Mi"
        cpu: "200m"
EOF

# See its QoS class
kubectl get pod resource-demo -o jsonpath='{.status.qosClass}'

# See resource requests on the node
kubectl describe node minikube | grep -A6 "Allocated resources:"

# Cleanup
kubectl delete pod resource-demo

Expected QoS output:

Burstable

(Because requests < limits. Set them equal for Guaranteed.)

Key Takeaways

✅ Quick Check

Q1: A container with limits.cpu: 500m suddenly needs 800m CPU to handle a traffic spike. What happens?

Q2: You set requests.memory: 128Mi and limits.memory: 128Mi. What QoS class does this pod get?

Q3: Your node has 4Gi of RAM. You have 40 pods each with requests.memory: 128Mi and no limits set. A burst of traffic causes one pod to use 2Gi. What happens?

Lab: Run, Inspect, Break, and Debug Pods

⏱️ ~30 min hands-on

Objectives

• Create and inspect pods using both methods
• Use kubectl exec to run commands inside a container
• Read logs including from previous (crashed) containers
• Trigger and diagnose an OOMKilled container
• Debug a pod stuck in CrashLoopBackOff
• Use an init container to gate a pod’s startup

Setup

# Create a dedicated namespace
kubectl create namespace pod-lab
kubectl config set-context --current --namespace=pod-lab

Exercise 1: Create and Inspect a Pod

What we’re doing: Create a pod both ways and compare the experience.

# Method A: Imperative
kubectl run nginx-imperative --image=nginx:1.25

# Method B: Declarative
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: nginx-declarative
  namespace: pod-lab
  labels:
    method: declarative
    app: nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.25
    ports:
    - containerPort: 80
    resources:
      requests:
        memory: "32Mi"
        cpu: "50m"
      limits:
        memory: "64Mi"
        cpu: "100m"
EOF

# Watch both come up
kubectl get pods -w

Expected output:

NAME                READY   STATUS              RESTARTS   AGE
nginx-declarative   0/1     ContainerCreating   0          2s
nginx-imperative    0/1     ContainerCreating   0          5s
nginx-declarative   1/1     Running             0          4s
nginx-imperative    1/1     Running             0          7s

Now deep-inspect:

# Full YAML of what's actually stored in K8s
kubectl get pod nginx-declarative -o yaml

# Human-readable summary with events
kubectl describe pod nginx-declarative

Look for these in describe output:

• QoS Class: — should be Burstable
• Node: — which minikube node
• IP: — the pod’s internal IP
• Events: — the pull/start sequence

Exercise 2: Exec and Port-Forward

What we’re doing: Get inside a running container and access it from your laptop.

# Get a shell inside the nginx container
kubectl exec -it nginx-declarative -- bash

# Inside the container — explore
hostname          # Pod name
cat /etc/hostname # Same
env | grep KUBERNETES  # K8s injects these env vars
curl localhost    # Access nginx from inside
exit

# Port-forward to access from your host
kubectl port-forward pod/nginx-declarative 8080:80 &
curl http://localhost:8080
echo "Response received!"
kill %1

Expected curl output:

<!DOCTYPE html>
<html>
<head><title>Welcome to nginx!</title></head>
...

💡 What just happened? kubectl exec opens a shell inside the running container’s filesystem and process space. port-forward tunnels traffic from your laptop’s port 8080 through the kubectl proxy to the pod’s port 80.

Exercise 3: Logs and Log Tailing

# View nginx access logs
kubectl logs nginx-declarative

# Follow logs in real time
kubectl logs nginx-declarative -f &

# In another command, generate some traffic
kubectl port-forward pod/nginx-declarative 8080:80 &
for i in {1..5}; do curl -s http://localhost:8080 > /dev/null; done

# You should see log entries appear in the -f stream
kill %1 %2

# Logs since a timestamp
kubectl logs nginx-declarative --since=5m

# Last N lines
kubectl logs nginx-declarative --tail=20

Exercise 4: Trigger and Diagnose CrashLoopBackOff

What we’re doing: Intentionally create a crashing pod and walk through the full debugging flow.

# Pod that exits immediately with an error message
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: crashloop-demo
  namespace: pod-lab
spec:
  containers:
  - name: app
    image: busybox
    command: ["sh", "-c", "echo 'ERROR: Database connection refused at postgres:5432'; exit 1"]
  restartPolicy: Always
EOF

Watch the CrashLoopBackOff progression:

kubectl get pod crashloop-demo -w

Expected output:

NAME              READY   STATUS              RESTARTS   AGE
crashloop-demo    0/1     ContainerCreating   0          1s
crashloop-demo    0/1     Error               0          3s
crashloop-demo    0/1     CrashLoopBackOff    1          8s
crashloop-demo    0/1     Error               2          28s
crashloop-demo    0/1     CrashLoopBackOff    3          52s

Now debug:

# Step 1: Get the error from logs (current run — might be empty after restart)
kubectl logs crashloop-demo

# Step 2: Get logs from the PREVIOUS crash
kubectl logs crashloop-demo --previous

# Step 3: See restart count and last exit code
kubectl describe pod crashloop-demo | grep -A15 "Containers:"

# Step 4: Check via JSONPath
kubectl get pod crashloop-demo -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'

Expected output from --previous logs:

ERROR: Database connection refused at postgres:5432

Key fields from describe:

    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 14 Jul 2026 10:00:05 +0000
      Finished:     Mon, 14 Jul 2026 10:00:05 +0000
    Ready:          False
    Restart Count:  4

# Cleanup
kubectl delete pod crashloop-demo

Exercise 5: Trigger an OOMKill

What we’re doing: Set a very low memory limit and watch the container get killed by the kernel.

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: oomkill-demo
  namespace: pod-lab
spec:
  containers:
  - name: memory-hog
    image: polinux/stress
    command: ["stress"]
    args: ["--vm", "1", "--vm-bytes", "150M", "--vm-hang", "1"]
    resources:
      requests:
        memory: "50Mi"
      limits:
        memory: "100Mi"   # Container will try to use 150M but limit is 100M
  restartPolicy: Never
EOF

Watch it get OOMKilled:

kubectl get pod oomkill-demo -w

Expected output:

NAME           READY   STATUS      RESTARTS   AGE
oomkill-demo   0/1     Pending     0          1s
oomkill-demo   1/1     Running     0          3s
oomkill-demo   0/1     OOMKilled   0          5s
oomkill-demo   0/1     Completed   0          5s

Diagnose it:

# Confirm OOMKill via exit code (137 = killed by signal 9)
kubectl get pod oomkill-demo -o jsonpath='{.status.containerStatuses[0].state.terminated.reason}'
# Output: OOMKilled

kubectl get pod oomkill-demo -o jsonpath='{.status.containerStatuses[0].state.terminated.exitCode}'
# Output: 137

# Full terminated state
kubectl describe pod oomkill-demo | grep -A10 "Last State:"

💡 What just happened? The container tried to allocate 150Mi of RAM, but the limit was 100Mi. The kernel’s OOM killer terminated the process. Exit code 137 = killed by SIGKILL (128 + 9). The fix: either raise the memory limit or fix the memory leak in your app.

kubectl delete pod oomkill-demo

Exercise 6: Init Container Gate

What we’re doing: Use an init container to simulate a dependency check that gates the main app.

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: init-demo
  namespace: pod-lab
spec:
  initContainers:
  - name: delay-start
    image: busybox
    command: ["sh", "-c", "echo 'Init: running preflight checks...'; sleep 10; echo 'Init: all checks passed!'"]

  containers:
  - name: app
    image: nginx:1.25
    resources:
      requests:
        memory: "32Mi"
        cpu: "50m"
      limits:
        memory: "64Mi"
        cpu: "100m"
EOF

Watch the init container run first:

kubectl get pod init-demo -w

Expected progression:

NAME        READY   STATUS     RESTARTS   AGE
init-demo   0/1     Init:0/1   0          2s
init-demo   0/1     Init:0/1   0          5s
init-demo   0/1     PodInitializing   0   11s
init-demo   1/1     Running    0          12s

See the init container logs:

kubectl logs init-demo -c delay-start

Expected output:

Init: running preflight checks...
Init: all checks passed!

kubectl delete pod init-demo

🔥 Break It! Challenge

What happens when you try to schedule a pod that’s too big for the node?

# Minikube has ~2GB RAM. Request more than exists.
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: toobig-pod
  namespace: pod-lab
spec:
  containers:
  - name: app
    image: nginx:1.25
    resources:
      requests:
        memory: "100Gi"   # 100GB on a 2GB node
        cpu: "64"         # 64 cores on a 2-core node
EOF

# See it stuck in Pending
kubectl get pod toobig-pod

# Find out WHY it's Pending
kubectl describe pod toobig-pod | grep -A10 "Events:"

Expected event:

Warning  FailedScheduling  0/1 nodes are available: 1 Insufficient memory, 1 Insufficient cpu.

The Scheduler found zero nodes that could satisfy the request. The pod stays Pending indefinitely.

kubectl delete pod toobig-pod

Cleanup

# Delete all pods in the lab namespace
kubectl delete pod --all -n pod-lab

# Switch back to default namespace
kubectl config set-context --current --namespace=default

# Delete the lab namespace
kubectl delete namespace pod-lab

What We Learned

Chapter 4: Workload Controllers

⏱️ Total chapter time: ~60 min (30 min reading + 30 min lab)

After this chapter, you will be able to: Deploy applications using the right controller for the job, perform zero-downtime rolling updates, rollback a bad deployment, and understand when to use each controller type.

What’s Inside

Prerequisites

• Completed Chapter 3 (understand pods)
• minikube status shows Running

4.1 ReplicaSets — Desired State and Self-Healing

⏱️ ~5 min read

TL;DR: A ReplicaSet ensures N copies of a pod are always running. If one dies, it creates a replacement. You almost never create ReplicaSets directly — Deployments manage them for you. But understanding them explains how Deployments work.

What a ReplicaSet Does

# replicaset.yaml
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: nginx-rs
spec:
  replicas: 3               # I want 3 pods running at all times
  selector:
    matchLabels:
      app: nginx            # I own pods with this label
  template:                 # Pod template — used to create new pods
    metadata:
      labels:
        app: nginx          # Must match selector above
    spec:
      containers:
      - name: nginx
        image: nginx:1.25

The ReplicaSet Controller runs a continuous loop:

graph LR
    RS[ReplicaSet Controller]
    RS -->|Count pods with\nlabel app=nginx| C{Actual vs Desired}
    C -->|"Actual: 3 = Desired: 3"| OK[✅ Nothing to do]
    C -->|"Actual: 2 < Desired: 3"| CREATE[Create 1 pod from template]
    C -->|"Actual: 4 > Desired: 3"| DELETE[Delete 1 pod]
    CREATE --> RS
    DELETE --> RS

The Label Selector — The Critical Link

The ReplicaSet knows which pods it “owns” via a label selector. This has an important implication:

⚠️ Warning: If you manually create a pod with a label that matches a ReplicaSet’s selector, the ReplicaSet will immediately delete it — it’s trying to maintain exactly N pods.

# A ReplicaSet with replicas: 3 and selector app=nginx
# If you manually add a 4th pod with the same label:
kubectl run extra --image=nginx --labels=app=nginx

# The ReplicaSet controller sees 4 pods when it wants 3
# → It immediately deletes one of them (randomly, often the one you just made)

Why Not Create ReplicaSets Directly?

One word: updates.

If you have a ReplicaSet running nginx:1.24 and you want to update to nginx:1.25, you’d need to delete the ReplicaSet and create a new one — causing downtime. There’s no rolling update.

Deployments solve this by managing two ReplicaSets during a rollout: the old one scales down as the new one scales up. The ReplicaSet is the mechanism; the Deployment is the orchestrator.

graph TB
    D[Deployment] --> RS1[ReplicaSet v1\nnginx:1.24\nreplicas: 0]
    D --> RS2[ReplicaSet v2\nnginx:1.25\nreplicas: 3]

The old RS stays (with 0 replicas) to allow rollbacks.

Try It

# Create a ReplicaSet directly (educational — not how you'd do it in practice)
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: nginx-rs
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx-rs-demo
  template:
    metadata:
      labels:
        app: nginx-rs-demo
    spec:
      containers:
      - name: nginx
        image: nginx:1.25
EOF

kubectl get pods -l app=nginx-rs-demo

# Delete one pod — watch it get recreated
POD=$(kubectl get pods -l app=nginx-rs-demo -o name | head -1)
kubectl delete $POD

# Immediately check — RS creates a replacement
kubectl get pods -l app=nginx-rs-demo

# Cleanup
kubectl delete replicaset nginx-rs

Key Takeaways

✅ Quick Check

Q1: You have a ReplicaSet with replicas: 5. You manually kubectl delete pod one of them. How many pods will exist 10 seconds later?

Q2: Can one ReplicaSet manage pods running different container images?

Q3: Why does a Deployment keep the old ReplicaSet around at 0 replicas after a rollout?

4.2 Deployments — Rolling Updates, Rollbacks, and Strategy

⏱️ ~8 min read

TL;DR: Deployments are how you deploy everything in Kubernetes. They manage ReplicaSets to give you zero-downtime rolling updates, instant rollbacks, and declarative version history. This is the controller you’ll use 90% of the time.

The Full Deployment YAML

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app

  strategy:
    type: RollingUpdate          # Default — zero downtime
    rollingUpdate:
      maxSurge: 1                # Allow 1 extra pod above desired during rollout
      maxUnavailable: 0          # Never go below desired count (zero downtime)

  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: app
        image: nginx:1.24        # ← version we'll upgrade
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "64Mi"
            cpu: "100m"
          limits:
            memory: "128Mi"
            cpu: "200m"
        readinessProbe:          # Deployment waits for this before continuing rollout
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          periodSeconds: 3

Rolling Update — What Actually Happens

With maxSurge: 1 and maxUnavailable: 0 on a 3-replica deployment:

sequenceDiagram
    participant D as Deployment
    participant RS1 as Old RS (v1)
    participant RS2 as New RS (v2)

    Note over D: kubectl apply (image: nginx:1.25)
    D->>RS2: Create RS v2
    RS2->>RS2: Create pod v2-1 (surge: now 4 pods)
    RS2-->>D: pod v2-1 Ready ✅
    D->>RS1: Scale down: delete pod v1-1 (back to 3)
    RS2->>RS2: Create pod v2-2 (4 pods again)
    RS2-->>D: pod v2-2 Ready ✅
    D->>RS1: Scale down: delete pod v1-2
    RS2->>RS2: Create pod v2-3
    RS2-->>D: pod v2-3 Ready ✅
    D->>RS1: Scale down: delete pod v1-3
    Note over D: Rollout complete. RS1 stays at 0.

The key: a pod must pass its readiness probe before the old one is deleted. This guarantees at least 3 healthy pods throughout.

Update Strategy Options

RollingUpdate (default)

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1          # or "25%" — extra pods above desired
    maxUnavailable: 0    # or "25%" — pods below desired allowed to be unavailable

Recreate

strategy:
  type: Recreate     # Kill ALL old pods, then create new ones

Causes downtime. Use only when old and new versions cannot run simultaneously (e.g., DB schema migrations with breaking changes).

Performing a Rolling Update

# Method 1: Edit the deployment file and kubectl apply
# (change nginx:1.24 → nginx:1.25 in deployment.yaml)
kubectl apply -f deployment.yaml

# Method 2: Set image imperatively
kubectl set image deployment/my-app app=nginx:1.25

# Method 3: Edit live
kubectl edit deployment my-app  # opens in $EDITOR

# Watch the rollout happen
kubectl rollout status deployment/my-app

Expected rollout output:

Waiting for deployment "my-app" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "my-app" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "my-app" rollout to finish: 1 old replicas are pending termination...
deployment "my-app" successfully rolled out

Rollback

# See rollout history
kubectl rollout history deployment/my-app

# Rollout history output:
# REVISION  CHANGE-CAUSE
# 1         <none>
# 2         <none>

# Add change-cause annotation for meaningful history
kubectl annotate deployment/my-app kubernetes.io/change-cause="Update to nginx 1.25"

# Rollback to previous version (RS v1 scales back up)
kubectl rollout undo deployment/my-app

# Rollback to a specific revision
kubectl rollout undo deployment/my-app --to-revision=1

# Watch the rollback
kubectl rollout status deployment/my-app

🏭 In Production: Always annotate deployments with --record (deprecated) or kubernetes.io/change-cause so rollout history is meaningful. “Deployed by CI build #1234 - ticket PROJ-567” is infinitely more useful than <none>.

Pausing and Resuming Rollouts

# Make multiple changes without triggering a rollout each time
kubectl rollout pause deployment/my-app

kubectl set image deployment/my-app app=nginx:1.25
kubectl set resources deployment/my-app -c app --limits=memory=256Mi,cpu=400m

# Apply all changes at once
kubectl rollout resume deployment/my-app

Scaling

# Scale manually
kubectl scale deployment my-app --replicas=10

# Or edit the YAML and kubectl apply
# In production: use HPA (Chapter 13) to autoscale

Try It

# Create a deployment with nginx:1.24
kubectl create deployment update-demo --image=nginx:1.24 --replicas=3

# Verify
kubectl get deploy update-demo
kubectl get pods -l app=update-demo

# Perform a rolling update to nginx:1.25
kubectl set image deployment/update-demo update-demo=nginx:1.25

# Watch it roll out
kubectl rollout status deployment/update-demo

# Confirm new image
kubectl get pods -l app=update-demo -o jsonpath='{range .items[*]}{.spec.containers[0].image}{"\n"}{end}'

# Check history
kubectl rollout history deployment/update-demo

# Rollback
kubectl rollout undo deployment/update-demo
kubectl rollout status deployment/update-demo

# Confirm original image restored
kubectl get pods -l app=update-demo -o jsonpath='{range .items[*]}{.spec.containers[0].image}{"\n"}{end}'

# Cleanup
kubectl delete deployment update-demo

Key Takeaways

✅ Quick Check

Q1: Your deployment has maxSurge: 0 and maxUnavailable: 1. During a rolling update of 5 replicas, how many pods are running at any given time?

Q2: A rollout is in progress. You update the deployment image again. What happens?

Q3: kubectl rollout undo fails with “no previous revision.” Why?

4.3 DaemonSets — One Per Node

⏱️ ~5 min read

TL;DR: A DaemonSet ensures exactly one pod runs on every node in the cluster. When new nodes are added, a pod is automatically scheduled on them. When nodes are removed, the pods are garbage collected.

When You Need Exactly One Pod Per Node

Some workloads are inherently node-level — they need to run on every machine:

📝 Note: Look at your kube-system namespace — kube-proxy is itself a DaemonSet. It runs on every node to maintain iptables rules.

DaemonSet YAML

# daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-collector
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: log-collector
  template:
    metadata:
      labels:
        app: log-collector
    spec:
      tolerations:
      # Allow scheduling on control plane nodes too
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule

      containers:
      - name: fluentd
        image: fluent/fluentd:v1.16
        resources:
          limits:
            memory: "200Mi"
            cpu: "100m"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true

      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:              # Mount the NODE's /var/log — not the pod's
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

The Key Difference: hostPath Volumes

DaemonSets typically mount host directories — they need to read files from the actual node filesystem. This is a hostPath volume, which is different from emptyDir or PersistentVolumes:

graph LR
    subgraph "Node A"
        HF["/var/log/ on Node"]
        subgraph "DaemonSet Pod"
            DS[log-collector\nmounts /var/log]
        end
        HF --> DS
    end
    subgraph "Node B"
        HF2["/var/log/ on Node"]
        subgraph "DaemonSet Pod"
            DS2[log-collector\nmounts /var/log]
        end
        HF2 --> DS2
    end

⚠️ Warning: hostPath volumes are powerful and dangerous. A pod with a hostPath mount to / can read the entire node filesystem. In production, restrict DaemonSet permissions with Pod Security Standards (Chapter 14).

DaemonSets vs Deployments

Targeting Specific Nodes

You can restrict a DaemonSet to a subset of nodes using nodeSelector or nodeAffinity:

spec:
  template:
    spec:
      nodeSelector:
        disk: ssd     # Only run on nodes labeled disk=ssd

# Label a node
kubectl label node minikube disk=ssd

# DaemonSet will now only schedule on labeled nodes

Try It

# On minikube (single node), a DaemonSet creates exactly 1 pod
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ds-demo
spec:
  selector:
    matchLabels:
      app: ds-demo
  template:
    metadata:
      labels:
        app: ds-demo
    spec:
      containers:
      - name: agent
        image: busybox
        command: ["sh", "-c", "while true; do echo 'Monitoring node...'; sleep 10; done"]
        resources:
          limits:
            memory: "32Mi"
            cpu: "50m"
EOF

# Exactly 1 pod on our 1-node cluster
kubectl get pods -l app=ds-demo -o wide

# See which node it's on
kubectl get pod -l app=ds-demo -o jsonpath='{.items[0].spec.nodeName}'

# Cleanup
kubectl delete daemonset ds-demo

Key Takeaways

✅ Quick Check

Q1: Your 10-node cluster runs a log-collector DaemonSet. You add 3 more nodes. What happens?

Q2: You want a DaemonSet to run on all nodes EXCEPT the control plane. How?

Q3: Can you kubectl scale a DaemonSet to 0?

4.4 StatefulSets — When Identity Matters

⏱️ ~6 min read

TL;DR: StatefulSets are for applications that need stable network identity and stable storage — databases, message queues, consensus systems. Each pod gets a persistent name (pod-0, pod-1) and its own PersistentVolumeClaim that follows it across restarts.

The Problem with Deployments for Stateful Apps

Deployments give each pod a random suffix (my-app-7f4b9c-x8p2q). If a pod is replaced, it gets a new random name, a new IP, and starts with an empty disk.

That’s fine for stateless services. But try running a database replica set where:

• Replica 1 needs to always be db-0 so others can find it
• db-0 needs its data to survive restarts
• Replicas must start in order (primary before secondaries)

Deployments can’t do this. StatefulSets can.

StatefulSet YAML

# statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongo
spec:
  serviceName: mongo          # Must match a Headless Service (see below)
  replicas: 3
  selector:
    matchLabels:
      app: mongo
  template:
    metadata:
      labels:
        app: mongo
    spec:
      containers:
      - name: mongo
        image: mongo:7
        ports:
        - containerPort: 27017
        volumeMounts:
        - name: data
          mountPath: /data/db    # Each pod mounts ITS OWN volume here

  volumeClaimTemplates:         # ← This is the StatefulSet superpower
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 1Gi

The volumeClaimTemplates section automatically creates a separate PVC for each pod:

• data-mongo-0 → mongo-0
• data-mongo-1 → mongo-1
• data-mongo-2 → mongo-2

The Four Guarantees of StatefulSets

graph TB
    subgraph "StatefulSet Guarantees"
        G1["1. Stable Pod Names\nmongo-0, mongo-1, mongo-2\n(not random hashes)"]
        G2["2. Stable Network Identity\nmongo-0.mongo.default.svc.cluster.local\n(predictable DNS)"]
        G3["3. Stable Storage\nEach pod keeps its own PVC\nacross restarts and reschedules"]
        G4["4. Ordered Operations\nStart: 0 → 1 → 2\nTerminate: 2 → 1 → 0"]
    end

The Required Headless Service

StatefulSets require a Headless Service (clusterIP: None) to provide stable DNS for individual pods:

# headless-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: mongo             # Must match StatefulSet's serviceName
spec:
  clusterIP: None         # ← Headless: no virtual IP, DNS returns pod IPs directly
  selector:
    app: mongo
  ports:
  - port: 27017

With this, each pod gets a stable DNS record:

mongo-0.mongo.default.svc.cluster.local  → 10.244.0.5
mongo-1.mongo.default.svc.cluster.local  → 10.244.0.6
mongo-2.mongo.default.svc.cluster.local  → 10.244.0.7

If mongo-1 is rescheduled to a different node, its DNS name stays mongo-1.mongo... — it just resolves to a new IP. Other services can always reach it by name.

When to Use StatefulSets

Use StatefulSets for:

• Databases (MongoDB, PostgreSQL, MySQL clusters)
• Message queues (Kafka, RabbitMQ)
• Consensus systems (Zookeeper, etcd)
• Any app that needs persistent, pod-specific identity

Don’t use StatefulSets for:

• Stateless web services (use Deployment)
• Batch jobs (use Jobs)
• Any app that doesn’t need stable identity or its own storage

🏭 In Production: Most teams use Helm charts (Chapter 12) for databases, which package StatefulSets with the right configuration. Don’t roll your own database StatefulSet for production without understanding your specific database’s clustering requirements.

Try It

# Create the headless service
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: nginx-headless
spec:
  clusterIP: None
  selector:
    app: nginx-sts
  ports:
  - port: 80
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx-sts
spec:
  serviceName: nginx-headless
  replicas: 3
  selector:
    matchLabels:
      app: nginx-sts
  template:
    metadata:
      labels:
        app: nginx-sts
    spec:
      containers:
      - name: nginx
        image: nginx:1.25
        resources:
          limits:
            memory: "64Mi"
            cpu: "100m"
EOF

# Watch them start IN ORDER (0, then 1, then 2)
kubectl get pods -l app=nginx-sts -w

# See stable pod names
kubectl get pods -l app=nginx-sts

# Delete the middle pod — it comes back with the SAME name
kubectl delete pod nginx-sts-1
kubectl get pods -l app=nginx-sts -w  # nginx-sts-1 returns with same name

# Cleanup
kubectl delete statefulset nginx-sts
kubectl delete service nginx-headless

Expected pod names:

NAME          READY   STATUS    RESTARTS   AGE
nginx-sts-0   1/1     Running   0          30s
nginx-sts-1   1/1     Running   0          25s
nginx-sts-2   1/1     Running   0          20s

Key Takeaways

✅ Quick Check

Q1: A StatefulSet pod kafka-2 gets rescheduled to a new node. Does it retain its data?

Q2: You scale a StatefulSet from 3 to 5 replicas. In what order do pods 3 and 4 start?

Q3: Can you use a regular (non-headless) Service with a StatefulSet?

4.5 Jobs and CronJobs — Run-to-Completion Workloads

⏱️ ~5 min read

TL;DR: Jobs run a task to completion (exit 0) and then stop. CronJobs run Jobs on a schedule. Use them for database backups, data processing, report generation — anything that has a start and end.

Jobs

A Job creates one or more pods, runs them to completion, and tracks success/failure.

# job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: db-backup
spec:
  completions: 1          # How many successful pod completions we need
  parallelism: 1          # How many pods to run in parallel
  backoffLimit: 3         # Retry up to 3 times before marking Job failed
  activeDeadlineSeconds: 300  # Kill the job if it runs longer than 5 min

  template:
    spec:
      restartPolicy: OnFailure   # Required — Never or OnFailure (not Always)
      containers:
      - name: backup
        image: postgres:16
        command:
        - sh
        - -c
        - |
          pg_dump -h $DB_HOST -U $DB_USER mydb | gzip > /backup/backup-$(date +%Y%m%d).sql.gz
          echo "Backup complete!"
        env:
        - name: DB_HOST
          value: postgres-svc
        - name: PGPASSWORD
          valueFrom:
            secretKeyRef:
              name: db-secret
              key: password

⚠️ Warning: Jobs require restartPolicy: OnFailure or Never. The default Always is not allowed — a Job that restarts forever never “completes.”

Parallel Jobs

Run multiple pods simultaneously for parallelizable work:

spec:
  completions: 10      # Need 10 successful completions total
  parallelism: 3       # Run 3 pods at a time
  backoffLimit: 5

graph LR
    J[Job\ncompletions:10\nparallelism:3]
    J --> P1[Pod 1 ✅]
    J --> P2[Pod 2 ✅]
    J --> P3[Pod 3 ✅]
    P1 --> P4[Pod 4 ✅]
    P2 --> P5[Pod 5 ✅]
    P3 --> P6[Pod 6 ✅]
    P4 & P5 & P6 --> DONE[Job Complete\n6 of 10... continues]

CronJobs

A CronJob creates a new Job on a schedule, using standard cron syntax.

# cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: daily-backup
spec:
  schedule: "0 2 * * *"         # Every day at 2:00 AM
  concurrencyPolicy: Forbid      # Don't start a new Job if the last one is still running
  successfulJobsHistoryLimit: 3  # Keep last 3 successful job records
  failedJobsHistoryLimit: 1      # Keep last 1 failed job record
  startingDeadlineSeconds: 60    # If missed schedule by 60s, skip this run

  jobTemplate:                   # The Job spec to run
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: backup
            image: postgres:16
            command: ["sh", "-c", "echo 'Running backup...' && sleep 5 && echo 'Done!'"]

CronJob Schedule Syntax

┌─── minute (0–59)
│ ┌─── hour (0–23)
│ │ ┌─── day of month (1–31)
│ │ │ ┌─── month (1–12)
│ │ │ │ ┌─── day of week (0–7, 0=Sunday)
│ │ │ │ │
* * * * *

"0 2 * * *"      → Every day at 2:00 AM
"*/15 * * * *"   → Every 15 minutes
"0 9 * * 1"      → Every Monday at 9:00 AM
"0 0 1 * *"      → First day of every month at midnight

💡 Tip: Use crontab.guru to validate cron expressions before applying them.

ConcurrencyPolicy Options

Use Forbid for anything that shouldn’t overlap (database backups, report generation). Use Allow for independent tasks.

Try It

# Create a simple job
cat <<'EOF' | kubectl apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: hello-job
spec:
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: hello
        image: busybox
        command: ["sh", "-c", "echo 'Job started!'; sleep 3; echo 'Job done!'; exit 0"]
EOF

# Watch it run to completion
kubectl get pods -l job-name=hello-job -w

# See Job status
kubectl get job hello-job

# Read the output
kubectl logs -l job-name=hello-job

# Create a CronJob (runs every minute for demo)
cat <<'EOF' | kubectl apply -f -
apiVersion: batch/v1
kind: CronJob
metadata:
  name: minutely-hello
spec:
  schedule: "* * * * *"
  successfulJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
          - name: hello
            image: busybox
            command: ["sh", "-c", "date; echo 'Cron tick!'"]
EOF

# Wait ~70 seconds, then see the created Jobs
sleep 70 && kubectl get jobs

# Cleanup
kubectl delete job hello-job
kubectl delete cronjob minutely-hello

Expected Job output:

NAME        STATUS     COMPLETIONS   DURATION   AGE
hello-job   Complete   1/1           4s         30s

Key Takeaways

✅ Quick Check

Q1: A Job with backoffLimit: 3 fails 4 times. What happens?

Q2: A CronJob with concurrencyPolicy: Forbid is scheduled every 5 minutes but takes 8 minutes to run. What happens?

Q3: You need to process 1000 independent work items, running 10 workers at a time. Which Job fields do you set?

Lab: Deploy, Scale, Update, and Rollback

⏱️ ~30 min hands-on

Objectives

• Deploy an application with 3 replicas using a Deployment
• Scale the Deployment up and down
• Perform a zero-downtime rolling update
• Observe ReplicaSet self-healing after pod deletion
• Trigger a bad rollout (wrong image) and roll back
• Create a CronJob and verify scheduled execution

Setup

kubectl create namespace workloads-lab
kubectl config set-context --current --namespace=workloads-lab

Exercise 1: Deploy and Inspect

What we’re doing: Deploy nginx and understand what gets created.

# Create a deployment declaratively
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webserver
  namespace: workloads-lab
  annotations:
    kubernetes.io/change-cause: "Initial deployment — nginx 1.24"
spec:
  replicas: 3
  selector:
    matchLabels:
      app: webserver
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: webserver
        version: "1.24"
    spec:
      containers:
      - name: nginx
        image: nginx:1.24
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "32Mi"
            cpu: "50m"
          limits:
            memory: "64Mi"
            cpu: "100m"
EOF

# Wait for rollout
kubectl rollout status deployment/webserver

Now inspect what was created:

# Deployment
kubectl get deployment webserver

# ReplicaSet created by the Deployment
kubectl get replicaset

# Pods created by the ReplicaSet
kubectl get pods -l app=webserver -o wide

# Full ownership chain
kubectl describe deployment webserver | grep -A3 "NewReplicaSet"

Expected output (deployment):

NAME        READY   UP-TO-DATE   AVAILABLE   AGE
webserver   3/3     3            3           30s

💡 What just happened? Creating the Deployment caused it to create a ReplicaSet, which created 3 pods. The chain is: Deployment → ReplicaSet → Pods. You only manage the Deployment.

Exercise 2: Scale Up and Down

# Scale to 5
kubectl scale deployment webserver --replicas=5
kubectl rollout status deployment/webserver
kubectl get pods -l app=webserver

# Scale back to 2
kubectl scale deployment webserver --replicas=2
kubectl get pods -l app=webserver -w  # Watch 3 pods terminate

# Scale back to 3 for the rest of the lab
kubectl scale deployment webserver --replicas=3
kubectl rollout status deployment/webserver

Exercise 3: Self-Healing — Delete a Pod

What we’re doing: Prove the ReplicaSet recreates killed pods.

# Get pod names
kubectl get pods -l app=webserver

# Delete one pod (replace with your actual pod name)
POD=$(kubectl get pods -l app=webserver -o name | head -1)
echo "Deleting: $POD"
kubectl delete $POD

# Immediately watch — new pod appears within seconds
kubectl get pods -l app=webserver -w

Expected output:

NAME                         READY   STATUS        RESTARTS   AGE
webserver-abc-x1             1/1     Running       0          5m
webserver-abc-x2             1/1     Running       0          5m
webserver-abc-x3             0/1     Terminating   0          5m   ← deleted
webserver-abc-x4             0/1     ContainerCreating   0   1s   ← NEW pod
webserver-abc-x4             1/1     Running       0          4s

The ReplicaSet detected the count dropped to 2 and immediately created a replacement.

Exercise 4: Rolling Update

What we’re doing: Update from nginx:1.24 to nginx:1.25 with zero downtime.

# In a separate watch (run this first)
kubectl get pods -l app=webserver -w &

# Perform the rolling update
kubectl set image deployment/webserver nginx=nginx:1.25
kubectl annotate deployment/webserver kubernetes.io/change-cause="Update to nginx 1.25" --overwrite

# Watch the rollout status
kubectl rollout status deployment/webserver

Expected watch output (one-at-a-time with maxSurge:1 maxUnavailable:0):

webserver-old-x1   1/1     Running             0   5m
webserver-old-x2   1/1     Running             0   5m
webserver-old-x3   1/1     Running             0   5m
webserver-new-y1   0/1     ContainerCreating   0   2s   ← new RS pod
webserver-new-y1   1/1     Running             0   5s   ← ready!
webserver-old-x1   1/1     Terminating         0   5m   ← old removed
webserver-new-y2   0/1     ContainerCreating   0   1s
...

# Kill the background watch
kill %1

# Confirm new image
kubectl get pods -l app=webserver -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].image}{"\n"}{end}'

# See rollout history
kubectl rollout history deployment/webserver

Expected history:

REVISION  CHANGE-CAUSE
1         Initial deployment — nginx 1.24
2         Update to nginx 1.25

Exercise 5: Bad Rollout + Rollback

What we’re doing: Deploy a broken image and roll back before it causes full outage.

# Deploy a non-existent image tag (simulating a bad push)
kubectl set image deployment/webserver nginx=nginx:this-tag-does-not-exist
kubectl annotate deployment/webserver kubernetes.io/change-cause="BROKEN — bad tag" --overwrite

# Watch it fail — new pods get ImagePullBackOff
kubectl get pods -l app=webserver -w &
sleep 20 && kill %1

# Check deployment status
kubectl rollout status deployment/webserver --timeout=30s

Expected output:

Waiting for deployment "webserver" rollout to finish: 1 out of 3 new replicas have been updated...
error: timed out waiting for the condition

# See the bad pods
kubectl get pods -l app=webserver

# One pod is ImagePullBackOff — but maxUnavailable:0 means the old pods stay!
# Current state: 3 old pods still running + 1 new (broken) pod trying

💡 Key insight: Because we set maxUnavailable: 0, the old pods are NOT removed until the new ones are Ready. The broken new pod can’t become Ready → the old pods keep serving traffic → no outage!

# Rollback immediately
kubectl rollout undo deployment/webserver
kubectl rollout status deployment/webserver

# Verify we're back on nginx:1.25
kubectl get pods -l app=webserver -o jsonpath='{range .items[*]}{.spec.containers[0].image}{"\n"}{end}'

# History now shows 4 revisions
kubectl rollout history deployment/webserver

Exercise 6: DaemonSet Demo

# Create a simple DaemonSet (1 pod on our 1-node cluster)
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-monitor
  namespace: workloads-lab
spec:
  selector:
    matchLabels:
      app: node-monitor
  template:
    metadata:
      labels:
        app: node-monitor
    spec:
      containers:
      - name: monitor
        image: busybox
        command: ["sh", "-c", "while true; do echo \"Node: $(hostname), Time: $(date)\"; sleep 5; done"]
        resources:
          limits:
            memory: "32Mi"
            cpu: "50m"
EOF

kubectl get pods -l app=node-monitor -o wide
kubectl logs -l app=node-monitor

# Can't scale a DaemonSet
kubectl scale daemonset node-monitor --replicas=0  # ← Error

Exercise 7: CronJob

# Run a job every minute
cat <<'EOF' | kubectl apply -f -
apiVersion: batch/v1
kind: CronJob
metadata:
  name: ticker
  namespace: workloads-lab
spec:
  schedule: "* * * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: Never
          containers:
          - name: tick
            image: busybox
            command: ["sh", "-c", "echo \"Tick at: $(date)\""]
EOF

# Watch for the job to fire (wait ~70 seconds)
echo "Waiting 70 seconds for first CronJob run..."
sleep 70
kubectl get jobs
kubectl get pods -l job-name

# Get the output
kubectl logs -l job-name --tail=5

Expected jobs output (after 2+ minutes):

NAME                  STATUS     COMPLETIONS   DURATION   AGE
ticker-28000001       Complete   1/1           4s         2m
ticker-28000002       Complete   1/1           4s         1m

🔥 Break It! Challenge

What happens when a Deployment’s maxUnavailable causes an outage during a bad rollout?

# Create a "risky" deployment with maxUnavailable:1
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: risky-deploy
  namespace: workloads-lab
spec:
  replicas: 3
  selector:
    matchLabels:
      app: risky
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1    # ← allows 1 pod to be missing
  template:
    metadata:
      labels:
        app: risky
    spec:
      containers:
      - name: app
        image: nginx:1.25
        resources:
          limits:
            memory: "32Mi"
            cpu: "50m"
EOF

kubectl rollout status deployment/risky-deploy

# Now push a broken image — this time 1 old pod WILL be removed first
kubectl set image deployment/risky-deploy app=nginx:broken-tag
kubectl get pods -l app=risky -w &
sleep 15 && kill %1

# State: 2 running old pods + 1 failed new pod = only 2 serving!
kubectl get pods -l app=risky

The lesson: maxUnavailable: 0 is a safety net. If you use maxUnavailable: 1 (or the 25% default), a bad rollout CAN reduce capacity. Always use maxUnavailable: 0 with a readinessProbe for zero-risk deployments.

Cleanup

kubectl config set-context --current --namespace=default
kubectl delete namespace workloads-lab

What We Learned

Chapter 5: Services — Exposing Your Applications

⏱️ Total chapter time: ~50 min (25 min reading + 25 min lab)

After this chapter, you will be able to: Connect pods using ClusterIP, expose applications externally with NodePort and LoadBalancer, use DNS to discover services, and debug connectivity failures.

What’s Inside

Prerequisites

• Completed Chapter 4 (comfortable with Deployments)
• minikube status shows Running

5.1 ClusterIP — Internal Communication

⏱️ ~6 min read

TL;DR: ClusterIP is the default Service type. It gives a group of pods a stable virtual IP that other pods can reach by name. Pod IPs change constantly; Service IPs never do.

The Problem Services Solve

Pods are ephemeral. Every time a pod restarts, it gets a new IP address. If Service A hardcodes Service B’s pod IP, it breaks the moment B’s pod is replaced.

graph LR
    subgraph "Without a Service"
        A[frontend\n10.244.0.5] -->|"hardcoded: 10.244.1.8"| B1[backend\n10.244.1.8]
        B1 -->|crashes!| DEAD[💀]
        B2[backend\n10.244.1.9] -.->|"new pod, new IP\nfrontend doesn't know"| Q[❓]
    end

graph LR
    subgraph "With a ClusterIP Service"
        A2[frontend] -->|"always: backend-svc\n10.96.45.123"| SVC[Service\nClusterIP\n10.96.45.123]
        SVC --> P1[backend pod\n10.244.1.9]
        SVC --> P2[backend pod\n10.244.1.10]
        SVC --> P3[backend pod\n10.244.1.11]
    end

The Service IP never changes. The pods behind it come and go — the Service automatically routes to healthy ones.

ClusterIP Service YAML

# service-clusterip.yaml
apiVersion: v1
kind: Service
metadata:
  name: backend-svc
spec:
  type: ClusterIP        # Default — omitting type also gives ClusterIP
  selector:
    app: backend         # Route traffic to pods with this label
  ports:
  - name: http
    port: 80             # Port the Service listens on
    targetPort: 8080     # Port on the pods to forward to
    protocol: TCP

kubectl apply -f service-clusterip.yaml
kubectl get svc backend-svc

Expected output:

NAME          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
backend-svc   ClusterIP   10.96.45.123   <none>        80/TCP    10s

The CLUSTER-IP is the stable virtual IP. EXTERNAL-IP: <none> means it’s not reachable from outside the cluster — by design.

How It Works Under the Hood

When a Service is created, the Endpoints controller builds a list of pod IPs matching the selector:

# See which pod IPs are behind a Service
kubectl get endpoints backend-svc

Expected output:

NAME          ENDPOINTS                                         AGE
backend-svc   10.244.0.4:8080,10.244.0.5:8080,10.244.0.6:8080   1m

kube-proxy on every node watches these Endpoints and programs iptables rules to load-balance traffic to those IPs.

graph TB
    Client[Client Pod] -->|curl backend-svc:80| KP[kube-proxy\niptables rules]
    KP -->|"random selection\n(round-robin)"| P1[Pod 10.244.0.4:8080]
    KP --> P2[Pod 10.244.0.5:8080]
    KP --> P3[Pod 10.244.0.6:8080]

Service DNS — The Real Power

Every Service gets an automatic DNS entry maintained by CoreDNS:

# Full DNS name format:
SERVICE-NAME.NAMESPACE.svc.cluster.local

# Examples:
backend-svc.default.svc.cluster.local
postgres.database.svc.cluster.local
redis.cache.svc.cluster.local

# Short form (same namespace only):
backend-svc

From any pod in the cluster:

# These all reach the same Service:
curl http://backend-svc                              # same namespace only
curl http://backend-svc.default                      # same cluster
curl http://backend-svc.default.svc.cluster.local    # fully qualified

🔗 Docker Parallel: In Docker Compose, containers reach each other by service name (http://backend). Kubernetes Services work the same way — but across multiple nodes and with automatic load balancing.

Port Mapping: port vs targetPort

ports:
- port: 80           # What OTHER pods use to reach this Service
  targetPort: 8080   # What port YOUR pods actually listen on

This decouples the internal implementation from the interface. Your app can change from port 8080 to 9000 — just update targetPort, no changes needed by consumers.

# Named targetPort (better — references container's port by name)
spec:
  containers:
  - name: app
    ports:
    - name: http
      containerPort: 8080

# Service can reference by name instead of number
spec:
  ports:
  - port: 80
    targetPort: http   # ← references the named port

Try It

# Deploy a backend
kubectl create deployment backend --image=nginx:1.25 --replicas=3

# Create a ClusterIP Service
kubectl expose deployment backend --port=80 --target-port=80 --name=backend-svc

# Verify
kubectl get svc backend-svc
kubectl get endpoints backend-svc

# Access from another pod (using DNS)
kubectl run curl-test --image=curlimages/curl --rm -it --restart=Never -- \
  curl -s http://backend-svc

# Cleanup
kubectl delete deployment backend
kubectl delete svc backend-svc

Key Takeaways

✅ Quick Check

Q1: A ClusterIP Service has 3 healthy pods and 1 crashed pod (all matching the selector). Does the Service route to the crashed pod?

Q2: You change a Deployment’s label from app: backend to app: backend-v2. The Service selector is app: backend. What happens?

Q3: Two Services in different namespaces are both named api. Can they coexist? How does a pod tell them apart?

5.2 NodePort — Exposing to the Outside

⏱️ ~5 min read

TL;DR: NodePort opens a port (30000–32767) on every node in the cluster and forwards traffic to your Service. It’s the simplest way to expose an app externally — but rarely used in production (Ingress is preferred).

How NodePort Works

# service-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
  name: web-nodeport
spec:
  type: NodePort
  selector:
    app: web
  ports:
  - port: 80          # ClusterIP port (internal)
    targetPort: 80    # Pod port
    nodePort: 30080   # External port on every node (30000–32767)
                      # Omit to let K8s auto-assign

graph LR
    Browser["Your Browser\nlocalhost:30080"] -->|NodePort| N1["Node\n:30080"]
    N1 --> SVC[Service ClusterIP]
    SVC --> P1[Pod :80]
    SVC --> P2[Pod :80]
    SVC --> P3[Pod :80]

Traffic flow:

1. External client hits any node on port 30080
2. Node forwards to the Service’s ClusterIP
3. Service load-balances to a pod

The NodePort Range

By default, nodePort values must be in 30000–32767. This range is intentionally high to avoid conflicts with system ports.

# Omit nodePort — K8s picks one for you
kubectl expose deployment web --type=NodePort --port=80

# See the assigned port
kubectl get svc web
# PORTS column: 80:31245/TCP  ← 31245 is the auto-assigned NodePort

Accessing on Minikube

On Minikube, the “node” is a VM or container with its own IP, not localhost:

# Get the Minikube IP
minikube ip
# Output: 192.168.49.2

# Access the NodePort
curl http://$(minikube ip):30080

# Or use the shortcut — minikube opens it in a browser
minikube service web-nodeport

# Or get the URL directly
minikube service web-nodeport --url

NodePort vs ClusterIP

🏭 In Production: NodePort has two problems: (1) clients must know the node IP, which changes when nodes are replaced, and (2) you can’t use standard ports 80/443. In production, use an Ingress (Chapter 6) or LoadBalancer Service instead.

Try It

# Deploy a simple web app
kubectl create deployment web --image=nginx:1.25 --replicas=2

# Expose with NodePort — let K8s pick the port
kubectl expose deployment web --type=NodePort --port=80

# See the assigned NodePort
kubectl get svc web

# Access it on Minikube
minikube service web --url

# Or manually:
NODE_PORT=$(kubectl get svc web -o jsonpath='{.spec.ports[0].nodePort}')
MINIKUBE_IP=$(minikube ip)
curl http://$MINIKUBE_IP:$NODE_PORT

# Cleanup
kubectl delete deployment web
kubectl delete svc web

Key Takeaways

✅ Quick Check

Q1: You have a 5-node cluster with a NodePort Service on port 31000. Client sends traffic to node 3. Node 3 has no matching pods — they all run on nodes 1 and 2. Does the request succeed?

Q2: Can you set nodePort: 80 to use a standard HTTP port?

Q3: You delete the NodePort Service and recreate it. Does it get the same NodePort number?

5.3 LoadBalancer — Cloud-Native Exposure

⏱️ ~5 min read

TL;DR: A LoadBalancer Service provisions a real cloud load balancer (AWS ELB, Azure ALB, GCP CLB) with a public IP automatically. On Minikube, use minikube tunnel to simulate this. In production, this is the standard way to expose services to the internet.

How LoadBalancer Works

# service-loadbalancer.yaml
apiVersion: v1
kind: Service
metadata:
  name: web-lb
spec:
  type: LoadBalancer
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 80

graph LR
    Internet -->|"Public IP\n203.0.113.42:80"| LB["Cloud Load Balancer\n(AWS ELB / Azure ALB / GCP CLB)"]
    LB --> N1[Node 1\nNodePort]
    LB --> N2[Node 2\nNodePort]
    N1 & N2 --> SVC[ClusterIP Service]
    SVC --> P1[Pod] & P2[Pod] & P3[Pod]

A LoadBalancer Service is really three services stacked:

1. A ClusterIP (internal routing)
2. A NodePort (on each node)
3. A cloud load balancer pointed at those NodePorts

When you kubectl apply a LoadBalancer Service in a cloud cluster, the cloud controller manager calls the cloud API to provision a real load balancer. The external IP appears in the EXTERNAL-IP column after ~30–60 seconds.

# Cloud cluster — shows a real external IP
kubectl get svc web-lb
# NAME    TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)        AGE
# web-lb  LoadBalancer   10.96.10.5     203.0.113.42    80:31456/TCP   45s

On Minikube: minikube tunnel

Minikube doesn’t have a cloud controller, so LoadBalancer Services stay in pending state without help:

kubectl get svc web-lb
# EXTERNAL-IP: <pending>  ← stuck without tunnel

Fix this with minikube tunnel — it creates a network route on your machine that acts as the load balancer:

# Run in a separate terminal (requires sudo on macOS/Linux)
minikube tunnel

# Now in your original terminal:
kubectl get svc web-lb
# EXTERNAL-IP: 127.0.0.1  ← now accessible on localhost!

curl http://127.0.0.1:80

⚠️ Warning: minikube tunnel requires elevated privileges and must stay running. If you close the terminal, the tunnel closes and EXTERNAL-IP goes back to pending.

When to Use LoadBalancer vs Ingress

💡 Tip: In production, most teams use one LoadBalancer that fronts an Ingress Controller (like NGINX), then use Ingress resources for path/host routing. This avoids paying for a separate cloud load balancer per service.

Try It

# Deploy app
kubectl create deployment web --image=nginx:1.25 --replicas=2

# Create LoadBalancer Service
kubectl expose deployment web --type=LoadBalancer --port=80

# Watch EXTERNAL-IP (starts as <pending>)
kubectl get svc web -w &

# In a new terminal, start the tunnel
# minikube tunnel   ← run this in another terminal

# After tunnel starts, EXTERNAL-IP becomes 127.0.0.1
# Kill the watch
kill %1

# Access it (with tunnel running)
curl http://127.0.0.1

# Cleanup
kubectl delete deployment web
kubectl delete svc web

Key Takeaways

✅ Quick Check

Q1: You apply a LoadBalancer Service on a bare-metal cluster with no cloud provider. What is EXTERNAL-IP?

Q2: You have 20 microservices. Should each get its own LoadBalancer Service?

Q3: A LoadBalancer Service has port: 443. Does this mean TLS is terminated at the load balancer?

5.4 Headless Services and DNS

⏱️ ~5 min read

TL;DR: A Headless Service (clusterIP: None) skips the virtual IP entirely. DNS returns the actual pod IPs directly. This is essential for StatefulSets and any app that needs to know which specific instance it’s talking to.

Regular vs Headless DNS

With a regular ClusterIP Service, DNS returns one stable virtual IP:

# Regular Service DNS lookup:
nslookup backend-svc.default.svc.cluster.local
→ Address: 10.96.45.123   (one ClusterIP)

With a Headless Service, DNS returns all matching pod IPs directly:

# Headless Service DNS lookup:
nslookup mongo.default.svc.cluster.local
→ Address: 10.244.0.5   (mongo-0's IP)
→ Address: 10.244.0.6   (mongo-1's IP)
→ Address: 10.244.0.7   (mongo-2's IP)

The client picks which IP to use — enabling client-side load balancing and direct pod addressing.

Headless Service YAML

# headless-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: mongo
spec:
  clusterIP: None       # ← This is what makes it headless
  selector:
    app: mongo
  ports:
  - port: 27017

That’s the only difference from a regular Service — clusterIP: None.

Per-Pod DNS (StatefulSets)

When a Headless Service is used with a StatefulSet (serviceName: mongo), each pod gets its own stable DNS record:

graph TB
    subgraph "Headless Service: mongo"
        DNS1["mongo-0.mongo.default.svc.cluster.local\n→ 10.244.0.5"]
        DNS2["mongo-1.mongo.default.svc.cluster.local\n→ 10.244.0.6"]
        DNS3["mongo-2.mongo.default.svc.cluster.local\n→ 10.244.0.7"]
    end
    
    Client[MongoDB\nClient Library] -->|"Primary:\nmongo-0.mongo..."| P0[mongo-0]
    Client -->|"Secondary:\nmongo-1.mongo..."| P1[mongo-1]
    Client -->|"Secondary:\nmongo-2.mongo..."| P2[mongo-2]

The MongoDB client can discover and address each replica independently — something impossible with a ClusterIP Service that hides individual pod identities.

DNS Record Types

# Inside a pod — check the DNS for a headless Service
kubectl run dns-test --image=busybox --rm -it --restart=Never -- \
  nslookup mongo.default.svc.cluster.local

# For a regular Service — one IP
# For a headless Service — multiple IPs (one per pod)

Headless Without a Selector

You can also create a headless Service with no selector — then manually manage the Endpoints. Useful for bridging to external databases:

# External database bridge
apiVersion: v1
kind: Service
metadata:
  name: external-db
spec:
  clusterIP: None
  ports:
  - port: 5432
---
apiVersion: v1
kind: Endpoints
metadata:
  name: external-db    # Must match Service name
subsets:
- addresses:
  - ip: 192.168.1.100  # Your external DB server IP
  ports:
  - port: 5432

Now pods can reach your external DB via external-db.default.svc.cluster.local:5432 — a fully internal DNS name pointing to an external resource.

Try It

# Create a headless Service backed by a Deployment
kubectl create deployment headless-demo --image=nginx:1.25 --replicas=3

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: headless-demo
spec:
  clusterIP: None
  selector:
    app: headless-demo
  ports:
  - port: 80
EOF

# Regular Service for comparison
kubectl expose deployment headless-demo --name=regular-svc --port=80

# Now compare DNS responses from inside a pod
kubectl run dns-test --image=busybox --rm -it --restart=Never -- sh -c "
  echo '--- Headless (multiple IPs):';
  nslookup headless-demo.default.svc.cluster.local;
  echo '--- Regular (one ClusterIP):';
  nslookup regular-svc.default.svc.cluster.local
"

# Cleanup
kubectl delete deployment headless-demo
kubectl delete svc headless-demo regular-svc

Expected output:

--- Headless (multiple IPs):
Server:    10.96.0.10
Address 1: 10.244.0.4
Address 2: 10.244.0.5
Address 3: 10.244.0.6

--- Regular (one ClusterIP):
Server:    10.96.0.10
Address 1: 10.96.45.200

Key Takeaways

✅ Quick Check

Q1: A StatefulSet pod kafka-1 is rescheduled to a different node with a new IP. Does kafka-1.kafka.default.svc.cluster.local still work?

Q2: A regular ClusterIP Service has 5 pods. You do an nslookup for the Service — how many IPs are returned?

Q3: When would you use a headless Service WITHOUT a StatefulSet?

Lab: Service Discovery and Connectivity Debugging

⏱️ ~25 min hands-on

Objectives

• Deploy two services and connect them via ClusterIP DNS
• Expose an app externally with NodePort
• Debug DNS resolution from inside a pod
• Observe and fix a broken label selector
• Inspect Endpoints to understand what a Service is routing to
• Use minikube tunnel with a LoadBalancer Service

Setup

kubectl create namespace svc-lab
kubectl config set-context --current --namespace=svc-lab

Exercise 1: ClusterIP — Wiring Two Services

What we’re doing: Deploy a frontend and backend, connect them using ClusterIP DNS.

# Deploy backend (simple nginx responding as our "API")
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  namespace: svc-lab
spec:
  replicas: 3
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
      - name: api
        image: nginx:1.25
        ports:
        - containerPort: 80
        resources:
          limits:
            memory: "64Mi"
            cpu: "100m"
---
apiVersion: v1
kind: Service
metadata:
  name: backend-svc
  namespace: svc-lab
spec:
  type: ClusterIP
  selector:
    app: backend
  ports:
  - port: 8080        # Service port clients use
    targetPort: 80    # Pod's actual port
EOF

# Verify
kubectl get deploy backend
kubectl get svc backend-svc
kubectl get endpoints backend-svc

Expected endpoints output:

NAME          ENDPOINTS                                         AGE
backend-svc   10.244.0.4:80,10.244.0.5:80,10.244.0.6:80       30s

Now test connectivity from a “frontend” pod:

# Run a curl pod as a frontend substitute
kubectl run frontend-test \
  --image=curlimages/curl \
  --rm -it --restart=Never \
  -- sh -c "
    echo 'Testing short DNS name:';
    curl -s -o /dev/null -w '%{http_code}' http://backend-svc:8080;
    echo '';
    echo 'Testing full DNS name:';
    curl -s -o /dev/null -w '%{http_code}' http://backend-svc.svc-lab.svc.cluster.local:8080;
    echo ''
  "

Expected output:

Testing short DNS name:
200
Testing full DNS name:
200

💡 What just happened? The pod resolved backend-svc via CoreDNS to the ClusterIP, which kube-proxy forwarded to one of the 3 backend pods. The 8080→80 port translation happened transparently.

Exercise 2: Inspect the DNS Infrastructure

What we’re doing: Look at CoreDNS and understand how Service DNS works.

# See CoreDNS pods (the cluster's DNS server)
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Run a debug pod and perform DNS lookups
kubectl run dns-debug \
  --image=busybox \
  --rm -it --restart=Never \
  -- sh -c "
    echo '=== DNS server configured for this pod ===';
    cat /etc/resolv.conf;
    echo '';
    echo '=== Lookup backend-svc (ClusterIP) ===';
    nslookup backend-svc.svc-lab.svc.cluster.local;
    echo '';
    echo '=== Lookup kube-dns (system service) ===';
    nslookup kube-dns.kube-system.svc.cluster.local
  "

Expected resolv.conf:

nameserver 10.96.0.10        ← CoreDNS ClusterIP
search svc-lab.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

The search domains are why short names like backend-svc work — the OS appends the search suffix automatically.

Exercise 3: Expose with NodePort

What we’re doing: Expose the backend to the outside world using NodePort.

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: backend-nodeport
  namespace: svc-lab
spec:
  type: NodePort
  selector:
    app: backend
  ports:
  - port: 80
    targetPort: 80
    nodePort: 30500
EOF

# Verify
kubectl get svc backend-nodeport

# Access via Minikube
MINIKUBE_IP=$(minikube ip)
echo "Accessing: http://$MINIKUBE_IP:30500"
curl -s -o /dev/null -w "HTTP Status: %{http_code}\n" http://$MINIKUBE_IP:30500

# Or use minikube shortcut
minikube service backend-nodeport -n svc-lab --url

Expected:

HTTP Status: 200

Exercise 4: Watch the Endpoints Live

What we’re doing: Watch how the Endpoints list updates as pods are added/removed.

# Start watching Endpoints
kubectl get endpoints backend-svc -w &

# Scale the backend up — watch new IPs appear
kubectl scale deployment backend --replicas=5
sleep 5

# Scale down — watch IPs disappear
kubectl scale deployment backend --replicas=1
sleep 5

# Kill the watch
kill %1

Expected watch output:

NAME          ENDPOINTS                                               AGE
backend-svc   10.244.0.4:80,10.244.0.5:80,10.244.0.6:80             5m
backend-svc   10.244.0.4:80,...+2 more...                            5m30s   ← scaled to 5
backend-svc   10.244.0.4:80                                          6m      ← scaled to 1

Scale back to 3 for the next exercise:

kubectl scale deployment backend --replicas=3

Exercise 5: Break It — Wrong Label Selector

What we’re doing: Misconfigure the Service selector and debug the resulting failure.

# Create a Service with a WRONG selector
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: broken-svc
  namespace: svc-lab
spec:
  type: ClusterIP
  selector:
    app: typo-backend    # ← WRONG — pods are labeled app=backend
  ports:
  - port: 80
    targetPort: 80
EOF

# Test connectivity — it fails
kubectl run curl-test \
  --image=curlimages/curl \
  --rm -it --restart=Never \
  -- curl --connect-timeout 3 http://broken-svc || echo "Connection failed!"

Expected:

curl: (28) Connection timed out after 3001 milliseconds
Connection failed!

Now diagnose:

# Step 1: Check Endpoints — is anything there?
kubectl get endpoints broken-svc
# Expected: broken-svc   <none>   ← empty! No pods match the selector

# Step 2: Describe the Service
kubectl describe svc broken-svc | grep -A5 "Selector:"
# Selector: app=typo-backend  ← wrong!

# Step 3: Fix it
kubectl patch svc broken-svc -p '{"spec":{"selector":{"app":"backend"}}}'

# Step 4: Verify Endpoints now populated
kubectl get endpoints broken-svc

# Step 5: Test connectivity — now succeeds
kubectl run curl-test \
  --image=curlimages/curl \
  --rm -it --restart=Never \
  -- curl -s -o /dev/null -w "HTTP Status: %{http_code}\n" http://broken-svc

💡 The debugging rule: If a Service isn’t routing, check Endpoints first. Empty Endpoints = selector mismatch. That’s the #1 Service failure cause.

Exercise 6: LoadBalancer with Minikube Tunnel

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: backend-lb
  namespace: svc-lab
spec:
  type: LoadBalancer
  selector:
    app: backend
  ports:
  - port: 80
    targetPort: 80
EOF

kubectl get svc backend-lb
# EXTERNAL-IP: <pending> initially

# Open a NEW terminal and run:
# minikube tunnel
# (keep it running while you test)

# Back in this terminal, wait a moment then:
kubectl get svc backend-lb
# EXTERNAL-IP: 127.0.0.1

curl -s -o /dev/null -w "HTTP Status: %{http_code}\n" http://127.0.0.1

🔥 Break It! Challenge

What happens when you send traffic to a ClusterIP from OUTSIDE the cluster?

# Get the ClusterIP of the backend-svc
CLUSTER_IP=$(kubectl get svc backend-svc -o jsonpath='{.spec.clusterIP}')
echo "ClusterIP: $CLUSTER_IP"

# Try to curl it directly from your host machine
curl --connect-timeout 3 http://$CLUSTER_IP:8080 || echo "Cannot reach ClusterIP from outside!"

# Try from inside the cluster (works fine)
kubectl run curl-inside \
  --image=curlimages/curl \
  --rm -it --restart=Never \
  -- curl -s -o /dev/null -w "HTTP Status: %{http_code}\n" http://$CLUSTER_IP:8080

The lesson: ClusterIPs are only routable inside the cluster. They exist in a virtual IP space managed by kube-proxy’s iptables rules on cluster nodes. Your host machine has no routes to that IP space (unless you use kubectl proxy or minikube tunnel).

Cleanup

kubectl config set-context --current --namespace=default
kubectl delete namespace svc-lab

What We Learned

Chapter 6: Ingress — HTTP Routing

⏱️ Total chapter time: ~55 min (25 min reading + 30 min lab)

After this chapter, you will be able to: Set up NGINX Ingress on Minikube, route HTTP traffic to multiple services using path and host rules, and terminate TLS with a self-signed certificate.

What’s Inside

Prerequisites

• Completed Chapter 5 (Services)
• minikube status shows Running

6.1 Ingress Controllers and Resources

⏱️ ~5 min read

TL;DR: Ingress is a two-part system: an Ingress Controller (a running pod that does the actual routing) and Ingress Resources (YAML rules that tell it where to send traffic). Without an Ingress Controller, Ingress Resources do nothing.

The Problem Ingress Solves

You have 5 microservices. Without Ingress:

• 5 LoadBalancer Services = 5 cloud load balancers = 5 external IPs = 5 monthly bills
• Clients need to know 5 different IPs/ports
• TLS certificates per service = complexity multiplied by 5

With Ingress:

• 1 LoadBalancer Service for the Ingress Controller
• 1 external IP for all services
• All routing rules defined in simple YAML
• TLS terminated once, centrally

graph LR
    Internet -->|"203.0.113.1:443"| LB[Cloud LB\n1 IP total]
    LB --> IC[Ingress Controller\nnginx pod]
    IC -->|"/api/*"| API[api-svc\n:8080]
    IC -->|"/auth/*"| AUTH[auth-svc\n:3000]
    IC -->|"/"| WEB[web-svc\n:80]
    IC -->|"admin.example.com"| ADMIN[admin-svc\n:9000]

The Two Parts

1. Ingress Controller

A running pod (or set of pods) that watches Ingress Resources and configures itself to route traffic. It’s NOT built into Kubernetes — you install it separately.

Popular controllers:

2. Ingress Resource

A Kubernetes object (kind: Ingress) that declares routing rules. The controller reads these and updates its config:

# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx          # Which controller handles this
  rules:
  - host: myapp.example.com
    http:
      paths:
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: api-svc
            port:
              number: 8080
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-svc
            port:
              number: 80

The Ingress Resource Anatomy

spec:
  ingressClassName: nginx    # Required in modern K8s (1.18+)
  
  tls:                       # Optional TLS config
  - hosts:
    - myapp.example.com
    secretName: myapp-tls
  
  rules:
  - host: myapp.example.com  # Match by hostname (optional — omit for any host)
    http:
      paths:
      - path: /api           # URL path to match
        pathType: Prefix     # Prefix | Exact | ImplementationSpecific
        backend:
          service:
            name: api-svc    # Route to this Service
            port:
              number: 8080

pathType options:

Key Takeaways

✅ Quick Check

Q1: You apply an Ingress Resource but no traffic is routed. What’s the most likely cause?

Q2: You have two Ingress Controllers installed (NGINX and Traefik). You apply an Ingress without ingressClassName. Which controller handles it?

Q3: pathType: Prefix with path /app. Does this match /application?

6.2 Setting Up NGINX Ingress on Minikube

⏱️ ~5 min read

TL;DR: One command enables the NGINX Ingress Controller on Minikube. After that, you need a way to reach it — either via minikube tunnel or by using Minikube’s built-in IP with the NodePort the controller exposes.

Enable the Minikube Ingress Addon

minikube addons enable ingress

Expected output:

💡  ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
    ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.10.x
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.x
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.x
🔎  Verifying ingress addon...
🌟  The 'ingress' addon is enabled

Verify It’s Running

# Check the controller pod (takes ~60 seconds to start)
kubectl get pods -n ingress-nginx

# Wait for it to be Ready
kubectl wait --namespace ingress-nginx \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=120s

Expected pod output:

NAME                                        READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-xxxxxxxxx-yyyyy    1/1     Running   0          90s

# See what Services the controller exposes
kubectl get svc -n ingress-nginx

Expected:

NAME                                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)
ingress-nginx-controller             NodePort    10.96.37.205   <none>        80:31xxx/TCP,443:32xxx/TCP
ingress-nginx-controller-admission   ClusterIP   10.96.88.14    <none>        443/TCP

The controller is exposed as a NodePort on Minikube, listening on ports 80 and 443.

The IngressClass

When the addon installs, it creates an IngressClass named nginx:

kubectl get ingressclass

Expected:

NAME    CONTROLLER             PARAMETERS   AGE
nginx   k8s.io/ingress-nginx   <none>       2m

Reference this in your Ingress Resources with ingressClassName: nginx.

Accessing Ingress on Minikube

Ingress on Minikube works best by adding hosts to /etc/hosts pointing to the Minikube IP:

# Get Minikube's IP
minikube ip
# Output: 192.168.49.2

# Add to /etc/hosts (requires sudo)
echo "$(minikube ip) myapp.local api.local" | sudo tee -a /etc/hosts

# Now these hostnames work:
curl http://myapp.local
curl http://api.local

Alternatively — use --header with curl to fake the Host header:

MINIKUBE_IP=$(minikube ip)

# Fake the host header — useful for testing without /etc/hosts
curl -H "Host: myapp.local" http://$MINIKUBE_IP

Architecture Overview

graph LR
    Browser -->|"myapp.local:80"| HOST["/etc/hosts:\nmyapp.local → 192.168.49.2"]
    HOST -->|"192.168.49.2:80"| NP["Minikube Node\nNodePort :31xxx"]
    NP --> IC["ingress-nginx-controller\npod"]
    IC -->|"Host: myapp.local\nPath: /api"| API[api-svc]
    IC -->|"Host: myapp.local\nPath: /"| WEB[web-svc]

Try It — Quick Smoke Test

# Verify the addon is enabled
minikube addons list | grep ingress

# Deploy a test app
kubectl create deployment smoke-test --image=nginx:1.25
kubectl expose deployment smoke-test --port=80

# Create an Ingress rule
cat <<'EOF' | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: smoke-test-ingress
spec:
  ingressClassName: nginx
  rules:
  - host: smoke.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: smoke-test
            port:
              number: 80
EOF

# Test with curl (fake host header)
curl -H "Host: smoke.local" http://$(minikube ip)

# Cleanup
kubectl delete deployment smoke-test
kubectl delete svc smoke-test
kubectl delete ingress smoke-test-ingress

Expected: nginx welcome page HTML.

Key Takeaways

✅ Quick Check

Q1: The NGINX Ingress controller pod is in CrashLoopBackOff. What happens to existing Ingress Resources?

Q2: Why does Minikube use NodePort for the Ingress controller instead of LoadBalancer?

Q3: You apply an Ingress with ingressClassName: traefik but only have NGINX Ingress installed. What happens?

6.3 Path-Based and Host-Based Routing

⏱️ ~7 min read

TL;DR: Ingress supports two routing dimensions: path-based (same domain, different URL paths → different services) and host-based (different domains → different services). You can combine both in one Ingress resource.

Path-Based Routing

All traffic to the same domain, split by URL path:

# path-based-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: path-routing
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2    # Strip the path prefix
spec:
  ingressClassName: nginx
  rules:
  - host: myapp.local
    http:
      paths:
      - path: /api(/|$)(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: api-svc
            port:
              number: 8080

      - path: /auth(/|$)(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: auth-svc
            port:
              number: 3000

      - path: /()(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: web-svc
            port:
              number: 80

graph LR
    C[Client] -->|"myapp.local/api/users"| IC[NGINX Ingress]
    C -->|"myapp.local/auth/login"| IC
    C -->|"myapp.local/"| IC
    IC -->|strips /api prefix → /users| API[api-svc :8080]
    IC -->|strips /auth prefix → /login| AUTH[auth-svc :3000]
    IC -->|passes /| WEB[web-svc :80]

The Rewrite Problem

Without the rewrite-target annotation, requests to /api/users would hit the backend service as /api/users. But your backend probably expects just /users — the /api prefix is just for Ingress routing.

# Without rewrite:
Client → /api/users → api-svc receives: /api/users  ← often breaks

# With rewrite-target: /$2
Client → /api/users → api-svc receives: /users  ← correct

The regex (/|$)(.*) captures the path after the prefix into group $2, and rewrite-target: /$2 sends just that part to the backend.

💡 Tip: For simpler cases where your backend CAN handle the full path (e.g., serving at /api/...), omit the rewrite annotation entirely and use pathType: Prefix.

Simple Path Routing (No Rewrite)

If your services handle the full path, keep it simple:

# simple-path-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: simple-routing
spec:
  ingressClassName: nginx
  rules:
  - host: myapp.local
    http:
      paths:
      - path: /api        # /api and all sub-paths → api-svc
        pathType: Prefix
        backend:
          service:
            name: api-svc
            port:
              number: 8080

      - path: /           # Everything else → web-svc
        pathType: Prefix
        backend:
          service:
            name: web-svc
            port:
              number: 80

⚠️ Warning: Order matters with Prefix paths. Always put more specific paths first (e.g., /api before /). NGINX Ingress picks the longest matching path, but listing specific paths first avoids ambiguity.

Host-Based Routing

Route different domains to different services — all on the same IP:

# host-based-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: host-routing
spec:
  ingressClassName: nginx
  rules:
  # Public-facing app
  - host: myapp.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-svc
            port:
              number: 80

  # Admin panel — separate domain
  - host: admin.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: admin-svc
            port:
              number: 9000

  # API subdomain
  - host: api.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-svc
            port:
              number: 8080

The NGINX controller reads the HTTP Host header to determine which rule applies.

Combining Path + Host

You can mix both dimensions in one Ingress:

spec:
  rules:
  - host: myapp.local            # Specific host
    http:
      paths:
      - path: /api               # AND specific path
        pathType: Prefix
        backend:
          service:
            name: api-svc
            port:
              number: 8080
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-svc
            port:
              number: 80

  - host: api.myapp.local       # API subdomain — all paths to api-svc
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-svc
            port:
              number: 8080

Default Backend

When no rule matches, you can specify a default backend:

spec:
  defaultBackend:           # Catch-all for unmatched requests
    service:
      name: default-404-svc
      port:
        number: 80
  rules:
  - ...

Without a defaultBackend, NGINX returns its own 404 page for unmatched requests.

Try It

# Deploy two services
kubectl create deployment app-a --image=nginx:1.25
kubectl create deployment app-b --image=nginx:1.25
kubectl expose deployment app-a --port=80 --name=svc-a
kubectl expose deployment app-b --port=80 --name=svc-b

# Customize their responses so we can tell them apart
kubectl exec -it deploy/app-a -- sh -c "echo 'Service A' > /usr/share/nginx/html/index.html"
kubectl exec -it deploy/app-b -- sh -c "echo 'Service B' > /usr/share/nginx/html/index.html"

# Create a path-based Ingress
cat <<'EOF' | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: path-demo
spec:
  ingressClassName: nginx
  rules:
  - host: demo.local
    http:
      paths:
      - path: /a
        pathType: Prefix
        backend:
          service:
            name: svc-a
            port:
              number: 80
      - path: /b
        pathType: Prefix
        backend:
          service:
            name: svc-b
            port:
              number: 80
EOF

MINIKUBE_IP=$(minikube ip)

# Test path routing
echo "=== /a should go to Service A ==="
curl -s -H "Host: demo.local" http://$MINIKUBE_IP/a

echo "=== /b should go to Service B ==="
curl -s -H "Host: demo.local" http://$MINIKUBE_IP/b

# Cleanup
kubectl delete deployment app-a app-b
kubectl delete svc svc-a svc-b
kubectl delete ingress path-demo

Key Takeaways

✅ Quick Check

Q1: You have two Ingress rules: path /api and path /. A request comes in for /apikeys. Which rule matches?

Q2: A client sends a request with Host: unknown.local — no Ingress rule matches this host. What does the user see?

Q3: You want /api/v1/users and /api/v2/users to go to the same backend service. What’s the simplest Ingress rule?

6.4 TLS Termination

⏱️ ~6 min read

TL;DR: The Ingress Controller handles TLS (HTTPS) so your backend services don’t have to. You create a TLS Secret containing a certificate and key, reference it in your Ingress, and the controller handles encryption/decryption transparently.

How TLS Termination Works

sequenceDiagram
    participant C as Browser (HTTPS)
    participant IC as Ingress Controller (TLS terminates here)
    participant S as Backend Service (plain HTTP)

    C->>IC: HTTPS request (encrypted)\nHost: myapp.local
    IC->>IC: Decrypt using TLS cert
    IC->>S: HTTP request (plain)\nforwarded to backend-svc:80
    S-->>IC: HTTP response
    IC-->>C: HTTPS response (encrypted)

Your backend pods only ever see plain HTTP. The Ingress Controller handles all the TLS complexity. This is called TLS termination at the edge.

Step 1: Create the TLS Secret

You need a certificate and private key. For local dev, generate a self-signed cert:

# Generate a self-signed certificate (valid for 365 days)
openssl req -x509 -nodes -days 365 \
  -newkey rsa:2048 \
  -keyout tls.key \
  -out tls.crt \
  -subj "/CN=myapp.local/O=myapp" \
  -addext "subjectAltName=DNS:myapp.local,DNS:api.local"

# Create a TLS Secret in Kubernetes
kubectl create secret tls myapp-tls \
  --cert=tls.crt \
  --key=tls.key

# Verify
kubectl get secret myapp-tls
kubectl describe secret myapp-tls

Expected secret output:

Name:         myapp-tls
Namespace:    default
Type:         kubernetes.io/tls

Data
====
tls.crt:  1234 bytes
tls.key:  1679 bytes

⚠️ Warning: Self-signed certificates cause browser “Not Secure” warnings. For production, use cert-manager with Let’s Encrypt to get free, auto-renewing trusted certificates.

Step 2: Reference the Secret in Ingress

# tls-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tls-ingress
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"    # Redirect HTTP → HTTPS
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - myapp.local             # Must match the certificate's CN/SAN
    secretName: myapp-tls     # The TLS Secret created above
  rules:
  - host: myapp.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-svc
            port:
              number: 80

The tls block tells the controller which certificate to use for which hostname. The rules block still uses http — it’s the routing rules, not the protocol. The controller handles the HTTPS→HTTP translation.

HTTP to HTTPS Redirect

The annotation nginx.ingress.kubernetes.io/ssl-redirect: "true" makes NGINX automatically redirect any HTTP request to HTTPS:

Client: GET http://myapp.local/
Ingress: 301 Redirect → https://myapp.local/
Client: GET https://myapp.local/ (now HTTPS)
Ingress: 200 OK (TLS terminated, forwarded to backend as HTTP)

Without this annotation, both HTTP and HTTPS work simultaneously.

Multiple Domains, Multiple Certs

You can have multiple TLS entries for different domains:

spec:
  tls:
  - hosts:
    - myapp.local
    secretName: myapp-tls        # cert for myapp.local

  - hosts:
    - admin.local
    secretName: admin-tls        # separate cert for admin.local

  rules:
  - host: myapp.local
    http:
      paths: [...]
  - host: admin.local
    http:
      paths: [...]

In Production: cert-manager + Let’s Encrypt

For production, never use self-signed certs. The standard setup:

# With cert-manager installed, just add this annotation:
metadata:
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"

spec:
  tls:
  - hosts:
    - myapp.example.com
    secretName: myapp-tls     # cert-manager creates and rotates this automatically

cert-manager watches Ingress Resources with this annotation, requests certificates from Let’s Encrypt, stores them as Secrets, and automatically rotates them before expiry. Zero manual cert management.

Try It

# Generate a self-signed cert
openssl req -x509 -nodes -days 365 \
  -newkey rsa:2048 \
  -keyout /tmp/tls.key \
  -out /tmp/tls.crt \
  -subj "/CN=tls-demo.local/O=demo" \
  2>/dev/null

# Create the Secret
kubectl create secret tls demo-tls \
  --cert=/tmp/tls.crt \
  --key=/tmp/tls.key

# Deploy a test app
kubectl create deployment tls-app --image=nginx:1.25
kubectl expose deployment tls-app --port=80 --name=tls-svc

# Create TLS Ingress
cat <<'EOF' | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tls-demo
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - tls-demo.local
    secretName: demo-tls
  rules:
  - host: tls-demo.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: tls-svc
            port:
              number: 80
EOF

MINIKUBE_IP=$(minikube ip)

# Test HTTP → HTTPS redirect
curl -v -H "Host: tls-demo.local" http://$MINIKUBE_IP 2>&1 | grep -E "< HTTP|Location"

# Test HTTPS directly (skip cert verification since self-signed)
curl -k -s -H "Host: tls-demo.local" https://$MINIKUBE_IP

# Cleanup
kubectl delete deployment tls-app
kubectl delete svc tls-svc
kubectl delete ingress tls-demo
kubectl delete secret demo-tls

Expected redirect output:

< HTTP/1.1 308 Permanent Redirect
Location: https://tls-demo.local/

Key Takeaways

✅ Quick Check

Q1: Your TLS certificate is for myapp.example.com but your Ingress host is app.example.com. What happens?

Q2: You enable ssl-redirect: true. A client sends a plain HTTP request. What response do they get?

Q3: Your TLS Secret’s certificate expires. What happens to traffic?

Lab: Multi-Service Ingress with TLS

⏱️ ~30 min hands-on

Objectives

• Enable and verify the NGINX Ingress Controller
• Deploy three services with distinct responses
• Configure path-based routing for two services under one domain
• Add host-based routing for a third service on a different domain
• Add TLS with a self-signed certificate
• Debug a broken Ingress rule

Setup

kubectl create namespace ingress-lab
kubectl config set-context --current --namespace=ingress-lab

# Ensure NGINX Ingress is enabled and ready
kubectl wait --namespace ingress-nginx \
  --for=condition=ready pod \
  --selector=app.kubernetes.io/component=controller \
  --timeout=120s

echo "Minikube IP: $(minikube ip)"

Exercise 1: Deploy Three Services

What we’re doing: Create three distinct services that we’ll route to via Ingress.

# Service 1: Web frontend
kubectl create deployment web --image=nginx:1.25 -n ingress-lab
kubectl expose deployment web --port=80 --name=web-svc -n ingress-lab
kubectl exec -n ingress-lab deploy/web -- \
  sh -c "echo '<h1>Welcome to Web Frontend</h1>' > /usr/share/nginx/html/index.html"

# Service 2: API
kubectl create deployment api --image=nginx:1.25 -n ingress-lab
kubectl expose deployment api --port=80 --name=api-svc -n ingress-lab
kubectl exec -n ingress-lab deploy/api -- \
  sh -c 'echo '"'"'{"service":"api","status":"ok"}'"'"' > /usr/share/nginx/html/index.html'

# Service 3: Admin panel (different domain)
kubectl create deployment admin --image=nginx:1.25 -n ingress-lab
kubectl expose deployment admin --port=80 --name=admin-svc -n ingress-lab
kubectl exec -n ingress-lab deploy/admin -- \
  sh -c "echo '<h1>Admin Panel</h1>' > /usr/share/nginx/html/index.html"

# Verify all running
kubectl get deploy,svc -n ingress-lab

Exercise 2: Path-Based Routing

What we’re doing: Route / to the web frontend and /api to the API service — both under myapp.local.

cat <<'EOF' | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: path-ingress
  namespace: ingress-lab
spec:
  ingressClassName: nginx
  rules:
  - host: myapp.local
    http:
      paths:
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: api-svc
            port:
              number: 80
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-svc
            port:
              number: 80
EOF

# Check Ingress status (ADDRESS should show Minikube IP)
kubectl get ingress path-ingress -n ingress-lab

# Test (using Host header to fake DNS)
MINIKUBE_IP=$(minikube ip)
echo "=== Testing / → web-svc ==="
curl -s -H "Host: myapp.local" http://$MINIKUBE_IP/

echo ""
echo "=== Testing /api → api-svc ==="
curl -s -H "Host: myapp.local" http://$MINIKUBE_IP/api

Expected output:

=== Testing / → web-svc ===
<h1>Welcome to Web Frontend</h1>

=== Testing /api → api-svc ===
{"service":"api","status":"ok"}

💡 What just happened? NGINX reads the Host header and the URL path, then selects the matching backend. The two services share a single IP but respond to different paths.

Exercise 3: Add Host-Based Routing

What we’re doing: Route traffic to the admin service using a separate hostname.

cat <<'EOF' | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: host-ingress
  namespace: ingress-lab
spec:
  ingressClassName: nginx
  rules:
  - host: admin.local          # Separate hostname → admin service
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: admin-svc
            port:
              number: 80
EOF

# Test host-based routing
MINIKUBE_IP=$(minikube ip)
echo "=== myapp.local → web-svc ==="
curl -s -H "Host: myapp.local" http://$MINIKUBE_IP/

echo ""
echo "=== admin.local → admin-svc ==="
curl -s -H "Host: admin.local" http://$MINIKUBE_IP/

# Optional: Add to /etc/hosts for browser testing
echo "# Add these for browser access:"
echo "echo '$(minikube ip) myapp.local admin.local' | sudo tee -a /etc/hosts"

Expected:

=== myapp.local → web-svc ===
<h1>Welcome to Web Frontend</h1>

=== admin.local → admin-svc ===
<h1>Admin Panel</h1>

Exercise 4: Add TLS

What we’re doing: Secure myapp.local with a self-signed certificate.

# Generate certificate covering both hosts
openssl req -x509 -nodes -days 365 \
  -newkey rsa:2048 \
  -keyout /tmp/lab.key \
  -out /tmp/lab.crt \
  -subj "/CN=myapp.local/O=ingress-lab" \
  -addext "subjectAltName=DNS:myapp.local,DNS:admin.local" \
  2>/dev/null

echo "Certificate generated."

# Create TLS Secret in ingress-lab namespace
kubectl create secret tls lab-tls \
  --cert=/tmp/lab.crt \
  --key=/tmp/lab.key \
  -n ingress-lab

kubectl get secret lab-tls -n ingress-lab

Update the path-based Ingress to use TLS:

cat <<'EOF' | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: path-ingress
  namespace: ingress-lab
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - myapp.local
    secretName: lab-tls
  rules:
  - host: myapp.local
    http:
      paths:
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: api-svc
            port:
              number: 80
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-svc
            port:
              number: 80
EOF

MINIKUBE_IP=$(minikube ip)

# Test HTTP → HTTPS redirect (expect 308)
echo "=== HTTP redirect check ==="
curl -v -H "Host: myapp.local" http://$MINIKUBE_IP/ 2>&1 | grep -E "< HTTP|Location"

# Test HTTPS directly (-k skips cert verification)
echo ""
echo "=== HTTPS response ==="
curl -k -s -H "Host: myapp.local" https://$MINIKUBE_IP/

# Inspect the certificate being served
echo ""
echo "=== Certificate info ==="
echo | openssl s_client -connect $MINIKUBE_IP:443 \
  -servername myapp.local 2>/dev/null | openssl x509 -noout -subject -dates

Expected:

=== HTTP redirect check ===
< HTTP/1.1 308 Permanent Redirect
Location: https://myapp.local/

=== HTTPS response ===
<h1>Welcome to Web Frontend</h1>

=== Certificate info ===
subject=CN=myapp.local, O=ingress-lab
notBefore=Jul 15 05:00:00 2026 GMT
notAfter=Jul 15 05:00:00 2027 GMT

Exercise 5: Debug a Broken Ingress

What we’re doing: Introduce a common Ingress mistake and diagnose it.

# Create a broken Ingress (references a Service that doesn't exist)
cat <<'EOF' | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: broken-ingress
  namespace: ingress-lab
spec:
  ingressClassName: nginx
  rules:
  - host: broken.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nonexistent-svc    # ← This service doesn't exist
            port:
              number: 80
EOF

# Test — gets 503 Service Unavailable
MINIKUBE_IP=$(minikube ip)
curl -s -o /dev/null -w "HTTP Status: %{http_code}\n" \
  -H "Host: broken.local" http://$MINIKUBE_IP/

Expected:

HTTP Status: 503

Now diagnose:

# Step 1: Check Ingress resource
kubectl describe ingress broken-ingress -n ingress-lab

# Step 2: Verify the backend Service exists
kubectl get svc -n ingress-lab | grep nonexistent  # Nothing found

# Step 3: Check NGINX Ingress controller logs
kubectl logs -n ingress-nginx \
  -l app.kubernetes.io/component=controller \
  --tail=20 | grep -i "error\|broken"

# Step 4: Fix it by pointing to a real service
kubectl patch ingress broken-ingress -n ingress-lab \
  --type='json' \
  -p='[{"op":"replace","path":"/spec/rules/0/http/paths/0/backend/service/name","value":"web-svc"}]'

# Verify the fix
curl -s -H "Host: broken.local" http://$MINIKUBE_IP/

🔥 Break It! Challenge

What happens when two Ingress resources claim the same host/path?

# Create a second Ingress that claims myapp.local/
cat <<'EOF' | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: duplicate-ingress
  namespace: ingress-lab
spec:
  ingressClassName: nginx
  rules:
  - host: myapp.local
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: admin-svc
            port:
              number: 80
EOF

# Which one wins?
for i in {1..5}; do
  curl -s -H "Host: myapp.local" http://$MINIKUBE_IP/ 2>/dev/null
  echo ""
done

Observation: NGINX uses the Ingress that was created first (by creation timestamp). The second one for the same host/path is effectively ignored (or causes inconsistent behavior). Check NGINX controller logs to see the conflict warning.

The lesson: Conflicting Ingress rules are a common production gotcha. Use namespaced Ingress resources carefully, and always check for conflicts after applying new Ingress rules.

kubectl delete ingress duplicate-ingress -n ingress-lab

Cleanup

kubectl config set-context --current --namespace=default
kubectl delete namespace ingress-lab

What We Learned

Chapter 7: ConfigMaps and Secrets

⏱️ Total chapter time: ~45 min (20 min reading + 25 min lab)

After this chapter, you will be able to: Externalize application configuration using ConfigMaps, manage sensitive data with Secrets, mount both as environment variables or files, and understand the security implications.

What’s Inside

Prerequisites

• Completed Chapter 3 (Pods)
• minikube status shows Running

7.1 ConfigMaps — Externalizing Configuration

⏱️ ~5 min read

TL;DR: A ConfigMap stores non-sensitive configuration as key-value pairs. It decouples your container image from its configuration — the same image can run in dev, staging, and prod with different ConfigMaps injected at runtime.

The Problem: Hardcoded Config

# Bad: config baked into the image
ENV DB_HOST=prod-db.internal
ENV MAX_CONNECTIONS=100
ENV LOG_LEVEL=info

Now you can’t reuse the same image in dev (different DB), staging (different log level), or if the DB hostname changes.

🔗 Docker Parallel: In Compose, you use .env files or environment: keys. In Kubernetes, ConfigMaps serve the same purpose — but they’re cluster objects that multiple pods can reference.

Creating ConfigMaps

Three ways to create a ConfigMap:

# Method 1: From literal values
kubectl create configmap app-config \
  --from-literal=DB_HOST=postgres-svc \
  --from-literal=DB_PORT=5432 \
  --from-literal=LOG_LEVEL=info \
  --from-literal=MAX_CONNECTIONS=100

# Method 2: From a .env file
# Create app.env:
# DB_HOST=postgres-svc
# DB_PORT=5432
# LOG_LEVEL=info
kubectl create configmap app-config --from-env-file=app.env

# Method 3: From a file (key = filename, value = file content)
kubectl create configmap nginx-config --from-file=nginx.conf

# Method 4: Declarative YAML (preferred for GitOps)

Declarative YAML (preferred):

# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: default
data:
  # Simple key-value pairs
  DB_HOST: "postgres-svc"
  DB_PORT: "5432"
  LOG_LEVEL: "info"
  MAX_CONNECTIONS: "100"

  # Multi-line config file content
  app.properties: |
    server.port=8080
    server.timeout=30
    feature.dark-mode=true

  nginx.conf: |
    server {
      listen 80;
      location / {
        proxy_pass http://backend:8080;
      }
    }

kubectl apply -f configmap.yaml
kubectl get configmap app-config
kubectl describe configmap app-config

What ConfigMaps Store

ConfigMaps hold strings only. For binary data, use the binaryData field (base64-encoded), but this is uncommon — Secrets handle sensitive binary data better.

data:
  key: "value"              # Simple string
  port: "5432"              # Numbers stored as strings — always quote them
  config.yaml: |            # Multi-line file content
    key: value
    nested:
      key: value

⚠️ Warning: ConfigMaps have a size limit of 1 MiB. They’re for configuration, not bulk data. If you need larger config files, use an init container to fetch them from an external source.

Try It

# Create a ConfigMap declaratively
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: demo-config
data:
  APP_ENV: "production"
  MAX_RETRIES: "3"
  config.json: |
    {
      "debug": false,
      "timeout": 30,
      "retries": 3
    }
EOF

# View it
kubectl get configmap demo-config -o yaml

# See the data fields
kubectl get configmap demo-config \
  -o jsonpath='{range .data}{@}{"\n"}{end}'

# Cleanup
kubectl delete configmap demo-config

Key Takeaways

✅ Quick Check

Q1: You update a ConfigMap while pods are running. Do the pods see the new values immediately?

Q2: Should you store a database password in a ConfigMap?

Q3: You have three environments (dev, staging, prod) each with different DB hostnames. What’s the recommended approach?

7.2 Secrets — Handling Sensitive Data

⏱️ ~6 min read

TL;DR: Secrets store sensitive data (passwords, tokens, keys). They look like ConfigMaps but are base64-encoded and have stricter access controls. Important: base64 is NOT encryption. Secrets are only as secure as your RBAC and etcd encryption configuration.

Creating Secrets

# From literals (most common)
kubectl create secret generic db-credentials \
  --from-literal=username=admin \
  --from-literal=password=s3cr3t!

# From files (e.g., SSH keys, TLS certs)
kubectl create secret generic ssh-key \
  --from-file=id_rsa=/path/to/private.key \
  --from-file=id_rsa.pub=/path/to/public.key

# TLS Secret (special type for Ingress)
kubectl create secret tls myapp-tls \
  --cert=tls.crt \
  --key=tls.key

# Docker registry credentials (for pulling private images)
kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=myuser \
  --docker-password=mypassword

Declarative YAML:

# secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque            # Generic secret type
data:
  username: YWRtaW4=   # base64("admin")
  password: czNjcjN0IQ==  # base64("s3cr3t!")

# Encode values yourself:
echo -n "admin" | base64      # YWRtaW4=
echo -n "s3cr3t!" | base64    # czNjcjN0IQ==

# Decode to verify:
echo "YWRtaW4=" | base64 --decode   # admin

⚠️ Warning: If you store Secrets in Git as YAML, the base64-encoded values are trivially decodable by anyone with repo access. Use tools like Sealed Secrets, External Secrets Operator, or HashiCorp Vault to store secrets safely in Git.

Secret Types

The base64 Reality Check

# Look at a Secret — the data appears base64-encoded
kubectl get secret db-credentials -o yaml

data:
  password: czNjcjN0IQ==   # "encrypted"? No. Just base64.
  username: YWRtaW4=

# Anyone with kubectl access can decode it in one command:
kubectl get secret db-credentials -o jsonpath='{.data.password}' | base64 --decode
# Output: s3cr3t!

Base64 is an encoding, not encryption. It’s there to handle binary data safely in YAML, not to protect the value.

What actually protects Secrets:

1. RBAC — restricts who can get or list Secrets
2. etcd encryption at rest — encrypts Secrets in the etcd database (must be enabled)
3. Namespace isolation — Secrets in namespace A can’t be read by pods in namespace B
4. Audit logging — who accessed which Secret and when

🏭 In Production: Enable etcd encryption at rest, restrict Secret access with RBAC, and never log Secret values. Consider an external secrets manager (AWS Secrets Manager, HashiCorp Vault) for critical secrets.

Using stringData (Easier YAML Authoring)

Instead of base64-encoding values yourself, use stringData:

# Kubernetes auto-encodes these to base64 on apply
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
stringData:              # Plain text — K8s encodes automatically
  username: admin
  password: s3cr3t!
  connection-string: "postgresql://admin:s3cr3t!@postgres-svc:5432/mydb"

kubectl apply -f secret.yaml

# Kubernetes stores it base64-encoded internally
kubectl get secret db-credentials -o yaml | grep -A3 "^data:"

💡 Tip: Use stringData in YAML files — it’s easier to read and less error-prone than manually base64-encoding values. Kubernetes handles the encoding internally.

Try It

# Create a Secret with stringData
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
stringData:
  DB_PASSWORD: "mySuperSecretPassword123"
  API_KEY: "sk-abcdef1234567890"
  JWT_SECRET: "my-jwt-signing-key-never-share-this"
EOF

# See it stored as base64
kubectl get secret app-secrets -o yaml

# Decode individual values
kubectl get secret app-secrets \
  -o jsonpath='{.data.DB_PASSWORD}' | base64 --decode
echo ""

# Cleanup
kubectl delete secret app-secrets

Key Takeaways

✅ Quick Check

Q1: A developer can kubectl get pods but you want to prevent them from reading Secrets. How?

Q2: You store a database password in a Secret. A bug in your app logs all environment variables to stdout. Is the password exposed?

Q3: What happens to a pod that references a Secret that doesn’t exist yet?

7.3 Environment Variables vs Volume Mounts

⏱️ ~5 min read

TL;DR: You can inject ConfigMap/Secret data into pods as environment variables (simple, but static) or as file mounts (more powerful, support live updates). Choose based on whether your app reads config from env vars or files.

Method 1: Environment Variables

Inject specific keys

spec:
  containers:
  - name: app
    image: myapp:latest
    env:
    # From a ConfigMap key
    - name: DB_HOST
      valueFrom:
        configMapKeyRef:
          name: app-config       # ConfigMap name
          key: DB_HOST           # Key inside ConfigMap

    # From a Secret key
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: db-credentials   # Secret name
          key: password          # Key inside Secret

    # Plain static value (for reference)
    - name: APP_VERSION
      value: "1.2.3"

Inject ALL keys from a ConfigMap/Secret

spec:
  containers:
  - name: app
    image: myapp:latest
    envFrom:
    - configMapRef:
        name: app-config       # All keys become env vars
    - secretRef:
        name: db-credentials   # All keys become env vars

# Inside the pod — all ConfigMap keys are env vars
kubectl exec -it my-pod -- env | grep -E "DB_HOST|LOG_LEVEL|MAX_CONN"

Env var limitations:

• Static at container start — updates to ConfigMap/Secret require pod restart
• If a key doesn’t exist in the ConfigMap, the pod fails to start
• All keys from envFrom become env vars — potential for name collisions

Method 2: Volume Mounts

Mount ConfigMaps and Secrets as files in the container’s filesystem:

spec:
  volumes:
  # ConfigMap as a volume
  - name: config-volume
    configMap:
      name: app-config          # Each key becomes a file

  # Secret as a volume
  - name: secret-volume
    secret:
      secretName: db-credentials
      defaultMode: 0400          # File permissions (owner read-only)

  containers:
  - name: app
    image: myapp:latest
    volumeMounts:
    - name: config-volume
      mountPath: /etc/config    # ConfigMap keys appear as files here
    - name: secret-volume
      mountPath: /etc/secrets   # Secret keys appear as files here
      readOnly: true

Inside the container:

/etc/config/
  DB_HOST          ← contains "postgres-svc"
  LOG_LEVEL        ← contains "info"
  app.properties   ← contains the multi-line properties content

/etc/secrets/
  username         ← contains "admin"
  password         ← contains "s3cr3t!"

Mounting Specific Keys as Files

Mount only specific keys, with custom filenames:

volumes:
- name: config-volume
  configMap:
    name: app-config
    items:
    - key: app.properties       # ConfigMap key
      path: application.properties  # File name inside the mount
    - key: nginx.conf
      path: nginx/nginx.conf    # Nested path

The Critical Difference: Live Updates

graph LR
    CM[ConfigMap\nupdated] -->|"Kubelet detects change\n~60 second delay"| VM[Volume Mount\n/etc/config/KEY\nupdated automatically ✅]
    CM -->|"Container was started\nwith env var value"| EV[Environment Variable\nNOT updated ❌\nrequires pod restart]

💡 Tip: For secrets especially, prefer volume mounts over env vars. A compromised process can dump all env vars trivially (os.environ in Python, process.env in Node.js). Reading a file requires explicit code. It’s a small but meaningful security improvement.

Try It

# Create both a ConfigMap and a Secret
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: demo-config
data:
  APP_ENV: production
  LOG_LEVEL: info
  config.json: |
    {"timeout": 30, "retries": 3}
---
apiVersion: v1
kind: Secret
metadata:
  name: demo-secret
type: Opaque
stringData:
  DB_PASSWORD: "supersecret"
  API_KEY: "key-12345"
EOF

# Pod using BOTH env vars and volume mounts
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: config-demo
spec:
  volumes:
  - name: config-vol
    configMap:
      name: demo-config
  - name: secret-vol
    secret:
      secretName: demo-secret
      defaultMode: 0400
  containers:
  - name: app
    image: busybox
    command: ["sh", "-c", "sleep 3600"]
    env:
    - name: APP_ENV
      valueFrom:
        configMapKeyRef:
          name: demo-config
          key: APP_ENV
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: demo-secret
          key: DB_PASSWORD
    volumeMounts:
    - name: config-vol
      mountPath: /etc/config
    - name: secret-vol
      mountPath: /etc/secrets
      readOnly: true
    resources:
      limits:
        memory: "32Mi"
        cpu: "50m"
EOF

kubectl wait pod config-demo --for=condition=Ready --timeout=30s

# Check env vars
kubectl exec config-demo -- env | grep -E "APP_ENV|DB_PASSWORD"

# Check volume-mounted files
kubectl exec config-demo -- ls /etc/config/
kubectl exec config-demo -- cat /etc/config/config.json

kubectl exec config-demo -- ls /etc/secrets/
kubectl exec config-demo -- cat /etc/secrets/API_KEY

# Cleanup
kubectl delete pod config-demo
kubectl delete configmap demo-config
kubectl delete secret demo-secret

Key Takeaways

✅ Quick Check

Q1: Your app reads config from environment variables. You update a ConfigMap. What must you do to apply the new config?

Q2: You mount a ConfigMap as a volume. The ConfigMap has 5 keys. How many files appear in the mount directory?

Q3: A pod has envFrom: - secretRef: name: my-secret. The Secret has a key MY_SECRET_VALUE. Another env var in the same pod is also named MY_SECRET_VALUE. Which one wins?

7.4 Immutable ConfigMaps and Secret Rotation

⏱️ ~4 min read

TL;DR: Mark ConfigMaps and Secrets as immutable: true for performance gains and accidental-change protection. For Secret rotation, update the Secret object and trigger a pod restart — or use volume mounts for zero-restart rotation.

Immutable ConfigMaps and Secrets

apiVersion: v1
kind: ConfigMap
metadata:
  name: stable-config
immutable: true          # Cannot be edited after creation
data:
  APP_VERSION: "2.1.0"
  FEATURE_FLAGS: "dark-mode,beta-dashboard"

apiVersion: v1
kind: Secret
metadata:
  name: stable-creds
immutable: true
stringData:
  API_KEY: "key-never-changes"

Benefits of immutable: true:

1. Performance — kubelet stops watching the ConfigMap/Secret for changes. At scale (thousands of ConfigMaps), this reduces API server load significantly
2. Safety — protects against accidental kubectl edit changes that would silently affect running pods
3. Auditability — forces you to create a new version, making changes explicit

Limitation: You cannot update an immutable ConfigMap or Secret. To change values:

# Must delete and recreate
kubectl delete configmap stable-config
kubectl apply -f new-stable-config.yaml

# Or create a new versioned name:
# stable-config-v2, then update pod specs to reference it

Secret Rotation

Rotating a secret (e.g., after a password change) without downtime:

With Volume Mounts (Zero-Downtime)

# 1. Update the Secret with the new value
kubectl patch secret db-credentials \
  --type='json' \
  -p='[{"op":"replace","path":"/data/password","value":"'$(echo -n "newpassword" | base64)'"}]'

# Or with stringData:
kubectl create secret generic db-credentials \
  --from-literal=password=newpassword \
  --dry-run=client -o yaml | kubectl apply -f -

# 2. Wait for kubelet to propagate the update to volume-mounted files (~60s)
# Your app must re-read the file on each use (not cache it at startup!)

If your app re-reads the mounted file on each request (or periodically), rotation is zero-downtime.

With Env Vars (Requires Rolling Restart)

# 1. Update the Secret
kubectl create secret generic db-credentials \
  --from-literal=password=newpassword \
  --dry-run=client -o yaml | kubectl apply -f -

# 2. Trigger a rolling restart so pods pick up new env vars
kubectl rollout restart deployment/my-app
kubectl rollout status deployment/my-app

Versioned ConfigMaps Pattern

A practical pattern for config changes that need explicit pod rollouts:

# Create a new versioned ConfigMap
kubectl create configmap app-config-v2 \
  --from-literal=LOG_LEVEL=debug \
  --from-literal=MAX_CONNECTIONS=200

# Update the Deployment to reference the new ConfigMap
# (edit deployment.yaml and change configMapKeyRef.name to app-config-v2)
kubectl apply -f deployment.yaml

# The rolling update ensures pods only use one config version at a time
# Old pods: app-config-v1 | New pods: app-config-v2

# After confirming the rollout is healthy, delete the old ConfigMap
kubectl delete configmap app-config-v1

This pattern ensures:

• Rollback is possible (old ConfigMap still exists during rollout)
• Exact config state is versioned and auditable
• No ambiguity about which pods use which config

Try It

# Create an immutable ConfigMap
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: fixed-config
immutable: true
data:
  REGION: "us-east-1"
  CLUSTER_NAME: "prod-cluster"
EOF

# Try to edit it — should fail
kubectl patch configmap fixed-config \
  --type='json' \
  -p='[{"op":"replace","path":"/data/REGION","value":"eu-west-1"}]'
# Error: configmap "fixed-config" is immutable

# Must delete and recreate to change
kubectl delete configmap fixed-config

# Cleanup

Key Takeaways

✅ Quick Check

Q1: You have 10,000 ConfigMaps in a large cluster. How does immutable: true help?

Q2: Your app caches the database password in memory at startup (reads env var once). You rotate the Secret and restart pods. Is there a window where old password is used?

Q3: Can you make a Secret immutable after it was created as mutable?

Lab: Configure a 12-Factor App

⏱️ ~25 min hands-on

Objectives

• Create ConfigMaps for app config and an nginx config file
• Create Secrets for database credentials and an API key
• Deploy an app that reads from both env vars and mounted files
• Update a ConfigMap and observe live propagation via volume mount
• Rotate a Secret and trigger a rolling restart
• Debug a missing ConfigMap reference

Setup

kubectl create namespace config-lab
kubectl config set-context --current --namespace=config-lab

Exercise 1: Create ConfigMaps

What we’re doing: Create two ConfigMaps — one for simple key-value config, one for a config file.

# ConfigMap 1: App settings
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: config-lab
data:
  APP_ENV: "production"
  LOG_LEVEL: "info"
  MAX_CONNECTIONS: "50"
  CACHE_TTL: "300"
  ALLOWED_HOSTS: "myapp.local,localhost"
EOF

# ConfigMap 2: NGINX config file (to be mounted as a file)
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
  namespace: config-lab
data:
  default.conf: |
    server {
        listen 80;
        server_name _;

        add_header X-Config-Source "ConfigMap" always;
        add_header X-App-Env "$APP_ENV" always;

        location /health {
            return 200 "OK\n";
            add_header Content-Type text/plain;
        }

        location / {
            root /usr/share/nginx/html;
            index index.html;
        }
    }
EOF

kubectl get configmap -n config-lab

Exercise 2: Create Secrets

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: config-lab
type: Opaque
stringData:
  DB_HOST: "postgres-svc.config-lab.svc.cluster.local"
  DB_USER: "appuser"
  DB_PASSWORD: "InitialPassword123!"
  DB_NAME: "appdb"
---
apiVersion: v1
kind: Secret
metadata:
  name: api-keys
  namespace: config-lab
type: Opaque
stringData:
  STRIPE_SECRET_KEY: "sk_test_abcdefghijklmnopqrstuvwx"
  SENDGRID_API_KEY: "SG.xxxxxxxxxxxxxxxx"
  JWT_SECRET: "my-jwt-signing-secret-minimum-32-chars-long"
EOF

# Verify Secrets exist (values are hidden)
kubectl get secrets -n config-lab

# Decode to verify (this is what an attacker would do too)
kubectl get secret db-credentials -n config-lab \
  -o jsonpath='{.data.DB_PASSWORD}' | base64 --decode
echo ""

Exercise 3: Deploy the App

What we’re doing: Deploy nginx with config injected via both env vars and volume mounts.

cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
  namespace: config-lab
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
    spec:
      volumes:
      # ConfigMap as a file (nginx config)
      - name: nginx-conf-vol
        configMap:
          name: nginx-config
      # Secret as files (credentials)
      - name: secret-vol
        secret:
          secretName: db-credentials
          defaultMode: 0400

      containers:
      - name: nginx
        image: nginx:1.25
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "32Mi"
            cpu: "50m"
          limits:
            memory: "64Mi"
            cpu: "100m"

        # Env vars from ConfigMap (simple values)
        envFrom:
        - configMapRef:
            name: app-config

        # Individual Secret values as env vars
        env:
        - name: STRIPE_KEY
          valueFrom:
            secretKeyRef:
              name: api-keys
              key: STRIPE_SECRET_KEY

        volumeMounts:
        # Mount nginx config file
        - name: nginx-conf-vol
          mountPath: /etc/nginx/conf.d
          readOnly: true
        # Mount credentials as files
        - name: secret-vol
          mountPath: /etc/secrets
          readOnly: true
EOF

kubectl rollout status deployment/webapp -n config-lab

Verify configuration was injected:

POD=$(kubectl get pods -n config-lab -l app=webapp -o name | head -1)

echo "=== Environment Variables from ConfigMap ==="
kubectl exec -n config-lab $POD -- env | grep -E "APP_ENV|LOG_LEVEL|MAX_CONN|CACHE_TTL"

echo ""
echo "=== Secret value as env var ==="
kubectl exec -n config-lab $POD -- sh -c 'echo "STRIPE_KEY starts with: ${STRIPE_KEY:0:10}..."'

echo ""
echo "=== Volume-mounted nginx config ==="
kubectl exec -n config-lab $POD -- cat /etc/nginx/conf.d/default.conf

echo ""
echo "=== Volume-mounted secret files ==="
kubectl exec -n config-lab $POD -- ls -la /etc/secrets/
kubectl exec -n config-lab $POD -- cat /etc/secrets/DB_USER
echo ""

Expected output (partial):

=== Environment Variables from ConfigMap ===
APP_ENV=production
LOG_LEVEL=info
MAX_CONNECTIONS=50
CACHE_TTL=300

=== Volume-mounted nginx config ===
server {
    listen 80;
    ...
    add_header X-Config-Source "ConfigMap" always;
    ...

=== Volume-mounted secret files ===
-r-------- 1 root root  11 DB_HOST
-r-------- 1 root root   7 DB_USER
-r-------- 1 root root  19 DB_PASSWORD
-r-------- 1 root root   5 DB_NAME

Exercise 4: Live Config Update via Volume Mount

What we’re doing: Update the nginx ConfigMap and watch the file change inside the container automatically.

# Record the current config file's last-modified time inside the pod
POD=$(kubectl get pods -n config-lab -l app=webapp -o name | head -1)
kubectl exec -n config-lab $POD -- stat /etc/nginx/conf.d/default.conf | grep Modify

# Update the ConfigMap — change a header value
kubectl patch configmap nginx-config -n config-lab \
  --type='merge' \
  -p='{"data":{"default.conf":"server {\n    listen 80;\n    server_name _;\n\n    add_header X-Config-Source \"ConfigMap-v2\" always;\n\n    location /health {\n        return 200 \"OK-v2\\n\";\n        add_header Content-Type text/plain;\n    }\n\n    location / {\n        root /usr/share/nginx/html;\n        index index.html;\n    }\n}\n"}}'

echo "ConfigMap updated. Waiting ~60 seconds for kubelet to propagate..."
sleep 65

# Check if the file changed (kubelet propagated the update)
kubectl exec -n config-lab $POD -- stat /etc/nginx/conf.d/default.conf | grep Modify
kubectl exec -n config-lab $POD -- grep "Config-Source" /etc/nginx/conf.d/default.conf

Expected:

# Before:
Modify: 2026-07-15 05:30:00.000000000 +0000

# After ~60 seconds:
Modify: 2026-07-15 05:31:05.000000000 +0000  ← changed!
    add_header X-Config-Source "ConfigMap-v2" always;  ← new value!

💡 Note: The file updated automatically without any pod restart. However, nginx doesn’t reload its config automatically — you’d need to send SIGHUP to nginx or use an init container/sidecar that watches for file changes and reloads.

Exercise 5: Secret Rotation

What we’re doing: Simulate a database password rotation.

# Current password
kubectl exec -n config-lab $POD -- cat /etc/secrets/DB_PASSWORD
echo ""

# Rotate the password
kubectl create secret generic db-credentials \
  -n config-lab \
  --from-literal=DB_HOST="postgres-svc.config-lab.svc.cluster.local" \
  --from-literal=DB_USER="appuser" \
  --from-literal=DB_PASSWORD="RotatedPassword456!" \
  --from-literal=DB_NAME="appdb" \
  --dry-run=client -o yaml | kubectl apply -f -

echo "Secret updated. Waiting ~60 seconds for kubelet propagation..."
sleep 65

# New password is live in the file — no pod restart needed
kubectl exec -n config-lab $POD -- cat /etc/secrets/DB_PASSWORD
echo ""

# However — the env var DB_PASSWORD is NOT updated (if it were set via env)
# Env vars are static: compare with STRIPE_KEY (set via env, not file):
kubectl exec -n config-lab $POD -- env | grep STRIPE_KEY

Expected:

# Before rotation:
InitialPassword123!

# After ~60 seconds:
RotatedPassword456!   ← updated via volume mount, no restart needed

Exercise 6: Debug a Missing ConfigMap

What we’re doing: Create a pod that references a non-existent ConfigMap and diagnose the failure.

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: missing-config-pod
  namespace: config-lab
spec:
  containers:
  - name: app
    image: nginx:1.25
    envFrom:
    - configMapRef:
        name: this-configmap-does-not-exist   # ← doesn't exist
    resources:
      limits:
        memory: "32Mi"
        cpu: "50m"
EOF

# Pod stays in Pending / CreateContainerConfigError
kubectl get pod missing-config-pod -n config-lab

# Describe to see the error
kubectl describe pod missing-config-pod -n config-lab | tail -15

Expected describe output (Events):

Warning  Failed     Error: configmap "this-configmap-does-not-exist" not found

Fix it:

# Create the missing ConfigMap
kubectl create configmap this-configmap-does-not-exist \
  -n config-lab \
  --from-literal=PLACEHOLDER=true

# Pod should start now (K8s retries)
kubectl get pod missing-config-pod -n config-lab -w

🔥 Break It! Challenge

What happens when a ConfigMap is deleted while pods are running?

# Create a pod with a volume-mounted ConfigMap
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: mounted-cm-pod
  namespace: config-lab
spec:
  volumes:
  - name: cfg
    configMap:
      name: app-config
  containers:
  - name: app
    image: busybox
    command: ["sh", "-c", "while true; do cat /cfg/LOG_LEVEL; sleep 5; done"]
    volumeMounts:
    - name: cfg
      mountPath: /cfg
    resources:
      limits:
        memory: "16Mi"
        cpu: "50m"
EOF

kubectl wait pod mounted-cm-pod -n config-lab --for=condition=Ready --timeout=30s
kubectl logs mounted-cm-pod -n config-lab

# Delete the ConfigMap while the pod is running
kubectl delete configmap app-config -n config-lab

# Check what happens to the pod and the files
sleep 5
kubectl get pod mounted-cm-pod -n config-lab   # Pod still running!
kubectl exec -n config-lab mounted-cm-pod -- cat /cfg/LOG_LEVEL  # Still readable!

# Key insight: deleting a ConfigMap doesn't kill running pods
# The mounted files remain accessible until the pod is restarted

The lesson: Deleting a ConfigMap doesn’t immediately affect running pods that have already mounted it. The files remain cached by kubelet. However, if the pod restarts, it will fail to start — it can’t mount a ConfigMap that no longer exists.

Cleanup

kubectl config set-context --current --namespace=default
kubectl delete namespace config-lab

What We Learned

Chapter 8: Persistent Storage — Volumes, PVs, and PVCs

⏱️ Total chapter time: ~55 min (25 min reading + 30 min lab)

After this chapter, you will be able to: Mount persistent storage to pods, understand the PV/PVC/StorageClass model, use dynamic provisioning on Minikube, and design stateful workloads that survive pod restarts.

What’s Inside

Prerequisites

• Completed Chapters 3 and 4 (Pods and Workloads)
• minikube status shows Running

8.1 The Storage Problem and Volume Types

⏱️ ~5 min read

TL;DR: Container filesystems are ephemeral — they vanish when the container restarts. Kubernetes Volumes solve this by attaching external storage to pods. There are many volume types; knowing which to use when is the skill.

The Ephemeral Filesystem Problem

graph LR
    P1["Pod (nginx)\nWrites /data/uploads/\n10,000 images"] -->|Pod crashes| DEAD[💀]
    NEW["New Pod (nginx)\nFresh container\nFilesystem empty"] -->|"Where's\n/data/uploads?"| Q[❓ Gone forever]

Any data written inside a container lives only as long as the container lives. This is by design — it makes containers stateless and reproducible. But databases, file storage, and caches need persistence.

🔗 Docker Parallel: In Docker/Compose, you use bind mounts (./data:/data) or named volumes (myvolume:/data). Kubernetes has equivalents — hostPath (like bind mount) and PersistentVolumes (like named volumes, but cluster-managed).

Volume Types — The Menu

Kubernetes supports many volume types. Here are the ones you’ll actually use:

emptyDir — Ephemeral Shared Storage

You’ve already seen this in Chapter 3 (sidecar pattern). Quick recap:

spec:
  volumes:
  - name: shared-data
    emptyDir: {}          # Created when pod starts, deleted when pod dies
    # emptyDir:
    #   medium: Memory    # Store in RAM instead of disk (faster, uses node RAM)
    #   sizeLimit: 500Mi  # Limit how much storage it can use

  containers:
  - name: writer
    volumeMounts:
    - name: shared-data
      mountPath: /data
  - name: reader
    volumeMounts:
    - name: shared-data
      mountPath: /shared

emptyDir survives container restarts within the same pod — but is deleted if the pod is rescheduled to another node.

hostPath — Node Filesystem Mount

spec:
  volumes:
  - name: node-logs
    hostPath:
      path: /var/log          # Actual directory on the node
      type: Directory         # Must exist | DirectoryOrCreate | File | Socket

  containers:
  - name: log-reader
    volumeMounts:
    - name: node-logs
      mountPath: /logs

⚠️ Warning: hostPath volumes are tied to a specific node. If the pod is rescheduled to a different node, it sees a different (empty or different-content) directory. Only use for DaemonSets (always on the same node) or single-node Minikube development.

The Standard Path: PersistentVolumeClaims

For anything beyond ephemeral sharing or DaemonSets, use PersistentVolumeClaims (PVCs). The PVC abstracts the actual storage backend — whether it’s an EBS volume, NFS server, or Minikube’s local storage:

spec:
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: my-pvc     # Reference a PVC by name

  containers:
  - name: database
    volumeMounts:
    - name: data
      mountPath: /var/lib/postgresql/data

The PVC is covered in detail in the next section.

Key Takeaways

✅ Quick Check

Q1: An nginx pod writes uploaded files to /tmp/uploads. The pod crashes and is restarted by the ReplicaSet controller. Are the uploaded files still there?

Q2: Two containers in the same pod need to share files. Which volume type is appropriate?

Q3: You have a DaemonSet that collects logs from /var/log on each node. Which volume type do you use?

8.2 PersistentVolumes and PersistentVolumeClaims

⏱️ ~7 min read

TL;DR: A PersistentVolume (PV) is a piece of actual storage provisioned by an admin. A PersistentVolumeClaim (PVC) is a pod’s request for storage. Kubernetes binds PVCs to PVs automatically. In practice, StorageClasses (next section) eliminate the need to create PVs manually.

The Three-Layer Model

graph TB
    DEV["Developer\nPod + PVC\n(requests 5Gi)"]
    OPS["Admin\nPersistentVolume\n(provisions 10Gi EBS volume)"]
    K8S["Kubernetes\nBinding\n(PVC → PV)"]

    DEV -->|creates| PVC[PersistentVolumeClaim\nsize: 5Gi\naccessMode: ReadWriteOnce]
    OPS -->|creates| PV[PersistentVolume\ncapacity: 10Gi\naccessMode: ReadWriteOnce\nAWS EBS vol-xxxxx]
    K8S -->|binds| BIND[PVC bound to PV\nPod gets the storage]
    PVC --> BIND
    PV --> BIND

• PV = the actual storage resource (an EBS volume, NFS export, etc.)
• PVC = a request for storage, written by the developer
• Binding = Kubernetes matches a PVC to a compatible PV

🔗 Docker Parallel: A PV is like a named Docker volume that exists on the host. A PVC is like a container’s volumes: request for that named volume. The difference: Kubernetes manages the matching automatically.

PersistentVolume (Manual Provisioning)

Admins create PVs to represent physical storage:

# pv.yaml — Admin-managed
apiVersion: v1
kind: PersistentVolume
metadata:
  name: postgres-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteOnce              # One node can mount read-write
  persistentVolumeReclaimPolicy: Retain   # Keep data after PVC deleted
  storageClassName: manual     # Label for matching with PVCs

  # The actual storage backend:
  hostPath:                    # Minikube dev only
    path: /mnt/postgres-data
  # In production, this would be:
  # awsElasticBlockStore:
  #   volumeID: vol-0abcd1234
  #   fsType: ext4

PersistentVolumeClaim (Developer-Facing)

Pods don’t reference PVs directly — they reference PVCs:

# pvc.yaml — Developer-managed
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pvc
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: manual     # Must match a PV's storageClassName
  resources:
    requests:
      storage: 5Gi             # Request 5Gi from the pool of PVs

kubectl apply -f pv.yaml
kubectl apply -f pvc.yaml

# Kubernetes binds them automatically
kubectl get pv,pvc

Expected output:

NAME                           CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM
persistentvolume/postgres-pv   10Gi       RWO            Retain           Bound    default/postgres-pvc

NAME                                 STATUS   VOLUME        CAPACITY   ACCESS MODES
persistentvolumeclaim/postgres-pvc   Bound    postgres-pv   10Gi       RWO

The PVC requested 5Gi but was bound to a 10Gi PV — PVs can be larger than the claim, but not smaller.

Using the PVC in a Pod

# pod-with-pvc.yaml
apiVersion: v1
kind: Pod
metadata:
  name: postgres
spec:
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: postgres-pvc   # Reference the PVC by name

  containers:
  - name: postgres
    image: postgres:16
    env:
    - name: POSTGRES_PASSWORD
      value: "password"
    - name: PGDATA
      value: /var/lib/postgresql/data/pgdata
    volumeMounts:
    - name: data
      mountPath: /var/lib/postgresql/data

PVC Lifecycle States

stateDiagram-v2
    [*] --> Pending : PVC created\nno matching PV yet
    Pending --> Bound : K8s found a matching PV
    Bound --> Released : Pod deleted PVC deleted\nPV still has data (Retain policy)
    Released --> Available : Admin manually cleans data
    Available --> Bound : New PVC can bind
    Bound --> Lost : PV backend becomes unavailable

# Check PVC status
kubectl get pvc postgres-pvc

# STATUS column:
# Pending  → waiting for a PV
# Bound    → matched and ready to use
# Lost     → the underlying PV disappeared

When PVC is Pending

The most common issue: a PVC stays Pending forever.

# Find out why
kubectl describe pvc my-pvc | grep -A10 "Events:"

Typical causes:

• No PV with matching storageClassName exists
• No PV with sufficient capacity (PVC requests 10Gi, only 5Gi PVs available)
• No PV with matching accessModes
• StorageClass doesn’t exist or its provisioner isn’t running

Try It

# Create a PV backed by hostPath (Minikube only)
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
  name: demo-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: manual
  hostPath:
    path: /tmp/demo-pv-data
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: demo-pvc
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: manual
  resources:
    requests:
      storage: 500Mi
EOF

# Check binding
kubectl get pv,pvc

# Use it in a pod
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: pvc-writer
spec:
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: demo-pvc
  containers:
  - name: writer
    image: busybox
    command: ["sh", "-c", "echo 'Hello from PVC!' > /data/test.txt && sleep 3600"]
    volumeMounts:
    - name: data
      mountPath: /data
    resources:
      limits:
        memory: "16Mi"
        cpu: "50m"
EOF

kubectl wait pod pvc-writer --for=condition=Ready --timeout=30s
kubectl exec pvc-writer -- cat /data/test.txt

# Delete the pod — data survives!
kubectl delete pod pvc-writer

# Recreate — data still there
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: pvc-reader
spec:
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: demo-pvc
  containers:
  - name: reader
    image: busybox
    command: ["sh", "-c", "cat /data/test.txt && sleep 60"]
    volumeMounts:
    - name: data
      mountPath: /data
    resources:
      limits:
        memory: "16Mi"
        cpu: "50m"
EOF

kubectl wait pod pvc-reader --for=condition=Ready --timeout=30s
kubectl logs pvc-reader  # "Hello from PVC!" — data survived pod deletion!

# Cleanup
kubectl delete pod pvc-reader
kubectl delete pvc demo-pvc
kubectl delete pv demo-pv

Key Takeaways

✅ Quick Check

Q1: A PVC requests 500Mi but the only available PV has 1Gi. Is the binding successful?

Q2: You delete a pod that uses a PVC. Is the PVC deleted?

Q3: Two pods in the same namespace try to use the same PVC with accessMode: ReadWriteOnce. What happens?

8.3 StorageClasses and Dynamic Provisioning

⏱️ ~6 min read

TL;DR: Instead of admins creating PVs by hand, a StorageClass tells Kubernetes how to automatically provision storage on demand. When a PVC is created, the StorageClass’s provisioner creates the backing volume automatically. This is how every major cloud Kubernetes offering works.

The Problem with Manual PVs

Manual provisioning doesn’t scale:

• Admin creates 10 PVs → developer creates 10 PVCs → bindings happen
• If a PVC needs 7Gi but only 5Gi PVs exist → stuck
• Admin must predict future storage needs
• Every cloud environment needs different PV specs

StorageClasses solve this with on-demand provisioning.

StorageClass YAML

# storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast-ssd
provisioner: kubernetes.io/aws-ebs    # Which plugin creates the volumes
parameters:
  type: gp3                           # Storage type (provider-specific)
  fsType: ext4
  encrypted: "true"
reclaimPolicy: Delete                 # Delete volume when PVC is deleted
allowVolumeExpansion: true            # Allow PVCs to grow (not shrink)
volumeBindingMode: WaitForFirstConsumer  # Create volume only when pod is scheduled

Popular provisioners:

How Dynamic Provisioning Works

sequenceDiagram
    participant D as Developer
    participant K as Kubernetes
    participant P as Cloud Provisioner
    participant S as Cloud Storage

    D->>K: Create PVC (5Gi, storageClass: fast-ssd)
    K->>P: "Someone needs 5Gi of fast-ssd"
    P->>S: Create EBS volume (5Gi, gp3)
    S-->>P: Volume ID: vol-abc123
    P->>K: Create PV (vol-abc123, 5Gi)
    K->>K: Bind PVC → PV
    K-->>D: PVC is Bound ✅

The developer writes a 5-line PVC YAML. Everything else is automatic.

Default StorageClass

Most clusters have a default StorageClass. If a PVC omits storageClassName, it uses the default:

# See StorageClasses in your cluster
kubectl get storageclass

# On Minikube:
# NAME                 PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE
# standard (default)   k8s.io/minikube-hostpath   Delete          Immediate

The (default) annotation means PVCs without a storageClassName use this class automatically.

# See which StorageClass is marked default
kubectl get storageclass -o jsonpath='{range .items[?(@.metadata.annotations.storageclass\.kubernetes\.io/is-default-class=="true")]}{.metadata.name}{"\n"}{end}'

Dynamic Provisioning in Practice

With dynamic provisioning, you only write the PVC — no PV needed:

# dynamic-pvc.yaml — No PV required, StorageClass handles it
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: database-storage
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: standard    # Minikube default (or fast-ssd in cloud)
  resources:
    requests:
      storage: 5Gi

kubectl apply -f dynamic-pvc.yaml

# A PV is automatically created and bound
kubectl get pv,pvc

volumeBindingMode: WaitForFirstConsumer

A critical setting for cloud clusters:

volumeBindingMode: Immediate         # Create volume NOW when PVC is created
volumeBindingMode: WaitForFirstConsumer  # Create volume only when a pod using the PVC is scheduled

Why WaitForFirstConsumer matters:

Cloud block storage (EBS, Azure Disk) is zone-specific. If the volume is created in us-east-1a but your pod gets scheduled to us-east-1b, the pod can’t attach the volume.

With WaitForFirstConsumer, Kubernetes waits until it knows which node (and zone) the pod will run on, then creates the volume in that same zone.

🏭 In Production: Always use WaitForFirstConsumer for cloud block storage StorageClasses. Immediate can cause unschedulable pods in multi-zone clusters.

Try It

# See Minikube's default StorageClass
kubectl get storageclass

# Create a PVC without specifying storageClassName (uses default)
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dynamic-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
EOF

# A PV was created automatically!
kubectl get pv,pvc

# See the auto-created PV details
kubectl describe pvc dynamic-pvc | grep -E "StorageClass|Volume:|Capacity"

# Use it
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: dynamic-test
spec:
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: dynamic-pvc
  containers:
  - name: app
    image: busybox
    command: ["sh", "-c", "echo 'Dynamic provisioning works!' > /data/test.txt && sleep 60"]
    volumeMounts:
    - name: data
      mountPath: /data
    resources:
      limits:
        memory: "16Mi"
        cpu: "50m"
EOF

kubectl wait pod dynamic-test --for=condition=Ready --timeout=30s
kubectl exec dynamic-test -- cat /data/test.txt

# Cleanup
kubectl delete pod dynamic-test
kubectl delete pvc dynamic-pvc
# PV is deleted automatically (reclaimPolicy: Delete)
kubectl get pv  # Should be gone

Key Takeaways

✅ Quick Check

Q1: You delete a PVC backed by a StorageClass with reclaimPolicy: Delete. What happens to the data?

Q2: A PVC in Pending state has storageClassName: fast-ssd. No StorageClass named fast-ssd exists. Will Kubernetes use the default StorageClass?

Q3: Can you resize a PVC from 5Gi to 10Gi after it’s bound?

8.4 Access Modes and Reclaim Policies

⏱️ ~5 min read

TL;DR: Access modes define how many nodes can mount a volume simultaneously. Reclaim policies define what happens to the underlying storage when a PVC is deleted. Getting these right prevents both data loss and runaway cloud costs.

Access Modes

graph TB
    subgraph "ReadWriteOnce (RWO)"
        N1[Node 1\nMounts RW] --> V1[(Volume)]
        N2[Node 2\n❌ Cannot mount] -.->|blocked| V1
    end
    subgraph "ReadWriteMany (RWX)"
        N3[Node 1\nMounts RW] --> V2[(NFS Volume)]
        N4[Node 2\nMounts RW] --> V2
        N5[Node 3\nMounts RW] --> V2
    end

The practical reality:

• ReadWriteOnce — works for almost all databases and single-replica apps
• ReadWriteMany — needed only for shared file storage across multiple pods on different nodes; requires NFS or cloud file storage
• Most cloud block storage (EBS, Azure Disk, GCP PD) is RWO only

spec:
  accessModes:
  - ReadWriteOnce    # Most common — put in a list even for a single mode

What Happens with Multiple Pods on the Same Node

ReadWriteOnce is per-node, not per-pod. Multiple pods on the same node can all mount the same RWO volume:

Node 1:
  pod-a → /data (mounted RWO) ✅
  pod-b → /data (same volume, same node) ✅

Node 2:
  pod-c → /data (different node) ❌ blocked

This is why StatefulSets use volumeClaimTemplates — each pod gets its OWN PVC. They don’t share a single RWO volume across nodes.

Reclaim Policies

When a PVC is deleted, what happens to the PV and its data?

# In PV spec (for manually created PVs):
spec:
  persistentVolumeReclaimPolicy: Retain

# In StorageClass (for dynamically provisioned):
reclaimPolicy: Delete

Choosing the Right Policy

graph TD
    Q{Is this data\ncritical / irreplaceable?}
    Q -->|Yes| R[Retain\n+ automated backup]
    Q -->|No or easily recreated| D[Delete]
    
    R --> RA[Production databases\nUser uploads\nTransaction logs]
    D --> DA[Caches\nTemp processing\nTest data]

⚠️ Warning: reclaimPolicy: Delete with a production database is a common disaster scenario. A developer runs kubectl delete pvc prod-db-pvc thinking it’s a dev environment — the EBS volume and all its data are gone instantly, with no recycle bin. Always use Retain for production databases.

The Retain Workflow (After PVC Deletion)

With Retain, the PV enters Released state. To reuse it:

# 1. PVC was deleted — PV is now Released
kubectl get pv
# STATUS: Released

# 2. Recover data (copy off the volume before doing anything)

# 3. To allow a new PVC to bind this PV:
kubectl patch pv my-pv -p '{"spec":{"claimRef":null}}'
# Now the PV is Available again

# 4. Create a new PVC that matches this PV's storageClass and capacity

In practice, most teams use Retain + a separate backup strategy (Velero, pg_dump, etc.) rather than trying to rebind old PVs.

Checking PV Status

# Full PV status overview
kubectl get pv

# Detailed info on a specific PV
kubectl describe pv my-pv

# Check which PVC is bound to a PV
kubectl get pv -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\t"}{.spec.claimRef.name}{"\n"}{end}'

PV Status phases:

Try It

# Create a PV with Retain policy
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
  name: retain-demo-pv
spec:
  capacity:
    storage: 100Mi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual
  hostPath:
    path: /tmp/retain-demo
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: retain-demo-pvc
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: manual
  resources:
    requests:
      storage: 100Mi
EOF

kubectl get pv,pvc

# Write data
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: retain-writer
spec:
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: retain-demo-pvc
  containers:
  - name: w
    image: busybox
    command: ["sh", "-c", "echo 'Important data!' > /data/important.txt && sleep 10"]
    volumeMounts:
    - name: data
      mountPath: /data
    resources:
      limits:
        memory: "16Mi"
        cpu: "50m"
EOF

kubectl wait pod retain-writer --for=condition=Ready --timeout=30s
kubectl exec retain-writer -- cat /data/important.txt
kubectl delete pod retain-writer

# Delete the PVC
kubectl delete pvc retain-demo-pvc

# PV is now Released — NOT deleted
kubectl get pv retain-demo-pv
# STATUS: Released  ← data preserved on disk

# The actual data is still on the node
minikube ssh "cat /tmp/retain-demo/important.txt"

# Cleanup
kubectl delete pv retain-demo-pv

Key Takeaways

✅ Quick Check

Q1: A Deployment with 3 replicas uses a single PVC with accessMode: ReadWriteOnce. All 3 pods are on the same node. Does this work?

Q2: You have reclaimPolicy: Delete on your production PostgreSQL PVC. A junior developer runs kubectl delete pvc postgres-data. What’s the damage?

Q3: Can you change a PV’s reclaimPolicy after it’s created?

Lab: Stateful App with Persistent Storage

⏱️ ~30 min hands-on

Objectives

• Create a PVC using dynamic provisioning (Minikube default StorageClass)
• Deploy PostgreSQL using that PVC
• Write data to the database and restart the pod — prove data survives
• Deploy a StatefulSet with volumeClaimTemplates (one PVC per pod)
• Observe PVC lifecycle: what happens when the StatefulSet is deleted
• Debug a pod stuck due to a missing PVC

Setup

kubectl create namespace storage-lab
kubectl config set-context --current --namespace=storage-lab

Exercise 1: Dynamic PVC for PostgreSQL

What we’re doing: Create storage dynamically and deploy PostgreSQL against it.

# Step 1: Create a PVC (StorageClass not specified → uses default)
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
  namespace: storage-lab
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 256Mi
EOF

# Check the PVC — starts Pending, then Bound
kubectl get pvc postgres-data -n storage-lab -w

Expected progression:

NAME            STATUS    VOLUME   CAPACITY   ACCESS MODES
postgres-data   Pending                                       ← before pod
postgres-data   Bound     pvc-xxx  256Mi      RWO             ← after pod scheduled

📝 Note: On Minikube with Immediate binding mode, the PVC may bind immediately. On cloud clusters with WaitForFirstConsumer, it stays Pending until a pod is scheduled.

# Step 2: Deploy PostgreSQL
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
  namespace: storage-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      volumes:
      - name: pgdata
        persistentVolumeClaim:
          claimName: postgres-data

      containers:
      - name: postgres
        image: postgres:16-alpine
        ports:
        - containerPort: 5432
        env:
        - name: POSTGRES_DB
          value: "appdb"
        - name: POSTGRES_USER
          value: "appuser"
        - name: POSTGRES_PASSWORD
          value: "labpassword"
        - name: PGDATA
          value: /var/lib/postgresql/data/pgdata
        volumeMounts:
        - name: pgdata
          mountPath: /var/lib/postgresql/data
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "200m"
---
apiVersion: v1
kind: Service
metadata:
  name: postgres-svc
  namespace: storage-lab
spec:
  selector:
    app: postgres
  ports:
  - port: 5432
    targetPort: 5432
EOF

kubectl rollout status deployment/postgres -n storage-lab

Exercise 2: Write Data, Restart Pod, Verify Persistence

What we’re doing: Prove that data in a PVC-backed PostgreSQL survives pod restarts.

# Step 1: Write data to PostgreSQL
POD=$(kubectl get pods -n storage-lab -l app=postgres -o name | head -1)

kubectl exec -n storage-lab $POD -- \
  psql -U appuser -d appdb -c "
    CREATE TABLE users (id SERIAL PRIMARY KEY, name TEXT, created_at TIMESTAMPTZ DEFAULT NOW());
    INSERT INTO users (name) VALUES ('Alice'), ('Bob'), ('Carol');
    SELECT * FROM users;
  "

Expected output:

CREATE TABLE
INSERT 0 3
 id | name  |          created_at
----+-------+-------------------------------
  1 | Alice | 2026-07-15 05:45:00.123456+00
  2 | Bob   | 2026-07-15 05:45:00.123456+00
  3 | Carol | 2026-07-15 05:45:00.123456+00
(3 rows)

# Step 2: Delete the pod (Deployment will recreate it)
kubectl delete pod -n storage-lab -l app=postgres

# Wait for new pod
kubectl rollout status deployment/postgres -n storage-lab

# Step 3: Query the NEW pod — data must still be there
NEW_POD=$(kubectl get pods -n storage-lab -l app=postgres -o name | head -1)
kubectl exec -n storage-lab $NEW_POD -- \
  psql -U appuser -d appdb -c "SELECT * FROM users;"

Expected (same data in new pod):

 id | name  |          created_at
----+-------+-------------------------------
  1 | Alice | 2026-07-15 05:45:00.123456+00
  2 | Bob   | 2026-07-15 05:45:00.123456+00
  3 | Carol | 2026-07-15 05:45:00.123456+00
(3 rows)

💡 What just happened? The new pod mounted the same PVC, which points to the same PersistentVolume. PostgreSQL found its data files at /var/lib/postgresql/data/pgdata exactly where it left them. Pod identity is irrelevant — the data lives in the PV.

Exercise 3: StatefulSet with volumeClaimTemplates

What we’re doing: Deploy a StatefulSet where each pod gets its OWN PVC automatically.

cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: data-store
  namespace: storage-lab
spec:
  serviceName: data-store
  replicas: 3
  selector:
    matchLabels:
      app: data-store
  template:
    metadata:
      labels:
        app: data-store
    spec:
      containers:
      - name: store
        image: busybox
        command: ["sh", "-c", "echo \"Pod: $HOSTNAME\" > /data/identity.txt && sleep 3600"]
        volumeMounts:
        - name: data
          mountPath: /data
        resources:
          limits:
            memory: "16Mi"
            cpu: "50m"
  volumeClaimTemplates:           # ← Creates one PVC per pod
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 64Mi
EOF

kubectl rollout status statefulset/data-store -n storage-lab

Watch three separate PVCs get created:

kubectl get pvc -n storage-lab

Expected:

NAME                  STATUS   VOLUME       CAPACITY   ACCESS MODES
data-data-store-0     Bound    pvc-aaa      64Mi       RWO    ← for data-store-0
data-data-store-1     Bound    pvc-bbb      64Mi       RWO    ← for data-store-1
data-data-store-2     Bound    pvc-ccc      64Mi       RWO    ← for data-store-2
postgres-data         Bound    pvc-ddd      256Mi      RWO

Verify each pod has its own isolated storage:

for i in 0 1 2; do
  echo "=== data-store-$i ==="
  kubectl exec -n storage-lab data-store-$i -- cat /data/identity.txt
done

Expected:

=== data-store-0 ===
Pod: data-store-0

=== data-store-1 ===
Pod: data-store-1

=== data-store-2 ===
Pod: data-store-2

Now delete and re-create data-store-1:

kubectl delete pod data-store-1 -n storage-lab
kubectl wait pod data-store-1 -n storage-lab --for=condition=Ready --timeout=60s

# It came back with the same PVC — identity preserved
kubectl exec -n storage-lab data-store-1 -- cat /data/identity.txt
# Pod: data-store-1  ← same content from the persistent volume

Exercise 4: StatefulSet Deletion and PVC Lifecycle

What we’re doing: Observe what happens to PVCs when a StatefulSet is deleted.

# Delete the StatefulSet (NOT the PVCs)
kubectl delete statefulset data-store -n storage-lab

# Check PVCs — they still exist!
kubectl get pvc -n storage-lab | grep data-store

Expected:

data-data-store-0   Bound   ...   ← Still there!
data-data-store-1   Bound   ...   ← Still there!
data-data-store-2   Bound   ...   ← Still there!

💡 Key insight: Deleting a StatefulSet does NOT delete its PVCs. This is intentional — it prevents accidental data loss. You must manually delete PVCs with kubectl delete pvc. This is a common source of surprise (and saved data) in production.

# Clean up PVCs explicitly
kubectl delete pvc -n storage-lab data-data-store-0 data-data-store-1 data-data-store-2

Exercise 5: Inspect Storage Details

# See all PVs in the cluster
kubectl get pv

# Which PVC is bound to which PV?
kubectl get pv -o custom-columns=\
'NAME:.metadata.name,CAPACITY:.spec.capacity.storage,STATUS:.status.phase,CLAIM:.spec.claimRef.name,POLICY:.spec.persistentVolumeReclaimPolicy'

# See StorageClass details
kubectl describe storageclass standard  # Minikube's default

# See full PVC details
kubectl describe pvc postgres-data -n storage-lab

🔥 Break It! Challenge

What happens when a pod references a PVC that doesn’t exist?

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: missing-pvc-pod
  namespace: storage-lab
spec:
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: this-pvc-does-not-exist
  containers:
  - name: app
    image: nginx:1.25
    volumeMounts:
    - name: data
      mountPath: /data
    resources:
      limits:
        memory: "32Mi"
        cpu: "50m"
EOF

# Pod stays in Pending
kubectl get pod missing-pvc-pod -n storage-lab

# See the exact error
kubectl describe pod missing-pvc-pod -n storage-lab | tail -10

Expected Events:

Warning  FailedMount   Unable to attach or mount volumes: ... persistentvolumeclaim "this-pvc-does-not-exist" not found

Fix it by creating the PVC:

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: this-pvc-does-not-exist
  namespace: storage-lab
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 64Mi
EOF

# Pod should start now
kubectl get pod missing-pvc-pod -n storage-lab -w

Cleanup

# Delete everything in the lab namespace
kubectl delete deployment postgres -n storage-lab
kubectl delete pvc postgres-data -n storage-lab
kubectl delete pvc this-pvc-does-not-exist -n storage-lab
kubectl delete pod missing-pvc-pod -n storage-lab
kubectl delete svc postgres-svc -n storage-lab

kubectl config set-context --current --namespace=default
kubectl delete namespace storage-lab

What We Learned

Chapter 9: Namespaces, RBAC, and Multi-Tenancy

⏱️ Total chapter time: ~55 min (25 min reading + 30 min lab)

After this chapter, you will be able to: Organize workloads into namespaces, control access using Roles and RoleBindings, create service accounts for pods, and design a basic multi-tenant cluster layout.

What’s Inside

Prerequisites

• Comfortable with kubectl (Chapter 2)
• minikube status shows Running

9.1 Namespaces — Cluster Partitioning

⏱️ ~5 min read

TL;DR: A namespace is a virtual cluster inside a physical cluster. Resources in different namespaces are isolated by name but share the same underlying nodes and networking. Use namespaces to separate teams, environments, or applications.

What Namespaces Do (and Don’t) Isolate

graph TB
    subgraph "Kubernetes Cluster"
        subgraph "namespace: team-a"
            PA1[web-deploy]
            PA2[web-svc]
            PA3[db-pvc]
        end
        subgraph "namespace: team-b"
            PB1[web-deploy]
            PB2[web-svc]
            PB3[db-pvc]
        end
        subgraph "namespace: monitoring"
            PM1[prometheus]
            PM2[grafana]
        end
        N1[Node 1] & N2[Node 2]
    end

Namespaces isolate:

• Resource names (two web-svc Services can coexist in different namespaces)
• RBAC policies (permissions are namespace-scoped)
• Resource quotas and limits
• Network policies (with a CNI that supports them)

Namespaces do NOT isolate:

• Nodes — all pods share the same node pool
• Node-level resources (CPU, RAM) — workloads from all namespaces compete
• Network traffic by default — cross-namespace pod communication is allowed

System Namespaces

kubectl get namespaces

Default namespaces every cluster starts with:

⚠️ Warning: Never deploy your own applications into kube-system. If you break something there, the entire cluster’s control plane can be affected.

Namespace-Scoped vs Cluster-Scoped Resources

Not everything lives in a namespace:

# Check if a resource type is namespace-scoped
kubectl api-resources --namespaced=true    # Namespace-scoped
kubectl api-resources --namespaced=false   # Cluster-scoped

Creating and Using Namespaces

# Create
kubectl create namespace team-a

# Declarative (preferred)
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: team-b
  labels:
    team: b
    environment: production
EOF

# Switch default namespace in your context
kubectl config set-context --current --namespace=team-a

# Or use -n per command
kubectl get pods -n kube-system

# See all resources in all namespaces
kubectl get pods -A        # Short for --all-namespaces
kubectl get deploy -A

Namespace Resource Quotas

Prevent one team’s namespace from consuming all cluster resources:

# resource-quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: team-a
spec:
  hard:
    requests.cpu: "4"           # Total CPU requests in namespace
    requests.memory: 4Gi        # Total memory requests
    limits.cpu: "8"             # Total CPU limits
    limits.memory: 8Gi          # Total memory limits
    pods: "20"                  # Max number of pods
    services: "10"              # Max number of Services
    persistentvolumeclaims: "5" # Max PVCs

kubectl apply -f resource-quota.yaml

# See current usage vs quota
kubectl describe resourcequota team-quota -n team-a

Try It

# Create two namespaces
kubectl create namespace team-a
kubectl create namespace team-b

# Create same-named resources in each — no conflict
kubectl run nginx --image=nginx:1.25 -n team-a
kubectl run nginx --image=nginx:1.25 -n team-b

# Both exist independently
kubectl get pods -n team-a
kubectl get pods -n team-b

# Cross-namespace DNS still works (Service required for DNS)
# From a pod in team-a:
# curl http://my-svc.team-b.svc.cluster.local

# Cleanup
kubectl delete namespace team-a team-b

Key Takeaways

✅ Quick Check

Q1: Team A’s web-svc Service is in namespace team-a. Team B’s pod needs to reach it. What DNS name do they use?

Q2: You apply a ResourceQuota to namespace dev with pods: "5". A developer tries to create a 6th pod. What happens?

Q3: Can you move a running pod from one namespace to another?

9.2 RBAC — Roles, ClusterRoles, and Bindings

⏱️ ~8 min read

TL;DR: RBAC (Role-Based Access Control) controls who can do what on which resources. A Role defines permissions. A RoleBinding says “this user/group/serviceaccount gets those permissions.” It’s the primary security mechanism for controlling kubectl access.

The RBAC Model

graph LR
    SUBJECT["Subject\n(User / Group / ServiceAccount)"] -->|bound by| RB[RoleBinding]
    RB --> ROLE["Role\n(set of permissions)"]
    ROLE --> RESOURCES["Resources\n(pods, secrets, deployments...)"]
    ROLE --> VERBS["Verbs\n(get, list, create, delete...)"]

Four object types:

Role — Define Permissions

# role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: team-a       # Only applies within team-a namespace
rules:
- apiGroups: [""]         # "" = core API group (pods, services, configmaps)
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]
  
- apiGroups: ["apps"]     # apps API group (deployments, replicasets)
  resources: ["deployments"]
  verbs: ["get", "list"]

Available verbs:

apiGroups reference:

# Find the API group for a resource
kubectl api-resources | grep deployment
# NAME          SHORTNAMES  APIVERSION   NAMESPACED  KIND
# deployments   deploy      apps/v1      true        Deployment
#                           ^^^^
#                           apiGroup = "apps"

kubectl api-resources | grep pod
# pods   po   v1    true   Pod
#              ^
#              v1 = core group → apiGroups: [""]

ClusterRole — Cluster-Wide or Reusable

# clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-reader           # No namespace — cluster-scoped
rules:
- apiGroups: [""]
  resources: ["nodes"]        # Nodes are cluster-scoped
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]

ClusterRoles can also be bound per-namespace via a RoleBinding (not ClusterRoleBinding) — useful for defining common permission sets once and reusing them:

# Bind the ClusterRole "view" to a user, but only in namespace "team-a"
kind: RoleBinding          # ← RoleBinding, not ClusterRoleBinding
subjects:
- kind: User
  name: alice
roleRef:
  kind: ClusterRole        # ← References a ClusterRole
  name: view               # Built-in ClusterRole

RoleBinding — Grant Permissions

# rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: alice-pod-reader
  namespace: team-a         # Only applies in team-a
subjects:
- kind: User               # User | Group | ServiceAccount
  name: alice              # The username
  apiGroup: rbac.authorization.k8s.io

roleRef:
  kind: Role               # Role | ClusterRole
  name: pod-reader         # Must exist in same namespace (for Role)
  apiGroup: rbac.authorization.k8s.io

Subject kinds:

# User (human)
- kind: User
  name: alice
  apiGroup: rbac.authorization.k8s.io

# Group (all users in a group)
- kind: Group
  name: team-a-developers
  apiGroup: rbac.authorization.k8s.io

# ServiceAccount (for pods — covered in section 9.3)
- kind: ServiceAccount
  name: my-service-account
  namespace: team-a         # ServiceAccounts are namespaced

Built-in ClusterRoles

Kubernetes ships with useful ClusterRoles you can reuse:

# Grant a user read-only access to a specific namespace
kubectl create rolebinding alice-viewer \
  --clusterrole=view \
  --user=alice \
  --namespace=team-a

# Grant cluster-admin to a user (use with extreme caution)
kubectl create clusterrolebinding alice-admin \
  --clusterrole=cluster-admin \
  --user=alice

Checking Permissions

# Can I do this?
kubectl auth can-i get pods --namespace=team-a
# yes

# Can another user do this?
kubectl auth can-i get secrets --namespace=team-a --as=alice
# no

# What can alice do?
kubectl auth can-i --list --as=alice --namespace=team-a

# Audit: see all RoleBindings in a namespace
kubectl get rolebindings -n team-a -o wide
kubectl get clusterrolebindings -o wide | grep alice

Try It

kubectl create namespace rbac-demo

# Create a Role that allows reading pods only
cat <<'EOF' | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: rbac-demo
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: test-user-pod-reader
  namespace: rbac-demo
subjects:
- kind: User
  name: test-user
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io
EOF

# Check what test-user can do
kubectl auth can-i get pods --namespace=rbac-demo --as=test-user       # yes
kubectl auth can-i delete pods --namespace=rbac-demo --as=test-user    # no
kubectl auth can-i get secrets --namespace=rbac-demo --as=test-user    # no
kubectl auth can-i get pods --namespace=default --as=test-user         # no (only in rbac-demo)

# Cleanup
kubectl delete namespace rbac-demo

Key Takeaways

✅ Quick Check

Q1: You want a pod to be able to list ConfigMaps in its own namespace. What subject type do you use in the RoleBinding?

Q2: You grant a user the view ClusterRole via a ClusterRoleBinding. What can they see?

Q3: A developer accidentally deletes a production Deployment. You add delete to their Role’s allowed verbs list after the fact. What should you actually do?

9.3 ServiceAccounts — Pod Identities

⏱️ ~6 min read

TL;DR: A ServiceAccount is a non-human identity for pods. When a pod needs to call the Kubernetes API (e.g., a CI runner listing pods, or an operator watching CRDs), it authenticates using its ServiceAccount’s token. By default, every pod gets the default ServiceAccount, which has almost no permissions.

Why Pods Need Identities

Some workloads need to interact with the Kubernetes API:

Without a ServiceAccount with the right Role, these tools fail with 403 Forbidden from the API server.

The Default ServiceAccount

Every namespace has a default ServiceAccount. All pods use it automatically unless you specify otherwise:

# See the default ServiceAccount
kubectl get serviceaccount default -n default

# Every pod has a token mounted automatically
kubectl run tmp --image=nginx --rm -it --restart=Never -- \
  cat /var/run/secrets/kubernetes.io/serviceaccount/token

The default SA has minimal permissions — by design. Don’t add permissions to it (every pod in the namespace inherits them).

Creating and Using a ServiceAccount

# serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: pod-lister
  namespace: monitoring

# role.yaml — what the SA is allowed to do
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-list-role
  namespace: default
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]

# rolebinding.yaml — connect SA to Role
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-lister-binding
  namespace: default
subjects:
- kind: ServiceAccount
  name: pod-lister
  namespace: monitoring          # SA is in 'monitoring' namespace
roleRef:
  kind: Role
  name: pod-list-role
  apiGroup: rbac.authorization.k8s.io

# pod using the ServiceAccount
spec:
  serviceAccountName: pod-lister  # Must be in same namespace as pod
  automountServiceAccountToken: true  # Default is true

The Mounted Token

When a pod uses a ServiceAccount, Kubernetes mounts a short-lived JWT token as a projected volume:

/var/run/secrets/kubernetes.io/serviceaccount/
├── token      ← JWT token to authenticate with the API server
├── ca.crt     ← CA cert to verify the API server's TLS
└── namespace  ← Current namespace name

# Inside a pod — call the K8s API using the mounted token
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)

curl -s \
  --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  -H "Authorization: Bearer $TOKEN" \
  https://kubernetes.default.svc/api/v1/namespaces/$NAMESPACE/pods | \
  python3 -m json.tool | grep '"name"'

Disabling Token Auto-Mount

For pods that don’t need API access, disable the token mount to reduce attack surface:

spec:
  automountServiceAccountToken: false  # No token mounted
  containers:
  - name: app
    image: nginx:1.25

Or disable it on the ServiceAccount itself (applies to all pods using it):

apiVersion: v1
kind: ServiceAccount
metadata:
  name: no-api-access
automountServiceAccountToken: false

Cross-Namespace Access

A pod can only use ServiceAccounts in its own namespace. For cross-namespace API access, bind the SA to Roles in other namespaces:

# Pod in 'monitoring' namespace has SA 'prometheus'
# Bind it to the 'view' ClusterRole in EVERY namespace:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus-cluster-view
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitoring          # SA lives in monitoring
roleRef:
  kind: ClusterRole
  name: view                     # Can view resources cluster-wide
  apiGroup: rbac.authorization.k8s.io

Try It

kubectl create namespace sa-demo

# Create ServiceAccount, Role, and RoleBinding
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: api-reader
  namespace: sa-demo
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: sa-demo
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: api-reader-binding
  namespace: sa-demo
subjects:
- kind: ServiceAccount
  name: api-reader
  namespace: sa-demo
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io
EOF

# Test: can the SA get pods?
kubectl auth can-i get pods \
  --namespace=sa-demo \
  --as=system:serviceaccount:sa-demo:api-reader      # yes

kubectl auth can-i delete pods \
  --namespace=sa-demo \
  --as=system:serviceaccount:sa-demo:api-reader      # no

kubectl auth can-i get secrets \
  --namespace=sa-demo \
  --as=system:serviceaccount:sa-demo:api-reader      # no

# Cleanup
kubectl delete namespace sa-demo

Key Takeaways

✅ Quick Check

Q1: An operator pod needs to watch and update Deployments across all namespaces. What RBAC objects do you create?

Q2: A pod uses the default ServiceAccount. An attacker compromises the pod. What can they do with the mounted token?

Q3: You create a ServiceAccount named deployer in namespace staging. Can a pod in namespace production use it?

9.4 Multi-Tenancy Patterns

⏱️ ~5 min read

TL;DR: Multi-tenancy means multiple teams or customers share one cluster safely. The standard Kubernetes approach uses namespaces + RBAC + ResourceQuotas + NetworkPolicy. For stronger isolation, use separate clusters.

Isolation Levels

graph LR
    subgraph "Soft Tenancy (Namespace-per-Team)"
        N1[namespace: team-a\nRBAC + Quota]
        N2[namespace: team-b\nRBAC + Quota]
        N1 & N2 --> CLUSTER1[Shared Cluster\nShared Nodes]
    end

    subgraph "Hard Tenancy (Cluster-per-Team)"
        CLUSTER2[Cluster A\nteam-a only]
        CLUSTER3[Cluster B\nteam-b only]
    end

The Namespace-Per-Team Pattern

The most common production approach:

# Per-team namespace with labels
apiVersion: v1
kind: Namespace
metadata:
  name: team-payments
  labels:
    team: payments
    environment: production
    cost-center: "123"

For each namespace, apply the standard set:

namespace/team-payments
├── ResourceQuota       ← CPU/memory/pod limits
├── LimitRange          ← Default requests/limits for pods without them
├── NetworkPolicy       ← Block cross-team traffic
├── Role + RoleBinding  ← Team-specific permissions
└── ServiceAccount      ← Per-team app identity

LimitRange — Default Resource Bounds

ResourceQuota limits the total for a namespace. LimitRange sets defaults for individual pods that forget to set resources:

# limitrange.yaml
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: team-payments
spec:
  limits:
  - type: Container
    default:              # Applied if container has no limits
      cpu: "500m"
      memory: "256Mi"
    defaultRequest:       # Applied if container has no requests
      cpu: "100m"
      memory: "128Mi"
    max:                  # No container in this namespace can exceed
      cpu: "2"
      memory: "2Gi"
    min:                  # No container can go below
      cpu: "50m"
      memory: "32Mi"

Without LimitRange, pods without resource specs become BestEffort QoS class — first to be evicted.

NetworkPolicy — Traffic Isolation

By default, all pods in a cluster can talk to all other pods. NetworkPolicy restricts this:

# deny-all-ingress.yaml — Block all incoming traffic by default
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
  namespace: team-payments
spec:
  podSelector: {}          # Applies to all pods in this namespace
  policyTypes:
  - Ingress
  # No ingress rules = deny all incoming traffic

---
# allow-same-namespace.yaml — Allow traffic within the namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-same-namespace
  namespace: team-payments
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector: {}      # Any pod in the same namespace

⚠️ Warning: NetworkPolicy requires a CNI plugin that supports it (Calico, Cilium, Weave). Minikube’s default CNI does NOT enforce NetworkPolicy. Apply them anyway — they’ll be enforced when you deploy to a production cluster with the right CNI.

Namespace Template (Standard Starter)

Here’s a complete namespace setup for a team:

# Script to create a fully configured team namespace
TEAM=payments
TEAM_NS=team-$TEAM

kubectl create namespace $TEAM_NS

# Resource limits
kubectl apply -f - <<EOF
apiVersion: v1
kind: ResourceQuota
metadata:
  name: ${TEAM}-quota
  namespace: ${TEAM_NS}
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 4Gi
    limits.cpu: "8"
    limits.memory: 8Gi
    pods: "20"
    services: "10"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: ${TEAM}-limits
  namespace: ${TEAM_NS}
spec:
  limits:
  - type: Container
    default:
      cpu: "500m"
      memory: "256Mi"
    defaultRequest:
      cpu: "100m"
      memory: "128Mi"
EOF

# Team ServiceAccount
kubectl create serviceaccount ${TEAM}-sa -n ${TEAM_NS}

# Bind 'edit' ClusterRole to team SA in their namespace
kubectl create rolebinding ${TEAM}-edit \
  --clusterrole=edit \
  --serviceaccount=${TEAM_NS}:${TEAM}-sa \
  -n ${TEAM_NS}

Key Takeaways

✅ Quick Check

Q1: Team A and Team B share a cluster with namespace-per-team. Team A’s pod sends a request directly to Team B’s pod IP. Is this blocked?

Q2: A developer in namespace team-a runs kubectl get pods -n team-b. They have the edit role in team-a. Can they see team-b’s pods?

Q3: A pod has no resource requests defined. The namespace has a LimitRange with defaultRequest.cpu: 100m. What CPU request does the pod actually have?

Lab: Lock Down a Namespace

⏱️ ~30 min hands-on

Objectives

• Create a team namespace with ResourceQuota and LimitRange
• Create a dev-user Role that allows read-write but not delete
• Verify the user can’t access other namespaces
• Create a ServiceAccount for an app pod to call the K8s API
• Test that ResourceQuota blocks over-limit pod creation
• Observe the LimitRange injecting default resources

Setup

# Create the team namespace
kubectl create namespace team-alpha

# Verify it exists
kubectl get namespace team-alpha

Exercise 1: Apply ResourceQuota and LimitRange

What we’re doing: Set namespace-level resource boundaries.

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ResourceQuota
metadata:
  name: alpha-quota
  namespace: team-alpha
spec:
  hard:
    requests.cpu: "2"
    requests.memory: 2Gi
    limits.cpu: "4"
    limits.memory: 4Gi
    pods: "5"
    services: "3"
    persistentvolumeclaims: "2"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: alpha-limits
  namespace: team-alpha
spec:
  limits:
  - type: Container
    default:
      cpu: "200m"
      memory: "128Mi"
    defaultRequest:
      cpu: "100m"
      memory: "64Mi"
    max:
      cpu: "1"
      memory: "512Mi"
EOF

# See the quota
kubectl describe resourcequota alpha-quota -n team-alpha
kubectl describe limitrange alpha-limits -n team-alpha

Expected quota output:

Name:              alpha-quota
Namespace:         team-alpha
Resource           Used   Hard
--------           ----   ----
limits.cpu         0      4
limits.memory      0      4Gi
pods               0      5
requests.cpu       0      2
requests.memory    0      2Gi
...

Exercise 2: RBAC for a Developer

What we’re doing: Create a Role that lets a developer manage deployments and view pods/logs, but NOT delete anything or touch secrets.

cat <<'EOF' | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: developer
  namespace: team-alpha
rules:
# Pods: read and exec, but not delete
- apiGroups: [""]
  resources: ["pods", "pods/log", "pods/exec"]
  verbs: ["get", "list", "watch"]

# Deployments: full management (but not delete)
- apiGroups: ["apps"]
  resources: ["deployments", "replicasets"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]

# Services: create and view
- apiGroups: [""]
  resources: ["services"]
  verbs: ["get", "list", "watch", "create", "update", "patch"]

# ConfigMaps: full access (not Secrets)
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

# Events: view for debugging
- apiGroups: [""]
  resources: ["events"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: dev-alice
  namespace: team-alpha
subjects:
- kind: User
  name: alice
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: developer
  apiGroup: rbac.authorization.k8s.io
EOF

echo "RBAC applied. Testing permissions..."

Test the permissions systematically:

# What alice CAN do
echo "=== Alice's allowed operations ==="
kubectl auth can-i get pods -n team-alpha --as=alice                      # yes
kubectl auth can-i list deployments -n team-alpha --as=alice              # yes
kubectl auth can-i create deployments -n team-alpha --as=alice            # yes
kubectl auth can-i update deployments -n team-alpha --as=alice            # yes
kubectl auth can-i get configmaps -n team-alpha --as=alice                # yes

echo ""
echo "=== What alice CANNOT do ==="
kubectl auth can-i delete pods -n team-alpha --as=alice                   # no
kubectl auth can-i delete deployments -n team-alpha --as=alice            # no
kubectl auth can-i get secrets -n team-alpha --as=alice                   # no
kubectl auth can-i get pods -n default --as=alice                         # no (wrong namespace)
kubectl auth can-i get pods -n kube-system --as=alice                     # no

Expected output:

=== Alice's allowed operations ===
yes
yes
yes
yes
yes

=== What alice CANNOT do ===
no
no
no
no
no

Exercise 3: Namespace Isolation Verification

# Create resources in other namespaces
kubectl run nginx-default --image=nginx:1.25 -n default

# Alice cannot see pods in other namespaces
kubectl auth can-i list pods -n default --as=alice          # no
kubectl auth can-i list pods -n kube-system --as=alice      # no
kubectl auth can-i list pods --all-namespaces --as=alice    # no

# But cluster-level admin can
kubectl auth can-i list pods -n default                     # yes (you're admin)
kubectl auth can-i list pods -n team-alpha --as=alice       # yes (alice's namespace)

# Cleanup
kubectl delete pod nginx-default -n default

Exercise 4: ServiceAccount for App API Access

What we’re doing: Create an app that reads ConfigMaps from its own namespace via the K8s API.

# Create the ServiceAccount
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: config-reader
  namespace: team-alpha
automountServiceAccountToken: true
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: configmap-reader
  namespace: team-alpha
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: config-reader-binding
  namespace: team-alpha
subjects:
- kind: ServiceAccount
  name: config-reader
  namespace: team-alpha
roleRef:
  kind: Role
  name: configmap-reader
  apiGroup: rbac.authorization.k8s.io
EOF

# Verify the SA can list configmaps
kubectl auth can-i list configmaps -n team-alpha \
  --as=system:serviceaccount:team-alpha:config-reader          # yes

kubectl auth can-i list secrets -n team-alpha \
  --as=system:serviceaccount:team-alpha:config-reader          # no

# Deploy a pod using this ServiceAccount
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-settings
  namespace: team-alpha
data:
  REGION: "us-east-1"
  TIER: "premium"
---
apiVersion: v1
kind: Pod
metadata:
  name: api-caller
  namespace: team-alpha
spec:
  serviceAccountName: config-reader
  containers:
  - name: app
    image: curlimages/curl:latest
    command: ["sh", "-c", "sleep 3600"]
    resources:
      limits:
        memory: "32Mi"
        cpu: "100m"
EOF

kubectl wait pod api-caller -n team-alpha --for=condition=Ready --timeout=60s

# Call the K8s API from inside the pod using the mounted SA token
kubectl exec -n team-alpha api-caller -- sh -c '
  TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
  NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)
  
  echo "=== Listing ConfigMaps (allowed) ==="
  curl -s \
    --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
    -H "Authorization: Bearer $TOKEN" \
    "https://kubernetes.default.svc/api/v1/namespaces/$NAMESPACE/configmaps" \
    | grep -o "\"name\":\"[^\"]*\"" | head -5
  
  echo ""
  echo "=== Listing Secrets (forbidden) ==="
  curl -s \
    --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
    -H "Authorization: Bearer $TOKEN" \
    "https://kubernetes.default.svc/api/v1/namespaces/$NAMESPACE/secrets" \
    | grep -o "\"reason\":\"[^\"]*\""
'

Expected output:

=== Listing ConfigMaps (allowed) ===
"name":"app-settings"
"name":"kube-root-ca.crt"

=== Listing Secrets (forbidden) ===
"reason":"Forbidden"

Exercise 5: ResourceQuota in Action

What we’re doing: Prove the quota blocks pod creation over the limit.

# Create pods up to the limit (quota allows 5 pods)
for i in 1 2 3 4; do
  kubectl run pod-$i --image=nginx:1.25 -n team-alpha
done

# Check quota usage
kubectl describe resourcequota alpha-quota -n team-alpha | grep -E "pods|cpu|memory"

Expected (partial):

pods    4    5   ← 4 used, limit 5

# Create one more (up to the limit)
kubectl run pod-5 --image=nginx:1.25 -n team-alpha

# Try to exceed — this should fail
kubectl run pod-6 --image=nginx:1.25 -n team-alpha

Expected error:

Error from server (Forbidden): pods "pod-6" is forbidden: exceeded quota: alpha-quota,
requested: pods=1, used: pods=5, limited: pods=5

Exercise 6: LimitRange Injects Defaults

What we’re doing: Create a pod without resource specs — watch LimitRange inject them.

# Pod with NO resource requests or limits
cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: no-resources-pod
  namespace: team-alpha
spec:
  containers:
  - name: app
    image: busybox
    command: ["sleep", "60"]
EOF

# Check what resources it actually got
kubectl get pod no-resources-pod -n team-alpha \
  -o jsonpath='{.spec.containers[0].resources}' | python3 -m json.tool

Expected (LimitRange injected these):

{
    "limits": {
        "cpu": "200m",
        "memory": "128Mi"
    },
    "requests": {
        "cpu": "100m",
        "memory": "64Mi"
    }
}

The pod was created without specifying any resources, but LimitRange automatically injected the namespace defaults.

🔥 Break It! Challenge

What happens when you try to create a pod that EXCEEDS the LimitRange’s max?

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: too-many-resources
  namespace: team-alpha
spec:
  containers:
  - name: app
    image: nginx:1.25
    resources:
      requests:
        memory: "1Gi"    # LimitRange max is 512Mi
        cpu: "2"         # LimitRange max is 1 CPU
      limits:
        memory: "2Gi"    # Way over max
        cpu: "4"         # Way over max
EOF

Expected error:

Error from server (Forbidden): error when creating "STDIN": pods "too-many-resources" is forbidden:
[maximum memory usage per Container is 512Mi, but limit is 2Gi.,
 maximum cpu usage per Container is 1, but limit is 4.]

The admission controller rejects the pod at creation time — it never reaches the scheduler.

Cleanup

kubectl delete namespace team-alpha
kubectl config set-context --current --namespace=default

What We Learned

Chapter 10: Health Checks — Probes and Graceful Shutdown

⏱️ Total chapter time: ~50 min (22 min reading + 28 min lab)

After this chapter, you will be able to: Configure liveness, readiness, and startup probes so Kubernetes knows when your app is healthy, and implement graceful shutdown so pods drain cleanly without dropping requests.

What’s Inside

Prerequisites

• Completed Chapter 3 (Pods) and Chapter 4 (Deployments)
• minikube status shows Running

10.1 Why Probes Exist — The Problem They Solve

⏱️ ~4 min read

TL;DR: Kubernetes can’t read your app’s source code. Without probes, it only knows whether the container process is running — not whether your app is actually healthy. Probes are how you teach Kubernetes what “healthy” means for your specific application.

The Two Failure Modes Probes Solve

1. The Zombie Process

graph LR
    APP[App process\nRunning ✅\nBut deadlocked 🧟] --> K8S[kubectl get pods\nStatus: Running\nREADY: 1/1]
    K8S --> TRAFFIC[Traffic still\nrouted here ❌\nRequests timing out]

The container is running, but the app is stuck in a deadlock, holding all worker threads, or has a corrupted internal state. Kubernetes doesn’t know — it just sees a live process.

Liveness probes fix this: they detect unhealthy-but-running apps and trigger a container restart.

2. The Not-Ready-Yet App

graph LR
    DEPLOY[Deployment\nrolling update] --> NEW[New pod\nProcess started\nbut warming up]
    NEW -->|"K8s sends traffic immediately\nbefore app is ready"| FAIL[User gets\n500 errors ❌\nduring warmup]

A new pod starts, the container process launches, and Kubernetes immediately routes traffic to it. But your app needs 30 seconds to load its cache, connect to the database, and finish initializing.

Readiness probes fix this: the pod only receives traffic when the probe passes.

The Three Probe Types