18.1 The Debugging Mental Model

⏱️ ~8 min read

TL;DR: Kubernetes debugging is systematic, not random. Follow a three-layer model: Is the pod running? Is the network reachable? Is the data accessible? Answer each layer in order before jumping to the next.

The Three-Layer Mental Model

Almost every Kubernetes problem falls into one of three layers:

graph TD
    A["🧩 Layer 1: Is the Pod Running?"] -->|yes| B["🌐 Layer 2: Is the Network Reachable?"]
    B -->|yes| C["💾 Layer 3: Is the Data Accessible?"]
    A -->|no| D["Debug pod lifecycle"]
    B -->|no| E["Debug services, DNS, NetworkPolicy"]
    C -->|no| F["Debug volumes, permissions"]

    style D fill:#ef4444,color:#fff
    style E fill:#f59e0b,color:#000
    style F fill:#8b5cf6,color:#fff

Never skip a layer. If pods aren’t running, there’s no point debugging the service. If the service is broken, there’s no point debugging the application logic.

The Universal Debugging Sequence

When something is broken, always run these commands in order:

# Step 1: Get the overall picture
kubectl get pods -n <namespace>
kubectl get events -n <namespace> --sort-by='.lastTimestamp' | tail -20

# Step 2: Zoom into the broken pod
kubectl describe pod <pod-name> -n <namespace>

# Step 3: Read the container logs
kubectl logs <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --previous  # logs from last crash

# Step 4: Get inside (if the pod is running but misbehaving)
kubectl exec -it <pod-name> -n <namespace> -- /bin/sh

# Step 5: Check related resources
kubectl describe service <svc-name> -n <namespace>
kubectl describe pvc <pvc-name> -n <namespace>

💡 Tip: kubectl describe is your best friend. The Events section at the bottom shows the timestamped history of what Kubernetes tried to do and where it failed.

Reading kubectl describe pod — What to Look At

kubectl describe pod my-broken-pod

Here’s what each section tells you:

Name:         my-broken-pod
Namespace:    default
Node:         minikube/192.168.49.2     ← which node it's on (empty if Pending)
Status:       Pending                   ← overall pod phase
Conditions:
  Initialized   True   ← init containers done
  Ready         False  ← is the pod serving traffic?
  PodScheduled  True   ← scheduler found a node

Events:
  Type     Reason            Age   From               Message
  ----     ------            ---   ----               -------
  Warning  FailedScheduling  5s    default-scheduler  0/1 nodes have enough memory
  Normal   Scheduled         4s    default-scheduler  assigned to minikube
  Normal   Pulling           3s    kubelet            Pulling image "myapp:v2"
  Warning  Failed            2s    kubelet            Failed to pull image: not found

The Events section reads like a log — newest at the bottom, problems labeled Warning. Start here.

Using kubectl Events for Cluster-Wide Visibility

# All events in the cluster, sorted by time
kubectl get events --all-namespaces --sort-by='.lastTimestamp' | tail -30

# Only Warning events
kubectl get events --all-namespaces --field-selector type=Warning

# Watch events live
kubectl get events --watch

The Ephemeral Debug Container

If a pod is running but has no shell (distroless images, scratch-based images), you can’t exec into it. Use an ephemeral debug container:

# Inject a debug container into a running pod (K8s 1.23+)
kubectl debug -it <pod-name> --image=busybox:1.36 --target=<container-name>

This starts a busybox container in the same pod namespace — you get a shell that shares the same network and can see the same processes.

# Alternatively, copy the pod and add a debug sidecar
kubectl debug <pod-name> -it --copy-to=debug-pod --image=busybox:1.36

Node-Level Debugging

When the problem is on the node itself (disk pressure, kubelet crash, etc.):

# Check node conditions
kubectl describe node minikube

# Look for these warning conditions:
#   MemoryPressure: True    → node is OOM, pods may be evicted
#   DiskPressure: True      → node disk is full, no new pods can start
#   PIDPressure: True       → too many processes
#   Ready: False            → kubelet is unhealthy

# SSH into the node to check kubelet logs
minikube ssh
sudo journalctl -u kubelet -f --since "5 min ago"

Quick Reference: Status vs. What it Means

Key Takeaways

✅ Quick Check

Q1: A pod is Running but your curl requests to its Service get Connection refused. In the three-layer model, which layer is broken, and what’s your first debugging command?

Q2: You run kubectl logs my-pod and get “Error from server (NotFound): pods ‘my-pod’ not found.” But kubectl get pods shows the pod. What’s likely happening?

Q3: A pod based on a distroless image (FROM scratch) is misbehaving. You want to run curl from inside the pod to test connectivity. How do you get a shell?