18.2 Pod Failures — CrashLoopBackOff, ImagePullBackOff, OOMKilled

⏱️ ~10 min read

TL;DR: The three most common pod failure modes each have a distinct fingerprint. Learn to recognize them immediately and drill straight to the cause.

Failure Mode 1: CrashLoopBackOff

What it means: Your container starts, crashes immediately, restarts, crashes again. Kubernetes backs off exponentially (10s → 20s → 40s → 5min cap) to avoid hammering the system.

kubectl get pods

NAME          READY   STATUS             RESTARTS   AGE
my-app-xxx    0/1     CrashLoopBackOff   8          12m

Diagnosis Decision Tree

graph TD
    A[CrashLoopBackOff] --> B{kubectl logs --previous?}
    B -->|Has output| C{Exit message?}
    B -->|Empty / OOMKilled| D[OOMKill — see section below]
    C -->|App error / panic| E[Fix the application code or config]
    C -->|Config not found| F[Missing ConfigMap, Secret, or env var]
    C -->|Permission denied| G[Wrong securityContext or RBAC]
    C -->|Connection refused| H[Dependency not ready — use initContainers]

Step 1 — Read the Crash Logs

# Current log (may be empty if crash is instant)
kubectl logs my-app-xxx

# Log from the PREVIOUS container run (before the restart)
kubectl logs my-app-xxx --previous

Common log patterns:

# Missing environment variable
Error: Required env var DB_URL is not set

# Wrong command/entrypoint
exec: "nonexistent-binary": executable file not found in $PATH

# Database connection refused
Error: dial tcp 10.96.100.5:5432: connect: connection refused

# Permission denied
Error: open /data/config.yaml: permission denied

Step 2 — Check Exit Code

kubectl describe pod my-app-xxx | grep -A 5 "Last State"

Last State:     Terminated
  Reason:       Error
  Exit Code:    1          ← non-zero = app crashed
  Started:      ...
  Finished:     ...

Common Fixes

# Fix 1: Missing env var — add it
env:
  - name: DB_URL
    valueFrom:
      secretKeyRef:
        name: db-secret
        key: url

# Fix 2: Dependency not ready — add an init container
initContainers:
  - name: wait-for-db
    image: busybox:1.36
    command: ['sh', '-c', 'until nc -z db-service 5432; do sleep 2; done']

Failure Mode 2: ImagePullBackOff / ErrImagePull

What it means: The kubelet tried to pull your container image and failed. Like CrashLoopBackOff, it backs off exponentially.

kubectl get pods

NAME          READY   STATUS             RESTARTS   AGE
my-app-xxx    0/1     ImagePullBackOff   0          5m

Diagnosis

kubectl describe pod my-app-xxx | grep -A 10 "Events:"

Events:
  Warning  Failed  2m  kubelet  Failed to pull image "my-app:v99": rpc error:
                                 code = NotFound desc = failed to pull...
                                 404 Not Found

Common Causes and Fixes

Fixing Auth Issues with imagePullSecret

# Create a secret with registry credentials
kubectl create secret docker-registry regcred \
  --docker-server=ghcr.io \
  --docker-username=myuser \
  --docker-password=mytoken \
  --docker-email=me@example.com

# Reference it in the pod spec
spec:
  imagePullSecrets:
    - name: regcred
  containers:
    - name: app
      image: ghcr.io/myuser/my-app:v1

Quick Tag Verification

# Check if the image actually exists locally/remotely
docker pull my-app:v99  # try it locally first

# Or with crane (a registry inspection tool)
crane manifest my-app:v99

Failure Mode 3: OOMKilled

What it means: Your container tried to use more memory than its limits.memory allows. The kernel’s OOM killer sent SIGKILL (signal 9). Exit code will be 137.

kubectl describe pod my-app-xxx | grep -A 10 "Last State"

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Started:      ...
  Finished:     ...

Diagnosis

# Check current memory usage
kubectl top pod my-app-xxx

# Check the limit
kubectl get pod my-app-xxx -o yaml | grep -A 5 resources

resources:
  requests:
    memory: "128Mi"
  limits:
    memory: "256Mi"    ← if the app needs 512Mi, it'll be OOMKilled

Fix Options

Option 1 — Increase the limit (if the usage is legitimate):

resources:
  requests:
    memory: "256Mi"
  limits:
    memory: "512Mi"

Option 2 — Fix the memory leak (if usage should be lower):

• Profile the app with pprof (Go), jmap (Java), or memory_profiler (Python)
• Look for unbounded caches, missing connection pool limits, or request body not closed

Option 3 — Enable GOMEMLIMIT (for Go apps):

env:
  - name: GOMEMLIMIT
    value: "200MiB"   # tells Go's GC to be more aggressive before hitting the limit

⚠️ Warning: OOMKilled pods will restart and immediately get OOMKilled again if you don’t fix the root cause. The RESTARTS count will climb. Don’t just bump the limit without understanding why memory is growing.

Failure Mode 4: Pending (No Suitable Node)

kubectl describe pod my-app-xxx | grep -A 5 Events

Events:
  Warning  FailedScheduling  30s  default-scheduler  
    0/1 nodes are available: 1 Insufficient memory.

Common causes:

Key Takeaways

✅ Quick Check

Q1: A pod exits with code 137. Is this a bug in your application code, or something Kubernetes did?

Q2: kubectl logs my-pod --previous returns nothing. The pod is in CrashLoopBackOff. What should you check next?

Q3: Your pod is ImagePullBackOff on a private GitHub Container Registry image. The image definitely exists and you can pull it locally with docker pull. What’s missing?