4.3 DaemonSets — One Per Node

⏱️ ~5 min read

TL;DR: A DaemonSet ensures exactly one pod runs on every node in the cluster. When new nodes are added, a pod is automatically scheduled on them. When nodes are removed, the pods are garbage collected.

When You Need Exactly One Pod Per Node

Some workloads are inherently node-level — they need to run on every machine:

📝 Note: Look at your kube-system namespace — kube-proxy is itself a DaemonSet. It runs on every node to maintain iptables rules.

DaemonSet YAML

# daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: log-collector
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: log-collector
  template:
    metadata:
      labels:
        app: log-collector
    spec:
      tolerations:
      # Allow scheduling on control plane nodes too
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
        effect: NoSchedule

      containers:
      - name: fluentd
        image: fluent/fluentd:v1.16
        resources:
          limits:
            memory: "200Mi"
            cpu: "100m"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true

      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:              # Mount the NODE's /var/log — not the pod's
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

The Key Difference: hostPath Volumes

DaemonSets typically mount host directories — they need to read files from the actual node filesystem. This is a hostPath volume, which is different from emptyDir or PersistentVolumes:

graph LR
    subgraph "Node A"
        HF["/var/log/ on Node"]
        subgraph "DaemonSet Pod"
            DS[log-collector\nmounts /var/log]
        end
        HF --> DS
    end
    subgraph "Node B"
        HF2["/var/log/ on Node"]
        subgraph "DaemonSet Pod"
            DS2[log-collector\nmounts /var/log]
        end
        HF2 --> DS2
    end

⚠️ Warning: hostPath volumes are powerful and dangerous. A pod with a hostPath mount to / can read the entire node filesystem. In production, restrict DaemonSet permissions with Pod Security Standards (Chapter 14).

DaemonSets vs Deployments

Targeting Specific Nodes

You can restrict a DaemonSet to a subset of nodes using nodeSelector or nodeAffinity:

spec:
  template:
    spec:
      nodeSelector:
        disk: ssd     # Only run on nodes labeled disk=ssd

# Label a node
kubectl label node minikube disk=ssd

# DaemonSet will now only schedule on labeled nodes

Try It

# On minikube (single node), a DaemonSet creates exactly 1 pod
cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ds-demo
spec:
  selector:
    matchLabels:
      app: ds-demo
  template:
    metadata:
      labels:
        app: ds-demo
    spec:
      containers:
      - name: agent
        image: busybox
        command: ["sh", "-c", "while true; do echo 'Monitoring node...'; sleep 10; done"]
        resources:
          limits:
            memory: "32Mi"
            cpu: "50m"
EOF

# Exactly 1 pod on our 1-node cluster
kubectl get pods -l app=ds-demo -o wide

# See which node it's on
kubectl get pod -l app=ds-demo -o jsonpath='{.items[0].spec.nodeName}'

# Cleanup
kubectl delete daemonset ds-demo

Key Takeaways

✅ Quick Check

Q1: Your 10-node cluster runs a log-collector DaemonSet. You add 3 more nodes. What happens?

Q2: You want a DaemonSet to run on all nodes EXCEPT the control plane. How?

Q3: Can you kubectl scale a DaemonSet to 0?