16.3 Kubernetes Manifest Validation in CI

⏱️ ~6 min read

TL;DR: Validating Kubernetes manifests in CI catches broken YAML, schema violations, security misconfigurations, and policy violations before they reach the cluster — where they’d cause failed deployments or runtime security issues. The three layers: syntax (kubectl --dry-run), schema (kubeconform), and policy (Kyverno CLI or OPA conftest).

Why Validate Manifests in CI?

Without manifest validation in CI, problems surface only after kubectl apply:

Layer 1: kubectl dry-run (Syntax + Server-Side Validation)

# Client-side dry-run (no cluster needed): checks YAML syntax
kubectl apply -f manifests/ --dry-run=client

# Server-side dry-run (requires cluster): validates against live API schema
# (catches deprecated API versions, field validation, admission webhooks)
kubectl apply -f manifests/ --dry-run=server

# Useful for Kustomize and Helm in CI:
kustomize build overlays/production | kubectl apply --dry-run=server -f -
helm template my-app ./chart --values values-prod.yaml | kubectl apply --dry-run=server -f -

Limitation: --dry-run=server requires cluster access. In CI, you’d connect to a staging or test cluster.

Layer 2: kubeconform (Schema Validation, No Cluster Needed)

kubeconform validates manifests against the Kubernetes JSON schema — no cluster required. It catches deprecated API versions and unknown fields:

# Install
curl -L https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz \
  | tar -xz && sudo mv kubeconform /usr/local/bin/

# Validate a directory of manifests
kubeconform -summary -output tap manifests/

# Validate against a specific Kubernetes version (important for upgrade planning)
kubeconform -kubernetes-version 1.29.0 manifests/

# Validate Kustomize output
kustomize build overlays/production | kubeconform -summary -

# Validate Helm output
helm template my-app ./chart --values values-prod.yaml | kubeconform -summary -

# Include CRD schemas (for ArgoCD Application, Prometheus Rules, etc.)
kubeconform \
  -schema-location default \
  -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' \
  manifests/

Example output:

Summary: 12 resources found in 4 files - Valid: 11, Invalid: 1, Errors: 0, Skipped: 0
manifests/ingress.yaml - Ingress networking.k8s.io/v1beta1 my-ingress failed validation:
  For Kubernetes 1.29.0: could not find schema for networking.k8s.io/v1beta1/Ingress
  (API removed in 1.22, use networking.k8s.io/v1)

Layer 3: Kyverno CLI (Policy Validation)

Kyverno can run in the CLI (without a cluster) to evaluate policies against manifests:

# Install Kyverno CLI
curl -LO https://github.com/kyverno/kyverno/releases/latest/download/kyverno-cli_linux_x86_64.tar.gz
tar -xvf kyverno-cli_linux_x86_64.tar.gz && sudo mv kyverno /usr/local/bin/

# Apply a policy file against manifests
kyverno apply policies/ --resource manifests/

# Apply built-in Pod Security policies
kyverno apply \
  https://github.com/kyverno/policies/raw/main/pod-security/restricted/require-run-as-non-root-user/require-run-as-non-root-user.yaml \
  --resource manifests/deployment.yaml

Example custom policies for CI:

# policies/require-resource-limits.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce
  rules:
  - name: check-container-limits
    match:
      any:
      - resources:
          kinds: [Deployment, StatefulSet, DaemonSet]
    validate:
      message: "All containers must have CPU and memory limits defined"
      pattern:
        spec:
          template:
            spec:
              containers:
              - resources:
                  limits:
                    cpu: "?*"
                    memory: "?*"
---
# policies/no-latest-tag.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: no-latest-tag
spec:
  validationFailureAction: Enforce
  rules:
  - name: check-image-tag
    match:
      any:
      - resources:
          kinds: [Deployment, StatefulSet, DaemonSet, Pod]
    validate:
      message: "Images must not use 'latest' tag"
      pattern:
        spec:
          =(initContainers):
          - image: "!*:latest"
          containers:
          - image: "!*:latest"
---
# policies/require-labels.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-labels
spec:
  validationFailureAction: Enforce
  rules:
  - name: check-labels
    match:
      any:
      - resources:
          kinds: [Deployment]
    validate:
      message: "Deployments must have 'app', 'version', and 'team' labels"
      pattern:
        metadata:
          labels:
            app: "?*"
            version: "?*"
            team: "?*"

Layer 4: OPA Conftest (General Policy Engine)

Conftest uses Open Policy Agent (OPA) Rego for more complex, custom policies:

# Install conftest
curl -L https://github.com/open-policy-agent/conftest/releases/latest/download/conftest_Linux_x86_64.tar.gz \
  | tar -xz && sudo mv conftest /usr/local/bin/

# Run policies against manifests
conftest test manifests/ --policy policies/

# Example Rego policy (policies/deny-root.rego):

# policies/deny-root.rego
package main

deny[msg] {
  input.kind == "Deployment"
  container := input.spec.template.spec.containers[_]
  not container.securityContext.runAsNonRoot
  msg := sprintf("Container '%s' must set runAsNonRoot=true", [container.name])
}

deny[msg] {
  input.kind == "Deployment"
  not input.spec.template.spec.securityContext.runAsNonRoot
  msg := sprintf("Deployment '%s' must set pod-level runAsNonRoot=true", [input.metadata.name])
}

warn[msg] {
  input.kind == "Deployment"
  not input.metadata.labels.team
  msg := sprintf("Deployment '%s' is missing 'team' label", [input.metadata.name])
}

Complete CI Validation Job (GitHub Actions)

# .github/workflows/validate-manifests.yaml
name: Validate Kubernetes Manifests

on:
  pull_request:
    paths:
    - 'k8s/**'
    - 'helm/**'

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4

    - name: Install kubeconform
      run: |
        curl -L https://github.com/yannh/kubeconform/releases/latest/download/kubeconform-linux-amd64.tar.gz \
          | tar -xz && sudo mv kubeconform /usr/local/bin/

    - name: Install Kyverno CLI
      run: |
        curl -LO https://github.com/kyverno/kyverno/releases/latest/download/kyverno-cli_linux_x86_64.tar.gz
        tar -xvf kyverno-cli_linux_x86_64.tar.gz && sudo mv kyverno /usr/local/bin/

    - name: Install Kustomize
      run: |
        curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash
        sudo mv kustomize /usr/local/bin/

    # Step 1: kubectl dry-run client (syntax only)
    - name: kubectl dry-run (client)
      run: kubectl apply -f k8s/base/ --dry-run=client

    # Step 2: Schema validation with kubeconform
    - name: kubeconform schema validation
      run: |
        kustomize build k8s/overlays/production | \
          kubeconform -summary -kubernetes-version 1.29.0 -strict -

    # Step 3: Policy validation with Kyverno
    - name: Kyverno policy check
      run: |
        kustomize build k8s/overlays/production > /tmp/manifests.yaml
        kyverno apply policies/ --resource /tmp/manifests.yaml

    # Step 4: Helm lint + schema check
    - name: Helm lint
      run: helm lint ./helm/my-app/ --values helm/my-app/values-prod.yaml

    - name: Helm kubeconform
      run: |
        helm template my-app ./helm/my-app/ --values helm/my-app/values-prod.yaml | \
          kubeconform -summary -kubernetes-version 1.29.0 -

✅ Quick Check

Q1: What’s the difference between kubectl apply --dry-run=client and --dry-run=server?

Q2: Why is kubeconform preferred over kubectl apply --dry-run=client for schema validation in CI?

Q3: You have a Kyverno policy that blocks latest tags. A developer adds image: nginx:latest to a Deployment. At which point in the GitOps pipeline is this caught?