51

Admission Controllers

Video: Day 51 — Admission Controllers & Webhooks • Theme: the gatekeepers that mutate and validate every request after authn/authz.

Key terms

TermMeaning
Admission controllerCode that intercepts requests after authn/authz, before storage
Mutating webhookCan change the object (add defaults, sidecars, labels)
Validating webhookCan only accept or reject the object
AdmissionReviewThe request/response object exchanged with a webhook
failurePolicyFail or Ignore if the webhook is unreachable
Built-in controllerCompiled-in admission plugin (e.g. NamespaceLifecycle)
Dynamic admissionWebhook-based controllers registered at runtime

Problem & solution

Authentication tells the api-server who you are; authorization tells it what you may do. But you still need policy on the content of objects: inject a sidecar, force resource limits, block :latest images, deny privileged pods. RBAC cannot express that.

Solution: Admission controllers run inside the api-server request path, after authn/authz and after schema validation, but before the object is persisted to etcd. They can mutate the object and/or validate it. Policy engines like Kyverno and OPA Gatekeeper plug in here as webhooks so you write policy as data, not code.

The analogy

After a trucker proves who they are and what they may bring in, a customs inspector still examines the actual paperwork before anything is filed: a first inspector may stamp-correct a form, adding a missing seal or a default value, and a second may reject it outright if it breaks the rules. Nothing reaches the port ledger until the papers pass. In Kubernetes that inspector is an admission controller, where a mutating webhook stamp-corrects the object, a validating webhook (Kyverno) accepts or rejects it, and only then is it written to etcd.

Graph legend — each Kubernetes node maps a customs concept to Kyverno admission:

Graph nodeMaps toWhat it does
the Pod being createdthe incoming objectThe request subject to policy before storage
Kyverno mutating webhookMutatingWebhookConfiguration (Kyverno)Applies mutate rules (defaults, labels, sidecars)
Kyverno validating webhookValidatingWebhookConfiguration (Kyverno)Applies validate rules to accept or reject
etcdcluster datastorePersists the object only after all checks pass

Where this fits in the cluster

The same cluster entities appear in every day's notes; the diagram below shows where this day's topic fits.

Graph legend — each node is a component in the Kyverno admission path:

Graph nodeMaps toWhat it does
api-server runs admission before the writekube-apiserverCalls webhooks before persisting objects
etcdcluster datastoreStores objects that pass admission
Kyverno admission controller pod and Servicethe Kyverno Deployment + ServiceEvaluates ClusterPolicies and answers AdmissionReviews
AdmissionReviewthe webhook payloadCarries the object to Kyverno and the decision back

The request path: where admission sits

Admission has two phases, and order matters: all mutating controllers run first (so later validators see the final object), then schema validation runs, then all validating controllers run.

Graph legend — each node is a stage of the api-server request path:

Graph nodeMaps toWhat it does
api-server authn then authzauthentication + authorizationConfirms who you are and what you may do
Mutating admissionmutating webhooksRewrites the object (defaults, labels, sidecars)
object schema validationapi-server validationChecks the object against its OpenAPI schema
Validating admissionvalidating webhooks (Kyverno)Accepts or rejects based on policy
persisted to etcdstorageWrites the final object only if all allow

If any controller rejects, the whole request fails and nothing is stored.

Built-in admission controllers

The api-server compiles in many plugins, enabled via a flag. Several are on by default in a kubeadm cluster.

# inspect what the api-server has enabled
ps -ef | grep kube-apiserver | tr ' ' '\n' | grep admission

# typical flag in /etc/kubernetes/manifests/kube-apiserver.yaml
#   --enable-admission-plugins=NodeRestriction,...

Common built-ins:

  • NamespaceLifecycle — blocks objects in terminating/non-existent namespaces.
  • LimitRanger — applies a namespace LimitRange (default/req limits).
  • ResourceQuota — enforces namespace quotas.
  • ServiceAccount — auto-mounts the default service account token.
  • NodeRestriction — limits what a kubelet can modify.
  • DefaultStorageClass — stamps the default class on PVCs without one.
  • MutatingAdmissionWebhook and ValidatingAdmissionWebhook — call your webhooks.

Dynamic admission with webhooks

Two special built-ins let you plug in your own logic at runtime. Kyverno and OPA Gatekeeper register these configurations for you; here is what one looks like under the hood — the api-server sends matching operations to the policy engine as an AdmissionReview.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: kyverno-resource-validating-webhook-cfg   # created by Kyverno
webhooks:
  - name: validate.kyverno.svc-fail
    admissionReviewVersions: ["v1"]
    sideEffects: NoneOnDryRun
    failurePolicy: Fail                 # block requests if Kyverno is down
    rules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
    clientConfig:
      service:
        name: kyverno-svc
        namespace: kyverno
        path: /validate
      caBundle: <base64-CA-cert>        # api-server verifies the webhook TLS
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations
kubectl describe validatingwebhookconfiguration kyverno-resource-validating-webhook-cfg

The webhook returns an AdmissionReview with allowed: true|false. A mutating webhook returns a base64 JSONPatch in patch to modify the object.

Native validation: ValidatingAdmissionPolicy (CEL)

Since 1.30, ValidatingAdmissionPolicy (GA) lets you write validation rules in CEL directly in the api-server — no external webhook server to run, host, or keep TLS-current.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-resource-limits
spec:
  matchConstraints:
    resourceRules:
      - apiGroups: ["apps"]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["deployments"]
  validations:
    - expression: "object.spec.template.spec.containers.all(c, has(c.resources.limits))"
      message: "every container must set resource limits"

Use cases and failure modes

  • Policy/governance: block :latest, require labels, deny privileged pods — Kyverno and OPA Gatekeeper are the two dominant webhook-based admission systems (policy as CRDs).
  • Injection: service meshes (Istio/Linkerd) use a mutating webhook to add the proxy sidecar automatically.
  • Failure policy: Fail is safe-by-default but can wedge the cluster if the policy engine is down (you may be unable to create the very pods that back it). Scope rules tightly and exempt critical namespaces with namespaceSelector.

End-to-end: a pod create through admission

The full path a CREATE pod takes through the two webhook phases.

Graph legend — each node is a real step a Pod create takes through admission:

Graph nodeMaps toWhat it does
kubectl create podclient requestSubmits the Pod
api-server authn and authzauthentication + authorizationGates who/what before admission
Mutating webhooks called in turnmutating phaseLets engines rewrite the Pod
Sidecar injected via JSONPatchmutation resultAdds defaults/sidecars to the object
Object schema validationapi-serverValidates the mutated object
Kyverno validating webhook calledKyvernoEvaluates ClusterPolicy validate rules
Request rejected, nothing storeddeny pathFails the whole request on policy violation
Object persisted to etcdstorageStores the Pod only if all validators allow

End-to-end example: a Kyverno policy that requires a label

A complete walkthrough using the real Kyverno policy engine: install Kyverno, apply a ClusterPolicy that rejects any Pod missing the label team, then show an allowed apply versus a rejected one — no custom webhook server to build.

Step 1 — install Kyverno (it registers its own webhooks).

kubectl create -f \
  https://github.com/kyverno/kyverno/releases/download/v1.12.5/install.yaml
kubectl -n kyverno rollout status deploy/kyverno-admission-controller
# deployment "kyverno-admission-controller" successfully rolled out

# Kyverno auto-creates the validating/mutating webhook configurations
kubectl get validatingwebhookconfigurations | grep kyverno
# kyverno-resource-validating-webhook-cfg   ...

Step 2 — apply a ClusterPolicy that requires the team label.

# require-team-label.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-team-label
spec:
  validationFailureAction: Enforce      # reject; use Audit to only report
  background: true
  rules:
    - name: check-team-label
      match:
        any:
          - resources:
              kinds: ["Pod"]
      validate:
        message: "every Pod must set the label 'team'"
        pattern:
          metadata:
            labels:
              team: "?*"                # any non-empty value
kubectl apply -f require-team-label.yaml
# clusterpolicy.kyverno.io/require-team-label created

kubectl get clusterpolicy require-team-label
# NAME                  ADMISSION   BACKGROUND   READY   AGE
# require-team-label    true        true         True    10s

Step 3 — allowed apply (the Pod carries the required label).

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: good-pod
  namespace: default
  labels:
    team: payments
spec:
  containers:
    - name: app
      image: nginx:1.27
EOF
# pod/good-pod created

kubectl get pod good-pod
# NAME       READY   STATUS    RESTARTS   AGE
# good-pod   1/1     Running   0          6s

Step 4 — rejected apply (label missing -> Kyverno denies, nothing stored).

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: bad-pod
  namespace: default
spec:
  containers:
    - name: app
      image: nginx:1.27
EOF
# Error from server: admission webhook "validate.kyverno.svc-fail" denied the
#   request:
#   resource Pod/default/bad-pod was blocked due to the following policies:
#   require-team-label: check-team-label: 'every Pod must set the label team'

kubectl get pod bad-pod
# Error from server (NotFound): pods "bad-pod" not found

Step 5 — confirm scope and failure behaviour.

# switch the policy to Audit: violations are reported, not blocked
kubectl patch clusterpolicy require-team-label --type merge \
  -p '{"spec":{"validationFailureAction":"Audit"}}'
kubectl get policyreport -A           # violations now show up as reports

# Kyverno's webhook uses failurePolicy Fail by default; if Kyverno is down,
# matching Pod creates are blocked until it recovers
kubectl -n kyverno get deploy kyverno-admission-controller

Graph legend — each participant is a real actor in the Kyverno admission flow:

ParticipantMaps toWhat it does
kubectlclientSends the Pod create request
api-serverkube-apiserverOrchestrates authn/authz and admission
authn / authzauthentication + authorizationConfirms identity and permissions
Mutating webhooksmutating phaseRun before validation (no change here)
Kyverno validating webhookKyverno validateEvaluates require-team-label and decides
etcdcluster datastoreStores the Pod only when Kyverno allows it

Key takeaways

  • Admission runs after authn/authz, before etcd — it governs object content.
  • Mutating webhooks run first (change objects); validating webhooks run last (accept/reject).
  • Many controllers are built in (LimitRanger, ResourceQuota, NodeRestriction, ...).
  • Kyverno / OPA Gatekeeper add dynamic policy as CRDs; mind failurePolicy and tight rules.
  • ValidatingAdmissionPolicy (CEL) does in-process validation with no webhook server.

Checklist

  • [ ] Listed --enable-admission-plugins on the api-server
  • [ ] Explained mutating-before-validating ordering
  • [ ] Installed Kyverno and inspected the webhook configurations it created
  • [ ] Wrote a ClusterPolicy and saw it reject a Pod missing a label
  • [ ] Described the risk of failurePolicy: Fail with the engine down
  • [ ] Wrote a CEL ValidatingAdmissionPolicy rule