Kubernetes Operators

A CRD gives you a typed object, but a stateful app like Prometheus needs domain logic: provision a StatefulSet, generate config from ServiceMonitors, roll upgrades, recover after failure. Doing that by hand does not scale and is error-prone.

Solution: The Operator pattern packages a CRD (the desired state) with a controller that runs a continuous reconcile loop — it watches your CRs and performs the same actions a skilled human operator would, automatically. The Prometheus Operator is the canonical example: you write a Prometheus CR and it builds and maintains the whole monitoring stack.

Some ships are so specialized, a deep-sea tanker say, that only a veteran dock-hand knows the full routine: how to berth it, fuel it, run safety checks, and recover it after a storm. Rather than wake that expert every time, the port hires an automated expert dock-hand that watches for that ship type and runs the whole routine itself, constantly comparing the order sheet to reality and fixing any drift. In Kubernetes that tireless dock-hand is an Operator controller (the Prometheus Operator), the order sheet it reads is your Custom Resource spec (a Prometheus), and the specialized ship it keeps shipshape is the managed workload (the Prometheus StatefulSet) it drives toward the desired state.

Graph legend — each Kubernetes node maps a port concept to the Prometheus Operator:

The same cluster entities appear in every day's notes; the diagram below shows where this day's topic fits.

Graph legend — each node is a cluster component involved in running the operator:

Every controller runs the same level-triggered loop. It does not care how it got an event; it always re-derives actions from the current desired vs observed state, which makes it self-healing.

Graph legend — each node is a stage of the operator's reconcile loop:

Key properties:

• Level-triggered, not edge-triggered: a missed event self-corrects on the next resync.
• Idempotent: running reconcile twice yields the same result.
• Owner references: child objects are garbage-collected when the CR is deleted.

An operator is two things shipped together: the CRD-defined desired state the user edits, and the controller Deployment that owns the logic. Here is a real Prometheus CR:

# the CRD-defined desired state the user edits
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: k8s
  namespace: monitoring
spec:
  version: v2.54.0
  replicas: 2
  serviceAccountName: prometheus-k8s
  serviceMonitorSelector: {}        # scrape all ServiceMonitors
  resources:
    requests:
      memory: 400Mi
status:
  availableReplicas: 0              # the operator fills this in

kubectl apply -f prometheus.yaml
kubectl get prometheuses -n monitoring
kubectl describe prometheus k8s -n monitoring   # events show reconcile actions

The controller (the prometheus-operator Deployment) owns the logic. Operators like this are built with controller-runtime (Kubebuilder / Operator SDK); the heart is a Reconcile method. You install the operator from its release bundle:

# install the Prometheus Operator (CRDs + controller Deployment)
kubectl create -f \
  https://github.com/prometheus-operator/prometheus-operator/releases/download/v0.76.0/bundle.yaml
kubectl rollout status deploy/prometheus-operator

For the Prometheus example, one pass of the loop typically:

• ensures a StatefulSet named prometheus-k8s with spec.replicas,
• generates the scrape config from selected ServiceMonitor/PodMonitor objects into a Secret the pods mount,
• ensures the governing Service for the StatefulSet,
• updates status.availableReplicas from observed pods.

kubectl get statefulset,svc,secret -n monitoring -l app.kubernetes.io/name=prometheus
kubectl get prometheus k8s -n monitoring -o jsonpath='{.status.availableReplicas}'

Because it is level-triggered, deleting the StatefulSet makes the next reconcile recreate it — the operator continuously repairs drift.

• Frameworks: Kubebuilder and Operator SDK (Go), plus Ansible- and Helm-based operators for simpler cases.
• OLM (Operator Lifecycle Manager): installs operators, manages their CRDs, RBAC, and performs version upgrades; OperatorHub.io is the catalog.
• Capability Levels describe maturity: 1) Basic install, 2) Seamless upgrades, 3) Full lifecycle, 4) Deep insights (metrics/alerts), 5) Auto-pilot (auto-scaling, auto-tuning).

Real-world examples: Prometheus Operator, cert-manager, etcd operator, Strimzi (Kafka), cloud database operators.

The full flow from a user's edit to repaired, reported state.

Graph legend — each node is a real step in reconciling a Prometheus CR:

A complete walkthrough using the real Prometheus Operator. We install it, create a Prometheus CR, watch the controller drive reality to match, delete the owned StatefulSet, and watch the loop repair the drift.

Step 1 — install the operator (CRDs + controller Deployment).

kubectl create namespace monitoring
kubectl create -f \
  https://github.com/prometheus-operator/prometheus-operator/releases/download/v0.76.0/bundle.yaml
# ... customresourcedefinition prometheuses.monitoring.coreos.com created
# ... deployment.apps/prometheus-operator created

kubectl rollout status deploy/prometheus-operator
# deployment "prometheus-operator" successfully rolled out

kubectl logs deploy/prometheus-operator | head
# level=info msg="Starting Prometheus Operator" version=0.76.0
# level=info msg="connection established" cluster-version=v1.30

Step 2 — grant the Prometheus pods scrape RBAC, then create the CR (the desired state).

# prometheus.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus-k8s
  namespace: monitoring
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: k8s
  namespace: monitoring
spec:
  version: v2.54.0
  replicas: 2
  serviceAccountName: prometheus-k8s
  serviceMonitorSelector: {}        # scrape all ServiceMonitors
  resources:
    requests:
      memory: 400Mi

kubectl apply -f prometheus.yaml
# prometheus.monitoring.coreos.com/k8s created

Step 3 — watch the controller reconcile to the desired state.

kubectl logs deploy/prometheus-operator -f
# level=info msg="sync prometheus" key=monitoring/k8s
# level=info msg="creating statefulset" statefulset=prometheus-k8s replicas=2

# the operator created an owned StatefulSet to match spec.replicas
kubectl get statefulset prometheus-k8s -n monitoring
# NAME             READY   AGE
# prometheus-k8s   2/2     30s

kubectl get prometheus k8s -n monitoring
# NAME   VERSION   DESIRED   READY   AGE
# k8s    v2.54.0   2         2       30s

# the StatefulSet carries an owner reference to the Prometheus CR (GC on delete)
kubectl get statefulset prometheus-k8s -n monitoring \
  -o jsonpath='{.metadata.ownerReferences[0].kind}'
# Prometheus

Step 4 — prove it is level-triggered: delete the StatefulSet and watch it heal.

kubectl delete statefulset prometheus-k8s -n monitoring
# statefulset.apps "prometheus-k8s" deleted

# the next reconcile recreates it from desired state
sleep 5
kubectl get statefulset prometheus-k8s -n monitoring
# NAME             READY   AGE
# prometheus-k8s   2/2     4s

Step 5 — scale via the CR; the controller converges.

kubectl patch prometheus k8s -n monitoring --type merge -p '{"spec":{"replicas":3}}'
# prometheus.monitoring.coreos.com/k8s patched

kubectl get prometheus k8s -n monitoring
# NAME   VERSION   DESIRED   READY   AGE
# k8s    v2.54.0   3         3       2m

Graph legend — each node is a real step in the install-and-reconcile walkthrough:

• An Operator = a CRD (desired state) + a controller (the logic).
• The reconcile loop is level-triggered, idempotent, and self-healing.
• Controllers use owner references so children clean up with the CR.
• Build with Kubebuilder / Operator SDK; distribute/upgrade with OLM.
• Operators encode an expert's run-book — provisioning, backups, failover, upgrades.

• [ ] Explained how an operator differs from a plain CRD
• [ ] Described the reconcile loop and why it is level-triggered
• [ ] Installed the Prometheus Operator and watched it create an owned StatefulSet
• [ ] Edited a Prometheus CR spec and observed the controller converge
• [ ] Named the role of Operator SDK and OLM