StatefulSets

This course is built around the public HelixWorks Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.

Source: scripts/failure.sh

pod_name="$(kubectl -n helixworks get pods -l app.kubernetes.io/name=helixworks -o jsonpath='{.items[0].metadata.name}')"
    if [ -z "$pod_name" ]; then
      echo "No HelixWorks pod found. Run: make deploy" >&2
      exit 1
    fi
    echo "Deleting pod $pod_name. The Deployment should create a replacement."
    kubectl -n helixworks delete pod "$pod_name"
    kubectl -n helixworks rollout status deployment/helixworks --timeout=120s

Code to reality

Declared intent

Delete one known Pod deliberately and test controller-driven recovery.

Interpreter

The shell invokes kubectl; the API records deletion and the Deployment controller notices the deficit.

Software effect

One Pod terminates and the ReplicaSet creates a replacement with a different identity.

Hardware effect

The node releases the old container resources and starts a new container sandbox.

Observable evidence

The rollout command reaches success and kubectl get pods shows a new ready Pod after the deletion.

Deployments treat pods as interchangeable cattle — random names, shared or no persistent storage, parallel rollout. That breaks stateful systems (databases, queues, clustered stores) which need a stable name, their own durable disk, and a predictable start order (e.g. bring up the primary before replicas).

Solution: A StatefulSet gives each pod a stable ordinal identity, its own PVC via volumeClaimTemplates, stable DNS through a headless Service, and ordered, one-at-a-time deploy/scale/update.

At the port, a line of numbered ships, Ship-0, Ship-1, Ship-2, each keeps a fixed berth and a name posted on the harbor board so anyone can find it directly, and each owns its own named warehouse unit on the quay. When a ship sails and returns, it gets the same number, the same berth listing, and the same warehouse reattached, nothing is shuffled. A StatefulSet is exactly this: pods get stable ordinal names like postgres-0, a headless Service gives each a fixed DNS name, and volumeClaimTemplates give each its own PVC that survives and reattaches by name across restarts.

Graph legend — each Kubernetes node maps a port concept to the real PostgreSQL StatefulSet:

The quickest way to grasp StatefulSets is to compare them with Deployments side by side. The differences all stem from identity and storage:

Pods are named <set>-<ordinal> (postgres-0, postgres-1, ...). With a headless Service each pod gets a deterministic DNS name, so replicas can find the primary by name regardless of IP changes.

   pod DNS:   postgres-0.postgres.default.svc.cluster.local
              postgres-1.postgres.default.svc.cluster.local
              <pod>.<headless-svc>.<namespace>.svc.cluster.local

A headless Service (clusterIP: None) returns the pod A records directly instead of a single VIP — that is what powers per-pod DNS, so replicas can stream WAL from postgres-0 by name.

apiVersion: v1
kind: Service
metadata:
  name: postgres            # serviceName referenced by the StatefulSet
spec:
  clusterIP: None           # headless -> per-pod DNS
  selector: { app: postgres }
  ports:
    - port: 5432
      name: postgresql

Each replica gets its own PVC rendered from the template (pgdata-postgres-0, pgdata-postgres-1, ...). Those PVCs survive pod deletion and reattach by name, so each PostgreSQL instance keeps its data directory.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres     # MUST match the headless Service name
  replicas: 3
  selector:
    matchLabels: { app: postgres }
  template:
    metadata:
      labels: { app: postgres }
    spec:
      containers:
        - name: postgres
          image: postgres:16
          ports:
            - { containerPort: 5432, name: postgresql }
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: password
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          volumeMounts:
            - name: pgdata
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: pgdata
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi

StatefulSets change pods in a strict order rather than all at once. This is how create, scale, and update sequence the ordinals:

   create / scale up:   postgres-0 (Ready) -> postgres-1 -> postgres-2  (in order)
   scale down:          postgres-2 -> postgres-1 -> postgres-0          (reverse order)
   rolling update:      highest ordinal first, down to postgres-0, one at a time

kubectl get statefulset postgres
kubectl get pods -l app=postgres -o wide   # postgres-0, postgres-1, postgres-2 in order
kubectl scale statefulset postgres --replicas=5   # adds postgres-3 then postgres-4
kubectl get pvc                            # pgdata-postgres-0 ... persist independently

# stage an update: only ordinals >= partition are updated
kubectl patch statefulset postgres -p \
  '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":2}}}}'

A few behaviors surprise people in practice, especially around PVCs and ordering. Keep these in mind:

   - PVCs are NOT deleted when you delete the StatefulSet (delete them by hand)
   - deleting a pod reschedules it with the SAME name and reattaches its PVC
   - podManagementPolicy: Parallel skips ordering for faster, order-agnostic apps
   - updateStrategy: OnDelete = you delete pods manually to pick up a new template
   - a stuck postgres-0 (not Ready) blocks postgres-1 from ever starting (OrderedReady)

Every node below names the actual resource or field so you can trace the bring-up straight into the PostgreSQL StatefulSet manifest.

Graph legend — every node is a real step in the ordered StatefulSet rollout:

A full walkthrough using the real postgres image: deploy a headless Service plus a 3-replica StatefulSet with volumeClaimTemplates, watch ordered creation, resolve a peer by its stable DNS name, write to one pod's PVC, scale, and prove the PVC reattaches by name.

1. Create the password Secret, then apply the headless Service and the StatefulSet together:

kubectl create secret generic postgres-secret --from-literal=password='S3cretPg!'

apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  clusterIP: None
  selector: { app: postgres }
  ports:
    - { port: 5432, name: postgresql }
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 3
  selector:
    matchLabels: { app: postgres }
  template:
    metadata:
      labels: { app: postgres }
    spec:
      containers:
        - name: postgres
          image: postgres:16
          ports: [{ containerPort: 5432, name: postgresql }]
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef: { name: postgres-secret, key: password }
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          volumeMounts:
            - { name: pgdata, mountPath: /var/lib/postgresql/data }
  volumeClaimTemplates:
    - metadata:
        name: pgdata
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi

2. Watch the pods come up one at a time, in order:

kubectl get pods -l app=postgres -w
# expected order: postgres-0 Running -> postgres-1 Running -> postgres-2 Running

3. Confirm each replica got its own PVC, rendered from the template:

kubectl get pvc
# expected: pgdata-postgres-0, pgdata-postgres-1, pgdata-postgres-2  (each Bound, 10Gi)

4. Create a table in postgres-0, then resolve postgres-0 by its stable DNS name from a client pod:

kubectl exec postgres-0 -- psql -U postgres -c \
  "CREATE TABLE ships(id int); INSERT INTO ships VALUES (0);"

kubectl run pgclient --rm -it --restart=Never --image=postgres:16 --env PGPASSWORD='S3cretPg!' -- \
  psql -h postgres-0.postgres.default.svc.cluster.local -U postgres -c "SELECT * FROM ships;"
# expected: one row, id = 0  (served by postgres-0 via its stable DNS name)

5. Scale out and confirm the new ordinals append in order:

kubectl scale statefulset postgres --replicas=5
kubectl get pods -l app=postgres
# expected: postgres-3 then postgres-4 created after postgres-0..postgres-2 stay Ready
kubectl get pvc
# expected: pgdata-postgres-3, pgdata-postgres-4 now also Bound

6. Prove sticky identity: delete postgres-0, watch it return with the SAME name and its data intact:

kubectl delete pod postgres-0
kubectl wait --for=condition=Ready pod/postgres-0 --timeout=60s
kubectl exec postgres-0 -- psql -U postgres -c "SELECT * FROM ships;"
# expected: id = 0 still present   (PVC pgdata-postgres-0 reattached by name)

7. Scale down and observe reverse-order termination (PVCs remain):

kubectl scale statefulset postgres --replicas=3
kubectl get pods -l app=postgres
# expected: postgres-4 then postgres-3 terminate first (reverse order)
kubectl get pvc
# expected: pgdata-postgres-3, pgdata-postgres-4 still present (StatefulSet keeps PVCs)

Graph legend — each participant in the sequence is a real actor in the StatefulSet flow:

These are the failures you are most likely to meet with StatefulSets:

   - pods stuck Pending          -> no headless Service, or serviceName mismatch
   - postgres-1 never starts     -> postgres-0 not Ready blocks ordered rollout
   - storage shared by mistake   -> use volumeClaimTemplates, not one shared PVC
   - PVCs linger after delete    -> StatefulSet leaves PVCs; clean up manually
   - no per-pod DNS              -> Service must be clusterIP: None (headless)
   - rollout too slow            -> set podManagementPolicy: Parallel if order-agnostic

• StatefulSets give pods stable ordinal names and sticky identity.
• A headless Service (clusterIP: None) provides per-pod DNS.
• volumeClaimTemplates create one durable PVC per pod that reattaches by name.
• Deploy/scale is ordered (up in order, down in reverse); a blocked pod stalls the rest.
• PVCs persist after pod or StatefulSet deletion — clean them up deliberately.
• Use partition to stage updates and Parallel policy when ordering is unneeded.

• [ ] Explained StatefulSet vs Deployment (identity, storage, ordering)
• [ ] Created a headless Service and matched serviceName
• [ ] Deployed a PostgreSQL StatefulSet with volumeClaimTemplates and saw per-pod PVCs
• [ ] Scaled up/down and observed ordered create/reverse delete
• [ ] Resolved a pod by its postgres-0.postgres...svc DNS name
• [ ] Staged a rolling update with a partition