OOM-Killed Enterprise Workflow: Incident Anatomy

This course is built around the public HelixWorks Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.

Source: gitops/apps/forge/base/workloads.yaml

readinessProbe: { httpGet: { path: /healthz, port: http }, periodSeconds: 5 }
          livenessProbe: { httpGet: { path: /healthz, port: http }, periodSeconds: 10 }
          resources: { requests: { cpu: 50m, memory: 64Mi }, limits: { cpu: 500m, memory: 256Mi } }

Code to reality

Declared intent

Separate traffic readiness, process recovery, scheduling reservation, and runtime resource ceiling.

Interpreter

The scheduler reads requests while kubelet probes health and the container runtime enforces cgroup limits.

Software effect

Unready Pods leave Service endpoints and repeatedly unhealthy containers restart without replacing the Pod.

Hardware effect

The node reserves CPU and memory and can throttle CPU or terminate a process that exceeds its memory cgroup.

Observable evidence

Endpoint membership, restart count, resource metrics, events, and an OOMKilled status distinguish each mechanism.

Enterprise problem: the enterprise AI workflow platform's workflow-platform disappears during a traffic spike because its container is killed for exceeding a memory boundary, and a blind restart or larger node can hide the cause while readers keep seeing errors. Whole-course context: The incoming evidence is the physical-to-pod memory model: requests reserve scheduler capacity, limits become Linux control-group ceilings, and neither creates RAM. Today’s slice: We trace one failed HTTP request through the application, container, pod, node, and Linux kernel, then separate immediate restoration from a justified resource change. End-of-day evidence: An incident record links the OOMKilled termination, limit, node condition, traffic replay, rollout revision, and unaffected control request. Still unsolved: Fleet autoscaling, cloud-provider capacity, and GitOps-controlled promotion are deferred to later days.

The thesis is simple: OOMKilled is evidence of a boundary crossing, not a diagnosis of why memory grew. The useful response follows observe → locate the enforcing boundary → change one thing → replay → compare.

The same outage can present as a 502, a restart, or a Pending replacement, so one signal cannot identify the failed boundary. Use a short evidence ladder and stop when a competing explanation remains possible.

A command is useful only when the operator can say which state it reads or changes underneath. This mapping prevents a YAML edit from being mistaken for new hardware.

kubectl get pod enterprise-workflow-api-abc -n workflow-platform -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'
kubectl describe pod enterprise-workflow-api-abc -n workflow-platform
kubectl top pod enterprise-workflow-api-abc -n workflow-platform --containers
kubectl describe node worker-3
kubectl rollout status deployment/enterprise-workflow-api -n workflow-platform

resources:
  requests:
    memory: 384Mi
  limits:
    memory: 768Mi

Pressure to restore service can erase evidence or introduce an unsafe resize, so recovery and diagnosis must interleave. Preserve the old pod status and interval first, restore through the least invasive reversible action, then test the hypothesis.

1. Capture the failed request time, pod UID, previous termination, events, resource spec, node condition, and traffic/usage interval.
2. Restore availability with a known-good rollback, traffic shedding, or additional healthy replicas when capacity permits.
3. Classify the boundary: container cgroup, node pressure, or scheduler placement.
4. Profile memory growth and decide whether the peak is legitimate, reducible, or leaking.
5. Change one resource or application variable through a new revision.
6. Replay representative traffic; compare reader errors, latency, usage, restarts, and an unaffected control.

Decision rule: raise a limit only for a measured legitimate peak with node headroom; fix the application for unbounded growth; add compatible nodes when justified requests cannot fit. A bigger machine can postpone a leak, but it cannot turn an unbounded process into a bounded one.

An OOM incident is a chain from a customer request to a Linux enforcement event, with Kubernetes recording desired and observed state around it. A reliable operator names the boundary, preserves immutable evidence, applies one reversible change, and measures the same customer path again.

• Requests affect placement; limits affect runtime enforcement; neither adds RAM.
• OOMKilled, node-pressure eviction, and Pending placement require different actions.
• Pod UID, revision, interval, and control probes make the root-cause claim falsifiable.

A recovery is incomplete if the enterprise AI workflow platform appears healthy but the evidence cannot show why or whether the failure will recur. Confirm each item against a real incident record.

• [ ] Captured failed request, pod UID, previous termination, events, and node conditions.
• [ ] Separated container OOM, node pressure, and unschedulable placement.
• [ ] Preserved original evidence before restarting or replacing the workload.
• [ ] Mapped every configuration change to its scheduler, kernel, or hardware effect.
• [ ] Replayed traffic and recorded positive, negative, and unaffected-control results.