OOM-Killed Enterprise Workflow: Incident Anatomy

Enterprise problem: the enterprise AI workflow platform's workflow-platform disappears during a traffic spike because its container is killed for exceeding a memory boundary, and a blind restart or larger node can hide the cause while readers keep seeing errors. Whole-course context: The incoming evidence is the physical-to-pod memory model: requests reserve scheduler capacity, limits become Linux control-group ceilings, and neither creates RAM. Today’s slice: We trace one failed HTTP request through the application, container, pod, node, and Linux kernel, then separate immediate restoration from a justified resource change. End-of-day evidence: An incident record links the OOMKilled termination, limit, node condition, traffic replay, rollout revision, and unaffected control request. Still unsolved: Fleet autoscaling, cloud-provider capacity, and GitOps-controlled promotion are deferred to later days.

The thesis is simple: OOMKilled is evidence of a boundary crossing, not a diagnosis of why memory grew. The useful response follows observe → locate the enforcing boundary → change one thing → replay → compare.

An outage becomes longer when responders cannot distinguish a container limit breach from whole-node exhaustion. These use cases make customer restoration and safe diagnosis independently observable.

Recovery without falsifiable acceptance conditions can merely move the outage to another node. Each story therefore requires both positive customer evidence and a negative or unaffected control.

The customer-visible symptom begins outside Kubernetes, so starting with a manifest edit loses the causal chain. The smallest flow follows the request to its failed process and back to a verified reader response before expanding into internal evidence.

A restart reason alone cannot show whether the node was also unhealthy, so the incident system must join Kubernetes desired state with kernel and customer observations. A control group, or cgroup, is the Linux accounting and enforcement boundary used for the container's processes.

An incident cannot be audited if mutable pod names are the only join key, because replacements reuse labels but receive new identities. The records below preserve desired state, observed runtime state, and customer proof without pretending they share one owner.

Generated-application database: Not created in this slice — the platform's domain data is unchanged; Kubernetes, observability, deployment, and incident stores already own the evidence required to recover and review the outage.

The same outage can present as a 502, a restart, or a Pending replacement, so one signal cannot identify the failed boundary. Use a short evidence ladder and stop when a competing explanation remains possible.

A command is useful only when the operator can say which state it reads or changes underneath. This mapping prevents a YAML edit from being mistaken for new hardware.

kubectl get pod enterprise-workflow-api-abc -n workflow-platform -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'
kubectl describe pod enterprise-workflow-api-abc -n workflow-platform
kubectl top pod enterprise-workflow-api-abc -n workflow-platform --containers
kubectl describe node worker-3
kubectl rollout status deployment/enterprise-workflow-api -n workflow-platform

resources:
  requests:
    memory: 384Mi
  limits:
    memory: 768Mi

Pressure to restore service can erase evidence or introduce an unsafe resize, so recovery and diagnosis must interleave. Preserve the old pod status and interval first, restore through the least invasive reversible action, then test the hypothesis.

1. Capture the failed request time, pod UID, previous termination, events, resource spec, node condition, and traffic/usage interval.
2. Restore availability with a known-good rollback, traffic shedding, or additional healthy replicas when capacity permits.
3. Classify the boundary: container cgroup, node pressure, or scheduler placement.
4. Profile memory growth and decide whether the peak is legitimate, reducible, or leaking.
5. Change one resource or application variable through a new revision.
6. Replay representative traffic; compare reader errors, latency, usage, restarts, and an unaffected control.

Decision rule: raise a limit only for a measured legitimate peak with node headroom; fix the application for unbounded growth; add compatible nodes when justified requests cannot fit. A bigger machine can postpone a leak, but it cannot turn an unbounded process into a bounded one.

An OOM incident is a chain from a customer request to a Linux enforcement event, with Kubernetes recording desired and observed state around it. A reliable operator names the boundary, preserves immutable evidence, applies one reversible change, and measures the same customer path again.

• Requests affect placement; limits affect runtime enforcement; neither adds RAM.
• OOMKilled, node-pressure eviction, and Pending placement require different actions.
• Pod UID, revision, interval, and control probes make the root-cause claim falsifiable.

A recovery is incomplete if the enterprise AI workflow platform appears healthy but the evidence cannot show why or whether the failure will recur. Confirm each item against a real incident record.

• [ ] Captured failed request, pod UID, previous termination, events, and node conditions.
• [ ] Separated container OOM, node pressure, and unschedulable placement.
• [ ] Preserved original evidence before restarting or replacing the workload.
• [ ] Mapped every configuration change to its scheduler, kernel, or hardware effect.
• [ ] Replayed traffic and recorded positive, negative, and unaffected-control results.