Kubernetes Memory: From Physical RAM to Pods

Enterprise problem: A customer-facing API is restarting with OOMKilled, while a batch job is Pending with Insufficient memory; treating both as “the cluster needs more RAM” risks wasting money without restoring either workload. Whole-course context: The SWE course has established workloads, infrastructure, and operations; today turns their resource settings and runtime evidence into one end-to-end capacity model. Today’s slice: We separate physical machine memory, node allocatable memory, scheduler reservations, and Linux enforcement, then choose the smallest layer that resolves each failure. End-of-day evidence: A reviewer can trace a 32 GiB node budget, explain an OOM or Pending event, and name the exact container, pod, node, or cluster change required. Still unsolved: Workload-specific heap profiling, cloud-node pricing, and production rollout policy remain environment-dependent decisions.

An ambiguous memory request can send the wrong team toward the wrong change, prolonging an outage or buying idle machines. These two use cases distinguish a workload ceiling from missing placement capacity before anyone edits a resource.

Memory changes are safe only when an accountable actor can state both the intended outcome and the evidence that would disprove it. These stories make the resize and capacity-addition decisions observable instead of relying on a generic “more memory” ticket.

A resource edit crosses several owners, so losing the sequence can confuse configuration with physical capacity. The happy path changes a container ceiling only after checking node headroom; the recovery path adds a machine because no existing per-node budget can fit the pod.

The same word—memory—appears in hardware, Kubernetes, and Linux with different meanings, so a flat component diagram hides the real failure boundary. The architecture below follows physical bytes upward and shows where scheduling stops and runtime enforcement begins.

Capacity decisions fail when live usage, declared reservations, and physical supply are stored as if they were one number. Each record below has a distinct owner and lifecycle, so an operator can reconstruct why a workload was placed, killed, evicted, or left Pending.

Generated-application database: Not created in this slice — Kubernetes desired state, node status, kernel counters, infrastructure state, metrics, and events already provide the durable control-plane and evidence records required by both flows.

An operator cannot decide whether to edit YAML or buy capacity until the location of the shortage is clear. A pod ultimately uses electrical charge stored in physical dynamic random-access memory (DRAM), but several ownership and accounting layers stand between the process and those chips.

1. DRAM: Memory modules attached to a server’s CPU sockets physically hold active pages.
2. Cloud host or bare-metal server: On bare metal, Linux sees installed DRAM directly. In a cloud, a hypervisor assigns a virtual machine a memory size backed by provider hardware.
3. Node operating system: Linux uses some memory for its kernel, page cache, kubelet, container runtime, and system daemons.
4. Kubernetes node: Kubelet reports detected capacity.memory, then exposes a smaller allocatable.memory budget for pods after reservations and eviction headroom.
5. Linux cgroup: A control group accounts for a container’s processes and enforces their memory ceiling.
6. Container and pod: A container is a process boundary; a pod is the scheduling unit containing one or more containers, all on the same node.

Virtual address space does not create physical RAM. It gives a process addresses that the kernel maps to resident pages, reclaimable cache, or—only when configured—swap storage.

Confusing a request with a limit produces opposite failures: an oversized request leaves a pod Pending, while an undersized limit kills a process that might have fit on the node. Kubernetes scheduling and Linux enforcement therefore use separate values.

resources:
  requests:
    memory: "2Gi"
  limits:
    memory: "3Gi"

The request does not pre-allocate a contiguous 2 GiB block of DRAM, and the process need not consume it. The limit does not add memory to the node; it permits this cgroup to compete for up to 3 GiB of the node’s real memory. A pod’s placement cost includes the requests of its containers, relevant init-container rules, and any declared pod overhead.

A machine label such as “32 GiB” overstates what workloads can safely reserve, so capacity planning must show every subtraction. This example separates physical supply, Kubernetes placement budget, and live demand.

An 8 GiB-request pod stays Pending even if a dashboard temporarily shows 10 GiB available: the scheduler protects declared reservations and sees only 6 GiB of request budget. A 4 GiB request with a 10 GiB limit can schedule, but several such pods may collectively use more than the node can supply if they peak together. That is memory overcommit, not free capacity.

All three symptoms mention memory, but responding to the wrong one can move rather than solve the outage. The decisive question is whether enforcement failed inside one cgroup, node safety failed across workloads, or placement failed before the pod ever ran.

Swap moves selected memory pages to slower storage and is configuration-dependent in Kubernetes environments. It can change failure timing and latency, but it does not make disk equivalent to DRAM or remove the need for a truthful node budget.

A safe change request must name both the object being changed and the physical consequence; otherwise several teams may each interpret “increase memory” differently. The smallest effective layer depends on whether one process, one pod, one node, or total fleet capacity is the bottleneck.

On bare metal, more physical memory may literally mean installing compatible DIMMs and rebooting the server. In a cloud node group, it usually means replacing or adding virtual machines with a larger memory-backed instance shape. In both cases kubelet must register the resulting capacity before the scheduler can use it.

Autoscalers operate on different axes, so choosing one by name rather than by bottleneck can amplify cost or preserve the original failure. Vertical scaling changes a pod’s resource envelope, horizontal scaling changes pod count, and cluster scaling changes machine count.

Vertical Pod Autoscaler (VPA) can recommend or apply new resource requests according to policy. A normal spec change often creates replacement pods; in-place resize can reduce disruption when the cluster, runtime, and workload support it, but it still cannot exceed real node capacity.

Horizontal Pod Autoscaler (HPA) adds or removes replicas from a scalable workload. It improves throughput or redundancy when work can be divided, but every new replica adds requests and usage somewhere in the cluster.

Cluster Autoscaler reacts to pods that cannot schedule by adding compatible nodes from a configured group. It fixes placement capacity, not a container that repeatedly crosses an unchanged 512 MiB limit.

Current-use charts alone cannot explain why Kubernetes scheduled or killed something, so diagnosis must combine declared state, node state, and events. These commands collect each view without mutating the cluster.

# Requests, limits, last termination state, placement, and pod events
kubectl describe pod api-7d9c -n payments

# Live container usage when Metrics Server is available
kubectl top pod api-7d9c -n payments --containers

# Capacity, allocatable, aggregate requests/limits, and MemoryPressure
kubectl describe node worker-a
kubectl top node worker-a

# Scheduling, eviction, and OOM-related event chronology
kubectl get events -n payments --sort-by=.lastTimestamp

Read the last container state for OOMKilled, the pod events for Insufficient memory, and the node conditions for MemoryPressure. Compare live usage with requests and limits, but size from a representative time window and a workload profile rather than a single sample.

Changing capacity during an incident is safer when each branch has a falsifiable observation. This sequence starts from the visible symptom and ends at the narrowest change that can remove it.

1. Pending with Insufficient memory: keep a justified request; add a compatible node or larger node shape, or correct an inflated request.
2. OOMKilled while the node is healthy: profile the process, distinguish leak from legitimate peak, then raise the container request and limit only with headroom.
3. Node MemoryPressure or evictions: restore node headroom, reduce unsafe overcommit, rebalance pods, or add node capacity.
4. Throughput pressure without a per-process memory ceiling: test horizontal scaling before making every replica larger.
5. Declared requests diverge from representative usage: use VPA recommendations as evidence, load-test the proposed envelope, and roll out gradually.

The most useful ticket replaces “increase Kubernetes memory” with a statement such as: “Raise the API container request from 1 GiB to 2 GiB and limit from 2 GiB to 3 GiB because load test run-184 peaked at 2.4 GiB without a leak; place the replacement only on nodes with at least 3 GiB headroom, and roll back on any OOM or node-pressure event.”

Memory incidents repeat when teams remember values but forget ownership boundaries, so retain the hierarchy rather than a single rule. Physical DRAM belongs to a machine, Kubernetes exposes a safe per-node budget, the scheduler reserves requests, and the Linux kernel enforces container cgroups.

• A cluster is a collection of separate node memory pools, not one giant shared RAM bank.
• Capacity is detected machine memory; allocatable is the smaller pod budget after reservations and eviction headroom.
• Requests govern placement; limits govern runtime enforcement; actual usage is a third value.
• A cgroup OOM kill, a kubelet eviction, and a scheduler Pending event require different fixes.
• Raising a limit permits more use of existing node RAM; resizing or adding nodes changes underlying capacity.
• VPA changes the vertical envelope, HPA changes replica count, and Cluster Autoscaler changes node count.

A memory change is not ready until another engineer can reproduce the diagnosis and see which boundary owns the fix. Use this checklist before approving the rollout.

• [ ] I can trace memory from physical DRAM through the host, node, cgroup, container, and pod.
• [ ] I can calculate node allocatable memory and remaining schedulable requests.
• [ ] I can explain why a request neither pre-allocates DRAM nor acts as a hard ceiling.
• [ ] I can distinguish OOMKilled, eviction under MemoryPressure, and Pending with Insufficient memory.
• [ ] I have included app containers, sidecars, init-container rules, and pod overhead in the capacity review.
• [ ] I have profiled a suspected leak before feeding it a larger limit.
• [ ] I can name whether this change resizes a container, pod, node, or cluster.
• [ ] I can justify VPA, HPA, or Cluster Autoscaler from the measured bottleneck.
• [ ] I have captured pre-change evidence, a positive control, rollout proof, and rollback conditions.