Kubernetes Memory: From Physical RAM to Pods

This course is built around the public HelixWorks Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.

Source: gitops/apps/forge/base/workloads.yaml

readinessProbe: { httpGet: { path: /healthz, port: http }, periodSeconds: 5 }
          livenessProbe: { httpGet: { path: /healthz, port: http }, periodSeconds: 10 }
          resources: { requests: { cpu: 50m, memory: 64Mi }, limits: { cpu: 500m, memory: 256Mi } }

Code to reality

Declared intent

Separate traffic readiness, process recovery, scheduling reservation, and runtime resource ceiling.

Interpreter

The scheduler reads requests while kubelet probes health and the container runtime enforces cgroup limits.

Software effect

Unready Pods leave Service endpoints and repeatedly unhealthy containers restart without replacing the Pod.

Hardware effect

The node reserves CPU and memory and can throttle CPU or terminate a process that exceeds its memory cgroup.

Observable evidence

Endpoint membership, restart count, resource metrics, events, and an OOMKilled status distinguish each mechanism.

Enterprise problem: A customer-facing API is restarting with OOMKilled, while a batch job is Pending with Insufficient memory; treating both as “the cluster needs more RAM” risks wasting money without restoring either workload. Whole-course context: The Kubernetes course has established containers, nodes, workloads, scheduling, and cluster operations; today connects their resource settings and runtime evidence into one end-to-end capacity model. Today’s slice: We separate physical machine memory, node allocatable memory, scheduler reservations, and Linux enforcement, then choose the smallest layer that resolves each failure. End-of-day evidence: A reviewer can trace a 32 GiB node budget, explain an OOM or Pending event, and name the exact container, pod, node, or cluster change required. Still unsolved: Workload-specific heap profiling, cloud-node pricing, and production rollout policy remain environment-dependent decisions.

An operator cannot decide whether to edit YAML or buy capacity until the location of the shortage is clear. A pod ultimately uses electrical charge stored in physical dynamic random-access memory (DRAM), but several ownership and accounting layers stand between the process and those chips.

1. DRAM: Memory modules attached to a server’s CPU sockets physically hold active pages.
2. Cloud host or bare-metal server: On bare metal, Linux sees installed DRAM directly. In a cloud, a hypervisor assigns a virtual machine a memory size backed by provider hardware.
3. Node operating system: Linux uses some memory for its kernel, page cache, kubelet, container runtime, and system daemons.
4. Kubernetes node: Kubelet reports detected capacity.memory, then exposes a smaller allocatable.memory budget for pods after reservations and eviction headroom.
5. Linux cgroup: A control group accounts for a container’s processes and enforces their memory ceiling.
6. Container and pod: A container is a process boundary; a pod is the scheduling unit containing one or more containers, all on the same node.

Virtual address space does not create physical RAM. It gives a process addresses that the kernel maps to resident pages, reclaimable cache, or—only when configured—swap storage.

Confusing a request with a limit produces opposite failures: an oversized request leaves a pod Pending, while an undersized limit kills a process that might have fit on the node. Kubernetes scheduling and Linux enforcement therefore use separate values.

A machine label such as “32 GiB” overstates what workloads can safely reserve, so capacity planning must show every subtraction. This example separates physical supply, Kubernetes placement budget, and live demand.

All three symptoms mention memory, but responding to the wrong one can move rather than solve the outage. The decisive question is whether enforcement failed inside one cgroup, node safety failed across workloads, or placement failed before the pod ever ran.

A safe change request must name both the object being changed and the physical consequence; otherwise several teams may each interpret “increase memory” differently. The smallest effective layer depends on whether one process, one pod, one node, or total fleet capacity is the bottleneck.

Autoscalers operate on different axes, so choosing one by name rather than by bottleneck can amplify cost or preserve the original failure. Vertical scaling changes a pod’s resource envelope, horizontal scaling changes pod count, and cluster scaling changes machine count.

Vertical Pod Autoscaler (VPA) can recommend or apply new resource requests according to policy. A normal spec change often creates replacement pods; in-place resize can reduce disruption when the cluster, runtime, and workload support it, but it still cannot exceed real node capacity.

Horizontal Pod Autoscaler (HPA) adds or removes replicas from a scalable workload. It improves throughput or redundancy when work can be divided, but every new replica adds requests and usage somewhere in the cluster.

Cluster Autoscaler reacts to pods that cannot schedule by adding compatible nodes from a configured group. It fixes placement capacity, not a container that repeatedly crosses an unchanged 512 MiB limit.

Current-use charts alone cannot explain why Kubernetes scheduled or killed something, so diagnosis must combine declared state, node state, and events. These commands collect each view without mutating the cluster.

# Requests, limits, last termination state, placement, and pod events
kubectl describe pod api-7d9c -n payments

# Live container usage when Metrics Server is available
kubectl top pod api-7d9c -n payments --containers

# Capacity, allocatable, aggregate requests/limits, and MemoryPressure
kubectl describe node worker-a
kubectl top node worker-a

# Scheduling, eviction, and OOM-related event chronology
kubectl get events -n payments --sort-by=.lastTimestamp

Read the last container state for OOMKilled, the pod events for Insufficient memory, and the node conditions for MemoryPressure. Compare live usage with requests and limits, but size from a representative time window and a workload profile rather than a single sample.

Changing capacity during an incident is safer when each branch has a falsifiable observation. This sequence starts from the visible symptom and ends at the narrowest change that can remove it.

1. Pending with Insufficient memory: keep a justified request; add a compatible node or larger node shape, or correct an inflated request.
2. OOMKilled while the node is healthy: profile the process, distinguish leak from legitimate peak, then raise the container request and limit only with headroom.
3. Node MemoryPressure or evictions: restore node headroom, reduce unsafe overcommit, rebalance pods, or add node capacity.
4. Throughput pressure without a per-process memory ceiling: test horizontal scaling before making every replica larger.
5. Declared requests diverge from representative usage: use VPA recommendations as evidence, load-test the proposed envelope, and roll out gradually.

The most useful ticket replaces “increase Kubernetes memory” with a statement such as: “Raise the API container request from 1 GiB to 2 GiB and limit from 2 GiB to 3 GiB because load test run-184 peaked at 2.4 GiB without a leak; place the replacement only on nodes with at least 3 GiB headroom, and roll back on any OOM or node-pressure event.”

Memory incidents repeat when teams remember values but forget ownership boundaries, so retain the hierarchy rather than a single rule. Physical DRAM belongs to a machine, Kubernetes exposes a safe per-node budget, the scheduler reserves requests, and the Linux kernel enforces container cgroups.

• A cluster is a collection of separate node memory pools, not one giant shared RAM bank.
• Capacity is detected machine memory; allocatable is the smaller pod budget after reservations and eviction headroom.
• Requests govern placement; limits govern runtime enforcement; actual usage is a third value.
• A cgroup OOM kill, a kubelet eviction, and a scheduler Pending event require different fixes.
• Raising a limit permits more use of existing node RAM; resizing or adding nodes changes underlying capacity.
• VPA changes the vertical envelope, HPA changes replica count, and Cluster Autoscaler changes node count.

A memory change is not ready until another engineer can reproduce the diagnosis and see which boundary owns the fix. Use this checklist before approving the rollout.

• [ ] I can trace memory from physical DRAM through the host, node, cgroup, container, and pod.
• [ ] I can calculate node allocatable memory and remaining schedulable requests.
• [ ] I can explain why a request neither pre-allocates DRAM nor acts as a hard ceiling.
• [ ] I can distinguish OOMKilled, eviction under MemoryPressure, and Pending with Insufficient memory.
• [ ] I have included app containers, sidecars, init-container rules, and pod overhead in the capacity review.
• [ ] I have profiled a suspected leak before feeding it a larger limit.
• [ ] I can name whether this change resizes a container, pod, node, or cluster.
• [ ] I can justify VPA, HPA, or Cluster Autoscaler from the measured bottleneck.
• [ ] I have captured pre-change evidence, a positive control, rollout proof, and rollback conditions.