Kubernetes Networking (CNI & Container Runtimes)

Kubernetes makes a hard promise: every pod gets its own IP and any pod can reach any other pod without NAT. Kubernetes itself does not implement that — it delegates to a CNI plugin. Understanding the runtime (containerd/runc) and the CNI is what lets you debug "pod stuck in ContainerCreating" or "pods can't talk across nodes".

Solution: Let a CNI plugin give every pod a routable IP (no NAT) and kube-proxy turn Service VIPs into load-balanced pod delivery, on top of containerd/runc.

A port full of berths is useless until a contractor lays the internal roads so a truck at any berth can reach any other berth, and puts up road signs that steer trucks to the right pier. Kubernetes hands that job to a plugin: the CNI builds the flat pod network that lets any pod reach any pod without NAT, while kube-proxy is the signage that routes Service traffic to the right pods.

The same cluster entities appear in every day's notes; the diagram below shows where this day's topic fits.

The kubelet does not run containers directly; it speaks the CRI to a high-level runtime, which calls a low-level OCI runtime.

Graph legend — each node is a real piece of the runtime stack:

• containerd: pulls images, manages container/snapshot lifecycle (the CRI runtime).
• runc: the OCI binary that actually clone()s the process with namespaces.
• Docker is not required on nodes; Kubernetes removed dockershim in 1.24.

Cluster networking breaks down into four distinct problems, and a different piece solves each one.

   1. container <-> container in a pod   -> same network namespace (localhost)
   2. pod <-> pod (same or other node)   -> the CNI plugin (flat, no NAT)
   3. pod <-> Service                     -> kube-proxy (ClusterIP -> pod)
   4. external -> Service                 -> NodePort / LoadBalancer / Ingress

When the kubelet creates a pod, it calls the CNI plugin to set up the pod's network namespace: allocate an IP from the node's slice of the pod CIDR, create a veth pair into the pod, and program routes so packets reach pods on other nodes.

Graph legend — Calico wiring a pod's network on CNI ADD:

Popular CNIs: Calico (BGP/eBPF, NetworkPolicy), Cilium (eBPF), Flannel (simple overlay), Weave. Pick one; it installs to /etc/cni/net.d and /opt/cni/bin.

kube-proxy runs on every node and turns a Service's stable ClusterIP into load-balanced delivery to ready pod IPs, using iptables or IPVS rules.

kubectl -n kube-system get ds kube-proxy
kubectl get endpointslices -l kubernetes.io/service-name=my-svc  # the targets

These commands show the active CNI config, the running containers, and which node each pod landed on, the first things to check when networking misbehaves.

# CNI config + plugin binaries on a node
ls /etc/cni/net.d/        # the active CNI config
ls /opt/cni/bin/          # the plugin binaries

# runtime view (no Docker)
sudo crictl ps            # running containers
sudo crictl pods          # pod sandboxes

# pod IPs and which node
kubectl get pods -o wide

# pod stuck ContainerCreating -> almost always CNI:
kubectl describe pod <pod>     # look for "failed to set up sandbox" / CNI errors

Most CNI issues trace back to a small set of recurring mistakes; these are the ones to rule out first.

   - No CNI installed         -> nodes NotReady, pods stuck ContainerCreating
   - pod CIDR mismatch        -> CNI config CIDR != kubeadm --pod-network-cidr
   - cross-node traffic fails -> firewall blocks the overlay (VXLAN 8472 / BGP 179)
   - cgroup driver mismatch   -> containerd SystemdCgroup must match the kubelet
   - two CNIs installed       -> conflicting config in /etc/cni/net.d

When a pod is created the kubelet calls the CNI to wire its network and allocate an IP.

Graph legend — pod creation through Calico, end to end:

• Kubernetes guarantees the model (pod IP, no NAT); the CNI plugin implements it.
• Runtime chain: kubelet -> containerd (CRI) -> runc (OCI); no Docker needed.
• Four problems: in-pod (localhost), pod-pod (CNI), pod-Service (kube-proxy), external (Ingress/LB).
• "Stuck ContainerCreating" is almost always a CNI problem — describe the pod.
• Match the pod CIDR and open the CNI's transport ports between nodes.

• [ ] Explained kubelet -> containerd -> runc and why Docker isn't required
• [ ] Listed the 4 networking problems and what solves each
• [ ] Found /etc/cni/net.d and /opt/cni/bin on a node
• [ ] Used crictl ps/crictl pods and kubectl get pods -o wide
• [ ] Diagnosed a ContainerCreating pod via kubectl describe