On-Premises Kubernetes: From Rack to Cluster

Enterprise problem: Moving Ziba on premises can satisfy locality or hardware-control requirements, but the organization becomes responsible for every failed disk, switch path, certificate, control-plane member, and spare worker that a cloud provider previously hid. Whole-course context: The incoming incident evidence shows that a pod limit and node capacity are different boundaries; today follows capacity downward into racks and upward into a resilient cluster. Today’s slice: We design a kubeadm-based high-availability cluster, its supporting load balancer, network, storage, power, and replacement workflow. End-of-day evidence: A rack-to-request topology, machine inventory, declarative bootstrap inputs, failure drill, and Ziba traffic replay prove the owned platform boundary. Still unsolved: Managed-cloud responsibility, provider-specific services, and automated just-in-time capacity remain later decisions.

The thesis is that on-premises Kubernetes is not merely Kubernetes installed on local servers; it is a product assembled from facilities, machines, networks, storage, control-plane quorum, and operating procedures. The team must own every interface between those three boxes.

A cluster can report healthy while one shared power feed or API endpoint remains a single point of failure. These use cases prove both service availability and recoverability under a named physical failure.

Owning hardware is useful only when responsibilities can be executed by named operators under pressure. The stories bind provisioning and replacement to observable customer and infrastructure outcomes.

A server in a rack is not yet schedulable capacity, so the flow must expose every transformation from powered metal to a serving pod. The recovery path begins with a reader-visible degradation and ends only when physical inventory and Kubernetes state agree again.

Kubernetes redundancy cannot compensate for two logical nodes sharing one physical failure domain. The design therefore makes facility, network, control-plane, worker, and storage boundaries visible before mapping pods onto them.

Replacement becomes dangerous when a Kubernetes node name is mistaken for a physical asset, because disks, network addresses, credentials, and old membership can survive the machine. The ownership model links them while keeping facility, platform, and application state separate.

Generated-application database: Not created in this slice — Ziba's application data remains in its existing store; infrastructure inventory, cluster desired state, etcd backups, and incident evidence are sufficient for the platform build and replacement flows.

Buying aggregate memory does not ensure that one pod fits or survives a machine failure. Capacity must be planned per node and per failure domain after subtracting operating-system, Kubernetes, and eviction reservations.

Declarative files do not rack servers or create network paths; they instruct already prepared software to establish cluster state. Every code review should name the prerequisite physical action and resulting durable state.

apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.36.0
controlPlaneEndpoint: k8s-api.example.internal:6443
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
systemReserved:
  cpu: 500m
  memory: 1Gi
kubeReserved:
  cpu: 500m
  memory: 1Gi

sudo kubeadm init --config cluster.yaml --upload-certs
kubectl get --raw='/readyz?verbose'
kubectl get nodes -o wide
kubectl -n kube-system get pods

Initial installation is a small fraction of ownership, so an on-premises decision must budget recurring work. The operating model should assign detection, authority, action, and proof for each layer.

An on-premises cluster turns organization-owned metal into schedulable machines through a long chain of independently failing systems. Kubernetes reconciles workloads only after power, network, storage, operating system, runtime, and control-plane quorum are already working.

• Design failure domains physically before spreading replicas logically.
• Treat kubeadm as bootstrap automation, not a managed platform.
• Join asset, node, build, and incident identities so replacement is safe and auditable.

A cluster is not production-ready merely because every node is currently Ready. Verify that the owned system can lose, replace, restore, and upgrade components while preserving customer evidence.

• [ ] Mapped racks, power, switches, storage, control-plane members, and workers.
• [ ] Proved API endpoint and etcd behavior during one-member loss.
• [ ] Planned N-1 worker capacity from requests and per-node allocatable memory.
• [ ] Versioned bootstrap inputs without committing secrets.
• [ ] Replaced or fenced a failed node and replayed Ziba traffic.