On-Premises Kubernetes: From Rack to Cluster
Build the machine, network, storage, and control-plane chain that keeps an enterprise AI workflow platform running when your team owns the data centre.
Run it in the public monorepo
This course is built around the public HelixWorks Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.
Source: infra/stacks/prod/main.tf
vpc_cidr = "10.30.0.0/16"
kubernetes_version = var.kubernetes_version
node_instance_types = ["m7i.large"]
node_min_size = 3
node_max_size = 20
deletion_protection = true
Code to reality
- Declared intent
- Declare the production cloud substrate with explicit network, Kubernetes, capacity, and protection policy.
- Interpreter
- Terraform passes environment policy into the shared module and the AWS provider reconciles managed resources.
- Software effect
- One EKS control plane manages a bounded node group while deletion protection applies to the managed data tier.
- Hardware effect
- AWS allocates network ranges, managed control-plane capacity, at least three billable EC2 workers, and protected database storage.
- Observable evidence
- Terraform outputs, AWS inventory, Kubernetes node readiness, and an application probe connect declaration to reality.
The enterprise problem and today’s slice
Enterprise problem: Moving an enterprise AI workflow platform on premises can satisfy locality or hardware-control requirements, but the organization becomes responsible for every failed disk, switch path, certificate, control-plane member, and spare worker that a cloud provider previously hid. Whole-course context: The incoming incident evidence shows that a pod limit and node capacity are different boundaries; today follows capacity downward into racks and upward into a resilient cluster. Today’s slice: We design a kubeadm-based high-availability cluster, its supporting load balancer, network, storage, power, and replacement workflow. End-of-day evidence: A rack-to-request topology, machine inventory, declarative bootstrap inputs, failure drill, and enterprise workflow traffic replay prove the owned platform boundary. Still unsolved: Managed-cloud responsibility, provider-specific services, and automated just-in-time capacity remain later decisions.
The thesis is that on-premises Kubernetes is not merely Kubernetes installed on local servers; it is a product assembled from facilities, machines, networks, storage, control-plane quorum, and operating procedures. The team must own every interface between those three boxes.
From hardware bill of materials to schedulable bytes
Buying aggregate memory does not ensure that one pod fits or survives a machine failure. Capacity must be planned per node and per failure domain after subtracting operating-system, Kubernetes, and eviction reservations.
| Layer | Enterprise workflow platform choice | Failure if omitted | Decision rule |
|---|---|---|---|
| Facility | Independent power and cooling paths | One utility event removes every replica | Place required replicas across genuinely independent domains |
| Network | Redundant switches, routable node/pod/service plan, stable API endpoint | Nodes partition or clients lose API access | Validate loss of each path and preserve address ownership in IPAM |
| Control plane | Three members for quorum and API load balancing | One control-plane host becomes cluster-wide outage | Use odd quorum membership and tested backups/restores |
| Workers | Per-node CPU/RAM shape plus spare failure capacity | Enterprise workflow replacement cannot fit after one node fails | Plan N-1 schedulable requests, not average live usage |
| Storage | Replication, attach semantics, backup and restore | Rescheduled pod cannot reach durable content | Prove recovery point and recovery time with restore drills |
Bootstrap code versus rack and software effects
Declarative files do not rack servers or create network paths; they instruct already prepared software to establish cluster state. Every code review should name the prerequisite physical action and resulting durable state.
apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.36.0
controlPlaneEndpoint: k8s-api.example.internal:6443
networking:
podSubnet: 10.244.0.0/16
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
systemReserved:
cpu: 500m
memory: 1Gi
kubeReserved:
cpu: 500m
memory: 1Gi
sudo kubeadm init --config cluster.yaml --upload-certs
kubectl get --raw='/readyz?verbose'
kubectl get nodes -o wide
kubectl -n kube-system get pods
| Code or action | Software/control-plane effect | Physical/network/storage effect | Deployment proof |
|---|---|---|---|
| Rack and cable approved machines | No Kubernetes object exists yet | Servers receive redundant power, switch links, and management reachability | Asset, port, power-path, and burn-in records |
kubeadm init | Creates certificates, static control-plane pods, initial etcd and join material on the first host | Uses that host's CPU, RAM, disks, clock, and network; creates no redundancy alone | API readiness, etcd membership, bootstrap log, immutable config revision |
| Join two more control planes | Adds API/controller/scheduler and etcd members according to topology | Consumes three distinct machines and network paths | Quorum and API endpoint survive one member loss |
| Install CNI | Deploys the Container Network Interface implementation and node agents | Programs host networking and possibly switch/router integrations | Pods communicate across nodes under policy |
| Create StorageClass/CSI integration | Defines dynamic volume behavior and deploys storage controllers/drivers | Allocates or attaches real array/local storage through provider APIs | Enterprise workflow volume survives pod replacement and restore drill |
| Apply enterprise workflow Deployment | Controllers schedule processes onto workers | Consumes node CPU/RAM and pulls image bytes across the network | Revision, Ready replicas, endpoints, and user probe |
kubeadm is a cluster bootstrap building block and can be integrated with provisioning systems, but its single-control-plane path has a single etcd failure point unless you deliberately build high availability and backups (Creating a cluster with kubeadm).
Operate what the cloud used to hide
Initial installation is a small fraction of ownership, so an on-premises decision must budget recurring work. The operating model should assign detection, authority, action, and proof for each layer.
| Responsibility | Recurring work | Decisive evidence |
|---|---|---|
| Facilities and hardware | Power/cooling tests, firmware, failed-part replacement, spares | Environmental telemetry, asset lifecycle, failure drill |
| Network and load balancing | Switch/router maintenance, IPAM/DNS, API and ingress load balancers | Path tests, configuration revision, failover duration |
| Linux and runtime | OS/kernel/runtime patching, image distribution, time sync | Compliance scan, rollout wave, rollback proof |
| Kubernetes control plane | Certificates, etcd backup/restore, version upgrades, admission and API availability | Restore drill, skew check, API SLO |
| Worker fleet | Capacity, drain/rebuild, labels/taints, kubelet configuration | N-1 scheduling test and replacement lead time |
| Workloads | Requests/limits, replicas, disruption budgets, application profiling | Enterprise workflow replay, restart/latency/error evidence |
Use on premises when legal locality, disconnected operation, specialized hardware, predictable utilization, or existing data-centre capability outweigh the operational burden. Avoid it when the organization cannot staff 24/7 ownership across facilities, network, storage, Linux, security, and Kubernetes; local hardware does not remove those responsibilities.
Key takeaways
An on-premises cluster turns organization-owned metal into schedulable machines through a long chain of independently failing systems. Kubernetes reconciles workloads only after power, network, storage, operating system, runtime, and control-plane quorum are already working.
- Design failure domains physically before spreading replicas logically.
- Treat kubeadm as bootstrap automation, not a managed platform.
- Join asset, node, build, and incident identities so replacement is safe and auditable.
Checklist
A cluster is not production-ready merely because every node is currently Ready. Verify that the owned system can lose, replace, restore, and upgrade components while preserving customer evidence.
- [ ] Mapped racks, power, switches, storage, control-plane members, and workers.
- [ ] Proved API endpoint and etcd behavior during one-member loss.
- [ ] Planned N-1 worker capacity from requests and per-node allocatable memory.
- [ ] Versioned bootstrap inputs without committing secrets.
- [ ] Replaced or fenced a failed node and replayed enterprise workflow traffic.