Karpenter Node Provisioning on AWS

Enterprise problem: KEDA can request ten Ziba workers, but if their memory requests cannot fit existing nodes they remain Pending while the publishing backlog grows; a broad fixed node group may solve the incident by buying months of idle RAM. Whole-course context: The incoming evidence is a bounded KEDA/HPA replica target plus Pending reasons, pod requests, affinity, and tolerations; today maps those constraints to Amazon Elastic Compute Cloud (EC2) capacity. Today’s slice: We configure Karpenter NodePool and AWS EC2NodeClass, follow the resulting NodeClaim to a virtual machine and physical host, and bound disruption. End-of-day evidence: A run correlates Pending pod UID, NodeClaim, EC2 instance ID, registered Node, workload start, consolidation, denial path, cost tags, and timestamps. Still unsolved: A provider-neutral on-premises versus AWS versus Google Cloud platform decision remains for the capstone.

The thesis is that Karpenter does not resize a pod: it turns unschedulable pod constraints into a new node lifecycle. The smallest complete model is Pending demand, a provisioning decision, and an actual machine that registers with Kubernetes.

The customer needs publishing capacity, not merely an EC2 API success, so machine creation must end in a running workload. A second use case proves that policy rejects incompatible, over-budget, or unauthorised supply.

Dynamic provisioning is unsafe when application teams can indirectly create arbitrary machines, so acceptance includes both placement and governance. These stories make the Kubernetes request, AWS resource, and physical consequence traceable.

Provisioning crosses two control planes—Kubernetes and AWS—so an EC2 instance is only an intermediate result. The happy path ends when the pod runs; the denied path ends with an explainable constraint mismatch and no unauthorised machine.

Node provisioning fails when Kubernetes policy and AWS configuration drift apart, because neither control plane alone proves a usable Node. The design follows the request from scheduler evidence through Karpenter state to EC2, networking, storage, IAM, and kubelet registration.

Deleting a Kubernetes Node object is not equivalent to proving that its billed EC2 instance and disks disappeared, so lifecycle records must cross-reference without merging authority. Kubernetes owns scheduling intent; AWS owns the infrastructure resources and billing state.

Generated-application database: Not created in this slice — Kubernetes and AWS control planes plus Git and evidence storage already own the durable provisioning, policy, audit, and lifecycle records.

An unbounded NodePool can turn one malformed pod into expensive or privileged infrastructure, so constrain architecture, capacity type, family, generation, zones, and aggregate resources. This current v1 example uses a portable NodePool and AWS-specific EC2NodeClass; replace discovery tags and role with values created for the cluster.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: ziba-general
spec:
  template:
    metadata:
      labels:
        workload.ziba.dev/tier: general
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: ziba-general
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["eu-west-2a", "eu-west-2b", "eu-west-2c"]
      expireAfter: 720h
  limits:
    cpu: "200"
    memory: 800Gi
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m
    budgets:
      - nodes: "20%"
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: ziba-general
spec:
  role: KarpenterNodeRole-ziba-production
  amiSelectorTerms:
    - alias: al2023@latest
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: ziba-production
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: ziba-production
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 80Gi
        volumeType: gp3
        encrypted: true
        deleteOnTermination: true
  metadataOptions:
    httpEndpoint: enabled
    httpProtocolIPv6: disabled
    httpPutResponseHopLimit: 1
    httpTokens: required
  tags:
    application: ziba
    environment: production
    owner: platform

For reproducible production rollouts, resolve and pin an approved immutable AMI alias rather than leaving @latest in Git. Karpenter requires each NodePool to reference a NodeClass; the AWS EC2NodeClass controls AMI discovery, subnet and security-group selection, node role or instance profile, storage, metadata options, and tags (Karpenter NodePools, Karpenter EC2NodeClasses).

A NodeClaim marked launched is not customer capacity until kubelet registers and the pod runs, so trace every transition. The final physical host is AWS-owned, while the EC2 VM, EBS volume, elastic network interface, and workload are customer-account resources with separate lifecycle evidence.

Karpenter reasons from pod requests and scheduling constraints, not actual post-start memory usage. A worker with a 1 GiB request and 4 GiB real peak can still trigger node pressure after placement; first repair truthful requests and limits, then let node provisioning respond to them.

Cloud consoles alone cannot explain why Kubernetes wanted a machine, while Kubernetes alone cannot prove whether EC2 billed or terminated it. Correlate both control planes and retain the identifiers.

kubectl get pods -A --field-selector=status.phase=Pending
kubectl describe pod -n ziba ziba-render-worker-example
kubectl get nodepools,nodeclaims
kubectl describe nodepool ziba-general
kubectl describe ec2nodeclass ziba-general
kubectl get nodes -L karpenter.sh/nodepool,karpenter.sh/capacity-type,node.kubernetes.io/instance-type
aws ec2 describe-instances --filters Name=tag:karpenter.sh/nodepool,Values=ziba-general

Choosing by popularity obscures the real design decision: whether capacity comes from predeclared groups or is synthesized from workload constraints. Both respond to Pending pods and both require truthful requests, cloud permissions, disruption policy, and observation.

Provisioning retries can amplify cost and delay when the root cause is an impossible pod constraint or broken bootstrap. Recover from the earliest failing boundary instead of repeatedly widening NodePool policy.

Dynamic nodes are safe only when workload intent, cloud supply, and physical runtime evidence remain connected. Karpenter expands the machine layer after pod scaling—it does not replace workload autoscaling.

• NodePool declares portable constraints and lifecycle policy; EC2NodeClass declares AWS launch details.
• NodeClaim represents one requested node and links Kubernetes intent to a provider instance.
• EC2 creates a VM backed by AWS physical hardware; kubelet registration turns that VM into a Kubernetes Node.
• Karpenter uses requests and scheduling constraints, so dishonest memory requests produce dishonest machine choices.
• NodePool limits, IAM, network selectors, immutable AMIs, encrypted disks, tags, and disruption budgets are governance controls.
• Cluster Autoscaler scales preconfigured groups; Karpenter auto-provisions from allowed constraints.

A production NodePool is not ready merely because a test instance launched. Review the complete customer-to-hardware and deletion path.

• [ ] Pending reasons and resource requests are justified by observed workload evidence.
• [ ] NodePools are mutually understandable, bounded, and do not overlap accidentally.
• [ ] EC2NodeClass resolves only approved subnets, security groups, IAM role, immutable AMI, encrypted storage, and tags.
• [ ] Karpenter controller permissions are least-privilege and CloudTrail-visible.
• [ ] A successful test correlates pod, NodeClaim, instance, Node, workload, and customer outcome.
• [ ] A forbidden constraint creates no unauthorised NodeClaim or EC2 instance.
• [ ] Node bootstrap, leaked-instance detection, interruption, consolidation, and rollback are rehearsed.
• [ ] PodDisruptionBudgets and Karpenter disruption budgets preserve the required availability.
• [ ] Cost alerts and aggregate capacity limits cover reconciliation races and unexpected demand.