61

AWS EKS: From VPC to Running Blog

Trace Ziba from AWS networking and an EKS control plane to EC2-backed pods, identity, load balancing, and OOM recovery.

The enterprise problem and today’s slice

Enterprise problem: An Amazon Elastic Kubernetes Service (EKS) cluster can be “active” while Ziba remains unreachable, unschedulable, over-privileged, or OOM-killed because the customer still owns the virtual network, worker capacity, workload identity, manifests, and service outcome. Whole-course context: The incoming shared-responsibility matrix separates provider-operated control-plane infrastructure from customer configuration and workloads; today instantiates that model in AWS. Today’s slice: We follow Terraform and Kubernetes declarations through an Amazon Virtual Private Cloud (VPC), EKS control plane, managed node group, AWS load balancer integration, and Ziba pods. End-of-day evidence: Resource IDs, Kubernetes identities, rollout revision, network path, denied AWS action, OOM recovery, and reader probe form one deployment record. Still unsolved: Karpenter's just-in-time nodes, event-driven pod scaling, and GitOps promotion are deferred.

The thesis is that EKS manages a Kubernetes control plane, not the whole application platform. Start with AWS, pods, and readers; then expand the network, compute, identity, and ownership interfaces that must all work.

Customer use cases

Creating an EKS resource is not a customer outcome, and conflating the two leaves broken network and identity paths undiscovered. These use cases prove deployment and recovery through the complete AWS-to-reader chain.

Use case IDActorCustomer jobSuccess outcomeDenial or recovery evidence
D61-UC-01AWS platform engineerProvision a private EKS foundation and expose Ziba through an approved public edgeHealthy Ziba replicas run across failure domains and readers receive the expected page through the AWS load balancerUnapproved network access and workload AWS API calls are denied while approved probes succeed
D61-UC-02Blog on-call engineerRecover an OOM-killed Ziba pod without confusing EKS control-plane health with worker or workload healthJustified revision rolls out on nodes with headroom and replay completes without another OOMUnsafe resize stays Pending or is denied; control plane and unaffected replica remain healthy

Actor-centred user stories

EKS operations cross AWS and Kubernetes identities, so acceptance must join both planes without implying that permission in one grants the other. These stories name the network, workload, denial, and recovery observations required.

Story IDUse case IDsUser storyObservable acceptance conditions
D61-US-01D61-UC-01As an AWS platform engineer, I want a reproducible VPC-to-ingress deployment, so that Ziba is reachable only through approved pathsTerraform outputs identify VPC, subnets, cluster and node group; Kubernetes shows Ready nodes/endpoints; public probe passes; private API and denied workload action tests behave as designed
D61-US-02D61-UC-02As a blog on-call engineer, I want OOM evidence connected to EC2-backed node capacity, so that I change the application boundary or compute supply for the right reasonPod UID, node UID, EC2 instance ID, limit, allocatable headroom, revision, replay, and unaffected replica are recorded

End-to-end product flows

An EKS deployment crosses two declarative control planes, so skipping the order can create pods before networking, nodes, or identities exist. The flow begins with a customer-approved release and ends with browser and denial evidence, not with an API returning success.

Flow IDUse case IDsPathTriggerNumbered stepsTerminal evidence
D61-FLOW-01D61-UC-01HappyPlatform engineer promotes Ziba's AWS environment revision1. Terraform creates VPC, subnets, routes, roles, EKS cluster, and managed node group.
2. Engineer obtains scoped cluster access.
3. Platform controllers and workload identity are installed/configured.
4. Ziba resources are applied.
5. AWS load balancer path becomes healthy.
6. Public probe and denied AWS API test run.
Terraform state/run ID, AWS resource ARNs, cluster/node UIDs, deployment digest, load-balancer DNS, positive probe, denied action, environment, actor, timestamp
D61-FLOW-02D61-UC-02RecoveryReader errors correlate with a Ziba OOMKilled termination1. Preserve request, pod, node, and AWS instance evidence.
2. Check limit breach, node pressure, requests, and managed-node-group capacity.
3. Choose code fix, justified resource revision, or node capacity action.
4. Roll out one change.
5. Replay traffic and compare unaffected replica.
6. Seal incident.
Old termination, node/instance mapping, infrastructure or manifest diff, rollout revision, replay result, and immutable incident ID

System design derived from the flows

The EKS control plane and worker nodes occupy different ownership and network boundaries, so drawing them as one box conceals routing, security-group, IAM, and capacity failures. The design expands the AWS-managed and customer-managed VPCs and keeps AWS authorization distinct from Kubernetes authorization.

Use case IDEntry pointResponsible servicesAuthoritative storeFailure evidence
D61-UC-01Terraform pipeline, AWS APIs, and Kubernetes APIVPC, subnets/routes/NAT or endpoints, IAM, EKS, managed node group, VPC CNI, load balancer controller, Kubernetes Deployment/Service/IngressTerraform backend and AWS resource APIs plus Kubernetes API desired/status stateNode join failure, exhausted subnet addresses, unauthorized API, unhealthy target, absent endpoints, or denied workload identity test mismatch
D61-UC-02Incident console, Kubernetes API, EC2/EKS telemetryKubelet/runtime, scheduler, Deployment controller, managed node group, CloudWatch/metrics pipeline, application profilerKubernetes status/events, AWS instance/node-group state, telemetry and incident storeOOMKilled, MemoryPressure, Pending Insufficient memory, failed node update/provisioning, or replay regression

AWS documents two VPCs in an EKS design: an AWS-managed VPC hosts the control plane, while a customer-managed VPC hosts nodes and customer infrastructure such as load balancers (EKS subnet architecture). The EKS architecture currently places control-plane instances across multiple Availability Zones and offers managed node groups, self-managed nodes, Karpenter, and other compute modes with different ownership (Amazon EKS architecture).

Data model and ownership

Troubleshooting stalls when an EC2 instance ID, Kubernetes node UID, pod UID, IAM role, and Terraform address cannot be joined. These records preserve references across planes while leaving authority with AWS IAM, Kubernetes RBAC, the workload controller, and the application data owner respectively.

Generated-application database: Not created in this slice — Ziba's existing content database remains application-owned; Terraform state, AWS resources, Kubernetes state, telemetry, and deployment evidence model the infrastructure slice.

Record or entityStore and ownerPrimary keyForeign key or opaque referenceTenant keyMaterial constraintLifecycle and deletionUse case IDs
EKSClusterResourceAWS EKS API, owned by AWS service for managed fields and customer cloud team for configurationCluster ARNVPC/subnet/security-group and IAM role referencesAWS account IDName/region unique in account; endpoint, version, logging, and access configuration match policyCreated, upgraded, deleted through reviewed infrastructure run; AWS retention follows service policyD61-UC-01, D61-UC-02
ManagedNodeGroupAWS EKS/EC2 APIs, owned by customer platform team with AWS lifecycle automationNode-group ARNCluster ARN, launch-template version, subnet and node-role referencesAWS account ID plus cluster nameDesired/min/max bounds and immutable-image policy; nodes span approved subnetsCreated, updated with drain/replacement, scaled, deleted; run evidence retainedD61-UC-01, D61-UC-02
KubernetesWorkloadRevisionKubernetes API store, owned by Deployment controller and blog teamDeployment UID plus revisionOpaque image digest, IAM role/service-account mapping, node and pod UIDsNamespaceImage digest immutable; resource, spread, probe, and service-account policy pass admissionRolled out, rolled back, superseded, retained by rollout/audit policyD61-UC-01, D61-UC-02
DeploymentEvidenceCustomer evidence store, owned by platform operationsEnvironment plus release/run IDTerraform state version, AWS ARNs, cluster/node/pod UIDs, probe and denial IDsAWS account plus service keyPositive and denied tests use the promoted revision and named environmentSealed after promotion/incident, retained, exported, and expired by policyD61-UC-01, D61-UC-02

Build the AWS foundation in dependency order

Creating resources out of order produces misleading secondary failures, such as a node that cannot join because its routes, IAM role, DNS, or security path is incomplete. Build and prove each dependency before relying on the layer above it.

  1. Account and region: establish quotas, IAM administration, audit logging, encryption policy, support path, and failure-domain requirements.
  2. VPC and subnets: allocate non-overlapping addresses and enough pod IP capacity; decide public/private routing, NAT cost/availability, and VPC endpoints.
  3. EKS control plane: select endpoint access, version/upgrade policy, logging, encryption, authentication, and control-plane subnets.
  4. Compute: choose managed node groups initially, with instance types, image, subnets, labels, taints, scaling bounds, and update policy.
  5. Cluster integrations: configure CNI, DNS, storage, metrics, load-balancer reconciliation, and workload identity through reviewed versions.
  6. Ziba workload: apply immutable image, requests/limits, replicas, topology spread, probes, disruption policy, Service, and ingress.
  7. Evidence: test public path, private API restrictions, denied AWS permission, node loss, OOM recovery, and rollback.

Managed node groups automate EC2 provisioning and draining around updates, but AWS states that customers remain responsible for deploying patched EKS-optimized AMI releases to their node groups; custom AMIs increase that ownership (EKS managed node groups).

Terraform and manifests versus AWS and hardware effects

Terraform code declares AWS resources while Kubernetes YAML declares desired cluster objects; neither line by itself proves a packet path or available RAM. Map each declaration to the API action, real compute, and customer observation it should cause.

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "REVIEW_AND_PIN"

  cluster_name    = "ziba-prod"
  cluster_version = var.kubernetes_version
  subnet_ids      = module.vpc.private_subnets

  eks_managed_node_groups = {
    general = {
      instance_types = ["m7i.large"]
      min_size       = 3
      max_size       = 6
      desired_size   = 3
    }
  }
}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ziba-blog
  namespace: blog
spec:
  replicas: 3
  template:
    spec:
      serviceAccountName: ziba-blog
      containers:
        - name: blog
          image: ACCOUNT.dkr.ecr.REGION.amazonaws.com/ziba@sha256:REPLACE_ME
          resources:
            requests:
              memory: 384Mi
            limits:
              memory: 768Mi
Code or actionAWS effectKubernetes/software effectHardware and deployment reality
VPC module and private subnetsCreates addresses, route tables and network boundaries in customer accountSupplies reachability and pod/node address substrateNo server exists yet; address exhaustion can still prevent pods/nodes
EKS cluster resourceRequests an AWS-operated regional control plane and interfaces into selected subnetsProduces Kubernetes API endpoint and control-plane behaviorControl-plane hosts run in AWS-managed VPC and are not customer EC2 nodes
Managed node group with three m7i.largeCreates group/launch configuration and three EC2 workers across eligible subnetsKubelets join and report three separate allocatable budgetsAWS schedules VMs backed by physical hosts; RAM remains per node, not one shared pool
Kubernetes DeploymentStores desired replica/template state in EKS APIControllers create pods; scheduler selects eligible nodesContainer processes consume EC2 node CPU and memory
Service/Ingress plus AWS controllerController calls Elastic Load Balancing APIs and configures targetsService endpoints track Ready podsNetwork flows cross a real AWS load balancer, security groups, subnets and pod/node targets
Memory limit increaseNo EC2 instance is resizedReplacement container receives higher cgroup limitUses more of the same node's finite RAM unless compute capacity is separately changed

Pin and review module/provider versions in real code; the placeholder above deliberately avoids asserting a current release. Terraform state contains sensitive infrastructure metadata and requires a protected remote backend, locking, encryption, and least-privilege access.

OOM recovery on EKS

EKS control-plane health does not imply Ziba or its EC2 node has enough memory. Diagnose from the container outward and change AWS capacity only after proving that the scheduler—not merely the process limit—is the constrained layer.

kubectl -n blog get pod -l app=ziba-blog -o wide
kubectl -n blog describe pod POD_NAME
kubectl describe node NODE_NAME
aws eks describe-nodegroup --cluster-name ziba-prod --nodegroup-name general
aws ec2 describe-instances --instance-ids INSTANCE_ID
kubectl -n blog rollout status deployment/ziba-blog
EvidenceInterpretationSmallest safe action
OOMKilled, node healthy, legitimate bounded peakContainer cgroup ceiling is too lowRaise request/limit through reviewed revision if per-node headroom is proven
OOMKilled, unbounded application growthApplication does not remain within a stable envelopeFix/profile code; temporary traffic control may restore service
Replacement Pending with Insufficient memoryNo eligible node has request capacityScale/change node group or later use dynamic provisioning; keep justified request
Node MemoryPressure and evictionsAggregate runtime pressure threatens nodeReduce/rebalance load, add compatible nodes, and preserve kube/system headroom
API unavailable but nodes/workload path observations differControl-plane/network failure, not proof of workload OOMSeparate customer VPC DNS/routes/security from AWS-managed service status and escalate with evidence

Decision rule: Kubernetes YAML changes pod placement and cgroup boundaries; Terraform changes AWS infrastructure supply. Use the former for a correctly bounded workload envelope and the latter when justified pods cannot fit or survive the required failure domain.

Key takeaways

EKS operates the Kubernetes control plane across AWS infrastructure while Ziba executes on customer-selected compute and network paths. Reliable operation joins AWS and Kubernetes identities, keeps their authorization planes distinct, and tests the public outcome plus a denied path.

  • An active EKS cluster is necessary but insufficient for a reachable application.
  • Managed node groups automate lifecycle steps but retain customer choices and patch rollout duties.
  • OOM recovery changes manifests, application code, or worker supply according to the failed boundary.

Checklist

An EKS deployment is incomplete if the team can show resources but not trace a request or denial across them. Verify each item with one immutable environment run.

  • [ ] Linked VPC, subnet, EKS, node-group, EC2 node, Kubernetes node, pod, and revision IDs.
  • [ ] Proved public Ziba access and denied unapproved cluster/AWS access.
  • [ ] Distinguished AWS IAM from Kubernetes access and RBAC.
  • [ ] Mapped Terraform and YAML to actual provider, software, VM, and memory effects.
  • [ ] Replayed OOM recovery and preserved an unaffected positive control.