Development EKS as the Fast Feedback Environment

This course is built around the public Zheta Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.

Source: infra/modules/environment/main.tf

resource "aws_subnet" "private" {
  count             = 2
  vpc_id            = aws_vpc.this.id
  availability_zone = local.azs[count.index]
  cidr_block        = cidrsubnet(var.vpc_cidr, 4, count.index + 8)
  tags = merge(local.tags, {
    "kubernetes.io/role/internal-elb" = "1"
  })
}

Code to reality

Declared intent

Place environment workloads across two private availability-zone subnets.

Interpreter

Terraform and the AWS provider create subnets and label them for internal Kubernetes load balancers.

Software effect

EKS and its controllers discover private network placement from stable subnet IDs and tags.

Hardware effect

AWS allocates isolated address ranges and routes network interfaces across two physical zones.

Observable evidence

Terraform outputs, AWS subnet inventory, EKS node addresses, and a private connectivity probe agree.

The source tables below remain the detailed contract. Begin with these customer paths:

• D22-UC-01
  • Person: Product developer
  • Job: Deploy an immutable Forge revision to isolated dev and exercise a generated support application
  • Observable result: Dev nodes are Ready, Forge services use the requested digest, and create-to-preview returns traceable evidence
• D22-UC-02
  • Person: Platform administrator
  • Job: Grant least-privilege developer access without granting AWS account or generated-app authority
  • Observable result: Developer can inspect and reconcile the dev namespace only

Now turn each customer job into a story with a result that an engineer can check:

• D22-US-01
  • Story: As a product developer, I want a reproducible dev deployment, so that every code change produces customer-visible evidence before promotion
  • Observable acceptance: Evidence links source revision, image digest, cluster/node/pod UIDs, lifecycle response, environment, time, run, and trace
• D22-US-02
  • Story: As a platform administrator, I want dev access mapped to a narrow Kubernetes role, so that provider membership does not imply production or…
  • Observable acceptance: Accepted dev read and denied privileged writes identify actor, resource, scope, precondition, expected and observed results, and audit IDs

Finally trace each story through the system that owns its state and the evidence that proves the outcome:

• D22-FLOW-01
  • Trigger: Developer requests deployment of a reviewed digest
  • Responsible systems: Terraform, AWS EKS, managed node group, kubelet, Kubernetes controllers, Forge services
  • Authoritative state: Terraform backend and AWS APIs for infrastructure; Kubernetes API for workloads; Forge stores for product state
  • Owned record: DevClusterRevision
  • Observable evidence: Account, cluster ARN, node/pod UIDs, Git SHA, digest, response, environment, timestamp, run, and trace IDs
  • Failure signal: Active cluster with zero Ready nodes, Pending pod, digest mismatch, absent endpoint, or failed lifecycle response
• D22-FLOW-02
  • Trigger: Developer attempts a production-context or cluster-admin mutation
  • Responsible systems: AWS IAM, EKS access entry, Kubernetes RBAC and audit, Forge authorization
  • Authoritative state: IAM and EKS access configuration plus Kubernetes RBAC; generated app database for app roles
  • Owned record: DevAccessMapping
  • Observable evidence: Denied verb/resource/context, allowed dev read, IAM session, Kubernetes audit ID, environment, time, and test ID
  • Failure signal: Unexpected yes from kubectl auth can-i, wrong context, missing audit, or cross-plane role assumption

Enterprise problem: Developers lose feedback or bypass controls when a shared cluster is slow, over-privileged, or reachable through undocumented identities. Whole-course context: The incoming dev VPC plan is reviewable but proves no deployed network; today defines the evidence required from the first Amazon Elastic Kubernetes Service (EKS) control plane and worker capacity. Today’s slice: The real stack names zheta-forge-dev, uses a private-only API, a managed system node group, an EKS access entry, control-plane logs, Secrets encryption, and Pod Identity; the cloud overlay remains fail-closed at zero replicas until real ECR digests and Secrets exist. End-of-day evidence: Only an owner-approved apply from connected-network access may produce cluster ARN, nodes, digest, response, denial, account, time, run, and trace IDs. Still unsolved: No claim of deployed dev or customer response is made without that evidence.

An Active EKS API is not a customer-visible application, so acceptance must continue through nodes, pods, and a Forge lifecycle probe. The dev environment optimizes fast bounded feedback, not production availability claims.

AWS IAM and Kubernetes role-based access control (RBAC) are separate authorization systems, so both must be tested. These stories join them only through explicit, revocable mappings.

The release path crosses infrastructure, cluster, workload, and product controllers, so each owner must produce its own observation. The negative path proves the environment boundary.

EKS manages the Kubernetes control plane, while the team still owns VPC integration, nodes, access, workloads, and product evidence. That responsibility split explains why ACTIVE alone is insufficient.

Cluster access and generated-app access must not share a role record, or infrastructure permission could expose customer rows. This slice uses existing generated-application storage only for the previewed app.

Generated-application database: Required in this slice — the generated application owns its dev tenant, user, and preview domain records independently of provider and cluster access.

A workload check can query the wrong cluster and look healthy, so print identity and context before observations. Run this labelled snippet from the public zheta-kubernetes-lab monorepo using stable paths infra/stacks/dev and gitops/apps/forge/overlays/dev.

: "${AWS_PROFILE:?set the approved AWS profile}" "${AWS_REGION:?set the intended AWS region}" "${EXPECTED_ACCOUNT_ID:?set the intended 12-digit AWS account}"
actual_account="$(aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"
[ "$actual_account" = "$EXPECTED_ACCOUNT_ID" ] || { echo "AWS account mismatch" >&2; exit 1; }
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks update-kubeconfig --name zheta-forge-dev --alias zheta-forge-dev
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks describe-cluster --name zheta-forge-dev --query 'cluster.{arn:arn,status:status,public:resourcesVpcConfig.endpointPublicAccess}'
kubectl --context zheta-forge-dev get nodes
./scripts/verify-release.sh dev
kubectl --context zheta-forge-dev -n zheta-forge port-forward service/control-plane 18080:8080
# In another terminal, only after the port-forward is ready:
FORGE_URL=http://127.0.0.1:18080 FORGE_TOKEN="${FORGE_TOKEN:?set the explicit bearer token}" ./scripts/smoke-product.sh

Run the AWS and Kubernetes commands from inside the VPC or a connected network because the module disables the public EKS endpoint; AWS documents that a private-only endpoint is not reachable from the public internet (EKS cluster endpoint). verify-release.sh fails closed until four real ECR digests and four durable Secrets exist.

Dev convenience cannot erase authorization boundaries. The provider plane owns developer membership and project policy; the hosted-runtime plane owns the dev cluster, namespace, identity, secrets, and deployment; the generated application owns preview users, roles, sessions, and data predicates.

Decision rule: revoke each mapping independently, and retire preview traffic and generated-app data before deleting the dev runtime or EKS foundation.

Development is the first AWS evidence environment, not a small production account.

• Trace EKS from VPC and nodes to a product response.
• Test IAM, Kubernetes RBAC, and app authorization separately.
• Promote immutable digests; never copy dev credentials or state.

The dev cluster is useful only when feedback is fast and bounded.

• [ ] Caller account and kubeconfig context name dev
• [ ] Nodes, workloads, digest, and customer lifecycle are correlated
• [ ] Privileged write is denied while an allowed dev read succeeds
• [ ] Preview data has an explicit retirement path