Development EKS as the Fast Feedback Environment
Create the first managed Kubernetes cluster without confusing an Active control plane with a usable product.
System map · Day 22
Whole-system design
Five stable layers. Today's work is expanded and linked; the rest stays in context.
Product and authority
Covered — People and product entry points · Identity and policy · HelixWorks control plane · Generated application plane
Delivery and desired state
Covered — Git desired state · CI and immutable artifacts · Argo CD reconciliation
Cloud and orchestration
Covered — Accounts, VPC, DNS, and private paths
Terraform and AWS APIs
Source-backed today
Creates the development substrate with explicit identity, capacity, outputs, and a reversible deletion path.
Kubernetes or EKS control plane
Source-backed today
Provides a private fast-feedback Kubernetes API whose context is printed before every observation.
Compute and traffic
Covered — Platform service workloadsAhead — Ambient mesh data plane
Worker compute
Source-backed today
Supplies a bounded two-to-six-node development group rather than sharing production compute or quotas.
Generated app workloads
Design target · not proved
The development overlay is intentionally blocked until all placeholder images become real account-scoped immutable digests.
Storage and evidence
Covered — Infrastructure state · Cluster desired and live state · Product data and artifacts · Evidence and observability
The enterprise problem and today’s slice
Enterprise problem: Developers lose feedback or bypass controls when a shared cluster is slow, over-privileged, or reachable through undocumented identities. Whole-course context: The incoming dev VPC plan is reviewable but proves no deployed network; today defines the evidence required from the first Amazon Elastic Kubernetes Service (EKS) control plane and worker capacity. Today’s slice: The real stack names helixworks-forge-dev, uses a private-only API, a managed system node group, an EKS access entry, control-plane logs, Secrets encryption, and Pod Identity; the cloud overlay remains fail-closed at zero replicas until real ECR digests and Secrets exist. End-of-day evidence: Only an owner-approved apply from connected-network access may produce cluster ARN, nodes, digest, response, denial, account, time, run, and trace IDs. Still unsolved: No claim of deployed dev or customer response is made without that evidence.
Customer outcome and implementation focus
The customer outcome is a reviewable development eks as the fast feedback environment change, not a collection of requirements. This day starts with the implementation boundary, then uses the command or manifest below to produce positive, denied, and recovery evidence.
Components in focus
Terraform, AWS EKS API, managed node groups, CoreDNS, and Argo CD bootstrap; EC2 worker compute; isolated dev state backend, Kubernetes etcd, and logs/object evidence; cache: not involved.
This map names the implementation boundary for this day. The service or controller changes only the state it owns; runtime and audit evidence let the operator distinguish a declared change from an effective one.
Create a development EKS environment that can be destroyed safely
Verify from substrate to customer
A workload check can query the wrong cluster and look healthy, so print identity and context before observations. Run this labelled snippet from the public helixworks-kubernetes-lab monorepo using stable paths infra/stacks/dev and gitops/apps/forge/overlays/dev.
: "${AWS_PROFILE:?set the approved AWS profile}" "${AWS_REGION:?set the intended AWS region}" "${EXPECTED_ACCOUNT_ID:?set the intended 12-digit AWS account}"
actual_account="$(aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"
[ "$actual_account" = "$EXPECTED_ACCOUNT_ID" ] || { echo "AWS account mismatch" >&2; exit 1; }
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks update-kubeconfig --name helixworks-forge-dev --alias helixworks-forge-dev
aws --profile "$AWS_PROFILE" --region "$AWS_REGION" eks describe-cluster --name helixworks-forge-dev --query 'cluster.{arn:arn,status:status,public:resourcesVpcConfig.endpointPublicAccess}'
kubectl --context helixworks-forge-dev get nodes
./scripts/verify-release.sh dev
kubectl --context helixworks-forge-dev -n helixworks-forge port-forward service/control-plane 18080:8080
# In another terminal, only after the port-forward is ready:
FORGE_URL=http://127.0.0.1:18080 FORGE_TOKEN="${FORGE_TOKEN:?set the explicit bearer token}" ./scripts/smoke-product.sh
Run the AWS and Kubernetes commands from inside the VPC or a connected network because the module disables the public EKS endpoint; AWS documents that a private-only endpoint is not reachable from the public internet (EKS cluster endpoint). verify-release.sh fails closed until five real ECR image digests and four durable Secrets exist. The fifth image is the inherited broker: cloud environments use SNS/SQS and keep its Deployment at zero replicas, but still pin its tag as part of the immutable release.
Key takeaways
Development is the first AWS evidence environment, not a small production account.
- Trace EKS from VPC and nodes to a product response.
- Test IAM, Kubernetes RBAC, and app authorization separately.
- Promote immutable digests; never copy dev credentials or state.
Checklist
The dev cluster is useful only when feedback is fast and bounded.
- [ ] Caller account and kubeconfig context name dev
- [ ] Nodes, workloads, digest, and customer lifecycle are correlated
- [ ] Privileged write is denied while an allowed dev read succeeds
- [ ] Preview data has an explicit retirement path