Design Event Scaling, Prove the Current Launch Veto
Distinguish a future scaling design from capabilities that the repository actually implements today.
System map · Day 28
Whole-system design
Five stable layers. Today's work is expanded and linked; the rest stays in context.
Product and authority
Covered — People and product entry points · Identity and policy · HelixWorks control plane · Generated application plane
Delivery and desired state
Covered — Git desired state · CI and immutable artifacts · Argo CD reconciliation
Cloud and orchestration
Covered — Terraform and AWS APIs · Accounts, VPC, DNS, and private paths
Kubernetes or EKS control plane
Design target · not proved
A future KEDA control loop would coordinate replica desired state, but no KEDA objects exist in the repository.
Compute and traffic
Covered — Platform service workloadsAhead — Ambient mesh data plane
Worker compute
Design target · not proved
Managed-node capacity is bounded but fixed; no Karpenter controller currently responds to pending workloads.
Generated app workloads
Design target · not proved
Workloads have fixed replicas and resource requests; generation remains synchronous rather than queue-driven.
Storage and evidence
Covered — Infrastructure state · Cluster desired and live state · Product data and artifacts
Evidence and observability
Design target · not proved
Queue depth, pending Pods, node readiness, and scale-down evidence are required before the scaling launch veto can clear.
The enterprise problem and today’s slice
Enterprise problem: A plausible autoscaling diagram can be mistaken for a deployed product capability. That creates both launch risk and surprise AWS spend. Whole-course context: Day 22 proved that generation is currently a synchronous control-plane-to-generator HTTP call; the outbox/broker path carries audit evidence, not generation jobs. Today’s slice: KEDA, Karpenter, a generation-job queue, a worker contract, and their AWS prerequisites are absent from the current repository. We design the future two-loop boundary, then turn that absence into a fail-closed launch veto. End-of-day evidence: Source checks show the missing controllers and async job path, while the synchronous product test remains the positive control. Still unsolved: No queue-driven pod or node scaling may be claimed until implementation, deployment, and an observed burst dossier exist.
Customer outcome and implementation focus
The customer outcome is a reviewable design event scaling, prove the current launch veto change, not a collection of requirements. This day starts with the implementation boundary, then uses the command or manifest below to produce positive, denied, and recovery evidence.
Components in focus
KEDA scaler, Karpenter provisioner, Kubernetes scheduler/HPA, and queue/metric source; pods and EC2 nodes; queue/metrics stores, Kubernetes etcd, and provider inventory; cache: metrics cache is derivative, not scaling authority.
This map names the implementation boundary for this day. The service or controller changes only the state it owns; runtime and audit evidence let the operator distinguish a declared change from an effective one.
Coordinate workload demand with node capacity
Prove absence before designing the two loops
Source absence is the decisive current evidence. The future design can use KEDA’s event metrics to feed HPA demand (KEDA scaling deployments) and Karpenter NodePools to constrain machine supply (Karpenter NodePools), but official documentation does not make those controllers exist in this repository.
: "${AWS_PROFILE:?set the approved AWS profile}" "${AWS_REGION:?set the intended AWS region}" "${EXPECTED_ACCOUNT_ID:?set the intended 12-digit AWS account}"
actual_account="$(aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"
[ "$actual_account" = "$EXPECTED_ACCOUNT_ID" ] || { echo "AWS account mismatch" >&2; exit 1; }
test ! -e argocd/addons/keda.yaml
test ! -e argocd/addons/karpenter-prod.yaml
! rg -n 'kind: (ScaledObject|NodePool|EC2NodeClass)' argocd gitops
! rg -n 'GenerationJob|enqueue_generation|generation_queue' services
rg -n 'self\.generator\.generate|artifact\.generated' services/control_plane/domain.py
echo "LAUNCH VETO: asynchronous generation and elastic node supply are not implemented"
Key takeaways
Elasticity will be two control loops joined by scheduling evidence; today it is a documented launch gap.
- Do not claim KEDA, Karpenter, or queued generation from a design diagram.
- Bound replicas, machine types, zones, aggregate resources, disruption, and cost.
- Finish at job latency and customer artifact, not controller status.
Checklist
Scaling is production-ready only when it is effective, bounded, and reversible.
- [ ] Repository absence checks are captured at an immutable revision
- [ ] Synchronous generation and separate audit delivery are traced truthfully
- [ ] Launch decision explicitly vetoes elastic-generation claims
- [ ] Future acceptance requires burst, denial, scale-down, and cost evidence