28

Design Event Scaling, Prove the Current Launch Veto

Distinguish a future scaling design from capabilities that the repository actually implements today.

System map · Day 28

Whole-system design

Five stable layers. Today's work is expanded and linked; the rest stays in context.

Product and authority

Covered — People and product entry points · Identity and policy · HelixWorks control plane · Generated application plane

Delivery and desired state

Covered — Git desired state · CI and immutable artifacts · Argo CD reconciliation

Cloud and orchestration

Covered — Terraform and AWS APIs · Accounts, VPC, DNS, and private paths

Kubernetes or EKS control plane

Design target · not proved

A future KEDA control loop would coordinate replica desired state, but no KEDA objects exist in the repository.

Compute and traffic

Covered — Platform service workloadsAhead — Ambient mesh data plane

Worker compute

Design target · not proved

Managed-node capacity is bounded but fixed; no Karpenter controller currently responds to pending workloads.

Generated app workloads

Design target · not proved

Workloads have fixed replicas and resource requests; generation remains synchronous rather than queue-driven.

Storage and evidence

Covered — Infrastructure state · Cluster desired and live state · Product data and artifacts

Evidence and observability

Design target · not proved

Queue depth, pending Pods, node readiness, and scale-down evidence are required before the scaling launch veto can clear.

The enterprise problem and today’s slice

Enterprise problem: A plausible autoscaling diagram can be mistaken for a deployed product capability. That creates both launch risk and surprise AWS spend. Whole-course context: Day 22 proved that generation is currently a synchronous control-plane-to-generator HTTP call; the outbox/broker path carries audit evidence, not generation jobs. Today’s slice: KEDA, Karpenter, a generation-job queue, a worker contract, and their AWS prerequisites are absent from the current repository. We design the future two-loop boundary, then turn that absence into a fail-closed launch veto. End-of-day evidence: Source checks show the missing controllers and async job path, while the synchronous product test remains the positive control. Still unsolved: No queue-driven pod or node scaling may be claimed until implementation, deployment, and an observed burst dossier exist.

Customer outcome and implementation focus

The customer outcome is a reviewable design event scaling, prove the current launch veto change, not a collection of requirements. This day starts with the implementation boundary, then uses the command or manifest below to produce positive, denied, and recovery evidence.

Components in focus

KEDA scaler, Karpenter provisioner, Kubernetes scheduler/HPA, and queue/metric source; pods and EC2 nodes; queue/metrics stores, Kubernetes etcd, and provider inventory; cache: metrics cache is derivative, not scaling authority.

This map names the implementation boundary for this day. The service or controller changes only the state it owns; runtime and audit evidence let the operator distinguish a declared change from an effective one.

Coordinate workload demand with node capacity

Prove absence before designing the two loops

Source absence is the decisive current evidence. The future design can use KEDA’s event metrics to feed HPA demand (KEDA scaling deployments) and Karpenter NodePools to constrain machine supply (Karpenter NodePools), but official documentation does not make those controllers exist in this repository.

: "${AWS_PROFILE:?set the approved AWS profile}" "${AWS_REGION:?set the intended AWS region}" "${EXPECTED_ACCOUNT_ID:?set the intended 12-digit AWS account}"
actual_account="$(aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"
[ "$actual_account" = "$EXPECTED_ACCOUNT_ID" ] || { echo "AWS account mismatch" >&2; exit 1; }
test ! -e argocd/addons/keda.yaml
test ! -e argocd/addons/karpenter-prod.yaml
! rg -n 'kind: (ScaledObject|NodePool|EC2NodeClass)' argocd gitops
! rg -n 'GenerationJob|enqueue_generation|generation_queue' services
rg -n 'self\.generator\.generate|artifact\.generated' services/control_plane/domain.py
echo "LAUNCH VETO: asynchronous generation and elastic node supply are not implemented"

Three authorization planes and lifecycle rule

Submitting an application job does not grant permission to change NodePools, and a Karpenter role does not read tenant data. Provider policy owns plans, runtime policy owns capacity, and the generated application owns jobs and artifacts.

Decision rule: scale pods for work and nodes for unschedulable constraints; retire jobs and workers before scaler policy, drain nodes before instance termination, and verify AWS billing state after Kubernetes cleanup.

Key takeaways

Elasticity will be two control loops joined by scheduling evidence; today it is a documented launch gap.

  • Do not claim KEDA, Karpenter, or queued generation from a design diagram.
  • Bound replicas, machine types, zones, aggregate resources, disruption, and cost.
  • Finish at job latency and customer artifact, not controller status.

Checklist

Scaling is production-ready only when it is effective, bounded, and reversible.

  • [ ] Repository absence checks are captured at an immutable revision
  • [ ] Synchronous generation and separate audit delivery are traced truthfully
  • [ ] Launch decision explicitly vetoes elastic-generation claims
  • [ ] Future acceptance requires burst, denial, scale-down, and cost evidence