Design Event Scaling, Prove the Current Launch Veto

This course is built around the public Zheta Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.

Source: gitops/apps/forge/base/workloads.yaml

serviceAccountName: control-plane
      securityContext: { runAsNonRoot: true, fsGroup: 65532, fsGroupChangePolicy: OnRootMismatch, seccompProfile: { type: RuntimeDefault } }
      containers:
        - name: control-plane
          image: zheta-forge/control-plane:v1
          envFrom: [{ configMapRef: { name: forge-environment } }, { secretRef: { name: control-plane-secrets } }]
          ports: [{ name: http, containerPort: 8080 }]
          readinessProbe: { httpGet: { path: /healthz, port: http }, periodSeconds: 5 }
          livenessProbe: { httpGet: { path: /healthz, port: http }, periodSeconds: 10 }
          resources: { requests: { cpu: 50m, memory: 64Mi }, limits: { cpu: 500m, memory: 256Mi } }

Code to reality

Declared intent

Give one microservice a distinct workload identity, hardened runtime, health contract, and capacity bounds.

Interpreter

Kubernetes admission, scheduler, kubelet, and workload-identity integration interpret different fields.

Software effect

The Pod runs as the control-plane ServiceAccount and publishes readiness only after its health endpoint responds.

Hardware effect

The scheduler reserves requested CPU and memory; the runtime enforces the limit and Linux security context.

Observable evidence

Pod identity, security context, readiness, resource metrics, and a denied excess-authority probe prove the contract.

The source tables below remain the detailed contract. Begin with these customer paths:

• D28-UC-01
  • Person: Publishing operator
  • Job: Determine whether Forge can safely accept an asynchronous generation burst
  • Observable result: The review records that current generation is synchronous and blocks the burst claim
• D28-UC-02
  • Person: Cloud governance engineer
  • Job: Define the bounded future pod-and-node scaling contract
  • Observable result: The design names metrics, limits, identities, zones, disruption, cost, and terminal product evidence

Now turn each customer job into a story with a result that an engineer can check:

• D28-US-01
  • Story: As a publishing operator, I want queue age to drive bounded worker capacity, so that jobs complete without a permanent oversized fleet
  • Observable acceptance: Evidence links queue, KEDA/HPA, pod, NodeClaim, instance, Node, job, latency, scale-down, environment, and run
• D28-US-02
  • Story: As a cloud governance engineer, I want incompatible capacity requests denied, so that workload labels cannot purchase arbitrary infrastructure
  • Observable acceptance: Denial names pod constraints, NodePool revision, zero matching claims/instances, approved control, cost window, time, and audit ID

Finally trace each story through the system that owns its state and the evidence that proves the outcome:

• D28-FLOW-01
  • Trigger: Reviewer invokes current generation
  • Responsible systems: KEDA, external metrics, HPA, scheduler, Karpenter, EC2 Fleet, kubelet, Forge worker
  • Authoritative state: SQS for jobs; Kubernetes API for scaling/scheduling; AWS API for instances; Evidence Store for outcome
  • Owned record: GenerationJob
  • Observable evidence: Request, artifact, audit event, service endpoints, immutable revision, environment, time, and trace ID
  • Failure signal: Stale metric, capped HPA, Pending pod, failed NodeClaim, EC2 capacity error, unregistered node, OOM, or missed queue age
• D28-FLOW-02
  • Trigger: Release claims elastic generation
  • Responsible systems: Argo CD, admission, KEDA, Karpenter, IAM, budgets/audit
  • Authoritative state: Git and Kubernetes policy state plus AWS infrastructure/audit state
  • Owned record: ScalingPolicyRevision
  • Observable evidence: Missing-source list, rejected decision, reviewer, revision, environment, timestamp, and successful synchronous trace
  • Failure signal: Bound exceeded, forbidden requirement accepted, disruption violation, missing cost tag, or positive control failure

Enterprise problem: A plausible autoscaling diagram can be mistaken for a deployed product capability. That creates both launch risk and surprise AWS spend. Whole-course context: Day 22 proved that generation is currently a synchronous control-plane-to-generator HTTP call; the outbox/broker path carries audit evidence, not generation jobs. Today’s slice: KEDA, Karpenter, a generation-job queue, a worker contract, and their AWS prerequisites are absent from the current repository. We design the future two-loop boundary, then turn that absence into a fail-closed launch veto. End-of-day evidence: Source checks show the missing controllers and async job path, while the synchronous product test remains the positive control. Still unsolved: No queue-driven pod or node scaling may be claimed until implementation, deployment, and an observed burst dossier exist.

Pod scaling and node provisioning solve different layers and can fail independently. Customer evidence must end with completed generated work, not a replica or instance count.

A rising replica count can hide Pending pods, and a launched instance can fail to register. These stories connect every control loop to customer work and cost.

The happy path follows demand down to physical CPU and memory and back to a completed job. The denial proves policy without broad destructive capacity changes.

KEDA changes workload demand; Karpenter changes machine supply. Kubernetes scheduling connects them by exposing unsatisfied pod constraints.

Deleting a Kubernetes Node object does not prove its EC2 instance stopped billing, so lifecycle evidence crosses systems without merging authority. Generated artifacts remain application-owned outputs.

Generated-application database: Not changed in this slice. There is no current GenerationJob aggregate. Artifacts remain in the existing provider-owned artifact store; a future job record needs an explicit owner and tenant key before async work is implemented.

Source absence is the decisive current evidence. The future design can use KEDA’s event metrics to feed HPA demand (KEDA scaling deployments) and Karpenter NodePools to constrain machine supply (Karpenter NodePools), but official documentation does not make those controllers exist in this repository.

: "${AWS_PROFILE:?set the approved AWS profile}" "${AWS_REGION:?set the intended AWS region}" "${EXPECTED_ACCOUNT_ID:?set the intended 12-digit AWS account}"
actual_account="$(aws --profile "$AWS_PROFILE" --region "$AWS_REGION" sts get-caller-identity --query Account --output text)"
[ "$actual_account" = "$EXPECTED_ACCOUNT_ID" ] || { echo "AWS account mismatch" >&2; exit 1; }
test ! -e argocd/addons/keda.yaml
test ! -e argocd/addons/karpenter-prod.yaml
! rg -n 'kind: (ScaledObject|NodePool|EC2NodeClass)' argocd gitops
! rg -n 'GenerationJob|enqueue_generation|generation_queue' services
rg -n 'self\.generator\.generate|artifact\.generated' services/control_plane/domain.py
echo "LAUNCH VETO: asynchronous generation and elastic node supply are not implemented"

Submitting an application job does not grant permission to change NodePools, and a Karpenter role does not read tenant data. Provider policy owns plans, runtime policy owns capacity, and the generated application owns jobs and artifacts.

Decision rule: scale pods for work and nodes for unschedulable constraints; retire jobs and workers before scaler policy, drain nodes before instance termination, and verify AWS billing state after Kubernetes cleanup.

Elasticity will be two control loops joined by scheduling evidence; today it is a documented launch gap.

• Do not claim KEDA, Karpenter, or queued generation from a design diagram.
• Bound replicas, machine types, zones, aggregate resources, disruption, and cost.
• Finish at job latency and customer artifact, not controller status.

Scaling is production-ready only when it is effective, bounded, and reversible.

• [ ] Repository absence checks are captured at an immutable revision
• [ ] Synchronous generation and separate audit delivery are traced truthfully
• [ ] Launch decision explicitly vetoes elastic-generation claims
• [ ] Future acceptance requires burst, denial, scale-down, and cost evidence