KEDA: Event-Driven Pod Scaling

Enterprise problem: Ziba’s blog was OOM-killed during a publish surge, yet keeping idle workers running all night wastes money; scaling on CPU alone cannot see queued posts and may add more memory-hungry pods than any node can hold. Whole-course context: The incoming evidence separates a container limit breach from a Pending pod and node pressure; today turns external work demand into a bounded replica decision. Today’s slice: We configure Kubernetes Event-driven Autoscaling (KEDA), its ScaledObject, authentication boundary, and generated HorizontalPodAutoscaler (HPA), while leaving nodes to a separate node autoscaler. End-of-day evidence: A queue-depth test records scale 0→N→0, the generated HPA, pod placements, node headroom, denied credentials, and an immutable run ID. Still unsolved: AWS-specific just-in-time node provisioning and the final platform decision are deferred.

The thesis is that KEDA does not add RAM or create virtual machines: it converts an external event signal into a desired pod count, after which Kubernetes still has to create and place those pods. Start with three boxes and expand each boundary only when its responsibility is clear.

Customers experience backlog and failed publishing, not scaler internals, so the use cases must make both latency and safety visible. These two cases distinguish serving legitimate work from refusing an unauthorised or unsafe scaling path.

Scaling can look successful while silently violating cost or credential boundaries, so acceptance must include negative evidence. Each story ties the customer result to a measurable control-plane and runtime outcome.

An event-driven scale path crosses the queue, KEDA, Kubernetes controllers, and node placement, so skipping a hop makes diagnosis guesswork. The flow starts at Ziba’s publish action and ends with queue, pod, and denial evidence.

KEDA, HPA, and the scheduler solve different layers, so treating them as rival autoscalers hides the physical bottleneck. KEDA produces demand metrics and activation, HPA changes replica intent, and the scheduler plus a node autoscaler determine whether those pods get machines.

Scaling becomes unauditable when event demand, credentials, desired replicas, and physical placement are collapsed into one record. The ownership model keeps the external queue, Kubernetes control plane, and node runtime independently inspectable and revocable.

Generated-application database: Not created in this slice — the blog’s existing queue plus Kubernetes desired state, Secrets, status, metrics, and evidence store are sufficient; KEDA does not require a new application database.

A scaler that lacks explicit bounds can turn a healthy queue spike into node exhaustion, so begin with one target, one trigger, and a hard maximum. This Redis example is runnable once Redis, KEDA, the ziba-render-worker Deployment, and the named Secret exist in namespace ziba.

apiVersion: v1
kind: Secret
metadata:
  name: redis-scaler-auth
  namespace: ziba
type: Opaque
stringData:
  password: replace-with-a-secret-manager-delivered-value
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: redis-scaler-auth
  namespace: ziba
spec:
  secretTargetRef:
    - parameter: password
      name: redis-scaler-auth
      key: password
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: ziba-render-worker
  namespace: ziba
spec:
  scaleTargetRef:
    name: ziba-render-worker
  pollingInterval: 15
  cooldownPeriod: 120
  minReplicaCount: 0
  maxReplicaCount: 12
  triggers:
    - type: redis
      metadata:
        address: redis.ziba.svc.cluster.local:6379
        listName: render-jobs
        listLength: "10"
        activationListLength: "0"
      authenticationRef:
        name: redis-scaler-auth

TriggerAuthentication separates authentication parameters from the ScaledObject and can use Secrets or cloud workload identity; a namespaced TriggerAuthentication must be in the same namespace as its ScaledObject (KEDA authentication). The Redis scaler divides queue work by the configured target to derive the external metric; use the exact scaler fields supported by the installed KEDA version (KEDA Redis Lists scaler).

A rising replica count is not proof of added service capacity because new pods can remain Pending or repeat the original OOM. Follow one queue observation through each controller and then down to a node’s CPU and DRAM.

KEDA complements rather than replaces HPA and node autoscaling. Kubernetes documentation describes node autoscaling as reacting to pod scheduling constraints and cloud/node configuration, while horizontal workload scaling changes replica count; combining them creates the chain load → pods → nodes (Kubernetes node autoscaling).

A dashboard that shows “replicas = 10” can conceal an authentication error, Pending pods, or OOM restarts, so collect evidence at every boundary. The test is complete only when queue age improves and the physical placement remains safe.

kubectl -n ziba apply -f keda-render-scaling.yaml
kubectl -n ziba get scaledobject,triggerauthentication,hpa
kubectl -n ziba describe scaledobject ziba-render-worker
kubectl -n ziba get pods -o wide --watch
kubectl -n ziba get events --sort-by=.lastTimestamp
kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.allocatable.cpu,MEMORY:.status.allocatable.memory

Choosing KEDA for every workload creates needless external dependencies, while refusing it leaves event backlog invisible to CPU-based policies. Use the demand signal closest to the customer’s waiting work, then keep replica and machine controls separate.

Event-driven scaling fails when teams mistake a desired replica count for delivered physical capacity. Keep the signal, pod, and machine loops observable as separate systems.

• KEDA turns external events into activation and external metrics; HPA changes target replicas.
• KEDA can scale an eligible workload to zero, but CPU/memory-only triggers cannot wake it from zero without a live metric source.
• TriggerAuthentication separates scaler authority from the application container and is independently revocable.
• More replicas mean more container cgroups competing for per-node CPU and RAM; they do not add hardware.
• Pending pods require compatible node capacity, while OOM-killed pods require a per-container memory diagnosis.

A KEDA rollout is incomplete until the queue, controllers, pods, nodes, and denial path agree. Use this list for Ziba’s production readiness review.

• [ ] The event signal measures customer work rather than a convenient proxy.
• [ ] minReplicaCount, maxReplicaCount, polling, cooldown, and HPA behavior are bounded.
• [ ] The scaler credential is least-privilege, not embedded in the worker image, and can be revoked independently.
• [ ] Requests and limits come from representative worker evidence, including the prior OOM.
• [ ] Nodes can fit the maximum practical burst or a node autoscaler is tested.
• [ ] A zero-to-N-to-zero test records queue age, HPA target, pod UIDs, node UIDs, timestamps, and run ID.
• [ ] Invalid authentication produces an alert and does not harm the frontend positive control.
• [ ] The rollback path preserves queued work and restores a known-good scaler revision.