Package Kubernetes Workloads with Kustomize
Reuse one workload design while keeping dev, staging, and production differences small, explicit, and reviewable.
Run it in the public monorepo
This course is built around the public Zheta Kubernetes Lab monorepo. The excerpt below is runnable source, not pseudocode.
Source: gitops/apps/forge/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: [../../base]
commonAnnotations: { zheta.io/release-state: blocked-unpinned }
configMapGenerator:
- { name: forge-environment, namespace: zheta-forge, behavior: merge, literals: [ENVIRONMENT=prod] }Code to reality
- Declared intent
- Reuse the Forge base while keeping production deliberately blocked until immutable image evidence exists.
- Interpreter
- Kustomize merges the base and environment data; the release verifier rejects the blocking annotation.
- Software effect
- Production renders shared manifests but remains an explicitly unauthorized release candidate.
- Hardware effect
- No production workload capacity should be allocated from this blocked source state.
- Observable evidence
- The rendered annotation and a failing release verifier prove that absence of digests vetoes deployment.
Start with the people and the result they need
The source tables below remain the detailed contract. Begin with these customer paths:
- D17-UC-01
- Person: Platform engineer
- Job: Render one Zheta Forge service for local, dev, staging, and production
- Observable result: Each environment shares container, probes, security context, and labels while replicas, domains, and capacity vary explicitly
- D17-UC-02
- Person: Release reviewer
- Job: Reject an overlay that removes a required guardrail
- Observable result: Unsafe change never reaches GitOps reconciliation and prior rendered digest remains promotable
Turn each customer job into a testable story
Now turn each customer job into a story with a result that an engineer can check:
- D17-US-01
- Story: As a platform engineer, I want one base with small overlays, so that an important workload fix applies consistently without erasing environment policy
- Observable acceptance: Rendered object set, tool versions, source revision, image digest, environment diff, run, and output digest are recorded
- D17-US-02
- Story: As a release reviewer, I want rendered policy validation, so that a template cannot hide a privileged workload or missing production limit
- Observable acceptance: Invalid overlay is denied, exact rendered field is identified, and the last approved render still passes
Add real state and observable proof
Finally trace each story through the system that owns its state and the evidence that proves the outcome:
- D17-FLOW-01
- Trigger: Engineer proposes a service packaging change
- Responsible systems: Kustomize renderer, scripts/verify-platform.sh, release verifier, diff reviewer
- Authoritative state: Git for source; render evidence for output digest
- Owned record: PackageRevision
- Observable evidence: Actor, service, image digest, environment, expected and observed objects, source revision, tool version, timestamp, run, render digest, and diff
- Failure signal: Blocked cloud overlay, mutable image, nondeterministic render, hidden secret, unexpected object, or broad environment diff
- D17-FLOW-02
- Trigger: Reviewer evaluates overlay that removes a memory limit
- Responsible systems: Policy engine, diff reviewer, promotion controller
- Authoritative state: Policy repository for guardrails; promotion store for approved render
- Owned record: EnvironmentOverlay
- Observable evidence: Denial rule, offending object and field, prior digest, unaffected render digest, environment, time, source revision, and run ID
- Failure signal: Missing resource limit, mutable tag, privileged pod, absent workload identity, or prior digest changed
The enterprise problem and today’s slice
Enterprise problem: Copying complete Kubernetes manifests for every Zheta Forge service and environment causes security fixes, resource limits, and labels to drift, while an overly clever template can hide what production will run. Whole-course context: The incoming evidence is a service map with explicit contracts and owners; today packages those services for Kubernetes without duplicating their whole desired state. Today’s slice: We use the repository’s actual Kustomize base-and-overlay design under gitops/apps/forge; there is no Forge Helm chart, so rendered Kubernetes objects are the honest review boundary. End-of-day evidence: A render run proves all three application archetypes use the same base, each environment changes only declared policy, blocked cloud overlays fail closed until digest promotion, no secret values appear, and output records source revision, tool version, environment, time, run, and digest. Still unsolved: Infrastructure versus workload state, Terraform, and AWS accounts remain deferred.
Customer use cases
Packaging is valuable only when it prevents environment drift without obscuring customer-impacting differences. These use cases cover reusable deployment and rejected unsafe variance.
| Use case ID | Actor | Customer job | Success outcome | Denial or recovery evidence |
|---|---|---|---|---|
| D17-UC-01 | Platform engineer | Render one Zheta Forge service for local, dev, staging, and production | Each environment shares container, probes, security context, and labels while replicas, domains, and capacity vary explicitly | Rendered diff reveals every material variance and contains no plaintext secret |
| D17-UC-02 | Release reviewer | Reject an overlay that removes a required guardrail | Unsafe change never reaches GitOps reconciliation and prior rendered digest remains promotable | Schema or policy denial names missing limit, probe, identity, or forbidden image tag |
Actor-centred user stories
A folder hierarchy is not DRY if reviewers cannot predict the final Kubernetes objects. These stories treat deterministic rendering and policy checks as the product of packaging.
| Story ID | Use case IDs | User story | Observable acceptance conditions |
|---|---|---|---|
| D17-US-01 | D17-UC-01 | As a platform engineer, I want one base with small overlays, so that an important workload fix applies consistently without erasing environment policy | Rendered object set, tool versions, source revision, image digest, environment diff, run, and output digest are recorded |
| D17-US-02 | D17-UC-02 | As a release reviewer, I want rendered policy validation, so that a template cannot hide a privileged workload or missing production limit | Invalid overlay is denied, exact rendered field is identified, and the last approved render still passes |
End-to-end product flows
GitOps cannot safely reconcile source fragments it has not rendered exactly as the cluster will see them. The flows produce and validate complete Kubernetes objects before promotion.
| Flow ID | Use case IDs | Path | Trigger | Numbered steps | Terminal evidence |
|---|---|---|---|---|---|
| D17-FLOW-01 | D17-UC-01 | Happy | Engineer proposes a service packaging change | 1. Resolve gitops/apps/forge/base. 2. Apply named environment overlay. 3. Render complete objects. 4. Validate schemas and blocked/digest-pinned policy. 5. Diff against approved render. 6. Record output digest. | Actor, service, image digest, environment, expected and observed objects, source revision, tool version, timestamp, run, render digest, and diff |
| D17-FLOW-02 | D17-UC-02 | Denied | Reviewer evaluates overlay that removes a memory limit | 1. Render candidate. 2. Policy detects missing limit. 3. Block promotion. 4. Preserve prior approved digest. 5. Render unrelated archetype as positive control. | Denial rule, offending object and field, prior digest, unaffected render digest, environment, time, source revision, and run ID |
The engineer’s source becomes the full object set that a Kubernetes API would receive. Review happens on that result, not only on template fragments.
System design derived from the flows
Packaging claims must match the repository or learners will run nonexistent commands. Zheta Forge currently uses Kustomize only: one stable base plus explicit local, dev, staging, and production overlays.
| Use case ID | Entry point | Responsible services | Authoritative store | Failure evidence |
|---|---|---|---|---|
| D17-UC-01 | Packaging pull request | Kustomize renderer, scripts/verify-platform.sh, release verifier, diff reviewer | Git for source; render evidence for output digest | Blocked cloud overlay, mutable image, nondeterministic render, hidden secret, unexpected object, or broad environment diff |
| D17-UC-02 | Render policy gate | Policy engine, diff reviewer, promotion controller | Policy repository for guardrails; promotion store for approved render | Missing resource limit, mutable tag, privileged pod, absent workload identity, or prior digest changed |
Git and the GitOps controller own rendered workload declarations; Terraform must not also declare these Deployments. gitops/apps/forge/base is the DRY invariant, each overlay has the SRP of environment variance, and rendering injects that variance without changing service MVC/DI/PubSub boundaries.
Data model and ownership
Rendered YAML is build output, but losing its provenance makes production impossible to reconstruct. Durable evidence binds source, toolchain, environment, and output digest without duplicating ownership of live Kubernetes state.
Generated-application database: Not created in this slice — packaging produces workload declarations and evidence; generated-app domain databases are consumed by services but not created or owned by the renderer.
| Record or entity | Store and owner | Primary key | Foreign key or opaque reference | Tenant key | Material constraint | Lifecycle and deletion | Use case IDs |
|---|---|---|---|---|---|---|---|
| PackageRevision | Git repository, owned by Platform Engineering | commit_sha plus package_path | Opaque chart and base version refs | None — reusable package has no customer tenant | Dependencies and image references are pinned; secret values are forbidden | Propose, review, merge, deprecate, retain in Git history | D17-UC-01, D17-UC-02 |
| EnvironmentOverlay | Git repository, owned by Environment Owner | commit_sha plus overlay_path | PackageRevision path reference | environment_id | Overlay may change only allowlisted environment fields | Create per environment, review independently, supersede, retain history | D17-UC-01, D17-UC-02 |
| RenderEvidence | Evidence store, owned by Delivery Platform | render_digest | Opaque package revision, overlay, toolchain, and policy refs | environment_id | Same immutable inputs must produce same object digest | Generate, approve or deny, promote approved digest, retain through audit window | D17-UC-01, D17-UC-02 |
The rendered policy result advances to approval or denial, and the evidence preserves the exact output digest. This lets promotion reuse identical bytes while each environment retains its own policy boundary.
Render before reconciliation
Source fragments do not reveal their combined effect, so render and inspect the complete object set. Run this labelled snippet from the public monorepo; the base is gitops/apps/forge/base, overlays are under gitops/apps/forge/overlays, and scripts/verify-platform.sh is the current aggregate gate.
kubectl kustomize gitops/apps/forge/base > /tmp/forge-base.yaml
kubectl kustomize gitops/apps/forge/overlays/local > /tmp/forge-local.yaml
./scripts/verify-platform.sh
./scripts/verify-release.sh staging # Fails closed until real ECR digests and required Secrets exist.
| Effect field | What happens |
|---|---|
| Declared intent | Render the actual Forge base/local overlay and require staging to be digest-pinned before release |
| Interpreter | kubectl kustomize, repository verification scripts, and the named cluster API used by verify-release.sh |
| Software effect | No live workload changes during render and server dry run; evidence files and policy results are created |
| Hardware effect | Local or CI CPU, memory, disk, and network serve rendering and validation only |
| Evidence | Full object YAML, server validation, policy result, diff, source revision, and render digest |
Key takeaways
Package once, overlay only real environment differences, and review the rendered objects.
- Helm packages reusable components; Kustomize composes bases and overlays.
- DRY means one authoritative invariant, not one environment with hidden switches.
- GitOps owns workloads only after deterministic rendering and policy approval.
Checklist
Packaging is ready when production output is predictable before reconciliation.
- [ ] Chart, image, and tool versions are pinned
- [ ] Overlay changes only allowlisted environment fields
- [ ] Render contains no plaintext secrets or mutable tags
- [ ] Invalid overlay and unaffected positive control are recorded