Package Kubernetes Workloads with Kustomize
Reuse one workload design while keeping dev, staging, and production differences small, explicit, and reviewable.
System map · Day 17
Whole-system design
Five stable layers. Today's work is expanded and linked; the rest stays in context.
Product and authority
Covered — People and product entry points · Identity and policy · HelixWorks control plane · Generated application plane
Delivery and desired state
Git desired state
Source-backed today
Stores one reusable workload base plus explicit environment overlays as the reviewed desired state.
CI and immutable artifacts
Source-backed today
Pins configuration and immutable image inputs before a release is eligible for promotion.
Argo CD reconciliation
Source-backed today
Consumes the rendered Kustomize output only after validation makes the resulting object set visible.
Cloud and orchestration
Covered — Terraform and AWS APIs · Accounts, VPC, DNS, and private paths · Kubernetes or EKS control plane
Compute and traffic
Covered — Worker compute · Platform service workloadsAhead — Ambient mesh data plane
Generated app workloads
Source-backed today
Receives environment-specific configuration without duplicating the entire workload manifest.
Storage and evidence
Covered — Product data and artifacts · Evidence and observabilityAhead — Infrastructure state · Cluster desired and live state
The enterprise problem and today’s slice
Enterprise problem: Copying complete Kubernetes manifests for every HelixWorks service and environment causes security fixes, resource limits, and labels to drift, while an overly clever template can hide what production will run. Whole-course context: The incoming evidence is a service map with explicit contracts and owners; today packages those services for Kubernetes without duplicating their whole desired state. Today’s slice: We use the repository’s actual Kustomize base-and-overlay design under gitops/apps/forge; there is no HelixWorks Helm chart, so rendered Kubernetes objects are the honest review boundary. End-of-day evidence: A render run proves all three application archetypes use the same base, each environment changes only declared policy, blocked cloud overlays fail closed until digest promotion, no secret values appear, and output records source revision, tool version, environment, time, run, and digest. Still unsolved: Infrastructure versus workload state, Terraform, and AWS accounts remain deferred.
Customer outcome and implementation focus
The customer outcome is a reviewable package kubernetes workloads with kustomize change, not a collection of requirements. This day starts with the implementation boundary, then uses the command or manifest below to produce positive, denied, and recovery evidence.
Components in focus
Helm renderer, Kustomize overlay builder, Argo CD repo server, and application controller; controller pods; Git repository and Kubernetes API are authoritative; cache: repo-server clone cache is an optimisation, never deployment truth.
This map names the implementation boundary for this day. The service or controller changes only the state it owns; runtime and audit evidence let the operator distinguish a declared change from an effective one.
Render Helm and Kustomize inputs before GitOps applies them
Render before reconciliation
Source fragments do not reveal their combined effect, so render and inspect the complete object set. Run this labelled snippet from the public monorepo; the base is gitops/apps/forge/base, overlays are under gitops/apps/forge/overlays, and scripts/verify-platform.sh is the current aggregate gate.
kubectl kustomize gitops/apps/forge/base > /tmp/forge-base.yaml
kubectl kustomize gitops/apps/forge/overlays/local > /tmp/forge-local.yaml
./scripts/verify-platform.sh
./scripts/verify-release.sh staging # Fails closed until real ECR digests and required Secrets exist.
Key takeaways
Package once, overlay only real environment differences, and review the rendered objects.
- Helm packages reusable components; Kustomize composes bases and overlays.
- DRY means one authoritative invariant, not one environment with hidden switches.
- GitOps owns workloads only after deterministic rendering and policy approval.
Checklist
Packaging is ready when production output is predictable before reconciliation.
- [ ] Chart, image, and tool versions are pinned
- [ ] Overlay changes only allowlisted environment fields
- [ ] Render contains no plaintext secrets or mutable tags
- [ ] Invalid overlay and unaffected positive control are recorded