Ship with Kustomize and ApplicationSet

Enterprise problem: A release assembled with copied YAML and pushed from CI can drift between environments, leak production cluster credentials, and leave no single reviewed record of what should run.

Whole-course context: Day 01 produced a Kubernetes cluster, a pinned Argo CD installation, and a declarative bootstrap boundary; today consumes that running control plane and a Git repository.

Today’s slice: We create a small hello-gitops workload, reuse a Kustomize base through development and production overlays, and let a least-privilege ApplicationSet generate both Argo CD Applications.

End-of-day evidence: One Git revision renders valid manifests, produces two bounded Applications, reaches Synced and Healthy, and repairs an observed live-state drift.

Still unsolved: Human SSO, external secrets, staged production approval, rollback drills, alerts, backups, and controller upgrades remain for Day 03.

Copying manifests or giving a pipeline production credentials makes routine delivery both inconsistent and over-privileged, so a safe slice must separate authoring from reconciliation. These use cases cover an engineer proposing a release and a platform operator constraining and recovering it.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D02-UC-01	Application engineer	Propose one reviewed application revision for development and production without editing the cluster	The same Kustomize base renders both overlays, the approved digest is visible in Git, and Argo reports both generated Applications at the intended revision	CI rejects an unpinned image or invalid manifest before merge; no production kubeconfig is present in the job
D02-UC-02	Platform operator	Limit where and what the delivery controller may reconcile, then recover declared state after drift	The AppProject admits only the named repository, namespaces, cluster, and resource kinds; self-heal restores a changed workload to Git	A forbidden kind or destination is denied, and a read-only observation shows the unaffected allowed Application remains Synced and Healthy

A Git commit is not sufficient evidence if nobody can state what must be observed after it merges or what should be denied. These stories turn delivery and platform boundaries into falsifiable acceptance conditions.

Story ID	Use case IDs	User story	Observable acceptance conditions
D02-US-01	D02-UC-01	As an application engineer, I want to promote a digest-pinned image by pull request, so that development and production differences stay explicit and reviewable	Both overlays render locally; schema and policy checks pass; the merge SHA becomes each generated Application’s observed revision; CI has no production cluster credential
D02-US-02	D02-UC-02	As a platform operator, I want Argo CD constrained by an AppProject and driven by an ApplicationSet, so that unauthorized sources, destinations, and kinds cannot enter the reconciliation loop	Two and only two Applications are generated; an out-of-project destination is rejected; live replica drift is restored from Git; the permitted positive control remains healthy

Delivery can appear successful while bypassing review or leaving CI as a hidden cluster administrator, so the path must begin with a developer-visible pull request and end with controller evidence. The recovery path proves both a denial boundary and reconciliation of an allowed resource.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D02-FLOW-01	D02-UC-01	Happy	Engineer opens a pull request changing an overlay image digest	1. Render both overlays from the shared base.<br>2. Validate Kubernetes schemas and policy.<br>3. Review and merge the digest change.<br>4. Argo polls or receives a webhook and reads the merged revision.<br>5. ApplicationSet renders the two Application resources.<br>6. Argo compares desired and live objects, then reconciles only the difference.<br>7. Observe revision, sync, health, image, and response.	Pull-request URL, merge SHA, rendered digest, generated Application names, Synced and Healthy status, rollout condition, and HTTP response captured with environment and timestamp
D02-FLOW-02	D02-UC-02	Recovery	Operator observes a forbidden manifest attempt or live replica drift	1. Submit the change through Git review rather than a cluster command.<br>2. AppProject rejects a source, destination, or kind outside its allow-list.<br>3. Preserve the denial condition and an allowed positive control.<br>4. For permitted live drift, Argo detects OutOfSync against Git.<br>5. Self-heal restores the declared replica count.<br>6. Observe the denied condition or restored resource without mutating it again.	Project-condition message or policy failure, exact resource and scope, before/after replica observation, allowed Application health, Git revision, environment, timestamp, and immutable CI run or audit identifier

If CI and Argo CD can both write workloads, a failed release has two possible reconcilers and no trustworthy owner. This design gives CI read-only access to repository content, gives Argo cluster credentials, and narrows Argo’s authority through one AppProject.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D02-UC-01	Pull request changing apps/hello-gitops/overlays/<environment>/kustomization.yaml	Git provider, CI validators, ApplicationSet controller, Argo repository server, application controller, Kubernetes controllers	Git commit on the protected delivery branch, owned by the application team under review policy	Failed render/schema/policy job, unmerged pull request, Application ComparisonError, failed sync operation, or workload health condition
D02-UC-02	Platform pull request changing bootstrap/argocd/project.yaml or application-set.yaml	Git provider, bootstrap reconciler, AppProject admission rules, ApplicationSet controller, Argo application controller, Kubernetes API	Bootstrap Git revision plus Argo CD AppProject, ApplicationSet, and generated Application custom resources in the cluster control plane	InvalidSpecError, destination or resource denial, generated-Application mismatch, persistent OutOfSync, or missing self-heal event while the allowed positive control stays healthy

The Argo CD control plane owns comparison and synchronization. Kubernetes workload controllers own pods after the desired objects reach the API. The generated hello application owns no user database and grants no application-user identity; a Git-provider account, Argo account, Kubernetes service account, and future application user remain separate authorities.

Treating generated Applications or live Kubernetes objects as the primary source can make a controller artifact look like independently editable business state. Git owns desired delivery state, Argo owns reconciliation records, and Kubernetes owns observed runtime status; generated Applications are derived and must not be hand-edited.

Generated-application database: Not created in this slice — hello-gitops is a stateless web response, while Git revisions and Argo/Kubernetes control-plane records provide all durable desired-state and delivery evidence required today.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
DeliveryRevision	Git repository, owned by application engineering and branch protection	Commit SHA	Parent commit SHA and pull-request URL	Repository plus environment path	Immutable commit; protected branch requires passing checks and review; rendered images must use a digest	Created by merge, retained by Git policy, reverted by a new commit rather than rewritten	D02-UC-01
EnvironmentOverlay	Git repository, owned by the application team for dev or production approvers for prod	Repository path plus commit SHA	Local reference to the shared Kustomize base	Environment name	Overlay contains only a real environment delta and must render exactly the allowed resource set	Evolves through pull requests; removal is a reviewed deletion and does not bypass Argo pruning policy	D02-UC-01, D02-UC-02
ProjectPolicy	Kubernetes API in the Argo namespace, owned by platform engineering	AppProject name hello-gitops	Exact Git repository URL and Kubernetes destination server are opaque external references	None — AppProject is cluster-scoped Argo control-plane policy	Source, destination namespaces, and allowed resource kinds are explicit allow-lists	Reconciled from bootstrap Git; policy changes are audited; deletion is blocked while governed Applications remain	D02-UC-02
ApplicationSetDefinition	Kubernetes API in the Argo namespace, owned by platform engineering	ApplicationSet name hello-gitops	Project name and Git repository URL	Environment element	List has unique environment/name/path tuples; missing template keys fail rendering	Reconciled from bootstrap Git; adding or removing an element creates or retires derived Applications under policy	D02-UC-01, D02-UC-02
GeneratedApplication	Kubernetes API in the Argo namespace, owned by the ApplicationSet controller	Application name hello-gitops-<environment>	Owner reference to ApplicationSet; source revision and destination are opaque bindings	Environment name	Derived spec must match the ApplicationSet template; direct edits are overwritten and are not authority	Created and updated by ApplicationSet; removed only when its generator element is reviewed out	D02-UC-01, D02-UC-02
WorkloadObservation	Kubernetes API status/events plus Argo operation history, owned by Kubernetes and Argo controllers	Object UID plus resource version or operation ID	Opaque Git revision and generated Application name	Kubernetes namespace	Observation must bind revision, environment, resource, expected state, observed state, and time	Live status follows object lifecycle; immutable delivery evidence is exported under audit retention	D02-UC-01, D02-UC-02

Environment folders often become copies that drift, while a single giant folder makes every team an owner of everything. This tree keeps shared workload shape in one base, true environment differences in overlays, and platform policy separate from application delivery.

This is DRY—do not repeat yourself—because Deployment, Service, configuration, and network policy have one definition. It follows the single-responsibility principle (SRP) because application engineers own apps/, platform engineers own bootstrap/argocd/, security owns policy/, and CI only evaluates those owners’ artifacts.

Create the directories, copy the following files, initialize Git, and push the repository to https://github.com/your-org/hello-gitops-delivery.git. Replace that exact example URL in both Argo manifests with the real repository URL; do not broaden it to a wildcard.

hello-gitops-delivery/
├── apps/
│   └── hello-gitops/
│       ├── base/
│       │   ├── deployment.yaml
│       │   ├── service.yaml
│       │   ├── network-policy.yaml
│       │   ├── index.html
│       │   └── kustomization.yaml
│       └── overlays/
│           ├── dev/
│           │   └── kustomization.yaml
│           └── prod/
│               ├── disruption-budget.yaml
│               └── kustomization.yaml
├── bootstrap/
│   └── argocd/
│       ├── project.yaml
│       └── application-set.yaml
└── policy/
    └── kubernetes.rego

A minimal Deployment that omits health, resources, or process restrictions can reconcile successfully while remaining unsafe to operate. The base defines the invariant workload once, and only rendered overlays—not the base alone—are delivery targets.

apps/hello-gitops/base/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-gitops
  labels:
    app.kubernetes.io/name: hello-gitops
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: hello-gitops
  template:
    metadata:
      labels:
        app.kubernetes.io/name: hello-gitops
    spec:
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: web
          image: nginxinc/nginx-unprivileged
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
          volumeMounts:
            - name: content
              mountPath: /usr/share/nginx/html
              readOnly: true
            - name: cache
              mountPath: /var/cache/nginx
            - name: tmp
              mountPath: /tmp
          resources:
            requests:
              cpu: 25m
              memory: 32Mi
            limits:
              cpu: 200m
              memory: 128Mi
          readinessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 2
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 10
            periodSeconds: 10
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
            readOnlyRootFilesystem: true
            runAsUser: 101
            runAsGroup: 101
      volumes:
        - name: content
          configMap:
            name: hello-gitops-content
        - name: cache
          emptyDir: {}
        - name: tmp
          emptyDir: {}

apps/hello-gitops/base/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: hello-gitops
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: hello-gitops
  ports:
    - name: http
      port: 80
      targetPort: http

apps/hello-gitops/base/network-policy.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: hello-gitops
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: hello-gitops
  policyTypes: ["Ingress", "Egress"]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              access: hello-gitops
      ports:
        - protocol: TCP
          port: 8080
  egress: []

apps/hello-gitops/base/index.html contains configuration without secrets:

<!doctype html>
<html lang="en">
  <meta charset="utf-8">
  <title>Hello GitOps</title>
  <h1>Hello from reviewed Git</h1>
  <p>Argo CD reconciled this response.</p>
</html>

apps/hello-gitops/base/kustomization.yaml generates a content-addressed ConfigMap, so a content edit changes its name and rolls the pod template automatically:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
  - network-policy.yaml
configMapGenerator:
  - name: hello-gitops-content
    files:
      - index.html

The workload does not mount a service-account token, runs as a non-root numeric user, drops Linux capabilities, uses a read-only root filesystem, and declares probes plus resource requests and limits. The empty egress list is intentional because this static server makes no outbound calls; add narrowly scoped DNS or service rules only when a real dependency exists.

Changing shared YAML independently in each environment creates invisible skew and makes promotion a copy operation. Kustomize overlays import the same base and declare only replica, disruption, and digest differences.

apps/hello-gitops/overlays/dev/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
replicas:
  - name: hello-gitops
    count: 1
images:
  - name: nginxinc/nginx-unprivileged
    newName: nginxinc/nginx-unprivileged
    digest: sha256:65e3e85dbaed8ba248841d9d58a899b6197106c23cb0ff1a132b7bfe0547e4c0

apps/hello-gitops/overlays/prod/disruption-budget.yaml protects one healthy replica during voluntary disruption:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: hello-gitops
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: hello-gitops

apps/hello-gitops/overlays/prod/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
  - disruption-budget.yaml
replicas:
  - name: hello-gitops
    count: 2
images:
  - name: nginxinc/nginx-unprivileged
    newName: nginxinc/nginx-unprivileged
    digest: sha256:65e3e85dbaed8ba248841d9d58a899b6197106c23cb0ff1a132b7bfe0547e4c0

The multi-platform digest above resolved for nginxinc/nginx-unprivileged:1.27-alpine on the publication date; verify the publisher, signature or provenance, vulnerability policy, and changelog before deliberately replacing it. A digest makes the selected bytes immutable, while a tag alone can move.

An Argo installation with a broad default project can turn one compromised repository into cluster-wide write access. The AppProject below allows one exact source, one in-cluster API, two namespaces, and only the namespaced kinds used by this lesson plus Namespace creation.

bootstrap/argocd/project.yaml:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: hello-gitops
  namespace: argocd
spec:
  description: Least-privilege delivery for hello-gitops
  sourceRepos:
    - https://github.com/your-org/hello-gitops-delivery.git
  destinations:
    - server: https://kubernetes.default.svc
      namespace: hello-dev
    - server: https://kubernetes.default.svc
      namespace: hello-prod
  clusterResourceWhitelist:
    - group: ""
      kind: Namespace
  namespaceResourceWhitelist:
    - group: ""
      kind: ConfigMap
    - group: ""
      kind: Service
    - group: apps
      kind: Deployment
    - group: networking.k8s.io
      kind: NetworkPolicy
    - group: policy
      kind: PodDisruptionBudget

This policy does not grant Secret, Role, RoleBinding, custom-resource, or arbitrary cluster-scoped writes. Repository credentials, if the repository is private, are bootstrap-owned secret references supplied outside plaintext Git; they do not belong in this application repository.

Hand-writing one Application per environment repeats policy and lets a missing value silently target the wrong place. A list generator makes the supported environment inventory explicit, while Go-template strictness fails on any absent key.

bootstrap/argocd/application-set.yaml:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: hello-gitops
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - list:
        elements:
          - environment: dev
            namespace: hello-dev
            path: apps/hello-gitops/overlays/dev
          - environment: prod
            namespace: hello-prod
            path: apps/hello-gitops/overlays/prod
  template:
    metadata:
      name: 'hello-gitops-{{ .environment }}'
      labels:
        app.kubernetes.io/part-of: hello-gitops
        environment: '{{ .environment }}'
    spec:
      project: hello-gitops
      source:
        repoURL: https://github.com/your-org/hello-gitops-delivery.git
        targetRevision: main
        path: '{{ .path }}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{ .namespace }}'
      syncPolicy:
        automated:
          enabled: true
          prune: true
          selfHeal: true
          allowEmpty: false
        syncOptions:
          - CreateNamespace=true
          - PruneLast=true

CreateNamespace=true is bounded because the AppProject and list generator name exactly hello-dev and hello-prod; it removes an otherwise imperative bootstrap step. PruneLast=true delays deletions until other synchronization work is healthy, reducing replacement risk. allowEmpty: false prevents a broken path or empty render from pruning the entire application, while prune: true removes reviewed deletions and selfHeal: true reverses live drift.

Do not edit either generated Application: the ApplicationSet controller owns its spec and will restore the template. To pause or change automated behavior, review a change to the ApplicationSet template; changing one generated child is transient configuration drift.

A pipeline that runs deployment commands still owns production credentials even if the manifests live in Git, so it has not achieved the key GitOps security boundary. Inversion of control (IoC) means the in-cluster controller observes reviewed Git and decides how to converge the cluster.

The Git provider authenticates reviewers and protects the branch. CI may read the repository and publish checks, but it holds neither a production kubeconfig nor an Argo API token capable of syncing. Argo CD holds in-cluster authority, and the AppProject reduces that authority to the declared source, destinations, and kinds.

No step issues a direct cluster write or an imperative Argo creation or sync command. Day 01’s bootstrap reconciler installs the AppProject and ApplicationSet declarations; from this point onward, reviewed Git changes are the only desired-state input.

Argo can faithfully reconcile malformed or unsafe intent, so pull requests need fast local and CI rejection before desired state changes. Rendering, schema validation, and policy validation catch different defect classes and should all run from a clean checkout.

Install pinned versions of Kustomize or use the Kustomize bundled with your pinned kubectl; install kubeconform for schema validation and conftest for policy. Then render each delivery target without contacting a cluster:

set -euo pipefail
work_dir="$(mktemp -d)"
trap 'rm -rf "$work_dir"' EXIT

for environment in dev prod; do
  kubectl kustomize "apps/hello-gitops/overlays/${environment}" \
    > "${work_dir}/${environment}.yaml"
  kubeconform -strict -summary -ignore-missing-schemas \
    "${work_dir}/${environment}.yaml"
  conftest test "${work_dir}/${environment}.yaml" --policy policy
done

policy/kubernetes.rego rejects mutable image references and missing compute bounds:

package kubernetes

deny contains message if {
  input.kind == "Deployment"
  some container in input.spec.template.spec.containers
  not contains(container.image, "@sha256:")
  message := sprintf("%s uses an image that is not pinned by digest", [container.name])
}

deny contains message if {
  input.kind == "Deployment"
  some container in input.spec.template.spec.containers
  not container.resources.requests.cpu
  message := sprintf("%s has no CPU request", [container.name])
}

deny contains message if {
  input.kind == "Deployment"
  some container in input.spec.template.spec.containers
  not container.resources.limits.memory
  message := sprintf("%s has no memory limit", [container.name])
}

Pin the validator versions in the CI image or tool manifest and record their versions with the run. The CI identity needs permission to publish a check result, not permission to read a production cluster; any KUBECONFIG or Argo deployment token in that job should fail the pipeline’s credential audit.

Promoting by rebuilding an image tag or copying an entire directory can change unreviewed bytes or reintroduce environment skew. Promotion is one Git pull request that changes the production overlay’s digest to the digest already proven in development.

 images:
   - name: nginxinc/nginx-unprivileged
     newName: nginxinc/nginx-unprivileged
-    digest: sha256:OLD_REVIEWED_DIGEST
+    digest: sha256:NEW_DIGEST_PROVEN_IN_DEV

The pull request should link the development Application revision, rollout health, vulnerability and provenance results, and the exact digest. A production approver reviews only the deliberate delta; after merge, Argo pulls the commit and reconciles production. Never retag a different image under an approved digest claim, and never make a UI sync the promotion mechanism.

Rollback follows the same ownership rule: revert the faulty digest commit through a reviewed pull request. Git then contains both the failed decision and its correction, and automated sync converges to the previously known digest without CI acquiring cluster credentials.

A green initial deployment proves only the easiest path, so an operator needs evidence that unsafe intent stops and allowed drift recovers. Run these drills in a disposable branch or approved test environment, and preserve the allowed positive control throughout.

Drill	Safe injection	Expected controller result	Read-only terminal proof
Missing template field	Remove namespace from one list element in a pull request	missingkey=error prevents that Application template from rendering	ApplicationSet condition names the missing key; existing allowed Application remains healthy
Forbidden resource	Add a ClusterRole to the overlay in a pull request	CI policy should fail first; AppProject also blocks the kind if it reaches comparison	Failed CI run or Application condition names the denied group/kind
Forbidden destination	Change one element to kube-system in a pull request	AppProject rejects the destination	Application condition names the invalid destination; hello-dev stays healthy
Empty render	Point a test element at an empty directory	allowEmpty: false prevents automated deletion	Existing managed resources remain listed and the Application reports comparison failure
Live replica drift	In a controlled lab only, alter the live replica count through an approved fault injector	Self-heal returns the Deployment to the Git-rendered count	Successive read-only observations show OutOfSync then declared replicas restored
Bad image digest	Promote a nonexistent digest in a disposable branch	Rollout becomes degraded without changing the approved Git history	Pod waiting reason, Application health, and unchanged previous Git revision are captured before revert

Do not use a manual edit as the normal operating path. If an approved chaos tool injects live drift, its identity, target, start time, expected result, observed result, environment, cleanup, and immutable run ID are part of the evidence; recovery authority remains with Argo’s control loop.

A dashboard screenshot can hide the wrong revision or namespace, so completion needs machine-readable observations tied to Git. These commands read Argo and Kubernetes state only; they do not create, patch, delete, or synchronize anything.

# Exactly two generated Applications, their revision, sync, and health
kubectl get applications.argoproj.io -n argocd \
  -l app.kubernetes.io/part-of=hello-gitops \
  -o custom-columns='NAME:.metadata.name,REVISION:.status.sync.revision,SYNC:.status.sync.status,HEALTH:.status.health.status'

# Generator and project conditions explain denial or rendering failures
kubectl get applicationsets.argoproj.io hello-gitops -n argocd -o yaml
kubectl get appprojects.argoproj.io hello-gitops -n argocd -o yaml

# Runtime image, availability, and Service endpoints
kubectl get deployment -n hello-dev hello-gitops \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}{.status.availableReplicas}{"\n"}'
kubectl get deployment -n hello-prod hello-gitops \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}{.status.availableReplicas}{"\n"}'
kubectl get service,endpoints -n hello-dev
kubectl get events -n hello-dev --sort-by=.metadata.creationTimestamp

For the HTTP response, use an already-approved internal probe pod labelled access=hello-gitops or the organization’s read-only synthetic monitor. The NetworkPolicy deliberately denies arbitrary callers, so weakening it merely to make a local curl pass would invalidate the security evidence.

Capture actor, resource, scope, precondition, expected result, observed result, environment, UTC timestamp, Git SHA, and immutable CI/audit ID for both positive and negative paths. Synced proves live manifests match desired manifests; Healthy separately proves Argo’s health assessment, so require both.

Examples copied without their version context age into unsafe defaults, so use primary documentation to re-check fields and controller behavior before upgrading. These links are the source basis for this lesson and should be reviewed alongside the pinned Argo CD and Kubernetes changelogs.

Primary sources: Kubernetes Kustomize, Argo declarative setup, Projects, ApplicationSet, ApplicationSet specification, automated sync, and sync options.

The course publishes against the Day 01 Argo CD pin and this date’s Kubernetes APIs. Before changing a chart, controller, CRD, validator, or base image, review release notes, render in a clean environment, repeat denial and drift drills, and promote the resulting digest by pull request.