Promote, Secure, Recover, and Operate GitOps

Enterprise problem: A release that depends on console changes, mutable image tags, or a privileged CI kubeconfig cannot prove what reached production; recovery becomes a second outage when the original operator is unavailable.

Whole-course context: Day 01 produced a code-owned cluster and pinned Argo CD bootstrap, and Day 02 produced a hello-gitops Kustomize base, environment overlays, AppProject, and ApplicationSet. Those reviewed repositories and their commit history are today’s incoming evidence.

Today’s slice: Promote the same verified image digest from development to production, constrain human and controller authority, reconcile drift, observe failure, recover through Git, and reconstruct the delivery system.

End-of-day evidence: A merged promotion or recovery commit maps to an immutable digest, Argo application revision, healthy workload, policy results, notification, and timestamped incident record.

Still unsolved: Cloud-provider-specific identity, multi-region application data recovery, and organisation-specific regulatory retention need separate threat models and tested recovery objectives.

Production delivery fails when the customer’s release, security, and recovery jobs are treated as one privileged operation; one mistake can then bypass review or erase evidence. These use cases separate the jobs and make denial or recovery visible.

Use case ID	Actor	Customer job	Success outcome	Denial or recovery evidence
D03-UC-01	Release manager	Promote the exact artifact proven in development into production through a pull request	Production reports the reviewed Git revision and the same immutable image digest that passed development checks	A failing policy or missing approval blocks merge; a revert PR restores the last known-good digest and records both commits
D03-UC-02	Platform security engineer	Keep repository, destination, resource, identity, and secret authority least-privileged while Argo repairs unauthorised drift	AppProject, SSO/RBAC, external-secret references, and self-heal converge without exposing plaintext credentials	Conftest denies an unsafe manifest, RBAC denies a mutation, and Argo records the drift plus the repaired resource version
D03-UC-03	Site reliability engineer	Detect degradation, preserve incident evidence, restore the GitOps control plane, upgrade it safely, or retire an application	Alerts and notifications identify the revision; a pinned bootstrap reconstructs Argo and its applications within the recovery objective	Read-only evidence identifies the failed revision; a tested restore or reviewed deletion commit reaches a healthy or intentionally absent terminal state

Vague roles such as “the DevOps team” hide who may approve, observe, or recover a change, so unsafe authority accumulates. Actor-centred stories give each person a bounded action and an observable acceptance test.

Story ID	Use case IDs	User story	Observable acceptance conditions
D03-US-01	D03-UC-01	As a release manager, I want a pull request to copy a development-proven digest into the production overlay, so that production cannot silently run a different artifact	The diff changes only the production digest; required checks and approval pass; Argo reports the merge SHA and the workload reports that digest
D03-US-02	D03-UC-01, D03-UC-03	As an incident commander, I want rollback to be a reviewed Git revert, so that desired state, audit history, and automatic reconciliation agree during recovery	The revert names the bad commit and incident; no Argo rollback command is used; the last known-good digest becomes Synced and Healthy
D03-US-03	D03-UC-02	As a platform security engineer, I want SSO groups, AppProjects, and destination-side secret retrieval to bound authority, so that neither a developer nor Argo can read or deploy more than required	A permitted read succeeds; an application mutation is denied; plaintext Secret policy passes; the external-secret controller alone retrieves the value
D03-US-04	D03-UC-02, D03-UC-03	As an on-call engineer, I want drift and reconciliation failures to page the owning team with revision evidence, so that repair is fast and attributable	Prometheus fires within the stated threshold; Argo Notifications includes application, project, revision, sync, and health; durable logs retain the event
D03-US-05	D03-UC-03	As a disaster-recovery lead, I want pinned infrastructure and bootstrap declarations plus recoverable secret sources, so that a replacement cluster can be reconstructed without console memory	A clean environment reaches the expected cluster, Argo chart, ApplicationSet-generated applications, digest, and health using reviewed OpenTofu and Git inputs

A production change can look successful at merge time yet fail during reconciliation or runtime, leaving customers with no trustworthy terminal state. These flows begin at a visible release or operations action and end with evidence rather than a deployment verb.

Flow ID	Use case IDs	Path	Trigger	Numbered steps	Terminal evidence
D03-FLOW-01	D03-UC-01	Happy	Release manager opens a production promotion pull request	1. Automation reads the digest already proven by the development revision.<br>2. The PR changes only the production overlay digest.<br>3. CI renders and validates every overlay.<br>4. Required owners approve and merge.<br>5. Argo pulls the merge revision and reconciles production.<br>6. The manager observes Synced, Healthy, and the running digest.	Actor, repository, PR, approvals, source and merge SHAs, digest, Argo revision, environment, health, and timestamps
D03-FLOW-02	D03-UC-01, D03-UC-02	Denied	A contributor proposes a mutable tag, plaintext Secret, forbidden kind, or unreviewed production change	1. Branch protection prevents direct production-branch mutation.<br>2. CI renders the proposed state.<br>3. Conftest, schema, or secret scanning rejects it.<br>4. Merge remains blocked.<br>5. Current production stays at its known-good revision.	Failed rule and check run, denied PR SHA, unchanged production revision and digest, plus an unaffected healthy probe
D03-FLOW-03	D03-UC-01, D03-UC-03	Recovery	On-call declares the newly promoted revision unhealthy	1. The incident record names the bad promotion SHA and last known-good SHA.<br>2. An engineer opens a revert branch and PR.<br>3. Normal validation and emergency approval policy run.<br>4. The revert merges as a new commit.<br>5. Argo reconciles that commit.<br>6. On-call observes the old digest healthy and closes the incident.	Incident ID, bad and revert SHAs, approval, digest transition, sync result, health probe, notification, and recovery time
D03-FLOW-04	D03-UC-02	Recovery	Argo detects a live Deployment field that differs from reviewed Git	1. The controller compares live state with the tracked revision.<br>2. Self-heal schedules reconciliation.<br>3. Argo restores the declared value.<br>4. Security reviews audit and Kubernetes events.<br>5. The owner verifies Git and live state match.	Diff path, actor or service identity when available, application revision, reconciliation operation, restored resource version, and timestamp
D03-FLOW-05	D03-UC-02, D03-UC-03	Failure	Reconciliation remains OutOfSync or health becomes Degraded beyond the alert threshold	1. Prometheus evaluates Argo metrics.<br>2. Alertmanager routes the page by project or team label.<br>3. Argo Notifications sends application and revision context.<br>4. On-call gathers read-only evidence.<br>5. The incident selects fix-forward, Git revert, or control-plane restore.	Alert fingerprint, notification delivery, application condition, Git revision, resource events, incident decision, and immutable log location
D03-FLOW-06	D03-UC-03	Recovery	Disaster-recovery lead declares the cluster or Argo control plane unrecoverable in place	1. The lead selects a signed Git recovery tag and versioned OpenTofu state.<br>2. The trusted infrastructure pipeline creates the replacement cluster.<br>3. The bootstrap state installs the pinned Argo chart and root declarations.<br>4. External-secret controllers repopulate required runtime secrets from the replicated secret service.<br>5. ApplicationSet recreates applications from Git.<br>6. The lead compares expected digests, sync, health, alerts, and recovery time with the exercise record.	Recovery tag, state version, plan/apply run IDs, chart version, application revisions and digests, secret readiness without values, RTO result, and sign-off

If CI, Argo, and operators all mutate production, incident responders cannot identify the authoritative writer and credentials spread across systems. The design therefore gives each flow one entry point, one desired-state source, and narrowly responsible reconcilers.

Use case ID	Entry point	Responsible services	Authoritative store	Failure evidence
D03-UC-01	Pull request against environments/prod	Git host branch protection and CODEOWNERS; CI render/policy jobs; Argo application controller; Kubernetes Deployment controller	Protected Git repository owns desired manifests and promotion history; OCI registry owns immutable artifact bytes and attestations	Blocked check or approval, Argo sync condition, Kubernetes rollout condition, digest mismatch, and PR-to-revision audit gap
D03-UC-02	Reviewed policy, AppProject, RBAC, or external-secret declaration	CI policy engine; Argo server SSO/RBAC; AppProject admission boundary; Argo application controller; external-secret controller; cloud secret service	Protected platform Git owns policy and references; identity provider owns groups; cloud secret service owns secret values; Kubernetes API owns observed resource state	Policy denial, authentication or RBAC denial, AppProject condition, secret-controller condition, Argo diff, audit event, and self-heal result
D03-UC-03	Incident, recovery, upgrade, or retirement pull request plus approved infrastructure run	Prometheus and Alertmanager; Argo Notifications; log archive; trusted OpenTofu runner; Argo/ApplicationSet controllers; backup and secret services	Incident system and immutable log archive own evidence; versioned OpenTofu backend owns infrastructure state; signed Git tag owns declarations; secret service owns required recovery values	Alert fingerprint, missing backup component, failed plan/apply run, unavailable secret reference, Application condition, RTO/RPO breach, or incomplete prune

CI has read-only repository and registry access during validation and no production kubeconfig. Argo owns workload reconciliation with a service account limited to its AppProject destinations and kinds; a trusted bootstrap pipeline alone assumes short-lived infrastructure authority for cluster or Argo reconstruction.

Recovery fails when teams back up controller caches but omit Git revisions, artifact bytes, identity mappings, or secret sources that actually reconstruct service. This ownership model distinguishes durable authority from rebuildable observations and states deletion explicitly.

Generated-application database: Not created in this slice — hello-gitops remains stateless, while protected Git, OCI, Kubernetes control-plane status, identity, secret-service metadata, and incident evidence durably support promotion and recovery.

Record or entity	Store and owner	Primary key	Foreign key or opaque reference	Tenant key	Material constraint	Lifecycle and deletion	Use case IDs
Desired-state revision	Protected Git repository owned by the platform source team	Commit SHA	Parent commit SHA and opaque PR ID	Repository or platform team ID	Production paths require passing checks, CODEOWNER approval, and non-force-pushed history	Created by merge; retained to policy; reverted by a new commit; repository retirement requires archive and approval	D03-UC-01, D03-UC-02, D03-UC-03
Promotion record	Git pull request and release evidence store owned by release engineering	PR ID plus merge SHA	Opaque development evidence run ID and OCI digest	Application team ID	Production digest must equal a development-proven digest and the PR changes only approved environment fields	Closed or merged with immutable audit; retained through release policy; deletion follows evidence-retention approval	D03-UC-01, D03-UC-03
Artifact and attestation	OCI registry owned by supply-chain engineering	Repository plus sha256 digest	Opaque build commit and CI run IDs	Registry project ID	Digest is immutable; signature, provenance, and vulnerability policy must bind to that digest	Retained beyond rollback window; garbage collection is blocked while any environment or evidence record references it	D03-UC-01, D03-UC-02, D03-UC-03
Application declaration	Kubernetes API in the Argo control cluster, reconciled by ApplicationSet	Application UID	Git URL, path, target revision, destination cluster reference, and AppProject name	AppProject name	Generator output must stay within fixed project and destination boundaries; direct generated-Application edits are overwritten	Recreated from Git; deletion comes from a reviewed generator change and cascades only under declared finalizer/prune policy	D03-UC-01, D03-UC-02, D03-UC-03
Application observation	Kubernetes API status subresource owned by Argo application controller	Application UID plus observed revision	Opaque desired-state commit SHA and destination resource UIDs	AppProject name	Sync and health must refer to the observed revision; status is evidence but not desired-state authority	Continuously replaced; durable copies flow to metrics and logs; expires with the Application after retention export	D03-UC-01, D03-UC-02, D03-UC-03
Identity-to-role mapping	Identity provider groups plus declarative argocd-rbac-cm, owned jointly by identity and platform security	IdP group ID plus Argo role name	Opaque user subject IDs; no copied passwords	Organisation ID	Default authenticated role has minimal permissions; production mutation is absent; group mappings are independently revocable	Joiner/mover/leaver lifecycle in IdP; Git review changes roles; audit retained; built-in admin stays disabled	D03-UC-02, D03-UC-03
External secret reference	Protected Git manifest owned by platform security and secret metadata owned by cloud security	Namespace, name, and remote secret key reference	Opaque secret-service version or alias; never the secret value	Cluster and application team ID	Plaintext Kubernetes Secret manifests are denied; workload access and controller retrieval grants are separate and least-privileged	Reference deleted by reviewed prune; value versions rotate and expire in secret service; recovery copies follow secret-service policy	D03-UC-02, D03-UC-03
Incident evidence bundle	Append-only log/object store owned by reliability engineering	Incident ID plus evidence timestamp	PR, commit, digest, Application UID, alert fingerprint, and infrastructure run IDs	Service or owning team ID	Actor, resource, scope, precondition, expected, observed, environment, time, and immutable identifier are mandatory	Written during release and incident flows; retained to policy; legal hold overrides expiry; verified deletion emits a tombstone	D03-UC-01, D03-UC-02, D03-UC-03
Recovery set	Versioned OpenTofu backend, Git mirror, OCI retention policy, and secret-service replica owned by platform recovery	Exercise or recovery-set ID	State version, signed Git tag, chart version, digest inventory, secret replica generation	Platform environment ID	Every referenced component must be recoverable together and pass scheduled restore tests within RTO and RPO	Created before risky upgrades and on schedule; expired only after a newer tested set exists; destruction requires two-person approval	D03-UC-03

Copy-pasted environments drift, mixed ownership makes reviews meaningless, and push-based pipelines accumulate cluster credentials. Apply DRY (one reusable definition), SRP (one reason for each module to change), and IoC (the controller pulls desired state) to keep the delivery path explainable.

• DRY: the base owns common Deployment, Service, probes, and resource policy; overlays contain only genuine environment variance such as replicas and digest.
• SRP: application teams own workload intent, platform security owns policy and roles, release engineering owns promotion automation, and recovery engineering owns tested reconstruction.
• IoC: CI proves a commit but never calls Kubernetes or Argo mutation APIs; Argo pulls Git and reconciles with cluster-local credentials.
• One reconciler per resource: OpenTofu owns the cluster and Argo bootstrap; Argo owns post-bootstrap platform and application workloads. Never declare the same object in both states.

gitops/
├── apps/hello-gitops/base/                 # DRY workload structure
├── apps/hello-gitops/overlays/dev/         # environment-owned differences
├── apps/hello-gitops/overlays/prod/        # protected digest promotion
├── platform/argocd/projects/               # AppProject and sync windows
├── platform/argocd/applicationsets/        # environment discovery
├── platform/argocd/rbac/                   # SSO group mappings
├── platform/external-secrets/              # references, never values
├── platform/observability/                  # alerts and notifications
├── policy/                                  # Conftest rules
└── infra/{cluster,bootstrap}/               # separate OpenTofu states

Mutable tags make development evidence unrelated to production bytes, so a rollback may fetch different content under the same name. Promotion copies a sha256 digest already observed in development and changes no image build inputs.

# apps/hello-gitops/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
images:
  - name: nginxinc/nginx-unprivileged
    newName: nginxinc/nginx-unprivileged
    digest: sha256:65e3e85dbaed8ba248841d9d58a899b6197106c23cb0ff1a132b7bfe0547e4c0
patches:
  - path: replicas.yaml

The release job may prepare a branch, but only the protected pull request merges it. It reads the development evidence, verifies the registry attestation binds to that digest, updates the production overlay, and attaches those immutable IDs to the PR; it does not log in to the cluster.

# Run in a short-lived promotion branch; IMAGE_DIGEST comes from signed dev evidence.
test "${IMAGE_DIGEST#sha256:}" != "$IMAGE_DIGEST"
test "$(printf %s "${IMAGE_DIGEST#sha256:}" | wc -c | tr -d ' ')" = 64
yq -i '.images[] |= select(.name == "nginxinc/nginx-unprivileged").digest = strenv(IMAGE_DIGEST)' \
  apps/hello-gitops/overlays/prod/kustomization.yaml
test -z "$(git diff --name-only -- apps/hello-gitops/base policy platform)"
git diff -- apps/hello-gitops/overlays/prod

Branch protection requires successful validation, signed commits where available, one application-owner approval, one production-owner approval, linear non-force-pushed history, and resolved conversations. Use environment-specific CODEOWNERS so changing policy or production cannot be self-approved by the author.

Invalid YAML is only the easiest failure; a syntactically valid manifest can still request a forbidden kind, omit resources, or contain a credential. CI renders exactly what Argo will render, validates schemas and policy, and scans the repository before merge.

# .github/workflows/gitops-validate.yaml
name: gitops-validate
on:
  pull_request:
permissions:
  contents: read
jobs:
  validate:
    runs-on: ubuntu-latest
    container:
      # Organisation image contains pinned tofu, kustomize, kubeconform, conftest, and gitleaks.
      image: ghcr.io/example/platform-ci@sha256:7a51e2ed6e67c6e84cd0e5551ae781f54d60f4c2da4e8bb9ecba0a79decc8763
    steps:
      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
      - name: Validate OpenTofu roots
        run: |
          tofu fmt -check -recursive infra
          for root in infra/cluster infra/bootstrap; do
            tofu -chdir="$root" init -backend=false
            tofu -chdir="$root" validate
          done
      - name: Render and validate Kubernetes manifests
        run: |
          mkdir -p rendered
          for env in dev prod; do
            kustomize build "apps/hello-gitops/overlays/$env" > "rendered/$env.yaml"
            kubeconform -strict -summary -ignore-missing-schemas "rendered/$env.yaml"
            conftest test "rendered/$env.yaml" --policy policy
          done
      - name: Reject committed credentials
        run: gitleaks detect --source . --redact --no-banner

Pin every action by commit and every tool image by digest; a human-readable tag may appear only as a comment. -ignore-missing-schemas requires a reviewed exception for custom resources, while built-in Kubernetes resources remain strictly checked.

# policy/kubernetes.rego
package kubernetes.delivery

deny contains message if {
  input.kind == "Secret"
  message := sprintf("plaintext Secret %s is forbidden; declare an ExternalSecret reference", [input.metadata.name])
}

deny contains message if {
  input.kind == "Deployment"
  container := input.spec.template.spec.containers[_]
  not regex.match("@sha256:[a-f0-9]{64}$", container.image)
  message := sprintf("container %s must use an immutable sha256 digest", [container.name])
}

deny contains message if {
  input.kind == "Deployment"
  container := input.spec.template.spec.containers[_]
  not container.resources.requests.memory
  message := sprintf("container %s needs a memory request", [container.name])
}

Manual sync buttons turn reviewed Git into a suggestion and make recovery dependent on whoever has console access. Automated sync makes the tracked commit authoritative, while prune, self-heal, retry, windows, and waves remain explicit risk decisions.

# In the ApplicationSet template from Day 02
spec:
  syncPolicy:
    automated:
      enabled: true
      prune: true
      selfHeal: true
      allowEmpty: false
    retry:
      refresh: true
      limit: 5
      backoff:
        duration: 10s
        factor: 2
        maxDuration: 3m
    syncOptions:
      - ApplyOutOfSyncOnly=true
      - PruneLast=true

selfHeal restores unauthorised live drift; it does not legitimise testing drift through manual edits. Inspect drift with the UI, argocd app diff, or Kubernetes reads, then investigate the actor through audit logs. Change an ApplicationSet-managed application only at its generator or template source because direct child edits are overwritten.

Automated sync documents a crucial constraint: Argo rollback is unavailable while automated sync is enabled. Keep auto-sync on and rollback by reverting Git, producing a new reviewed desired-state revision.

Sync windows are justified only for a real business freeze, such as a regulated production blackout; otherwise they delay recovery and create false safety. Declare the window in the AppProject, keep manual bypass disabled, and change the window itself through emergency review.

spec:
  syncWindows:
    - kind: deny
      schedule: "0 22 * * 1-5"
      duration: 10h
      timeZone: Europe/London
      applications:
        - "hello-gitops-prod"
      manualSync: false

Use sync waves only where Kubernetes readiness cannot express ordering. For example, place an ExternalSecret at wave -1 and the dependent Deployment at wave 0; the Deployment must still fail safely until the target Secret exists because a wave is ordering, not proof that another controller finished.

A controller that can deploy any repository to any namespace is a cluster-admin pathway, and a shared local admin password defeats individual attribution. AppProjects restrict Argo’s deployment envelope; SSO (single sign-on) and RBAC (role-based access control) restrict human observation, while the built-in administrator is disabled after SSO is proven.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: platform-prod
  namespace: argocd
spec:
  sourceRepos:
    - https://github.com/example/gitops.git
  destinations:
    - server: https://kubernetes.default.svc
      namespace: hello-prod
  clusterResourceWhitelist: []
  namespaceResourceWhitelist:
    - group: ""
      kind: ConfigMap
    - group: ""
      kind: Service
    - group: apps
      kind: Deployment
    - group: external-secrets.io
      kind: ExternalSecret
  orphanedResources:
    warn: true

The production namespace is created by the bootstrap owner, not by this AppProject, so the application needs no cluster-scoped Namespace permission. The Projects guide explains repository, destination, and resource-kind boundaries.

# Pinned Argo Helm values, reconciled by the bootstrap OpenTofu state.
configs:
  cm:
    admin.enabled: "false"
    url: https://argocd.example.com
    oidc.config: |
      name: Corporate OIDC
      issuer: https://id.example.com/oauth2/default
      clientID: argocd-production
      clientSecret: $argocd-oidc:clientSecret
      requestedScopes: ["openid", "profile", "email", "groups"]
  rbac:
    policy.default: role:authenticated
    scopes: '[groups]'
    policy.csv: |
      p, role:prod-readonly, applications, get, platform-prod/*, allow
      p, role:prod-readonly, logs, get, platform-prod/*, allow
      g, platform-prod-readers, role:prod-readonly

role:authenticated intentionally has no additional policy lines; the RBAC guide warns that default-role permissions cannot later be removed with a deny. Test SSO group claims and read-only access before setting admin.enabled to false in a reviewed bootstrap change; retain a documented break-glass identity at the external identity provider, not a shared Argo account.

Secret values never enter Git, CI-rendered manifests, Argo’s repository cache, diffs, or notifications. Prefer a destination-cluster controller, as the official secret-management guidance recommends; SOPS is acceptable for bootstrap material only when ciphertext, recipient policy, key custody, rotation, and recovery are independently controlled.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: hello-gitops-runtime
  namespace: hello-prod
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: production-secrets
  target:
    name: hello-gitops-runtime
    creationPolicy: Owner
  data:
    - secretKey: API_TOKEN
      remoteRef:
        key: /hello-gitops/production
        property: api_token

The platform bootstrap owns ClusterSecretStore and workload identity; the application owns only its ExternalSecret. The controller’s machine grant retrieves the named remote path, while a human’s secret-admin role is separate, audited, and not inherited from Argo access.

Without revision-aware alerts, on-call sees “production is down” but cannot tell whether Git, Argo, Kubernetes, or the workload failed. Scrape Argo controller metrics, route low-cardinality team labels, archive controller and Kubernetes audit logs, and notify on state transitions.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: argocd-production
  namespace: monitoring
spec:
  groups:
    - name: argocd.delivery
      rules:
        - alert: ArgoApplicationOutOfSync
          expr: argocd_app_info{project="platform-prod",sync_status!="Synced"} == 1
          for: 15m
          labels:
            severity: warning
            team: platform
          annotations:
            summary: "{{ $labels.name }} remains out of sync"
        - alert: ArgoApplicationDegraded
          expr: argocd_app_info{project="platform-prod",health_status=~"Degraded|Missing"} == 1
          for: 5m
          labels:
            severity: critical
            team: platform
          annotations:
            summary: "{{ $labels.name }} health is {{ $labels.health_status }}"

The official metrics reference defines argocd_app_info and its sync and health labels. Also alert when Argo scrape targets are down, but keep that controller-availability signal separate from application health.

# Application metadata in the ApplicationSet template
annotations:
  notifications.argoproj.io/subscribe.on-sync-failed.slack: platform-prod
  notifications.argoproj.io/subscribe.on-health-degraded.slack: platform-prod
  notifications.argoproj.io/subscribe.on-sync-succeeded.slack: release-audit

Argo Notifications configuration and channel destinations are declarative. Delivery tokens arrive through the external-secret controller, and templates include application, project, target revision, sync status, health status, environment, and a link to the durable incident or release record without including secret values.

An incident response that mutates Argo state may make the dashboard green while Git immediately reapplies the bad revision. Gather read-only evidence first, then choose a reviewed fix-forward or revert so the recovery survives the next reconciliation.

# Read-only evidence: no sync, set, rollback, patch, apply, or delete commands.
date -u +%Y-%m-%dT%H:%M:%SZ
git show --show-signature --stat "$BAD_REVISION"
argocd app get hello-gitops-prod -o json > application-observation.json
argocd app history hello-gitops-prod > application-history.txt
argocd app diff hello-gitops-prod > desired-live.diff || true
kubectl -n hello-prod get deployment,pods,events -o wide
kubectl -n hello-prod get deployment hello-gitops -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Copy the evidence to the incident’s append-only store with its environment, time, actor, command version, and checksums. Avoid storing raw Kubernetes Secrets, environment variables, tokens, or unredacted controller configuration.

# Recovery changes Git; branch protection still requires the emergency approvers.
git switch --create "incident/${INCIDENT_ID}-revert" origin/main
git revert --no-edit "$BAD_PROMOTION_COMMIT"
git show --stat --oneline HEAD
git push --set-upstream origin HEAD
gh pr create --title "revert(prod): ${INCIDENT_ID} restore known-good digest" \
  --body "Reverts ${BAD_PROMOTION_COMMIT}; evidence: ${INCIDENT_EVIDENCE_URL}"

Do not invoke UI history rollback, CLI rollback or sync mutations, or a temporary manual image edit. When the PR merges, automatic reconciliation applies the new revert commit; closure requires Synced, Healthy, expected digest, customer probe, alert resolution, and preserved evidence.

A backup that contains only Argo’s Kubernetes objects cannot rebuild a deleted cluster if Git, artifacts, OpenTofu state, or secret-service recovery material are missing. Define recovery point objective (RPO) and recovery time objective (RTO), then test the whole dependency set on a clean replacement environment.

Back up or replicate these independently owned assets:

1. Protected desired-state and infrastructure Git repositories, signed tags, branch rules, and CODEOWNERS in a second administrative boundary.
2. Versioned and encrypted OpenTofu state, state locks, provider lock files, and short-lived workload-identity configuration.
3. OCI artifacts, signatures, provenance, and vulnerability attestations for every retained deployment digest.
4. External secret values, versions, encryption or SOPS recovery keys, and workload-identity grants through the secret service’s supported backup process.
5. Identity-provider application configuration and group-to-role mappings without copying user passwords.
6. Incident, audit, Argo controller, notification, and Kubernetes audit evidence in durable storage.

The official disaster-recovery guide describes Argo export/import for installations with imperative or otherwise non-reconstructable state. This course deliberately treats such state as migration debt: Git-first reconstruction avoids an argocd admin import mutation and proves that declarations are sufficient.

# Trusted recovery runner; backend and cloud identity arrive from its environment.
git clone --branch "$SIGNED_RECOVERY_TAG" "$INFRA_REPOSITORY" infra-recovery
git -C infra-recovery verify-tag "$SIGNED_RECOVERY_TAG"
tofu -chdir=infra-recovery/infra/cluster init
tofu -chdir=infra-recovery/infra/cluster plan -out=cluster.plan
tofu -chdir=infra-recovery/infra/cluster apply cluster.plan
tofu -chdir=infra-recovery/infra/bootstrap init
tofu -chdir=infra-recovery/infra/bootstrap plan -out=bootstrap.plan
tofu -chdir=infra-recovery/infra/bootstrap apply bootstrap.plan

# Read-only reconstruction proof.
kubectl -n argocd get deployment,statefulset,pods
kubectl -n argocd get applications,argocdprojects,applicationsets
argocd app list -o wide
kubectl -n hello-prod get deployment hello-gitops -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

The restore is complete only when application revisions and digests match the recovery tag, external-secret readiness is proven without printing values, alerts and notifications are tested, and measured RTO/RPO are attached to the exercise. A green Argo pod alone is not recovery evidence.

An unpinned controller upgrade can change custom-resource definitions or reconciliation behaviour across every application at once. Upgrade one reviewed version step at a time, preserve a tested recovery set, and make the trusted bootstrap pipeline—not an operator terminal—the only writer.

The course starts from Helm chart 9.5.17; keep that exact pin and provider lock file until an automated dependency PR proposes a reviewed target. Before merging, read every intervening minor-version note in the official upgrade overview and the chart changelog; major upgrades require the documented backup and compatibility work.

# infra/bootstrap/versions.auto.tfvars: the PR changes one auditable value.
argocd_chart_version = "9.5.17"

Use this upgrade gate:

1. Create and verify a complete recovery set; record current chart, Argo, Kubernetes, CRD, Git, and digest versions.
2. Render the Helm diff and OpenTofu plan in CI; validate CRD changes, removed flags, RBAC, SSO, repository credentials, and ApplicationSet behaviour.
3. Apply the reviewed pin to a disposable recovery cluster, then development; exercise login, generation, sync, self-heal, revert, notification, backup, and reconstruction.
4. Observe at least one normal reconciliation window and compare error rate, queue latency, resource use, sync duration, and controller restarts.
5. Approve the production bootstrap run using short-lived identity; hold the prior recovery set and artifact until the rollback window expires.
6. If the controller regresses, revert the version-pin commit and let the trusted bootstrap pipeline apply the reviewed plan; never hand-edit the Helm release.

Deleting an Argo Application in the UI can remove the control record without proving whether workloads, retained volumes, secret values, DNS, artifacts, or audit evidence were handled. Retirement is a staged Git change with explicit retention and a verifiable absence result.

1. Open a decommission PR that blocks new promotions, names the data owner, exports any required generated-application data, and records retention decisions.
2. Remove traffic declaratively and observe zero expected traffic for the agreed drain period.
3. Rotate or revoke workload identity and remote secret access at their owning systems; do not print or copy secret values.
4. Remove the ApplicationSet generator element in a reviewed PR. The declared Application finalizer and prune policy remove managed runtime resources; protected retained resources need their own approved lifecycle.
5. Verify through read-only queries that the Application and intended resources are absent, retained artifacts remain protected for the rollback window, and alerts do not mistake intentional absence for an outage.
6. Expire Git, artifact, secret, backup, and incident records only under their separate retention policies; emit a deletion tombstone or audit event for each owner.

GitOps degrades into clickops unless normal releases, emergencies, access changes, recovery, upgrades, and deletion all use the same declared path. Use this checklist in release reviews and quarterly recovery exercises.

• [ ] Development and production refer to an immutable sha256 digest, and promotion proves they are the same artifact.
• [ ] Production changes arrive only through protected pull requests with environment-specific ownership.
• [ ] CI passes tofu fmt, tofu validate, kustomize build, kubeconform, conftest, and a redacted secret scan.
• [ ] CI has no production kubeconfig or Argo mutation token; Argo pulls reviewed Git.
• [ ] Automated sync has prune, self-heal, retry, and allowEmpty: false set intentionally.
• [ ] Rollback is a Git revert or corrective commit, never UI history rollback while auto-sync is enabled.
• [ ] A clean-room restore proves expected ApplicationSet output, revisions, digests, secret readiness, alerts, RTO, and RPO.
• [ ] Argo chart and providers stay pinned; upgrades traverse documented compatibility steps and retain a tested recovery set.