Making It Safe in GitOps: Health, Deletion, and Admission

GitOps engines infer resource health from status conditions. Provider-managed resources usually have bundled health rules that read those conditions correctly — a failed cloud operation surfaces as Degraded. But the composite claim type is a custom API group the engine doesn't recognise, so by default it is reported Healthy the moment it exists, regardless of whether the underlying cloud resources actually came up. That is false-green, and it is worse than a plain failure because it hides one.

The fix is a single custom health rule for the composite group that mirrors the bundled provider logic: not-yet-ready composites report Progressing or Degraded, never Healthy. Why this matters beyond tidy dashboards: the claim is ordered in an earlier sync-wave than the workload Deployment, precisely so the identity exists before pods try to assume it. If the composite is false-green, the wave advances early, pods start without their role, and they fail with permission errors. Honest health is the backstop that makes the ordering real. Rule of thumb: a custom resource type that reports Healthy-on-existence is a silent outage waiting to happen — add explicit health for every custom group, and test the failure path (does a broken claim actually block the next sync-wave?) rather than trusting the green.

Because the controller reconciles to match the claim, deleting a claim means deleting its resources — which is right for a stateless identity and catastrophic for a stateful store. The policy is therefore type-dependent: stateless resources clean themselves up, stateful ones never delete automatically.

Stateless (IAM role, service account):
    claim deleted → resource DELETED     (clean lifecycle, no orphan identities)

Stateful (bucket, cache, database):
    claim deleted → resource RETAINED    (data-loss guard; delete is manual & explicit)

Two latches make this robust. First, retain by default for anything holding data — an accidental delete must never be able to destroy state. Second, protect even the stateless resources against accidental removal with Prune=false plus finalizers, so only an explicit claim deletion — not a transient sync glitch or a mis-scoped Application prune — ever tears anything down. The verified behaviour to prove at the pilot: delete an identity claim and confirm the cloud role is gone with no orphan left behind; delete a stateful claim and confirm the data survives.

Rule of thumb: decide deletion policy per resource type before you ship — default stateful to retain, make every automatic deletion trace to a deliberate human action, and verify both the no-orphan cleanup and the data-survives cases on real claims.

Schema validation on the claim is necessary but not sufficient, because someone could bypass the claim entirely and apply a raw provider resource, or the controller's broad minting identity could be tricked into an additive grant that escalates. Admission policies are the enforcement layer that sits at the cluster boundary and rejects what shouldn't exist at all. Three policies do the heavy lifting:

These ship co-located with the composition they guard, gated per cluster, and — crucially — are rolled out Audit first, then Enforce, per environment (dev → test → prod), after a soak in audit mode confirms they wouldn't wrongly block legitimate claims. A safety policy that fails open (an Ignore failure policy) is deliberate: a broken policy engine should not take down provisioning, but the ceiling (the cloud permission boundary) still holds underneath. Rule of thumb: enforce at admission what schema validation can't see — raw resources that bypass the claim, and additive grants that escalate — and always promote a new policy Audit → Enforce per environment so you learn what it would block before it blocks it.

The last safety system is visibility. A dashboard for controller and provider health, alerts on reconcile failures and on the controller pod approaching its resource limits, and an operational runbook turn "a team says their role didn't appear" into "we saw the failed reconcile and knew why." The controller is a long-running reconcile loop; its memory footprint tracks the number of managed resources, so a request/limit set to the observed peak with headroom prevents the slow-creep OOM that otherwise arrives months later.

Rule of thumb: ship the dashboard, alerts, and runbook with the platform, not after the first incident — a provisioning system you can't observe is one whose failures you learn about from the teams it blocked.

• False-green is the core GitOps hazard: the custom composite type reports Healthy-on-existence by default; add one custom health rule so a failed claim shows Progressing/Degraded and blocks the next sync-wave.
• Deletion is type-dependent: stateless identities delete cleanly (no orphan) on claim delete; stateful stores retain by default, and Prune=false + finalizers ensure only an explicit delete ever removes anything.
• Admission policies enforce what schema can't: an AWS permissions boundary on every role, GCP additive-only grants, required ownership tags — rolled out Audit → Enforce per environment.
• Safety policies fail open by design (a broken engine mustn't halt provisioning) because the cloud permission boundary is the ceiling underneath.
• Ship observability (dashboard, alerts, runbook) with the platform; size the controller to its observed peak to avoid slow-creep OOM.

• [ ] I can explain false-green health, why it's worse than a plain failure, and how it breaks sync-wave ordering.
• [ ] I can state a type-dependent deletion policy and describe the two latches (retain-by-default, explicit-delete-only).
• [ ] I can name the three admission policies and explain what each catches that schema validation cannot.
• [ ] I can explain why safety policies fail open and what still enforces the ceiling when they do.
• [ ] I can list the observability pieces and explain why controller memory must be sized to observed peak.