Drawing the Boundary: The Hard Type-Split with Terraform
The whole model rests on one line: which cloud resources the in-cluster controller may own, and which stay with Terraform. Draw that line loosely and the two systems fight over the same objects; draw it as an unambiguous, enforceable test and they coexist cleanly forever. This day turns "foundation vs app-scoped" into a single-axis rule, works the genuinely hard case (a grant that crosses the line), and shows how to keep the boundary honest as the platform grows.
Why the boundary is load-bearing
Terraform and an in-cluster controller both reconcile continuously. If a single cloud object is ever owned by both, each will revert the other's changes in an endless loop — a reconcile war with no winner. So the boundary is not documentation; it is the invariant that keeps the system stable. The safe configuration is two disjoint domains, no cloud object owned by both.
Rule of thumb: the type-split is not a style preference, it is the precondition that prevents two reconcilers from co-managing one object — design it as an invariant you can enforce, not a convention you hope people follow.
The test: one axis, lifecycle plus sharing
The classification collapses to a single question about lifecycle and sharing. Everything else is noise.
Owned by one app, created and destroyed with it, referenced only by it → app-scoped (controller). Shared across apps, part of the network/security/account perimeter, or longer-lived than any single app → foundation (Terraform).
| Resource | Owner | Why |
|---|---|---|
| An app's own bucket / service account / IAM role / cache | Controller | Single-app lifecycle |
| VPC, subnets, project / account | Terraform | Perimeter, shared |
| Shared KMS key, org / base IAM, DNS zone | Terraform | Multi-tenant, security |
| A data store used by many apps | Terraform | Shared lifecycle |
The test is deliberately blunt so it produces the same answer for everyone. When a resource is genuinely ambiguous, there is a default: treat it as foundation and raise it for an explicit decision. The self-service layer must never silently claim something that might belong to the perimeter. Rule of thumb: encode the boundary as one testable axis with a safe default (foundation) — a rule that needs a judgement call every time is a rule that will drift.
The hard case: cross-boundary grants
The messy case is not classification — it is a grant that crosses the line. An app-scoped identity (controller-owned) frequently needs access to a foundation resource: permission to decrypt with a shared KMS key, or to read a shared bucket. The dangerous shortcut is to let the controller write a policy attachment onto the foundation object. Now two systems manage one object again, and the boundary has leaked.
The convention that keeps the split clean: the controller owns the app-side principal; Terraform owns any grant on a foundation resource. The app requests access through its claim; the foundation plane issues the grant on its own object. The controller never writes policy onto something Terraform owns.
Rule of thumb: when access must cross the boundary, split the two halves — the requesting principal belongs to the self-service plane, the grant on the shared resource belongs to the foundation plane — so that no single cloud object ever has two owners.
Greenfield makes the pilot safe
There is a reason to start app-scoped and greenfield: if you never migrate an existing Terraform-managed resource, there are no shared objects at all during the pilot, so boundary leakage cannot physically happen yet. New app-scoped resources are created fresh under the controller; everything Terraform holds today stays exactly where it is. Migration of existing resources is explicitly out of scope — you don't move a service's current role out of Terraform state into the controller, you mint a new per-service identity and bind the workload to it.
| Decision | Choice | Effect |
|---|---|---|
| Existing TF-managed resources | Leave in Terraform | No state surgery, no dual ownership |
| New app-scoped resources | Controller (greenfield) | Born under the new model, clean lifecycle |
| Foundation (VPC, DNS, base IAM, shared keys) | Terraform, untouched | Perimeter unchanged |
Rule of thumb: greenfield-first is a safety property, not just a scoping convenience — with nothing migrated, the worst failure mode of the boundary (two owners on one object) is impossible during the riskiest, earliest phase.
Keeping the boundary honest over time
A clean split on day one erodes unless something enforces it, because every tempting shortcut involves the self-service layer reaching across the line. The defence is layered, each layer independent so a gap in one is caught by the next.
| Layer | Enforces |
|---|---|
| Claim schema validation (CEL on the XRD) | Requested capabilities ⊆ allowlist; owner required; obvious escalations rejected |
| Composition template | Only known capabilities expand; provider/account pinned, not claim-chosen |
| Admission policy (Kyverno) | Reject raw resources that bypass the claim path; block perimeter-owned kinds |
| Cloud-side permission boundary | A hard ceiling the minted identities can never exceed, whatever the claim says |
Eventually, "which side owns this?" should be answerable from an ownership registry rather than tribal memory — a machine-checkable list that the pre-production hardening turns from convention into a gate. The registry is the definition the type-split test compiles into. Rule of thumb: a boundary defended by one control is one bug from leaking; layer independent checks (schema, template, admission, cloud ceiling) so that no single failure lets the self-service plane claim or grant across the line.
Key takeaways
- Two continuously-reconciling systems make disjoint object ownership the only stable configuration; the type-split is that invariant.
- The classification is one axis — lifecycle plus sharing: single-app-and-dies-with-it → controller; shared/perimeter/long-lived → Terraform; ambiguous → default to foundation and raise it.
- The hard case is the cross-boundary grant: the controller owns the app-side principal, Terraform issues the grant on its own foundation object — never the reverse.
- Greenfield-first makes boundary leakage impossible during the pilot because nothing is migrated and no object has two owners.
- Keep the boundary honest with layered enforcement (schema, composition, admission, cloud permission boundary) and, eventually, a machine-checkable ownership registry.
Checklist
- [ ] I can explain why overlapping ownership causes a reconcile war and why disjoint ownership is the only stable state.
- [ ] I can apply the lifecycle-plus-sharing test to a new resource and say what to do when the answer is ambiguous.
- [ ] I can describe a cross-boundary grant and split it correctly: app-side principal to the controller, grant-on-foundation to Terraform.
- [ ] I can explain why greenfield-first removes the worst boundary failure mode during the pilot.
- [ ] I can list the layered enforcement controls and explain why one control is not enough to keep the boundary honest.