Drawing the Boundary: The Hard Type-Split with Terraform
The whole model rests on one line: which cloud resources the in-cluster controller may own, and which stay with Terraform. Draw that line loosely and the two systems fight over the same objects; draw it as an unambiguous, enforceable test and they coexist cleanly forever. This day turns "foundation vs app-scoped" into a single-axis rule, works the genuinely hard case (a grant that crosses the line), and shows how to keep the boundary honest as the platform grows.
Customer outcome and implementation focus
An app team can create only a resource whose lifecycle belongs to that app; a platform team retains network, cluster, and shared IAM foundations. The implementation enforces one owner per cloud object, so controllers cannot reconcile each other’s changes forever.
| Story ID | User story | Observable acceptance |
|---|---|---|
| D03-US-01 | As a platform owner, I want every cloud object assigned to one reconciler, so that drift has one repair authority. | Inventory labels identify Terraform or Crossplane, never both. |
| D03-US-02 | As an app team, I want a bounded grant to a shared foundation, so that I can use it without owning it. | The app role is created while the shared network/resource policy remains Terraform-owned. |
Components in focus
The type split is a data-ownership rule. Terraform state is the foundation record; etcd is the claim/controller record; provider state is the real cloud record. No cache can arbitrate ownership.
| Boundary | Controller and compute | Authoritative record | Failure evidence |
|---|---|---|---|
| Foundation network/base IAM | Terraform runner and provider API | Terraform state backend | Plan detects an unexpected object or owner label. |
| App role/bucket | Crossplane provider pod | Claim and managed-resource status in etcd | Provider ReconcileError with cloud operation ID. |
| Shared-resource grant | Terraform policy plus generated role reference | Terraform state and provider policy | Admission rejects a claim trying to mutate foundation policy. |
Enforce ownership as a machine-checkable label
Documentation cannot stop a reconcile war. Label every managed object and reject a Crossplane claim that requests a foundation type; the admission controller interprets this before any provider API call.
metadata:
labels:
platform.example.com/owner: crossplane
platform.example.com/scope: app
spec:
parameters: { resourceClass: app-bucket }
Declared intent: mark this resource as app-scoped and Crossplane-owned. Interpreter: admission policy and composition controller. Software effect: an invalid foundation class is denied before managed-resource creation. Hardware effect: only controller/API-server CPU is used on denial; successful reconciliation later allocates cloud capacity. Evidence: policy denial log for foundation-network and a successful inventory query for the labelled bucket.
Why the boundary is load-bearing
Terraform and an in-cluster controller both reconcile continuously. If a single cloud object is ever owned by both, each will revert the other's changes in an endless loop — a reconcile war with no winner. So the boundary is not documentation; it is the invariant that keeps the system stable. The safe configuration is two disjoint domains, no cloud object owned by both.
Rule of thumb: the type-split is not a style preference, it is the precondition that prevents two reconcilers from co-managing one object — design it as an invariant you can enforce, not a convention you hope people follow.
The test: one axis, lifecycle plus sharing
The classification collapses to a single question about lifecycle and sharing. Everything else is noise.
Owned by one app, created and destroyed with it, referenced only by it → app-scoped (controller). Shared across apps, part of the network/security/account perimeter, or longer-lived than any single app → foundation (Terraform).
| Resource | Owner | Why |
|---|---|---|
| An app's own bucket / service account / IAM role / cache | Controller | Single-app lifecycle |
| VPC, subnets, project / account | Terraform | Perimeter, shared |
| Shared KMS key, org / base IAM, DNS zone | Terraform | Multi-tenant, security |
| A data store used by many apps | Terraform | Shared lifecycle |
The test is deliberately blunt so it produces the same answer for everyone. When a resource is genuinely ambiguous, there is a default: treat it as foundation and raise it for an explicit decision. The self-service layer must never silently claim something that might belong to the perimeter. Rule of thumb: encode the boundary as one testable axis with a safe default (foundation) — a rule that needs a judgement call every time is a rule that will drift.
The hard case: cross-boundary grants
The messy case is not classification — it is a grant that crosses the line. An app-scoped identity (controller-owned) frequently needs access to a foundation resource: permission to decrypt with a shared KMS key, or to read a shared bucket. The dangerous shortcut is to let the controller write a policy attachment onto the foundation object. Now two systems manage one object again, and the boundary has leaked.
The convention that keeps the split clean: the controller owns the app-side principal; Terraform owns any grant on a foundation resource. The app requests access through its claim; the foundation plane issues the grant on its own object. The controller never writes policy onto something Terraform owns.
Rule of thumb: when access must cross the boundary, split the two halves — the requesting principal belongs to the self-service plane, the grant on the shared resource belongs to the foundation plane — so that no single cloud object ever has two owners.
Greenfield makes the pilot safe
There is a reason to start app-scoped and greenfield: if you never migrate an existing Terraform-managed resource, there are no shared objects at all during the pilot, so boundary leakage cannot physically happen yet. New app-scoped resources are created fresh under the controller; everything Terraform holds today stays exactly where it is. Migration of existing resources is explicitly out of scope — you don't move a service's current role out of Terraform state into the controller, you mint a new per-service identity and bind the workload to it.
| Decision | Choice | Effect |
|---|---|---|
| Existing TF-managed resources | Leave in Terraform | No state surgery, no dual ownership |
| New app-scoped resources | Controller (greenfield) | Born under the new model, clean lifecycle |
| Foundation (VPC, DNS, base IAM, shared keys) | Terraform, untouched | Perimeter unchanged |
Rule of thumb: greenfield-first is a safety property, not just a scoping convenience — with nothing migrated, the worst failure mode of the boundary (two owners on one object) is impossible during the riskiest, earliest phase.
Keeping the boundary honest over time
A clean split on day one erodes unless something enforces it, because every tempting shortcut involves the self-service layer reaching across the line. The defence is layered, each layer independent so a gap in one is caught by the next.
| Layer | Enforces |
|---|---|
| Claim schema validation (CEL on the XRD) | Requested capabilities ⊆ allowlist; owner required; obvious escalations rejected |
| Composition template | Only known capabilities expand; provider/account pinned, not claim-chosen |
| Admission policy (Kyverno) | Reject raw resources that bypass the claim path; block perimeter-owned kinds |
| Cloud-side permission boundary | A hard ceiling the minted identities can never exceed, whatever the claim says |
Eventually, "which side owns this?" should be answerable from an ownership registry rather than tribal memory — a machine-checkable list that the pre-production hardening turns from convention into a gate. The registry is the definition the type-split test compiles into. Rule of thumb: a boundary defended by one control is one bug from leaking; layer independent checks (schema, template, admission, cloud ceiling) so that no single failure lets the self-service plane claim or grant across the line.
Before and after, side by side
When Terraform and Crossplane can both reconcile the same cloud object, each controller can undo the other's work and neither state record remains trustworthy. The target state assigns foundation resources to Terraform and single-service resources to Crossplane, with explicit grants crossing the boundary without transferring ownership.
On the right, each object has exactly one lifecycle owner and one authoritative state record. The access grant is intentionally visible as a relationship between the planes: it permits use of a shared foundation capability without letting the application claim create, update, or delete that foundation resource.
Key takeaways
- Two continuously-reconciling systems make disjoint object ownership the only stable configuration; the type-split is that invariant.
- The classification is one axis — lifecycle plus sharing: single-app-and-dies-with-it → controller; shared/perimeter/long-lived → Terraform; ambiguous → default to foundation and raise it.
- The hard case is the cross-boundary grant: the controller owns the app-side principal, Terraform issues the grant on its own foundation object — never the reverse.
- Greenfield-first makes boundary leakage impossible during the pilot because nothing is migrated and no object has two owners.
- Keep the boundary honest with layered enforcement (schema, composition, admission, cloud permission boundary) and, eventually, a machine-checkable ownership registry.
Checklist
- [ ] I can explain why overlapping ownership causes a reconcile war and why disjoint ownership is the only stable state.
- [ ] I can apply the lifecycle-plus-sharing test to a new resource and say what to do when the answer is ambiguous.
- [ ] I can describe a cross-boundary grant and split it correctly: app-side principal to the controller, grant-on-foundation to Terraform.
- [ ] I can explain why greenfield-first removes the worst boundary failure mode during the pilot.
- [ ] I can list the layered enforcement controls and explain why one control is not enough to keep the boundary honest.