Standing Up Crossplane v2: Controller, Providers, and Identity

The platform installs as a small stack of charts, wired into the GitOps app-of-apps and enabled per cluster. Order matters: the core CRDs and controller must exist before providers, providers before their configs, configs before any claim. Crossplane expresses this with sync-waves so the GitOps engine applies them in sequence rather than all at once.

Rule of thumb: an in-cluster provisioner is a dependency chain, not a bag of manifests — encode the order (core → providers → configs → claims) as sync-waves so a cold install converges deterministically instead of racing.

A practical warning that saves days: use Crossplane v2 GA, and do not resurrect an older v1-era spike if one exists in the repo. v2's APIs (namespaced composite resources, the current Composition pipeline model) differ enough that building on dead v1 prior-art costs more than starting fresh. Treat any abandoned crossplane-providers-config from a prior experiment as archaeology, not a foundation — and retire it explicitly (including any static cloud keys it left behind) as part of standing up the new install.

Rule of thumb: when a major version changed the API model, an old internal spike is a liability, not a head start — read it for lessons, then delete it so nobody wires the new system to the dead one.

The single hardest prerequisite is the identity the Crossplane controller assumes to mint cloud resources. Until this exists and each cloud trusts it, providers cannot authenticate and nothing reconciles. It is provisioned in Terraform (it is foundation — a controller identity is shared infrastructure, not app-scoped), one per cluster, and wired into the provider configs.

Two details bite in practice. First, naming must include the cluster: GCP service-account IDs are project-global and AWS role names are account-global, so when several clusters share one project or account, an identity named only by environment collides. Every controller (and later every minted) identity carries the cluster in its name. Second, on GCP the controller SA usually needs an explicit exception in any org-level "deny IAM policy changes" guardrail, or its setIamPolicy calls are refused with a 403.

Rule of thumb: stand up the minting identity before anything else and treat it as foundation — providers that can't authenticate produce the most confusing failure mode in the whole system (everything renders, nothing appears in the cloud).

With identity in place, the provider packages install and their configs bind them to it. Each provider config hardcodes which identity to use — the claim never chooses its provider or credentials. That single decision (pin the provider config in the platform layer, not the claim) closes a confused-deputy hole where a crafted claim could point itself at an over-privileged identity. Providers reaching Healthy and provider configs authenticating is the gate for enabling any composition.

Rule of thumb: the provider config is a platform secret, not a claim input — pin it in the chart so a claim can express what it needs but never which credential mints it.

The platform goes live first on exactly two clusters — one GKE, one EKS — and provisions exactly one resource type (an IAM role claim), with no real service cut over yet. "Platform ready" is a distinct milestone from "services using it": the controller, providers, the first claim type, GitOps health, observability, and guardrails all land and are validated before a single production workload changes how it gets its identity.

Validation at this stage is deliberately offline-then-live: a helm template render proves the composition emits the right resources with the right deletion policy and tags (a negative control confirms the guardrails would reject a bad claim), then a live claim proves mint → bind → ready → delete-with-no-orphan on both clouds. Rule of thumb: separate "platform ready" from "in use" as explicit milestones — prove the whole path on synthetic claims first, so the first real service to cut over is exercising a system that has already been shown to work and to clean up after itself.

• The platform is four ordered charts — core, providers, provider configs, compositions — wired into GitOps and sequenced with sync-waves.
• Build on Crossplane v2 GA; treat any old v1 spike as archaeology and retire it (and its static credentials) rather than extending it.
• The controller identity (AWS IRSA role / GCP Workload Identity SA, one per cluster, provisioned in Terraform) gates everything; without it providers can't authenticate and nothing reconciles.
• Identity names must include the cluster (project/account-global namespaces collide otherwise), and GCP controllers often need an explicit deny-policy exception to call setIamPolicy.
• Pin the provider config in the platform layer so a claim never chooses its credential; reach providers Healthy before enabling compositions.
• Ship to a two-cluster pilot with zero service cutover, validating offline-render then live claim (mint → bind → ready → delete-no-orphan) before anyone depends on it.

• [ ] I can name the four charts and explain why sync-wave ordering is required for a cold install to converge.
• [ ] I can explain why to build on v2 and retire an old v1 spike rather than extend it.
• [ ] I can describe the controller identity on each cloud and why it is the prerequisite that gates all provisioning.
• [ ] I can explain why identity names must include the cluster and what the GCP deny-policy exception is for.
• [ ] I can justify pinning the provider config in the chart rather than accepting it from the claim.
• [ ] I can distinguish "platform ready" from "in use" and describe the offline-then-live validation for the pilot.