Choosing the Control Plane: Crossplane vs KCC vs ACK

It is tempting to choose the "most production-ready" tool, but every one of these is GA and widely run. A generic "production-ready" badge launders over the fit questions that actually decide the outcome: how each behaves under your GitOps setup, your cloud spread, and your ownership boundary. So the useful comparison re-tests each claim against the target stack — two clouds (a GKE side and an EKS side), a GitOps app-of-apps, and a hard split where no cloud object is owned by both Terraform and the controller.

Rule of thumb: never let "X is production-ready" end a tool decision — re-test the specific claims that matter (deletion, drift, identity, GitOps health, topology) against your own stack, because generic maturity and fit-for-you are different questions.

This is the highest-stakes dimension, because all three tools delete the live cloud resource by default when the custom resource is deleted. Under a GitOps app-of-apps, an accidental prune, a mis-scoped Application, or a stray kubectl delete can cascade into real cloud deletion. What differs is where you can set a safe default.

For an app-scoped model the default state is equally dangerous on all three; the mitigation differs only in where a blanket safe default can live. Whichever you choose, you pair it with GitOps Prune=false plus finalizers so only an explicit claim delete ever removes anything. Rule of thumb: treat default-delete as the first thing you neutralise — set retain where you can set it broadest, and never rely on the tool default alone.

All three run a reconcile loop that continuously corrects drift. That is a feature — until a Terraform-owned object and a controller-owned object are ever the same object, at which point the two systems enter a reconcile war, each reverting the other forever. This is why the hard type-split from Day 01 is not a nice-to-have but a precondition: keep the domains disjoint and the fight cannot start. (Under a greenfield pilot with no migration, it structurally cannot occur — there are no shared objects yet.)

Rule of thumb: overlapping ownership between an IaC tool and an in-cluster controller is not a bug to fix later, it is a state to make impossible by construction — disjoint object ownership is the only stable configuration.

Here the tools diverge structurally. Crossplane runs one control plane spanning both clouds (with per-cloud provider packages underneath). KCC is one controller for all of GCP — but GKE-only. ACK is one controller per AWS service: IAM and object storage are separate deployments, charts, and identities, so the AWS side is "1 + N", not one.

Rule of thumb: count the operating surfaces a choice creates, not just the features — one control plane is fewer things to run but a bigger single blast radius; per-cloud, per-service controllers distribute risk at the cost of more moving parts.

The controller holds the credential that mints cloud IAM, so the controller's own identity is the real blast radius — more than any resource it manages. KCC's namespaced mode is strongest here (a per-namespace identity, scopable with IAM Conditions). ACK scopes per service controller, but its IAM controller is the dangerous one — without a permission boundary it can mint roles more privileged than itself. Crossplane is weakest by default (one identity per provider), so scoping leans on multiple provider configs plus an AWS permission boundary.

Rule of thumb: evaluate the minting identity before the minted resources — a self-service IAM system is only as safe as the ceiling on the controller that issues the grants, and that ceiling (permission boundary, IAM Conditions, allowlist) is mandatory, not optional.

None of the three sync cleanly out of the box; each needs custom health and diff handling or it reports false-green on cloud failures. Crossplane's managed resources may need custom health wiring (and, depending on version, a tracking-method change). KCC and ACK both write state back into the spec, creating a permanent diff that needs per-type ignoreDifferences. All three benefit from custom health logic so a failed cloud operation shows red, not green.

Rule of thumb: assume any in-cluster provisioner is silently healthy until proven otherwise — budget for custom health and diff suppression as part of adoption, and test the failure path (does a broken cloud op actually show unhealthy?) before you trust the dashboards.

Now the six dimensions resolve into a choice, and it hinges on one premise. If you are not building a unified, portable claim API, a hard type-split gains little from a single control plane, and per-cloud tools fit better — KCC's namespaced identity is the cleanest app-scoped fit on GCP, and cloud-isolated controllers keep blast radius per-cloud. But if you are building Compositions — one platform-authored claim surface that app teams use identically on both clouds — then Crossplane's abstraction layer is the product, and neither KCC nor ACK can offer a single claim API across both clouds.

The target here is explicitly the second: a single claim (XRole, XBucket) that behaves the same for every team regardless of cloud. So the choice is Crossplane v2 — chosen because of the Composition goal, not because it out-matures the others. (Note the subtle point: an AWS grant is a policy statement and a GCP grant is a role binding; they are not portable, so the Composition standardises the claim, not the cloud semantics — more on that on Day 05.) Rule of thumb: the right in-cluster provisioner is a function of whether you're building a shared abstraction — decide that first; the tool falls out of it.

• All three tools are GA; the decision is about fit to your stack and goal, not maturity — re-test each production-ready claim against your clouds, GitOps, and ownership boundary.
• Deletion defaults to destroy on every tool; differences are only in where a safe retain default can live. Pair with Prune=false + finalizers regardless.
• Continuous drift correction makes a disjoint type-split with Terraform a precondition — overlap causes an unwinnable reconcile war.
• Topology differs structurally: Crossplane is one control plane (concentrated blast radius); KCC/ACK are per-cloud and, for ACK, per-service (1+N surfaces).
• The controller's own identity is the real blast radius; a permission boundary / allowlist on the minting identity is mandatory.
• The decision is premise-dependent: build a unified cross-cloud claim API → Crossplane; don't → per-cloud KCC+ACK fit better. Here the goal is the unified API, so Crossplane v2.

• [ ] I can explain why "production-ready" is the wrong axis and name the six dimensions that actually decide the tool.
• [ ] I can state the default deletion behaviour of all three tools and the mitigation that does not depend on the tool default.
• [ ] I can explain why continuous drift correction makes a disjoint Terraform boundary a precondition, not a nicety.
• [ ] I can contrast one-control-plane vs 1+N topology and say what each does to blast radius and operational load.
• [ ] I can state the premise (unified claim API or not) that flips the decision, and justify choosing Crossplane under that premise.