Rolling Out: Pilot → Cutover → Prod Without a Big Bang
A system this central can't arrive all at once. This final day is the rollout: a three-phase plan that separates *standing the platform up* from *cutting services over* from *going to production*, the accepted risks you carry into each phase and how you fence them, and the definition of done that says when it's actually finished — a quantified velocity win, not just "it runs."
The phase model, and why the seam is where it is
The rollout's key decision is where to put the seam. It is drawn so that Phase 1 changes no service's behaviour at all — the entire platform (controller, identity model, the first claim type, health, observability, guardrails, and a disabled service-chart helper) lands and is validated before a single workload changes how it gets its identity. Cutover starts only in Phase 2. New resource types wait for Phase 3.
| Phase | Goal | Explicitly not yet |
|---|---|---|
| 1 — Platform ready | Controller + XRole claim live and validated on the pilot pair; health, obs, guardrails, disabled helper | No service cut over |
| 2 — Cutover (non-prod) | Real services flip to per-service identity — dev first, then test | No production change |
| 3 — Prod (GA) + widen | Progressive prod by cluster class; add XServiceAccount / XBucket | — |
Rule of thumb: put the rollout seam between "platform ready" and "in use" — proving the whole path on synthetic claims before any real cutover means the first service to move is exercising a system already shown to work and to clean up after itself.
Cutover is a reversible, observed flip
Phase 2 doesn't rewrite anything risky; it flips an enable flag on a service-chart helper that was shipped disabled in Phase 1. The helper templates the claim and the ServiceAccount annotation from the same inputs (no status read-back), in an earlier sync-wave than the Deployment. The cutover per service is: mint the new identity → flip the KSA annotation → observe under soak → revoke the old shared grant. Each step is reversible until the last, and the old shared identity stays in place until the new one is proven.
Rule of thumb: make cutover a sequence of reversible steps with the old path live until the new one is proven — mint-then-observe-then-revoke, never mint-and-revoke-at-once, so any surprise is a rollback, not an incident.
Accepted risks and how they're fenced
Not every risk is closed before production; some are accepted with an explicit fence, which is different from being ignored. Naming them is part of the plan.
| Risk | Disposition | Fence |
|---|---|---|
| Claim delete → cloud delete (IAM) | Verified safe | Delete leaves no orphan; stateful types retain |
| Composite false-green in GitOps | Handled | Custom health rule for the composite group |
| Cross-cluster identity impersonation in a shared non-prod project | Accepted (non-prod only) | Prod is single-cluster-per-project, so the vector is absent; hardening tracked |
| Broad controller minting role | Accepted, fenced in-cluster | The claim-side allowlist (schema render + admission) is the fence, not the cloud role |
| GitOps tracking-label collision | Verified non-issue | Custom instance label means the documented collision doesn't apply |
The pattern worth internalising: a risk you can't cheaply close at the cloud layer (a broad controller role) is fenced at the claim layer instead (allowlist at render and admission), and a risk that only exists in a shared non-prod topology is accepted because production doesn't share that topology. Rule of thumb: distinguish closed, handled, and accepted-with-a-fence risks explicitly — an accepted risk with a named fence and a condition under which it disappears is a decision; an unnamed one is a surprise.
The prod gate and the definition of done
Production is not gated by the dev-pilot bar; it has a higher one. Before Phase 3, the pre-production hardening must be in place: the permission boundary on the minting role, the admission allowlist promoted to Enforce, and the auditable type-split. Then prod rolls by cluster class (multi-tenant, then single-tenant per cloud) rather than all at once.
And "done" is defined in outcomes, not activity:
Done = provisioning of app-scoped resources live in production on both clouds, app teams self-serving end-to-end through GitOps, hardening in place, and a quantified velocity win — turnaround time saved versus the old platform-PR path — across N self-serving teams.
Rule of thumb: write the definition of done as a measured outcome (teams self-serving, turnaround time saved) not a checklist of shipped components — a platform that "runs" but nobody self-serves on hasn't finished the job it was built for.
The whole arc, in one view
Every day fed this: the problem (Day 01), the tool decision (Day 02), the boundary (Day 03), the platform (Day 04), the claim API and profiles (Days 05–06), the safety systems (Day 07), and the rollout (today). The throughline is a single discipline — replace a ticket and a shared credential with a reviewed, allowlisted, self-service claim, and move in reversible steps that always leave the old path standing until the new one is proven.
Key takeaways
- The rollout seam sits between platform ready and in use: Phase 1 stands everything up with zero cutover; Phase 2 cuts services over in non-prod; Phase 3 goes to prod by cluster class and widens resource types.
- Cutover is reversible: flip a pre-shipped (disabled) helper, mint → observe under soak → revoke the old grant last, old identity live until the new one is proven.
- Risks are closed, handled, or accepted-with-a-fence, each named — a broad controller role fenced at the claim allowlist; a non-prod-only impersonation vector accepted because prod doesn't share the topology.
- Production has a higher gate than the pilot (boundary + admission-enforce + auditable type-split) and rolls by cluster class.
- Done is a measured outcome — teams self-serving end-to-end and a quantified turnaround-time win — not a list of shipped components.
Checklist
- [ ] I can explain why the rollout seam is placed between platform-ready and cutover, and what Phase 1 deliberately excludes.
- [ ] I can describe the reversible cutover sequence and why the old shared grant is revoked last.
- [ ] I can distinguish closed, handled, and accepted-with-a-fence risks and give an example of each.
- [ ] I can state what the production gate adds over the pilot bar and why prod rolls by cluster class.
- [ ] I can state a definition of done written as a measured outcome rather than a checklist.