Rolling Out: Pilot → Cutover → Prod Without a Big Bang
A system this central can't arrive all at once. This final day is the rollout: a three-phase plan that separates *standing the platform up* from *cutting services over* from *going to production*, the accepted risks you carry into each phase and how you fence them, and the definition of done that says when it's actually finished — a quantified velocity win, not just "it runs."
Customer outcome and implementation focus
The service team gets faster, safer infrastructure changes only after the platform control loop and its failure paths have been exercised. Roll out the same immutable controller/configuration artifacts from pilot to production with explicit environment ownership; do not rebuild a different system per environment.
| Story ID | User story | Observable acceptance |
|---|---|---|
| D08-US-01 | As a platform owner, I want a pilot with no workload cutover, so that controller failure is discovered before customer traffic depends on it. | Pilot claims reconcile and denial drills pass while all services retain old identities. |
| D08-US-02 | As a service owner, I want reversible per-service cutover, so that a bad identity policy can be rolled back without a fleet outage. | Reverting one claim restores the old path and a separate pilot service remains on the new path. |
Components in focus
| Environment layer | Owner and compute | State/storage | Evidence |
|---|---|---|---|
| Local/pilot cluster | Platform team; controller pods | Disposable etcd and test cloud account | Reconciliation and bounded failure drill. |
| Development/staging | Environment owner; separate cluster/account | Separate Git overlay, secrets, and state backend | Promotion revision and trace IDs. |
| Production | Production owner; HA control-plane/provider services | Separate account, cluster, network, backups, cloud IAM | Customer request, cloud audit, cost/resource identity. |
| Application | Service team pods | Application DB/cache remain untouched by identity migration | Allowed and denied service calls. |
Promote one tested artifact through explicit overlays
The rollout configuration changes policy and ownership per environment, not the controller image or claim API. GitOps interprets the overlay and records the exact revision that reached each cluster.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: crossplane-prod }
spec:
source: { path: platform/overlays/prod, targetRevision: 4c0ffee }
syncPolicy: { automated: { prune: false, selfHeal: true } }
Declared intent: promote an immutable Git revision to the production overlay while preventing automatic destructive prune. Interpreter: Argo CD renders the overlay and Crossplane/provider controllers reconcile its claims. Software effect: Kubernetes objects and cloud resources converge toward that revision. Hardware effect: controller pods consume cluster capacity; provider actions allocate cloud resources only after policy permits. Evidence: Argo revision, Crossplane readiness, cloud audit IDs, a successful customer request, a denied out-of-scope request, and a tested rollback revision.
The phase model, and why the seam is where it is
The rollout's key decision is where to put the seam. It is drawn so that Phase 1 changes no service's behaviour at all — the entire platform (controller, identity model, the first claim type, health, observability, guardrails, and a disabled service-chart helper) lands and is validated before a single workload changes how it gets its identity. Cutover starts only in Phase 2. New resource types wait for Phase 3.
| Phase | Goal | Explicitly not yet |
|---|---|---|
| 1 — Platform ready | Controller + XRole claim live and validated on the pilot pair; health, obs, guardrails, disabled helper | No service cut over |
| 2 — Cutover (non-prod) | Real services flip to per-service identity — dev first, then test | No production change |
| 3 — Prod (GA) + widen | Progressive prod by cluster class; add XServiceAccount / XBucket | — |
Rule of thumb: put the rollout seam between "platform ready" and "in use" — proving the whole path on synthetic claims before any real cutover means the first service to move is exercising a system already shown to work and to clean up after itself.
Cutover is a reversible, observed flip
Phase 2 doesn't rewrite anything risky; it flips an enable flag on a service-chart helper that was shipped disabled in Phase 1. The helper templates the claim and the ServiceAccount annotation from the same inputs (no status read-back), in an earlier sync-wave than the Deployment. The cutover per service is: mint the new identity → flip the KSA annotation → observe under soak → revoke the old shared grant. Each step is reversible until the last, and the old shared identity stays in place until the new one is proven.
Rule of thumb: make cutover a sequence of reversible steps with the old path live until the new one is proven — mint-then-observe-then-revoke, never mint-and-revoke-at-once, so any surprise is a rollback, not an incident.
Accepted risks and how they're fenced
Not every risk is closed before production; some are accepted with an explicit fence, which is different from being ignored. Naming them is part of the plan.
| Risk | Disposition | Fence |
|---|---|---|
| Claim delete → cloud delete (IAM) | Verified safe | Delete leaves no orphan; stateful types retain |
| Composite false-green in GitOps | Handled | Custom health rule for the composite group |
| Cross-cluster identity impersonation in a shared non-prod project | Accepted (non-prod only) | Prod is single-cluster-per-project, so the vector is absent; hardening tracked |
| Broad controller minting role | Accepted, fenced in-cluster | The claim-side allowlist (schema render + admission) is the fence, not the cloud role |
| GitOps tracking-label collision | Verified non-issue | Custom instance label means the documented collision doesn't apply |
The pattern worth internalising: a risk you can't cheaply close at the cloud layer (a broad controller role) is fenced at the claim layer instead (allowlist at render and admission), and a risk that only exists in a shared non-prod topology is accepted because production doesn't share that topology. Rule of thumb: distinguish closed, handled, and accepted-with-a-fence risks explicitly — an accepted risk with a named fence and a condition under which it disappears is a decision; an unnamed one is a surprise.
The prod gate and the definition of done
Production is not gated by the dev-pilot bar; it has a higher one. Before Phase 3, the pre-production hardening must be in place: the permission boundary on the minting role, the admission allowlist promoted to Enforce, and the auditable type-split. Then prod rolls by cluster class (multi-tenant, then single-tenant per cloud) rather than all at once.
And "done" is defined in outcomes, not activity:
Done = provisioning of app-scoped resources live in production on both clouds, app teams self-serving end-to-end through GitOps, hardening in place, and a quantified velocity win — turnaround time saved versus the old platform-PR path — across N self-serving teams.
Rule of thumb: write the definition of done as a measured outcome (teams self-serving, turnaround time saved) not a checklist of shipped components — a platform that "runs" but nobody self-serves on hasn't finished the job it was built for.
The whole arc, in one view
Every day fed this: the problem (Day 01), the tool decision (Day 02), the boundary (Day 03), the platform (Day 04), the claim API and profiles (Days 05–06), the safety systems (Day 07), and the rollout (today). The throughline is a single discipline — replace a ticket and a shared credential with a reviewed, allowlisted, self-service claim, and move in reversible steps that always leave the old path standing until the new one is proven.
Before and after, side by side
A big-bang migration revokes the shared identity before the new path has accumulated production evidence, so a single surprise becomes a fleet-wide incident. The target rollout separates platform readiness from service cutover and moves document-export-service through reversible mint, flip, soak, and revoke steps before widening the rollout.
The right side leaves a safe return path until the new identity has produced customer and operator evidence under soak. Only after successful own-resource access, denied cross-service access, healthy reconciliation, and an unaffected control does the rollout revoke the old grant and expand to the next environment.
Key takeaways
- The rollout seam sits between platform ready and in use: Phase 1 stands everything up with zero cutover; Phase 2 cuts services over in non-prod; Phase 3 goes to prod by cluster class and widens resource types.
- Cutover is reversible: flip a pre-shipped (disabled) helper, mint → observe under soak → revoke the old grant last, old identity live until the new one is proven.
- Risks are closed, handled, or accepted-with-a-fence, each named — a broad controller role fenced at the claim allowlist; a non-prod-only impersonation vector accepted because prod doesn't share the topology.
- Production has a higher gate than the pilot (boundary + admission-enforce + auditable type-split) and rolls by cluster class.
- Done is a measured outcome — teams self-serving end-to-end and a quantified turnaround-time win — not a list of shipped components.
Checklist
- [ ] I can explain why the rollout seam is placed between platform-ready and cutover, and what Phase 1 deliberately excludes.
- [ ] I can describe the reversible cutover sequence and why the old shared grant is revoked last.
- [ ] I can distinguish closed, handled, and accepted-with-a-fence risks and give an example of each.
- [ ] I can state what the production gate adds over the pilot bar and why prod rolls by cluster class.
- [ ] I can state a definition of done written as a measured outcome rather than a checklist.