Promote or Replay-Roll Back from Evidence
Consume the Day 02 cutover decision record to run a named promotion gate, diagnose failures bottom-up, and replay safely if green must be abandoned.
The enterprise problem and today’s slice
Enterprise problem: A technically complete green database can still harm customers if no person owns the promotion decision, if replication or readiness evidence is stale, or if blue is deleted before replay rollback has been tested.
Whole-course context: Day 01 created a compatibility contract; Day 02 produced a snapshot, offset, organization affinity, and keep-blue-or-discard-green decision; today consumes those artifacts to decide, execute, observe, and recover a production cutover.
Today’s slice: We require a named promoter and QA/evaluation gate, freeze and drain with explicit RPO/RTO, observe substrate through customer evidence, replay events idempotently on rollback, and bake before any blue deletion.
End-of-day evidence: A promotion or rollback receipt names the promoter, accepted gate inputs, organization cohort, freeze and drain observations, RPO/RTO result, customer probes, replay ledger, bake window, and deletion hold.
Still unsolved: This course does not replace disaster recovery design, legal retention policy, or a service-specific incident command process; it supplies the migration evidence those processes need.
Thesis: Promotion is an accountable decision over evidence, not the final step of a deployment pipeline; if the evidence cannot prove recovery, the safe outcome is no promotion or a replay rollback while blue remains intact.
This final day operates the release boundary rather than redesigning the service. It treats a successful health check as one signal among storage, replication, workload, routing, and customer-path evidence.
Customer use cases
An unnamed “go” decision makes ambiguous evidence somebody else’s problem. These use cases make the promoter accountable for a bounded cohort and give the recovery operator an executable path before any recovery database is deleted.
| Use case ID | Actor | Customer job | Success outcome | Denial or recovery evidence |
|---|---|---|---|---|
| D03-UC-01 | Named release promoter | Decide whether a prepared organization cohort may move to green | Promotion occurs only when QA, replication, freeze, drain, RPO/RTO, routing, and customer probes meet recorded thresholds | Missing owner or evidence, failed QA, stale readiness, or unresolved lag produces an explicit no-go receipt and blue remains active |
| D03-UC-02 | Recovery operator | Return an affected cohort to blue after promotion without losing or duplicating ingestion | Router returns atomically to blue, saved-offset events replay with idempotency, and customer probes recover before the bake deadline | Replay duplicate, missing event, cross-schema trace, or early-blue-deletion attempt stops recovery and escalates with immutable evidence |
Actor-centred user stories
If promotion evidence is spread across dashboards and no artifact says who accepted it, an incident begins with a debate rather than a recovery action. These stories make the gate and rollback ledger reviewable.
| Story ID | Use case IDs | User story | Observable acceptance conditions |
|---|---|---|---|
| D03-US-01 | D03-UC-01 | As a named release promoter, I want one signed gate receipt, so that every threshold and unresolved exception is visible before a cohort changes color | Receipt binds promoter identity, decision scope, QA result, replication state, frozen offset, drain result, RPO/RTO targets, router revision, probes, and expiry |
| D03-US-02 | D03-UC-02 | As a recovery operator, I want replay records keyed by source event identity, so that rollback can reprocess only missing work without double-applying it | Replay ledger records event ID, source offset, target result, idempotency outcome, old/new affinity revision, and a passing unaffected-organization control |
End-to-end product flows
The promotion path is incomplete if it never checks the customer request after routing changes, and the rollback path is unsafe if it assumes an event can be applied twice. RPO is the maximum acceptable data loss measured in time or events; RTO is the maximum acceptable time to restore the customer workflow.
| Flow ID | Use case IDs | Path | Trigger | Numbered steps | Terminal evidence |
|---|---|---|---|---|---|
| D03-FLOW-01 | D03-UC-01 | Happy | Named promoter opens the prepared cohort gate | 1. Resolve promoter, cohort, and non-expired Day 02 decision. 2. Verify blue/green identity, replication state, QA result, saved offset, backlog headroom, and drain completion. 3. Measure RPO/RTO against the accepted target. 4. Atomically set cohort affinity to green. 5. Run green customer read/write, stale-read, and timeout probes. 6. Start bake timer while retaining blue. | Signed gate receipt, affinity revision, old/new database identity, target and observed RPO/RTO, probe traces, bake deadline, environment, timestamp, and immutable promotion ID |
| D03-FLOW-02 | D03-UC-02 | Recovery | Customer probe or QA finding fails after promotion | 1. Stop further cohort promotion. 2. Confirm blue still exists and matches the retained recovery point. 3. Atomically return only the affected cohort to blue. 4. Reconcile saved-offset events through idempotency keys. 5. Verify no duplicate/missing event and an unaffected green or blue control as applicable. 6. Preserve green for diagnosis; do not delete blue early. | Rollback receipt, affected cohort, old/new affinity revisions, source offsets, replay outcomes, duplicate/missing counts, customer probes, retained database identities, environment, timestamp, and immutable incident ID |
The promoter is not a controller: the gate authorizes a conditional router update, while the router owns affinity state and the workload serves requests. A failed QA gate, ambiguous ownership, or absent evidence is a denied outcome, not a reason to override the gate. A synchronous write-read timeout needs bottom-up diagnosis because it can arise from exhausted storage/network capacity, a database connection path, replication lag, router affinity, or the application’s timeout budget.
System design derived from the flows
A dashboard can be green while the customer path fails, so diagnosis follows dependency order: substrate capacity, database/control-plane reachability, replication and storage state, workload readiness, router affinity, then the end-to-end organization request. This prevents deleting or replaying against an unverified target.
| Use case ID | Entry point | Responsible services | Authoritative store | Failure evidence |
|---|---|---|---|---|
| D03-UC-01 | Named promoter opens a cohort gate | Promotion policy service, QA/evaluation service, capacity monitor, replication monitor, readiness/drain controller, organization router | Signed promotion receipt and router affinity registry; database and monitoring systems remain authoritative for observed state | Missing promoter, expired decision, failed QA, RPO/RTO breach, readiness/drain mismatch, replication lag, or failed customer probe |
| D03-UC-02 | Recovery operator invokes cohort rollback | Incident controller, organization router, replay worker, durable ingestion service, idempotency store, evidence ledger | Router affinity revision, saved offset, replay ledger, and retained blue database | Blue absence, affinity conditional-write conflict, replay duplicate, missing event, stale read, or deletion-hold violation |
The operational rule is one desired-state authority per boundary: the promotion receipt authorizes one router update; the router reconciles organization affinity; the database owns stored data; the replay ledger owns duplicate suppression. Before any state surgery or deletion, independently inspect the real blue database identity and a known-good organization request. A healthy controller never proves an adjacent layer is healthy.
Data model and ownership
Rollback cannot be reproduced from an alert alone; it needs durable proof of what was promoted, what was replayed, and what deletion was prohibited. The service databases remain the customer-data authority while this day adds gate and replay evidence.
Generated-application database: Not created in this slice — service-owned blue/green databases hold customer data, while promotion, replay, and bake records provide the durable operational evidence.
| Record or entity | Store and owner | Primary key | Foreign key or opaque reference | Tenant key | Material constraint | Lifecycle and deletion | Use case IDs |
|---|---|---|---|---|---|---|---|
| PromotionReceipt | Evidence store, owned by promotion policy service | promotion_id | Day 02 CutoverDecision and router revision references | Cohort identifier | Named promoter, evidence digest, thresholds, decision, and expiry are mandatory; no implicit approver | Append-only; superseded by later receipt; retained through audit policy | D03-UC-01 |
| OrganizationAffinity | Router registry, owned by organization router | organization_id plus affinity_revision | PromotionReceipt or RollbackReceipt reference | organization_id | Conditional revision update; exactly one active color; request trace must echo revision | Updated by authorized transition; history retained through bake and rollback window | D03-UC-01, D03-UC-02 |
| ReplayLedgerEntry | Replay store, owned by recovery operator | source event_id plus target database identity | Saved offset and RollbackReceipt references | organization_id | Unique source event identity per target effect; duplicate is recorded as no-op, never silently applied | Created during recovery; retained until event retention and incident review expire | D03-UC-02 |
| BakeHold | Release evidence store, owned by named promoter | bake_hold_id | PromotionReceipt and blue database identity reference | Cohort identifier | Blue deletion is denied until deadline, healthy probes, and explicit retirement decision all exist | Created at promotion; expires only into a reviewed retire/extend decision | D03-UC-01, D03-UC-02 |
| RollbackReceipt | Incident evidence store, owned by recovery operator | rollback_id | PromotionReceipt, affinity revisions, and replay range references | Cohort identifier | Requires blue identity proof and post-replay customer probe; cannot claim success with unknown replay gaps | Append-only; retained with incident and release history | D03-UC-02 |
Promotion gate and bottom-up diagnosis
The gate exists because a migration can be “finished” at the database layer while still failing the customer. A no-go result is complete when it names the failing premise, owner, recovery boundary, and next observable check.
| Gate input | Required proof | Reject or recover when |
|---|---|---|
| Ownership | Named promoter and recovery operator; exact cohort scope | Owner is missing, scope overlaps, or decision is expired |
| Data | Snapshot identity, replication state, validation, RPO observation | Apply stopped, lag breaches target, or data validation is incomplete |
| Ingestion | Saved offset, freeze/drain result, backlog and disk/time headroom | Offset is absent, backlog grows beyond capacity, or drain still accepts work |
| Runtime | Storage/network capacity, workload readiness, router revision, timeout/staleness probes | Substrate is saturated, readiness is stale, or a trace crosses colors |
| Customer | Read/write and negative contract probes for the promoted organization plus unaffected control | QA fails, write-read is inconsistent, or evidence cannot identify a request |
| Recovery | Blue identity, replay plan, duplicate suppression, bake hold | Blue was deleted early, replay is untested, or duplicate behavior is unknown |
When a gate fails, diagnose in this order: verify CPU, memory, disk, and network headroom; then database endpoints and replication state; then Job/workload readiness and drain; then affinity revision and request trace; finally replay the customer probe. This catches a disk-exhausted frozen backlog or a stopped replication worker before blaming the router. Accidental early blue deletion is not a routine rollback: stop promotion, record the missing recovery point, restore only through the organization’s established backup/disaster-recovery process, and do not claim replay rollback is available.
Completion and next action
The final decision rule is: promote only with a named owner and complete evidence; roll back only while the retained blue point and replay ledger make it testable; delete blue only after the bake hold, healthy customer evidence, and an explicit retirement decision. These rules turn the listed edge cases—partial Jobs, replication limits, capacity exhaustion, cross-schema traffic, failed QA, duplicate replay, and ambiguous ownership—into bounded, observable outcomes.
Key takeaways
- Promotion authority, evidence scope, and expiry must be explicit.
- RPO and RTO are acceptance criteria that require measurement, not labels on a runbook.
- Replay rollback needs a retained blue target and idempotency keyed by source event identity.
- Bake before deletion preserves a recovery option when the first customer path reveals a missed edge case.
Checklist
- [ ] I can name the promoter, cohort, decision expiry, and evidence digest for a cutover.
- [ ] I can diagnose a timeout from capacity through database, workload, routing, and customer probe.
- [ ] I can atomically route one organization back to blue and replay only missing events.
- [ ] I can explain why early blue deletion invalidates the normal rollback claim.