Isolate a Breaking Migration with Blue/Green Data
Use the Day 01 compatibility contract to decide when an incompatible schema needs a point-in-time copy, a durable-ingestion freeze, and atomic per-organization routing.
The enterprise problem and today’s slice
Enterprise problem: Example Knowledge Service must replace a text identifier with a typed relationship and enforce referential integrity, but old and new representations cannot safely share one write path; mixed requests can create records that neither version can interpret.
Whole-course context: Day 01 supplied the compatibility contract and Job evidence; today consumes that bundle to choose blue/green isolation for a breaking migration and produces the cutover decision record that Day 03 will promote or replay.
Today’s slice: We distinguish a point-in-time snapshot from a replicated green database, freeze durable ingestion at a saved offset, measure backlog and disk headroom, route whole organizations atomically, and retain blue until promotion evidence is accepted.
End-of-day evidence: A decision record binds snapshot ID, replication health, frozen offset, backlog estimate, headroom, cohort routing version, validation results, and a pre-promotion choice to keep blue or discard green.
Still unsolved: Final production promotion authority, RPO/RTO acceptance, post-promotion replay rollback, and deletion after a bake period are deliberately deferred to Day 03.
Thesis: Blue/green protects an incompatible data model only when the boundary is the entire organization’s state and routing is atomic; random request weighting creates two schemas for one customer.
This day covers data-copy and traffic-isolation mechanics. It does not assume a provider-managed replication product or claim that a green health signal alone proves customer correctness.
Customer use cases
An incompatible relationship change cannot be protected by sending ten percent of individual requests somewhere else, because a customer’s next request may read a different schema. These use cases treat an organization as the indivisible cutover unit.
| Use case ID | Actor | Customer job | Success outcome | Denial or recovery evidence |
|---|---|---|---|---|
| D02-UC-01 | Data migration engineer | Create and validate green data for the new typed relationship without changing blue customer traffic | Green starts from a named snapshot, catches up through a named replication boundary, and passes relationship validation | Replication lag, unsupported DDL, type conversion error, or missing foreign-key cascade keeps the organization on blue |
| D02-UC-02 | Release operator | Prepare an organization-level cutover without losing ingestion events | Ingestion freezes at a durable saved offset, backlog fits proven disk/time headroom, and one organization resolves to exactly one color | Backlog growth, disk threshold breach, stale offset, or cross-schema request denies promotion and preserves blue |
Actor-centred user stories
If the router makes a probabilistic decision per request, a successful write can be invisible to the following read. These stories demand a stable cohort record, where a cohort is a named set of organizations sharing the same release treatment.
| Story ID | Use case IDs | User story | Observable acceptance conditions |
|---|---|---|---|
| D02-US-01 | D02-UC-01 | As a data migration engineer, I want green validation to name its snapshot and replication position, so that a passing query cannot hide a stale or structurally incomplete copy | Validation receipt includes snapshot ID, replication position, source/target counts, invalid relationship count, and the exact schema revision tested |
| D02-US-02 | D02-UC-02 | As a release operator, I want each organization pinned to one color, so that no request can write blue and read green during an incompatible migration | Router trace for the organization shows one cohort revision and color; an attempted conflicting affinity is denied; a different organization remains an unaffected blue control |
End-to-end product flows
Green is not “a replica with a new name.” A snapshot is a fixed copy at one instant; replication is the continuing application of later changes. The flow must prove both before it freezes an ingestion stream, which is a durable append-only sequence identified by offsets.
| Flow ID | Use case IDs | Path | Trigger | Numbered steps | Terminal evidence |
|---|---|---|---|---|---|
| D02-FLOW-01 | D02-UC-01 | Happy | Migration engineer approves a breaking type-and-relationship plan | 1. Record blue schema and snapshot identity. 2. Create green from the snapshot. 3. Apply the green-only schema transformation. 4. Start replication from the matching position. 5. Validate type conversion, orphan relationships, and cascade policy. 6. Keep blue serving traffic while green QA runs. | Snapshot ID, replication start/current positions, lag, conversion rejects, foreign-key validation, green QA run ID, blue health observation, environment, timestamp, and immutable plan digest |
| D02-FLOW-02 | D02-UC-02 | Failure | Operator begins the ingestion freeze and headroom check | 1. Stop new ingestion admission at a named boundary. 2. Persist the last accepted offset. 3. Measure backlog rate, retention window, free disk, and expected catch-up time. 4. Reject promotion if any threshold is unknown or exceeded. 5. Resume blue ingestion or discard green according to the decision record. | Freeze ID, saved offset, backlog bytes/messages, disk and time forecasts, threshold decision, blue positive-control request, recovery action, environment, timestamp, and immutable evidence ID |
Logical replication has explicit limits: PostgreSQL does not replicate DDL changes, and schema changes must therefore be coordinated separately; conflicts during apply can stop replication rather than silently making target data correct (logical replication restrictions, logical replication conflicts). The reusable rule is to treat schema transformation, replication health, and data validation as three distinct proofs.
System design derived from the flows
The system needs separate authoritative records for cohort affinity, ingestion position, and cutover decision. Conflating them lets a healthy replication worker accidentally authorize customer routing.
| Use case ID | Entry point | Responsible services | Authoritative store | Failure evidence |
|---|---|---|---|---|
| D02-UC-01 | Reviewed breaking-migration plan | Snapshot service, replication worker, green transformer, validator, database control plane | Blue/green database metadata and immutable migration plan | Snapshot mismatch, replication apply stop, unsupported DDL, conversion reject, orphan count, or validation digest mismatch |
| D02-UC-02 | Operator selects prepare cohort cutover | Ingestion admission controller, durable ingestion service, capacity monitor, organization router, cutover policy service | Saved ingestion offset, capacity observation store, organization affinity registry, cutover decision record | Frozen-backlog threshold breach, disk exhaustion forecast, missing offset, router affinity conflict, or cross-schema trace |
At the substrate layer, green consumes storage for the snapshot, replica WAL or equivalent change stream, transformed indexes, and frozen backlog; measure free disk and write throughput before calling the freeze safe. At the application layer, the router resolves organization_id from one durable affinity record before it selects blue or green. A write-read timeout or stale read is evidence to investigate routing, replica lag, and connection timeouts—not permission to retry against the other color.
Data model and ownership
A blue/green plan fails auditability when “green is ready” has no snapshot, offset, cohort, or decision behind it. This slice creates migration-control state and retains customer data in the service-owned databases.
Generated-application database: Not created in this slice — blue and green remain service-owned knowledge databases; routing, freeze, and validation records are durable operational control-plane state.
| Record or entity | Store and owner | Primary key | Foreign key or opaque reference | Tenant key | Material constraint | Lifecycle and deletion | Use case IDs |
|---|---|---|---|---|---|---|---|
| SnapshotRecord | Migration control store, owned by data migration engineering | snapshot_id | Opaque blue database revision and log position | None — one migration artifact may span organizations | Snapshot position and source schema digest are immutable and must match replication start | Created before green; retained through bake and rollback; deleted only after signed retirement | D02-UC-01 |
| ReplicationObservation | Observability store, owned by data migration engineering | replication_observation_id | SnapshotRecord and opaque source/target positions | None — operational global record | Lag, apply state, conflict state, and schema revision are all required; healthy applies do not waive validation | Append-only observations retained through incident window | D02-UC-01 |
| IngestionFreeze | Ingestion control store, owned by release operations | freeze_id | Durable ingestion service and saved offset reference | None — stream boundary is global | Exactly one active freeze per stream; stored offset is monotonic and must be acknowledged before admission closes | Created for preparation; resumed or superseded by explicit decision; retained as evidence | D02-UC-02 |
| OrganizationAffinity | Router registry, owned by release operations | organization_id plus affinity_revision | Opaque CutoverDecision reference | organization_id | One active color per organization; conditional update prevents conflicting cohort assignments | Created or changed by named promotion; retained for rollback; removed only after data retirement | D02-UC-02 |
| CutoverDecision | Evidence store, owned by named release authority | cutover_decision_id | SnapshotRecord, freeze, QA, and router revision references | Cohort identifier, not an authorization transfer | Must state keep-blue or discard-green before promotion; missing evidence is a denial | Append-only; superseded only by a later signed decision | D02-UC-01, D02-UC-02 |
The decision rule is clear: use expand/migrate/contract when the old and new representations can coexist; use blue/green when correctness requires an incompatible representation or isolated validation. Never use probabilistic request weighting across incompatible schemas. Before promotion, keep blue if replay rollback is viable and validated; discard green if QA fails, replication is ambiguous, ownership is unclear, or capacity evidence is incomplete.
Edge cases and the Day 03 handoff
The dangerous cases are not exotic: a type cast can reject one historical value; an absent cascade can leave orphan children; DDL can fail to reach green; frozen ingestion can fill disk; a router can pin a write to blue and a read to green; QA can pass only a stale replica. Each has the same recovery pattern: stop the unsafe transition, preserve blue as the positive control, capture the exact boundary and evidence, then either repair and revalidate green or discard it.
Key takeaways
- A snapshot is fixed history; replication is continuing change capture; neither replaces validation.
- Freeze ingestion at a durable offset and prove backlog, disk, and time headroom before cutover.
- Atomic organization affinity is a correctness boundary, not an experiment percentage.
- Failed QA or incomplete evidence means retain blue and reject promotion.
Checklist
- [ ] I can name the snapshot ID, replication position, and schema digest that green validated.
- [ ] I can explain why DDL, replication apply, and data correctness need different checks.
- [ ] I can calculate whether a frozen backlog fits available disk and recovery time.
- [ ] I can prove one organization never crosses schemas during a write-read sequence.