Change a Live Schema Without Breaking Old Code
Build a compatibility-first migration for Example Knowledge Service, then prove both the old and new application versions can serve the same data.
The enterprise problem and today’s slice
Enterprise problem: A customer needs Example Knowledge Service to add a required source_kind field while requests continue, but an application release and a database change cannot be assumed to arrive at the same instant; an old process can fail on a new constraint and a new process can fail on an absent field.
Whole-course context: This day creates the compatibility contract and migration evidence that Day 02 uses to decide whether a breaking change needs a second database and a cohort cutover; Day 03 turns that evidence into a promotion or rollback decision.
Today’s slice: We use an additive, or backward-compatible, schema change, an idempotent Kubernetes Job, readiness and draining controls, and contract probes for the application and database boundary.
End-of-day evidence: A reviewed migration record shows the Job can be retried safely, source_kind is populated, old and new releases pass their stated compatibility probes, and drained instances stop receiving new work before shutdown.
Still unsolved: Replacing an incompatible type, changing identity semantics, or requiring a no-downtime data copy is deliberately deferred to the blue/green decision in Day 02.
Thesis: A safe live migration is not “run SQL before deploy”; it is a transition contract in which every live version can read and write every state that the migration exposes.
This course stays inside application-to-database delivery, durable ingestion, routing, and operating evidence. It intentionally does not prescribe a cloud provider, a database product beyond the cited examples, or a particular continuous-delivery tool.
Customer use cases
An apparently harmless field addition becomes an outage when one live version assumes the field exists or is non-null before another version can tolerate it. These use cases make the customer-visible read and write contract explicit.
| Use case ID | Actor | Customer job | Success outcome | Denial or recovery evidence |
|---|---|---|---|---|
| D01-UC-01 | Service engineer | Add source_kind to a live knowledge record without interrupting reads or writes | Both release versions accept records during the transition and new records converge to a populated value | A compatibility probe rejects a new NOT NULL constraint before backfill; the old release remains a positive control |
| D01-UC-02 | Release operator | Run and retry the backfill without duplicating or corrupting records | The migration Job records a durable checkpoint and converges after an interrupted attempt | Job status, row counts, and idempotency keys identify partial state; retry changes only unfinished rows |
Actor-centred user stories
If a schema change is described only as a database task, nobody owns the old process still serving customers. These stories bind the data contract to observable release behavior.
| Story ID | Use case IDs | User story | Observable acceptance conditions |
|---|---|---|---|
| D01-US-01 | D01-UC-01 | As a service engineer, I want old and new releases to handle a missing or defaulted source_kind, so that a rolling deployment cannot expose an incompatible record shape | Old release reads a record with the new column; new release reads a pre-backfill record; both write paths pass contract probes with request and record identifiers |
| D01-US-02 | D01-UC-02 | As a release operator, I want a resumable backfill Job, so that a pod restart does not turn a partial migration into a manual repair | An interrupted Job resumes from its checkpoint, preserves already-completed rows, and emits one immutable run ID and one unaffected-record probe |
End-to-end product flows
A rolling release is unsafe if the database contract narrows before the last old process drains. The happy path therefore expands first, while the recovery path proves that a partial Job can converge instead of being blindly rerun.
| Flow ID | Use case IDs | Path | Trigger | Numbered steps | Terminal evidence |
|---|---|---|---|---|---|
| D01-FLOW-01 | D01-UC-01 | Happy | Engineer approves an additive migration | 1. Add nullable source_kind with a documented default behavior. 2. Deploy code that reads absence as unknown and writes the new field. 3. Run old/new read and write contract probes. 4. Backfill in bounded batches. 5. Drain old instances from readiness before termination. 6. Add the enforcing constraint only after zero missing values and no old processes that write remain. | Migration revision, schema version, old/new image digests, probe results, zero-null query, drained-instance count, environment, timestamp, and immutable release ID |
| D01-FLOW-02 | D01-UC-02 | Recovery | The migration Job stops after processing a batch | 1. Inspect the Job run ID, checkpoint, failed batch, and database lock/connection state. 2. Verify a completed row and an untouched row. 3. Correct the bounded failure. 4. Retry with the same idempotency key. 5. Re-run counts and both release probes. | Failed and retry Job IDs, checkpoint range, expected versus observed counts, successful replay result, unaffected positive control, environment, timestamp, and immutable evidence bundle |
An idempotent operation reaches the same intended result if repeated: setting an absent source_kind to unknown is idempotent; incrementing a counter is not. Kubernetes Jobs retry failed Pods according to their retry policy, so the Job must persist enough progress to make retries safe rather than assume each attempt is fresh (Kubernetes Job documentation).
System design derived from the flows
The design needs one owner for each state transition; otherwise application deployment, SQL tooling, and a Job can overwrite one another’s assumptions. The migration manifest declares desired work, the Kubernetes Job controller retries Pods, and the database remains authoritative for data and migration checkpoints.
| Use case ID | Entry point | Responsible services | Authoritative store | Failure evidence |
|---|---|---|---|---|
| D01-UC-01 | Reviewed schema-and-release change | Release controller, Example Knowledge Service, readiness endpoint, database migration executor | Versioned migration repository for intent; knowledge database for schema and records | Old/new contract probe failure, readiness still accepting traffic while terminating, null count above zero, or schema revision mismatch |
| D01-UC-02 | Approved backfill Job run | Kubernetes Job controller, migration worker, database checkpoint table, observability pipeline | Knowledge database checkpoint and per-record migration marker | Retry count exhausted, duplicate mutation attempt, missing checkpoint, lock timeout, or completed/untouched control mismatch |
The cluster schedules the Job’s worker Pod onto CPU, memory, disk, and network capacity; the worker changes database rows; a query, Job condition, and customer probe prove the effect. Do not infer a finished migration from kubectl apply alone. In PostgreSQL, a transaction is all-or-nothing only inside its own boundary; a long backfill still needs batch-level checkpoints and an application contract around it.
Data model and ownership
Without a record of the intended schema and each batch’s result, an operator cannot distinguish an idempotent retry from an unsafe duplicate. This day creates durable application data and migration evidence, but no separate generated application database.
Generated-application database: Not created in this slice — the service-owned knowledge database, migration checkpoint, and immutable release evidence are sufficient durable state.
| Record or entity | Store and owner | Primary key | Foreign key or opaque reference | Tenant key | Material constraint | Lifecycle and deletion | Use case IDs |
|---|---|---|---|---|---|---|---|
| KnowledgeRecord | Knowledge database, owned by Example Knowledge Service | knowledge_record_id | source_id is an opaque upstream reference | organization_id | source_kind may be absent only during the named compatibility window; readers must map absence to unknown | Created by ingestion; updated by idempotent backfill; retained and deleted under organization retention policy | D01-UC-01, D01-UC-02 |
| SchemaContract | Versioned migration repository, owned by service engineering | schema revision digest | Release digest and database revision are opaque build references | None — one service contract | Expand, migrate, and contract order is immutable for a release train; enforcing constraint is blocked until compatibility proof | Created in review; superseded by later revision; retained with release evidence | D01-UC-01 |
| MigrationCheckpoint | Knowledge database, owned by migration worker | migration_run_id plus batch cursor | SchemaContract digest and Job UID | None — operational record is global to one run | Unique record range plus idempotency key; completed batches cannot be re-applied with a different payload digest | Created at Job start; retained for audit; expired only after recovery retention | D01-UC-02 |
| CompatibilityProbe | Evidence store, owned by release operations | probe_run_id | Opaque release digest, schema revision, and request trace | organization_id when a customer record is used | Must capture old and new images, expected and observed read/write result, and positive control | Append-only; detailed payloads expire by privacy policy while result metadata remains | D01-UC-01, D01-UC-02 |
The contract makes expand/migrate/contract a decision rule: expand when both schemas can coexist, migrate until evidence shows convergence, and contract only after every old process that reads or writes is gone. A type mismatch, a missing foreign-key cascade, or a new non-null field is not a signal to “try it in production”; either add a compatibility representation or choose the isolated blue/green path next.
Curriculum ledger and the next decision
A three-day course must accumulate artifacts rather than repeat deployment advice. This ledger is the prerequisite graph and handoff contract for the capstone decision.
| Day | Incoming concepts or artifact | One new concept and capability | End-to-end learner job | Visual delta | Reproducible evidence | Handoff to the next day |
|---|---|---|---|---|---|---|
| 01 | Familiar release and database change | Compatibility-first expand/migrate/contract | Make one additive field change safely and recover a partial Job | Adds migration worker, checkpoint, readiness, and evidence to the request path | Old/new probes, zero-null count, Job checkpoint, drain observation | Compatibility contract and migration evidence bundle |
| 02 | Compatibility contract and evidence bundle | Blue/green isolation plus atomic cohort routing | Choose isolation for an incompatible schema and validate green before promotion | Adds snapshot, replication, durable ingestion offset, router, and cohort state | Snapshot ID, replication lag, offset, headroom, cohort probes, QA report | Cutover decision record and retained blue recovery point |
| 03 | Cutover decision record and retained blue recovery point | Named promotion gate and replay rollback | Promote, diagnose, or roll back from evidence without deleting recovery state early | Adds named promoter, operational gate, replay ledger, and delete hold | Gate receipt, RPO/RTO observations, promotion/replay tests, bake record | Signed completion or explicit unresolved-risk record |
Key takeaways
- Additive migration is safe only when old and new code both tolerate the transition state.
- An idempotent Job needs a checkpoint, a bounded batch, and a positive control.
- Readiness removes new traffic; draining finishes owned in-flight work before termination.
- Contract after measurable convergence, never immediately after a schema command succeeds.
Checklist
- [ ] I can explain why
ADD COLUMN NOT NULLis usually not the first live step. - [ ] I can make a migration retry-safe with a cursor, idempotency key, and row-level condition.
- [ ] I can prove old and new releases read and write the transition state.
- [ ] I can identify a type or relationship change that should move to blue/green isolation.