Distributed System Patterns II: Data & Coordination

Yesterday's patterns kept a system standing when a component misbehaved. Today's patterns solve a different class of problem: how to model, store, and coordinate data once it is spread across many services and machines. A single database with a single transaction is the simplest world; the moment data lives in several places, you can no longer wrap everything in one atomic step, and new patterns fill that gap.

The recurring tension: a classic database transaction gives you ACID guarantees — a group of changes that all succeed or all fail together, leaving the data consistent — but a transaction cannot span independent services. So these patterns each answer some version of "how do we keep data correct, or eventually correct, when no single transaction can cover the whole operation?"

As before, each pattern gets the problem, the pattern, and a one-line "when to use." They are a vocabulary for data-heavy designs, not a sequence to apply in order.

CQRS splits a system's operations into two separate models: commands that change data (write) and queries that read it. Instead of one model serving both, you build a write model optimized for correctness and a read model optimized for fast, convenient reads.

• Problem. Reads and writes pull one data model in opposite directions — writes want a normalized, consistent shape; reads want denormalized, query-shaped views. Serving both from one model makes each worse.
• Pattern. Separate the two. Commands update the write store; that change propagates to one or more read stores shaped exactly for the queries they serve. The read side is often eventually consistent — it catches up a moment after the write.
• When to use. Read and write workloads differ sharply in shape or scale, and you can tolerate the read side lagging the write side slightly.

Event sourcing stores the full history of what happened as an ordered log of immutable events, rather than storing only the current state. The current state is derived by replaying the events from the start. Instead of a row that says balance = 80, you keep deposited 100, withdrew 20, and compute the balance.

• Problem. Storing only current state throws away history — you cannot answer "how did we get here?", audit changes, or rebuild a different view of the past.
• Pattern. Treat the append-only event log as the source of truth. Rebuild current state, or any alternate view, by folding over the events. It pairs naturally with CQRS: events are the write side, and read models are projections built from the event stream.
• When to use. You need a full audit trail, temporal queries ("what did this look like last Tuesday?"), or the ability to build new read views from history.

A saga manages a transaction that spans multiple services, where no single database transaction can cover the whole operation. It breaks the operation into a sequence of local transactions — one per service — and, if a later step fails, runs compensating transactions that undo the earlier steps. A compensating action is the deliberate reverse of a step: not a rollback the database gives you for free, but explicit business logic like "refund the payment" or "release the reservation."

• Problem. An operation like "place an order" touches payment, inventory, and shipping — each its own service with its own database. There is no shared transaction to make all three atomic.
• Pattern. Run each step as a local transaction. If step 3 fails, invoke compensating transactions for steps 2 and 1, in reverse, to leave the system in a consistent state. The saga trades atomic all-or-nothing for eventual, compensated consistency.
• When to use. A business operation must span multiple services and you can define a sensible undo for each step.

Sagas come in two coordination styles:

The compensating-transaction flow is what makes a saga safe when a step fails midway:

Read the diagram as a forward path that unwinds on failure: each successful step leaves a trail, and the compensation walks that trail backward, undoing exactly what was done.

Scatter-gather fans a single request out to many workers in parallel (scatter), then combines their responses into one result (gather). It is the pattern behind search — query many shards at once, merge the hits.

• Problem. Answering one request requires consulting many sources or partitions, and doing them one after another would be far too slow.
• Pattern. Send the request to all participants simultaneously, wait for responses (up to a deadline), and aggregate. A slow or dead participant is handled by a timeout and a partial result rather than blocking the whole answer.
• When to use. A request naturally decomposes into independent sub-queries whose results merge — search, price comparison, querying every shard.

Leader election is how a group of interchangeable nodes agrees on exactly one to act as coordinator — the single node responsible for some job, like assigning work or ordering writes. If the leader dies, the group elects a new one.

• Problem. Some job must be done by exactly one node at a time (to avoid duplicate or conflicting work), but any node could be that one, and the current one might fail.
• Pattern. The nodes run an election so that exactly one becomes leader; the rest follow and watch for its failure. Safe election rests on consensus — the process by which a group of nodes agrees on one value despite crashes and lost messages — typically via a store like etcd or ZooKeeper (see the replication-and-consensus day). A fencing token, an ever-increasing number handed to each new leader, stops a stale old leader from still acting.
• When to use. A task requires a single coordinator — a primary writer, a scheduler, a partition owner — and must survive that coordinator failing.

Publish-subscribe (pub/sub) decouples the sender of a message from its receivers. A publisher emits an event to a topic without knowing who listens; any number of subscribers register interest in that topic and each receives a copy. A message broker (a middleman that stores and routes messages) sits between them.

• Problem. A service that produces an event should not have to know, or call, every service that cares about it — that couples them tightly and breaks when consumers change.
• Pattern. Publishers send events to named topics on a broker; subscribers consume from topics independently. Adding a new consumer requires no change to the producer. Delivery is asynchronous, so producer and consumer need not be up at the same instant.
• When to use. One event has many interested, evolving consumers, and you want producers and consumers to scale and deploy independently.

Sharding splits one dataset across multiple databases (shards), each holding a slice of the data keyed by some shard key, so that no single machine holds — or serves — all of it.

• Problem. The data, or the write traffic, outgrows what one database can hold or handle.
• Pattern. Partition the data by a key (user id, region) across independent shards; each request routes to the shard that owns its key. This is covered in depth on the sharding day — here it is one tool among the data patterns.
• When to use. A single database can no longer hold the data volume or absorb the write throughput.

The strangler fig pattern migrates a legacy system to a new one incrementally, routing a growing slice of traffic to the new implementation while the old one keeps running, until the old system is fully replaced and can be removed. The name comes from a vine that grows around a tree and gradually takes its place.

• Problem. A big-bang rewrite — replace the whole legacy system at once — is high-risk and often fails, but you cannot freeze the system for a long migration either.
• Pattern. Put a routing layer (a facade) in front of the legacy system. Reimplement one capability at a time in the new system and redirect just that capability's traffic to it. Repeat until nothing routes to the legacy system, then delete it.
• When to use. You must modernize or replace a running system without downtime and without betting everything on a single cutover.

When a transaction must span services, there are two families of answer, and choosing between them is a core distributed-data decision. Two-phase commit (2PC) is the strongly consistent option: a coordinator asks every participant to prepare (phase one), and only if all vote yes does it tell them all to commit (phase two); any "no" aborts everyone. It keeps the data strictly consistent but holds locks across all participants for the whole exchange, so one slow or crashed participant can block the rest.

The saga is the loosely coupled option from earlier today: no distributed locks, each step commits locally, and failures are handled by compensation rather than a coordinated abort. It stays available and scalable at the cost of only eventual consistency and the effort of writing compensating actions.

The rule of thumb: reach for 2PC when the operation is short, the participants are few, and a wrong result is unacceptable; reach for a saga when the operation is long-running or spans many services and you would rather stay available and reconcile than lock everyone and block.

These patterns are not rivals — they combine into a coherent data architecture, and it helps to see what problem each one owns.

A realistic write-heavy service stacks several: commands are event-sourced, a CQRS read model is a projection of those events, cross-service steps run as a saga, downstream services learn of changes via pub/sub, and the whole store is sharded by tenant. Each pattern earns its place by solving one named problem — the same discipline as the resilience day.

• These patterns answer one question in many forms: how to keep data correct, or eventually correct, when no single transaction can span the whole operation.
• CQRS separates write and read models; event sourcing stores the history of events and derives state by replay — the two pair naturally.
• Sagas replace cross-service atomicity with local steps plus compensating transactions, coordinated by choreography (events) or orchestration (a central driver).
• Scatter-gather fans out and merges; leader election picks one failover-safe coordinator via consensus; pub/sub decouples producers from evolving consumers.
• Strangler fig migrates a live system incrementally instead of a risky big-bang cutover.
• 2PC gives strong, immediate consistency but locks all participants; a saga stays available with eventual, compensated consistency — pick by operation length and participant count.

• [ ] Can you explain how CQRS and event sourcing complement each other?
• [ ] Can you describe a saga's compensating transactions and how they unwind a failed step?
• [ ] Do you know the difference between saga choreography and orchestration?
• [ ] Can you trace a scatter-gather request from fan-out to merged response, including a slow participant?
• [ ] Can you state why leader election depends on consensus and what a fencing token prevents?
• [ ] Can you contrast 2PC and saga on consistency, locking, and availability under failure?
• [ ] Can you explain how the strangler fig pattern avoids a big-bang rewrite?