Event-Driven Architecture

An event is a record of something that already happened — a fact, stated in the past tense: OrderPlaced, PaymentCaptured, EmailBounced. An event-driven architecture is a system whose parts react to those facts instead of calling each other directly. When a service does its work, it publishes an event announcing what it did, and any other service that cares reacts to it. The service that publishes is the producer; each service that reacts is a consumer. The crucial property is that the producer does not know who — if anyone — is listening.

Contrast the two shapes directly. In a direct (synchronous) call, the caller names the callee, waits for it, and depends on it being awake:

In an event-driven shape, the order service publishes one fact and walks away; the interested services react on their own:

The order service in the second picture has no idea inventory, payment, email, or analytics exist. That inversion — the producer announces, consumers subscribe — is the whole idea. It buys loose coupling (you can add a fifth reactor without touching the order service) and independent evolution (each side deploys and scales alone). It costs directness: the flow of cause and effect is now spread across many services reacting to a stream of facts, which is harder to trace than a single call stack. The rest of this day is how to get the benefit while keeping the cost honest.

Two kinds of message travel through these systems and confusing them causes real design mistakes. A command is an instruction to do something, named in the imperative: PlaceOrder, ChargeCard, SendEmail. It is addressed to one specific handler, it expects that handler to act, and the sender usually wants a result. An event is a notification that something already happened, named in the past tense: OrderPlaced, CardCharged, EmailSent. It is addressed to no one in particular, it expects nothing, and any number of consumers may react — or none.

The distinction matters because it decides who is in control. With a command, the sender is asserting authority: "make this happen," and it holds the outcome. Naming a message as a command is admitting a directed dependency — the sender needs that one handler. With an event, the publisher gives up all authority over what follows: it states a fact and the reactors decide, independently, what that fact means for them. Naming a message as an event is a promise that you will not depend on any particular reaction.

Aspect	Command	Event
Grammar	Imperative — PlaceOrder	Past tense — OrderPlaced
Intent	"Do this" (an instruction)	"This happened" (a fact)
Recipients	Exactly one handler	Zero to many reactors
Who decides the effect	The sender	Each reactor, independently
Coupling direction	Sender depends on handler	Publisher depends on no one
If nobody handles it	A bug — the work is undone	Fine — the fact still happened

A practical tell: if renaming the message from DoX to XHappened would feel wrong because you are counting on something specific to occur, it is a command — keep it a direct call or a targeted queue message. If the past-tense name reads naturally and you genuinely do not care how many parties react, it is an event — publish it and let the world subscribe. Rule of thumb: model instructions with one owner as commands; model facts that others may care about as events, and never smuggle a command in past-tense clothing.

"Event-driven" is not one thing. There are three levels of how much an event carries and how much authority the event log holds, and picking the wrong level is a common source of pain. They form a ladder from thin-and-simple to fat-and-powerful.

Level 1 — Event notification. The event is a thin ping: "order 123 changed, go look." It carries an id and little else. A consumer that wants details calls back to the producer to fetch them. This keeps events tiny and keeps the producer as the single authority on its data, but every reaction now triggers a callback, and the producer is back in the request path it was trying to escape.

Level 2 — Event-carried state transfer. The event is fat: it carries the state a consumer needs — the order's items, address, total — so no callback is required. Consumers can keep their own local copy of just the fields they use and never bother the producer. The producer is now truly decoupled and can even be offline while consumers work, but events are larger, they duplicate data, and a consumer's copy can be momentarily stale.

Level 3 — Event sourcing. The event log stops being a notification channel and becomes the source of truth itself. You do not store current state and emit events about it; you store the events and derive current state by replaying them. "Event sourcing" means the ordered log of every change is the database, and any current value (an order's status, an account balance) is computed by folding the events from the beginning — a fold being the act of starting from an empty value and applying each event in turn to accumulate the present state.

Event sourcing gives a perfect audit trail (every change is a stored fact, nothing is overwritten), time travel (replay to any past point), and the freedom to build brand-new views from history. It costs the most: you must design events as immutable facts forever, replay can be slow so you keep snapshots (a saved fold up to some point, so replay starts near the end instead of the beginning), and querying "current state" is no longer a plain row read.

Level	Event carries	Consumer needs a callback?	Log is the source of truth?	Cost
1. Notification	Just an id / "it changed"	Yes — fetches detail	No	Producer back in the request path
2. State transfer	The state consumers need	No	No	Bigger events, duplicated + slightly stale data
3. Event sourcing	Every change, forever	No	Yes	Replay, snapshots, immutable event design

Most systems live comfortably at level 2, reach for level 1 when data is huge or sensitive enough that copying it everywhere is unwise, and adopt level 3 only for domains where the full history genuinely matters (ledgers, audit, anything where "how did we get here" is a real question). Rule of thumb: default to event-carried state transfer; add callbacks only when events must stay thin, and reach for event sourcing only when the history itself is a first-class asset.

Once several services react to events, a multi-step business process (place an order, then reserve stock, then charge the card, then ship) has to be coordinated somehow. There are two coordination styles, and the choice shapes how the whole system feels to build and debug.

Choreography is decentralized: there is no conductor. Each service reacts to events and emits its own, and the overall process emerges from the chain of reactions — like dancers who each know their own cue rather than following a caller. Inventory reacts to OrderPlaced and emits StockReserved; payment reacts to StockReserved and emits PaymentCaptured; shipping reacts to PaymentCaptured. No single component knows the whole flow.

Orchestration is centralized: one component, the orchestrator (often a workflow engine), knows the whole process and drives it step by step, sending commands and waiting for replies. It calls inventory, waits, calls payment, waits, calls shipping. The steps live in one place you can read top to bottom.

The trade-off is real and runs in opposite directions. Choreography maximizes decoupling — each service knows only its own inputs and outputs, and you add a step by adding a subscriber — but the end-to-end process exists nowhere as a single artifact, so understanding "what happens when an order is placed" means tracing events across many services, and debugging a stuck flow is detective work. Orchestration makes the process visible and easy to change in one place, and failure handling (retry step 2, compensate step 1) has an obvious home — but the orchestrator becomes a component every step depends on, a form of recentralized coupling, and it must not become a bottleneck or a single point of failure.

Dimension	Choreography	Orchestration
Control	Distributed across reactors	One central driver
Coupling	Loosest — services know only their events	Steps coupled to the orchestrator
Where the flow lives	Nowhere — emergent from reactions	In one readable place
Adding a step	Add a subscriber, touch nothing else	Edit the orchestrator
Visibility / debugging	Hard — trace events across services	Easy — read the workflow
Failure handling	Each service handles its own	Centralized retries + compensation
Best when	Few steps, loose reactions	Many steps, needs oversight

Rule of thumb: use choreography for short, loosely related reactions where decoupling is the prize; reach for an orchestrator when a process has many ordered steps, needs a clear owner for failures, or must be observable as one thing.

Event-driven systems are asynchronous by nature — a producer publishes and moves on, and consumers react whenever they get around to it — so their default consistency model is eventual consistency: after a change, the various parts of the system disagree for a short window, then converge to the same answer once every event has propagated. There is a moment when the order service says "placed" but the shipping service has not yet reacted. This is not a bug to be fixed; it is the price of decoupling, and the job is to design so that the brief disagreement is harmless.

The main tool for living with it is the read model (also called a projection): a purpose-built, read-optimized copy of data, assembled by a consumer from the events it receives, and kept only to answer queries. Instead of asking the order service for details every time, a consumer folds the events it cares about into a local shape tuned for its own reads — a shipping dashboard, a search index, a summary table.

This is the lightweight version of an idea called CQRS (Command Query Responsibility Segregation) — the practice of separating the write side (which handles changes and emits events) from the read side (one or more projections shaped for queries), so each can be modeled and scaled on its own terms. You do not need the full pattern to benefit: even a single projection built from events, kept eventually consistent, lets reads be fast and independent while writes stay simple. The catch is that a projection lags the events feeding it, so a read can briefly show yesterday's answer; you accept that lag deliberately where it is safe and fall back to a synchronous read only where a stale answer would actually hurt. Rule of thumb: treat eventual consistency as the default, build read-optimized projections from the event stream, and reserve strong, synchronous reads for the few places where staleness is genuinely unacceptable.

Event-driven architecture does not remove coupling; it moves it. Services no longer couple through direct calls, but they couple through the shape of the events — the fields, types, and meaning that producers write and consumers read. This is quieter and easier to miss: a producer can break three consumers by renaming a field, and nothing in the code will point from cause to effect. The event's structure is a contract, and it needs the same care you would give an API.

The event contract (also called the schema — the agreed structure of an event: its fields, their types, and what they mean) has to evolve without breaking the consumers already reading it. Two disciplines keep that safe:

• Additive changes only. Adding a new optional field is safe — old consumers ignore what they do not read. Renaming or removing a field, or changing a type, is a breaking change that silently corrupts consumers who expected the old shape. Evolve by adding, never by mutating in place.
• Upcasters for old events. An upcaster is a small function that transforms an event written in an old shape into the current shape as it is read, so a single up-to-date handler can process history without caring which version produced each event. This matters most under event sourcing, where old events live forever and must stay readable as the schema moves on.

A schema registry makes the contract explicit and enforceable. It is a shared service that stores every event's schema and its version, and checks each new version for backward compatibility before a producer is allowed to publish it — so an accidental breaking change is rejected at publish time instead of discovered later as corrupted consumers.

Here is a small event payload showing an additive change done safely — a new coupon field appears in v2, and every older consumer keeps working because it simply never reads it:

{
  "eventType": "OrderPlaced",
  "schemaVersion": 2,
  "orderId": "order-123",
  "items": [{ "sku": "A1", "qty": 2 }],
  "total": 40.00,
  "coupon": "SUMMER10"
}

Rule of thumb: treat every event as a versioned public contract — evolve it additively, upcast old versions on read, and register schemas so a breaking change is caught before it ships, not after it silently corrupts a consumer.

Because events travel asynchronously and delivery layers redeliver on any doubt, event-driven systems must be built to survive duplicates, partial failures, and poison messages. Four techniques carry most of that weight, and each answers a specific failure.

Idempotent consumers. An operation is idempotent when performing it twice leaves the world exactly as performing it once would — reacting to the same OrderPlaced twice must not ship two boxes or charge two cards. Since a consumer will occasionally see the same event more than once, every consumer carries a stable identifier for the event (an event id) and checks whether it has already handled that id before acting; if so, it skips the work. Idempotency is not optional here — it is the price of admission.

The outbox pattern (one-line recap). A service that must both change its database and publish an event faces a trap: two separate systems cannot commit together, so a crash between "row written" and "event published" leaves them permanently out of step. The outbox pattern avoids this by writing the event into an outbox table inside the same database transaction as the business change, so both commit atomically; a background relay then reads that table and publishes to the broker, retrying until each event lands. The point: it makes "change state and announce it" a single, crash-safe act.

Dead-letter handling. A poison event fails every time it is processed — a malformed payload, a reference to something deleted. Left alone it is retried forever, blocking everything behind it. The fix is a dead-letter queue (DLQ): after a capped number of failed attempts, the event is moved aside into a separate quarantine queue so healthy events keep flowing, and it is preserved (not dropped) so a human or a repair job can inspect it, fix the cause, and put it back. A DLQ nobody watches is a silent data-loss bucket, so alert on its depth.

Replay. Because an event log keeps its events rather than deleting them on read, you can replay — reset a consumer to an earlier position and feed it the history again. This is how you rebuild a projection after a bug, seed a brand-new consumer with the full past, or recover from a failure by reprocessing. Replay only works if consumers are idempotent, which is one more reason idempotency is foundational.

Failure introduced	Defense	What it guarantees
Same event delivered twice	Idempotent consumer + event id	Reacting twice changes the world once
Crash between DB write and publish	Outbox pattern + relay	State change and its event commit together
An event that always fails	Dead-letter queue + alert	Quarantine, inspect, unblock the flow
Bad projection / new consumer / recovery	Replay from the log	Rebuild state from stored history

Rule of thumb: make every consumer idempotent, publish through an outbox so state and event are atomic, dead-letter poison events instead of retrying forever, and rely on a retained log so you can replay to rebuild or recover.

Event-driven architecture is powerful and genuinely costly, so it earns its place only against specific needs — and reaching for it by default is over-engineering. Start by asking what the interaction actually is.

Prefer a plain synchronous call when the caller needs the result now to continue, when there is exactly one obvious handler, and when the flow is simple enough that a stack trace should tell the whole story. Asking a service for a value and waiting for it is not old-fashioned; it is the right tool when the answer is needed immediately and directness aids debugging.

Prefer a simple queue (one producer handing work to one pool of workers) when you need to decouple timing — move slow work off the request path, absorb bursts, retry failures — but the work still has a single owner. A queue buys asynchrony without the fan-out and inversion of full event-driven design.

Reach for event-driven architecture when the fact of a change genuinely has many independent interested parties, when those parties should evolve and deploy without coordinating, and when you expect new reactions to be added over time that the producer should not have to know about. The classic fit is one business fact — an order was placed — that must fan out to unrelated concerns (stock, payment, notification, analytics) that will keep multiplying.

But know the failure modes it introduces, because they are the flip side of its strengths:

• Event storms. One event triggers reactions that emit more events that trigger more reactions — a cascade that amplifies load unpredictably, sometimes in a loop. Bound fan-out and watch for cycles.
• Ordering surprises. Events can arrive out of the order they were produced; a consumer that assumes OrderPaid always precedes OrderShipped will misbehave when they swap. Depend on order only where you have guaranteed it (for example, by keying related events to the same partition).
• Duplicate reactions. Redelivery means the same event fires a reaction twice; without idempotency, that double-ships or double-charges.
• Hidden coupling through event shape. The worst failure mode is invisible: consumers quietly grow to depend on a field or a subtle meaning, and the producer, seeing only its own code, breaks them by "harmlessly" changing an event. The coupling is real but shows up nowhere in the call graph.

Rule of thumb: use a direct call when a result is needed now, a queue when timing must decouple but one owner remains, and full event-driven architecture only when many independent parties must react to a fact and keep multiplying — and go in expecting storms, reordering, duplicates, and schema coupling as the standing costs.

Let us make it concrete with an e-commerce flow and run the same business process both ways. A customer places an order, and four things must follow: inventory reserves the stock, payment charges the card, email sends a confirmation, and analytics records the sale. The fact at the center is OrderPlaced.

First, the choreography version. The order service publishes OrderPlaced as a fat, state-carrying event and walks away; each service reacts on its own, and some emit their own events that others react to in turn:

Notice the properties. The order service depends on no one; adding a loyalty-points service later means it just subscribes to OrderPlaced, and nothing upstream changes. But the end-to-end story — placed, reserved, charged, confirmed — lives nowhere as a single artifact, so if confirmations stop arriving, you trace events across four services to find where the chain broke. And if payment fails after stock is reserved, someone must react to un-reserve it (a compensating reaction), and that recovery logic is scattered across the services rather than sitting in one place.

Now the orchestration version. A single order-workflow orchestrator knows all four steps and drives them, sending commands and reacting to their outcomes:

Here the whole process is one readable thing: you can open the orchestrator and see reserve, charge, email, record, in order, with the failure path (charge failed, so release the stock) sitting right beside the happy path. The cost is that every step now depends on the orchestrator, and it must be made highly available and kept from becoming a bottleneck. Same business flow, opposite trade-off: choreography spreads the process out for maximum decoupling and minimum visibility; orchestration gathers it into one place for maximum visibility at the price of a central dependency. A common real-world answer is to mix them — orchestrate the money-critical core (reserve, charge, compensate) where oversight and failure handling matter most, and let peripheral reactions (email, analytics) simply subscribe to events and choreograph themselves.

This table collapses the day into decisions and the tool each one points to.

Question / concern	Reach for	Why
Is this a fact or an instruction?	Event (past tense) vs command (imperative)	Facts have many reactors; instructions have one owner
How much should the event carry?	Notification / state transfer / sourcing	Thin + callback, fat + no callback, or the log is truth
Consumer needs no callback to the producer	Event-carried state transfer (level 2)	Fat events + a local copy decouple fully
The full history is a first-class asset	Event sourcing (level 3)	Store events, fold to current state, replay anytime
Loose reactions, few steps	Choreography	Maximum decoupling, no central driver
Many ordered steps, needs oversight	Orchestration	One readable flow, central failure handling
Fast reads off the event stream	Projection / read model (light CQRS)	Query-shaped copy, eventually consistent
Event shape must evolve safely	Additive changes + upcasters + schema registry	Old consumers keep working; breaks caught at publish
Same event may arrive twice	Idempotent consumer + event id	Reacting twice changes the world once
Change state and announce it atomically	Outbox pattern + relay	No lost events from a two-system write
An event that always fails	Dead-letter queue + alert	Quarantine and unblock the flow
Rebuild a view or recover	Replay from the retained log	Reprocess history into a fresh projection
Result needed now, one handler	Direct synchronous call	Event-driven would only add cost

• Event-driven means parts react to events — facts stated in the past tense — instead of calling each other directly; the producer announces and does not know who reacts.
• An event is a fact with zero-to-many reactors and no expected outcome; a command is an instruction to one handler with an expected result — never disguise a command as an event.
• The three levels are notification (thin ping, consumer calls back), event-carried state transfer (fat event, no callback), and event sourcing (the log is the source of truth, current state is a fold of events); most systems want level 2.
• Choreography decentralizes the flow across reactors for loose coupling but poor visibility; orchestration centralizes it in one driver for visibility at the price of a central dependency — mixing them is common.
• Eventual consistency is the default; build read-optimized projections from the stream (light CQRS) and reserve synchronous reads for where staleness truly hurts.
• Events are contracts: evolve them additively, upcast old versions on read, and use a schema registry so breaking changes are caught at publish time, not as silent consumer corruption.
• Make every consumer idempotent, publish through an outbox so state and event commit atomically, dead-letter poison events, and keep a retained log so you can replay to rebuild or recover.
• Choose a direct call when a result is needed now, a simple queue when only timing must decouple, and full event-driven architecture only when many independent parties must react to a fact — expecting event storms, reordering, duplicates, and hidden schema coupling as the standing costs.