Managing Shared State

Shared state is any data that more than one process, service, user, or machine can read or write. The moment two things can touch the same value, you have to decide what "correct" means when they touch it at the same time. That decision is the whole subject of this day.

Everyday examples: a user session, a shopping cart, an account balance, an inventory count, a workflow status, a document being edited, the position of a job queue, a distributed lock, a rate-limit counter, a feature flag.

On a single server the story is boring. One machine holds the value in memory and serializes every access.

Spread that same value across several servers reading and writing a shared database, cache, or queue, and the boring story becomes the hard one: two servers must never make conflicting assumptions about the same value.

The core question for the rest of the day: when more than one thing can read or write the same state, how do we keep the system correct enough?

The cleanest strategy for shared state is to have less of it. Every piece of state that two writers can mutate is a place where a race, a lost update, or a duplicate side effect can hide. Before reaching for locks and clever protocols, try to remove the sharing.

Prefer designs that lean on:

• Stateless services that keep no durable data in local memory.
• Durable database state as the single source of truth.
• Immutable events (append, never overwrite).
• Idempotent operations that survive being retried.
• A single owner for each piece of state.
• Async workflows with an explicit, persisted status.

Avoid designs that rely on:

• Important state living only in one server's memory.
• Hidden side effects that other code cannot see.
• Multiple services writing the same row with no coordination.
• Cross-service distributed transactions unless they are truly unavoidable.

A good default: first make the service stateless and push durable shared state into a database or queue, then carefully define who is allowed to write each thing.

A stateless application server keeps no durable user state in its own memory. Every request carries or fetches whatever it needs from a shared store, so any server can serve any request. "Stateless" does not mean the server has no variables mid-request; it means nothing important is lost if that server dies between requests.

Why this matters: any server can handle any request, servers scale horizontally, a failed server is simply replaced, and deploys get easier because no session dies with a machine.

The classic failure is storing something like a cart in local memory. The user's next request lands on a different server and the cart has vanished. Put the cart in a shared store such as Redis or the database instead, and every server can read it.

The trade-off is honest: an external state store adds a network hop of latency. You pay that latency to buy correctness and scalability, and it is almost always worth it.

State ownership means naming exactly one component that is allowed to write each kind of state. Everyone else reads it or subscribes to its changes. A single writer removes a whole class of "who wrote this last?" bugs before they can happen.

The dangerous version has the Orders service, the Payments service, and an Admin tool all writing payment status directly. Three writers, no coordination, and the true state becomes whatever wrote last.

The safe version makes the Payments service the only writer for payment state. Other services read it or react to events it publishes. As a rule of thumb: do not let multiple services mutate the same state directly; assign ownership and coordinate through APIs or events.

A concurrency problem happens when two operations touch the same state at the same time and the interleaving produces a wrong result. The most famous is the race condition: two operations "race," and the winner depends on timing rather than on any rule you intended.

Both users read "1 available," both believe they can buy, and the store oversells. Nothing crashed; the logic was simply run twice on stale reads.

The standard defenses each appear later in this day:

• A database transaction that makes read-check-write atomic.
• A row-level lock so only one writer proceeds at a time.
• Optimistic concurrency control that detects the clash at write time.
• A reservation model that holds stock before charging.
• A single-threaded queue per key so updates to one item never overlap.

Pessimistic locking assumes conflicts are likely, so it locks the data first, does the update safely, then releases the lock. "Pessimistic" because it pays the cost of locking up front rather than gambling that no one else shows up.

In SQL, a row lock inside a transaction looks like this:

BEGIN;
SELECT * FROM inventory WHERE item_id = 123 FOR UPDATE;
-- check quantity
-- decrement quantity
COMMIT;

The FOR UPDATE clause makes any other transaction that wants that row wait until this one commits. Use it when conflicts are common, correctness is critical, and the locked section is short.

For high-value correctness such as decrementing checkout inventory, a short database transaction with row-level locking is often the right tool.

Optimistic concurrency control assumes conflicts are rare, so it does not lock anything up front. Instead it reads a version number with the data and only commits the write if that version is unchanged, detecting the clash at write time.

The write itself carries the guard in its WHERE clause:

UPDATE documents
SET content = 'new content', version = version + 1
WHERE document_id = 123 AND version = 7;

If zero rows are updated, someone changed the row after you read it, so your write is stale and you retry or return a conflict. Use this when conflicts are rare, you want high throughput, and callers can retry or merge. It fits document edits, profile updates, config changes, and admin workflows.

Rule of thumb: if conflicts are rare, use optimistic concurrency with a version field and either retry or return a conflict to the caller.

A distributed lock coordinates work across multiple machines by making them all agree that only one holder may proceed at a time. It is genuinely useful, and also one of the easiest things to get subtly wrong, so reach for it only when nothing simpler fits.

Good uses: stopping two workers from running the same scheduled job, guarding one short critical section, or avoiding duplicate expensive work. Dangerous uses: relying on a lock for financial correctness, wrapping a long-running workflow, using a lock with no timeout, or using one with no fencing token.

Why locks are risky:

• The holder can crash while holding the lock.
• The lock can expire (its TTL runs out) while the work is still running, so two workers now believe they hold it.
• A network partition makes it ambiguous who holds the lock.
• A retried operation can duplicate its side effects.

Two safety mechanisms are non-negotiable when you must use one. A TTL (time-to-live) is an expiry so a crashed holder cannot freeze the resource forever. A fencing token is a number that increases every time the lock is granted; the protected resource rejects any write carrying a token older than the highest it has seen, so a stalled old holder that wakes up after its TTL cannot corrupt state.

Rule of thumb: avoid distributed locks unless necessary; when you use one, add a TTL, a fencing token, and idempotent writes. For real correctness, prefer a database transaction or a single-writer design over a lock.

An operation is idempotent when repeating it has no additional effect after the first success. Send the same request twice and the world ends up in the same state as sending it once. This is the single most important defense against the retries that distributed systems produce constantly.

Retries are unavoidable: clients retry, networks fail mid-response, workers crash, queues redeliver, and payment APIs receive duplicate submissions. The fix is a client-generated idempotency key — a unique id the caller attaches to a request so the server can recognize a repeat.

Store enough to recognize and answer a repeat safely. A typical idempotency table holds one row per key:

idempotency_key | request_hash | status | response | created_at

The request_hash lets you detect a key reused for a different payload (a client bug) rather than silently returning the wrong response. Use idempotency for payments, order creation, email sending, file uploads, job submission, and any external API call. Rule of thumb: because retries are unavoidable, make write operations idempotent using client-generated idempotency keys.

A queue hands workers units of work, which turns "handle this now, in the request" into "record it and process it soon." It is one of the best tools for shared work state because it decouples the producer from the consumer.

Benefits: it smooths traffic spikes, moves slow work off the request path, enables retries, and lets you cap concurrency. But a queue is itself shared state, and it introduces its own hazards:

• Duplicate delivery (the same message twice).
• Out-of-order processing.
• Poison messages (a message that always fails).
• Partial failure part way through handling.
• Retry storms when many messages fail and retry at once.

The defenses are a checklist worth memorizing: idempotent workers, a dead-letter queue for messages that exhaust their retries, a retry limit, a visibility timeout (a window during which a picked-up message is hidden from other workers), a job-status table, and a deduplication key. The load-bearing assumption behind all of them: a message can be delivered more than once, so consumers must be idempotent.

"Exactly-once processing" sounds like the obvious goal, but end-to-end exactly-once delivery is extremely hard and rarely what you actually need. The queue can redeliver, and the worker can crash after acting but before acknowledging. Chasing a guarantee the infrastructure cannot give you leads to fragile designs.

The practical target instead: at-least-once delivery with idempotent consumers. Accept that a message may arrive twice, and make the effect of processing it happen once logically.

Concrete example: a message says "send the invoice email for invoice 123." Before sending, the worker checks whether invoice_email_sent is already recorded for 123. If yes, it skips; if no, it sends and records the fact in the same durable step. The redelivery is harmless because the second run finds the record and does nothing.

Leader election chooses exactly one active coordinator among many identical nodes, so that a job meant to run once does not run on every node. The other nodes stand by, ready to take over if the leader disappears.

Typical uses: one scheduler fires periodic jobs, one coordinator assigns partitions, or one node performs cluster maintenance. Common building blocks are ZooKeeper, etcd, Consul, Kubernetes leases, and — for simpler systems — database advisory locks.

Rule of thumb: if only one instance should coordinate work, use leader election or a managed scheduler rather than a local cron on every server, which would fire the same job N times.

User state — who is logged in, what they can do — has to survive across requests that may land on different servers. There are three standard ways to hold it, trading simplicity, scale, and revocation against each other.

Option A — server-local session. State lives in one server's memory. Simple and fast, but hard to scale, lost when that server dies, and it forces sticky routing that complicates load balancing.

Option B — shared session store. Any server reads the session from Redis or a session database. Scales horizontally and survives server failure, at the cost of an external dependency that must stay reliable.

Option C — stateless signed token. The client carries a cryptographically signed token (such as a JWT) that any server can verify without a lookup. Scales beautifully, but revocation is harder, token expiry and rotation matter, and encoded permissions can go stale.

Rule of thumb: for simple auth, stateless tokens scale well; when revocation or dynamic permissions matter, combine tokens with a server-side session or a live permission check.

A counter looks trivial and turns out to be one of the trickiest shared values at scale, because every increment is a write to the same spot. The right design depends entirely on how correct the count must be.

Match the mechanism to the requirement:

For very high write rates, a sharded counter splits one hot value into many sub-counters that are summed on read, spreading the write load. Rule of thumb: choose the counter design by correctness — views can be approximate, inventory cannot.

A cache is a second copy of data kept close to the reader for speed. Because it is a copy, it can disagree with the source of truth, and that disagreement is the shared-state risk you sign up for the moment you add a cache.

The common pattern is cache-aside: read from the cache, fall back to the database on a miss, and invalidate the cache on write.

The hazards: stale entries, a cache stampede (many clients rebuild the same expired key at once), a lost invalidation, and hot keys hammered by traffic. Defenses include TTLs, explicit invalidation on write, request coalescing (let one rebuild feed many waiters), versioned cache keys, and background refresh. One hard line: never let a cache become the authority for security or authorization decisions.

Rule of thumb: treat the cache as a performance optimization, not a correctness layer — the database stays the source of truth.

A long-running operation — one that spans several steps, external calls, or minutes of wall-clock time — should keep its progress in explicit, persisted state rather than in a variable that dies with the process. If the only record of progress is "the code is currently on line 40," a crash loses everything.

The weak design starts the job and hopes every step finishes. The strong design writes a status the whole system can see:

pending
processing
waiting_for_external_callback
completed
failed
retrying

Persist that status in a job table that the workers update as they go:

Why it pays off: users can check progress, workers can retry safely from a known point, failures are visible instead of silent, and operations can resume after a crash. Rule of thumb: for long-running operations, persist workflow state in a job table rather than keeping it in memory.

A saga runs a multi-step workflow across several services when a single database transaction cannot span them all. Instead of one all-or-nothing commit, each step commits locally, and if a later step fails, earlier steps are undone by explicit compensating actions (an operation that reverses a completed step).

A checkout saga might be: create order, reserve inventory, charge payment, confirm order, send email. If the payment step fails, the compensating action releases the reserved inventory so nothing is left dangling.

The trade-offs: a saga avoids brittle distributed transactions and makes failure handling explicit, but it demands a carefully designed state machine and it tolerates temporary inconsistency between steps. Rule of thumb: for cross-service workflows, avoid distributed transactions and use a saga with explicit compensating actions.

The single-writer pattern routes every write for a given entity through one owner, so writes to that entity never overlap. If all updates for account 123 go to the same worker, they are applied one at a time, and most race conditions simply cannot occur.

Common forms: one actor per document, one partition owner per account, one service that owns order state, or one queue partition per user id. The key is a partitioning rule that always sends the same entity to the same owner.

Rule of thumb: if ordering matters per entity, route all updates for the same entity to the same partition or owner.

Designing shared state means naming the ways it can break before it breaks, then attaching a defense to each. The failures below are not exotic; they happen in normal operation, and a good design has an answer ready for every one.

The paired defenses form the toolkit for the whole day: idempotency, transactions, retry with backoff, dead-letter queues, version fields, state machines, the outbox pattern, and monitoring plus reconciliation jobs that catch what slips through.

The outbox pattern solves one specific, common failure: you write a row to the database and then try to publish an event about it, but the publish fails, so other services never learn the change happened. The two actions — write and publish — are not atomic, and the gap between them loses events.

The trick: inside the same database transaction that writes the business row, write the event into an outbox table. Since both live in one transaction, they commit together or not at all. A separate background worker then reads unpublished outbox rows, publishes them to the broker, and marks them published — retrying safely until each one lands.

Rule of thumb: to reliably publish an event after a database write, use the outbox pattern so no event is lost in the gap between the database and the broker.

When you face a new piece of shared state, work through it in a fixed order rather than jumping to a mechanism. The mechanism is the last decision, not the first.

Worked reasoning, checkout inventory: the critical state is the stock count, which many users can try to buy at once. Overselling is unacceptable, so keep inventory in the database and change it inside a transaction or a reservation flow. Checkout requests should carry an idempotency key because clients retry. Non-critical state such as search indexing or analytics can be async and eventually consistent. Notice the pattern: identify the state, decide how correct it must be, then pick ownership and concurrency control to match.

A few worked examples show how the requirement drives the design:

• Shopping cart (medium correctness): keep the cart in Redis or the database, keep inventory in the database with a transaction or reservation, and give checkout an idempotency key. The cart can be eventually persisted; inventory cannot oversell.
• Payment creation (high correctness): client idempotency key → API → payments database → external provider, guarded by a unique payment record, a retry-safe provider call, webhook deduplication, and a payment state machine.
• Collaborative document editing (high correctness, but the conflict model matters): client edits flow through a collaboration service into an operation log. A plain version check is often too coarse, so choose an explicit conflict-resolution model. OT (operational transform — rewrite each edit against concurrent edits so they compose) and CRDTs (conflict-free replicated data types — data structures that merge automatically without a coordinator) are the two standard answers, picked by product need.
• Scheduled job (correctness depends): do not run cron on every app server, or the job fires N times. Use a managed scheduler feeding a queue of workers, or elect a leader so only the leader enqueues the job.

This table collapses the day into problem-to-solution pairs. Each solution buys correctness at some cost, named in the last column.

The one-paragraph summary to keep in your head: make application servers stateless, keep durable state in the database, define a single owner for each state transition, and use idempotency for write APIs. For conflicts, choose transactions or locks for critical state and optimistic concurrency when conflicts are rare. For async work, assume duplicate delivery and make consumers idempotent.

• Shared state is any data more than one process can read or write; the hard part is what "correct" means when they do it at once.
• The first move is to reduce shared mutable state: stateless servers, durable DB state, immutable events, single owners.
• Give each piece of state one writer; let everyone else read or subscribe.
• For conflicts, pick pessimistic locking when they are common and optimistic concurrency when they are rare.
• Retries are unavoidable, so make writes idempotent with client-generated keys; assume at-least-once delivery, not exactly-once.
• Distributed locks need a TTL and a fencing token, and are still weaker than a transaction or single-writer design.
• Long-running work needs explicit, persisted status; cross-service work needs sagas with compensations.
• Publish events reliably with the outbox pattern, treat caches as speed not truth, and plan a defense for every failure mode.