Distributed System Patterns I: Resilience

A distributed system is a set of services running on separate machines, talking over a network. The moment you split work across machines, new failure modes appear that a single program never faces: a call can be slow, a dependency can be down, a burst of traffic can arrive all at once. Resilience patterns are small, reusable answers to these failures — each one names a specific problem and a specific move that contains it.

The through-line for the whole day: a failure in one component must not become a failure of the whole system. Left unmanaged, a single slow dependency can tie up every thread, exhaust every connection, and take down services that had nothing to do with the original fault. This is a cascading failure — a local problem spreading outward until the system collapses. Every pattern below is a wall built to stop that spread.

Each pattern is presented the same way: the problem it solves, the pattern itself, and a one-line "when to use." Read them as a toolbox, not a checklist — you reach for the one that matches the pressure you can actually name.

An ambassador is a helper process that sits next to your service and handles the messy details of talking to the outside world on its behalf. Your application code makes a plain local call; the ambassador adds retries, timeouts, TLS (encrypted connections), service discovery (finding where the remote service lives), and metrics before forwarding the request.

• Problem. Every service reimplements the same cross-cutting client concerns — retry logic, connection encryption, monitoring — and they drift apart, each slightly wrong in its own way.
• Pattern. Move those concerns out of the application and into a co-located proxy. The app calls localhost; the ambassador owns the network complexity.
• When to use. You have many services, in many languages, that all need the same outbound-call behavior and you do not want to rewrite it in each.

app code  ->  ambassador (localhost)  ->  remote service
              retry / TLS / discovery / metrics

A sidecar is the general idea behind the ambassador: a companion container or process deployed alongside your main service, sharing its lifecycle, that provides supporting features without changing the service's code. The ambassador is one kind of sidecar (an outbound proxy); other sidecars handle logging, configuration, or security.

• Problem. You want to add capabilities — log shipping, a config watcher, a proxy — to a service without editing and redeploying the service itself.
• Pattern. Package the capability as a separate process that runs next to the service and shares its network and disk. The main service stays focused on business logic.
• When to use. A capability is orthogonal to the business logic and useful to attach uniformly across many services — the model a service mesh (a dedicated layer that manages service-to-service traffic) is built on.

A circuit breaker is a wrapper around a call to a remote dependency that watches for failures and, once they cross a threshold, stops making the call for a while — failing fast instead of waiting on something that is clearly broken. The name comes from household electrical breakers that trip to prevent a fire.

• Problem. A dependency is failing or slow. Every caller keeps trying, each call blocks a thread until it times out, and the pile-up of stuck threads drags the caller down too — the classic cascade.
• Pattern. Track recent failures. While the dependency looks healthy the breaker is closed and calls pass through. When failures cross a threshold the breaker trips open and calls fail immediately without touching the dependency. After a cooldown it goes half-open and lets a few trial calls through: if they succeed it closes again, if they fail it re-opens.
• When to use. Any synchronous call to a dependency that can fail or hang, where you would rather return a fast error (or a fallback) than block.

The three states and the transitions between them are the heart of the pattern:

Read the state machine as a self-healing loop: closed is normal, open is protection, half-open is the careful probe that decides whether the dependency has recovered. Opening the circuit does double duty — it protects the caller from stuck threads and gives the failing dependency room to recover instead of being hammered while it is down.

A retry simply tries a failed call again, on the assumption that some failures are transient — a dropped packet, a momentary blip. But naive retries are dangerous, so the pattern comes with two modifiers.

• Problem. Retrying immediately and in lockstep turns a small hiccup into a stampede: every client retries at the same instant, and the synchronized wave (a thundering herd) overwhelms a dependency that was just starting to recover.
• Pattern. Exponential backoff waits longer after each failed attempt — 100ms, then 200ms, 400ms, 800ms — so a struggling dependency gets increasing breathing room. Jitter adds a small random offset to each wait so clients do not all retry at the same moment, spreading the load out. Cap the number of attempts, and only retry operations that are safe to repeat (see idempotency below).
• When to use. Transient, retryable failures on operations that are idempotent — safe to run more than once without changing the result, so a duplicate does no harm.

# attempt with capped exponential backoff + jitter
delay = min(base * 2 ** attempt, max_delay)
sleep(delay + random(0, jitter))

The pairing matters: backoff alone still lets synchronized clients retry together; jitter alone does not give a dependency increasing recovery time. Together they spread retries across both time and clients.

A timeout is an upper bound on how long you will wait for a call before giving up and treating it as failed. Without one, a call to a hung dependency waits forever.

• Problem. A remote call has no built-in deadline. If the dependency hangs, the calling thread and its resources are held indefinitely, and enough held threads exhaust the caller.
• Pattern. Set an explicit deadline on every network call. When it elapses, abandon the call and return an error (or a fallback). Timeouts are what make circuit breakers and retries possible — a failure has to be detectable before it can be counted.
• When to use. Every synchronous network call, without exception. A call with no timeout is a latent hang.

A bulkhead takes its name from a ship's hull, which is divided into sealed compartments so that a breach in one does not flood the whole vessel. In software, a bulkhead isolates resources — thread pools, connection pools — so that a failure confined to one pool cannot drain the resources the rest of the system depends on.

• Problem. All callers share one pool of threads or connections. One slow dependency consumes the entire pool, and now every unrelated request starves too, even requests that never touch the slow dependency.
• Pattern. Give each dependency (or each class of work) its own isolated pool with its own limit. When one pool is exhausted, only the work that uses it is affected; everything else keeps flowing.
• When to use. One service calls several dependencies of differing reliability, and you cannot let the flakiest one starve the others.

The picture makes the isolation concrete: dependency B is saturated and its pool is full, but pool A is untouched, so requests to A keep succeeding. The failure is sealed in its compartment.

Rate limiting caps how many requests a client (or the system as a whole) may make in a window of time. Throttling is the act of slowing or rejecting requests once that cap is reached. The two words describe the same defense from opposite ends: the limit is the rule, throttling is its enforcement.

• Problem. A single client, a buggy loop, or a traffic spike sends far more requests than the system can serve, degrading service for everyone.
• Pattern. Count requests per client per window and reject or delay anything over the limit — commonly with a token bucket, where each request spends a token and tokens refill at a fixed rate, allowing short bursts but bounding the sustained rate. Return a clear signal (HTTP 429, "too many requests") so well-behaved clients back off.
• When to use. Any public entry point, or any shared resource that must be protected from a single greedy caller or a sudden surge.

A health check is an endpoint a service exposes that reports whether it is able to do its job. A heartbeat is a periodic signal a service emits to say "I am still alive." Together they let the rest of the system know which instances are usable right now.

• Problem. A load balancer or orchestrator keeps sending traffic to an instance that has crashed, hung, or lost its own dependencies, and users hit the dead instance.
• Pattern. Each instance answers a health-check probe (often GET /health) with healthy or unhealthy; a good check verifies real readiness, such as database connectivity, not just that the process is running. Missed heartbeats mark an instance dead so it is removed from rotation and, where possible, restarted.
• When to use. Any pool of instances behind a load balancer or managed by an orchestrator — which is to say, essentially every production service.

A load balancer (LB) sits in front of a pool of identical service instances and spreads incoming requests across them, so no single instance is overwhelmed while others sit idle. It is both a scaling tool and a resilience tool: combined with health checks, it routes around instances that have failed.

• Problem. One instance cannot handle the traffic, and if it dies every request in flight dies with it. You need many instances to look like one reliable endpoint.
• Pattern. Place an LB in front of the pool. It distributes requests — round-robin (each instance in turn), least-connections (whoever is least busy), or by a hash of the client — and, using health checks, sends traffic only to instances currently reporting healthy.
• When to use. Any time you run more than one instance of a stateless service, which is the default for anything that must scale or stay available.

No single pattern is a full defense; they compose, each covering a gap the others leave open. A typical outbound call is wrapped in several at once, and it helps to see the order in which they act.

A single call to a dependency commonly stacks them: a timeout bounds the wait, a retry with backoff handles a transient blip, a circuit breaker trips if failures persist, and a bulkhead ensures that even a full-blown outage of this dependency cannot drain the threads other work needs. Layered this way, a local fault stays local — which is the entire point.

• Resilience patterns each name one failure and one containment move; the shared goal is to stop a local fault from cascading into a system-wide one.
• Ambassador and sidecar move cross-cutting concerns out of application code into a co-located companion process.
• Circuit breakers fail fast through a closed → open → half-open cycle, protecting the caller and giving the dependency room to recover.
• Retries need exponential backoff and jitter, and only apply to idempotent operations; timeouts belong on every network call.
• Bulkheads isolate resource pools so one saturated dependency cannot starve the rest; rate limiting caps load at the entry point.
• Health checks and load balancing turn a pool of instances into one endpoint that routes around failure.
• The patterns compose — a real outbound call wears timeout, retry, circuit breaker, and bulkhead at the same time.

• [ ] Can you explain the closed/open/half-open circuit-breaker states and each transition?
• [ ] Do you know why retries need both exponential backoff and jitter, not just one?
• [ ] Can you state which operations are safe to retry, and why idempotency is the test?
• [ ] Can you describe how a bulkhead isolates one failing dependency from the rest?
• [ ] Do you know the difference between rate limiting (the rule) and throttling (the enforcement)?
• [ ] Can you name what a health check should actually verify beyond "process is running"?
• [ ] Can you list which resilience patterns wrap a single outbound call and in what order they act?