Caching, In Depth

A cache is a small, fast store that holds copies of values you have already computed or fetched, so the next request for the same value skips the slow work. The slow work is usually a trip to the source of truth — the authoritative store that owns the real data, typically a database. Caching exists for two reasons: to cut latency (how long one request waits for its answer) and to reduce load on that source of truth so it survives more traffic.

The economics are simple. Reading a value from memory takes microseconds; reading it from a database across the network takes milliseconds, sometimes tens of milliseconds under load. When the same value is read far more often than it changes, most of those database trips are pure waste. A cache turns repeated expensive reads into cheap ones, and in doing so it shields the database from traffic it would otherwise buckle under. The rule of thumb: cache data that is read much more often than it is written, and whose slight staleness you can tolerate.

A cache is not one thing in one place. The same value can be cached at several layers — points along the path a request travels from the user's device to the database — and each layer catches a different share of traffic. The closer a cache sits to the user, the faster the response and the more load it removes from everything behind it, but the less control you have over what is stored and when it clears.

Each layer trades reach against control. A browser cache lives on the user's own device, so a hit costs zero network — but you cannot reach in to clear it, and it only helps that one user. A CDN (Content Delivery Network — a fleet of servers spread across the globe so content is served from a location near each user) caches at the edge, the server geographically closest to the requester; it absorbs enormous read traffic for shared content but is awkward for anything user-specific. A reverse proxy (a server that sits in front of your application and forwards requests to it, such as Nginx or Varnish) can cache responses for all users of one deployment. An in-process cache lives in the application's own memory — the fastest server-side option, but each server has its own copy, so they drift apart. A distributed cache (a separate memory service like Redis or Memcached that all your servers share over the network) gives every server one consistent view at the cost of a network hop.

The layers are cumulative, not competing. A well-built system uses several at once, letting each catch what it is best placed to catch before the request falls through to the next.

A read strategy answers one question: when a value is missing from the cache, who is responsible for fetching it from the source of truth and putting it there? There are two common answers, and the difference is about which piece of code owns that responsibility.

In cache-aside (also called lazy loading), the application is in charge. It checks the cache first; on a hit (the value is present) it returns immediately, and on a miss (the value is absent) it reads the database itself, writes the result back into the cache, and returns it. The cache is just a dumb box the application talks to on both sides. The flow in words:

Check cache for the key.
On hit, return the cached value.
On miss, read the value from the database.
Write that value into the cache with a TTL.
Return the value.

In read-through, the cache itself is in charge. The application only ever asks the cache; when the cache does not have the value, the cache (through a loader you configured) fetches it from the database, stores it, and hands it back. The application never talks to the database for reads at all.

Cache-aside is the most common default because it is simple and degrades gracefully: if the cache disappears, reads still work, just slower. Reach for read-through when you want the loading logic centralized and you are comfortable making the cache a hard dependency of every read.

A write strategy decides what happens to the cache when data changes. The tension is always the same: the database is the source of truth and must be updated, but the cache holds a copy that will now be wrong unless you do something about it. The three common answers differ in when the database write happens and whether the new value lands in the cache at all.

In write-through, every write updates the cache and the database together, synchronously, before the write is acknowledged. Reads that follow immediately see the fresh value, and the cache is never behind the database. The cost is that every write now pays for two updates instead of one, so writes are slower.

In write-behind (also called write-back), the write updates the cache immediately and returns, while the database update is queued and flushed a moment later in the background. Writes are fast and bursts are absorbed, because many cache updates can be batched into fewer database writes. The danger is durability: if the cache node dies before the flush, those writes are lost, and the database is briefly behind the cache.

In write-around, the write goes straight to the database and skips the cache entirely — the matching cache entry is simply invalidated or left to expire. Nothing pollutes the cache with data that may never be read again. The trade-off is that the very next read of that key is guaranteed to miss and hit the database.

These are not mutually exclusive with the read strategies — a common pairing is cache-aside for reads plus write-around (invalidate on write) for writes, which keeps the cache lazily populated and never stale for long.

A cache has finite memory, so it cannot keep everything. When it fills up and a new value needs room, an eviction policy decides which existing entry to throw out. Separately, entries can also leave the cache on their own when their TTL (time-to-live — an expiry set on an entry, after which it is treated as absent) elapses. Picking the right policy is about matching the eviction rule to how your data is actually accessed.

LRU (Least Recently Used) evicts the entry that has gone unused the longest. It assumes that something touched recently will be touched again soon — a good bet for most workloads, which is why LRU is the usual default. LFU (Least Frequently Used) evicts the entry with the fewest accesses over time; it protects genuinely popular keys better than LRU, but it can cling to items that were hot long ago and are now cold. FIFO (First In, First Out) evicts in insertion order regardless of use — simple and cheap, but it will happily throw out a hot key just because it is old. TTL expiry is not really an eviction contest at all: each entry carries its own expiry and disappears when its time is up, independent of memory pressure. Random eviction just picks any entry; it sounds crude but is extremely cheap and, surprisingly, performs close to LRU on many workloads because it avoids the bookkeeping cost.

In practice these combine: most caches run TTL expiry (so nothing is ever unboundedly stale) alongside an eviction policy like LRU (so the cache stays within its memory budget). Start with LRU plus a TTL unless you have measured a reason to do otherwise.

There is an old joke that there are only two hard things in computer science: cache invalidation, naming things, and off-by-one errors. The joke lands because invalidation — deciding when a cached copy has gone wrong and removing or refreshing it — genuinely is one of the hardest parts of caching. The cache holds a copy of data that lives elsewhere; the moment that data changes, the copy is a lie, and you have to catch every such lie without clearing so aggressively that the cache stops helping.

There are four broad approaches, and real systems mix them. With TTL-based invalidation you do nothing on write and simply let each entry expire on its own schedule; it is the simplest approach and requires no coordination, but it accepts staleness up to the length of the TTL. With explicit invalidation the code that writes to the database also deletes the matching cache key, so the next read misses and reloads fresh data; it is precise but requires you to remember every place a write can happen. With versioned or namespaced keys, you never overwrite or delete — instead you change the key. A user's profile might be cached under user:42:v7, and bumping a version counter to v8 makes every old entry instantly unreachable and free to be evicted later; this sidesteps the delete-everywhere problem but needs a version to track. With write-time update (the write-through pattern from earlier), the write pushes the new value straight into the cache so it is never stale in the first place.

The honest summary: TTL buys simplicity at the price of staleness, explicit invalidation buys freshness at the price of discipline, and versioned keys buy an atomic cutover at the price of a version to manage. Most systems lean on TTL as a safety net and layer explicit invalidation on the paths where staleness would hurt.

Because a cache is a copy, it can disagree with the source of truth for a while. A stale read is a read that returns an old value because the cache has not yet caught up with a change in the database. Usually a brief window of staleness is fine — that tolerance is exactly what makes the data cacheable. The case that bites is read-after-write consistency: the expectation that a user who just made a change immediately sees their own change reflected back.

The sequence above is the classic trap. The user renames themselves, the database is updated, but the cache still holds the pre-change copy, so the very next page load shows the old name — and the user concludes the save failed. The fix is to make the write path touch the cache, not just the database: either delete the cached key on write (explicit invalidation) so the reload misses and reloads fresh, or overwrite the cached value on write (write-through) so the reload sees the new value directly. A softer option is to let the user's own client hold the value it just wrote and show that, while background caches catch up. The principle: whenever a cache sits in front of data a user can edit, the write must invalidate or update the cache, or that user will read their own change stale.

Caches introduce their own failure modes, and the most infamous is the stampede (also called a thundering herd). It happens when a popular key expires or is missing and, in the same instant, many concurrent requests all miss the cache and all rush to the database to recompute the same value at once. The database, which the cache existed to protect, is suddenly hit by the full crowd — sometimes hard enough to fall over, which makes every request slower, which makes the pile-up worse.

The mitigations all aim to stop the crowd from doing the same work in parallel. Request coalescing — also called single-flight, meaning only one in-flight computation per key is allowed — lets the first miss compute the value while the others wait for its result instead of computing their own. A per-key lock is the mechanism that enforces this: the first request grabs the lock, recomputes, populates the cache, and releases; the rest find the fresh value on their retry. Jittered TTLs add a small random offset to each entry's expiry so that keys written together do not all expire at the same instant. Early or background recompute refreshes a hot entry slightly before it expires, on a background task, so requests never actually find it missing.

The rule of thumb: for any expensive, hot value, coalesce concurrent misses into a single recompute and jitter the expiry so the whole cache does not refresh in lockstep.

A hot key is a single cache entry that receives a wildly disproportionate share of traffic — a celebrity's profile, a viral post, a global feature flag read on every request. Even a distributed cache stores each key on one node, so a hot key funnels all its traffic onto that one node, which can saturate its CPU or network while the rest of the cluster sits idle.

Three mitigations help. Putting a small local in-process cache in front of the distributed cache means each server answers most reads of the hot key from its own memory, so only a trickle reaches the shared node. Key replication stores copies of the hot key on several nodes and spreads reads across them, dividing the load. Splitting the key breaks one hot entry into several sub-keys (for example, sharding a global counter into ten counters that are summed on read) so the traffic lands on multiple nodes. The principle: detect the hot key first through metrics, then fan its read traffic across more memory — local, replicated, or split — so no single node carries it alone.

Cache penetration happens when requests repeatedly ask for a key that does not exist in the source of truth at all. Because there is nothing to cache, every such request misses the cache and then misses the database too, so the cache provides no protection — and a malicious client can exploit this by hammering random non-existent keys to drive load straight through to the database.

The standard defense is to cache the negative result: when the database returns nothing for a key, store a small placeholder (a null or empty marker) in the cache with a short TTL, so repeat requests for that missing key are answered by the cache instead of reaching the database. The short TTL keeps the negative entry from lingering if the key later comes into existence. For very large key spaces, a compact membership filter in front of the cache can reject known-absent keys outright before any lookup. The rule: cache misses that are legitimately empty, not just hits, so that "this does not exist" is itself a cheap cached answer.

A cache avalanche is a stampede at scale: instead of one hot key expiring, a large batch of entries all expire at nearly the same moment, so a flood of misses hits the database all at once. It commonly happens after a cache restart (everything reloads with the same TTL and therefore expires together) or when many keys were written in the same warm-up burst with an identical expiry.

The main defense is the same jitter used against stampedes, applied broadly: give each entry a TTL with a random spread (say, a base of ten minutes plus or minus a minute) so expirations scatter across time instead of clustering on one instant. Staggering the warm-up — populating the cache gradually rather than all at once — has the same effect. Combined with request coalescing so that whatever does expire together still only recomputes once per key, jittered TTLs turn a potential flood into a manageable trickle.

A CDN caches your content on edge servers distributed around the world, so a user in one region is served from a nearby machine instead of your origin on another continent. This is the layer that makes static and shared content — images, scripts, stylesheets, videos, and cacheable API responses — fast globally while removing that traffic from your own servers entirely. What the edge caches, and for how long, is controlled almost entirely through HTTP headers.

The key header is Cache-Control, which the origin sends alongside a response to tell the edge (and the browser) how to cache it. A directive like Cache-Control: public, max-age=3600 says "anyone may cache this for one hour"; no-store says "never cache this"; and s-maxage sets a separate lifetime specifically for shared caches like the CDN. Because edge-cached content can outlive a change at the origin, CDNs offer a purge (also called invalidation) operation to force the edge to drop a cached item before its TTL — but purges propagate across the global fleet with some lag, so they are a correction, not an instant switch. A common pattern avoids purges altogether by putting a content hash in the filename (app.9f2c1.js): the URL changes whenever the content changes, so a new version is simply a new, uncached URL and the old one can expire on its own. The rule of thumb: cache aggressively at the edge with long lifetimes for content that is either immutable or safe to serve slightly stale, and use versioned URLs so you rarely need to purge.

A cache you do not measure is a cache you cannot trust, because its whole value is statistical — it only helps to the degree that it actually catches traffic. A handful of metrics tell you whether it is earning its place and warn you before a failure mode bites.

The headline number is the hit rate — the fraction of lookups answered from the cache — with its mirror the miss rate (the fraction that fell through to the source of truth). A high hit rate means the cache is doing its job; a falling one means your keys, TTLs, or size need attention. p99 latency (the 99th-percentile response time — the value that only the slowest one percent of requests exceed) matters more than the average, because caches are meant to tame the slow tail, and a bad p99 often reveals stampedes or hot keys that the average hides. The eviction count — how many entries the cache pushed out to make room — tells you whether the cache is too small: a high, steady eviction rate means entries are being thrown away before they can be reused, which quietly drags the hit rate down.

Pulling the layers together, a sound default stacks caches so that each one catches what it is best placed to catch, and only genuine misses fall through to the next. A request should have to travel the full depth to the database only rarely.

The browser handles repeat visits for one user at zero network cost. The CDN edge absorbs shared static and global content near each user. The application's in-process cache answers the very hottest keys straight from local memory and shields the shared cache from hot-key traffic. The distributed cache gives every server one consistent view of warm data and takes the bulk of read load off the database. Finally the database, the source of truth, is reached only for true misses and for writes. Each layer removes load from everything behind it, so the database — the most expensive and least scalable component — sees the smallest share of traffic. Do not add all four at once; add each layer when a measured bottleneck justifies it, exactly as you would any other scaling move.

A one-glance summary of the choices in this chapter.

• Cache to cut latency and to shield the source of truth; cache what is read far more than it is written and can tolerate brief staleness.
• Caches live at many layers — browser, CDN edge, reverse proxy, in-process, distributed — and the layers are cumulative, each removing load from the one behind it.
• Cache-aside (the app fetches on a miss) is the simple default read strategy; read-through moves that logic into the cache.
• Choose a write strategy by when the database update happens: write-through for immediate freshness, write-behind for write throughput, write-around when data is rarely re-read soon.
• Run LRU plus a TTL by default; other eviction policies are answers to specific access patterns.
• Invalidation is the hard part: TTL buys simplicity, explicit deletes buy precision, versioned keys buy atomic cutovers — most systems use TTL as a net plus explicit invalidation where it counts.
• Any cache in front of user-editable data must be invalidated or updated on write, or users read their own changes stale.
• Design for the four failure modes from the start: coalesce and jitter against stampedes and avalanches, front hot keys with local memory, and cache negative results against penetration.
• Measure hit rate, miss rate, p99 latency, and evictions — a cache is only worth what its numbers prove.