Caching Levels — The Six-Layer Hierarchy from CPU to CDN

Every real system has six cache layers sitting between a user request and a disk. CPU L1/L2/L3 caches run at nanosecond speed and are invisible to your code. The OS page cache holds your hot files in RAM automatically. Your app-process cache (Caffeine, a Python dict) keeps computed results in memory for milliseconds. Redis or Memcached stores shared state across every server. The database buffer pool hides most disk reads. CDN edge nodes serve the whole page before it ever reaches your data center. A request that hits every layer warm costs 20 ms. A request that misses every layer costs 600 ms. That 30× gap is decided entirely by how well you understand — and tune — these six layers.

Caching is one of those topics where you can learn each technique in isolation — Redis for session storage, CDN for static files, Caffeine for in-process lookups — without ever seeing the full picture. The full picture is this: all six layers are active simultaneously for every production request, and the difference between a well-tuned stack and a poorly-tuned one is measured in hundreds of milliseconds and in database servers you don't have to buy.

The Cold-Cache Horror Story: A Checkout Flow in Trouble

Picture an e-commerce checkout page. A user clicks "Buy Now" and their browser fires a request. Here is what happens on a system where nobody has thought carefully about caching layers — every cache is cold or misconfigured:

1. Browser cache miss — the CDN URL has no Cache-Control header, so the browser always re-fetches. +0 ms saved.
2. CDN miss — the checkout HTML is marked Cache-Control: no-store out of excess caution. Every request hits the origin. +30 ms network to origin.
3. Nginx reverse proxy — no proxy-side cache configured. Request passes through. +2 ms routing.
4. Redis miss — the user's cart, session, and pricing data are not in Redis (TTL expired). +1 ms network to Redis, then fall through to DB.
5. Database query — five SQL joins to assemble cart, shipping options, and product details. The query itself takes 60 ms on the primary. +60 ms.
6. Response assembly + serialization — another 15 ms server-side. +15 ms.
7. Response travels back — another 30 ms network to browser. +30 ms.

Total: roughly 138 ms per checkout page load — before accounting for the three round trips a browser needs for TLS and TCP. At p99 (with DB query variability), this regularly spikes to 800 ms. Their load tests show they need eight database replicas to handle Black Friday traffic.

The Warm-Cache Story: Same Team, Three Months Later

The team audits each cache layer and makes targeted fixes. Here is the same request after the optimization:

1. Browser cache hit — static assets (JS bundles, CSS, images) have content-hash URLs and a one-year Cache-Control: immutable header. Browser serves them without a network request. ~0 ms for assets.
2. CDN edge hit — the shared parts of the checkout HTML (header, footer, product thumbnails) are now cached at the CDN with a 60-second TTL and surrogate-key invalidation on price changes. Served from the CDN edge in 8 ms. +8 ms for the shell.
3. Redis hit — user cart, pricing rules, and shipping options are cached in Redis with a 5-minute TTL. Redis answers in 0.6 ms. +0.6 ms.
4. Database buffer pool hit — the small remaining DB read (user's saved address) lands in Postgres shared_buffers and returns in 0.3 ms — no disk touch. +0.3 ms.
5. Response assembly + network back — 5 ms + 8 ms. +13 ms.

Total: roughly 22 ms per checkout page load. p99 is under 80 ms. And because the DB now sees only the cache misses (roughly 2% of requests after layered caching), the load test reveals they can handle Black Friday on two replicas instead of eight — a difference worth tens of thousands of dollars per month in infrastructure cost.

The waterfall makes the contrast visceral. In the cold path, the majority of wall-clock time is spent in two places: the CDN round-trip (because caching was disabled) and the database disk I/O (because five joins bypassed every cache). In the warm path, those two expensive phases don't happen. The request returns from shallow layers — CDN and Redis — and the system is 6× faster with 4× fewer database servers needed. The six-layer hierarchy isn't academic. It is the architectural blueprint for every latency improvement your system will ever achieve.

Before diving into each layer individually, you need one picture that holds the whole hierarchy in your head. Here it is: imagine a pyramid. At the very top — the smallest, fastest, most expensive tier — sit your CPU caches. At the very bottom — the largest, slowest, cheapest tier — sits the CDN serving billions of users globally. A request starts at the top of the pyramid for the layer closest to the CPU. If it misses, it falls to the next layer. And the next. Each fall costs progressively more time.

The pyramid shape is not arbitrary — it reflects a hard physical reality. As you move from top to bottom: capacity grows (KB → GB → TB → PB), latency grows (1 ns → 50 ms), and cost per byte drops (SRAM at $10/MB → DRAM at $0.005/MB → SSD at $0.0001/MB → CDN storage at fractions of a cent). You cannot get all three wins at once. Speed costs space. Space costs money. This triangle defines where each layer sits.

Reading the pyramid: each layer you descend pays a latency penalty and gains capacity. CPU L1 is baked onto the processor die itself — it is the single fastest memory humans have ever built, but it holds only 32–64 KB per core. By the time you reach the CDN edge, you are talking petabytes across thousands of servers worldwide, but each cache hit takes 5–50 ms depending on how far the user is from the nearest PoP. The pyramid is not a hierarchy of importance — all six layers run simultaneously in every production request. It is a hierarchy of speed and cost.

The most counterintuitive thing about this pyramid is who controls each layer. Layers 1–2 (CPU caches, OS page cache) are controlled entirely by the hardware and kernel — your application code cannot directly address them. Layer 3 (app-process cache) you control explicitly in your application code. Layer 4 (distributed cache) is a separate service you operate and configure. Layer 5 (DB buffer pool) is controlled by your database configuration — a single parameter like Postgres's shared_buffers or MySQL's innodb_buffer_pool_size can double or halve the number of disk reads your database generates. Layer 6 (CDN) is controlled by HTTP headers your application sends. Each layer has a different control knob, and knowing which knob to turn for which problem is the core skill this page builds.

Caching has its own jargon, and a lot of the terms look similar but mean subtly different things. We'll warm up with the plain-English idea first, then introduce the name — so when you bump into these in docs, code reviews, or interviews, you already have a mental hook for what each one means and why anyone cares.

The Fundamentals

When your program asks for a piece of data and it is already sitting in the cache, ready to go — that is called a cache hit. When the cache doesn't have it and has to go fetch it from the layer below — that is a cache miss.

The fraction of requests that find data in the cache is the hit ratio. This is the single most important number in cache performance. A 90% hit ratio means only 10% of traffic reaches the next layer; a 50% hit ratio means half of all traffic falls through — very different outcomes. The flip side is the miss penalty — the extra time you pay every time the cache doesn't deliver.

The data your system accesses most often right now is called the working set. If your cache is large enough to hold the entire working set, hit ratios will be high and stable. If the working set is larger than your cache, items get evicted before they can be reused — a condition called cache thrashing.

Write Policies

When data changes, you have three strategies for keeping the cache in sync with the source. If you write to the cache AND the backing store simultaneously — guaranteeing they match at all times — that is write-through. It is safe and consistent, but every write still pays the cost of hitting the backing store.

If you write only to the backing store and skip the cache entirely (forcing the next read to re-fetch) — that is write-around. Good for data that is written once and not immediately re-read — it prevents polluting the cache with cold data.

If you write only to the cache and lazily flush to the backing store later — that is write-back (or write-behind). It is fastest because the caller doesn't wait for the DB, but it introduces risk: if the cache crashes before flushing, data is lost.

Cache Locality

The reason CPU caches are so effective is that programs naturally exhibit locality — they tend to access the same data repeatedly, and data near recently-accessed data. When a program uses the same variable or memory address over and over — think a loop counter — that is temporal locality. When a program accesses sequentially adjacent memory locations — think iterating an array — that is spatial locality. Both principles explain why caches at every layer are so effective — they work because programs are not random in what they access.

Cache Relationships (Inclusive vs Exclusive)

When you have multiple cache layers, you have to decide: can the same data item live in more than one layer at a time? If yes, the caches are inclusive — each layer is a superset of the layer above it. If an item lives in exactly one layer at any time, the caches are exclusive — evicting from one layer moves the item to the next rather than discarding it. AMD CPUs use exclusive L2/L3 caches; Intel traditionally used inclusive L3 (this has evolved). For distributed systems, the inclusive vs exclusive choice maps to whether you populate multiple cache tiers simultaneously (inclusive) or only cache at the tier that served the miss (exclusive).

Hardware Cache Coherence

On a multi-core CPU, each core has its own L1 and L2 cache — meaning the same memory address can exist in multiple cores' caches simultaneously. If one core changes a value, all other cores with a copy need to know. The hardware protocol that manages this is called MESI (Modified, Exclusive, Shared, Invalid). Each cache line is tagged with one of these four states, and the hardware automatically coordinates between cores when a write happens. This is invisible to your code but explains why false sharing — two threads writing to adjacent variables on the same 64-byte cache line — can cause surprising performance cliffs: every write invalidates the other core's copy and forces a round-trip through L3.

The concept map shows how all the vocabulary connects. Hit ratio and miss penalty together determine end-to-end latency — you want maximum hit ratio and minimum miss penalty at every layer. Working set size tells you whether a given cache is large enough to hold hot data without thrashing. Write policies decide consistency trade-offs. Locality explains why caches work at all. Inclusive vs exclusive decides total capacity. MESI is the hardware implementation of consistency at the CPU level.

With the vocabulary and pyramid mental model in place, let's walk each layer one by one — what it is, who controls it, the latency numbers, the typical capacity, and (critically) what makes it the right layer for certain types of data. Later blocks dive deep into each; here the goal is a clear, side-by-side picture.

Layer 1 — CPU L1 / L2 / L3 Caches (1–30 ns)

The fastest memory in any computer is not RAM — it is the cache baked directly onto the processor die. There are actually three tiers of this on-die cache, nested inside the CPU itself, named by how close they sit to each core: "level 1" is closest and smallest, "level 3" is furthest and largest. The innermost one — closest to a single core — is called L1 cache, holds 32–64 KB per core and responds in about 1 nanosecond — that is 4 CPU clock cycles at 4 GHz. The middle tier, L2 cache, holds 256 KB to ~4 MB and responds in 4–12 ns. The outermost, L3 cache, is shared across all cores and holds 4–192+ MB depending on the CPU, responding in 15–50 ns.

Your application code cannot directly address L1/L2/L3 caches. You cannot say "put this variable in L1." The CPU hardware manages these caches automatically — the programmer's job is to write code with good cache-friendly access patterns. Accessing array elements in order (sequential → excellent spatial locality) is dramatically faster than jumping around a linked list (random pointers → terrible spatial locality). This matters for database internals, serialization, and hot-path server code. A cache-unfriendly inner loop can run 10–100× slower than an equivalent cache-friendly one — not because of algorithm complexity but because of L1/L2 thrashing.

Layer 2 — OS Page Cache (~100 ns, GBs of DRAM)

When your application reads a file — say, a database reading a data page from disk — the operating system does something clever: it keeps a copy of that file's data in ordinary RAM (DRAM) even after the read is done. The next time anyone reads the same data, the OS serves it directly from RAM without touching the disk. This memory region managed by the kernel is called the OS page cache.

Why does this matter? Because your database is really two caches at the OS level: its own buffer pool (Layer 5) AND the OS page cache. When Postgres reads a data page and it's not in shared_buffers, the kernel checks its page cache first. If the page is in the page cache (a kernel-level cache miss from Postgres's perspective, but a page-cache hit from the OS perspective), the read takes ~100 ns. Only if the page cache also misses does the OS issue an actual disk I/O — 50 µs for an SSD, 5–10 ms for an HDD. This two-level structure at the OS layer is often overlooked but explains why databases see much better performance than naïve latency calculations suggest.

Layer 3 — App-Process Cache (100 ns–1 µs, MBs–GBs)

When you store results in a HashMap inside your Java application, or a dict in Python, or use a library like Caffeine — that is Layer 3: the app-process cache. It lives inside your application's own memory (heap), so access is just a hash lookup — no network, no IPC, no syscall. Latency is effectively the same as accessing any other in-memory data structure: 100 ns to a few microseconds.

The critical limitation of Layer 3 is that it is local to one process. If you have ten application servers, each has its own independent in-process cache. If server A caches a user's profile, server B knows nothing about it. This makes Layer 3 best for shared-nothing, read-heavy, globally valid data — reference data that is the same for all servers (configuration values, product catalog snippets, static rate limits), not per-user state. For per-user or frequently-updated shared state, you need Layer 4.

Layer 4 — Distributed Cache: Redis / Memcached (0.5–2 ms, GBs–TBs)

When you need a cache that is visible to all your application servers simultaneously, you reach for a distributed cache. Redis and Memcached are separate server processes — your application connects to them over a TCP socket, pays a network round-trip, and gets back a cached value in 0.5–2 ms.

That network round-trip is the key trade-off: Layer 4 is ~1,000–2,000× slower than Layer 3 for any single lookup, but it is shared. Every server in your fleet sees the same data. This makes it ideal for: user sessions, rate limit counters, cart contents, real-time leaderboards, feature flags, and any state that is per-user but needs to survive a process restart or be visible to multiple servers. The canonical rule: if the data lives in your database and you want to serve it without a DB query, Redis is your Layer 4.

Layer 5 — Database Buffer Pool (5 µs–15 ms, GBs on DB Server)

Your database has its own cache, and it is arguably the most powerful cache in the entire stack for database-heavy applications. The database buffer pool caches raw data pages in RAM on the database server itself. Postgres calls it shared_buffers. MySQL/InnoDB calls it innodb_buffer_pool_size. The concept is the same: a chunk of the DB server's RAM is set aside for caching recently-read database pages.

When a query reads a row, the database loads the entire 8 KB page containing that row into the buffer pool. Future queries for any row on that same page are served from RAM in ~5 µs instead of requiring a disk read (50 µs for NVMe SSD, 5–10 ms for HDD). For a production database with a well-tuned buffer pool and a working set that fits in RAM, 90–99% of reads never touch storage at all. The critical tuning fact: Postgres's default shared_buffers is just 128 MB regardless of how much RAM the server has. On a 32 GB server, a single config change — shared_buffers = 8GB — can eliminate most disk I/O entirely.

Layer 6 — CDN Edge (5–50 ms, GBs–PBs, Globally Distributed)

The outermost cache layer sits between the Internet and your origin servers. A CDN edge node is a server in a data center physically close to the user — often within a city or metropolitan area. When the edge node has a cached copy of the requested resource (an HTML page, a JSON API response, an image), it serves the response directly: ~5–30 ms to the user regardless of where your origin server lives.

CDN caching is controlled by HTTP headers your application sends — Cache-Control, Surrogate-Control, Vary, ETag. The cache key is typically the URL. CDNs are ideal for: static files (JS/CSS/images with content-hash URLs, cached for a year with Cache-Control: immutable), shared page shells (header, footer, product grids), and public API responses (search results, product listings). They are not ideal for personalized responses (shopping carts, authenticated pages) without careful key partitioning or ESI (Edge Side Includes).

The comparison table highlights the most important pattern: as you move down the layers, latency multiplies by 10–1,000× at each step, while capacity also multiplies by 10–1,000×. There is no layer that is "better" — each layer has a different role. The goal is to ensure that the hottest data (most frequently accessed) lives in the fastest layer that can hold it, and that each layer is sized large enough to achieve a high hit ratio for its working set.

Here is the insight that separates engineers who design caches from engineers who just use them: cache hit ratios across multiple layers do not add — they multiply. And multiplication is far more powerful than addition.

The Cascading Multiplication

Imagine a four-layer cache stack where each layer has a 90% hit ratio. You might intuitively think "90% + 90% + 90% + 90% = 360%" which doesn't even make sense, or maybe "90% average across layers." But the real math works like this: the 10% of requests that miss Layer 1 reach Layer 2. Layer 2 hits 90% of those — so 10% × 10% = 1% of original requests reach Layer 3. Layer 3 hits 90% of those — so 1% × 10% = 0.1% of original requests reach Layer 4. Layer 4 hits 90% — so only 0.1% × 10% = 0.01% of original requests reach the backing store (disk).

The formula is straightforward: the probability of a request reaching layer N is the product of all miss rates for layers 1 through N-1.

Put differently: you need 10,000 requests at Layer 1 before a single request reaches disk. If your Layer 1 is processing 10,000 requests per second, your disk sees 1 request per second. That is the power of compounding.

The cascade diagram makes the compounding visceral: each layer's bar is 10× shorter than the one before it, because each layer absorbs 90% of what reaches it. By the time you reach the disk, the traffic is a tiny sliver — 10 requests per second from an original 10,000. The database doesn't need to scale to match your application traffic; it only needs to handle the residual after every upstream cache layer has done its job.

The End-to-End Latency Formula

The same compounding logic applies to average latency. The end-to-end average latency of a multi-level cache stack is not the slowest layer's latency — it is the probability-weighted sum across all layers.

Let's work through a realistic example with five layers and real latency numbers. The numbers below are representative for a typical production web application with a well-tuned Redis cluster, reasonable DB buffer pool, and CDN in front:

The worked example reveals something surprising: even though disk I/O is 10 ms — roughly 33,000× slower than an app-cache hit — the disk contributes only about 2% of the average latency. Why? Because only 0.075% of requests ever reach it. The slowest layer barely matters for the average, because the upstream layers filter out virtually all traffic. This has a counterintuitive implication for optimization: spending engineering effort to improve L1 hit ratio (e.g., from 85% to 92%) has a larger impact on average latency than tuning the disk read path by 10×, because the disk path is already so rare.

When a Single 50% Layer Is Worth Adding

The math also reveals why adding a mediocre cache layer with only 50% hit ratio can still be worthwhile. If your stack currently sends 1,000 requests per second to the database, adding a layer with 50% hit ratio reduces database traffic to 500 requests per second — for free, without any database tuning. Whether 500 req/s of saved DB load justifies the operational cost of the new cache layer depends on your specific system, but the math is always in your favor as long as the cache layer is cheaper per request than the layer below it. Redis at 0.5 ms is dramatically cheaper to scale than a Postgres query at 10 ms — so even a 50% Redis hit ratio dramatically reduces DB scaling costs.

Before your code ever touches Redis or a database, it runs on a CPU that is doing its own caching — silently, automatically, and at speeds so fast that most engineers never think about them. Understanding this layer won't let you configure it (the hardware controls it entirely), but it will explain why some algorithms are 10× faster than others on identical hardware, and why "cache-friendly" code is a real engineering concern worth talking about in a systems interview.

The Speed Gap Problem: Why CPU Caches Had to Exist

Here is the fundamental problem that forced CPU caches into existence: modern CPUs execute instructions in roughly 0.25–0.5 nanoseconds (a 3 GHz clock ticks every 0.33 ns), but reading from DRAM — the ordinary RAM in your computer — takes roughly 80 nanoseconds. That is a 200× speed gap. Without caching, the CPU would spend most of its time waiting for data, running at roughly 0.5% of its theoretical peak speed.

The solution: build a small chunk of extra-fast memory and bolt it onto the CPU itself. The memory used inside CPU caches is a different type than the RAM in your DIMM slots — it uses six transistors per bit instead of one, so each bit is wired up to respond instantly without needing the periodic "refresh" pulse ordinary RAM needs. This faster type is called SRAM (Static RAM), and the slower bulk type used in main memory is DRAM (Dynamic RAM). SRAM is faster but six times more expensive per bit, so you can only afford a tiny amount. The hardware splits that tiny amount into a three-level hierarchy:

The memory hierarchy pyramid. Each level down is roughly 5–10× slower, 5–30× larger, and much cheaper per byte. The CPU checks L1 first; on a miss it checks L2, then L3, then DRAM. Every miss to a lower level costs more cycles.

What the CPU Actually Caches: Cache Lines, Not Individual Bytes

Here is a detail that trips up many people: the CPU doesn't cache individual bytes or even individual integers. Instead, every time it pulls something from RAM it grabs a fixed-size neighborhood — typically a 64-byte chunk on Intel/AMD x86 and most ARM64 server and mobile cores — and stuffs the whole chunk into the cache. Apple's M-series chips are a notable exception and use 128-byte lines. Those chunks are called cache lines. Why 64 bytes specifically? Because programs almost never read just one byte and stop. If your code touches address X, it will very likely touch X+1, X+2, ... X+63 within a few instructions — a pattern called spatial locality. Loading the whole neighborhood at once means one slow trip to DRAM pays for dozens of future fast reads.

This has a practical consequence for data structure design: an array of 64-bit integers stores 8 integers per cache line. Traversing the array sequentially loads a new cache line every 8 elements, but each load serves the next 7 reads for free — very efficient. A linked list, by contrast, stores each node at a random heap address; every pointer dereference may require loading an entirely new cache line, potentially causing a full DRAM round-trip for every node. Same algorithmic complexity (O(n)), but the array version can be 5–10× faster in practice purely because of cache efficiency.

Temporal and Spatial Locality: Why Caches Work at All

CPU caches work because real programs aren't random — they have patterns. Two key patterns make caching effective:

• Temporal locality — if you accessed a piece of data once, you'll probably access it again soon. A loop counter used a million times fits this pattern perfectly. The CPU keeps it in L1 and never pays the DRAM cost after the first load.
• Spatial locality — if you accessed address X, you'll likely access nearby addresses soon. Array traversal, struct field access, and sequential file reads all exhibit this. The cache-line mechanism exploits it automatically.

When your code violates these patterns — accessing memory in a random stride, jumping around a huge linked list — you "thrash the cache" and effectively reduce the CPU to running at DRAM speeds. The difference between cache-friendly and cache-hostile code on the same hardware is routinely 5–10×.

MESI Coherence: Keeping Multi-Core Caches Consistent

A modern CPU has 8–128 cores, each with its own L1 and L2 cache. If two cores both cache the same memory address and one of them writes to it, the other's cached copy is now stale and would return the wrong answer if used. The hardware fixes this by giving every cached 64-byte chunk a little status tag — "is this copy still trustworthy, or has somebody else changed it?" — and broadcasting updates between cores whenever a write happens. The protocol that defines the four possible tag values and the rules for switching between them is called MESI. The name is just the four states stitched together:

The hardware circuitry managing MESI transitions runs entirely without software intervention. From your code's perspective, multi-core caches appear coherent. From a performance perspective, however, MESI coherence creates one dangerous trap.

False Sharing: The Silent Performance Killer

Imagine two threads on two different cores, each incrementing their own independent counter. Thread A increments counter_a; Thread B increments counter_b. Completely independent operations — in theory, they should run at full speed with zero contention. But if counter_a and counter_b happen to live in the same 64-byte cache line (e.g., they are adjacent fields in the same struct), every write by Thread A invalidates Thread B's cached copy of that line via MESI, and vice versa. Neither thread is actually touching the other's data, but the CPU's cache coherence mechanism can't tell — it only sees cache lines, not individual variables. This is false sharing: two threads contending over a cache line they don't logically share.

False sharing in action. Both cores read the same 64-byte cache line into their L1 caches. When Core 0 writes counter_a, MESI sends an invalidation to Core 1 — even though Core 1 only cares about counter_b. Both threads slow to DRAM speeds despite touching logically independent data. The fix is to pad each counter to 64 bytes so they live in separate cache lines.

Every Linux server you've ever run an application on has been quietly caching disk data in RAM behind your back — automatically, since the 1990s. This is the OS page cache, and it is genuinely the cheapest performance win in systems engineering because you didn't have to configure, buy, or install anything. It just works. Understanding it lets you reason about when it's working for you, when workloads fight against it, and how to tune it for your specific situation.

How the Page Cache Works: The Kernel's RAM Recycler

When your application calls read() on a file, the kernel doesn't fetch bytes straight from disk and hand them to your process. Instead, it reads an entire page (4 KB on x86 Linux) into a kernel-managed RAM buffer, then copies the data to your process's address space. The crucial part: after satisfying your read() call, the kernel keeps that page in RAM. The next read() — by you, by another process, or by the OS itself — is served entirely from RAM with no disk involved.

Where does this RAM come from? The kernel uses all RAM not currently needed by processes as page cache. If a process needs more heap memory, the kernel evicts cold pages from the page cache (using a variant of LRU) to reclaim space. The page cache is not a fixed allocation — it expands and contracts dynamically based on system pressure. This is why the output of free -h can be confusing at first glance:

A common mistake: seeing free = 0.8 GB and panicking that the server is out of memory. The 26 GB in buff/cache is available to processes on demand. The kernel will evict page cache pages to make room. available is the honest metric for "how much RAM can a new process get right now."

The read() path through the OS page cache. On a hit, the kernel copies data from a RAM page (~100 ns). On a miss, it fetches from disk, stores the page in the cache, then copies to your process. Critically, the page stays in cache after the miss — subsequent reads are free.

Why Databases Love (and Sometimes Fight) the Page Cache

Every database you've ever used benefits from the page cache by default. Postgres reads its data files through the page cache, and the page cache is often the reason Postgres feels fast even when shared_buffers is set to a modest 2 GB on a 64 GB server — the OS is quietly caching another 40 GB of hot pages underneath. This creates a double-buffering situation: both Postgres's own buffer pool and the OS page cache hold copies of the same data. It's technically wasteful (RAM used twice), but in practice the two levels serve different purposes — shared_buffers knows it's holding database pages and evicts intelligently; the page cache just sees bytes and does generic LRU.

MySQL InnoDB takes a different approach: it sets the O_DIRECT flag on its data files, which bypasses the page cache entirely and uses its own buffer pool exclusively. The motivation is to avoid double-buffering: if you've allocated 60 GB to the InnoDB buffer pool, there's no point caching the same data again in the page cache. InnoDB's buffer pool has smarter eviction policies (it knows about hot vs cold pages in its B-tree), so excluding the OS cache is a net win when RAM is the constraint.

Scan Eviction: The Page Cache's Achilles Heel

The page cache's biggest weakness is scan eviction. Imagine running a full-table scan on a 200 GB table while your production database also serves live OLTP queries. The scan reads 200 GB of pages sequentially, filling the page cache as it goes. Because the OS can't tell the difference between "scan I'll never see again" and "hot page accessed a thousand times today," it happily evicts hot pages to make room for scan data. After the scan finishes, all your hot OLTP pages are gone — every subsequent query causes a cache miss and a disk read. The database spends the next several minutes re-warming the cache. This is called cache pollution.

The fix is to tell the kernel about your access pattern so it stops treating the scan like hot data. Linux gives you two system calls for this — one for normal file reads, one for memory-mapped files. They are called fadvise() (file advise) and madvise() (memory advise), and they let you whisper hints like "I'm scanning, don't bother keeping these pages" or "I'll be jumping around randomly, skip the pre-fetch." Used together, they make a scan walk through the page cache without pushing out the hot OLTP data:

Postgres historically used posix_fadvise(POSIX_FADV_WILLNEED) driven by effective_io_concurrency to prefetch pages during bitmap heap scans and similar operations (modern Postgres 17+ has moved sequential scans onto the streaming-I/O / async-I/O framework instead). On the application side, POSIX_FADV_DONTNEED is the surgical lever you reach for when you specifically want to prevent a one-shot batch scan from polluting the page cache. It's a rare but powerful tuning lever for mixed OLTP + analytical workloads.

tmpfs and shmfs: Explicit In-RAM Filesystems

Sometimes you want a guarantee that certain data always lives in RAM and never gets written to disk at all — temporary files, in-memory IPC buffers, shared-memory regions between processes. Linux offers a special filesystem that looks and acts like any other directory but is secretly backed by the page cache and swap rather than a real disk. It is called tmpfs (temporary filesystem). Files you write to a tmpfs mount stay entirely in RAM. Postgres's shared memory segment lives on a tmpfs mount (the familiar /dev/shm path), which is why Postgres restart is so fast — shared_buffers is in RAM-backed shared memory the kernel never has to read from disk.

The in-process application cache is the cache you build yourself, inside your application's own memory. Unlike the CPU cache (hardware-controlled) or the OS page cache (kernel-controlled), this one is yours to design: you choose what to store, how much, how long, and what eviction policy to use. It is the fastest cache you can write application code against — because the data is already in the same process, there is no network hop, no serialization, no protocol overhead. A hit costs roughly 100–500 ns on a modern JVM, compared to 0.5–2 ms for Redis. That is a 1,000–5,000× speed difference.

Why Not Just Use a HashMap? The Problem With Unbounded Caches

The simplest in-process cache is a HashMap: put data in on first access, read from it on subsequent accesses. This works until your JVM runs out of heap memory and starts either throwing OutOfMemoryError or spending 80% of its time in GC. An in-process cache without a size bound is just a memory leak with aspirations. You need at minimum: a maximum size (so memory is bounded), an eviction policy (so the least-useful entries are dropped first), and optionally a TTL (so stale data expires).

Caffeine: The Modern Best-in-Class (Java)

For Java, Caffeine is the de-facto standard. It replaced Guava's cache as the go-to library (Spring's @Cacheable uses Caffeine by default when it's on the classpath). The reason it pulled ahead is its eviction algorithm. The old default — "drop whatever was used longest ago" (plain LRU) — has a known weakness: a one-time scan can shove out keys that have been hammered a million times. Caffeine instead remembers how often each key has been used recently, and protects frequently-used keys from one-shot scans. The technique has a mouthful of a name — W-TinyLFU (Windowed Tiny Least Frequently Used) — but the idea is just "favour popular keys over recently-touched-once keys." Here is a minimal production-ready Caffeine setup:

Key points from this snippet: maximumSize bounds memory; expireAfterWrite bounds staleness; get(key, loader) is the preferred read pattern because Caffeine serializes concurrent loads for the same key — if 100 threads request the same missing key simultaneously, only one DB call fires and the other 99 wait for that result. This prevents the cache stampede (also called "thundering herd") where a sudden miss causes a flood of identical DB queries.

The Multi-Instance Problem: Why In-Process Cache Hit Ratio Degrades at Scale

Here is the catch with in-process caching: when you run multiple application instances (horizontal scaling), each instance has its own private cache. Instance A might cache user profile 42; instance B might not. A request routed to instance B misses and hits the database, even though the exact same data is sitting warm in instance A's heap. Your effective hit ratio is per instance, not across the whole fleet.

With 3 instances and random load balancing, the same hot key (user 42) is only cached in 1 of 3 instances. Roughly two-thirds of requests miss locally and fall through to the database. The fix is either sticky sessions (fragile, counter to horizontal scaling) or a combined strategy: a small Caffeine cache for the very hottest keys, backed by Redis for cross-instance sharing.

The L-Shape Cache Strategy: In-Process + Distributed Together

The best pattern for production is a two-level lookup: check the in-process Caffeine cache first (1 µs); if missed, check Redis (1 ms); if missed, query the database and populate both caches. This is called the "L-shape" or "layered" cache strategy. The Caffeine tier holds the hottest ~1,000–10,000 keys; Redis holds the full warm data set across all instances. The Caffeine layer absorbs the top 5–10% of traffic at microsecond speed. The Redis layer handles the rest at millisecond speed. Together, fewer than 1% of requests reach the database.

Equivalents in Other Languages

A distributed cache solves the one problem that in-process caching cannot: shared state across multiple application servers. Instead of each server maintaining its own private copy of cached data, all servers talk to a central (or clustered) cache tier. Every server sees the same data, invalidations propagate instantly, and the cache can be sized independently of your application server heap. The tradeoff — and it is a real one — is a network round-trip for every cache operation: typically 0.5–2 ms on a well-connected internal network, compared to ~100–500 ns for in-process.

Redis vs Memcached: Which One and When

In practice, Redis wins the majority of greenfield decisions because it is more flexible. The overhead difference between Redis and Memcached is rarely the bottleneck — network latency and serialization cost typically dwarf the few-microsecond difference in the servers themselves. Pick Memcached only when you've benchmarked a specific bottleneck that Memcached solves and Redis doesn't.

Redis Cluster: Horizontal Sharding via Consistent Hashing

A single Redis node caps out at roughly 25–50 GB of useful cache data (beyond that, GC-equivalent pauses on the fork for RDB snapshots start to hurt). Redis Cluster scales horizontally by splitting the keyspace into 16,384 hash slots. Every key is hashed to a slot (using CRC16 mod 16384), and every cluster node owns a contiguous range of slots. The client library calculates which slot a key belongs to before sending the request, connecting directly to the right node. This is a form of consistent hashing: adding a new node means redistributing only some of the slots, not rehashing the entire keyspace.

Redis Cluster splits the 16,384 hash slots across primary nodes. The client library computes the slot locally and connects directly to the right node — no central router hop. Each primary has at least one replica for failover. When you add a new node, only a proportional subset of slots moves; everything else stays in place.

Reducing Round-Trips: Pipelining and Lua Scripts

The biggest cost of a distributed cache is the network: 0.5–2 ms per round-trip. If your request handler makes 20 independent Redis calls, that's 10–40 ms of pure network overhead before the first byte of your business logic runs. Two Redis features cut this dramatically:

Pipelining via MULTI/EXEC — batch multiple commands into one TCP write. The server buffers them and sends all responses back in one reply. Zero-copy batching:

Lua scripts — run arbitrary logic on the Redis server, eliminating the network round-trip for the "read → decide → write" pattern. For example: "if the rate-limit counter for this IP is under 100, increment and return 'allowed'; otherwise return 'blocked'" — in Lua, this is one atomic server-side operation. In application code, it would be a GET, then a conditional SET — two round-trips with a potential race between them.

Failure Modes: Hot-Key Meltdown and Network Partitions

Every database that reads data from disk would be unbearably slow if it had to do an actual disk read for every row you touch. The database's solution is exactly the same as the OS page cache — keep recently-accessed data in RAM — but smarter: instead of caching generic file bytes, the database caches structured pages (8 KB blocks for Postgres, 16 KB for InnoDB) that it understands as B-tree nodes, heap tuples, or index entries. Because the database knows what these pages contain, it can make better eviction decisions than a generic kernel cache.

Postgres: shared_buffers and the Double-Buffering Dance

Postgres manages its own in-memory cache of database pages, called the shared buffer pool, configured by shared_buffers. The conventional starting point is 25% of total RAM — so on a 64 GB DB server, set shared_buffers = 16GB. Why only 25%? Because Postgres reads its data files through the OS page cache (it does not use O_DIRECT by default), and the OS page cache can beneficially cache the same pages again in the remaining 75% of RAM. This double-caching is technically redundant but provides a practical buffer: if shared_buffers is cold after a restart, the OS page cache (which the kernel persistently maintains across Postgres restarts) may already have hot pages. Postgres will re-populate shared_buffers from the OS cache at ~100 ns per page, not from disk at 0.1–8 ms per page.

Postgres Eviction: Clock-Sweep

When shared_buffers is full and Postgres needs to load a new page, it has to throw something out to make room. How does it decide? Imagine the buffer pool as a circular table of slots, and a single hand pointing at one slot — like a clock. Each slot has a small counter that goes up every time something touches that page. The hand walks slowly around the ring, and at each stop it decrements the counter by one. When a counter reaches zero, that slot is the loser and gets evicted. This sweeping clock-hand approach is called the clock-sweep algorithm, and the reason Postgres uses it is mostly about avoiding lock contention: a strict "least recently used" list would require every cache hit to update a global linked list, which becomes a hotspot under high concurrency. Clock-sweep approximates LRU using only local counter increments, no shared list, no global lock.

InnoDB: Buffer Pool + O_DIRECT

MySQL's InnoDB storage engine takes a more aggressive approach to memory management: allocate a large fraction of RAM to the InnoDB buffer pool and use O_DIRECT to bypass the OS page cache entirely. The motivation is clear: if you're giving InnoDB 50 GB of buffer pool on a 64 GB server, there's no point having the OS also cache those same pages — you'd have no RAM left for anything else.

InnoDB Midpoint Insertion: Protecting Hot Pages From Scans

InnoDB has a clever twist that directly addresses the scan-pollution problem we saw in the OS page cache. The buffer pool is still arranged as a list with "hot" at the head and "evict next" at the tail, but instead of dropping new pages straight at the head — where they'd push the truly hot pages closer to eviction — InnoDB drops them partway down the list. That partway-down spot is called the midpoint (by default, about 3/8 of the way up from the tail). New pages enter here. A page only graduates to the front of the list (the "young" or hot sublist) if it gets touched a second time. Pages that are only touched once after insertion drift through the back half (the "old" sublist) and get evicted from the tail without ever disturbing the truly hot data.

The key insight: a full-table scan floods new pages in at the midpoint. If they are only read once (typical for a scan), they never get promoted to the young list and are quickly evicted without displacing the truly hot pages that live at the head. The hot OLTP pages survive the scan untouched.

InnoDB's scan-resistant LRU. New pages enter at the midpoint (boundary between young and old sublists). If a page is accessed again soon, it gets promoted to the head of the young list and survives for a long time. Scan pages that are never re-accessed slowly drift toward the tail of the old sublist and are evicted first — never displacing truly hot pages.

Hit Ratio: The #1 Database Performance Metric

The buffer pool hit ratio tells you what percentage of page reads are served from RAM vs disk. This single number summarizes your database's cache efficiency better than any other metric. A rough guide:

• ≥ 99.5% (InnoDB ≥ 997/1000) — excellent. Almost all reads from RAM. Very few disk reads.
• 98–99.5% — good for a working set that's 90%+ hot. Occasional cold reads.
• 90–98% — 2–10% of reads are disk reads. Investigate: working set larger than buffer pool? Scan pollution? Need to increase buffer pool or add caching above the DB.
• < 90% — serious I/O pressure. Page queries will be slow. The buffer pool is significantly undersized for the working set, or a batch scan is polluting it.

Every layer we've covered so far lives inside your data center — on the CPU die, in the kernel, in the JVM, in a Redis cluster, or in the database server. The CDN edge cache is different: it lives outside your data center entirely, in hundreds of small server rooms scattered across the world's cities, each one close to the users in that region. CDN providers call each of these little server rooms a Point of Presence, or PoP for short — just a name for "a CDN data center near users." Its job is to serve your content before a user's request ever reaches your origin servers. For a user in Singapore, that means the response travels from a CDN node in Singapore (~5 ms round-trip) instead of from your US-east origin (~200 ms). The CDN is the one cache that reduces latency not by making your server faster, but by moving the response physically closer to the person asking for it.

What a CDN Can (and Cannot) Cache

The conventional wisdom is "CDNs cache static files." The more accurate picture in 2024 is "CDNs cache anything with a cache key and a reasonable TTL." The canonical categories, from easiest to hardest:

• Static assets — JavaScript bundles, CSS, images, fonts, video segments. Cache-Control: immutable, max-age=31536000 (one year) using content-hash URLs. Hit ratios of 99%+ are achievable. These are so static that the only invalidation trigger is deploying a new build with a new content hash.
• Shared HTML fragments — product pages, blog posts, marketing pages. Cache with a moderate TTL (60 s–5 min) and use surrogate keys (Cloudflare Cache-Tags, Fastly Surrogate-Key header) to invalidate all related pages instantly when content changes. Hit ratios of 80–95%.
• API responses — increasingly common. Product listings, search results, pricing queries — all can be cached at the CDN edge when they are not user-personalized. Use Cache-Control: public, s-maxage=30 (cache at CDN for 30 seconds, revalidate). Hit ratios of 50–80% for popular queries.
• Personalized content — responses that contain user-specific data (a user's cart, their name in the header) must not be cached at shared CDN edge nodes. The fix is to split pages into a cacheable shell + dynamic user-specific fragment loaded client-side (Edge Side Includes, or a separate API call after initial render).

Cache Key: What Makes a CDN Response Unique

A CDN identifies a cacheable response by its cache key — by default, the URL including query string. Two requests for the same URL with the same cache key are served the same cached response. This is why cache keys need careful design:

Tiered Caching: Shield Nodes Cut Origin Load

In a basic CDN setup, every edge PoP has its own cache. A cache miss at the Singapore PoP goes directly to your origin. A cache miss at the Tokyo PoP also goes directly to your origin. A cache miss at the Sydney PoP does the same. If you have 300 PoPs and a CDN miss rate of 20%, each of those 300 PoPs independently hammers your origin. The traffic that reaches your origin server is 300× amplified relative to what a single-PoP scenario would generate.

The solution is CDN shielding (also called "origin shield" or "tiered caching"): add a mid-tier cache layer between the edge PoPs and your origin. The Singapore, Tokyo, and Sydney edge nodes all forward misses to a single regional shield node (say, Tokyo shield), which itself caches the response. Your origin only sees one request even if three edge PoPs missed simultaneously for the same URL. The shield absorbs the long-tail miss traffic; your origin sees only the cache misses the shield itself couldn't serve.

The full six-layer request path. A CDN edge hit costs 5–30 ms total and is the default happy path for static and shared content. A full cold miss through all layers to disk costs 100–600 ms. Crucially, the CDN and the origin-side caches (Redis, buffer pool) are complementary: the CDN serves shared public content; Redis serves user-specific computed state; the buffer pool absorbs DB page reads. No single layer replaces the others.

CDN in the Six-Layer Context: What It Does That No Other Layer Can

The key insight that ties this section to the full hierarchy: the CDN is the only layer that operates on the user's side of the network. Every other cache layer (in-process, Redis, buffer pool) is co-located with your application — they reduce work on your servers, but they do nothing about the physics of data traveling across the Atlantic or Pacific. The CDN collapses the geographic distance problem. It doesn't make your app smarter; it makes the content physically closer.

This also defines exactly what the CDN is bad at: anything that requires real-time user-specific computation. A user's shopping cart, their personalized recommendations, their account balance — these need data from your database that only your application can assemble. The CDN can serve the shell of the page in 10 ms; Redis and the buffer pool assemble the personalized data in another 2–3 ms; together they give you a total page experience under 20 ms without the CDN ever touching the private data.

Invalidation: The Hard Problem at the CDN Layer

The most dangerous CDN failure mode isn't a miss — it's serving a stale hit. If a product price changes and the CDN is still serving the old price from a 5-minute cached response, customers see incorrect prices. CDNs solve this with two mechanisms: TTL expiry (wait for the cache to age out naturally) and instant purge. Purge needs a clever trick, because a single product change might affect dozens of cached pages: the product page itself, the category page, the homepage, the search results. Listing every URL to purge by hand is tedious and error-prone. Instead, CDNs let you attach labels to each cached response — a product page can be labelled "product-42, category-electronics, homepage-featured" — and then you purge by label, not by URL. These labels are called surrogate keys (called Cache-Tags on Cloudflare, Surrogate-Key on Fastly). One API call ("purge everything tagged product-42") clears every cached response touching that product, across every PoP, in a few hundred milliseconds.

The six cache layers exist simultaneously in every production system — but that doesn't mean every piece of data belongs in every layer. Putting data in the wrong layer wastes memory, introduces stale-data bugs, or misses the latency benefit you were hoping for. The decision isn't hard once you understand the three questions that drive it: Who shares this data? How often does it change? How large is it?

The Three Axes That Drive Layer Selection

Before picking a layer, ask these three questions about the data you want to cache:

1. Scope — who needs it? Data that every server in every region needs (public homepage HTML) is a CDN candidate. Data that only the current server process needs (a parsed config file) belongs in an in-process cache. Data that all servers in one region share (a logged-in user's session) belongs in a distributed cache like Redis.
2. Mutability — how often does it change? Static assets that change only on deploy → CDN with long TTLs. Price data updated every few minutes → distributed cache with short TTL. A user's shopping cart, mutated on every click → distributed cache with write-through on every change. A CPU-computed result like a cryptographic hash → in-process cache with a bounded capacity.
3. Size — how much memory does it occupy? A parsed JWT token is a few hundred bytes and belongs in-process. Gigabytes of product catalog belong in Redis, not in every app-server heap. Video files belong in the CDN, not in Redis.

The decision tree above walks you through the dominant question at each branch. A public homepage with the same HTML for all users? CDN, because geographic proximity gives you the biggest latency win. A user's shopping cart? Redis, because multiple app servers need to read it — you can't keep it in one server's heap. A single server's parsed YAML config? In-process map, because every network hop is wasted cost for data only that process needs.

The matrix above is worth internalizing. Notice the pattern: anything user-specific and mutable goes to Redis, because you need it shared across servers but it must not be cached at CDN (which would serve your cart to someone else). Anything DB-internal goes to the buffer pool — that layer is managed automatically by Postgres or MySQL, and you only tune it by configuring memory allocation. Financial records don't live in any cache as their primary store — durability requirements mean the disk is the record of truth, and any cache layer holding a financial value is secondary at best.

Common Placement Mistakes and Why They Happen

The most frequent mistake teams make is reaching for Redis reflexively — for everything. Config values that never change and are only read by one service end up in Redis even though an in-process map would be 200× faster and involve no network hop. The motivation is understandable: Redis is already in the stack, so it feels "easy." But every Redis GET is a network round-trip (~0.5 ms), and if you're calling it 50 times per request for config lookups, you've added 25 ms of latency for data that could live in a local map refreshed every 60 seconds.

The second common mistake is caching per-user data at the CDN level. CDN caching is fundamentally designed for shared content — the same response, delivered to many users. The moment you cache a response that varies by user identity, you risk the CDN serving one user's data to another. Some CDN providers support Vary headers or user-aware cache segments, but these are complex to configure correctly. The safer default: anything with user-specific data should bypass CDN caching entirely and go to Redis or the app layer.

Here is the uncomfortable truth about multi-level caching: adding a cache layer doesn't just make reads faster, it creates a new copy of your data. And the moment you have multiple copies, you have a consistency problem — when the real data changes, who tells the copies? This is why invalidation is often called the hardest problem in caching. Phil Karlton's famous quip — "There are only two hard things in computer science: cache invalidation and naming things" — is funny because it's deeply true. Let's understand why, and what to do about it.

The Layers That Take Care of Themselves

Good news first: the lowest two layers — CPU cache and OS page cache — handle their own consistency automatically, and you never need to think about them.

CPU caches use a hardware protocol called MESI (Modified, Exclusive, Shared, Invalid). When one CPU core writes to a memory address, the MESI protocol automatically broadcasts an invalidation signal to other cores that hold a cached copy of that address. This happens at the hardware level, in nanoseconds, without any software involvement. You never write application code to manage CPU cache coherence.

The OS page cache uses write-back semantics managed by the kernel. When your database writes to a file, the kernel updates the page cache immediately — so any subsequent read on the same machine sees fresh data. Across machines (NFS), page cache coherence is more complex, but for a typical single-node database setup, you don't manage this yourself.

The Layers Where You're Responsible

The layers that your application code manages — in-process cache, distributed cache (Redis/Memcached), and CDN — do not have any automatic invalidation mechanism. When the source data changes, nothing tells them automatically. You have to build that mechanism yourself. This is where bugs live.

Three strategies exist for keeping application-managed cache layers consistent with the source of truth:

1. Write-through — when the app writes to the database, it also writes the new value to the cache at the same time, synchronously. The cache is always current. Cost: every write hits both DB and cache; write latency doubles.
2. Write-behind (write-back) — the app writes only to the cache immediately, and a background job flushes to the database later. Lowest write latency; risk of data loss if the cache crashes before the flush. More on this in Section 15.
3. Explicit invalidation (delete on write) — the app writes to the database, then deletes (or marks stale) the cache entry. The next read is a cache miss and repopulates from the DB. Simple and safe, but causes a brief miss storm after writes. This is the most common pattern in practice.

The diagram above shows what a correctly implemented cascading invalidation looks like: a single write event fans out to every layer that might hold a copy. The red "forgotten layer" on the right is the classic production bug. One layer is missed — usually the in-process cache on individual app servers — and it serves the old price for as long as its TTL lasts. Users on servers that got the invalidation see $25; users on servers that didn't see $29. Both prices are "real" at different layers of the stack, which is worse than just being consistently wrong.

Three Invalidation Patterns in Practice

1. Pub/Sub Fan-Out (Redis PUBLISH)

The most common way to keep every server's in-process cache fresh in a multi-server fleet is to use Redis as a tiny one-way notification service: one server shouts "the price for product 42 changed!" and every other server hears it and clears the affected key. This shout-and-listen pattern is called publish/subscribe, or pub/sub for short, and "fanning out" just means the one shout reaches many listeners. When a server writes a new value, it publishes an invalidation message to a shared channel. All servers — including itself — subscribe to that channel and evict the affected key from their local in-process cache when they receive the message.

The advantage: all servers learn about the invalidation in milliseconds. The limitation: Redis pub/sub is fire-and-forget — if a server is temporarily disconnected, it misses the message and its in-process cache stays stale until the TTL expires. For high-stakes data (prices, inventory), you pair pub/sub with a short TTL as a fallback safety net.

2. CDC-Driven Cascading Invalidation

The pub/sub approach above only catches writes that go through your application. A migration script, an admin running a manual SQL update, or a different microservice writing to the same DB will all bypass the invalidation. A safer trick: have something watch the database itself and emit a notification for every row that changes, no matter who changed it. Databases already keep an internal log of every change for replication purposes (Postgres calls it the WAL, MySQL calls it the binlog); a tool can tap into that log and turn each row change into a message. This watch-the-log pattern is called Change Data Capture, or CDC. Tools like Debezium implement it for most databases. A separate invalidation service consumes the CDC stream and clears the appropriate keys in Redis and CDN. The benefit is that all writes are caught. The cost is operational complexity — you're adding Kafka or a similar message broker to the architecture.

3. Versioned Key Fingerprinting

Instead of invalidating, you change the key. When a product's data changes, you append a version hash to the cache key: product:42:v7 becomes product:42:v8. Old caches silently expire because no one asks for :v7 anymore. New reads always hit :v8, which is a miss on first access but is then populated correctly. This sidesteps the invalidation problem entirely — you never delete, you rotate. The trade-off: old versioned keys accumulate in the cache until TTL expires, wasting memory.

This stale-instance diagram describes a real class of production incident. A team deploys a new price. Redis is correctly invalidated. They even configured pub/sub to broadcast. But Servers 2, 3, and 4 were temporarily under load during the pub/sub broadcast and their subscriber threads were backlogged. For the next 45 minutes — until the in-process Caffeine cache TTL expired — 75% of checkout requests showed the old price. The fix is not complex: combine pub/sub with short in-process TTLs (60 seconds as a ceiling), and during high-stakes deploys, restart app servers in rolling order to flush all in-process caches cleanly.

We've talked a lot about what happens when a read hits or misses a cache. But caching strategies are equally important for writes. When a user updates their cart, changes a password, or completes a purchase, where does that data go first — and in what order? The answer determines your write latency, your consistency guarantees, and how much data you can lose if a server crashes. There are three main strategies, and knowing when to reach for each one is a key engineering judgment.

Let's walk through each strategy in plain terms, then look at how they compose in a real multi-level stack.

Write-Through — Consistency First

Write-through means: when you write data, you write it to the cache and the database in the same synchronous operation, before returning success to the client. The cache is always up to date because it gets every write immediately. The cost is that every write pays the latency of two storage operations: the cache write plus the database write. For a typical setup, that's ~0.5 ms to Redis plus ~5 ms to Postgres — roughly doubling write latency compared to bypassing the cache.

This is the right strategy for financial transactions, inventory counts, and anything where stale reads are unacceptable. The slightly higher write latency is a reasonable price to pay for the guarantee that every read from the cache reflects the latest confirmed write.

Write-Around — Protecting the Cache from Junk

Write-around means: writes go directly to the database and skip the cache entirely. The cache is only populated on cache misses during reads. This pattern makes sense when you're writing data that will rarely or never be read from the cache — for example, a bulk import of historical records, log ingestion, or append-only analytics events. Caching these writes would evict useful data from the cache for no benefit, since the written data won't be read back through the cache path.

The downside: the first read after a write is always a cache miss, which adds the database read latency. For frequently-read data, write-around is the wrong choice — you get the worst of both worlds (write goes to DB, read still misses the cache).

Write-Back — Performance at the Cost of Durability

Write-back is the most aggressive strategy: writes land in the cache immediately and return success to the client, then a background flush thread pushes dirty cache entries to the database asynchronously. This gives you the absolute lowest write latency — cache writes are microseconds to nanoseconds, vs 5–15 ms for database writes. The risk: if the cache server crashes before the flush, you lose the writes that hadn't been persisted. For counters (page views, "likes"), losing a few counts is acceptable. For financial records, it is not.

How These Strategies Compose in a Multi-Level Stack

In practice, different layers use different write strategies simultaneously. Here's how a typical e-commerce system stacks them:

The key insight from this table: different data types use different write strategies simultaneously within the same application. User profiles use write-through to Redis because consistency matters — you must not serve a user's stale name to their friends seconds after an update. View counters use write-back because losing a few counts is fine, and the throughput requirement is enormous. Static assets use write-around entirely — you never put a JS bundle in Redis or the app heap — and at CDN you effectively sidestep invalidation by using a new URL with a content hash on every deploy.

Once you understand the six layers individually, the next step is seeing how real engineering teams wire them together. At scale, you don't pick one layer — you compose several into named architectural patterns. The four patterns below represent the most-used approaches in production today, from the simple default that most teams reach for, to the sophisticated multi-region hierarchies that companies like Facebook and Netflix built to serve billions of requests.

(a) Cache-Aside (Look-Aside) — The Default Pattern

The most common pattern by a large margin. The application manages the cache explicitly: on a read, check the cache first; if it misses, fetch from the database and populate the cache; on a write, write to the database and then invalidate (or update) the cache entry. The application code is in full control — there is no caching middleware silently doing things for you.

Why it's the default: it's simple to understand, easy to debug (you can trace exactly which line populated a cache entry), and compatible with any database or cache store. Most Redis + Postgres architectures use this pattern. The downside is that the responsibility for cache correctness sits entirely in application code — if a developer forgets to invalidate after a write, the bug is silent and difficult to find in production.

(b) Read-Through — Transparent Cache Population

In read-through, the cache itself is responsible for fetching from the database on a miss — the application only ever talks to the cache. A cache library like Caffeine (Java) or a Redis module with a loader function handles the miss transparently. The application says "get me product:42" and the cache returns it, fetching from the DB if necessary, without the application seeing the DB at all.

This centralizes the cache-loading logic in one place and ensures the "populate on miss" step is never forgotten. The trade-off: it can be harder to implement correctly for complex queries, and the application loses fine-grained control over when and what gets cached.

(c) Cache-of-Caches / Hierarchical — Two-Tier Architecture

When a single shared cache cluster becomes the bottleneck, teams add a local in-process cache in front of the distributed cache. Reads check local memory first; only misses reach the shared Redis or Memcached cluster. This pattern cuts Redis round-trips dramatically for hot keys that every app server needs repeatedly.

Twitter, Facebook, and many large-scale systems use this pattern. The naming varies — "L1/L2 cache," "local cache + regional cache," "near cache" — but the structure is the same: a fast, small, in-process layer absorbs the hottest traffic, and a larger, shared, network-accessed layer handles the rest.

The code above shows the hierarchy in action. Caffeine acts as the L1 gatekeeper — if the key is in local memory, the call returns in microseconds without a network hop. If L1 misses, Caffeine's CacheLoader function fires, checks Redis (L2), and only if that misses too does it reach the database. From the caller's perspective, it's a single get call. The layered fallback is invisible.

(d) Multi-Tier Heterogeneous — Netflix EVCache Pattern

Netflix built its own globally-distributed cache layer on top of Memcached. They named it EVCache — short for "Ephemeral Volatile cache," signalling up front that the data can be lost on a node crash and that's fine, because every entry can be rebuilt from the source. EVCache is a well-documented example of a multi-tier heterogeneous cache. Netflix uses Memcached (not Redis) for raw speed on hot ephemeral data — Memcached has lower overhead per operation and better CPU efficiency for simple key-value lookups. Redis is used separately for data that needs data structures (sets, sorted sets, pub/sub). The two stores coexist in the same architecture, each used for what it does best.

EVCache adds zone-aware routing: within an AWS region, Netflix runs one Memcached cluster per availability zone. Reads prefer the local zone's cluster to minimize cross-zone latency. Writes replicate across all zones asynchronously. This means a zone failure doesn't lose cached data — other zones still hold copies — and normal reads stay within the same data center rack, keeping latency under 1 ms.

Facebook TAO — A Real-World Two-Tier Example

The social-graph data Facebook serves is mostly a giant collection of two things: objects (a user, a post, a photo) and associations between them (Alice is friends with Bob; Bob liked this photo). Facebook's caching system is literally named after those two shapes — TAO stands for "The Associations and Objects." Published in a USENIX ATC 2013 paper by Bronson et al., it is one of the most studied real-world caching architectures. TAO sits in front of MySQL and serves Facebook's social graph — friends, likes, comments — as a two-tier cache.

TAO's architecture encodes a key insight: for a globally-distributed social graph, writes need strong consistency (all writes go to the leader MySQL) but reads can tolerate brief lag (follower regions cache data that may be a few hundred milliseconds behind the leader). The two-tier design — TAO cache servers in front of MySQL, per region — means most social graph reads are served from in-region memory without any cross-Atlantic network hop. The consistency cost is bounded: when a user updates their profile, followers in EU will see the change within seconds once the async invalidation propagates. That's an acceptable trade-off for a social feed. It would not be acceptable for a bank balance.

The cache-of-caches diagram shows the steady-state topology used by many large-scale Java services: each app server has a bounded Caffeine cache (typically 5,000–50,000 entries) that absorbs the hottest fraction of keys with microsecond latency. The shared Redis cluster handles everything the L1 doesn't hold — much larger working set, shared across all instances. Only genuine misses at both layers reach the database. In a well-tuned deployment, the database might see 1–3% of total read traffic. The rest is absorbed by the two-layer hierarchy.

Knowing what layers exist and what write strategies to use is only half the battle. The other half is knowing the mistakes teams make in production — the easy-to-reach-for decisions that feel right but quietly destroy performance, introduce stale data, or cause catastrophic failures under load. Here are the seven most damaging pitfalls, each with a clear explanation of why the mistake is tempting, what goes wrong, and how to fix it.

The goal of these exercises is not to test whether you memorized the layer names. It's to build the habit of reasoning through multi-level cache decisions — the kind of thinking an interviewer is looking for when they describe a system and ask "where would you add caching?"

Theory says caches make things faster. Production says: it depends on how you configure them. The four incidents below are composite sketches drawn from real patterns — exact companies are softened, but every failure mode is documented in public postmortems or engineering blogs. Read each one to understand not just what broke, but why that specific layer interaction caused the failure.

Textbook cache descriptions make it sound like a choice between Redis and Memcached. Real production systems layer four or five different caching mechanisms simultaneously, each chosen for a specific reason. The architectures below are drawn from public engineering posts, USENIX papers, and conference talks — they show how the six-layer hierarchy plays out at scale.

Facebook TAO (USENIX ATC 2013, Bronson et al.): writes go to the leader region and propagate to followers via async invalidation. Each follower region reads from its local TAO cache backed by a MySQL replica — reads never cross regions. This means EU users always get low latency but may briefly see stale data after a write.

No two production stacks are identical. Facebook TAO skips CDN and in-process caches because social graph data is too personalized to cache at the edge. Netflix skips in-process caches because EVCache latency is already under 0.5 ms intra-VPC. Shopify leads with CDN because static content cacheable at the edge handles the bulk of flash-sale traffic.

These aren't beginner mistakes — they're the mental-model errors that experienced engineers make when reasoning about cache hierarchies. Each one is distinct from the pitfalls in S17 (which focused on implementation gotchas). These are wrong beliefs about how caches work, and correcting them changes how you design systems.

Knowing the theory is necessary but not sufficient. This section is a practical five-stage playbook for tuning a multi-level cache stack in a running production system. Follow it in order — skipping stages leads to optimizing the wrong layer.

The five stages must be followed in order. Tuning eviction before measuring hit ratios is guessing. Right-sizing before locating the bottleneck layer is optimizing the wrong thing.

Everything from this page distilled into quick-reference cards and a glossary. Bookmark this section for interview prep.

The Latency Hierarchy (commit to memory)

Key Rules of Thumb

Glossary

cache hierarchy

The ordered set of cache layers from fastest/smallest (CPU L1) to slowest/largest (CDN), each layer catching misses from the layer above.

MESI protocol

The cache coherence protocol used between CPU cores: each cache line is in one of four states — Modified, Exclusive, Shared, Invalid. Prevents two cores seeing different values for the same memory address.

false sharing

When two threads modify different variables that happen to live on the same CPU cache line, causing constant MESI invalidations even though the threads aren't sharing data logically. Solved by padding structs to align variables to separate cache lines.

page cache

The Linux kernel's in-RAM cache of recently-read or recently-written file data. File I/O goes through the page cache automatically; Postgres, for example, benefits from it without any configuration.

buffer pool

The database's own in-process cache of recently-accessed database pages. Postgres calls it shared_buffers; MySQL calls it the InnoDB buffer pool. It sits in front of the OS page cache for database-specific I/O.

working set

The subset of data actively accessed in a given time window. Once a cache is large enough to hold the working set, additional capacity yields near-zero improvement in hit ratio.

temporal locality

The tendency to re-access the same data soon after accessing it. High temporal locality makes caching effective — the same key is likely in cache when requested again.

spatial locality

The tendency to access data near recently-accessed data. Exploited by CPU cache lines (fetch 64 bytes at once) and OS page cache (fetch 4 KB at once, even if only 8 bytes needed).

miss penalty

The latency cost paid when a cache lookup fails and must fall through to the next (slower) layer. Miss penalty × miss rate = average added latency from misses.

write-through / write-around / write-back

Three strategies for handling writes in a cache. Write-through writes to both cache and backing store synchronously. Write-around skips the cache on write. Write-back (write-behind) writes to the cache and flushes to the backing store asynchronously.

cache stampede

When a popular cached item expires and many clients simultaneously attempt to rebuild it, overwhelming the backing store with identical requests. Prevented by mutex locking and TTL jitter.

TAO

Facebook's "The Associations and Objects" distributed cache system, described in the USENIX ATC 2013 paper by Bronson et al. Designed for graph data with a write-to-leader, read-from-local-cache pattern across multiple regions.

Caching Levels is one layer of a larger system design picture. Here are the six most directly connected topics — each one deepens a specific aspect of what you learned here.