The cache-aside pattern
Lesson 3-1 said caches hold copies, but not who puts the copies there or when. That decision is a caching pattern, and choosing one is the first thing you do when adding a cache to real code. Cache-aside (also called lazy loading) is the most common pattern in production backends, and you can write it in six lines. On every read:
- Look in the cache first
- Hit: return the cached value, done
- Miss: read from the database, store a copy in the cache, return it
The first request for any key is slow (a miss), and every request after that is fast, until the entry is removed. The cache fills itself with exactly the data people actually ask for, which is why it is called lazy.
In the simulation below, a Python dict named cache plays the role of Redis and a dict named db plays the database. The pattern is identical in production, just with network calls instead of dict lookups.
Cache-aside in six lines
Watch the first read of each user miss, and every repeat hit.
db = {"user:1": "Ada", "user:2": "Grace"}
cache = {}
def get_user(key):
if key in cache:
print("cache hit:", key)
return cache[key]
print("cache miss:", key, "-> reading database")
value = db[key]
cache[key] = value
return value
get_user("user:1")
get_user("user:1")
get_user("user:2")
get_user("user:1")Output
cache miss: user:1 -> reading database cache hit: user:1 cache miss: user:2 -> reading database cache hit: user:1
Four calls produced two database reads, and the last line is the payoff. user:1 was requested three times and read from the database once.
The three lines after the miss message are the entire pattern: read the source, store a copy, return it. Storing before returning is what makes the next request a hit, and forgetting that one line gives you a cache that never fills.
cache[key] = value writing to the cache on a read path is the part that surprises people. Reads mutate the cache, which is why cache-aside is also called lazy loading, since the cache populates itself from actual demand rather than from a guess.
A fifth call to get_user("user:2") would hit, since that key was already loaded. Every key follows the same one-miss-then-hits shape.
"user:1" as a key follows the usual convention of type:id, and it is worth adopting. The prefix makes keys readable in a debugging session and lets you reason about which group of keys to clear.
Note what this simple version omits, which is expiry and failure handling. Entries here live forever and the database is assumed to answer, and the next two lessons are about both of those gaps.
Measuring the hit rate
The same pattern with counters, over a request sequence with repeats.
db = {"u1": "Ada", "u2": "Grace", "u3": "Linus"}
cache = {}
hits = 0
misses = 0
requests = ["u1", "u2", "u1", "u1", "u3", "u2", "u1"]
for key in requests:
if key in cache:
hits += 1
else:
misses += 1
cache[key] = db[key]
print("hits:", hits)
print("misses:", misses)
print("hit rate:", str(round(hits / len(requests) * 100)) + "%")Output
hits: 4 misses: 3 hit rate: 57%
The first appearance of each of u1, u2, and u3 is a miss, and everything after is a hit, so misses equal the number of distinct keys. That relationship is the most useful thing in this block, since it means the hit rate depends on how much traffic repeats rather than on anything about the cache.
Fifty-seven percent looks disappointing next to the 90% from lesson 3-1, and the reason is the sample size. Three distinct keys out of seven requests is a lot of variety, and real traffic over an hour asks for the same popular items thousands of times.
That is also why the hit rate climbs on its own as a cache runs. The fixed cost of one miss per key is paid once and amortized over every later request, so a longer window shows a better number.
hits / len(requests) is the definition worth remembering, meaning hits over total requests rather than hits over misses. Getting that ratio backward is a common slip when reading a dashboard.
Note that this cache is unbounded, and it grows by one entry per distinct key forever. Real caches have a memory limit and must throw something away, which is the eviction problem in lesson 3-4.
The first request after a fresh deploy
A cache miss, so the database is read and the cache is filled for later requests.
An empty cache is called a cold cache, and every first read is a miss that fills it. Latency during that period is the uncached latency, which is the 25 ms from unit 1 rather than the sub-millisecond hit.
This is also why restarting a cache in production is scary. All that traffic suddenly hits the database at once until the cache rewarms, so a 90% hit rate becoming 0% for a minute means ten times the database load arriving instantly.
That failure has a name worth knowing, which is a thundering herd. Many simultaneous misses for the same key all read the database at the same time, and the database can be overwhelmed by requests that a warm cache would have answered for free.
Some teams pre-fill or warm caches before big events for exactly this reason. A script reads the popular keys before traffic arrives, paying the miss cost when nobody is waiting, which is standard practice before a product launch or a sale.