Hot keys

When hashing is not enough

Hash sharding spreads keys evenly, but not necessarily traffic. If one key is vastly more popular than the rest, the shard that owns it melts while its neighbors idle. That key is a hot key, and it is a real, recurring production problem.

Classic examples:

• A celebrity's profile on a social network (one user_id, millions of readers)
• A viral post or video
• A flash-sale product page

No resharding scheme fixes this, because the load is concentrated in a single key that must live somewhere. The standard remedies:

1. Cache it. Unit 3 to the rescue: a popular key is exactly what caches are best at
2. Split the key. Store celebrity#1, celebrity#2, ... on different shards and pick one at random per read
3. Replicate the hot data extra times and spread reads across the copies

One popular key against three thousand normal ones

Nine thousand requests hit one celebrity key and a thousand spread across a thousand normal users.

import hashlib
from collections import Counter

def shard_of(key):
    return int(hashlib.md5(key.encode()).hexdigest(), 16) % 4

requests = ["celebrity"] * 9000 + ["user" + str(i) for i in range(1000)]
load = Counter(shard_of(k) for k in requests)
for shard in sorted(load):
    print("shard", shard, "handled", load[shard], "requests")

Output

shard 0 handled 9216 requests
shard 1 handled 248 requests
shard 2 handled 279 requests
shard 3 handled 257 requests

The 1,000 normal users split evenly at about 250 per shard, which is the hash function working perfectly. Every celebrity request lands on shard 0, which is also the hash function working perfectly.

That is the uncomfortable point of this output. Nothing is broken, the distribution is doing exactly what lesson 5-1 promised, and the load is still catastrophically uneven.

The hash function balances distinct keys and knows nothing about how often each key is read. One key is one key whether it is requested once or nine thousand times, so popularity is invisible to the scheme.

Counter from collections tallies occurrences in one pass, and it is worth reaching for whenever you need a frequency count. Doing it by hand with a dict works and takes four more lines.

Note that shard 0 also owns its share of normal users, which is why the count is 9,216 rather than exactly 9,000. The hot key is on top of a normal workload, not instead of one.

What share of traffic one shard absorbed

9,216 / 10,000 is 92% of all traffic on one of four shards, while the others sit near 2.5% each.

Even with perfect hash sharding, one popular key defeats the distribution. Adding shards makes the imbalance worse in relative terms, since the other shards get quieter and shard 0 does not.

Look at what this means for capacity planning. Four shards give you the write and read capacity of roughly one shard, so three quarters of the hardware you paid for is idle while the fourth is failing.

The failure is also localized in a confusing way. Requests for the celebrity time out, and so do requests for the ordinary users who happen to live on shard 0, so the symptom looks like a partial outage with no obvious pattern.

The first fix is almost always the unit 3 cache, because a single hot key with an unchanging value is the ideal cache resident. Ninety-two percent of traffic asking for the same value is a 92% hit rate waiting to be collected.

Handling a flash sale with what you already have

Serve the product page from the cache and CDN, so the hot shard is barely touched.

A hot page with identical content for every viewer is the perfect cache and CDN resident. One database read per TTL window can serve millions of viewers, so a 10-second TTL turns a million requests into six database reads a minute.

The CDN half matters as much as the cache half. Static content served from an edge location near the user never reaches your infrastructure at all, so the traffic is absorbed before it becomes your problem.

Resharding cannot help concentration on one key, and it is worth being firm about why. The key has to live on some shard, so no assignment scheme moves load off it, and lesson 5-1's expensive migration would buy nothing.

A permanent dedicated database is heavy machinery for a one-hour spike. It might be right for a permanently hot dataset, and the flash sale ends, so the operational burden would outlive the problem.

Note the general pattern here, because it recurs. When load concentrates on a small amount of unchanging data, the answer is almost always a copy closer to the reader rather than a rearrangement of the source.