Read replicas and replication lag

Followers can earn their keep

Followers exist for safety, but they hold a full copy of the data, so why not let them answer read queries? Used this way a follower is called a read replica.

Most apps read far more than they write. A social feed might see 100 reads per write. Send all writes to the leader and spread reads across three replicas, and you have roughly quadrupled read capacity without touching your schema.

The catch: the replication log takes time to ship and replay, typically milliseconds, sometimes seconds under load. That delay is replication lag, and it means a replica is always slightly behind the leader. A read from a replica can return data that is a moment out of date. Whether that matters depends entirely on the query, and the next simulation makes the problem concrete.

Replication lag, made concrete

The leader has applied 3 writes and the follower only 2, so each returns a different balance.

leader_log = ["balance=100", "balance=80", "balance=95"]
follower_applied = 2
print("Leader has applied", len(leader_log), "writes ->", leader_log[-1])
print("Follower has applied", follower_applied, "writes ->", leader_log[follower_applied - 1])
print("Replication lag:", len(leader_log) - follower_applied, "write(s)")

Output

Leader has applied 3 writes -> balance=95
Follower has applied 2 writes -> balance=80
Replication lag: 1 write(s)

leader_log[-1] is the last element, meaning the newest write, and the follower reads one position earlier. Both answers are values the system genuinely held, and only one of them is current.

Note that the follower is not wrong so much as behind. It holds a consistent past state rather than a corrupted one, which is an important distinction, since a lagging replica never invents data.

Measuring lag in writes is the clearest way to see the mechanism, and production systems usually report it in seconds. Both measures matter, since a replica one write behind on a quiet system may be seconds behind by the clock.

Look at the specific numbers to see why this is dangerous for money. The balance went 100, then 80, then 95, so a stale read of 80 is neither the current value nor the original one, and any decision made on it is based on a moment that has passed.

Lag is also not constant, which is what makes it hard to reason about. It is milliseconds on an idle system and can stretch to seconds or minutes during a heavy write burst, exactly when correctness matters most.

Read your own writes

The classic lag bug: Ada posts a comment (a write, to the leader), the page reloads and fetches comments (a read, from a lagging replica), and her comment is missing. She assumes the site ate it and posts again. Now it is there twice.

The standard fix is read-your-own-writes routing: for a short window after a user writes, send that user's reads to the leader, where the write definitely exists. Everyone else can keep reading replicas, because they never knew the comment existed and cannot miss it.

This is your first taste of a theme unit 7 makes precise: replicas that receive changes with a delay are eventually consistent, and systems are designed around who can tolerate how much staleness.

Routing a read to the leader

read(where) returns the newest entry from the leader, or the follower's latest applied value otherwise.

leader_log = ["balance=100", "balance=80", "balance=95"]
follower_applied = 2

def read(where):
    if where == "leader":
        return leader_log[-1]
    return leader_log[follower_applied - 1]

print("Read your own write from the replica:", read("replica"))
print("Read your own write from the leader:", read("leader"))

Output

Read your own write from the replica: balance=80
Read your own write from the leader: balance=95

The routing decision is one if, and that is genuinely the shape of the fix in production. Read-your-own-writes is a routing rule rather than a new mechanism, so no data model changes.

The two output lines are the same query answered two ways, both from a healthy system. That is the uncomfortable part of replica reads, since correctness now depends on where a query was sent.

In real code the where argument is not passed by hand, and it is derived. A common implementation records the timestamp of a user's last write in their session and routes their reads to the leader for a few seconds after it.

The cost is leader load, which is the thing replicas existed to reduce. Sending every read to the leader for safety gives back all the capacity you gained, so the window has to be short and applied only to users who just wrote.

Note that the same trick has a more precise variant worth knowing. Instead of a time window, the client remembers the log position of its write and the router picks any replica that has replayed at least that far, which gives correctness without pinning traffic to the leader.

Which query is safe on a lagging replica

Rendering the public list of a restaurant's reviews.

Public review lists tolerate seconds of staleness, and nobody can tell. A reader who sees 47 reviews instead of 48 has no way to know a newer one exists, and no decision they make depends on it.

The withdrawal check is unsafe because it risks approving overdrafts on stale balances. A replica showing the pre-withdrawal balance can authorize a second withdrawal of money that is already gone, and the loss is real.

The just-submitted form hits the read-your-own-writes bug from this lesson. The user knows what they wrote, so they are the one person who can detect the staleness, which is what makes it a visible bug rather than an invisible one.

The fresh login code may not have replicated yet, and the failure is total rather than cosmetic. A login code written to the leader a second ago and read from a replica does not exist, so a valid code is rejected.

Route by staleness tolerance, meaning money and own-writes to the leader, and everything else to replicas. The question to ask for each query is who would notice and what it would cost them.