Replication: leader and followers

One database is a single point of failure

Unit 2 gave you many app servers, but they all talk to one database box. If its disk dies, your product and its data are gone. That is unacceptable, so databases are replicated: full copies kept on multiple machines.

The standard arrangement is leader-follower (also called primary-replica):

• The leader is the only node that accepts writes
• Every write is recorded in an ordered replication log, and each follower replays that log to stay a faithful copy
• If the leader dies, a follower is promoted to be the new leader, a process called failover

Why one leader? If two nodes accepted writes independently, they could both change the same row at once and disagree forever. Funneling writes through one node sidesteps that conflict entirely. You pay for it with a write ceiling, which unit 5 addresses.

Replaying the replication log

The leader records every write in order, and a follower applies them one by one.

leader_log = ["INSERT post 1", "UPDATE post 1", "INSERT post 2"]
follower_state = []
for entry in leader_log:
    follower_state.append(entry)
    print("follower replayed:", entry)
print("follower is a faithful copy:", follower_state == leader_log)

Output

follower replayed: INSERT post 1
follower replayed: UPDATE post 1
follower replayed: INSERT post 2
follower is a faithful copy: True

Order is the whole trick, because both nodes apply the same log in the same order, so the follower ends as an exact copy. Swap entries one and two and the follower would try to update a post that does not exist yet.

That the log is a list rather than a set is the essential detail. Replication ships an ordered sequence of changes rather than a bundle of final values, which is what makes it work for updates and deletes as well as inserts.

follower_state == leader_log is the invariant a real system checks continuously, usually by comparing how far each follower has replayed. That position is called replication lag, and it is the subject of the next lesson.

Removing leader_log entries one at a time shows the follower tracking whatever the log contains. The follower has no opinions and no logic of its own, and that is deliberate, since a follower that decided anything for itself could diverge.

Note that this replay is also how backups and point-in-time recovery work. Keeping the log means any past state can be reconstructed by replaying up to a chosen entry, which is why the log matters beyond replication.

Why all writes go through one leader

So concurrent writes to the same row cannot happen on two nodes and conflict.

One write path means one authoritative order of changes, so replicas can simply replay the log and always agree. The order is decided in exactly one place, which is what makes the previous block's invariant hold.

Consider what two leaders would mean for one row. Node A sets a price to 10 and node B sets it to 12 at the same moment, both accept their write, and now the two nodes disagree with no fact in the system saying which is correct.

Multi-leader systems exist and must detect and resolve conflicting writes, which is genuinely hard. The resolutions available are all unsatisfying, meaning last-write-wins discards data, and merging requires the application to define what merging means.

Single-leader is the default because it trades some write capacity for a much simpler correctness story. The trade is explicit, since the leader's throughput is your write ceiling, and unit 5 is about what to do when you hit it.

Counting write downtime during failover

10 seconds to notice plus 20 seconds to promote gives 30 seconds without writes.

Note what did not happen, since the absence is the point. No data was lost because the followers had the log, and reads could continue from the surviving followers throughout.

Compare that with unit 1, where the same disk failure destroyed the product. Replication converts a catastrophe into a 30-second incident, which is the single largest reliability improvement in this course.

Detection time is the tunable half of that 30 seconds and it has a floor. Checking more aggressively shortens the outage and raises the risk of promoting a follower because of a brief network hiccup, which produces two nodes believing they are the leader.

That failure has a name worth knowing, which is split brain, and it is why real failover involves a majority vote rather than a single monitor's opinion. Lesson 7-2 covers the quorum machinery that makes such a vote safe.