Stateless services

The state problem

Work through the round-robin setup from lesson 2-2 with real users and something breaks immediately: logins.

State is any data a server remembers between requests. The classic example is a session: after login, the server stores "session42 belongs to ada" so later requests with that session ID are trusted.

If server A stores that session in its own memory, and the load balancer sends ada's next request to server B, then B has never heard of session42. Ada is suddenly logged out, but only sometimes, depending on where the LB sends her. These bugs are miserable to debug.

A service is stateless when any server can handle any request, because servers keep nothing important in local memory. Stateless services are what make horizontal scaling safe.

The broken setup and the fix, side by side

Each server has private memory, so a login on A is invisible to B, and a shared store fixes it.

server_memory = {"A": {}, "B": {}}
server_memory["A"]["session42"] = "ada"
print("Server B sees session42:", server_memory["B"].get("session42"))

shared_store = {"session42": "ada"}
print("Shared store sees session42:", shared_store["session42"])

Output

Server B sees session42: None
Shared store sees session42: ada

None on the first line is the logged-out user. Ada authenticated successfully a moment ago, and the server now handling her request has no record of it, so she is asked to log in again.

.get() returns None instead of crashing when a key is missing, which is exactly what a real session lookup does. That is why the bug presents as an unexpected logout rather than a 500, and why it is easy to miss in testing.

The nested dict is the honest picture of separate processes. Nothing connects server_memory["A"] to server_memory["B"], just as nothing connects the memory of two machines in a rack.

The fix is one line, and moving the data outside both servers is the entire idea of statelessness. Neither server owns the session, so it does not matter which one the load balancer picked.

Note how intermittent this bug is with round robin across three servers. Roughly one request in three works, which means it reproduces locally almost never and in production constantly, and that combination is what makes it miserable to debug.

Where state goes instead

Three standard homes for state that used to live in server memory:

1. The database. Anything durable (users, posts, orders) was already here
2. A shared in-memory store like Redis: a tiny, very fast key-value database all app servers share. Perfect for sessions. You will meet Redis again as a cache in unit 3
3. The client. Signed tokens (like JWTs) let the browser carry its own proof of login, so servers store nothing

Some things resist this, like websocket connections for live chat, and we will handle that honestly in the chat capstone (lesson 10-2).

The unit 2 recipe, used by nearly every web backend on earth: load balancer, N stateless app servers, shared database and session store.

Which setup scales from 3 servers to 30

Servers that keep no local state and read sessions from Redis.

Stateless servers are interchangeable, so adding more just works. The thirtieth server needs no data migration and no special handling, because it can answer any request the moment it starts.

Local session memory breaks logins, which is this lesson's bug, and it gets worse with more servers rather than better. At 30 servers a user's session is on the right machine one time in thirty.

Local files vanish for other servers, which is the same problem wearing different clothes. An upload written to disk on server A is a 404 when the next request lands on B, and the fix is the same, meaning shared storage.

Splitting the user table across app servers means each server can only serve some users, and now the load balancer has to know which. That couples routing to data layout and gives up the interchangeability that made scaling easy.

State belongs in shared stores behind the app tier. The app servers become pure request handlers, and every hard question about data lives in exactly one place.

Sizing the fleet with room to lose one

The answer is 10 servers.

Covering 1,700 RPS at 200 RPS per server needs ceil(1700 / 200) = 9 servers. Dividing gives 8.5, and rounding up is not optional, since 8 servers cover only 1,600 RPS and the remaining 100 arrive anyway.

To survive one failure at peak you need 9 healthy servers even when one is dead, so run 10. The spare is not idle, since all ten share the traffic during normal operation and each one runs at 85% of what it could do.

This is called N+1 provisioning, and it only works because the servers are stateless and interchangeable. Losing any one of the ten is the same event, which is what makes a single spare sufficient.

Real fleets often go further and provision N+2, or spread across availability zones so a whole data center failing is survivable. The arithmetic is identical, and only the definition of one failure changes.