Failure spreads through calls
Lesson 8-1 warned that microservices turn function calls into network calls. Here is the concrete disaster that follows, the one behind many famous outages. The recommendations service gets slow, taking 30 seconds instead of 20 ms. The product page service calls it and waits. Waiting requests hold threads and connections, so the product page service runs out of capacity, and the home page service calling it starts waiting too. One sick service has taken down three healthy ones. That is a cascading failure, and the root cause was not the slowness, it was the unlimited waiting.
Three defenses, applied at every network call between services:
- Timeout. Never wait longer than a budget, like 200 ms. A slow dependency becomes a fast, explicit error instead of a held thread
- Fallback. When the call fails, degrade instead of erroring the whole page: render the product page without the recommendations strip. Most users never notice, and the page stays up
- Circuit breaker. After several consecutive failures, stop calling the sick service at all for a while and go straight to the fallback. That sheds load off the struggling service so it can actually recover, the same logic that made lesson 6-2 back off its retries. Periodically one trial call is let through, and if it succeeds the circuit closes again
A circuit breaker
After 3 consecutive failures the circuit opens, and later calls serve the fallback immediately.
FAILURE_LIMIT = 3 consecutive_failures = 0 circuit_open = False calls = ["ok", "fail", "fail", "fail", "ok", "ok"] for i, result in enumerate(calls, 1): if circuit_open: print("call", i, "-> skipped, circuit open, fallback served") continue if result == "ok": consecutive_failures = 0 print("call", i, "-> success") else: consecutive_failures += 1 print("call", i, "-> failure", consecutive_failures, "of", FAILURE_LIMIT) if consecutive_failures >= FAILURE_LIMIT: circuit_open = True print("circuit opened: stop calling the sick service")
Output
call 1 -> success call 2 -> failure 1 of 3 call 3 -> failure 2 of 3 call 4 -> failure 3 of 3 circuit opened: stop calling the sick service call 5 -> skipped, circuit open, fallback served call 6 -> skipped, circuit open, fallback served
Calls 5 and 6 would have succeeded, and the breaker skipped them anyway. That is the deliberate trade, meaning a short window of unnecessary fallbacks in exchange for never hammering a struggling dependency.
consecutive_failures = 0 on success is what makes the counter measure a sustained problem rather than a total. Occasional isolated failures are normal in a distributed system, and only a run of them indicates a sick dependency.
The skipped calls cost nothing, which is the operational benefit. An open circuit fails in microseconds instead of waiting for a timeout, so the caller's threads stay free and the cascade from the intro cannot start.
Real breakers have a third state between open and closed, usually called half-open. After a cooldown, one trial call is allowed through, and its result decides whether to close the circuit or stay open, which is how the system recovers without a human.
Note that the breaker needs a fallback to be useful. Opening the circuit converts a slow failure into a fast one, and only the fallback turns it into a working page, which is why the two are always designed together.
A worst-case latency budget
Three hops, each with a 2-second timeout and one retry.
timeout_s = 2 retries = 1 hops = 3 per_hop = timeout_s * (1 + retries) print("Worst case per hop (s):", per_hop) print("Worst case for the request (s):", per_hop * hops)
Output
Worst case per hop (s): 4 Worst case for the request (s): 12
timeout_s * (1 + retries) is the per-hop figure, and the 1 + is the original attempt. Forgetting it is the standard off-by-one in this calculation and understates every budget by one timeout.
Twelve seconds from three 2-second timeouts is the number worth sitting with. Nobody chose a 12-second budget, and it emerged from three local decisions that each looked conservative.
Multiplying by hops assumes the calls are sequential, which is the common case and not the only one. Parallel calls take the slowest rather than the sum, so restructuring a chain into a fan-out is one real way to shrink the worst case.
Note that the user is gone long before 12 seconds. Browsers, load balancers, and mobile clients all have their own timeouts, so the request is usually abandoned at 30 seconds or less while the servers keep working on it.
That abandoned work is its own problem, and it is why deadlines matter more than timeouts. Without a deadline propagated down the chain, every service keeps grinding on a request whose caller stopped listening.
Budgets compound
Twelve seconds of worst case from three innocent 2-second timeouts is the lesson: timeout budgets compound along call chains. Deep chains therefore need small per-hop budgets, very few retries, and ideally an overall deadline passed down the chain, with each service subtracting the time already spent before calling the next. It is also one more quiet argument for lesson 8-1's honesty about microservices: every service boundary you add is another place where timeouts, fallbacks, and breakers must be designed, tested, and monitored.
In an interview, saying "every cross-service call gets a timeout, a fallback, and a circuit breaker" the moment you draw your first service-to-service arrow is exactly the failure story lesson 9-2 says to volunteer before being asked.
What a well-designed checkout does when payments is down
It fails fast through its open circuit breaker and shows a clear try-again-shortly message, while the rest of the site stays healthy.
Timeout plus breaker means checkout fails fast and stays contained. No held threads, no cascade into the rest of the site, and no retry storm hammering payments while it is trying to recover.
Containment is the property to emphasize, and it is the whole point of the lesson. Browsing, search, and the product pages keep working, so a payments outage costs the checkout flow instead of the entire site.
Note that payments gets an honest error message rather than a silent fallback, because money is not a feature you degrade quietly. This is unit 7's CP reasoning appearing at the service level, since an unclear outcome on a payment is worse than a clear failure.
The message wording is part of the design rather than an afterthought. Try again shortly tells the user the order was not placed and that retrying is safe, which prevents the duplicate-submission behavior that idempotency keys then have to clean up.
Compare the alternative that most systems ship by default, which is checkout hanging for 30 seconds and then erroring. Same outcome for the order, far worse for the user, and it consumes the capacity that kept the rest of the site alive.