Seeing inside a distributed system
With one box you could read one log file. With 10 services on 40 machines, knowing what the system is doing becomes its own engineering discipline: observability. Its three pillars:
- Logs: timestamped text records of events (
payment failed for order 812: card declined), shipped from every machine into one searchable place - Metrics: numbers over time, requests per second, error rate, latency, queue depth (your lesson 6-1 alarm), graphed on dashboards with alerts
- Traces: the story of one request as it hops across services, each hop timed, so you can see exactly where 480 of the 500 ms went
The habit you started in lesson 1-2, measure where time goes before fixing anything, is observability in miniature. Metrics tell you that something is wrong, traces tell you where, logs tell you why.
Why averages lie about latency
Ten requests, compared by average, median, and 90th percentile.
latencies = sorted([12, 15, 11, 240, 14, 13, 16, 18, 12, 300]) average = sum(latencies) / len(latencies) p50 = latencies[4] p90 = latencies[8] print("average (ms):", average) print("p50 (ms):", p50) print("p90 (ms):", p90) print("worst (ms):", latencies[-1])
Output
average (ms): 65.1 p50 (ms): 14 p90 (ms): 240 worst (ms): 300
The typical request took 14 ms, and two slow outliers dragged the average to 65. The average describes no actual request here, since nothing took anywhere near 65 ms.
That is the specific failure of an average, and it is worth naming. It is pulled by extremes, so a metric meant to describe the typical experience ends up describing neither the typical case nor the bad one.
Percentiles avoid that because they are positions in the sorted list rather than arithmetic on the values. latencies[4] is the median and no outlier can move it, which is why the number stays honest as the tail gets worse.
p90 at 240 ms against a p50 of 14 ms is the shape to recognize. A large gap between the two means a system with two behaviors rather than one slow system, which usually points at cache misses or a dependency timing out.
This is why dashboards show p50, p95, and p99 instead of averages. Reading them together tells a story, since p50 is the experience you designed for and p99 is the experience that generates support tickets.
Note that percentiles need care when combined across machines, and that trips people up. Averaging the p99 of ten servers does not give the fleet's p99, so real monitoring systems keep histograms rather than pre-computed percentiles.
An error-rate alert
140 errors out of 200,000 requests, checked against a 0.05% threshold.
requests = 200000 errors = 140 error_rate = errors / requests * 100 print("error rate:", str(round(error_rate, 3)) + "%") print("alert!" if error_rate > 0.05 else "healthy")
Output
error rate: 0.07%
alert!Seven hundredths of a percent triggers an alert, which looks paranoid until you read it as 140 people. Error budgets are tight because a rate that sounds negligible is a real number of failed requests.
The rate rather than the count is what belongs in the threshold, and that is the design decision here. A hundred and forty errors is alarming at 200,000 requests and unremarkable at 20 million, so alerting on the count would fire constantly during traffic peaks.
round(error_rate, 3) keeps three decimals because the interesting values live there. Rounding to a whole percent would display 0% and hide exactly the condition being watched.
The conditional expression "alert!" if error_rate > 0.05 else "healthy" is Python's ternary, and it reads in the order value, condition, alternative. It suits a two-way choice like this and gets unreadable with more branches.
Note what a threshold like this cannot see, which is a partial failure. A single broken endpoint can be 100% failing while the site-wide rate stays under the threshold, which is why real monitoring breaks the rate down by endpoint and by service.
What to check when the average looks fine
The p95 and p99 latency, which reveal the slow tail the average hides.
You just saw a 14 ms median coexist with a 65 ms average and 300 ms outliers, so a healthy-looking 40 ms average is entirely compatible with a badly slow tail.
Averages bury the tail, and at scale the tail is busy. At a million requests a day, a bad p99 means 10,000 slow experiences daily, which is plenty to generate complaints while every dashboard stays green.
The users reporting the problem are also not random, which makes the tail worse than it looks. Slow requests concentrate on users with the most data, the largest carts, or the least cached profiles, so the same people hit the tail repeatedly and conclude the app is broken.
Trust the reports over the dashboard when they disagree, because the reports are measurements too. A metric that contradicts consistent user experience is usually the wrong metric rather than wrong users.
Latency questions are percentile questions. Getting into the habit of asking at which percentile whenever someone quotes a latency number is a small change that prevents a lot of wasted investigation.
How many slow requests a bad p99 produces
1% of 2,000,000 is 20,000 requests a day experiencing 3-second-or-worse responses.
p99 means 99% are faster, so 1% are at or beyond it. The median user gets 30 ms, and twenty thousand requests a day are a hundred times slower than that.
Twenty thousand is the number that makes the case, since it is not an edge case at that volume. If those requests belong to even a few thousand distinct users, that is a steady stream of complaints about an app whose average latency is excellent.
Note that p99 is a floor rather than a description of those requests. Some of the 20,000 took 3 seconds and others took 30, and the p99 figure says nothing about how far the tail extends, which is why p99.9 and the max are also worth watching.
That is why teams set targets on p99 rather than averages. A service level objective phrased as p99 under 500 ms is a promise about the worst common experience, which is the thing users actually notice.
Common causes of a long tail are all things this course has covered: cache misses from unit 3, replication lag detours from unit 4, and queue backlogs from unit 6. Garbage collection pauses and a single slow shard from lesson 5-2 round out the usual list.