Seeing inside a distributed system
With one box you could read one log file. With 10 services on 40 machines, knowing what the system is doing becomes its own engineering discipline: observability. Its three pillars:
- Logs: timestamped text records of events (
payment failed for order 812: card declined), shipped from every machine into one searchable place - Metrics: numbers over time, requests per second, error rate, latency, queue depth (your lesson 6-1 alarm), graphed on dashboards with alerts
- Traces: the story of one request as it hops across services, each hop timed, so you can see exactly where 480 of the 500 ms went
The habit you started in lesson 1-2, measure where time goes before fixing anything, is observability in miniature. Metrics tell you that something is wrong, traces tell you where, logs tell you why.
Code exercise · python
Run this latency analysis of 10 requests. Averages hide problems: compare the average against the median (p50) and the 90th percentile (p90). Percentile p90 means 90% of requests were at least this fast.
Code exercise · python
Your turn: build an error-rate alert. Out of 200,000 requests, 140 returned errors. Print the error rate as a percent rounded to 3 decimals with a % sign, then print "alert!" if it exceeds 0.05 percent, otherwise "healthy".
Quiz
Users report the app feels slow, but your dashboard shows average latency is a healthy 40 ms. What should you check next?
Problem
A service handles 2,000,000 requests per day with a p99 latency of 3 seconds (and p50 of 30 ms). Roughly how many requests per day are as slow as 3 seconds or worse?