Cold starts, honestly

The catch

Salespeople skip this slide, so here it is. When an event arrives and no copy of your function is warm, AWS must create one: provision a micro-VM, load your code, start the runtime. That pause is a cold start, and the request that triggered it waits through all of it: commonly 100 to 1000 ms, worse for big deployment packages and heavy runtimes, and it happens again per concurrent copy when traffic jumps.

After the run, AWS keeps the copy warm for a while, so the next event skips the pause. Under steady traffic most invocations are warm, and cold starts hide in the slowest fraction of requests, which is exactly where users notice them. Engineers measure that slow fraction with percentiles: sort all response times, and p99 is the value that 99% of requests come in under, leaving only the slowest 1% above it. A cold start barely moves the average, but it shows up instantly at p99, which is why latency budgets are usually written as percentiles, not averages.

Honest guidance: for background jobs and pipelines like lesson 7-2's thumbnails, cold starts are irrelevant. For latency-sensitive user-facing APIs, they are a real cost you weigh against never patching a server, and mitigations (like paying for pre-warmed capacity) exist but cancel some of the savings.

One cold start hiding from the median

Ten requests hit a function, and one of them paid an 812 ms cold start. With only 10 samples, the sorted list's 5th value stands in for the median and the 10th is the worst case.

latencies="15 14 16 15 812 15 14 16 15 15"
sorted=$(printf '%s\n' $latencies | sort -n | paste -sd' ' -)
echo "sorted ms: $sorted"
sum=0; for l in $latencies; do sum=$((sum + l)); done
echo "average: $((sum / 10)) ms"
echo "median (p50): $(echo $sorted | cut -d' ' -f5) ms"
echo "worst (p100): $(echo $sorted | cut -d' ' -f10) ms"

Output

sorted ms: 14 14 15 15 15 15 15 16 16 812
average: 94 ms
median (p50): 15 ms
worst (p100): 812 ms

Three numbers describe the same ten requests and tell three different stories. The median says 15 ms, which is what nine users out of ten actually experienced. The worst case says 812 ms, which is what one user experienced.

The average, at 94 ms, describes nobody. No request took 94 ms, and no user would recognize that number. That is the case for percentiles over averages in one line: the average is dragged by the outlier without ever pointing at it, while p50 and p100 together tell you both that the service is fast and that something occasionally is not.

The weighted average cost of cold starts

If 2 of every 100 requests hit an 800 ms cold start and the rest take 15 ms, this computes the weighted average.

cold_ms=800
warm_ms=15
cold_per_100=2
sum=$((cold_per_100 * cold_ms + (100 - cold_per_100) * warm_ms))
printf 'average latency: %d.%02d ms\n' $((sum / 100)) $((sum % 100))

Output

average latency: 30.70 ms

Two percent of requests roughly doubled the average, from 15 ms to 30.7 ms. That is the arithmetic reason cold starts get argued about: a small fraction of slow requests has an outsized effect on any single summary number.

The printf prints integer division and remainder as a decimal, because bash has no floating-point arithmetic. sum / 100 gives the whole milliseconds and sum % 100 gives the hundredths.

The same calculation for a heavier function

A heavier function cold-starts in 1200 ms, runs warm in 20 ms, and after a marketing push 5 of every 100 requests are cold.

cold_ms=1200
warm_ms=20
cold_per_100=5
sum=$((cold_per_100 * cold_ms + (100 - cold_per_100) * warm_ms))
printf 'average latency: %d.%02d ms\n' $((sum / 100)) $((sum % 100))

Output

average latency: 79.00 ms

What changed

• All three variables move: 1200, 20, and 5. The formula is untouched.
• The average nearly tripled, from 30.7 ms to 79 ms, even though warm requests barely changed at all. The cold fraction is doing essentially all of the damage.
• Both inputs moved in the same direction at once, which is the realistic case. A traffic surge raises the cold fraction, and a heavier deployment package raises the cold start duration, so the two effects compound rather than trading off.

The workload that should worry least

The nightly 02:00 report job from lesson 7-2 should worry least about cold starts.

Nobody is waiting on it. An extra second at 02:00 is invisible, and even an extra ten seconds would be, because the only consumer of the output reads it hours later.

Compare that with the alternatives. A checkout API has a user watching a spinner and a measurable drop-off for every extra second. A trading API has a latency budget written in the requirements. And a burst of traffic after a TV ad is the worst case of all, because a sudden wave of concurrent requests means many copies cold-starting at once, so a large share of those users hit the slow path simultaneously.

The general rule falls out of this: cold starts matter in proportion to how directly a human is waiting on the response.