Context switches, waiting, and the cost of it all

What a context switch saves

To pause a process mid-instruction-stream and resume it later as if nothing happened, the OS must save its context: every CPU register, the program counter from lesson 1-2, and memory-management state. Restoring another process's context reloads all of that.

Each switch costs on the order of microseconds, plus a hidden tax. The CPU's fast caches are full of the old process's data, so the new one starts slow while it refills them.

Thousands of unnecessary switches per second add up. That is why locks queueing threads, from lesson 5-3, and having far more busy threads than cores can actually make software slower rather than faster.

Turn-taking is cheap enough to feel free, and it is never actually free.

The scheduler's best trick, skipping the waiters

Recall the process states from lesson 3-1: running, ready, and waiting. The scheduler only gives slices to ready processes, so a process that is waiting, whether for a disk read, a network reply, or a time.sleep, costs nearly zero CPU.

time.sleep(0.2) does not spin the CPU for 0.2 seconds. It tells the OS to wake the process in 0.2 seconds and the process leaves the ready queue entirely.

This distinction decides how to scale a program. Adding threads helps an I/O-bound workload because the extra threads are mostly blocked, while adding threads beyond the core count to a CPU-bound workload only adds context switches. The comparison below measures both cases.

A busy loop next to a sleep

One block works the CPU hard, the other asks the OS to be woken later.

import time

start = time.perf_counter()
total = 0
for i in range(1_000_000):
    total += i
cpu_time = time.perf_counter() - start

start = time.perf_counter()
time.sleep(0.2)
wait_time = time.perf_counter() - start

print("busy loop result:", total)
print("sleep took at least 0.2s:", wait_time >= 0.2)
print("the CPU was working during the loop and idle during the sleep")

Output

busy loop result: 499999500000
sleep took at least 0.2s: True
the CPU was working during the loop and idle during the sleep

The busy loop makes the CPU perform a million additions, while the sleep makes it do nothing at all, since the OS simply does not schedule this process until the timer fires.

time.perf_counter() is a high-resolution stopwatch. The code compares durations rather than printing them, because exact timings differ on every run and every machine.

Wall-clock time versus CPU time

Two different clocks measured across the same sleep.

import time

start = time.perf_counter()
cpu_start = time.process_time()
time.sleep(0.3)
wall = time.perf_counter() - start
cpu = time.process_time() - cpu_start

print("wall clock time at least 0.3s:", wall >= 0.3)
print("CPU time under 0.05s:", cpu < 0.05)

Output

wall clock time at least 0.3s: True
CPU time under 0.05s: True

time.perf_counter measures wall-clock time, what a stopwatch would show, while time.process_time counts only the time the CPU actually spent running this process. A 0.3-second sleep passes on the stopwatch and costs almost zero CPU, because the scheduler never ran the process.

This gap is exactly what profilers report for I/O-bound programs: lots of elapsed time and almost no compute. Replacing the sleep with the busy loop from the previous block makes the two measurements nearly equal, which is the signature of a CPU-bound program.

The cost of a thread that is waiting

A web server thread that spends 95% of its life waiting for database replies costs the CPU almost nothing while it waits, because blocked threads leave the ready queue and are simply not scheduled.

Blocked means off the ready queue. The scheduler never wastes a slice on it, and the OS wakes it only when the reply arrives.

This is why a modest server can juggle thousands of mostly-waiting connections. The limiting resource in that design is memory for the per-thread stacks rather than CPU, which is also why async frameworks exist to make the per-connection cost smaller still.

Naming the save-and-restore of CPU state

Saving one process's registers and program counter and then loading another's so it can resume is a context switch.

The OS saves the running process's context, meaning its registers, program counter, and memory state, picks the next ready process, and restores that one's context. The saved bundle of CPU state is what the word context refers to.

The OS performs thousands of these per second, and they are fast enough to be invisible while never being free. That is the reason a program creating hundreds of busy threads on an 8-core machine can measurably slow down.