Queues and workers

What is wrong with transcoding inside the request

The user, the connection, and one whole server thread are all held hostage for 40 seconds.

The lesson 1-1 request cycle is meant to finish in tens of milliseconds. Slow work done inside it blocks the user, times out connections, and starves the server of capacity for everyone else.

Do the arithmetic to see how bad the capacity problem is. A server holding 40-second requests can only handle a handful at a time, so a fleet sized for 200 fast requests per second collapses to a few uploads at once.

The connection is likely to fail anyway, which makes the whole design pointless. Load balancers and browsers have their own timeouts, often 30 or 60 seconds, so a job that takes longer produces an error even when the work succeeded.

The fix is to answer immediately and do the slow work somewhere else, and that somewhere is a queue plus workers.

The producer-worker pattern

A message queue is a service that stores small job descriptions until something is ready to process them, releasing them in arrival order. Its purpose is separation: the code that discovers work no longer has to be the code that does the work, or do it at the same moment. Think of it as a waiting line for jobs, sitting between the servers that discover work and the servers that do it:

1. The app server (the producer) receives the upload, saves the raw file, and pushes a small message onto the queue: {"job": "transcode", "video": 123}. This takes a millisecond
2. It immediately answers the user: got it, processing
3. Separate worker processes pull messages off the queue, one at a time, and do the slow work at their own pace

Real queue systems include RabbitMQ, Amazon SQS, and Kafka.

What the queue buys you:

• Fast responses: users never wait on slow work
• A shock absorber: a traffic spike piles messages into the queue instead of crashing servers, and workers catch up afterward
• Independent scaling: too much work waiting? Add workers, nothing else changes

A queue absorbing a burst

Each second some jobs arrive and one worker processes up to 3.

from collections import deque

queue = deque()
arrivals = [5, 8, 2, 0, 1, 0]
WORKER_RATE = 3

for second, arriving in enumerate(arrivals):
    for j in range(arriving):
        queue.append("job")
    done = 0
    while queue and done < WORKER_RATE:
        queue.popleft()
        done += 1
    print("second", second, "| arrived", arriving, "| processed", done, "| waiting", len(queue))

Output

second 0 | arrived 5 | processed 3 | waiting 2
second 1 | arrived 8 | processed 3 | waiting 7
second 2 | arrived 2 | processed 3 | waiting 6
second 3 | arrived 0 | processed 3 | waiting 3
second 4 | arrived 1 | processed 3 | waiting 1
second 5 | arrived 0 | processed 1 | waiting 0

Eighteen jobs arrived in bursts of up to 8 per second, and a worker that can only do 3 per second finished all of them. Nothing failed and nothing was dropped, which is the shock-absorber property in one output block.

Watch the waiting column rise to 7 and come back to 0. That number is the backlog, and its shape here is exactly what healthy bursty traffic looks like, meaning it grows during a spike and drains during the quiet.

The cost is latency rather than failure, and it is worth being explicit. The jobs that arrived in second 1 finished around second 3, so a user waited a few seconds for work that was accepted instantly.

deque is a list optimized for adding at one end and removing at the other, which is a perfect queue. Using a plain list with pop(0) works and gets slower as the queue grows, since every removal shifts every remaining element.

enumerate(arrivals) gives the index and the value together, which is what supplies the second number. The while queue and done < WORKER_RATE condition stops on either an empty queue or a used-up worker, which is why second 5 processes only 1.

Doubling the workers

The same arrivals with capacity 6 per second, meaning two workers.

from collections import deque

queue = deque()
arrivals = [5, 8, 2, 0, 1, 0]
WORKER_RATE = 6

for second, arriving in enumerate(arrivals):
    for j in range(arriving):
        queue.append("job")
    done = 0
    while queue and done < WORKER_RATE:
        queue.popleft()
        done += 1
    print("second", second, "| arrived", arriving, "| processed", done, "| waiting", len(queue))

Output

second 0 | arrived 5 | processed 5 | waiting 0
second 1 | arrived 8 | processed 6 | waiting 2
second 2 | arrived 2 | processed 4 | waiting 0
second 3 | arrived 0 | processed 0 | waiting 0
second 4 | arrived 1 | processed 1 | waiting 0
second 5 | arrived 0 | processed 0 | waiting 0

The peak backlog dropped from 7 to 2, and the queue is empty in most seconds. Jobs now wait a fraction of a second instead of several, which is the user-visible improvement.

One constant changed and nothing else, which is the independent-scaling property from the intro. Adding capacity to the worker tier required no change to producers, to the queue, or to the job format.

The while loop already stops early when the queue is empty, so quiet seconds process fewer jobs. Second 3 processes 0 because there was nothing to do, not because capacity was missing, and that distinction matters when reading a real dashboard.

Note that the second 1 backlog of 2 is not a problem to fix. Provisioning enough workers to leave the queue empty at every instant means paying for idle capacity most of the time, and absorbing brief bursts is the queue's job.

The number to size against is the sustained arrival rate rather than the peak. Average arrivals here are 2.67 per second, so even the single worker had enough long-run capacity, and the second worker bought latency rather than throughput.

What a steadily growing queue depth tells you

Work is arriving faster than workers can finish it, and the backlog will keep growing until you add workers or shed load.

A queue that grows without draining means the sustained arrival rate exceeds the processing rate. That is an arithmetic fact rather than a tuning problem, so no amount of making individual jobs slightly faster fixes a persistent deficit.

The queue is buying you time rather than solving the imbalance, and eventually memory or user patience runs out. A queue that has grown for an hour holds jobs whose results nobody is waiting for anymore.

There is a compounding failure worth anticipating. Jobs that carry a deadline or a timeout can expire while waiting, so a deep queue starts producing failures for work that would have succeeded, and those failures often get retried, which adds more load.

Queue depth is one of the most important dashboards in any async system. Bursty depth is normal and steady growth is an alarm, and alerting on the trend rather than the absolute number is what catches the problem early.