Work fails, so we retry
A worker pulls a job, calls a payment provider, and the network drops (as How the Internet Works warned it would). Did the charge happen? You often cannot know. The queue's answer is blunt: if a worker does not confirm a job finished, the job is handed out again. This is at-least-once delivery, and it is the standard guarantee, because the alternative risks silently losing jobs.
At-least-once has a sharp edge: the same job can run twice. If the first charge actually succeeded and the retry charges again, a customer just paid double.
The cure is idempotency: designing an operation so that running it twice has the same effect as running it once. Setting balance = 80 is idempotent. balance = balance - 20 is not. Deleting a row is idempotent. Appending a row is not.
An idempotent payment handler
Every event carries a unique id, and the handler records the ids it has processed.
processed = set() def handle(event_id, amount): if event_id in processed: print("skipping duplicate", event_id) return processed.add(event_id) print("charging $" + str(amount), "for", event_id) handle("evt_1", 30) handle("evt_2", 15) handle("evt_1", 30)
Output
charging $30 for evt_1 charging $15 for evt_2 skipping duplicate evt_1
The retried delivery of evt_1 is harmless, which is the entire goal. The queue is free to deliver a job twice, and the handler makes the second delivery a no-op.
The identity of a job has to come from the producer rather than from the payload, which is the design decision hiding in that first argument. Two genuinely separate $30 charges look identical in their amounts and must not be deduplicated, so only an explicit id can tell a duplicate from a repeat.
In production the processed set is a database table or Redis, shared by all workers, and event_id is called an idempotency key. A local set would be useless, since a retry usually lands on a different worker than the original.
There is a race in this version worth knowing about. Two workers could check the set at the same moment and both proceed, so real implementations use a unique constraint on the key or an atomic set-if-absent, which makes the check and the claim one operation.
Note where the recording happens, which is before the charge rather than after. That ordering prevents a double charge if the process dies mid-operation and accepts the opposite risk, meaning a job recorded but never done, and which risk to take is a per-operation decision.
Every payment provider exposes this as an API parameter, so the pattern is not something you invent. Stripe and its peers accept an idempotency key on charge requests for exactly this reason.
Retry politely: exponential backoff
When a dependency fails, retrying instantly and forever makes things worse: a struggling service gets hammered by every client at once, a self-inflicted flood called a retry storm.
The standard discipline is exponential backoff: wait 1 second before the first retry, then 2, then 4, then 8, doubling each time, and give up after a few attempts. Failures get time to heal, and load on the sick service drops instead of spiking. Production systems also add jitter, a little randomness in each delay, so a thousand clients do not all retry in the same instant.
You saw this schedule from the receiving side in lesson 4-2's failover timing. Now compute it from the sender's side.
An exponential backoff schedule
Delays double after each retry, and the cumulative wait grows with them.
delay = 1 total_wait = 0 for attempt in range(1, 6): total_wait += delay print("retry", attempt, "after", delay, "s (total wait:", str(total_wait) + "s)") delay *= 2
Output
retry 1 after 1 s (total wait: 1s) retry 2 after 2 s (total wait: 3s) retry 3 after 4 s (total wait: 7s) retry 4 after 8 s (total wait: 15s) retry 5 after 16 s (total wait: 31s)
Five retries span 31 seconds, and that total is the number worth remembering. Doubling gives a long window from very few attempts, which is the property that makes backoff work.
Compare the load this places on a struggling service against retrying every second. Five attempts in 31 seconds is a sixth of the traffic, and the reduction gets larger with every additional retry.
The cumulative totals follow a pattern that is useful for estimating. Each total is one less than the next delay, so the whole schedule is roughly twice the final wait, which lets you pick a retry count from a target window.
delay *= 2 after the print is what produces the doubling, and moving it before the print would start the schedule at 2 seconds. The order of those two lines is the difference between a first retry that is prompt and one that is not.
Add jitter in production, meaning a random fraction added to or subtracted from each delay. Without it, a thousand clients that failed at the same instant retry at the same instant five more times, which recreates the flood the backoff was meant to prevent.
Note the limit on how long this should continue. Five retries is a reasonable default, and a schedule that keeps doubling forever eventually holds a job for hours, which is what dead letter queues in the next lesson are for.
How many emails a triple delivery sends
Exactly 1.
The first delivery sends the email and records the job id. Deliveries 2 and 3 find the id already recorded and do nothing, so the duplicates cost a set lookup each.
This pairing is the fundamental contract of async systems. The queue promises at-least-once delivery, the handler promises idempotency, and together they behave like exactly-once.
Neither half works alone, which is the reason to state it as a contract. An idempotent handler behind an at-most-once queue silently loses jobs, and a non-idempotent handler behind an at-least-once queue sends three welcome emails.
Exactly-once as a queue guarantee is largely a marketing claim, and knowing that is useful in an interview. Distributed systems cannot deliver it in general, so systems that advertise it are usually doing deduplication on your behalf, which is this pattern with the work moved.
The practical habit is to assume every handler will run more than once. Design each one so that is boring, and the retry behavior above stops being a source of bugs.