When retrying is the wrong answer
Lesson 6-2 taught workers to retry because most failures are temporary: a network blip heals, an overloaded service recovers. But some jobs fail for reasons that never heal. A message referencing a video that was deleted, a job whose code path has a bug, an event in a format the worker cannot parse. Such a job is a poison message, and pure retry logic turns it into a small disaster: it fails, at-least-once delivery hands it out again, it fails again, forever. Workers burn time on it endlessly, and in queues that process in order it can block every job behind it. On-call engineers meet this one within their first year.
The standard defense has two parts:
- Cap the attempts. Track how many times each message has been tried (real queue systems count this for you) and stop after a small number, like 3 or 5
- Move it aside, never delete it. A message that exhausts its attempts goes to a dead letter queue (DLQ): a separate queue that humans inspect. The data is preserved for debugging, the main queue keeps flowing, and after the bug is fixed the dead letters can be replayed through the main queue
Code exercise · python
Run this worker loop with a poison message in the queue. Good jobs succeed on their first attempt. The poison job exhausts its 3 attempts and is moved to the dead letter queue instead of looping forever.
Code exercise · python
Your turn: measure why attempt caps matter. Each poison message wastes a lesson 6-2 exponential backoff schedule of 5 attempts (delays of 1, 2, 4, 8, 16 seconds, doubling from 1) before reaching the DLQ. With 200 poison messages a day, print the seconds wasted per message and the total worker-seconds wasted per day, exactly as shown.
Quiz
Your dead letter queue, normally empty, gains 5,000 messages in ten minutes, all of the resize-image job type. What most likely happened?
Problem
A queue delivers each message at least once. Workers cap processing at 4 attempts per message, then dead-letter it. A poison message arrives. How many times does a worker run its handler on it before it lands in the DLQ?