Dead-letter queues and poison messages

The interviewer asks: "One message in your queue fails every time. What happens to the queue?"

A weak answer says "retry until it works" or "delete it". The first never ends. The second hides the bug. A strong answer limits the tries, moves the message to a separate place, raises an alarm and sends it again after the fix. At SDE-1 level, name the idea. At SDE-2 level, give the numbers.

Think about a post office. A letter with an unreadable address cannot be delivered. After a few tries, the post office puts it in a tray, and a person reads it. Other letters keep moving.

A poison message is a message that fails every time a worker tries it. The cause is in the message or the code, for example a phone number with letters in it. A retry cannot help. A short failure, such as a server that is down for a minute, is different. A retry helps there (lesson 54).

In lesson 58, a message without an ack goes back to the queue. A poison message comes back again and again, and each try uses worker time:

• workers busy = failing messages x seconds of work per try / seconds between tries

A dead-letter queue (DLQ) is a second queue for messages that failed too many times. You set a maximum number of tries on the main queue. Amazon SQS calls it maxReceiveCount. At the limit, the queue moves the message to the DLQ, with the reason and the number of tries.

Rules:

1. Retry with backoff and jitter (lesson 54) up to the limit. Five is a good start (our own advice).
2. Skip the tries for a clear permanent error, such as an unreadable message.
3. Give the DLQ an owner and an alarm. Alarm when it is not empty. A DLQ that nobody reads only hides the problem.
4. Keep DLQ messages longer than the main queue does. In SQS standard queues, the keeping time counts from the first send.
5. Replay after the fix. Move the messages back at a chosen speed. Handling a message twice must give the same result as once (idempotent, lesson 61). Replay can change the order, and old work may no longer make sense.

A food app sends a receipt SMS from a queue.

Assumptions (made up to teach with, not real data): 100,000 messages a day. 100 of them fail forever, because of a bad phone number. There are 20 workers. A try takes 1.2 seconds. A failed message returns after 30 seconds.

Step 1: no limit. Each day adds 100 messages. Workers busy = 100 x 1.2 / 30 = 4 for each day. All 20 are busy after 20 / 4 = 5 days, which is 500 messages. This answers the predict box.

Step 2: a DLQ with 5 tries. A poison message now uses 5 x 1.2 = 6 seconds. For 100 messages that is 600 seconds a day, out of 20 x 86,400 = 1,728,000. The cost is 0.035% of the work.

Step 3: replay. The SMS provider was down for about 12 hours, and 50,000 messages reached the DLQ. The normal load is 100,000 / 86,400 = 1.16 messages a second. The workers can do 20 / 1.2 = 16.67 a second. The spare speed is the difference: 16.67 - 1.16 = 15.5 a second. Replay at full speed delays every new SMS by about 50,000 / 16.67 = 3,000 seconds, which is 50 minutes. Replay at 10 a second takes 5,000 seconds (83 minutes), and new SMS are not delayed.

• A DLQ solves workers kept busy by failing messages. It costs a second queue, an alarm and an owner. I would use it on every queue.
• A low limit solves wasted work. It costs good messages in the DLQ in a short outage. I would raise it then.
• Replay solves recovery. It costs care with repeats, old messages and order.

"A DLQ fixes the failed message." It only keeps the message. A person must find the cause.

"Retry forever, so nothing is lost." In the example, 5 days of poison messages kept all 20 workers busy.

"Replay everything at once." In the example, that delays every new SMS by 50 minutes.

1. What is a poison message? Why does a retry not help it?
2. A queue has 10 workers. A failed job uses 2 seconds and returns every 20 seconds. Each day, 50 new messages fail forever. After how many days are all workers busy?
3. A DLQ holds 30,000 messages after an outage. The spare speed is 12 a second. Is replay at 20 a second a good idea?