Will my message arrive once, twice or never?

Money, orders and notifications all move between parts of a system as messages. The interviewer asks: "A message is sent between two services. Can it be lost? Can it arrive twice? How do you make sure a customer is paid cashback exactly once?"

A weak answer says "the system guarantees exactly once" and stops. A strong answer explains why a message can be lost or repeated, and what each fix costs. It says that a setting in the message system does not protect work outside it, such as a write to your own database.

At SDE-1 level you should name the three guarantees. At SDE-2 level you should say which order of steps gives which guarantee, and show it with numbers. At SDE-3 level you should also say what "exactly once" really means and where it stops.

Think about sending a parcel by courier and asking for a signed slip. If the slip does not come back, you do not know the truth. The parcel may be lost on the way. Or the parcel may have arrived and the slip may be lost on the way back. You have two choices. Send a second parcel, and the person may get two. Or do not send, and the person may get none.

Computers face the same problem. A network can lose any message. This includes the reply that says "I got it". The sender sees only silence. Lesson 54 said it: a timeout means "no answer", not "not done".

Acknowledgements

An acknowledgement (ack) is a reply that says "I got it" or "I finished". In a message system, acks happen at two places:

1. Producer to broker. The broker (the server that stores the messages, lesson 59) tells the producer "I have stored your message". RabbitMQ calls this a publisher confirm. If no confirm arrives, the producer does not know. If it sends again, the queue may now hold two copies.
2. Consumer to broker. The consumer tells the broker "I finished this message", and the broker removes it (lesson 58). In a log, the consumer saves its offset: the number of the next message to read (lesson 59). This is the same idea.

If a consumer dies without an ack, the broker gives the message to another consumer.

The three guarantees

A delivery guarantee says how many times a message is handled.

At-most-once. The message is handled zero times or one time. It is never repeated, but it can be lost. This happens when the consumer acks first and then does the work, or when the producer never sends again. If a crash comes after the ack and before the work is done, the work is gone.

At-least-once. The message is handled one time or more. It is not lost, but it can be repeated. This happens when the consumer does the work first and then acks, and when the producer sends again until it gets a confirm. If a crash comes after the work and before the ack, the broker sends the message again, and the work is done twice. This is the usual default. RabbitMQ with manual acks and Amazon SQS standard queues work this way.

Exactly-once. The effect of the message happens one time. Be careful with this one. A network can lose any reply, so the sender cannot tell "lost message" from "lost ack". Because of this, plain delivery cannot be exactly once. Real systems give exactly-once in effect. They send at-least-once, and they make sure a repeat does no harm. They use three tools:

• Remove repeats on the sending side. The broker remembers what a producer already sent. Amazon SQS FIFO queues do this for 5 minutes. Kafka has an idempotent producer for the same job.
• Do the read, the work and the write as one unit inside the broker. Kafka can do this with transactions. It works only when the output is also written to Kafka.
• Ignore repeats at the receiver. This is the only tool that works when the work happens outside the broker. An example is a change in your database or a call to a payment provider. Every message carries an ID. The consumer saves the ID together with the result, in one database transaction (lesson 21). When a message arrives with an ID that is already saved, the consumer skips it.

The Kafka documentation says the same. To get exactly-once with an outside system, store the consumer's position together with the output. A broker setting alone cannot do that.

The order of ack and work

This is the point to remember. A consumer takes a message, does the work, and acks. Only the order differs:

A repeat without a crash

A repeat can happen when nothing crashed. The broker waits for an ack only for an ack time limit (Amazon SQS: 30 seconds by default). Suppose the job takes 40 seconds. The time limit ends while the first consumer still works. The broker gives the message to a second consumer, and both finish. This is the timeout problem of lesson 54. Set the limit above the slowest job, or extend it while the job runs.

Which guarantee to choose

• At-most-once is the cheapest and fastest. Use it where a lost message costs little, for example a location ping every 3 seconds. The next ping replaces the lost one.
• At-least-once is the usual choice when work matters. The work must be safe to repeat (idempotent, lesson 9).
• Exactly-once in effect adds a saved ID and a check for every message. Use it when both loss and repeat cost money.

A wallet app gives Rs 10 cashback for each payment. Each payment creates one message. A consumer reads the message and adds Rs 10 to the user's wallet in the database.

Assumptions (made up to teach with, not real data):

• 500,000 messages per day.
• Consumers restart 6 times a day (deploys and crashes). At each restart, 50 messages are in progress.
• The ack time limit is 30 seconds. On a slow day, 1% of jobs take 40 seconds.

In progress at restarts per day = 6 x 50 = 300 messages.

Design A: ack, then work. A crash after the ack loses the work. All 300 in-progress messages can be lost. That is 300 x Rs 10 = Rs 3,000 of cashback never paid each day. As a share: 300 / 500,000 = 0.06%. No customer is paid twice. This is at-most-once, and it answers the predict box: the job is lost.

Design B: work, then ack. No message is lost. But if a consumer crashes after the database write and before the ack, the broker sends the message again. In the worst case all 300 are repeated: 300 x Rs 10 = Rs 3,000 paid twice each day. This is at-least-once.

The slow day. 1% of 500,000 = 5,000 jobs take 40 seconds. The limit is 30 seconds, so the broker gives each of them to a second consumer. No crash happened, but 5,000 x Rs 10 = Rs 50,000 is paid at least twice. This is more than the crash case. The fix is a limit above the slowest job, for example 120 seconds.

Design C: work, then ack, and ignore repeats. For each message, the consumer does this in one database transaction:

1. Try to save the message ID in a table that allows each ID only once (a unique constraint, lesson 22).
2. If the ID was new, add Rs 10 to the wallet.
3. If the ID already existed, do nothing.
4. Commit. Then send the ack.

A crash before the commit undoes everything, so the message comes again and is done. A crash after the commit but before the ack makes the message come again, and the saved ID makes the repeat do nothing. The result: 0 lost and 0 paid twice. The cost: one extra insert for each of the 500,000 messages, and a table of IDs that grows. This is exactly-once in effect.

The producer side. Suppose the payment service sends a message and the broker's confirm is lost. The service sends again, and the queue holds 2 copies of one payment. Design C still pays once, because both copies carry the same payment ID.

• At-most-once solves repeats and is the cheapest. It costs lost messages. I would use it for data where the next message replaces a lost one.
• At-least-once solves loss. It costs repeats, so the work must be safe to repeat. I would use it as the default and add the ID check where a repeat costs money.
• Exactly-once in effect solves both. It costs a saved ID, an extra write, and a rule for when to delete old IDs. I would use it for money, and not for a location ping.
• A longer ack time limit solves repeats on slow jobs. It costs a slower retry after a real crash, because the message stays hidden for longer. I would set it above the slowest normal job.

"The broker says exactly-once, so duplicates are impossible." A broker setting covers work inside the broker. A write to your own database or a call to another service is outside it. A crash between that write and the ack still repeats the message.

"At-most-once is a bug." It is a choice. For a location ping every 3 seconds, a lost message costs nothing, and a repeat is wasted work.

"If I get no ack, the message was lost, so sending again is safe." No ack means no answer. The message may have arrived. Sending again can create a copy.

1. Name the three delivery guarantees. Which order of ack and work gives at-most-once?
2. A system gets 2,000,000 messages per day. Consumers restart 8 times a day with 25 messages in progress each time. Each message pays Rs 5. With ack first, what is the most that can be lost in a day, in messages and in rupees? On another day the ack time limit is 30 seconds and 2% of jobs take 45 seconds. How many jobs are handled twice?
3. For each, choose a guarantee and say why: (a) a driver's location every 3 seconds, (b) a wallet credit, (c) an invoice email.