Skip to content

Trade-offs · Production decisions

At-least-once with idempotency vs exactly-once

How to choose between at-least-once delivery with idempotent handlers and a broker's exactly-once features, the numbers behind it, and what breaks.

Asked at: SDE-2Senior

The decision

A network can lose any message or any reply, so a sender cannot tell a lost message from a lost confirmation. At-least-once delivery means the sender tries again until it is sure, so a message arrives one time or more. The receiver must then be idempotent, which means that handling a message many times leaves the same result as handling it once. Exactly-once is the wish that the effect happens one time. Real systems give exactly-once in effect in two ways. You can add your own idempotency to at-least-once delivery. Or you can use a broker feature, such as transactions, that does the checking inside the broker. This page compares those two ways.

Choose at-least-once with idempotent handlers when

  • The work happens outside the broker: a write to your database, a payment, an SMS, an email.
  • Messages cross more than one system or more than one broker.
  • The cost of a duplicate is high (money), so you want a check you control and can test.

Choose broker exactly-once features when

  • The whole flow stays inside one broker: read a message, process it, and write the result to the same broker.
  • The job is stream processing, such as counting or joining streams, where the output is also a stream.
  • Your platform team already runs the feature and knows its limits.

The numbers that change the answer

Worked example: a cashback service in a payments app. Assumptions: 1,000,000 messages a day, 0.1% delivered twice (after a crash, or a late ack), and Rs 50 per cashback. A healthy system may see fewer duplicates, so measure your own rate.

Cost of no protection (calculation).

  • Duplicates = 1,000,000 x 0.001 = 1,000 a day.
  • Overpaid = 1,000 x Rs 50 = Rs 50,000 a day.
  • In a year: Rs 50,000 x 365 = Rs 18,250,000 (1.8 crore).

Cost of the idempotency check (assumption plus calculation). The handler saves the message ID and the result in one transaction. Assumption: the handler takes 5 ms, and the check adds 0.5 ms. One worker handles 1,000 / 5 = 200 messages a second without the check and 1,000 / 5.5 = 181.8 with it. That is 5.5 / 5 = 1.1 times the work, so you need about 10% more worker capacity.

Size of the ID table (assumption plus calculation). Assumption: 64 bytes a row, kept for 7 days. Rows = 1,000,000 x 7 = 7,000,000. Size = 7,000,000 x 64 = 448,000,000 bytes = 448 MB. Small.

How long to keep IDs. Keep each ID longer than the longest time a duplicate can arrive. Assumption: the broker keeps messages for up to 3 days, so a duplicate can arrive for up to 3 days, and 7 days is safe. If IDs expire sooner, a late duplicate passes.

A cited limit. Amazon SQS FIFO queues remove duplicate sends only inside a 5-minute window (see Sources). A retry after 6 minutes can create a second copy. So the broker feature protects only the part it can see.

Broker transactions (assumption). Results become visible to read_committed readers only after a commit. If the system commits every 100 ms, results arrive up to 100 ms late. That is fine for counting and too slow if a user waits.

What breaks with each choice

At-least-once with idempotent handlers.

  • The check and the work are not one unit. Two copies arrive together, both look, both see "new". Fallback: let the database refuse the second copy with a unique constraint, and save the ID in the same transaction as the work.
  • The ID is built from the wrong fields, so two different messages look the same and one is dropped. Fallback: use the ID the sender created, and test with a replay.
  • A side effect lies outside your transaction. An email is sent before the commit, then the commit fails, then the retry sends it again. Fallback: send the email from a message that is saved in the transaction (the outbox pattern), and pass an idempotency key to outside services.

Broker exactly-once.

  • The guarantee stops at the edge of the broker. In Kafka, the idempotent producer stops duplicates when a send is retried (within one producer session), and transactions write to several partitions all-or-nothing. A consumer set to read_committed then skips messages of aborted transactions. But the Kafka documentation says that exactly-once to another system needs that system to cooperate (see Sources). A write to your own database inside the handler is not covered. Fallback: store the consumer position in the same place as the result, as the Kafka documentation suggests, or use idempotent handlers anyway.
  • A false sense of safety: teams stop checking for duplicates. Fallback: test with a forced crash between the work and the ack.
  • An old producer instance that is still running, or a transaction that never ends and holds back read_committed readers. Fallback: fencing, which makes the broker refuse the older instance, and a time limit on transactions.

What it costs

  • People. Idempotent handlers: assumption 1 to 2 engineer-days for each consumer, less with a shared library. Broker transactions: weeks of platform setup.
  • Servers. About 10% more worker capacity and 448 MB of table in the example.
  • Changing your mind. Adding idempotency later is a safe and mostly local change: add the table, check at the top of the handler, and deploy. Moving from broker transactions to your own checks is the same work. Moving to broker transactions needs a platform that offers them, so it is the costly direction. Start with idempotent handlers, because you will need them anyway at the edge.

How each level answers

SDE-1: A message can be delivered more than once. Exactly-once means the effect happens one time. A handler is idempotent when a repeat does no harm.

SDE-2: I use at-least-once and make the handler idempotent with a message ID. I save the ID and the result in one database transaction, so a duplicate is skipped. This works with any broker.

Senior: At 0.1% duplicates, 1,000,000 messages a day give 1,000 duplicates and Rs 50,000 a day of overpaid cashback. The check adds 0.5 ms (about 10% more worker time) and 448 MB for 7 days of IDs. I keep IDs longer than the longest redelivery, and a unique constraint handles two copies that race. Broker transactions help only for flows inside the broker.

Staff: The business risk is Rs 18,250,000 a year of double payments, against a few engineer-days. So I make idempotency a platform rule: every consumer that changes money or sends a message uses the shared library, and review checks it. I roll it out first on the money consumers. I treat broker exactly-once as an optimization for stream jobs, not as the safety plan. I measure the duplicate rate, the duplicates skipped and the money mismatches found by reconciliation. If reconciliation finds zero double payments over a quarter, the choice was right.

Say it in the interview

I would use at-least-once delivery and make the cashback handler idempotent. The handler saves the message ID and the credit in one transaction with a unique constraint, so a duplicate is skipped. That saves Rs 50,000 a day at 1,000 duplicates. I would use broker transactions only for jobs that read and write inside the broker.

Common mistakes

  • Saying "the broker guarantees exactly-once, so we are safe". It covers messages inside the broker. Your database write and the payment call are outside.
  • Checking "does this ID exist?" and then doing the work. Two copies pass the check together. Use a unique constraint in the same transaction.
  • Keeping IDs for a short time. A late duplicate arrives after the ID is gone and is handled again.

Sources

Understand the trade-off. Practise explaining your choice.

In an AI mock interview, the interviewer can push on a choice like this one: it changes a number and asks if your answer still holds.

See the mock interviewsAll trade-offs