Queue vs pub/sub vs log

The decision

Parts of a system pass messages to each other. You choose how the messages are kept and who gets them. A queue gives each message to one worker and removes it after the worker confirms. Publish and subscribe (pub/sub) gives a copy of each message to every subscriber. A log is a list that only grows at the end. Each reader (a consumer) keeps its own place in the list, called an offset. Old messages stay for a set time, called the retention time.

Choose a queue when

• Each message is a job that one worker must do once, such as "send this SMS" or "resize this photo".
• You want the broker to route messages by rules, or to let urgent jobs go first.
• One slow job must not block the jobs behind it. A queue confirms each message on its own.
• Nobody needs to read a message again after it is done.

Choose pub/sub when

• One thing happened and several parts must react, such as "order placed".
• The sender must not know who listens, so you can add a listener without changing the sender.
• Messages are useful only now. A subscriber that is offline may miss them.

Choose a log when

• Several parts must read the same events, and each part needs its own place in them.
• You may need to replay: read old events again after a bug fix or for a new reader.
• Messages of one key, such as one order, must stay in order at high volume.
• You can pay for storage during the retention time. Apache Kafka keeps data for 7 days by default.

The numbers that change the answer

Assumption: 2,000 events per second, 500 bytes each, three services must read every event, one consumer handles 500 events per second.

The consumer count also sets the number of partitions (the parts a log is split into). Kafka lets one consumer in a group read each partition at a time. So 8 partitions allow at most 8 consumers in a group, which is 8 x 500 = 4,000 events per second.

Worked example at Indian scale (assumption). A ride app logs 2,000 trip events per second at peak. A bug gave wrong fares for the last hour. Events in one hour: 2,000 x 3,600 = 7,200,000. With a log, you fix the code and move the reader back one hour. The 8 consumers do 4,000 per second while 2,000 new events still arrive. Net catch-up: 4,000 - 2,000 = 2,000 per second. Time: 7,200,000 / 2,000 = 3,600 seconds, which is 1 hour. With a queue, the confirmed messages are gone, so you cannot replay.

What breaks with each choice

• Queue: a message that always fails keeps coming back (a poison message). Use a dead letter queue, a side queue for failed messages, to set it aside. A crash before the confirmation makes a job run twice, so jobs must be safe to repeat.
• Pub/sub: a subscriber that is down loses the messages sent meanwhile. Fix: give each subscriber its own queue that stores messages.
• Log: one slow message holds up the rest of its partition. Order exists only inside one partition, not between partitions. Changing the number of partitions later can move a key to another partition.

What it costs

• A queue is the cheapest to run and to understand. Moving to a log later means new code for offsets, keys and the partition plan. Plan for weeks of work (rough guess).
• A log costs storage and care. Here it is 604.8 GB for 7 days, before the extra copies kept for safety. You must also choose the key and the partition count well.
• Moving from a log back to a queue means less code, but you lose replay and the second reader.

How each level answers

SDE-1: A queue gives each message to one worker. Pub/sub gives every subscriber a copy. A log keeps messages so readers can read again.

SDE-2: I pick a queue for background jobs, because each job is done once. I pick a log when three services read the same events, because each one keeps its own place and I can replay.

Senior: At 2,000 events per second and 500 per consumer, I need 4 consumers, so I create 8 partitions keyed by order ID. The risk is a hot key that overloads one partition. The fallback is a separate queue for slow jobs.

Staff: I ask which readers will exist next year, and who pays for the 604.8 GB. I start with a queue for the one known job, and move to a log when a second reader or a replay need appears. To move, the sender writes to both for a week, then readers switch one by one. One team owns the event shape and changes it only by adding fields. I check the choice by consumer lag, the age of the oldest message, and the hours lost to replays.

Say it in the interview

I would use a log keyed by order ID, because three services read the same events and I may need to replay an hour. At 2,000 events per second that is about 605 GB for 7 days, and partitions limit my consumers. If only one job needs the messages, a queue is simpler.

Common mistakes

• Saying "Kafka is faster". Speed is not the question. Replay and many readers are.
• Adding more consumers than partitions. The extra consumers read nothing.
• Expecting order across the whole topic. Order holds only inside one partition, so choose the key by what must stay in order.
• Using plain pub/sub for important events. An offline subscriber misses them.

Sources

• Apache Kafka, design (one consumer per partition in a group, order inside a partition, rewinding to an old offset): https://kafka.apache.org/42/design/design/ (opens in a new tab) (checked 26 Oct 2026).
• Apache Kafka, broker settings (log.retention.hours default is 168 hours): https://kafka.apache.org/42/configuration/broker-configs/ (opens in a new tab) (checked 26 Oct 2026).
• All event rates, sizes and consumer speeds are assumptions made for teaching.