Queue, log or pub/sub (RabbitMQ-style vs Kafka-style)

Many designs pass messages between parts: orders, payments, chat messages, clicks. The interviewer asks: "Would you use RabbitMQ or Kafka here, and why?"

A weak answer says "Kafka is faster" or "I know Kafka better". A strong answer asks what the receiving parts need. Do they need to read old messages again? Do many different parts read the same message? Must the messages of one user stay in order? Then it names the kind of system that fits and what it costs.

At SDE-1 level you should know that two kinds of system exist and name one use for each. At SDE-2 level you should explain how a log is split to go faster, and how the parts that read share it. You should also work out the sizes. At SDE-3 level you should also say what breaks when one part that reads is slow.

Lesson 58 gave you the queue. A producer adds a message, a worker takes it, and the queue removes it after the ack. Here is a second way to keep messages.

Think about two ways to share news in a housing society (the group that runs an apartment building). First, the secretary gives each notice to one volunteer, who acts on it and throws it away. Second, the secretary pins every notice on a board in order. Anyone can read it at any time. Each person remembers the number of the next notice to read. Old notices stay on the board for a few days.

The first way is a queue. The second way is a log.

Queue style (RabbitMQ-style)

A broker is a server that stores messages and hands them out. In queue style, the broker keeps messages in queues, gives each message to one worker, and deletes it after the ack. A broker of this kind can also route. A message goes to one or more queues by rules, for example by the word in its label. Some queues also let an urgent message go before the others. This is a priority.

Queue style fits jobs: "send this SMS", "resize this photo". Each job is done by one worker and is then gone.

Log style (Kafka-style)

A log is a list of messages where new messages are added at the end and old ones are not changed. The broker does not delete a message when someone reads it. It keeps the message for the retention time. You can also set a size limit. (Kafka keeps data for 7 days by default.)

A consumer is a program that reads messages (lesson 58). In a log, the consumer asks the broker for messages.

Each message has an offset: its place in the log, counted from 0. The consumer remembers its own offset, the number of the next message to read. To read again from yesterday, the consumer sets the offset back. This is a replay.

Many consumers can read the same log, and each one keeps its own offset. So each one gets every message. This is pub/sub (lesson 58) built on a log.

Partitions

One log on one server has a limit on how fast it can take writes and serve reads. To go beyond it, split the topic into partitions. Each partition is its own log, and the partitions can sit on different servers.

The producer gives each message a key, for example an order ID. A hash turns the key into a number (lesson 43). The partition is chosen like this:

• partition number = hash(key) mod number of partitions

(mod gives the remainder after division.) The same key always goes to the same partition, while the number of partitions does not change. The messages inside one partition keep their order. Each partition has its own offsets, counted from 0, and a consumer keeps one offset for each partition it reads. There is no order between partitions. So choose a key so that the messages that must stay in order share a key.

Consumer groups

A consumer group is a set of consumers that have the same group name and share the work. The rule: each partition is read by one consumer in the group. This means:

• A group of 6 consumers on a topic of 12 partitions: each consumer reads 2 partitions.
• A group with more consumers than partitions: the extra consumers have nothing to read and wait.
• If a consumer dies, the broker gives its partitions to the others in the group.
• A second group with another name reads the whole topic again, with its own offsets.

So the number of partitions sets the most parallel work that one group can do.

Offset commit and lag

A consumer saves its offset from time to time. This is a commit. A consumer commits after it finishes the work. If it crashes, a new consumer starts at the last committed offset. Messages after that offset are read again, so the work must be safe to repeat.

Lag shows how far a group is behind. Lag = the offset where the next new message will be written - the committed offset. A growing lag is the same signal as a growing backlog in lesson 58.

A log has a cost. If one message is slow, the messages behind it in that partition wait. A queue acknowledges each message on its own, so one slow message does not block the others.

Event-driven design

A command asks for something to be done: "send the SMS". An event says that something happened: "OrderPlaced". Write events in the past tense.

In event-driven design, parts publish events, and other parts react. The part that publishes does not know who reads. A log fits this well, because a new consumer can join with its own group and read old events too.

The benefits: you add a new consumer without a change in the producer. The parts do not call each other directly, so a slow part cannot hold up the one that publishes (lesson 54). The costs:

• No single place shows the full chain of what happens after an event. Debugging needs an ID that travels with the event.
• Readers run at different speeds, so data is correct only after a short delay (lesson 40).
• Producer and consumers must agree on the shape of an event. If the producer renames a field, consumers can break (lesson 11).

Which one to pick

Ask these questions:

The two styles overlap. RabbitMQ also offers streams, which keep messages and allow replay. Kafka groups can work like workers on a queue. So answer from what the system must do, not from a product name.

A food app publishes an event to the topic order-events each time an order changes: placed, paid, packed, delivered.

Assumptions (made up to teach with, not real data):

• Peak: 6,000 events per second. Average: 2,000 events per second.
• One consumer processes 1,000 events per second.
• An event is 1 KB (1,000 bytes). The log keeps 7 days.

Step 1: how many partitions? At peak, 6,000 / 1,000 = 6 consumers. A group cannot use more consumers than partitions, so we need at least 6. We choose 12, to leave room to double. At peak, each partition gets 6,000 / 12 = 500 events per second.

Step 2: the group. The group loyalty has 6 consumers. Each reads 12 / 6 = 2 partitions, which is 2 x 500 = 1,000 events per second. That is exactly the consumer speed. With 14 consumers, 12 read one partition each, and 2 wait. This answers the predict box: 12.

Step 3: the key. The key is the order ID. Assume that hash(order 8841) mod 12 = 5. The events placed, paid and packed for order 8841 all go to partition 5, in this order. Without the key, "packed" could be read before "paid".

Step 4: more groups. The group analytics has 3 consumers (12 / 3 = 4 partitions each). It reads every event, and has its own offsets. Assume partition 5 has 1,000 messages, so the next offset to write is 1,000. The group loyalty has committed 990, so its lag is 1,000 - 990 = 10. The group analytics has committed 400, so its lag is 1,000 - 400 = 600.

Step 5: storage. 2,000 x 86,400 seconds = 172,800,000 events per day. Over 7 days: 172,800,000 x 7 = 1,209,600,000 events. At 1,000 bytes each: 1,209,600,000,000 bytes, about 1.2 TB for one copy.

Step 6: a consumer dies at peak. To keep the arithmetic simple, we count the speed of the whole group. We also assume the lag is spread evenly over the partitions. (In real life it can sit on the busiest consumers, and catching up then takes longer.) The group loyalty has 5 consumers left. Its speed is 5 x 1,000 = 5,000 events per second, against 6,000 arriving. Lag grows by 1,000 per second. After 10 minutes (600 seconds): 1,000 x 600 = 600,000 events behind. Starting a 6th consumer gives 6,000 per second. That equals the arrival rate, so the lag does not shrink. To catch up, use more consumers. With 12 consumers, the speed is 12,000. The lag shrinks by 12,000 - 6,000 = 6,000 per second, so 600,000 / 6,000 = 100 seconds. We cannot use more than 12 consumers.

Step 7: a replay. A bug in the points code gave wrong points for the last hour. We fix the code and set the offsets back one hour. Events in one hour: 2,000 x 3,600 = 7,200,000. New events still arrive at 2,000 per second, while the 6 consumers do 6,000 per second. Time to catch up = 7,200,000 / (6,000 - 2,000) = 1,800 seconds, which is 30 minutes. In queue style the acknowledged messages are gone, so this is not possible. The producer would have to send them again.

• A log solves replay, many independent consumers, and order per key at high volume. It costs storage for the retention time and the work of choosing the partition count and key. A slow message also holds up its partition. I would use a log when events are facts that several parts read, or when I may need to replay.
• A queue solves jobs that are done once, with routing, priority and per-message acknowledgement. It costs the loss of the message after the ack, so there is no replay, and one queue per consumer for pub/sub. I would use a queue for background jobs.
• Event-driven design solves tight links between parts. It costs a flow that is hard to follow. I would use direct calls when the caller needs the answer now.
• More partitions solve a limit on parallel work. They cost a fixed plan: adding partitions later changes hash(key) mod number, so a key can move.

"Kafka is just a faster RabbitMQ." They keep messages in different ways. A log keeps messages for replay and many consumers. A queue removes them after the ack, and routes them by rules.

"More consumers in the group always means faster." Only up to the number of partitions. In our example, the 13th and 14th consumers read nothing.

"The topic keeps all messages in order." Only each partition keeps its order. Messages with different keys can be read in a different order than they were written.

1. In your own words, what are an offset, a partition and a consumer group? Name one thing a log can do that a queue cannot.
2. A topic has 8 partitions and gets 4,000 events per second at peak. One consumer handles 750 events per second. How many consumers does a group need? If only 4 consumers run for 10 minutes, how far behind is the group?
3. A messaging app has three services that must read every message: notifications, search and analytics. Messages of one chat must stay in order. What do you choose for the key and for the groups?