Batch size and buffering vs latency

The decision

Batching means grouping many small items into one request or one write. Buffering means holding items for a short time so a group can form. Every request has a fixed cost, such as a network round trip, a disk flush or a commit. A batch shares that cost among many items, so throughput rises. The price is that an item waits for its group, so latency rises, and the items in the buffer are lost if the process dies. There are two settings. The batch size is the most items or bytes in a group. The wait time is the longest an item waits for the group to fill. The Kafka producer calls the wait time linger.ms.

Choose small batches or no buffering when

• The user waits for the result. A payment confirmation cannot sit in a buffer for 50 ms.
• Traffic is low. At 10 messages a second, a group of 100 never fills, so every item waits for the timer.
• Losing the buffered items is not acceptable, and you cannot replay them.
• The system has plenty of spare capacity, so the fixed cost per request is not a problem.

Choose large batches or longer buffering when

• Nobody is waiting for a single item: logs, metrics, analytics events, search indexing and copying data.
• The fixed cost per request is high compared with the cost per item. You are near the limit of a CPU, a network card or a database.
• Traffic is high enough that groups fill fast. At 2,000 messages a second, a 16 KB group of 1 KB messages fills in 8 ms.
• You can replay lost items from a source, or the loss is small.

The numbers that change the answer

Requests saved (calculation). Assumption: a producer sends 2,000 messages a second, each 1 KB. For now we ignore any size limit.

• Wait time 5 ms: 2,000 x 0.005 = 10 messages per group. Requests: 2,000 / 10 = 200 a second instead of 2,000.
• Wait time 50 ms: 2,000 x 0.05 = 100 messages per group. Requests: 20 a second.

The size limit hits first. A cited default of the Kafka producer is a batch size of 16,384 bytes (16 KB). At 1 KB per message that is 16 messages, which fills in 16 / 2,000 = 8 ms. A wait of 50 ms then changes nothing at this rate, because the size limit sends the group first. The setting that matters is the one that triggers first.

The default wait depends on the version (cited fact). The Kafka 4.1 documentation shows a linger.ms default of 5 ms. It says the default changed from 0 to 5 in Kafka 4.0. An older client may have no wait at all, so check your version (see Sources).

Throughput gain (assumptions and calculation). Assumption: each request costs 0.2 ms of CPU however large it is, and each message costs 0.01 ms. Cost per message = (0.2 + group size x 0.01) / group size.

The first step, from 1 to 10, is worth 7 times (33,333 / 4,762 = 7). The step from 16 to 100 adds less than 2 times (83,333 / 44,444 = 1.9), and it costs more waiting. The gain flattens because the per-message cost of 0.01 ms stays.

Latency cost (calculation). The extra delay is at most the wait time. At low traffic it is the whole wait. At 10 messages a second and 50 ms, a group holds 10 x 0.05 = 0.5 messages. There is no gain, and every item pays 50 ms. Our own simple test: batching helps when rate x wait time is well above 1, say 10 or more. In Kafka the rate that counts is the rate for one partition, because each partition has its own groups. With 20 partitions, 2,000 messages a second is only 100 a second each. A 5 ms wait then gives 0.5 messages per group.

Data at risk (calculation). At 2,000 messages a second and a 50 ms wait, 2,000 x 0.05 = 100 messages sit in memory. A crash can lose all 100 if you have not stored them elsewhere. With 5 ms it is 10. A producer buffer also has a limit, 32 MB by default in Kafka. When it fills, the producer blocks for up to 60 seconds by default and then fails. A slow server turns into stuck application threads (see Sources).

Tail effects. A big group is one unit for failure and retry. If a 100-message group fails, all 100 are retried. One slow item delays the 99 others.

What breaks with each choice

Too little batching.

• The system hits its CPU or network limit early, and cost per message is high. Fallback: raise the wait time in small steps and watch the p99.
• Brokers or databases get many tiny requests. Fallback: group by destination.

Too much batching.

• Users feel a delay. Fallback: separate paths, a fast path with no wait for user actions and a batch path for background data.
• A crash loses the buffer. Fallback: write to a local log first, or use acknowledgements that wait for storage.
• The buffer fills when the receiver slows, and threads block. Fallback: a bounded buffer with a clear rule: drop old items, or fail fast.
• Uneven traffic: groups are full at peak and nearly empty at night, so latency changes through the day. Fallback: use both a size limit and a wait limit.

What it costs

• Money: batching usually lowers it. Going from 4,762 to 33,333 messages a second per core is about 7 times fewer cores for the same work (33,333 / 4,762 = 7).
• People (Assumption): one engineer-day to change settings and test, and one more week for a separate fast path.
• Changing your mind: settings are configuration, so you can change them in minutes and watch p99. The hard part is the second path for urgent items.

How each level answers

SDE-1: Batching groups items into one request, which is faster per item. The cost is that an item waits.

SDE-2: For analytics events I use a wait time of 20 to 50 ms, because no user waits. For payments I send at once. I set both a size limit and a time limit.

Senior: At 2,000 messages of 1 KB a second, a wait of 5 ms gives groups of 10 and cuts requests by 10 times. The 16 KB size limit fills in 8 ms, so a longer wait does nothing. A group of 100 adds under 2 times throughput over 16 and holds 100 messages at risk. I would stay near 5 to 20 ms, and keep user actions on a no-wait path. I check the rate for each partition, not the total. If the process crashes I need a log to replay.

Staff: I ask which flows face customers and which run in the background. With product I set the wait time for each: tens of milliseconds for background, none for payment. I own the setting with the platform team. One dashboard shows cost per message, p99 delay and data in the buffer. I start with the default of my client version. I raise the wait when the CPU is above about 60 percent (Assumption) and the latency budget has room. I revisit it when traffic doubles, because the group size changes with rate.

Say it in the interview

For background events I would batch with a size limit of 16 KB and a wait of about 5 ms. At 2,000 messages a second that cuts requests by about 10 times, and the added delay is at most 5 ms. The risk is that a crash loses the buffered messages and a slow receiver fills the buffer. So I would keep user actions on a path with no wait, and use a bounded buffer and a replayable log.

Common mistakes

• Raising the wait time without checking the rate. At low traffic the group never fills, so you add delay for no gain.
• Tuning only one limit. The size limit may fire first.
• Batching user-facing writes. The user feels every millisecond of the wait.
• Forgetting the buffer is memory. A crash loses it, and a full buffer blocks the caller.

Sources

• Apache Kafka 4.1 documentation, producer configuration (linger.ms default 5 ms, and the page says the default changed from 0 to 5 in Kafka 4.0; batch.size default 16,384 bytes; buffer.memory default 33,554,432 bytes, and the producer blocks for max.block.ms, default 60,000 ms, before failing): https://kafka.apache.org/41/configuration/producer-configs/ (opens in a new tab) (checked 26 Oct 2026). Defaults differ between versions, so read the page for your version.
• The message rate, message size, per-request and per-message costs, the 60 percent CPU level and the effort figures are assumptions made for teaching. The group sizes, request counts and cost per message are our own calculations.