Separate reads from writes; store events (CQRS, event sourcing)

Some candidates add "CQRS and event sourcing" to every design because the words sound senior. The interviewer asks: "Why? What problem does it solve here, and what does it cost?"

A weak answer repeats the words and names a tool. A strong answer starts from a problem: this screen reads too much, or we must keep the full history. It says what the new parts are, what delay the user may see, and how much extra work the team takes on. It also says when plain tables are enough, which is most of the time.

At SDE-2 level you should explain a read copy that is filled from events. At SDE-3 level you should say when not to use any of this.

Think of a restaurant. The counter takes orders and writes a ticket. It follows strict rules: the dish exists, the price is right. A big board on the wall shows "order 981: cooking" for waiting customers. The board is built from the tickets, in the shape that people need to read. The two jobs differ, so they use different tools.

Now think of a bank passbook. It never erases a line. It only adds lines. The balance is the sum of all lines. To fix a mistake, the bank adds a new line.

Where one model hurts

In lesson 19 you counted the rows that each query reads, and you copied facts on purpose. In lesson 14 the example had 20 reads for each write. The write side wants clean tables, rules and transactions. The read side wants a shape for each screen. One set of tables for both fits neither job well: the reads are slow, or the writes are hard to keep right.

CQRS

A command asks the system to change something, such as "place this order". A query reads and changes nothing. Command Query Responsibility Segregation (CQRS) means that commands and queries use separate models. (Segregation means keeping things apart.)

• The write model keeps the clean tables of lesson 18. It checks the rules inside a transaction (lesson 21). It is the source of truth (lesson 19).
• A read model is a copy shaped for one screen. It is the "second table for one question" of lesson 19.

There are two levels:

1. Same database. The read tables are updated in the same transaction as the write tables (lesson 19). The read model is always right. It costs little, and it is a good first step.
2. Separate store. The read model lives in another store that suits the query. Events fill it. The order service saves the order and an outbox row in one transaction (lesson 64). A relay sends the row to a log (lesson 59). A projector is a consumer that reads these events and updates the read model.

The projector must handle repeats, because delivery is at-least-once (lesson 64). It saves the number of the last event it applied, in the same update as the change. It skips any event with a number it has already seen. The event key is the customer ID, so one customer's events stay in order (lesson 59).

The result is that the read model is eventually consistent (lesson 40). A short time passes between the save and the update. A user who places an order and opens "My orders" at once may not see it. Two fixes from lesson 40 work here. Show the new order from the save response. Or read that user's own orders from the write side for 30 seconds.

A read model can be thrown away. If a screen needs a new shape, build a new read model by replaying the events (lesson 59), and then switch to it.

Event sourcing

Normally the database holds the current state, and an update overwrites the old value. History is gone. Event sourcing stores every change as an event, which is a fact in the past tense (lesson 59), such as "OrderPlaced". Events go into an event store. It is append-only: events are added, and never changed or deleted. The stream of one order is its events, in order.

To find the current state, start empty and apply the events of the stream one by one. For a long stream, a snapshot helps. It is a saved copy of the state after event number N. You load the snapshot and apply only the later events.

Good events say what the user meant ("two seats reserved"), not only the new state ("seats left: 42"). To fix a mistake, append a new event that undoes it, such as "OrderCancelled". Do not edit the old one.

Two writers can append to one stream at the same moment. Use an expected number, as in optimistic locking (lesson 22). Each event row has a stream ID and an event number, with a unique constraint on the pair. A writer inserts number last + 1. If another writer got there first, the insert fails, and the writer reloads the stream and decides again.

The costs are real:

• Event shapes change. Old events keep the old shape, so the code must read every version (lesson 11).
• No questions about current state. You cannot ask "all orders that are still cooking" of an event store. A read model must answer that. So event sourcing nearly always comes with CQRS. CQRS can be used alone.
• Deleting a person's data is hard, because events never change. A common answer is to keep personal data elsewhere and put only its ID in the event.
• A rebuild takes time. The worked example gives a number.

When it pays

Think about CQRS when reads far outnumber writes and many screens need different shapes, or when reads and writes need different storage or scale. Think about event sourcing when you must keep the full history, or must be able to rebuild state from facts. Most data needs neither. Martin Fowler warns that CQRS adds risky complexity for most systems. Try cheaper steps first: an index (lesson 20), a copied field (lesson 19), a follower (lesson 39), or a cache (lesson 31). Use these ideas for one part of a system, such as orders, and not for all of it.

SwadKart, the food app of lesson 19. At the peak there are 120 new orders per second, and the screen "My orders" runs 3,000 times per second. With the clean tables it reads 100 rows each time. With copies in the same database (lesson 19) it reads 20 rows.

Assumptions (made up to teach with):

• Each order makes 5 events: 1 placed and 4 status changes (lesson 19).
• One projector handles 1,000 events per second. Each event updates one document.
• The delay from save to read model is about 1 second (outbox loop, log, projector).
• An event is 300 bytes. A current-state record is 1 KB (lesson 14). There are 1,000,000 orders a day. Applying one event in memory takes 0.1 ms.

Step 1: the read side. The read model is one document for each customer. It holds the 20 newest orders, with restaurant name, total and status. "My orders" reads 1 document. That is 3,000 documents per second. The clean tables read 3,000 x 100 = 300,000 rows (100 times more). The copies read 3,000 x 20 = 60,000 rows (20 times more).

Step 2: the cost. At the peak there are 120 x 5 = 600 events per second. The projector is 600 / 1,000 = 60% busy. The write side only adds one outbox insert for each order.

Step 3: the predict box. The delay is about 1 second, so after half a second the order is probably not there. Fix: for 30 seconds after a customer saves, read that customer's own orders from the write side. At the peak, up to 120 x 30 = 3,600 customers are in this 30-second period.

Step 4: the projector stops for 10 minutes. 10 minutes is 600 seconds, so the backlog is 600 x 600 = 360,000 events. It catches up at 1,000 - 600 = 400 events per second: 360,000 / 400 = 900 seconds, which is 15 minutes. With 2 projectors, split by customer ID, the speed is 2,000. Catching up is 360,000 / (2,000 - 600) = 257 seconds, about 4.3 minutes.

Step 5: events as storage. 1,000,000 orders x 5 events x 300 bytes = 1,500,000,000 bytes = 1.5 GB per day. A year is 1.5 x 365 = 547.5 GB. Current-state records would be 1,000,000 x 1 KB = 1 GB per day. Events take 1.5 times as much space, and they keep the history.

Step 6: snapshots. One order has 5 events: 5 x 0.1 ms = 0.5 ms to rebuild. A restaurant's money stream may have 100,000 events: 100,000 x 0.1 ms = 10,000 ms = 10 seconds. With a snapshot every 1,000 events, at most 1,000 x 0.1 ms = 100 ms. That is 100 times less.

Step 7: a rebuild. A bug gave wrong data in a read model, so we rebuild it. One year has 365,000,000 orders x 5 = 1,825,000,000 events. One projector at 1,000 per second needs 1,825,000 seconds, which is about 21 days (1,825,000 / 86,400 = 21.1). With 20 projectors, each on its own part of the log (lesson 59), it takes 1,825,000,000 / 20,000 = 91,250 seconds, about 25 hours. Plan for that.

Step 8: two writers on one stream. Order 981 has events 1 (placed), 2 (paid) and 3 (packed). The kitchen sends "cancel" and the rider app sends "dispatch" at the same time. Both read the stream at event 3 and try to insert event number 4. The unique key lets one insert win. The other fails, reloads, sees "cancelled" as event 4, and refuses to dispatch. No lock was held.

Step 9: a profile page. A customer's name and phone are read by key and changed once a month. A read model would save nothing: it reads 1 row already. Keep one table.

• A read model solves slow reads and awkward shapes. In the example, a screen went from 100 rows to 1 document. It costs a projector to run, a delay of about 1 second, and a rebuild plan. I would use it for a heavy screen after cheaper steps have failed.
• A separate store solves different storage and scale for reads. It costs a second system and eventual consistency. I would first try tables in the same database, updated in one transaction.
• Event sourcing solves full history and rebuilds from facts. It costs more space (1.5 times), events that change shape, a day-long rebuild, and a read model for every question. I would use it where history is the product, such as money movements.
• I would change the choice when the delay hurts users, or when a rebuild takes too long. Add snapshots and more projectors.

"CQRS means two databases and event sourcing." It does not. Separate models can live in one database, as in level 1. Event sourcing is a separate choice.

"Event sourcing is an audit log." An audit log is a side record, and the tables still hold the truth. In event sourcing the events are the truth, and the state is calculated from them.

"The read model always shows the newest data." It lags. In the example the delay is about 1 second. If the projector stops for 10 minutes, the delay grows to 10 minutes, and it takes 15 more minutes to catch up.

1. In one sentence each: what is CQRS, what is event sourcing, and what does a projector do? Which one can be used without the other?
2. A shop has 80 orders per second at the peak, with 6 events per order. One projector handles 800 events per second. (a) How many events per second at the peak? (b) How busy is the projector? (c) The projector stops for 5 minutes. How big is the backlog, and how long does it take to catch up?
3. A team wants event sourcing for the user profile page. The page shows a name, a phone number and a photo. It is edited once a month and read 500 times per second by user ID. What do you say?