Case study: E-commerce checkout and flash sale

Every shopping site needs checkout, and every big shopping site has a sale day. The interviewer asks: "Design checkout for an online shop. Now it is sale day, and one phone has 10,000 units for one million buyers. What happens?"

The question tests two things. First, can you follow an order from the cart to the payment, with clear stages? Second, do you know that only a few buyers can win, and that everyone else needs a fast and honest answer?

A weak answer lists services (cart, order, payment) and says "add more servers on sale day". A strong answer says where the stock is decided and how it stays correct. It says what happens to an order that is never paid. For the sale, it counts the load, finds the one part that cannot go faster, and protects that part.

At SDE-1 level you should list the parts and follow one order. At SDE-2 level you should stop overselling and handle unpaid orders. At SDE-3 level you should protect the one popular item and the rest of the store, with numbers.

Quick recap (lessons 22, 35, 58 and 66). Two buyers can both read "1 left" and both buy. The fix is one conditional update, such as UPDATE ... SET available = available - 1 WHERE available > 0. If 1 row changed, you won. If 0 rows changed, you lost. One row takes about 500 updates per second when each update takes 2 ms. A hold is a saved reservation with an end time. A queue lets slow work finish after the answer. A state machine is a list of states with a table of legal moves.

Think about a large shop on Diwali day. (Diwali is a big festival in India, and shops run big sales.) The shelves show the goods and prices. Each customer carries a basket. The stock room holds the real count of each item. At the counter, a clerk writes a bill and the customer pays. A guard at the door watches for thieves. Now one million people stand at the door for 10,000 phones. The shop lets in a few people at a time and tells the others the truth, quickly.

An online shop has the same parts. This lesson follows one order from the basket to a paid bill.

The parts

• Catalog. The products with prices and pictures. People read it all day, and it changes rarely. Pages come from a cache (lessons 31 and 32). Pictures and the page of a popular item come from a CDN (content delivery network). A CDN is a group of servers near the users that keep copies of pages and files. A label such as "Only 3 left" comes from a copy that is 1 to 2 seconds old. The label is for display. It never decides a sale.
• Cart. What the customer wants to buy. Store it by user ID in a key-value store (lesson 17), with a time to live of 30 days (Assumption). The cart holds no stock. The server reads the real price again when the order is placed. Never trust a price that comes from the phone.
• Inventory. The count of units for each product. It is the only place that decides who gets a unit. It uses the conditional update from lesson 22.
• Order. One row for each order, with a state (lesson 66).
• Fraud check. A quick risk check before the money is taken. It looks at simple signs, such as many orders to one address. It answers allow, review or block. A review order waits for a person. The check runs inside the request, with a short timeout (lesson 54). This bullet is all this lesson says about fraud.
• Payment. A separate service. A payment gateway is the service of a payment company that takes card and UPI payments for you. You call it with an idempotency key (lesson 61).

The path of one order

1. The customer taps Place order. The app reads the cart and the real prices.
2. The fraud check runs. Stop if it says block.
3. One database transaction (lesson 21) does three things:
   • It saves the order in the state PENDING_PAYMENT, with an end time for the hold. Assumption: the hold lasts 10 minutes.
   • It saves an outbox row (lesson 64), so the "order placed" message cannot be lost.
   • Last, for each item, it runs the conditional update on inventory, in product ID order (the same order each time avoids deadlock, lesson 22). If any update changes 0 rows, it rolls back, so the customer gets all items or none. The stock row is locked until the transaction ends (lesson 22), so this update comes last to keep the lock short.
4. The app answers "Order placed. Pay within 10 minutes." This is the promise, so it stays in the request.
5. The app calls the gateway with the order ID as the key. If the payment works, a conditional update moves the order from PENDING_PAYMENT to PAID. If it changes 0 rows, the order is no longer waiting, for example because the hold ended. Then you refund the money and tell the customer, as in lesson 22.
6. A timer job looks for orders still in PENDING_PAYMENT after the end time. It moves each one to EXPIRED with a conditional update. Only when that update changes 1 row does it add the units back, in the same transaction. So the units return once, even if the job runs twice.
7. After PAID, a queue (lesson 58) does the slow work: the SMS, the email, the invoice PDF and the message to the warehouse.

The legal moves of the order:

EXPIRED and CANCELLED are final states.

The fraud check, the stock update and the order row stay in the request. The customer needs their result, and the next step depends on them (lesson 58). The SMS, the email, the invoice and the warehouse message can wait, so they go to the queue.

When far too many people come

On a sale day, more people come than the system can serve. Three moves help.

1. Slow down (back-pressure). Put a line in front of the step that cannot go faster. A line is a queue (lesson 58). People enter the step at the speed of its slowest part. Lesson 63 said you cannot slow customers. A line works here because joining it is cheap and the customer sees a place.
2. Say no, quickly and cheaply (load shedding). Refuse some requests on purpose, with a clear answer such as "sold out". Do it before they use the costly parts.
3. Do less (graceful degradation). Switch off extras that are not needed to buy, such as recommendations and "people also viewed" lists. Show saved pages and old stock labels.

Use them for the two problems of a sale day.

Problem 1: the hot item. A flash sale is a short sale of a small stock at a special price. The hot item is the one item that everyone wants. All buyers want one stock row, and more servers do not make one row faster. Use the first two moves:

• Slow down. The row takes about 500 updates per second. So the line lets buyers in at 70% of that, which is 350 per second (the planning target of lesson 15).
• Say no cheaply. Joining the line costs much less work than placing an order. When the stock reaches zero, a tiny status file at the CDN says "sold out". It has four values (soon, open, few left, sold out), and the CDN keeps it for 1 second. All waiting buyers read it.

A unique constraint on (product, customer) for live orders (lesson 22) limits each customer to 1 unit.

Problem 2: the rest of the store. The other pages still get ten times the usual load. Warm the cache before the sale starts, and use request coalescing (lesson 35). Start extra servers before the sale, because the load grows faster than new servers start (Assumption: they need a few minutes). Switch off the extras first, and protect checkout most.

Big Billion Days is the name of a large yearly sale of an Indian shopping site. Here the words mean any big sale day.

Assumptions (made up to teach with, not real data):

• A flash item: 10,000 phones, on sale at 12 noon. 2,000,000 people wait on its page. The page asks for the status every 5 seconds.
• 1,000,000 of them press Buy within 10 seconds.
• One order locks the stock row for 2 ms until its transaction ends, so the row takes 500 orders per second (lesson 22).
• An app server has 4 cores. A page request or an order step uses 8 ms of CPU, and a join of the line uses 1 ms. We plan for 70% busy (lesson 15).
• 30% of orders are never paid (as in lesson 22). The store has 10 times its usual load: 14,000 catalog requests per second.
• A database read uses 4 ms of CPU. The database has 4 cores, which give 4,000 ms of CPU per second (lesson 35). The cache hit rate is 95%.

Step 1: the page. 2,000,000 / 5 = 400,000 requests per second. That is 400,000 / 14,000 = 28.6 times the sale-day load of the whole store. One server does 4 x 1,000 / 8 = 500 requests per second, and 350 at 70%. The page alone would need 400,000 / 350 = 1,143 servers, so the page and the status file come from the CDN.

Step 2: the press. 1,000,000 / 10 = 100,000 presses per second. Only 10,000 phones exist, so 10,000 / 1,000,000 = 1% of the people can win. 990,000 people need a quick "no".

Step 3: the stock row. 100,000 / 500 = 200. The load is 200 times too high. This answers the predict box: 200 times, and 1% can win. The conditional update keeps the result correct. It cannot make the row faster.

Step 4: the line. The line admits 350 buyers per second, which is 70% of 500. All 10,000 phones sell in 10,000 / 350 = 28.6 seconds, about 29 seconds. The admitted orders use 350 x 8 ms = 2,800 ms of CPU per second. That is 2,800 / 4,000 = 70% of one 4-core server. We use 2 servers, so one is a spare.

A join costs 1 ms. One server does 4 x 1,000 / 1 = 4,000 joins per second, which is 2,800 at 70%. So 100,000 / 2,800 = 35.7, which is 36 servers. The total is 36 + 2 = 38 servers. Without the line, all 100,000 presses use the 8 ms step: 100,000 / 350 = 286 servers. The cheap no needs 286 / 38 = 7.5 times fewer servers.

Step 5: the unpaid orders. 30% of 10,000 = 3,000 orders are not paid. After 10 minutes the timer moves them to EXPIRED and adds 3,000 units back. The status file says "few left" again, and the line sells them in 3,000 / 350 = 8.6 seconds.

Step 6: the rest of the store. With a 95% hit rate, 5% of the reads reach the database: 14,000 x 0.05 = 700 per second. That is 700 x 4 = 2,800 ms, so 2,800 / 4,000 = 70% of the database. Now suppose the cache is empty at the start of the sale. All 14,000 reads go to the database: 14,000 x 4 = 56,000 ms, which is 56,000 / 4,000 = 14 times the capacity. So we warm the cache with the popular pages before the sale, and we use request coalescing.

Step 7: the fraud check stops answering. Assumption: 350 orders per second reach 5 servers with 200 workers each, which is 1,000 workers (lesson 54). With no timeout, every order holds a worker. All workers are stuck after 1,000 / 350 = 2.9 seconds. With a timeout of 100 ms, 350 x 0.1 = 35 workers are busy, which is 3.5%. We also need a rule for a timeout. Assumption: an order from a customer with paid orders before passes, and any other order goes to review.

• Reserving stock when the order is placed solves carts that never end in a sale. It costs a buyer who reaches checkout and finds the phone sold out. I would change it only if the business wants to promise the unit at the cart, and then I would use a short hold.
• One conditional update solves overselling. It costs a limit of about 500 updates per second for one row. I would change the design when one row gets more than that. For the hot item, I put a line in front.
• A line solves the hot row and the server cost (38 servers instead of 286). It costs waiting, and one more part to run. I would use it only for items with a rush.
• A cached page and stock label solve the page load (400,000 requests per second). They cost a label that is 1 to 2 seconds old.
• Switching off extras solves the load of the rest of the store. It costs some sales that recommendations would bring.
• A fraud check solves bad orders. It costs time and wrong alarms. I would use a short timeout and a rule for when it times out.

"Take the unit out of stock when the customer adds it to the cart." Most carts never end in a sale. The first 10,000 people to add the phone would block it for everyone. Take the unit when the order is placed, and give it back when the hold ends.

"The stock label on the page is the truth, so I check it before I take the order." The label is 1 to 2 seconds old. Many buyers can see "1 left" at the same time. Only the conditional update decides who gets the unit.

"On sale day, add more servers and the store will cope." Servers do not make one stock row faster: the row was 200 times over its limit. Servers also do not fix an empty cache: the database was 14 times over its capacity. First find the part that cannot grow.

1. Name the six parts of checkout. Which one decides who gets the last unit, and which one may show old data?
2. A hot item has 2,000 units. One stock row takes 4 ms per update. 50,000 people press Buy in 5 seconds. (a) How many presses per second? (b) What is the limit of the row? (c) How many times too high is the load? (d) The line admits buyers at 70% of the limit. How long until all units are sold?
3. A customer pays at minute 11. The hold ended at minute 10, and the unit now belongs to another buyer. What happens, and what does the customer see?