Cheap wins first: batching, compression, doing less

Interviewers say: "The peak needs 58 servers. What do you do before you buy them?" A weak answer is "add servers". A strong answer is "find the waste first", because less CPU per request means fewer servers.

Think about a delivery person with 21 parcels for one building. Making 21 trips is waste. One trip with all 21 parcels does the same job.

In lesson 15, one core does 1,000 ms of work per second. So a request that uses fewer milliseconds of CPU lets one server answer more requests. Servers = peak / (cores x 1,000 / CPU ms x 0.7). Waste is work that a request does but the user does not need. Measure first: time each step of one request, and fix the biggest.

There are three cheap wins.

1. Batching means doing many small jobs in one call. Each database call has a fixed cost: the user waits one round trip (0.5 ms in one data centre, lesson 14), and the app uses some CPU. A request loads 20 orders, then runs one query for the items of each order: 1 + 20 = 21 calls (lesson 12). One query for the items of all 20 orders makes 1 + 1 = 2 calls. Writes too: insert 100 rows in one statement. The cost: a big batch waits longer, uses more memory, and one bad row can fail the whole batch.
2. Compression means making the answer smaller before it travels. The client says Accept-Encoding: gzip (gzip is common). The server compresses the answer and says Content-Encoding: gzip. Text such as JSON gets much smaller. Photos and video are already compressed, so they gain nothing. Compressing costs CPU, so skip tiny answers.
3. Doing less means removing work. Send only the fields the screen needs (lesson 12). Return one page, not all rows (lesson 10). Add the missing index (lesson 20). Send an image at the size of the screen (lesson 14).

We reuse lesson 26. Assumptions (made up to teach with): peak 20,000 requests per second; servers with 4 cores, planned at 70% busy; Rs 10 per server per hour.

Step 1: the start. One server does 4 x 1,000 / 8 = 500 requests per second, and 500 x 0.7 = 350 at 70%. Servers = 20,000 / 350 = 57.1, so 58.

Step 2: time the steps. Assumption: the 8.0 ms of CPU has three parts. Each database call uses 0.1 ms: 21 x 0.1 = 2.1 ms. Each field of the answer uses 0.1 ms: 30 x 0.1 = 3.0 ms. The rest: 2.9 ms.

Step 3: fix two steps.

• Batching: 2 calls x 0.1 = 0.2 ms. We save 1.9 ms of CPU. The user also waits 19 x 0.5 = 9.5 ms less.
• Doing less: the screen needs 8 fields. 8 x 0.1 = 0.8 ms. We save 2.2 ms.
• New CPU = 8.0 - 1.9 - 2.2 = 3.9 ms.

Step 4: servers. One server does 4 x 1,000 / 3.9 = 1,026 (rounded), and 1,026 x 0.7 = 718. Servers = 20,000 / 718 = 27.9, so 28. This answers the predict box. We save 58 - 28 = 30 servers: Rs 300 per hour.

Step 5: compression. Assumptions: the answer is 8 KB, JSON shrinks to 25% of its size, and compressing costs 0.4 ms of CPU.

• Bandwidth before: 20,000 x 8 KB = 160 MB per second = 1,280 megabits per second. After: 40 MB per second = 320 megabits per second.
• CPU: 3.9 + 0.4 = 4.3 ms, so servers = 20,000 x 4.3 / 2,800 = 30.7, so 31.

Compression saved 960 megabits per second and cost 31 - 28 = 3 servers (Rs 30 per hour). It pays when the network is the limit.

• Fixing waste solves a big bill: here 30 servers fewer. It costs engineer time and new risk: big batches, extra CPU. I would add servers first if the peak is near and the fix takes weeks.

"Compression always makes it faster." It adds CPU work (3 servers here). A tiny answer or a photo gains nothing.

"A bigger batch is always better." One batch of 40,000,000 rows locks them all (lesson 25). Use a limit.

"Servers are the quick fix, and waste can wait." Waste costs Rs 300 for each hour that the extra servers run. But if the peak is tomorrow, add servers first.

1. Name the three cheap wins. Give one cost of each.
2. A request uses 10 ms of CPU. The peak is 12,000 requests per second. Servers have 8 cores, planned at 70%. How many servers before, and after cutting the CPU to 6 ms?
3. A team turns on compression for a service that sends photos. The CPU rises and the bytes do not change. Why, and what do you do?