Cost engineering: cost per request as a design input

The interviewer says: "Your design handles 10,000 requests per second. What does one request cost? Which design is cheaper?" A weak answer says "servers are cheap". A strong answer gives a number and names the part to cut.

A tea stall pays rent and a helper every day, even with no customers. It also pays for milk and sugar for each cup. One cup costs the daily bill divided by the cups sold.

Cost per request is the bill for a period divided by the requests in that period. The same works for one user, one GB or one prediction. Cost per 1,000 requests is the bill divided by the thousands of requests.

The bill has two parts (lessons 31 and 130):

• Fixed part: paid with no traffic, such as servers that run all month.
• Part for each request: compute time and memory, calls to a managed service, storage by kind, data moved out of the cloud or between zones, and logs.

So cost per request = fixed part / requests + per-request part. More traffic spreads the fixed part, so a busy service pays less for each request, until it must add servers. Use the whole bill, including idle servers. Capacity paid only while a job runs counts only those hours.

Three levers lower it: do less work for each request (lesson 53), use cheaper capacity (reserved for steady load, spot for work that can restart), or move work to a cheaper time or place, such as a night batch on idle servers. Alert on cost per request, not only on the total (it rises with traffic).

An order-status API. Assumptions (made up, not quotes): 30,000,000 requests a day, so 900,000,000 in 30 days. Six servers at Rs 10 an hour: 6 x 10 x 720 = Rs 43,200. A managed database charges Rs 0.02 per 1,000 reads, and each request reads twice. Logs are 1 KB per request at Rs 20 per GB. The answer is 2 KB at Rs 8 per GB sent out.

Design A.

• Reads: 1,800,000,000 / 1,000 x 0.02 = Rs 36,000.
• Logs: 900 GB x 20 = Rs 18,000. Data out: 1,800 GB x 8 = Rs 14,400.
• Total: 43,200 + 36,000 + 18,000 + 14,400 = Rs 111,600. Per 1,000 requests: 111,600 / 900,000 = Rs 0.124. Fixed Rs 0.048, per-request Rs 0.076.

Doubling (predict box). The per-request part doubles to Rs 136,800. The total is 43,200 + 136,800 = Rs 180,000 for 1,800,000,000 requests: Rs 0.100 per 1,000, so it goes down.

Design B. Add one cache server (Rs 7,200). Assumption: 80% of reads hit the cache, so reads cost Rs 7,200. Keep 1 in 10 log lines (all routine): Rs 1,800. Total: 43,200 + 7,200 + 7,200 + 1,800 + 14,400 = Rs 73,800, or Rs 0.082 per 1,000. This saves 37,800 a month. Assumption: the work takes 2 engineer-weeks at Rs 62,500 = Rs 125,000, so it pays back in 125,000 / 37,800 = 3.3 months.

Alert. Divide each day's bill by its requests. Alert at 25% above that weekday's usual value (our choice).

Not worth it. A job costs Rs 3,000 a month. Halving it saves Rs 1,500. One engineer-week is Rs 62,500, which is 41.7 months of savings, unless the job will grow.

Hotstar wrote on 19 Oct 2026 that its 7.5 KB playback signal has six consumers at 2.1 million messages per second. Most need about 1.1 KB, so it trimmed the message before fan-out. Its title says 350,000 dollars a year saved, a projection (company figures). Cost grows with bytes x consumers x rate. Link: https://blog.hotstar.com/the-kafka-optimization-that-saved-us-350k-a-year-48a0e05c3c08 (opens in a new tab)

Unit cost solves comparison: it shows which design is cheaper and when the bill outgrows traffic. It costs effort to count requests and split shared bills. I would use it for any large bill, not a small one.

"A lower bill means a lower cost per request." Traffic may have fallen.

"Servers are the cost." In design A they are 39%.

1. Define cost per request. Name the three levers.
2. A service has a fixed bill of Rs 20,000 and pays Rs 0.05 per 1,000 requests. It serves 500,000,000. Cost per 1,000?
3. A fix saves Rs 2,000 a month and takes 3 weeks at Rs 62,500 a week. Do it?