What makes the bill explode

The interviewer says: "Your cloud bill doubled, and traffic grew only 10%. Where do you look?" A weak answer is "ask for a discount". A strong answer splits the bill into parts, finds what changed, and fixes the biggest part first.

Think about a home electricity bill: units used x price per unit. A cloud bill works the same way: each part is units x price (lesson 15). Four usual things make it jump.

1. Servers that are too big or too many. You pay every hour, busy or not. Check the CPU at the busiest time (lesson 91). If it is far below your target of 70% (lesson 15), the servers are too big. Right-sizing means choosing the number and size of servers so that the peak is near the target, with one spare. It does not mean 100% busy.
2. Full price for steady load. On-demand servers are paid by the hour, with no promise. Reserved capacity is a promise to use a fixed amount for one or three years, at a lower hourly price. You pay the promise even if you use less. Reserve only the load that never goes away.
3. Runaway usage. One customer, job or bug uses far more than expected. Examples are a retry loop (lesson 54) or a customer script that calls your API in a loop. Split the bill by service and by day, and ask what changed (lesson 94). Then set a limit for each customer (lesson 56) and an alert on daily spend.
4. Log cost. Log size = requests x lines per request x bytes per line (lesson 91). Common causes are debug lines left on and whole request bodies in the log. Fixes: keep every error but only a sample of routine lines. Never log full bodies. Delete old logs after a set number of days.

The food-ordering app. Assumptions (made up): a server costs Rs 10 an hour, and a month has 720 hours. Traffic averages 200 requests per second and peaks at 1,000. One server handles 100 per second at full load. The team runs 20 servers. Logging costs Rs 20 per GB written.

Step 1: right-size. Now 20 servers handle 2,000 at full load, so the peak is 1,000 / 2,000 = 50% busy. One server at 70% handles 70. Peak servers = 1,000 / 70 = 14.3, so 15, plus 1 spare = 16. Before: 20 x 720 x 10 = Rs 144,000. After: 16 x 720 x 10 = Rs 115,200.

Step 2: reserve the steady part. Assumption: the quietest hour has 150 requests per second, so 150 / 70 = 2.1, so 3 servers always run. On demand: 3 x 720 x 10 = Rs 21,600. Assumption: a one-year promise costs Rs 6 an hour, so 3 x 720 x 6 = Rs 12,960.

Step 3: the logs. Each request writes 3 x 500 = 1,500 bytes. Per day: 200 x 86,400 x 1,500 = 25.92 GB. Per month: 25.92 x 30 = 777.6 GB, which costs Rs 15,552.

After the debug release: 12 x 1,000 = 12,000 bytes per request. That is 12,000 / 1,500 = 8 times more, which answers the predict box. The month costs 8 x 15,552 = Rs 124,416. That is more than the whole server bill of Rs 115,200.

Step 4: runaway usage. Assumption: each call to one endpoint sends an SMS that costs us Rs 0.50. A script calls it 5 times a second for 28,800 seconds (8 hours): 5 x 28,800 x 0.50 = Rs 72,000. A limit of 1 call a second would cap it at Rs 14,400.

• Right-sizing solves paying for idle servers. It costs spare room. I would keep more room when traffic grows fast.
• Reserved capacity solves a lower price. It costs a promise. I would reserve only what runs most hours.
• Limits solve a runaway bill. They cost some blocked users. I would add them early.

"Cut all the logs to save money." Keep every error and sample the routine lines.

"Reserved capacity is always cheaper." A server is worth reserving only if it runs more than 6 / 10 = 60% of the hours. A server that runs 40% of the hours costs 720 x 6 = Rs 4,320 reserved, against 0.4 x 720 x 10 = Rs 2,880 on demand.

1. Name the four usual reasons a bill jumps.
2. A service has 500 requests per second. Each writes 4 lines of 250 bytes. Logging costs Rs 20 per GB. What is the log cost for 30 days?
3. A team wants to reserve 12 servers for a year. Only 4 run all day. The other 8 run the 6 busiest hours of each day. On demand costs Rs 10 an hour and reserved Rs 6. Is this a good plan?