Testing a system: unit, integration, contract, load, chaos

The interviewer asks: "How will you know it works before users find out?" A weak answer is "just test it". A strong answer names each test and what it catches. At SDE-3 level, add a test that breaks a part on purpose.

Think of a bridge. Engineers test each bolt, then how parts fit, then trucks, then a broken cable.

• A unit test runs one small piece of code, a unit, such as a function. It catches wrong logic.
• An integration test runs your code with the real parts it uses, such as a real database. A fake is a stand-in that returns fixed answers. It can hide a mismatch, such as a wrong column name. A real part shows it.
• A contract test checks the agreement between a caller and a provider, the service it calls (lesson 11). The caller lists what it reads, such as field names. The provider's tests must pass against that list, so a rename fails before release.
• A load test sends many requests and measures p99 and errors (lessons 4 and 91) against a goal, your target. A breakpoint test raises the load until the goal breaks. A soak test runs for hours and finds slow leaks of memory.
• A chaos test breaks a part on purpose and checks that users stay safe.

Write many unit tests and fewer of the others (the test pyramid).

Servers needed

Choose a target CPU share: the most CPU you allow at peak, such as 80%. Servers needed = load / (rate per server x target CPU share). If the goal must hold when one server stops, add one more server, so the rest carry its load.

How to break a part on purpose

A fault is something going wrong. You can stop a server, slow a call, make a call return errors such as 503 (lesson 54), cut the network, or fill a disk. A proxy, a server that passes calls between two others, can add delay or errors with no code change. Then:

1. Write the steady state: p99 and error rate (lesson 91).
2. Write a hypothesis: "If one server stops, errors stay under 1%."
3. Limit the blast radius: how many users can be hurt. Start in staging, a copy of the system for tests. Then try a small share of live traffic (our advice).
4. Set a stop line, for example errors above 2%, and undo the fault if it is crossed.
5. Inject the fault, compare, and fix what you find.

The checkout of lesson 91. Assumptions: 125 requests per second per server at 100% CPU, 4 servers, peak 200, sale at 3 times the peak. Goal: p99 under 1,000 ms, CPU at most 80%. A coupon gives 10% off, at most Rs 50.

1. Unit. An order of Rs 800: 10% is Rs 80, the cap is Rs 50, so the user pays 800 - 50 = Rs 750. Code with no cap gives 720, and the test fails.
2. Integration. The code saves amount, but the real column is total. The fake accepted it. The real database fails.
3. Contract. The payment team renames status to state. Checkout's list says status, so the provider's run fails.
4. Load. Capacity is 4 x 125 = 500. Assumption: a test shows p99 700 ms at 400 requests per second (80% CPU), and over 1,000 ms at 600. The sale is 3 x 200 = 600. Then 600 - 500 = 100 requests have no room: 100 / 600 = 16.7%. Servers needed = 600 / (125 x 0.8) = 600 / 100 = 6.
5. Chaos. Hypothesis: with 6 servers and one stopped, the goal holds. Five remain and take 600 / 5 = 120 each, which is 96% CPU. The hypothesis fails. With 7 servers, six remain: 600 / 6 = 100 each, which is 80%. So the answer to the predict box is 6 servers, and 7 if one may stop.

Layered tests solve faults found late. They cost time to write and run. Load and chaos tests solve surprises on sale day. They cost a test copy of the system and some risk. I would repeat them after big changes.

"All unit tests pass, so the system works." The fakes hid the wrong column name and the renamed field.

"Chaos testing means breaking production at random." It is an experiment with a hypothesis and a stop line.

1. What does a contract test catch that a unit test with fakes cannot? Name the five steps of a chaos test.
2. One server handles 100 requests per second at 100% CPU. Peak is 450. CPU must stay at 75% or less. How many servers do you need, and how many if one may stop?
3. All unit tests pass, but checkout fails after release. The payment service now returns state, but the unit tests use a fake that returns status. What was missing?