Stop calling a broken service (circuit breaker, bulkhead)

The interviewer says: "The report server is stuck. Why do order pages stop too?" A weak answer says "add a timeout". A strong answer says that waiting calls fill the workers and keep hitting the failing server. It names two tools: stop the calls, and give each service its own workers.

A fuse box has a switch for each line. A faulty line trips its own switch. A ship has walls between rooms, so one flooded room does not sink it.

In lesson 54 a dependency is a server you call, and a call holds a worker while it waits. A timeout stops one call from waiting forever. But if the dependency is down, every call still waits the full timeout, and the failing server keeps getting calls.

A circuit breaker is code around the calls to one dependency. When many calls fail, it stops sending calls for a while. Each call fails at once: failing fast. It has three states:

1. Closed. Calls go through. The breaker counts recent results.
2. Open. At least 5 of the last 10 calls failed. Calls fail at once. A timer starts, for example 10 seconds.
3. Half-open. The timer ended. A few trial calls go through, for example 3. If they succeed, the breaker closes. If one fails, it opens again.

Timeouts, broken connections and 503 count as failures. Most 4xx do not (lesson 54). While the breaker is open, the caller returns an error, or the last value it saved. Do not retry a refused call. Use one breaker for each server or shard (one part of a split database, lesson 43).

A bulkhead gives each dependency its own limit on workers, called a pool. When the pool is full, a new call to that dependency fails at once. The other workers stay free. Pool size = calls per second x normal time per call x a spare factor (our own advice).

A breaker reacts after it sees failures. A bulkhead limits damage from the first second.

An API server has 200 workers. Assumptions (made up):

• 400 product views per second. Each calls the price server: 50 ms, timeout 300 ms.
• 50 order-status requests per second, 50 ms, no call.
• 20 statement downloads per second. Each calls the report server: 2 seconds, timeout 5 seconds.
• Breaker: 5 of the last 10 calls, 10 seconds open, 3 trial calls. An open breaker answers in 1 ms.

Step 1: normal. Busy workers: views 400 x 0.05 = 20, status 50 x 0.05 = 2.5, statements 20 x 2 = 40. Total 62.5 = 31%.

Step 2: price server down, timeout only. Price calls hold 400 x 0.3 = 120 workers. Total = 120 + 2.5 + 40 = 162.5, which is 81%.

Step 3: the same outage, with a breaker. The first calls fail at 300 ms, so the breaker opens after about 0.3 seconds. Price calls then hold 400 x 0.001 = 0.4 workers. Total = 0.4 + 2.5 + 40 = 42.9, which is 21%. The price server gets 3 trial calls per 10 seconds, about 0.3 per second: 400 / 0.3 = about 1,333 times fewer.

Step 4: the report server stalls. Assumption: a busy hour brings 60 statement requests per second, each stuck 5 seconds. Workers needed = 60 x 5 = 300. All 200 are stuck after about 200 / 60 = 3.3 seconds, and order pages fail too. This answers the predict box. A breaker is too late: the first call fails only at 5 seconds.

Step 5: a bulkhead. Normal need is 20 x 2 = 40 workers. We choose a pool of 60, which is 1.5 times that. Calls that can start per second = 60 / 5 = 12. The other 48 fail at once. The other 140 workers need 22.5, which is 16%.

A breaker solves workers stuck on a broken service. It costs refused calls that might have worked. A bulkhead solves one slow service using every worker. It costs idle workers in one pool. I would resize a pool when traffic grows, and skip the breaker for rare calls.

"A timeout is enough." In Step 2 a timeout still held 120 workers, and every call reached the failing server.

"A breaker fixes the broken service." It protects the caller. Someone must still fix the service.

"One big pool wastes nothing." True on a normal day. In Step 4 it lost all 200 workers.

1. Name the three breaker states. What is a bulkhead?
2. A server has 100 workers. A dependency is called 50 times per second, and each call waits 2 seconds. (a) How many workers does it hold? (b) With a pool of 30, how many calls per second start, and how many fail at once?
3. One breaker covers 8 shards. One shard fails, and all calls fail. What changes?