Debugging an incident: a method

The interviewer says: "It is 9 pm and orders are failing. You are the engineer on call. What do you do?" Nobody expects you to guess the cause. The interviewer wants a calm method.

A weak answer jumps to one guess, such as "the database is down", or reads logs for an hour while users keep failing. A strong answer has four steps and says why they come in that order. It stops the damage before it finds the full cause. It uses numbers from the graphs, not a feeling. It also tells people what is happening.

At SDE-1 level you should name the four steps. At SDE-2 level you should read a dashboard, narrow the problem and choose a safe first action. At SDE-3 level you should also weigh the risk of each action and stop the problem from coming back.

Think about a water leak at home. You see water on the floor. First you close the main tap, so the floor stops getting wetter. Then you look for the broken pipe. Then you repair it and fit a cheap alarm. You did not wait to find the pipe first.

An incident is a time when real users are hurt because a service is slow, wrong or down. The method has four steps:

Mitigate means to reduce or stop the harm to users, without knowing the full cause. It is not the fix.

Detect

An alert is a message that goes to a phone when a metric (lesson 91) crosses a line. Someone chose the line, for example "errors above 5%". The on-call engineer is the person whose turn it is to answer alerts.

When the alert arrives, do four things:

1. Write down the start time: when the numbers first went bad. It is before the alert.
2. Measure the harm: how many requests fail each second, and which feature.
3. Tell people. A status update has three facts: what users see, what you are doing, and when the next update comes.
4. In a big incident, name one leader who keeps the timeline and decides.

Narrow

Narrow means to make the search area smaller, step by step, until you can name the place. Ask three questions:

• When did it start? Find the time when the graph line suddenly goes up or down.
• What changed near that time? Make a list: a release (a new version of the code), a setting, or a change in traffic.
• Where is it? Which route, which server, which customer?

Then use lesson 91: a metric shows where, a trace shows which step is slow, and a log line shows why. A hypothesis is a guess that you can test. Write it down. Look for one fact that would prove it wrong. If you find none, you can act on it.

Reading a dashboard

A dashboard is a screen of graphs of your metrics. These rules help you read one:

1. Find the time the line changes. Compare it with your list of changes.
2. Look at all four golden signals (latency, traffic, errors, saturation). The patterns below are common.
3. Split the number. An overall number can hide the place. Suppose 5 servers share the traffic equally, and one server fails half of its requests. The overall error rate is 50% / 5 = 10%. Split the metric by server and the bad one is clear.
4. Ask what moved first. A high CPU can be an effect, not the cause. The signal that moved first is closer to the cause.
5. Use percentiles, not averages (lesson 4).

Mitigate

Mitigate before you fix, because users are hurt every minute. Pick the first action by four tests:

1. Fast: it works in minutes.
2. Safe: it should not make things worse.
3. Reversible: you can undo it.
4. Matches the evidence: it acts on the place you found.

These are the common actions:

Change one thing at a time, so that you can tell which one helped.

Then confirm. Watch the same metric that showed the problem. The incident is mitigated when that metric is normal again. Time to mitigate is the time from the start of the harm until it stops.

Fix

When users are safe, find the root cause: the reason that, if you remove it, stops the problem from happening again. Keep asking "why" until you reach something you can change. Then fix the cause. Then prevent a repeat with an alert that would fire sooner, a test, or a limit. Then write a postmortem: a short report with what users saw, a timeline, the cause, what you did and what you will change.

A postmortem is blameless. It asks what in the system allowed the mistake, not which person made it. If people are blamed, they hide mistakes.

We use the food-ordering app of lesson 3.

Assumptions (made up to teach with):

• 4 app servers sit behind a load balancer. It shares requests equally. Total traffic is 200 requests per second, so each server gets 200 / 4 = 50 requests per second.
• One server can handle 125 requests per second at 100% CPU. CPU grows with requests. So 50 / 125 = 40% CPU, as in lesson 91.
• An alert checks every 5 minutes. It fires if the error rate in the last 5 minutes is above 5%.
• At 19:40 a release changed some text on the home page. It went to all 4 servers.
• The app saves a receipt file on the local disk of the server for each order.
• The load balancer sends each server a health check: a small test that runs again and again. It stops sending requests to a server that fails the test. Here the test asks only "is the program running?".

Step 1: detect (20:10). The alert fires: error rate 9%. The graph shows the start time: 20:05. The harm is 200 x 9 / 100 = 18 failed requests per second. The alert came 5 minutes after the start. That is 300 seconds, so 18 x 300 = 5,400 requests had already failed. At 20:11 we send a status update: "Some customers cannot place orders. We are investigating. Next update in 20 minutes."

Step 2: narrow (20:11 to 20:15).

• When: errors start at 20:05.
• What changed: the release of 19:40 is 25 minutes earlier. It went to all 4 servers, so a bad release would hurt all four.
• Four signals: traffic is 200 requests per second (normal), errors 9%, p99 latency 300 ms (normal), CPU 40%. Nothing is full or slow, but requests fail.
• Split by server: the errors are 0% on servers 1, 2 and 4, and 36% on server 3. Check: 36% x 50 = 18 failed per second, and 18 / 200 = 9%.
• Disk gauge: the disk of server 3 reached 100% at 20:05. The other three are at 61%.
• Log of server 3: {"level":"ERROR","service":"checkout","msg":"cannot write receipt: no space left on device"}

Our hypothesis: the disk of server 3 is full, so it cannot save the receipt. A fact that would prove it wrong is a disk with free space. We see the opposite, so we can act on it.

This answers the predict box. One bad server out of 4 gives an overall rate of the failing share divided by 4. So the failing share is 9% x 4 = 36%. The first action is to remove server 3 from the load balancer.

Step 3: mitigate (20:16). We test the options against the four tests:

• Roll back the 19:40 release: slow, and it does not match the evidence, because 3 servers have it and work.
• Restart server 3: fast, but the disk stays full.
• Remove server 3 from the load balancer: under a minute, and the harm is on server 3 only.

Is it safe? Three servers get 200 / 3 = 66.7 requests per second each (rounded). CPU is 66.7 / 125 = 53%, which is below 100%. It is reversible: we can add the server back. Assumption: p99 stays near 300 ms at 53% CPU.

At 20:17 we confirm. The error rate of the last minute is 0%, and p99 is 310 ms, which fits our assumption. Time to mitigate: 20:05 to 20:16 = 11 minutes = 660 seconds. Failed requests: 660 x 18 = 11,880.

Step 4: fix (next morning). We keep asking why:

Why did requests fail? The disk of server 3 was full. Why? Receipt files stay on the local disk of each server. Why only server 3? Assumption: the other three have a job that deletes old receipts, and server 3 was set up by hand without it.

The root cause: servers are built differently, and they keep files on their own disk. We do these things:

• Move receipts to object storage (lesson 24, a store for files), so servers keep no files of their own (lesson 26).
• Alert when any disk is above 80%. This line is our choice.
• Make the health check write a small test file, so a server with a full disk is removed by itself.
• Write a blameless postmortem: 11,880 failed requests, 5 minutes from the start to the alert, 11 minutes from the start to mitigation, the cause and the changes above.

• Mitigating first solves the harm that grows every minute: here 18 failed requests per second. It costs a risk that the action goes wrong, and the cause stays until the fix. I would investigate longer first when the harm is small and the quick action is risky. An example is a failover that may lose recent writes.
• Narrowing with evidence solves wild guessing. It costs a few minutes of reading dashboards. I would undo the last change at once when the harm began minutes after a release and the dashboards show no clear place.
• A blameless postmortem solves repeat incidents. It costs an hour or more of several people's time. I would write a short one for every incident that hurt users.

"I must find the root cause before I do anything." Users are hurt every second: here 18 requests failed each second. Remove server 3 first. Find the full cause after that.

"The last release always causes the problem." Not always. The release of 19:40 was 25 minutes before the errors, and three servers with the same release worked fine. Undoing an innocent release wastes time and leaves the real problem.

"The error rate is back to 0%, so the incident is over." After we removed server 3, its disk was still full. The incident is over when the cause is fixed and a check exists to catch it sooner.

1. Name the four steps of the method in order. Why do you mitigate before you fix?
2. A service has 6 servers that share 300 requests per second equally. One server fails 30% of its requests. (a) What is the overall error rate? (b) You remove that server 14 minutes after the start. How many requests fail? (c) One server handles 100 requests per second at 100% CPU. What is the CPU of each server after you remove the bad one?
3. At 21:00 a release goes out. At 21:03 all servers show a 20% error rate at 200 requests per second. A teammate reads logs for 25 minutes to find the exact bad line of code. How many requests fail in that time? What should the team do instead?