How good is good enough: uptime and speed

In the requirements step you write numbers for downtime and for speed (lesson 3). The interviewer then checks if you understand what the numbers mean. A common answer is "the app works 99.9% of the time" or "the page opens in 1 second on average". These sound good. But can you say how many hours 99.9% allows? And what does the slowest user see, when the average is 1 second?

A weak answer repeats a number without meaning. A strong answer turns it into time: "99.9% is about 43 minutes of downtime in a month." A strong answer also looks at the users who wait longest, and not only at the average.

At SDE-1 level you should explain the idea in words. At SDE-2 level you should do the arithmetic and choose the target. At SDE-3 level you should also say what a target costs, and why a screen with many requests feels slower than one request.

Think of a tea stall. Suppose the stall is open 99 days out of every 100. That sounds almost perfect. But 1 day in 100 means it is closed for 3 to 4 days every year. If one of those days is a festival day, the owner loses the best sales of the year. A single percent hides how long the closed time really is. Now think about waiting time. The owner says: "A customer waits 2 minutes on average." But one customer in 20 waits 15 minutes, and that customer never comes back. The average does not show this.

So there are two questions about a system. Is it working? And when it works, how fast is it?

Uptime and the nines

Uptime is the share of time when the system works for users. Interviewers also call it availability. It means the same thing. Downtime is the rest of the time, when users cannot use the system (lesson 3). Uptime is written as a percentage.

You find the downtime with one formula:

• downtime = (1 - uptime) x total time

Engineers count the nines in the percentage. 90% is one nine, 99% is two nines, 99.9% is three nines, and so on. Use these totals: one year = 365 x 24 = 8,760 hours. One month of 30 days = 30 x 24 x 60 = 43,200 minutes.

Each extra nine cuts the downtime by 10 times. It also costs more, because you need more servers, more care and more people. So the question "how many nines?" is really "how much is an hour of failure worth to this business?"

One more warning. The percentage counts all minutes the same. But users do not. Ten minutes of downtime at 4 am hurts few users. Ten minutes at the busiest time hurts many.

Latency and percentiles

Latency is the time from the moment the client sends a request to the moment it gets the full answer. We measure it in seconds or in milliseconds (ms). 1 second = 1,000 ms. Every request has its own latency, and they are not the same. So one number cannot describe all of them.

The average adds all the latencies and divides by the number of requests. The average is easy, but it hides the slow requests. A few slow requests are only a small part of the total, so the average hardly moves. The users who get those slow answers still wait a long time.

A percentile fixes this. Sort the latencies from the fastest to the slowest. Take 100 requests as an example:

• p50 is the 50th in the list. Half of the requests are as fast as this or faster. It shows the typical request.
• p95 is the 95th in the list. 95 requests are as fast as this or faster, and 5 are slower.
• p99 is the 99th in the list. 99 requests are as fast as this or faster, and only 1 is slower.

The letter p stands for percentile. A target such as "p99 under 2 seconds" means that 99 of every 100 requests finish in 2 seconds or less. Do not confuse p99 with "the slowest request". The slowest request, called the maximum, is worse than p99.

Why does the slow end matter so much? A screen in an app often needs several requests, and the user waits for the slowest one. So a rare slow request is not rare at the level of a screen. The worked example shows this with numbers.

We go back to the food-ordering app from lesson 3. Its requirements said: at most 60 minutes of downtime per month, and the menu opens in under 2 seconds.

Assumptions (made up to teach with, not real traffic):

• A month has 30 days, so 43,200 minutes. A year has 8,760 hours.
• 5 lakh (500,000) orders arrive in the busiest 2 hours (as in lesson 3), spread evenly. 1 lakh = 100,000.
• Opening the menu screen makes 10 requests. The slow requests are random and do not depend on each other.

Part 1: which nines do we need?

• 60 minutes out of 43,200 = 60 / 43,200 = 0.00139, which is 0.139% of the month.
• So we need uptime of at least 100% - 0.139% = 99.861%.
• Two nines: 0.01 x 43,200 = 432 minutes of downtime. That is 7.2 hours, more than 60. Not enough.
• Three nines: 0.001 x 43,200 = 43.2 minutes. That is less than 60. Enough.
• Four nines: 0.0001 x 43,200 = 4.32 minutes. More than we need, and it costs more.

We choose three nines. In a year, that is 0.001 x 8,760 = 8.76 hours.

Now the warning about busy times. In the busiest 2 hours (120 minutes) there are 500,000 / 120 = 4,167 orders per minute (rounded from 4,166.7). If the 43.2 minutes of downtime fall in that time, the app misses 43.2 x 4,167 = about 180,000 orders. That is 180,000 / 500,000 = 36% of the evening's orders. The same 43.2 minutes at 4 am would cost almost nothing. So in the interview, also ask when the downtime may happen.

Part 2: the average and the percentiles. Take the 100 requests from the predict box: 98 take 100 ms and 2 take 5,000 ms.

• Average = (98 x 100 + 2 x 5,000) / 100 = (9,800 + 10,000) / 100 = 198 ms.
• Sorted list: the first 98 are 100 ms, the last 2 are 5,000 ms.
• p50 = the 50th = 100 ms.
• p95 = the 95th = 100 ms.
• p99 = the 99th = 5,000 ms.

An average of 198 ms looks very good against a target of 2 seconds. The p50 and the p95 look fine too. Only the p99 shows 5,000 ms, which is more than 2 seconds.

That is the answer to the predict box. The average hides the slow requests. The p95 hides them too in this case, because it allows 5 requests in 100 to be slower, and only 2 are slow. The p99 allows only 1 in 100, so it shows them. Which percentile to watch depends on how many slow requests you can accept.

Part 3: what a user sees. Each request has a 2 / 100 = 0.02 chance of being slow. A menu screen makes 10 requests. A request is fast with chance 0.98. For all 10 to be fast, we multiply 0.98 by itself 10 times. That is 0.98 to the power of 10 = 0.817. So the chance that at least one is slow is 1 - 0.817 = 0.183, which is 18.3%.

About 1 in 5 menu screens includes a 5-second request, although only 1 request in 50 is slow. The average showed none of this.

• Setting a number solves: the unclear words "highly available" and "fast". A number can be checked, and everybody means the same thing.
• A higher uptime costs money. Each extra nine means 10 times less downtime, and more servers and people. In lesson 3, going from 3 to 4 servers cost 33% more, just to survive one broken server. I would pick the nines from what an hour of failure costs the business, and I would change the choice when that cost changes.
• A target on p99 costs effort. It is harder to reach than a target on the average, because you must find and fix the rare slow requests. It also needs many requests to measure. With only 100 requests, the p99 depends on one or two of them.
• The average is cheap but misleading. I would still look at it, but I would set the target on a percentile.
• Choose the percentile from how many slow requests users can accept. p95 allows 5 in 100 to be slow. p99 allows 1 in 100.

"99.9% sounds almost perfect, so 99% is about the same." It is 10 times worse. 99% allows 87.6 hours of downtime a year, and 99.9% allows 8.76 hours. In a month that is 7.2 hours against 43.2 minutes. Always change the percentage into hours or minutes.

"If the average latency is low, users are happy." The average hides the slow requests. In the example the average was 198 ms, but 2 of every 100 requests took 5 seconds. A screen with 10 requests had a slow one 18.3% of the time.

"p99 is the worst case." It is not. p99 is the time that 99 of every 100 requests meet. One request in 100 is slower than p99, and the slowest request, the maximum, can be much slower. Say "p99" when you mean "the slow end that 1 in 100 users feel", and do not call it the worst case.

1. What is the p95 of a list of 100 request times? Why does the average hide slow requests?
2. A service promises 99.95% uptime. How many hours of downtime is that in a year? How many minutes in a 30-day month?
3. A teammate says: "Our average latency is 150 ms and our p99 is 4 seconds. The target is 2 seconds, so we are fine." A screen makes 10 requests, and 1% of requests take 4 seconds or more. What do you answer?