SLOs, error budgets and alerts

The interviewer asks: "How do you know checkout is reliable enough? When do you wake an engineer?" A weak answer says "100% uptime". A strong answer sets a target on what users feel, counts the failure allowed, and wakes someone only when that allowance burns too fast.

Think of a 30 GB monthly data pack. Use 10 GB on day one and you will run out in 3 days, so you act.

• An SLI (service level indicator) is a measured number that shows what users feel. An example is the share of checkout requests that succeed in 1,000 ms or less (lesson 91).
• An SLO (service level objective) is a target for the SLI over a window: 99.9% good requests over 30 days.
• The error budget is what the SLO allows to fail: 100% minus the SLO. For 99.9% it is 0.1% of requests, or 0.1% of 43,200 minutes = 43.2 minutes (lesson 4). While budget is left, ship changes. When it is gone, fix reliability first. A 100% target leaves no budget.
• The burn rate is how many times faster than the steady pace you use the budget: error rate / (1 - SLO). A burn rate of 1 uses the whole budget in exactly 30 days.

An alert is a message to a person when a number crosses a line. A page wakes the on-call engineer, the person responsible this week. A ticket can wait.

A fixed line such as "page above 10% errors" misses a small problem that lasts long. So alert on burn rate over a long window, and over a short window one twelfth as long. Both must be over the line, so the alert stops soon after a fix. Budget used = burn rate x window / 30 days. Time to alert = window x line / burn rate, if the burn starts at once. For 99.9%, the Google SRE Workbook suggests: page at burn 14.4 over 1 hour (2% of budget) or 6 over 6 hours (5%), and a ticket at 1 over 3 days (10%).

A runbook is a short page linked from the alert: what it means, where to look, the first safe step, whom to call.

An SLA (service level agreement) is a contract with a customer. It says what happens if the promise is missed, usually a service credit: part of the bill returned. A target with no consequence is an SLO. Keep the SLO stricter than the SLA. Your providers limit your SLA: if every checkout needs a 99.9% provider, you cannot promise 99.99%. A contract also says how uptime is measured.

SwadKart checkout. Assumptions (made up): SLO 99.9% over 30 days, 200 requests a second.

Step 1: budget. Requests = 200 x 86,400 x 30 = 518,400,000. Budget = 0.1% = 518,400 bad requests = 518,400 / 200 = 2,592 seconds = 43.2 minutes of full outage.

Step 2: a bad deploy. 5% fail for 30 minutes. Bad requests = 200 x 1,800 x 0.05 = 18,000, which is 3.5% of the budget. Burn rate = 5% / 0.1% = 50. The budget is gone in 720 hours / 50 = 14.4 hours.

Step 3: alerts. "Page above 10% errors" never fires at 5%. The burn rule fires after 1 hour x 14.4 / 50 = 0.29 hours, about 17 minutes. By then 200 x 1,037 x 0.05 = 10,368 requests have failed: 2% of the budget. The runbook says: open the dashboard (lesson 91); if a deploy was in the last hour, go back to the old version.

Step 4: a slow leak. 0.3% errors is burn rate 3: the budget lasts 30 / 3 = 10 days. A 10% line never fires, and 3 is below 6. The ticket rule fires after 3 days x 1 / 3 = 1 day. Then 3 x 24 / 720 = 10% of the budget is gone.

An SLO solves arguments about "reliable enough". It costs effort to measure. Burn alerts solve missed leaks. They cost rules to tune. I would change the target when users complain or cannot tell the difference.

"The target should be 100%." It leaves no budget.

"An SLO and an SLA are the same." An SLA has a consequence.

"Alert on a fixed error line." A 10% line missed both the deploy and the leak.

1. What are an SLI, an SLO, an SLA and an error budget?
2. SLO 99.95% over 30 days, 100 requests a second. (a) How many bad requests are allowed? (b) How many minutes of full outage? (c) Errors are 1%: burn rate, and hours the budget lasts?
3. SLO 99.9%, errors 0.2% all month. Does a page rule fire? A ticket rule?