Skip to content

Trade-offs · Senior and Staff

RPO and RTO vs the cost of each extra nine

How to choose recovery targets and an availability goal by weighing what each extra nine and each minute of lost data costs against what it saves.

Asked at: SeniorStaff

The decision

Two numbers define how a service survives a disaster. The RPO (recovery point objective) is the most data you accept to lose, counted in time. An RPO of 5 minutes means you may lose the last 5 minutes of writes. The RTO (recovery time objective) is the longest outage you accept before service returns. A third number is availability, written in nines: 99.9 percent has three nines. Each nine cuts allowed downtime by 10 times, and the cost of each nine usually grows much faster. The decision is how many nines, and how small an RPO and RTO, each part of your system should have.

Choose a lower target when

  • A short outage or a small data loss is cheap for the business, for example a report page or a recommendation box.
  • Your revenue per hour is small. The value of the next nine is a tiny share of revenue (see the numbers).

Choose a higher target when

  • Lost data cannot be rebuilt, for example payments, orders or messages that the user already saw confirmed.
  • A contract or a law sets the number, with penalties.
  • An outage hurts trust far beyond the lost sales, for example a payment or a login service that other teams depend on.
  • The service is a dependency of many others. Its target must be higher than theirs.

The numbers that change the answer

Downtime for each nine (calculation). A 30-day month has 30 x 24 x 60 = 43,200 minutes. Allowed downtime is 43,200 x (1 - availability).

AvailabilityDowntime a monthDowntime a year
99 percent432 minutes87.6 hours
99.9 percent43.2 minutes8.8 hours
99.99 percent4.3 minutes52.6 minutes
99.999 percent26 seconds5.3 minutes

At 99.99 percent a human cannot even join the call before the budget is gone, so the fourth nine needs automatic recovery.

What a nine is worth (calculation). The Google SRE book makes the same point with a service of 1 million dollars of yearly revenue. Moving from 99.9 to 99.99 percent gains about 900 dollars (cited fact; see Sources). Now the same arithmetic with an Assumption: yearly revenue of Rs 50 crore (500,000,000 rupees).

  • 99.9 to 99.99: 500,000,000 x 0.0009 = Rs 4.5 lakh a year.
  • 99.99 to 99.999: 500,000,000 x 0.00009 = Rs 45,000 a year.

Each nine is worth 10 times less than the one before. The same book says an improvement may cost 100 times more than the one before (a cited fact; see Sources). If a nine costs more than it adds in sales, buy it only for trust, a contract or a dependency reason.

Your goal is limited by your chain (calculation). Take three services in a row, each 99.9 percent. Then 0.999 x 0.999 x 0.999 = 0.997, which is 99.7 percent. That is about 129 minutes of downtime a month. To offer 99.99 percent, the dependencies need more nines than the promise, or a fallback.

Redundancy and the failover (calculation). Two independent copies, each 99.9 percent: failure of both is 0.001 x 0.001 = 0.000001, so 99.9999 percent. That is the best case, with a perfect failover. Now Assumption: the failover works 90 percent of the time. The system is down when the main copy fails (0.001) and either the second is down (0.001) or the failover fails (0.1). That is about 0.001 x 0.101 = 0.000101, which is 99.99 percent, or about 4.4 minutes a month. If the failover works 99 percent of the time, it is 0.001 x 0.011 = 0.000011, which is 99.9989 percent. The failover, not the copy, decides the nine.

RTO and the restore (calculation). Assumption: 2 TB of data and a restore speed of 200 MB per second. 2,000,000 MB / 200 = 10,000 seconds, which is 2.8 hours, before you start the application. At this size and speed, a backup alone cannot give an RTO under about 3 hours.

RPO and money (assumptions). An orders service takes 500 orders a second, each worth Rs 800. With copies 1 second behind, a sudden failure can lose 500 orders, which is 500 x 800 = Rs 400,000 (Rs 4 lakh). With a daily backup, the loss can be a full day. A synchronous copy gives an RPO of 0, but each write waits. Assumption: 20 ms to the other region, so each commit takes 20 ms longer.

What breaks with each choice

A low target.

  • A bad day costs more than planned, and nobody has a plan. Fallback: write what you will do, and cut scope, not safety.

A high target.

  • You pay for a standby that was never tested. Fallback: a drill every quarter with the real time measured.
  • The failover itself causes the outage, as in the 90 percent case above. Fallback: automate the steps, test them, and let a person approve.
  • Cost crowds out other work. Fallback: set targets for each tier.
  • Bad data is copied everywhere, so redundancy does not help. Fallback: point-in-time backups, which keep a non-zero RPO.

What it costs

All numbers are Assumptions, as multiples of a base bill.

TargetTypical setupBill
RPO 24 hours, RTO 4 hoursDaily backups, rebuilt on demand1.05 times
RPO 1 minute, RTO 30 minutesLive copy and a warm standby1.3 times
RPO about 0, RTO under 1 minuteTwo live regions2 times or more

People cost grows too (Assumption). A drill needs about 2 engineer-days each quarter for the first tier, and an on-call rotation of at least 4 people gives 24-hour cover. Changing your mind is cheap in one direction. Raising a target takes weeks to months. Lowering one saves money only after you remove the standby.

How each level answers

SDE-1: RPO is how much data I can lose. RTO is how long I can be down. More nines means less downtime.

SDE-2: I set the targets for each service. A payment needs a small RPO. A recommendation box can be down, so it gets a fallback and a low target.

Senior: At 99.99 percent I have 4.3 minutes a month, so recovery must be automatic. A restore of 2 TB takes 2.8 hours, so backup alone cannot meet a 30-minute RTO, but a warm standby can. I assume the failover works only 90 percent of the time. I test it until it works nearly always, because that number sets the nine.

Staff: I tie the number to money and trust. For Rs 50 crore of yearly revenue, the fourth nine is worth Rs 4.5 lakh in sales. So I buy it only for a contract or a dependency reason. I agree tiers with product, name one owner for each tier, and publish the target and the drill result. I raise a tier only when the loss from one incident, or a contract, is larger than the yearly extra bill. I start every tier on backups and move up one step at a time. I know the choice was right when the drill meets its numbers and the error budget is spent at a steady rate. I review the tiers each year.

Say it in the interview

I would set targets for each tier, not one for everything. For payments I would take an RPO under a minute and an RTO of about 30 minutes with a warm standby. For reports I would accept a daily backup. The risk is a failover that fails. I would drill it every quarter, because with a 90 percent failover even two copies give only about 99.99 percent.

Common mistakes

  • Chasing nines without a price. The fourth nine is worth Rs 4.5 lakh for Rs 50 crore of revenue. Compare it with the bill first.
  • Quoting the nines of one box. The user sees the chain. Three services of 99.9 percent give 99.7 percent.
  • Calling a backup a plan. A backup nobody has restored may not work, and a 2 TB restore takes hours.
  • Setting RPO and RTO as one number. They differ. A live copy gives a small RPO, but you still need time to switch.

Sources

Understand the trade-off. Practise explaining your choice.

In an AI mock interview, the interviewer can push on a choice like this one: it changes a number and asks if your answer still holds.