Multi-region: latency, failover, data residency

Most designs start in one city. Then the interviewer asks: "Many users are in Europe, and they say the app is slow. What happens if the whole city loses power? What if a country says its users' data must stay inside the country?"

A weak answer says "use two regions" and stops. A strong answer says what runs in the second region, how long users wait when a region fails, and how much data can be lost.

At SDE-1 level you should explain why far users are slow and why one region is one failure. At SDE-2 level you should work out the failover time and the lost writes. At SDE-3 level you should also compare one writer with two.

Think about a bank with its head branch in Mumbai and a second branch in Frankfurt. A customer in Europe can walk into the Frankfurt branch. For anything new, the clerk must phone Mumbai, which takes time. A courier brings a copy of the Mumbai ledger, so Frankfurt is a little behind. If Mumbai floods, Frankfurt can open for everyone, but the newest entries are missing.

A region is an area of the world where a cloud provider has its data centres (lesson 13).

Why more than one region

1. Far users are slow. The round trip (lesson 7) between Europe and Mumbai is long. The first request on a new connection needs 3 of them (lesson 36). A CDN helps with files, not with an order.
2. One region is one failure. A power or network fault can take the whole region, with the writes that nobody copied elsewhere.
3. The law or a contract. This is data residency (below).

Active-passive

In an active-passive design, one region serves all users. The other region, the standby, waits. It holds a copy of the data: the leader (lesson 39) sends its log to a follower in the standby region. The leader does not wait for it. The standby can be:

• Cold: only the database copy runs. After a failure, the app servers must start. Cheap and slow.
• Warm: a few app servers run, for example a quarter of the normal size. After a failure, more start.
• Hot: app servers of full size run all the time. Fast and costly.

Failover between regions

Lesson 41 gave the steps: detect, choose, promote, redirect. Between regions:

• The health checker must run outside the region that it watches, because a checker inside a dead region dies with it. Use several checkers, and act when most agree (lesson 41).
• Redirect means moving the users. With DNS, the name points to the new region, but resolvers keep the old answer until its TTL ends (lesson 8). A global load balancer is a load balancer (lesson 28) above the regions. It has one address, checks every region, and sends users to a healthy one. It does not wait for DNS caches, so the switch is faster. (Some products with this name use DNS inside and still wait for the TTL.)

Two numbers describe a failover (lesson 42). RTO (recovery time objective) is how long users are without service. RPO (recovery point objective) is how much data you may lose, as time. Here:

• RTO = time to notice + time to promote + time to start or grow the app servers + time to move users
• RPO = the writes that did not yet reach the second region. With a steady lag it is writes per second x lag (lesson 41). With a growing lag it is the backlog

Copies that fall behind across regions

Across regions, the copy is behind by the replication lag (lesson 40), and the road is long. The rules of lesson 40 hold:

• the backlog grows by (arrival rate - link rate) each second
• the lag of a new write = backlog / link rate
• the data at risk when the region dies = the backlog

A far user who wrote and then reads from the nearby copy needs read-your-writes (lesson 40). One way is to send that user's reads to the main region for N seconds, with N longer than the lag.

A write can also wait for the second region (synchronous copying, lesson 47). The RPO is then 0, and each such write pays one round trip between the regions. If the link breaks, those writes cannot finish.

Active-active

In an active-active design, both regions serve users. Geo routing in DNS (lesson 8) or the global load balancer sends each user to the nearest region. The question is who takes writes.

• Local reads, one writer. Each region answers reads from its own copy. All writes go to the leader in the main region. Far users get fast reads, but writes are not faster.
• Two writers. Each region takes writes. If two users change the same row in two regions at once, the regions hold two values and need a rule to choose. "The newest time wins" throws away one change, and clocks differ (lesson 49).

The common way to avoid conflicts is a home region. Each user belongs to one region, for example the nearest. All writes about that user go there (lesson 43 split data by a key in the same way). The other region may keep a read copy. The cost: a feature that mixes users of both regions reads old copies.

Active-active is not free. When a region fails, the other must carry all users, so each region must be sized for everyone.

Data residency

Data residency means that a law or a contract says where data must be stored. For example, the data of users in one country may have to stay in that country. Ask the legal team for the rule. Then:

• If the rule covers the country, the standby must be in that country too. Some providers have two regions in one country (see Sources).
• With a home region, each user's rows stay there.
• Every copy counts: backups (lesson 42), logs, caches and search copies.

An exam-preparation app has its servers and database in Mumbai. 20% of the users are in Europe, so the team adds a region there.

Assumptions (made up to teach with, not real data):

• A round trip: Europe to Mumbai 150 ms, and Europe to the European region 20 ms. The regions are 150 ms apart.
• A request needs 20 ms of server time. The first request on a new connection needs 3 round trips (lesson 36). The connection then stays open.
• One screen makes 6 requests, one after another.
• The app takes 1,000 writes per second of 2 KB each. The link between the regions carries 4 MB per second. The calm lag is 0.4 seconds.
• The Mumbai bill is Rs 10 lakh a month: Rs 6 lakh for app servers and Rs 4 lakh for the database. The link costs Rs 0.5 lakh (1 lakh = 100,000).

Step 1: the screen for a European user.

• From Mumbai: first request 3 x 150 + 20 = 470 ms. Each of the next 5: 150 + 20 = 170 ms. Total = 470 + 5 x 170 = 1,320 ms.
• From Europe with a local read copy: first request 3 x 20 + 20 = 80 ms. Each next one: 20 + 20 = 40 ms. Total = 80 + 5 x 40 = 280 ms. That is 1,320 / 280 = 4.7 times faster.
• A write is not faster. Directly to Mumbai: 150 + 20 = 170 ms. Through the European app to the Mumbai leader: 20 + 20 + 150 = 190 ms.
• A mistake: European app servers that read the Mumbai database. A request with 5 queries in a row needs 20 + 20 + 5 x 150 = 790 ms. That is 4.6 times slower than 170 ms.

Step 2: the failover time. Assumptions: a person needs 300 seconds to notice. A cold standby needs 120 seconds to start its app servers, and a warm one 90 seconds to grow. DNS moves users in up to 60 seconds, and a global load balancer in 5. Checks every 10 seconds with 3 failures give 30 seconds. We add the steps, one after another, for the worst case: notice + promote (15) + start + move.

Costs: cold = database 4 + link 0.5 = Rs 4.5 lakh. Warm adds a quarter of the app servers: 4 + 0.25 x 6 + 0.5 = Rs 6 lakh. Hot adds all: 4 + 6 + 0.5 = Rs 10.5 lakh.

This answers the predict box: 105 seconds at most, and 1,000 x 0.4 = 400 lost writes. Users were told that these were saved. Writes during the 105 seconds fail.

Step 3: a slow link. A fault leaves the link with a fifth of its speed: 4 / 5 = 0.8 MB per second.

• Writes need 1,000 x 2 KB = 2 MB per second, so the link is normally 50% busy. Now it carries 0.8 MB / 2 KB = 400 writes per second.
• The backlog grows by 1,000 - 400 = 600 writes per second. After 60 seconds: 600 x 60 = 36,000 writes (72 MB).
• The lag of a new write: 36,000 / 400 = 90 seconds.
• If Mumbai is lost now, the 36,000 writes are lost: 36,000 / 1,000 = 36 seconds of writes, not 0.4.
• A read-your-writes window of 30 seconds is shorter than the lag of 90. From the 31st second the user reads a copy that does not have the write yet.

Step 4: a write that waits. A payment waits for the second region: 1 + 150 + 1 = 152 ms, against 1 ms. Its RPO is 0.

Step 5: active-active with a home region. European users are homed in Europe, so their screen takes 280 ms. Each region is sized for all users: Rs 10 lakh plus Rs 10.5 lakh is Rs 20.5 lakh a month, 2.05 times the one-region bill. If Mumbai is lost, Indian users wait 30 + 15 + 5 = 50 seconds for their data to be promoted in Europe. European users notice nothing.

Residency. Suppose the law says that the data of Indian users must stay in India. The European region may not hold their rows. Mumbai needs a standby in another Indian region.

• A second region solves slow far users (280 ms, not 1,320) and the loss of a region. It costs Rs 4.5 to 10.5 lakh a month, and a lag that can lose writes. I would add one when many users are abroad, or when hours of downtime are not allowed.
• A hotter standby solves a long outage: 105 seconds, not 495. It costs more: Rs 10.5 lakh, not Rs 4.5 lakh.
• Waiting for the second region solves lost writes. It costs 152 ms for each such write, and writes that stop when the link breaks. I would use it only for money.
• A home region solves conflicts and residency. It costs old reads for features that mix users of two regions, and a full-size second region if it must carry everyone. I would use two writers for the same data only with a clear rule.

"A second region makes everything faster for far users." Reads from a local copy are faster: 280 ms, not 1,320. Writes are not: 190 ms, not 170. App servers that read a far database are slower: 790 ms.

"With a copy in another region, a region failure loses no data." The copy is behind. Calm times lose 400 writes, and a slow link loses 36,000.

"Active-active means no failover and no cost." Each region must carry everyone: 2.05 times the bill. A lost region's data still needs promotion.

1. What is the difference between active-passive and active-active? What are RTO, RPO and data residency?
2. Users in Singapore are 90 ms from Mumbai and 15 ms from a local region. A request needs 20 ms of server time. A screen has 5 requests: the first needs 3 round trips, and each of 4 more needs 1. (a) How long does the screen take from Mumbai, and from the local region with a local read copy? (b) Checks run every 15 seconds, 3 failures mean dead, promotion takes 15 seconds, the standby is hot, and the DNS TTL is 30 seconds. What is the RTO? (c) The app takes 500 writes per second, and the copy is 0.6 seconds behind. How many writes are lost?
3. A team puts app servers in Europe but keeps the only database in Mumbai. Each request makes 5 queries in a row, and Europe to Mumbai is 150 ms. Users say it is slower. What is the request time, and what do you change?