Failover: what happens to writes in flight

Every server can die. If your design has a leader and followers, the interviewer asks: "The leader dies at peak time. What happens?" This question tests whether you know what a copy of the data really protects.

A weak answer says "a follower takes over" and stops. A strong answer says how the system notices the failure, which follower takes over, how long writes stop, and which writes can be lost. It also says what happens when the old leader comes back.

At SDE-1 level you should explain that a follower becomes the new leader. At SDE-2 level you should work out how many writes are lost and how long writes stop. At SDE-3 level you should also explain why two leaders at the same time is dangerous, and how to prevent it.

Think of the school from lesson 39. The principal keeps the master register, and a clerk carries each new line to the classroom copies. One morning the principal falls ill. Someone must choose a new principal. A good choice is the teacher whose copy has the most lines, because that copy is the closest to the master. Everyone must then be told who the new principal is. Last, think about the old principal: she wrote some lines that the clerk never carried. When she comes back, she must not start giving orders again as if nothing happened.

In database words, from lesson 39: the leader takes all writes, and a follower copies the leader's log of changes. Failover means that a follower takes the leader's job when the leader fails. Making the follower the new leader is called promotion. After promotion the follower accepts writes.

The steps of a failover

1. Detect. Something must notice that the leader is not answering. A tool called the checker sends a health check (lesson 28), a small request on a timer. If a few checks in a row fail, it marks the leader as dead. The time to notice is the check interval x the number of failed checks.
2. Choose. Pick the follower that is least behind. Each follower knows its position in the log (lesson 40). The follower with the highest position has the most changes.
3. Promote. The chosen follower stops copying and starts to accept writes.
4. Redirect. The other followers must now copy the new leader. The apps must also send their writes to the new leader. Apps usually reach the leader through one fixed address, and the failover moves that address. An app that remembered the old address keeps trying the old leader until that memory expires.
5. Deal with the old leader. When it returns, it must come back as a follower, not as a leader.

Postgres can promote with one command, but it does not notice failures itself. You need another tool for the checks.

Writes in flight

A write in flight is a write that has been sent, but is not yet safe on the follower that will take over. There are two kinds.

• Accepted, not yet copied. The leader saved the change and told the app "saved". With asynchronous copying (lesson 39) it had not yet sent the change to the follower. When the leader dies, the follower never gets it. The new leader does not have this write, so it is lost, even though the app was told "saved". The number of lost writes is about writes per second x the lag in seconds. Postgres says the same: the loss grows with the replication delay at the time of failover.
• Sent, no answer. The app sent a write, and the leader died before it answered. The write may or may not have been saved. The app cannot know. If the change did reach the follower, and the user presses the button again, the same order is created twice.

With synchronous copying (lesson 39), the leader tells the app "saved" only after a follower has the change. If that follower becomes the new leader, no "saved" write is lost. The cost is the one you know from lesson 39: slower writes, and writes that wait when that follower is down.

Writes stop for a while

Between the death of the leader and the moment the apps use the new leader, nobody accepts writes. Reads from the followers still work. The time without writes is:

• time to notice + time to promote + time for the apps to find the new leader

When the old leader comes back

The old leader may hold changes that never left it. If it starts again as a leader, there are two leaders. This is split brain: two servers both think they are the leader and both accept writes. Their data then differs, and no rule says which one is right.

Split brain does not need a dead leader. Suppose only the network between the health checker and the leader breaks. The leader is alive, but the checker thinks it is dead and promotes a follower. Some apps still reach the old leader, and others reach the new one.

Three ways to prevent it:

1. Stop the old leader before you promote. Shut it down, or cut its access to the disk and the network.
2. Let a leader step down by itself. A leader that cannot reach the checker or any follower for some seconds stops accepting writes.
3. Do not trust one checker. Use several, and promote only when more than half of them agree that the leader is dead.

When an old leader returns, rebuild it as a follower of the new leader. The changes it never sent are thrown away, or saved for a person to check. Postgres has a tool for this, called pg_rewind.

Fast or careful?

An automatic failover is fast. A person who must wake up, log in and check is slow, but less likely to promote by mistake. The notice time is also a choice. A short time finds a real failure sooner. But it also treats a slow leader as a dead one, for example a leader stuck for 8 seconds.

SwadKart, the food-delivery app of lessons 39 and 40, has one leader and two followers. The leader dies.

Assumptions (made up to teach with, not real traffic):

• 300 order writes per second.
• Follower 1 is 0.4 seconds behind (the calm lag of lesson 40). Follower 2 is 0.9 seconds behind.
• Health checks run every 5 seconds. Three failed checks in a row mean "dead", so noticing takes 3 x 5 = 15 seconds.
• Promotion takes 5 seconds. The apps need 10 seconds to find the new leader.
• An order costs Rs 400 on average. A write needs 20 ms from sending to the answer.

Step 1: how long writes stop.

• 15 + 5 + 10 = 30 seconds.
• Writes that fail in this time: 300 x 30 = 9,000.
• Three nines allow 43.2 minutes a month, which is 2,592 seconds (lesson 4). One failover of 30 seconds uses 30 / 2,592 = 1.2% of that.

Step 2: choose the follower. Follower 1 lacks the last 0.4 seconds of changes. Follower 2 lacks the last 0.9 seconds. We promote follower 1. Follower 2 is behind it, so it can copy the new leader from where it stopped.

Step 3: lost writes.

• Promote follower 1: 300 x 0.4 = 120 writes are lost.
• If we had promoted follower 2: 300 x 0.9 = 270 writes lost.
• The 120 orders are worth 120 x Rs 400 = Rs 48,000. The customers were told "saved" and the bank may have charged them, but the new leader has no order.

This answers the predict box: about 120.

Step 4: writes with no answer. At the moment of the crash, 300 writes per second x 0.020 seconds = 6 writes were waiting for an answer. For each of them, the app does not know if the order exists. A customer who presses Pay again may now have two orders. The safe step is to look for the order before you create it again. Also compare with the payment gateway, which is the company that takes the card payment and keeps its own list of charges.

Step 5: synchronous copying. If the leader waits for follower 1 before it says "saved" (lesson 39), then follower 1 has every "saved" write. After promotion of follower 1, 0 writes are lost. The cost: each write takes 2.5 ms and not 1 ms, and if follower 1 is down, writes wait.

Step 6: the old leader returns. Its disk holds the 120 changes that never left. We start it as a follower of follower 1. Those 120 changes are removed or saved aside. If we started it as a leader instead, there would be two leaders.

Step 7: split brain. Suppose the network between the health checker and the leader breaks for 60 seconds. The leader is alive. The checker promotes follower 1. Assumption: one third of the app servers can still reach the old leader.

• Writes to the old leader: 300 x 1/3 = 100 per second. In 60 seconds: 100 x 60 = 6,000 orders.
• The new leader never sees these 6,000 orders. Both leaders continue the same sequence of order numbers (lesson 23), so two different orders can get the same number.

Step 8: a short check time or a long one. Assumption: the leader freezes for 8 seconds, for example on a slow disk. This happens 6 times in a year. A real crash happens once in a year. A needless failover (a failover of a leader that is alive) loses the same writes as a real one.

The long time costs 10 more seconds per real crash: 300 x 10 = 3,000 failed writes in the year. The short time costs 720 lost writes and 6 x 20 = 120 seconds without writes, which is 300 x 120 = 36,000 failed writes. In this example the long time is better.

Step 9: by hand. Assumption: a person needs 15 minutes = 900 seconds. Writes that fail: 300 x 900 = 270,000, against 9,000 for automatic failover.

• Failover solves a dead leader: writes start again in 30 seconds instead of the 15 minutes (900 seconds) a person needs.
• Failover costs lost writes with asynchronous copying (120 in the example, Rs 48,000). It also costs a time without writes (30 seconds) and a risk of two leaders.
• Synchronous copying solves lost writes. It costs slower writes and writes that wait when the follower is down (lesson 39).
• A short check time solves a long wait after a real crash. It costs needless failovers.
• I would change it when the data is money or orders: I would wait for one follower and promote that one. I would also change it when a wrong promotion is worse than a short wait: then a person confirms it.

"A follower always has every write, so failover loses nothing." Not with asynchronous copying. The follower is behind. In the example 300 x 0.4 = 120 writes were only on the leader.

"Failover is instant." It takes the time to notice, to promote and for the apps to find the new leader: 30 seconds in the example. Writes fail in that time (9,000 of them).

"When the old leader is repaired, it should take its job back." It holds changes the new leader does not have. If both lead, you have split brain. In the example the old leader took 6,000 orders that the new leader never saw. Bring it back as a follower.

1. Write the steps of a failover in order. What is split brain?
2. A leader takes 500 writes per second. Follower A is 0.2 seconds behind and follower B is 0.6 seconds behind. Health checks run every 4 seconds, and three failed checks mean "dead". Promotion takes 6 seconds, and the apps need 9 seconds to find the new leader. (a) Which follower do you promote, and how many writes are lost? (b) How long do writes stop? (c) How many writes fail in that time?
3. After a failover, support reports customers who were charged with no order, and some who have two orders. Give one reason for each, and say what you check first.