Why do I see old data after saving? (replica lag)

A user changes the name on their profile and presses Save. The page says "saved". The user opens the profile, and it shows the old name. Interviewers ask about this because many designs read from copies of the database, and the copies are a little behind.

A weak answer says "the data is the same everywhere" or does not notice the problem. A strong answer says how far behind the copies can be, and which reads must still show the new data. For example: "After a user saves, that user's reads go to the leader for 30 seconds. Other users read from the followers."

At SDE-1 level you should explain why the old data appears. At SDE-2 level you should also give a fix and its cost. At SDE-3 level you should also work out how long the delay can last when the load changes.

Go back to the school in lesson 39. The principal writes a new line in the master register. A clerk carries it to the classroom copies. You write your new phone number in the principal's office, walk straight to your classroom, and look at the copy. The clerk has not arrived yet. Your old number is still there. Wait a little and the new number appears.

Remember lesson 39. The leader takes the writes. The followers copy the leader's log of changes and answer reads. Replication lag is the time between a change on the leader and the same change being visible on a follower. Some people say "replica lag". A change takes these steps:

1. The leader saves the change.
2. The leader sends it over the network.
3. The follower receives it and saves it.
4. The follower applies it, which means it makes the change in its own copy of the data.

Only after step 4 can a read on the follower see the change. So a read that comes less than the lag after the write shows old data. A read that comes later shows the new data.

What makes the lag longer

The lag is never zero. In Postgres it is typically under one second when the follower keeps up. It grows when the follower cannot keep up:

• A burst of writes. A follower applies changes in order. If changes arrive faster than it can apply them, a line of waiting changes builds up. This waiting line is the backlog.
• A busy follower. A follower that is very busy with reads has less CPU left to apply changes.
• A slow network between the leader and the follower.

When the backlog is more than zero, you can find the lag:

• backlog grows by (changes arriving per second - changes applied per second) each second
• lag in seconds = backlog / changes applied per second

More followers do not make the log apply faster, because every follower applies the whole log. A faster follower does. More followers help only when a busy follower is short of CPU: they take reads away, so it has more CPU left for the log.

Read-your-writes

Read-your-writes is a promise to one user: after I save something, my own next read shows it. It says nothing about other users. They may see the old value for a short time, and often that is fine. You need it only for data a user has just changed, such as their own profile or order.

Ways to give read-your-writes

1. A time window on the leader. For N seconds after a user writes, that user's reads go to the leader. The leader always has the newest data. Choose N longer than the lag you expect and longer than the time the user takes to read again. The cost is extra work for the leader.
2. Always read your own data from the leader. The user's own profile and orders come from the leader. Other people's data comes from followers. This is simple. The cost is leader work for all of those reads.
3. Remember the position. When the leader saves a change, it gives back a position in the log. Postgres calls it the log sequence number. The app keeps it. For the next read, the app uses a follower only if that follower has applied the log up to that position. Otherwise it waits a little or asks the leader. This needs more code, but it needs no fixed window.
4. Make the write wait. Use synchronous copying with the setting that waits until the follower has applied the change (remote_apply, lesson 39). The cost is slower writes. Every follower that answers reads must be one of the followers the leader waits for.
5. Show the new value yourself. After Save, the page shows the value from the save response and does not read again. The next page load can still show old data.

Going back in time

There is a second problem. A user reads from one follower and sees a new comment. The user reloads, the app picks another follower that is further behind, and the comment is gone. The user sees time go backward. The fix is to send each user to the same follower every time, using a fixed rule on the user id. If that follower fails, move the user to another one.

The first meaning of "eventual"

Suppose the writes stop. The followers apply the rest of the log, and soon every follower holds exactly the same data as the leader. This is called eventual consistency: if no new writes arrive, all copies end up the same after some time. "Eventual" means "after some time, not at once". It does not say how long. It also says nothing about what a read shows before that time. Followers of one leader are eventually consistent.

This word "consistent" is not the C in ACID from lesson 21. Here it only means that the copies hold the same data.

We continue with SwadKart from lesson 39: two followers, and the leader as the only server that takes writes.

Assumptions (made up to teach with, not real traffic):

• 300 saves per second. Reads of the saved data go to followers.
• After each save, the user reads the saved data once, at a random moment in the next 20 seconds. Every moment is equally likely.
• When it is calm, the lag is 0.4 seconds. A follower can apply at most 2,000 changes per second, in order.
• To keep the sums simple, use the lag at the moment of the save for the whole 20 seconds.
• A read costs 4 ms of CPU. One server has 8 cores, which is 8,000 ms per second (lesson 15).
• Once a month a job writes 3,000 extra rows per second for 60 seconds. The leader writes them in big batches, so it can keep up. Each follower still applies every row.

Step 1: the predict box. A read shows the change only if it comes after the lag. With a lag of 2 seconds, the reload after 1 second shows the old number, and the reload after 3 seconds shows the new number.

Step 2: calm times. The read is at a random moment in 20 seconds. It shows old data when it comes within the first 0.4 seconds.

• Chance = 0.4 / 20 = 0.02, which is 2%
• Users who see old data = 300 x 0.02 = 6 users every second

Two in a hundred looks small. At this scale it is 6 users every second, even when nothing is wrong.

Step 3: the monthly job. Changes arrive at 300 + 3,000 = 3,300 per second. A follower applies 2,000 per second.

• The backlog grows by 3,300 - 2,000 = 1,300 changes per second.
• After 60 seconds the backlog is 1,300 x 60 = 78,000 changes.
• The lag is then 78,000 / 2,000 = 39 seconds.
• When the job ends, only 300 changes arrive per second. The follower removes the backlog at 2,000 - 300 = 1,700 per second. That takes 78,000 / 1,700 = 46 seconds (rounded from 45.9).

The lag is above its calm value for about 60 + 46 = 106 seconds. Both followers have this lag, because each applies the whole log. At the end of the job the lag is 39 seconds. For about 52 seconds in all, the lag is more than the 20 seconds in which users read again. That time runs from 31 seconds into the job to 22 seconds after it ends. In that time every user who saves (300 users per second) reads old data, 100% of them. After the backlog is gone, the lag returns to 0.4 seconds and all copies are the same again. That is eventual consistency.

Step 4: a fix, the 30-second window. For 30 seconds after a save, that user's read goes to the leader. Users read within 20 seconds, so all of those reads are inside the window and show the new data. Old data seen: 0%.

The cost: 300 reads per second move to the leader.

• 300 x 4 ms = 1,200 ms per second, which is 1,200 / 8,000 = 15% of the leader.
• In lesson 39 the leader used 2,550 ms for writes. Now it uses 2,550 + 1,200 = 3,750 ms, which is 47% (outside the job). This is below our 70% limit, so the leader has room.

A window must cover the time in which the user may read again. If the lag is 39 seconds and a user reads again after 35 seconds, a 30-second window is not enough.

• Reading from followers solves a leader that is too busy with reads. It costs old data for some reads: 2% of the saves in calm times, 100% during the job.
• The leader window solves users not seeing their own changes. It costs leader work (15% in the example) and a number to choose. I would change it when the lag is longer than the window, or when the leader has no room. Then I would use the log position, or make the write wait.
• Taking a follower out of use when its lag is too long solves reading old data. It costs work for the other servers. In the example both followers lag. The leader would get all 1,500 reads and the writes: 8,400 ms of 8,000 ms, which is 105% (lesson 39).
• Watch the lag. Postgres shows write_lag, flush_lag and replay_lag for each follower. Set an alarm on it.

"Replication lag only happens when something is broken." It is normal. In the example, a calm lag of 0.4 seconds still showed old data to 6 users every second.

"More followers make the lag shorter." Not by themselves. Each follower applies the whole log. In the example both followers were 39 seconds behind. A faster follower, or fewer writes, makes the lag shorter.

"Eventually consistent means data can be lost or can stay different." No. It means that when the writes stop, all copies become the same. Nothing is lost. The only problem is the unknown time, and what a read shows until then.

1. What is replication lag? What does read-your-writes promise? What does "eventual" mean in eventual consistency?
2. A follower applies at most 1,000 changes per second. Normally 200 changes arrive per second. A job adds 1,500 per second for 40 seconds. (a) How big is the backlog when the job ends? (b) What is the lag then? (c) How long after the job ends is the backlog gone?
3. A team gives each user a leader window of 5 seconds after they write. During a job the lag reaches 12 seconds, and users read again after about 8 seconds. What happens, and what could the team do?