Skip to content

Trade-offs · Production decisions

Synchronous vs asynchronous replication

When to make a database wait for its copies before it says saved, the numbers that change the answer, and what breaks with each choice.

Asked at: SDE-2Senior

The decision

A database leader sends every change to its followers. The question is when it tells the app "saved". With asynchronous replication, the leader says "saved" after its own disk write, and sends the change to followers later. With synchronous replication, the leader waits until a follower confirms that it has the change, and only then says "saved". Asynchronous is faster and keeps working when a follower is down. Synchronous loses no confirmed write when the leader dies, but it is slower and depends on the follower.

Choose synchronous when

  • The data is money, orders or anything you cannot rebuild, and losing the last second of changes is not acceptable.
  • A follower is close, in the same data centre or a nearby zone, so the wait is a few milliseconds.
  • You have at least two followers, so one slow or dead follower does not stop writes.

Choose asynchronous when

  • The follower is far away, for example in another city or region, and each wait would add tens of milliseconds.
  • The data can be rebuilt or the last few seconds are cheap to lose, such as a view counter or a cache of a feed.
  • Followers serve reads, and a little delay is acceptable (with read-your-writes for the user who just saved).

The numbers that change the answer

Assumptions for all numbers:

  • A disk write takes 1 ms.
  • A round trip inside one data centre takes 0.5 ms. Between two cities it takes 30 ms.
  • A server works 99.9% of the time, independently of other servers.

Time per write (calculation).

  • Asynchronous: 1 ms (the leader's disk only).
  • Synchronous, follower in the same data centre: 1 + 0.5 + 1 = 2.5 ms, the worst case when the steps run one after another.
  • Synchronous, follower in another city: 1 + 30 + 1 = 32 ms. That is 32 times slower than asynchronous.

Parallel commits needed (calculation). Assumption: 300 writes per second. Work in flight = rate x time per write.

  • Asynchronous: 300 x 0.001 s = 0.3 commits in flight.
  • Synchronous across cities: 300 x 0.032 s = 9.6, so about 10 commits must wait at the same time. Your connection pool must allow it. A client that writes one row after another manages only 1 / 0.032 = 31 writes per second.

Write availability (calculation). A 30-day month has 43,200 minutes.

  • Asynchronous: the leader only, 0.999. Downtime = 0.001 x 43,200 = 43.2 minutes.
  • Synchronous with one named follower: both must work, 0.999 x 0.999 = 0.998001. Downtime = (1 - 0.998001) x 43,200 = 86.4 minutes.
  • Synchronous with "any one of two followers": writes stop if the leader is down, or both followers are down. The chance of both followers down is 0.001 x 0.001 = 0.000001. Availability = 0.999 x (1 - 0.000001) = 0.998999001. Downtime = (1 - 0.998999001) x 43,200 = 43.24 minutes, almost the same as the leader alone.

Data at risk (calculation). Worked example: a food-delivery app. Assumptions: 300 writes per second, followers 0.4 seconds behind, average order value Rs 400.

  • Changes only on the leader = 300 x 0.4 = 120.
  • If the leader's disk is lost at that moment, 120 changes are gone. At Rs 400 each, that is 120 x 400 = Rs 48,000 of confirmed orders.
  • After a write burst the lag may reach 2 seconds (assumption). Then 300 x 2 = 600 changes, which is Rs 240,000 (2.4 lakh).

If the loss and the lost trust cost more than a second follower and 1.5 ms per write, go synchronous.

What breaks with each choice

Asynchronous.

  • The leader dies and a follower is promoted. The follower does not have the last changes, but users already saw "saved". The Postgres manual says the data loss is proportional to the delay at failover time (see Sources). Fallback: keep the old leader's log, compare it with the new leader, and replay or refund the missing orders.
  • An event may exist for an order that the new leader does not know. That is a ghost event. Fallback: the outbox pattern and a daily reconcile.
  • The old leader may return with extra changes. Fallback: fence it, and let it rejoin only after a rewind.

Synchronous.

  • A dead or slow follower stops commits. The Postgres manual says such commits may never complete if a synchronous standby crashes. Fallback: name several standbys and wait for any one of them (in Postgres, ANY 1 (s1, s2)). Or alert and switch to asynchronous after a timeout. The second fallback means you may lose data during that time, so a person should decide.
  • One slow disk on the follower slows every write. Fallback: watch the 99th percentile write time, not only the average.
  • The wait ends at a setting you chose. In Postgres, remote_write waits until the follower has received the change and handed it to its operating system. on waits until the change is safe on the follower's disk. remote_apply waits until reads on the follower can see it. The weaker setting is quicker, but a crash of the follower's whole machine can still lose data.

What it costs

  • Servers. Safe synchronous needs 2 followers (one to wait for, one spare): one more server than a single-follower asynchronous setup. Assumption: a follower costs the same as the leader.
  • People. You need alerts on lag and stuck commits, and a runbook. Assumption: 2 to 3 engineer-days.
  • Changing your mind. This is cheap. In Postgres you edit synchronous_standby_names on the primary and reload the configuration file, with no restart (see Sources). Test failover first. synchronous_commit can be changed at any time, for each transaction, so payments wait and view counters do not.

How each level answers

SDE-1: Asynchronous means the leader says "saved" before the follower has the change. Synchronous means it waits. Asynchronous is faster but can lose recent changes.

SDE-2: I use synchronous for payments and orders, because losing a confirmed order is worse than 2.5 ms. I use asynchronous for counters and feeds, because speed matters more and the data can be rebuilt.

Senior: With 300 writes per second and 0.4 seconds of lag, async risks 120 changes, about Rs 48,000. I wait for any one of two followers in the same region: writes cost 2.5 ms and write downtime stays near 43 minutes a month. If one follower dies, the other carries on. Across cities the cost is 32 ms, so the second city stays asynchronous.

Staff: I frame it as a recovery point objective, which is how much data the business can lose. I ask finance and support what a lost order costs. Then I set a rule for each data class: zero loss for payments, a few seconds for the rest. I track lag, stuck commits and lost-write incidents. After each failover test I check how many confirmed writes were missing. The platform team owns the failover runbook.

Say it in the interview

For payments I would wait for one of two nearby followers. It adds about 1.5 ms per write, and we lose no confirmed write if the leader dies. Feeds, counters and far cities stay asynchronous, because 120 lost changes cost little and 32 ms per write is too slow.

Common mistakes

  • Saying "synchronous is always safer". One named follower doubles your write downtime from 43.2 to 86.4 minutes a month in the example.
  • Waiting for a follower in another city. Each write then costs 32 ms, and the pool needs many more connections.
  • Forgetting that async loss is confirmed data. The user saw "saved", so it is a broken promise.
  • Never testing failover. A setting that was never tested can hide a stuck commit that only appears on the worst day.

Sources

Understand the trade-off. Practise explaining your choice.

In an AI mock interview, the interviewer can push on a choice like this one: it changes a number and asks if your answer still holds.

See the mock interviewsAll trade-offs