Skip to content

Trade-offs · Production decisions

Lease vs lock vs fencing token

How to choose between a plain lock, a lease with a time limit and a fencing token, with lease lengths, failure counts and the fallback.

Asked at: SDE-2Senior

The decision

Sometimes only one worker may run a job or change a record at a time. A plain lock is held until its owner releases it, or until the owner's connection ends. A lease is a lock with a time limit: if the owner does not renew it, it ends by itself. A fencing token is a number that grows each time the lock is granted. The data remembers the highest token it has seen and refuses a smaller one. These three are not rivals. They are three levels of safety.

Choose a plain lock when

  • All the work happens inside one database, and the lock is a row lock or an advisory lock. The database releases it when the transaction or the session ends.
  • The critical part is short, under a few hundred milliseconds, and never waits for a person or a network call.
  • A crash must release the lock without a timer. A database session does this for you.
  • One database already sees every writer, so servers need no shared lock.

Choose a lease when

  • The lock is shared by many servers, and the holder may crash. A time limit stops a dead holder from blocking work forever.
  • The lock only saves work, such as "run this nightly job once". If two workers run, nothing is damaged.
  • You can make the work safe to repeat, so a rare double run does no harm.
  • The holder can renew the lease while it works, and it stops when renewing fails.

Choose a fencing token when

  • A second holder would damage data, for example a stale holder that overwrites a total.
  • The thing you write to is yours to change, such as a database row, so it can check the token. An SMS provider cannot check it.
  • You have a counter that never repeats: one where a write reaches a majority of copies, or inside the data's own database.

In practice you use a lease plus a token, because the lease hands out the numbers.

The numbers that change the answer

All numbers here are assumptions for a nightly job that updates a total for each restaurant.

Lease length. 99 of 100 runs end within 20 seconds. The slowest 1 in 100 takes 90 seconds. A 30-second lease expires on the slow runs: 1% of 1,000 restaurants is 10 runs a night with a second holder. The fix is to renew. Our own advice, not a standard: renew every lease / 3. For a 30-second lease that is every 10 seconds. With 1,000 holders there are 1,000 / 10 = 100 renewals per second.

Wait after a crash. Work waits at most one lease. A 30-second lease waits 30 seconds. A 10-minute lease waits 600 seconds, which is 20 times longer (600 / 30).

Pauses that outlast the lease. A holder can freeze for a long time. Assumption: 1 run in 10,000 freezes for longer than the lease, and there are 5,000,000 runs a day. That is 5,000,000 / 10,000 = 500 runs a day where a second holder starts while the first still believes it holds the lock. Without a token, each one can overwrite good data. With a token, the database refuses a stale write once the newer holder has written, so up to 500 bad writes a day are stopped.

Lock server failover. Assumption: the lock server copies to a follower with a delay of 0.4 seconds, and it grants 50 locks per second. At the moment of a crash, 50 x 0.4 = 20 grants may not be copied. The new leader can grant the same locks again. This is why the token counter must live where a write reaches a majority, or in the data's own database. Assumption: that costs 5 ms per grant instead of 1 ms.

The check. The token check and the write must be one step:

sql
UPDATE totals SET amount = 42000, last_token = 33
WHERE restaurant_id = 77 AND last_token <= 33;

If it changes 0 rows, the holder lost the lock and must stop. The test is "less than or equal", so one holder can write many times.

What breaks with each choice

Plain lock.

  • It only works where the lock and the data share one database. A hung connection can hold a row lock for a long time, so set idle_in_transaction_session_timeout for holders and lock_timeout for waiters.
  • Two locks taken in different orders can deadlock. Lock in one fixed order.

Lease.

  • A lease can end while the holder is paused, so two holders run at once. Checking "do I still hold it" just before the write does not help, because a pause can come right after the check.
  • A clock jump on the lock server can end a lease early. Holders should measure their own time with a monotonic clock, which only moves forward.
  • A long lease means a long wait after a crash. A short one ends under healthy slow runs.

Fencing token.

  • The token counter can repeat after a failover if it sits on an asynchronous follower.
  • A resource that ignores the token gives no protection. For such work, make the action safe to repeat, or insert a unique row for each action before you do it.
  • The check refuses a stale write only after the newer holder has written once. Before that, the stale write passes, and the newer holder replaces it later. So write a whole new value, not a change to the old one.
  • The refused holder must handle the refusal: stop, take the lock again, and read fresh data.

What it costs

All numbers here are assumptions.

  • Plain lock: nothing new. It uses the database you have.
  • Lease: one shared store, a renewal loop and alarms. About a week to build and test well.
  • Fencing token: a counter, a column on each protected row, a check in each write, and tests that pause a holder. About one more week.
  • Changing your mind: a token column is a small change, but every writer must send the token. Plan a few days to find all writers.

How each level answers

The question is: "A job must run on one server at a time. What if that server freezes?"

SDE-1: A lock lets one worker in. A lease is a lock that ends after a set time, so a crashed worker does not block others.

SDE-2: I use a lease with renewal, because a crashed holder would otherwise block the job forever. The lease is longer than a normal run, and the worker renews it while it works.

Senior: A paused holder can still write after its lease ends, so the lease alone cannot protect data. I add a fencing token and make the database refuse any older token. With 1 in 10,000 of 5,000,000 daily runs pausing too long, the token can stop up to 500 stale writes a day.

Staff: I ask what a double run costs: wasted work allows a lease, and moving money needs a token or a conditional update. I roll out in steps: add the token column, make every writer send it, then turn the check on, all in one shared library. I track refused stale writes and failed renewals, because a rise in either means the leases are too short.

Say it in the interview

I would use a 30-second lease, renewed every 10 seconds, so a crash blocks work for at most 30 seconds. For data that must stay correct I add a fencing token, because a paused holder can still write after its lease ends. If the target cannot check tokens, I make the action safe to repeat with a unique row.

Common mistakes

  • Trusting a lease alone for correctness. A holder frozen for 45 seconds keeps working after a 30-second lease. Kleppmann explains this in the article linked below.
  • Checking the lock just before the write. The check and the write are two steps, and a pause fits between them.
  • Taking the token from an asynchronous follower. After a failover the number can repeat, and the check then passes for a stale holder.
  • Using a lock when one conditional update would do. A check and a write in one statement leaves no lock to lose.

Sources

Understand the trade-off. Practise explaining your choice.

In an AI mock interview, the interviewer can push on a choice like this one: it changes a number and asks if your answer still holds.