Trade-offs · Production decisions
Lease vs lock vs fencing token
How to choose between a plain lock, a lease with a time limit and a fencing token, with lease lengths, failure counts and the fallback.
Asked at: SDE-2Senior
The decision
Sometimes only one worker may run a job or change a record at a time. A plain lock is held until its owner releases it, or until the owner's connection ends. A lease is a lock with a time limit: if the owner does not renew it, it ends by itself. A fencing token is a number that grows each time the lock is granted. The data remembers the highest token it has seen and refuses a smaller one. These three are not rivals. They are three levels of safety.
Choose a plain lock when
- All the work happens inside one database, and the lock is a row lock or an advisory lock. The database releases it when the transaction or the session ends.
- The critical part is short, under a few hundred milliseconds, and never waits for a person or a network call.
- A crash must release the lock without a timer. A database session does this for you.
- One database already sees every writer, so servers need no shared lock.
Choose a lease when
- The lock is shared by many servers, and the holder may crash. A time limit stops a dead holder from blocking work forever.
- The lock only saves work, such as "run this nightly job once". If two workers run, nothing is damaged.
- You can make the work safe to repeat, so a rare double run does no harm.
- The holder can renew the lease while it works, and it stops when renewing fails.
Choose a fencing token when
- A second holder would damage data, for example a stale holder that overwrites a total.
- The thing you write to is yours to change, such as a database row, so it can check the token. An SMS provider cannot check it.
- You have a counter that never repeats: one where a write reaches a majority of copies, or inside the data's own database.
In practice you use a lease plus a token, because the lease hands out the numbers.
The numbers that change the answer
All numbers here are assumptions for a nightly job that updates a total for each restaurant.
Lease length. 99 of 100 runs end within 20 seconds. The slowest 1 in 100 takes 90 seconds. A 30-second lease expires on the slow runs: 1% of 1,000 restaurants is 10 runs a night with a second holder. The fix is to renew. Our own advice, not a standard: renew every lease / 3. For a 30-second lease that is every 10 seconds. With 1,000 holders there are 1,000 / 10 = 100 renewals per second.
Wait after a crash. Work waits at most one lease. A 30-second lease waits 30 seconds. A 10-minute lease waits 600 seconds, which is 20 times longer (600 / 30).
Pauses that outlast the lease. A holder can freeze for a long time. Assumption: 1 run in 10,000 freezes for longer than the lease, and there are 5,000,000 runs a day. That is 5,000,000 / 10,000 = 500 runs a day where a second holder starts while the first still believes it holds the lock. Without a token, each one can overwrite good data. With a token, the database refuses a stale write once the newer holder has written, so up to 500 bad writes a day are stopped.
Lock server failover. Assumption: the lock server copies to a follower with a delay of 0.4 seconds, and it grants 50 locks per second. At the moment of a crash, 50 x 0.4 = 20 grants may not be copied. The new leader can grant the same locks again. This is why the token counter must live where a write reaches a majority, or in the data's own database. Assumption: that costs 5 ms per grant instead of 1 ms.
The check. The token check and the write must be one step:
UPDATE totals SET amount = 42000, last_token = 33
WHERE restaurant_id = 77 AND last_token <= 33;If it changes 0 rows, the holder lost the lock and must stop. The test is "less than or equal", so one holder can write many times.
What breaks with each choice
Plain lock.
- It only works where the lock and the data share one database. A hung connection can hold a row lock for a long time, so set
idle_in_transaction_session_timeoutfor holders andlock_timeoutfor waiters. - Two locks taken in different orders can deadlock. Lock in one fixed order.
Lease.
- A lease can end while the holder is paused, so two holders run at once. Checking "do I still hold it" just before the write does not help, because a pause can come right after the check.
- A clock jump on the lock server can end a lease early. Holders should measure their own time with a monotonic clock, which only moves forward.
- A long lease means a long wait after a crash. A short one ends under healthy slow runs.
Fencing token.
- The token counter can repeat after a failover if it sits on an asynchronous follower.
- A resource that ignores the token gives no protection. For such work, make the action safe to repeat, or insert a unique row for each action before you do it.
- The check refuses a stale write only after the newer holder has written once. Before that, the stale write passes, and the newer holder replaces it later. So write a whole new value, not a change to the old one.
- The refused holder must handle the refusal: stop, take the lock again, and read fresh data.
What it costs
All numbers here are assumptions.
- Plain lock: nothing new. It uses the database you have.
- Lease: one shared store, a renewal loop and alarms. About a week to build and test well.
- Fencing token: a counter, a column on each protected row, a check in each write, and tests that pause a holder. About one more week.
- Changing your mind: a token column is a small change, but every writer must send the token. Plan a few days to find all writers.
How each level answers
The question is: "A job must run on one server at a time. What if that server freezes?"
SDE-1: A lock lets one worker in. A lease is a lock that ends after a set time, so a crashed worker does not block others.
SDE-2: I use a lease with renewal, because a crashed holder would otherwise block the job forever. The lease is longer than a normal run, and the worker renews it while it works.
Senior: A paused holder can still write after its lease ends, so the lease alone cannot protect data. I add a fencing token and make the database refuse any older token. With 1 in 10,000 of 5,000,000 daily runs pausing too long, the token can stop up to 500 stale writes a day.
Staff: I ask what a double run costs: wasted work allows a lease, and moving money needs a token or a conditional update. I roll out in steps: add the token column, make every writer send it, then turn the check on, all in one shared library. I track refused stale writes and failed renewals, because a rise in either means the leases are too short.
Say it in the interview
I would use a 30-second lease, renewed every 10 seconds, so a crash blocks work for at most 30 seconds. For data that must stay correct I add a fencing token, because a paused holder can still write after its lease ends. If the target cannot check tokens, I make the action safe to repeat with a unique row.
Common mistakes
- Trusting a lease alone for correctness. A holder frozen for 45 seconds keeps working after a 30-second lease. Kleppmann explains this in the article linked below.
- Checking the lock just before the write. The check and the write are two steps, and a pause fits between them.
- Taking the token from an asynchronous follower. After a failover the number can repeat, and the check then passes for a stale holder.
- Using a lock when one conditional update would do. A check and a write in one statement leaves no lock to lose.
Sources
- Martin Kleppmann, "How to do distributed locking" (locks for efficiency or correctness; leases and pauses; fencing tokens): https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html (opens in a new tab) (checked 26 Oct 2026).
- Postgres, explicit locking (row locks last until the transaction ends; advisory locks can be tied to a session or a transaction; deadlocks are detected): https://www.postgresql.org/docs/current/explicit-locking.html (opens in a new tab) (checked 26 Oct 2026).
- The run times, pause rate, delays and costs are assumptions made for teaching. The advice to renew every lease / 3 is our own. The counter that repeats after a failover is our own reasoning from asynchronous copying.
Understand the trade-off. Practise explaining your choice.
In an AI mock interview, the interviewer can push on a choice like this one: it changes a number and asks if your answer still holds.