Skip to content

Trade-offs · Foundations

Retry vs fail fast

When to retry a failed call and when to fail fast and return an error or a fallback, the numbers that decide, and what breaks with each.

Asked at: SDE-1SDE-2

The decision

A call to another service fails. You can retry: try the same call again after a short wait. Or you can fail fast: stop at once and return an error, or a simpler answer called a fallback. Retry helps when the failure is short and random. It hurts when the other service is overloaded or down, because every retry adds more load to a service that is already struggling.

Choose retry when

  • The failure is short and random, such as one lost message or a server that is restarting.
  • The call is safe to repeat. Reading data is safe. Creating a payment is not, unless it carries an idempotency key, a unique code that lets the server ignore a repeat.
  • You have time left. The user's deadline must allow another try.
  • Only one layer of your system retries.

Choose fail fast when

  • The other service is down or overloaded. More calls make it worse.
  • The call is not safe to repeat, and you have no idempotency key.
  • The user's deadline is almost used up.
  • You have a fallback, such as an older value from a cache.
  • Several layers call each other, and each might retry.

The numbers that change the answer

Retries help only when failures are independent. If a call fails 5% of the time by chance, three tries all fail with chance 0.05 x 0.05 x 0.05 = 0.000125. Success rises from 95% to 99.9875%. If the service is down, all three tries fail, and you sent three times the traffic.

Layers multiply. Assumption: the app, the API server and the order service each retry 3 times, so each makes up to 4 tries. One user action can reach the last service as 4 x 4 x 4 = 64 calls. Google's SRE book describes this effect, with 64 attempts in its example (see Sources).

Worked example at Indian scale. A bank app checks payment status on salary day. Assumptions: the status service handles 1,200 requests per second. Normal traffic is 1,000 per second. Then it starts to fail on every call, and clients retry.

PolicyCalculationLoad on the service
No retry1,000 x 11,000 per second (83%)
3 tries at one layer1,000 x 33,000 per second (250%)
3 layers, 4 tries each1,000 x 6464,000 per second
Retry budget of 10%1,000 + 0.10 x 1,0001,100 per second (92%)

A retry budget allows retries only up to a small share of normal traffic. The 10% is our assumption.

Waiting costs workers. Requests in progress = requests per second x seconds each takes. Assumption: the API server has 200 workers and gets 100 requests per second. With a 2-second timeout and 3 tries, a dead dependency holds each request for 6 seconds. That is 100 x 6 = 600 requests in progress, or 300% of the workers. With a 300 ms timeout and fail fast: 100 x 0.3 = 30, or 15%.

Jitter. If 1,000 clients fail together and all retry after 100 ms, 1,000 calls land in the same moment. With full jitter, each waits a random time from 0 to 200 ms. That is about 1,000 / 200 = 5 calls per ms. An AWS engineer's test in 2015 found that jitter cut the calls by more than half with 100 competing clients (see Sources).

What breaks with each choice

  • Retry: a retry storm. Retries slow the service, so more calls time out, so there are more retries. A timeout also does not mean the work failed. The server may have finished and the answer was lost, so a repeat can charge twice without an idempotency key.
  • Fail fast: users see errors for faults that would have passed in a moment. A circuit breaker can fail fast too early. It stops calls to a broken service for a while, and then tests with one call. Fix: tune the trip point, and always show a clear message or fallback.

What it costs

  • Retries cost extra traffic: up to 3x here. They also cost design work: idempotency keys, a backoff with jitter, a retry budget (days, rough guess).
  • Fail fast costs lost requests during short faults, and a fallback to build and test.
  • Changing the policy is a config change if you built it as one setting. It is a code change if retries are spread over many layers.

How each level answers

SDE-1: A retry tries again. Fail fast gives up at once. Retry only calls that are safe to repeat.

SDE-2: On calls that are safe to repeat, I retry timeouts and 503 errors up to 3 times with backoff and jitter, at one layer. I do not retry a payment without an idempotency key.

Senior: If the service is down, 3 tries at 1,000 requests per second give 3,000, which is 250% of its capacity. Across 3 layers it is 64,000. So I use one retry layer, a 10% retry budget, and a circuit breaker. The fallback is a cached value. The risk is a trip point that is too sensitive.

Staff: I ask what an error costs against what extra load costs. I set one rule for the company: retry at the edge only, with a budget, and every write carries an idempotency key. One platform team owns the client library, so the rule is in code and not in a wiki. To change old services, I count retries per layer first and remove the inner ones one by one. I check the choice by retry rate, error rate, and how long a dependency takes to recover after an outage.

Say it in the interview

I would retry at one layer only, up to 3 times with backoff and jitter, and only for calls that are safe to repeat. When the service is down, I would trip a circuit breaker and return a cached value. Three layers of retries can turn 1,000 calls per second into 64,000.

Common mistakes

  • Retrying at every layer. The calls multiply (4 x 4 x 4 = 64).
  • Retrying without a random wait (jitter). All clients return together.
  • Retrying a payment with no idempotency key. A timeout is not a failure report.
  • Retrying errors that cannot succeed, such as 400 (bad request). Every retry only adds load.

Sources

Understand the trade-off. Practise explaining your choice.

In an AI mock interview, the interviewer can push on a choice like this one: it changes a number and asks if your answer still holds.

See the mock interviewsAll trade-offs