Skip to content

Trade-offs · Senior and Staff

Consistency budget across services

How to decide how stale each service may be, by setting a delay budget for the whole user journey and sharing it among hops, with numbers and fallbacks.

Asked at: SeniorStaff

The decision

When one user action touches many services, each service may show data that is a little old. A consistency budget is the longest delay the user may see between an action and its effect everywhere. An example is "an order appears in the order list within 2 seconds, for 99 out of 100 orders". You then share that budget among the hops. You have three ways to build. Strong everywhere means each read goes to the owner of the data, or to its primary copy, and waits for the latest value. Eventual with a budget means each hop may lag, and you measure the lag. Tiered means strong for the few data types where a wrong value costs money, and a budget for the rest.

Choose strong everywhere when

  • Every service handles money, stock or permissions, where a wrong read loses money or breaks trust.
  • The journey has few hops, so the extra delay and the extra failure chance stay small.
  • A wrong answer is worse than a slow or failed answer, for example a bank balance before a withdrawal.

Choose eventual with a budget when

  • Most screens only show information: lists, feeds, search, dashboards.
  • Hops use queues and indexes, which are lagged by nature.
  • You need each service to stay up when another is down.
  • Reads are many, and read cost matters. For example, DynamoDB charges half as much for an eventually consistent read as for a strongly consistent one (see Sources).

Choose tiers when

  • Your product has both kinds of data, which is the usual case.
  • You can name each data type's owner and its allowed delay in a table that everyone can read.
  • You can afford a small amount of design work now to avoid a rewrite later.

The numbers that change the answer

Spending the budget (assumptions plus calculation). The journey: a user places an order and opens "My orders". Budget: 2,000 ms at the 99th percentile. The 99th-percentile delay of each hop, all assumptions:

HopDelay (ms)
Relay polls the outbox200
Broker100
Consumer queue wait800
Search index refresh1,000
Read replica lag500

Sum = 200 + 100 + 800 + 1,000 + 500 = 2,600 ms. Adding the 99th percentiles of each hop is a cautious estimate. All hops are rarely slow together, but the sum is simple and easy to check. The sum of 2,600 ms is over the budget of 2,000 ms by 600 ms. Each fix has a cost. Poll every 50 ms instead of 200 (more database queries). Scale the consumer to cut its wait to 300 ms (more servers). Refresh the index every 500 ms (more index work). New sum = 50 + 100 + 300 + 500 + 500 = 1,450 ms, which fits.

Cost of strong reads (assumptions plus calculation). If the order list asks 4 services in a row, each with a 15 ms call, the wait is 4 x 15 = 60 ms. Availability: Assumption: each service is up 99.9% of the time. All four must be up, so 0.999 to the power of 4 = 0.996, or 99.6%. A 30-day month has 43,200 minutes, so downtime is (1 - 0.996) x 43,200 = about 173 minutes. One service alone has 0.001 x 43,200 = 43.2 minutes. The chain is about 4 times worse.

A cheaper fix for the user's own view. Read your own writes: for 5 seconds (Assumption) after a user writes, read that user's data from the primary copy. Only that user pays the cost, and the rest of the traffic stays eventual. Assumption: 3% of reads happen in those 5 seconds, so only 3% of reads go to the primary copy.

Cited facts. DynamoDB reads are eventually consistent by default and cost half of strongly consistent reads. All reads from a global secondary index are eventually consistent. In a global table in the default mode, changes reach other Regions typically within a second (see Sources).

What breaks with each choice

Strong everywhere.

  • A slow service slows every journey that calls it, and one down service stops the chain (99.6% in the example).
  • The primary copy becomes the busiest point and a limit on scale. Fallback: cache with short lifetimes only where a small delay is fine.

Eventual with a budget.

  • A user sees an old value and acts on it, for example pays twice or reports "my order is missing". Fallback: read your own writes, a "processing" state, and idempotent actions.
  • A hop quietly gets slower and nobody notices. Fallback: measure lag for each hop and alert when a hop uses more than its share.
  • The sum looks fine on average and fails at the 99th percentile. Fallback: budget on percentiles, not averages.

Tiers.

  • The table of types and delays drifts from the code. Fallback: the owner reviews it each quarter, and the lag is shown on a dashboard.
  • A developer treats a lagged read as strong. Fallback: name the client calls by their guarantee, such as getOrderStrong and listOrdersLagged.

What it costs

  • People: Assumption: one engineer-week to measure lag at each hop, and two weeks for read-your-own-writes in the order service.
  • Reversible-first path. Start with tiers on paper: list your data types and give each an allowed delay, with money and permissions as strong. Measure the real lag before you change code. Add read-your-own-writes first because it is cheap. Move a type from lagged to strong when users hit a wrong value more than once in 1,000 sessions (Assumption: our example trigger). Move it also when a wrong value has a money cost. Move a type from strong to lagged when its availability, such as 99.6%, causes more pain than stale data would.
  • Changing your mind: moving from lagged to strong is a code change and more load. Moving from strong to lagged is easy in code but needs new screens for the "processing" state.

How each level answers

The question is: "A user places an order. How fresh must the order list be?"

SDE-1: After a write, a read from a copy can show old data for a short time. Strong reads avoid that and are slower.

SDE-2: The order list can be slightly old, so I read from a replica. I show the user's own new order straight from the write response, so it never looks missing.

Senior: I set a budget of 2 seconds at the 99th percentile and add the hops: 2,600 ms, which is 600 ms over. I fix the three biggest hops to reach 1,450 ms. I do not read strongly across four services, because 99.9% to the power of 4 is about 173 minutes of downtime a month. The fallback is the primary copy for the user's own reads for 5 seconds.

Staff: I make the budget a product decision with the owners of each journey, written as a table of data types, owners and delays. Money and permissions are strong; lists and search are lagged. I measure the lag of every hop, and the number of "missing order" tickets. The platform team provides the lag metrics and the client libraries. I review the table every quarter and when a journey adds a hop.

Say it in the interview

I would set one budget for the whole journey, such as 2 seconds at the 99th percentile for the order list. Then I would share it across the hops. In my example the hops sum to 2,600 ms, so I would cut the three slowest. Money and stock reads stay strong, and the rest stay lagged, with read-your-own-writes as the fix for the user's own view.

Common mistakes

  • Saying "eventual" without a number. Eventual means nothing until you give it a limit and a way to measure it.
  • Averaging. Users complain about the slow tail, so budget with the 99th percentile.
  • Making everything strong "to be safe". Four services at 99.9% each give 99.6%. Safety on paper becomes outages in practice.
  • Forgetting the user's own screen. Showing the user their own change from the response solves most complaints.

Sources

  • Amazon DynamoDB, read consistency: eventually consistent reads are the default and cost half of strongly consistent reads; reads from global secondary indexes are always eventually consistent; with global tables, changes replicate to other Regions typically within a second in the default mode. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadConsistency.html (opens in a new tab) (checked 26 Oct 2026)
  • Hop delays, the 2,000 ms budget, service availability, call times, the 5-second window, the 3% share of reads and the engineer-weeks are our assumptions. The 2,600 ms, 1,450 ms, 99.6% and 173 minutes are our calculations, shown above.

Understand the trade-off. Practise explaining your choice.

In an AI mock interview, the interviewer can push on a choice like this one: it changes a number and asks if your answer still holds.