Serving a model at scale: batch or real-time predictions, latency and cost

The interviewer says: "The model ranks restaurants for each user. Do you compute the lists at night, or when the user opens the app? How long may a request take? What does it cost? What do users see when the model service is down?"

A weak answer says "real time, because it is fresh", with no numbers. A strong answer asks how fresh the answer must be and how many user and restaurant pairs exist. It adds the time of each stage against a budget, works out the cost of 1,000 predictions for both designs, and names a fallback.

At SDE-2 level you should choose with numbers. At SDE-3 level you should also handle server start-up, a new model version, and a slow or dead model service.

Think of a restaurant. In the morning the cook prepares the most-ordered dishes and keeps them hot. A customer gets a dish at once, but it is hours old, and some dishes are never eaten. Or the cook makes each dish to order: fresh, but the customer waits.

A prediction is the answer of the model for one case (lesson 163). Serving means giving predictions to the product. There are two ways.

• Batch prediction. A job computes the predictions for everyone, for example for every pair of a user and a restaurant, each night. It stores them in a table or cache, with the user as the key. At request time the service reads the stored list.
• Real-time prediction. The service computes the predictions when the request arrives.

Two lanes. In batch, a night job scores every pair and stores a list for each user. When the user opens the app, the request only reads the stored list: fast, but old. In real time, the request arrives, a model service builds the features and scores the pairs now, and the answer is fresh, but the user waits.

Choose with three questions:

1. How fresh must the answer be? Batch uses facts from the night before. It cannot know where the user is now, the cart, or how busy a kitchen is.
2. How many pairs exist? Batch scores all of them, also for users who never open the app. Real time scores only the pairs asked for, but its servers must be ready all day.
3. What is the latency budget, the time you allow for one request? Real time must finish inside it.

A real-time answer can be cached for a short time (lesson 57). Only a repeated case benefits, and a score built from live features gets old fast.

A mix is common: a nightly list is the base, and real time re-scores a short part with fresh features. This is the precompute against compute-on-read choice of the trade-off pages. Cloud ML platforms offer both modes.

The latency budget in milliseconds

A real-time request has stages: the network call, fetching features (lesson 164), running the model, shaping the answer, and overhead. Give each stage its p99 time (lesson 4), not the average, and add them. This is only a first estimate, because delays can come together (one busy server slows every stage). A request that calls 5 services and waits for all of them is worse. Say each is slow in 1 request of 100, independently. Then 1 - 0.99 to the power of 5 = 4.9% of requests wait for a slow call. Set a timeout a little above the budget (lesson 82), with a fallback.

Model size and hardware

A model must fit in memory. The file size is weights x bytes for each weight, and running it needs some extra memory. Our model has 50,000,000 weights of 4 bytes, so the file is 200 MB.

An ordinary processor (CPU), with the cores of lesson 31, runs small models well. An accelerator, often a graphics processor, does thousands of the same multiply-and-add steps at once. It costs more each hour, suits large models, and wastes money when idle.

Dynamic batching

Dynamic batching means that the server waits a few milliseconds to collect requests, then runs them as one group. A group costs one fixed start-up time plus a small time for each item, so the start-up time is shared. Throughput rises, but each request waits longer. NVIDIA's Triton server lets you set the largest batch and the longest wait. Its guide says to try longer delays while you watch throughput and the latency budget.

Starting, scaling, releasing and falling back

• Cold start. A new server must read the model file and load it before it can answer. It should pass the health check (lesson 54) only after that. So autoscaling, which adds servers when load rises, lags behind a sudden rise.
• Canary for a new model version (lesson 127): a small share of requests goes to the new version while you compare p99 and errors. Going back means setting the "live" record to the old version.
• Fallback. When the model is slow or down, the product answers with something simpler: the nightly list or a list of popular restaurants. A circuit breaker (lesson 83) stops calls to a dead service.
• Cost per 1,000 predictions (lesson 161): the cost of a day divided by that day's predictions, in thousands.

SwadKart (lessons 163 and 164) shows 200 restaurants to a user who opens the app.

Assumptions (made up to teach with):

• 2,000,000 users have the app. 500,000 of them open it each day, 3 times each: 1,500,000 requests. Peak is 69.6 requests a second.
• Scoring one pair takes 0.05 ms on one core. Prices are assumptions (October 2026), not quotes. A 4-core server costs Rs 12 an hour, which is Rs 3 for each core-hour. Real time uses 3 servers, as in lesson 163. An accelerator costs Rs 120 an hour.
• The service has 50 ms at p99. Lesson 163 gave 30 ms in all, split here as: network 3, features 8, model 10, shaping the answer 4, overhead 5.

Step 1: predictions. Batch: 2,000,000 x 200 = 400,000,000 each night. Real time: 1,500,000 x 200 = 300,000,000 a day. Batch also scores the 1,500,000 users who do not open the app (2,000,000 - 500,000), which is 75% of its work.

Step 2: cost (the predict box). Batch work: 400,000,000 x 0.05 ms = 20,000 seconds = 5.56 core-hours, which costs 5.56 x Rs 3 = Rs 16.7 a night. Per 1,000 predictions: Rs 16.7 / 400,000 = Rs 0.00004 (rounded). Real time: 3 servers x Rs 12 x 24 hours = Rs 864 a day, so Rs 864 / 300,000 = Rs 0.00288 per 1,000, about 69 times more. Real time does less work (300,000,000 x 0.05 ms = 4.2 core-hours), but we pay for 3 x 4 x 24 = 288 core-hours, so the servers are 1.4% busy. The fleet is sized for the peak and for failures, while the batch job pays only for the hours it runs. Table storage is not counted. Both bills are small, so freshness decides.

Step 3: the budget. The stages add up to 3 + 8 + 10 + 4 + 5 = 30 ms, inside 50 ms. A timeout of 40 ms plus 5 ms to read the nightly list is 45 ms, so a fallback fits.

Step 4: a heavier model. Say the model takes 20 ms a pair on one core, 400 times more. For 200 pairs that is 4,000 ms of work, far above 30 ms. Batch: 400,000,000 x 20 ms = 8,000,000 seconds = 2,222 core-hours = Rs 6,667 a night. So use the mix: the nightly job builds the 200-pair list, and real time re-scores only the best 20 with fresh features.

Step 5: dynamic batching. The re-score runs on an accelerator. Assumption: for this heavy model, a batch takes 6 ms plus 0.1 ms for each pair. One request has 20 pairs: 6 + 2 = 8 ms, so 1,000 / 8 = 125 requests a second. Suppose the app grows to a peak of 300 requests a second, with accelerators at most 70% busy.

Batching halves the cost, and the price is time. With a longest wait of 4 ms, a request can wait for the batch before it, then for its own. For batches of 8 that is 22 + 4 + 22 = 48 ms, against 8 ms alone. The other stages take 3 + 8 + 4 + 5 = 20 ms of the 50 ms, so the model stage may use 30 ms, and batches of 8 are too slow. Batches of 3 take 6 + 6 = 12 ms and serve 3 x 1,000 / 12 = 250 requests a second, so 300 / (250 x 0.7) = 1.7, which is 2 accelerators. The worst case is 12 + 4 + 12 = 28 ms.

Step 6: start, release, fail. Cold start: reading 200 MB at 100 MB/s takes 2 s, and loading takes 3 s (assumption). A server takes 60 s to start, so a new one serves after 60 + 2 + 3 = 65 s. Fallback: assume 100 orders for every 1,000 opens with the model, 95 with the nightly list, 80 with a popular list. If the peak lasted one hour, an outage would hit 69.6 x 3,600 = 250,560 opens. Orders lost: nightly list 250.56 x 5 = 1,253; popular list 250.56 x 20 = 5,011; blank screen 250.56 x 100 = 25,056.

Batch solves a heavy model and a small, flat latency (a table read) at a low cost per prediction. It costs old answers, 75% wasted work and a table to store. Real time solves fresh answers. It costs a fleet sized for the peak, idle most of the day (69 times more per prediction here), plus a latency budget and a failure plan. I would use batch when answers can be a day old or the model is heavy, real time when fresh facts change the answer, and a mix for both.

Dynamic batching solves cost (2 accelerators, not 4) and costs latency (up to 28 ms with batches of 3, not 8). I would use it only when the budget has room.

"Real time is always better, because it is fresher." Here it cost 69 times more per prediction. A model of 20 ms a pair cannot score 200 pairs inside 30 ms.

"Dynamic batching only makes things faster." It adds waiting: the worst case went from 8 ms to 28 ms, even with small batches.

"If each of 5 services has a p99 of 20 ms, the request has a p99 of 20 ms." A request that waits for all 5 meets a slow call in 4.9% of cases. Measure the whole request.

1. What are batch and real-time prediction? Name the three questions that choose between them. What does dynamic batching trade?
2. A music app has 1,000,000 users and 100 songs for each. 400,000 users open the app each day, one request each. Scoring a pair takes 0.1 ms on one core, and a core-hour costs Rs 3. The real-time service has 3 servers at Rs 12 an hour. (a) How many predictions does the nightly batch make, and what does it cost? (b) How many predictions does real time make in a day, and what does the fleet cost? (c) What is the cost per 1,000 predictions for each?
3. The model service has a p99 of 80 ms, the budget is 50 ms, and the service takes 65 seconds to add a server. Traffic doubles every evening at 8 pm. What do you do?