The pieces of an ML system: data, features, training and serving

The interviewer says: "Design a system that decides which restaurants a food app shows first. It should learn from past orders." Many candidates argue about which learning method is best. The interviewer wanted to hear how data reaches the learning step, where the result runs, and what happens on a bad night.

A weak answer names a learning method and stops. A strong answer draws the whole path, from a tap in the app to a changed list on the screen. It gives numbers for the data, for how often learning runs, and for the time of one request. It also says how to go back when the new version is worse. Many strong answers start with a simple rule as a first version. Then the data and serving pieces are ready before a complex model arrives.

At SDE-1 level you should name the pieces and what flows between them. At SDE-2 level you should size them. At SDE-3 level you should also handle answers that arrive late, rollback, and who owns each decision.

Think of a tea-stall owner. Over the years the owner learned that on rainy evenings office workers order two teas, not one. It came from thousands of past orders. But using it takes one second: a worker walks up and the owner starts the kettle.

An ML system has the same two speeds. Learning is slow, reads a lot of past data, and runs now and then. Using is fast and runs for every request. Most of the design work is joining these two speeds.

Here are the words.

• A model is a function that was learned from past examples. It takes some numbers about a case and returns a number. For example, it takes facts about a user and a restaurant, and returns the chance that the user orders.
• A feature is one input number about the case. Examples: "orders this user placed in the last 7 days" and "distance to the restaurant in km".
• A label is the answer we want the model to predict. For example: "the user ordered from this restaurant, 1 for yes and 0 for no". The label is known only after the event, and sometimes much later.
• An example is one past case: its features and its label.
• Training finds the inner numbers of the model, called weights, so that its answers on past examples are close to their labels. A training job is a batch job that does this. A batch job reads a large, fixed input, writes an output and stops. A training job takes hours.
• A prediction is the answer of the model for one case. Serving means answering live requests with a model. Inference is another word for making predictions. This module says "serving".

The pieces and what flows between them

1. Events. The product logs what happened: what was shown, what the user tapped, what the user ordered. The events go to a log (lesson 87). Each event also records the version of the model that made the decision, so that a later result can be matched to it.
2. Training data. A batch job joins the events with the labels that came later. Each row is one example: features and a label. The rows are files in object storage (lesson 40).
3. Training job. It reads the rows and writes a model file: the weights and the rules for using them. The file gets a version number, for example v42.
4. Evaluation. Before release, score the new model on examples that training did not see. Release it only if it is better than the live model.
5. Model store. The file goes to object storage. A small record says which version is live.
6. Serving service. Several servers behind a load balancer (lesson 54) load the live file into memory. For each request they build the features, run the model and return the predictions.
7. Monitoring. Watch errors and speed, as for any service. Also watch what the model decided and what happened afterwards. Those outcomes become labels for the next training run, and the loop closes.

A loop of six boxes. Events log what users saw and did. A batch job joins them with labels into training data. The slow training job learns a model. Evaluation checks that it is better than the live model. The model file, a version such as v42, goes to the model store. The serving service answers every request quickly with it. What users do next comes back as new events and labels.

The model is a small box

In 2015, Google engineers drew a picture of real-world ML systems. The ML code is a small box in the middle. The data, checks, serving and monitoring around it are much larger. An answer about the model alone covers a small part.

What you decide and what the data scientist decides

A data scientist builds the model. This split is common, not a rule.

SwadKart is a made-up food-delivery app that earlier lessons use.

Assumptions (made up to teach with):

• 500,000 users open the app each day, 3 times each. Each open shows 20 restaurants. A typical user can order from 200 restaurants.
• 150,000 orders a day. A shown restaurant has label 1 if the user orders from it within 1 hour.
• One event is 300 bytes. One training row is 200 bytes (40 features of 4 bytes, plus ids and the label).
• Training uses the last 30 days. It runs once a day for 4 hours on 20 machines.
• The model has 50,000,000 weights of 4 bytes each.
• Scoring one user and one restaurant takes 0.05 ms of processor time.
• Peak load is 4 times the average (lesson 30). A server has 4 cores and should be at most 70% busy (lesson 31).
• A request must finish in 50 ms at p99 (lesson 4). Getting features and the network take 20 ms of that.

Step 1: events. Opens: 500,000 x 3 = 1,500,000. Shown restaurants: 1,500,000 x 20 = 30,000,000 a day. Size: 30,000,000 x 300 bytes = 9 GB a day. Share of label 1: 150,000 / 30,000,000 = 0.5%, so 1 row in 200 is a 1.

Step 2: training data. 30 x 30,000,000 = 900,000,000 rows. Size: 900,000,000 x 200 bytes = 180 GB for each training run. Rows with label 1: 30 x 150,000 = 4,500,000.

Step 3: the model file. 50,000,000 x 4 bytes = 200 MB. Fourteen versions take 14 x 200 MB = 2.8 GB. Training costs 20 machines x 4 hours = 80 machine-hours a day.

Step 4: serving load. Average opens: 1,500,000 / 86,400 = 17.36, which we round to 17.4 a second and use from here. Peak: 17.4 x 4 = 69.6 requests a second. Each request scores 200 restaurants, so 69.6 x 200 = 13,920 predictions a second. Work: 13,920 x 0.05 ms = 696 ms a second, which is 0.7 cores. At 70% busy we need 0.7 / 0.7 = 1 core. We run 3 servers, so one can fail and the rest still cope. That is 3 x 24 = 72 server-hours a day, close to the 80 machine-hours of training. But the shapes differ. Serving makes 1,500,000 x 200 = 300,000,000 predictions a day, and it never stops.

Step 5: one request. The model needs 200 x 0.05 ms = 10 ms. With 20 ms for features and network, the total is 30 ms, inside the 50 ms limit.

Step 6: how old is what the model knows? Training starts at 1 am and reads events up to midnight. It ends at 5 am. Checks take 2 hours, so the model goes live at 7 am and serves until the next 7 am. At 7 am the newest data is 7 hours old. At the next 7 am it is 7 + 24 = 31 hours old. The average is (7 + 31) / 2 = 19 hours.

Step 7: a failed night (the predict box). Serving servers keep the model file in memory. They do not call the training job. If training fails, v42 stays live and the user sees a normal list from v42. Each missed night adds 24 hours. After 3 missed nights the average age is 19 + 3 x 24 = 91 hours. So alert on model age, for example when it passes 36 hours (an assumption).

Step 8: a bad model. Say v43 goes live and orders fall from 100 to 90 for every 1,000 opens. Baseline: 150,000 / 1,500,000 x 1,000 = 100. At peak there are 69.6 x 60 = 4,176 opens a minute. The loss is 4,176 / 1,000 x 10 = about 42 orders a minute. Rollback means setting the live record back to v42. Servers load 200 MB again, and we assume that takes 1 minute. If the team sees the drop after 10 minutes, about 11 x 42 = 460 orders are lost. So keep old files and make the switch one step.

Real-world note (Razorpay, 13 Oct 2026). Razorpay wrote about a model that scores a checkout. The product uses the score to decide whether to offer cash on delivery. The real result, whether the order was returned, arrives days later. So the team records, for each decision, which inputs, which release and which rule produced it. Without that record they could not match the late result to the decision. Source: https://engineering.razorpay.com/inside-razorpays-rto-model-the-engineering-behind-its-ml-lifecycle-999fe75e66ec (opens in a new tab) (text read from the company's blog feed, 26 Oct 2026).

Daily batch training solves a model that follows the world, with a simple schedule and a cheap bill. It costs a model that is always 7 to 31 hours behind (19 on average), and a pipeline of seven pieces that can each fail. I would train more often when the world changes within hours and the new data pays for the machines. A versioned file and one "live" record solve rollback in about 1 minute, and cost a little storage (2.8 GB here).

"The model is the system." In the example the model file is 200 MB. A training run reads 180 GB, and the product logs 9 GB of events every day.

"A model that is retrained every night is always fresh." Its newest data is 7 hours old at least, and 19 hours old on average. After 3 failed nights it is 91 hours old.

"The model keeps learning from every request while it serves." In this design, serving only reads the weights. Learning happens in the next training run, from the events that the product logged, and it writes a new version. Some teams run that step every hour, but it is still a training step. In the example the model changes once a day.

1. What are a feature, a label and a training job? Name three pieces of an ML system that are not the model code.
2. A food app has 200,000 users a day who open the app 2 times each. Each open shows 10 restaurants. The training window is 28 days and a row is 100 bytes. Peak is 5 times the average, and each request scores 10 restaurants at 0.2 ms each. (a) How many rows does one training run read, and how many GB? (b) How many predictions a second at peak, and how many cores are busy?
3. A data scientist says model v43 is 5% better than v42 on past data and wants all servers to switch tonight. What do you ask for?