Training-serving skew and feature stores

The interviewer says: "The team trained a model and it looked good in tests. After release, it is clearly worse. What went wrong? How do you stop it from happening again?"

A weak answer says "get more data". A strong answer asks how each input number is made when the model learns and when it runs. It names the causes: two code paths, different time windows, and numbers that did not exist yet. It then names one place that keeps one definition of each number.

At SDE-2 level you should explain the cause and the fix. At SDE-3 level you should also say what each store holds, how fresh each number must be, and what freshness costs.

Think of a student who practises sums with a ruler in centimetres. In the exam the ruler is in inches. The skill is the same, but every number looks different. The model is the student. The ruler is the code that makes its input numbers.

Quick recap of lesson 163: a feature is one input number about a case, for example "orders this user placed in the last 7 days". A label is the answer to predict. An example is the features and the label of one past case. Serving means answering live requests with the model.

Training-serving skew means that a feature is made one way when the model learns and another way when it serves. The model is then worse live than in tests. Common causes:

• Two code paths. Training features come from SQL in the warehouse. Serving features come from application code. They can differ in rounding, time zones and the handling of a missing value.
• A different window. "Last 7 days" can mean the last 168 hours, or the last 7 calendar dates. An order at the edge is counted by one path and not by the other.
• A different age. Training uses a value that is exact to the second. Serving reads a value that a job refreshed this morning.
• The future. A training row uses a fact that was not known at the time of the example. This is leakage.

Point in time

A training row is a case from the past. Its as-of time is the moment when the prediction would have been made. A training set is point-in-time correct when each feature has only facts that production could have used at the as-of time. A fact counts from the time when production could use it, not from the time of the event. Feast, an open-source feature store, says its point-in-time joins stop future values from leaking into training.

The feature store

A feature store keeps one definition of each feature and two copies of its values.

1. The offline store keeps history: each value with its time. It is the warehouse of lesson 155. Training reads it in bulk.
2. The online store keeps the latest value for each key. It is a key-value store (lesson 33). Serving reads it by key in a few milliseconds.
3. One shared definition fills both, often through one job. It can be a daily batch job (lesson 155) or a stream job (lesson 154).

For training, the store joins each example with the value that the offline store had as of its time. For serving, it answers a lookup by key. This is the precompute idea of the trade-off page on precompute against compute on read.

Batch features and streaming features

A batch feature is made by a batch job on a schedule, for example every night. It is cheap, and its value is as old as the schedule. A streaming feature is updated by a stream processor as events arrive, so it is fresh in seconds. It costs a job that never stops, state to save, and late events (lesson 154). Choose by how fast the feature changes and how much an old value hurts. Serving a request live does not make every feature a streaming feature.

Log what serving used, and check for skew

The most direct protection is to log the features that the server used for each prediction and train on those rows. Training then sees what serving saw. To check for skew, take a sample of logged rows, compute the same features offline, and count the rows that differ for each feature. Alert when the share passes a limit.

SwadKart (the food app of lesson 163) predicts if a user orders from a shown restaurant. One feature is orders_7d.

Assumptions (made up to teach with):

• The prediction is made on Friday at 8 pm. The user ordered on Friday 3 pm (5 hours before), Thursday 2 pm (30 hours), Monday 4 pm (100 hours) and last Saturday 4 am (160 hours). The user ordered again on Friday 10 pm (2 hours after).
• The training SQL counts orders from 7 calendar dates back to the end of the same day.
• The online store is refreshed once a day at 2 am, with a window of 168 hours that ends at 2 am.

Step 1: three values for one number.

The answer to the predict box is 4. The training SQL counted an order from after the prediction, close to the label. So in training a higher number looks like a sign of an order, partly because the number holds the answer. Live, the model gets 3, with no leaked order: it learned from numbers that were sometimes too high.

Step 2: three fixes.

1. Write the feature once, as the orders in the 168 hours before t, and run that one definition for both stores.
2. Let the offline store record the values that the online store held, with their times. A point-in-time join for Friday 8 pm then returns the 2 am value, which is 3, the same as serving. The model learns from what it will see. If serving can only hold the 2 am value, training must hold it too, even though the exact value is 4.
3. Make the feature fresher, with a stream job, so that serving also returns 4.

Step 3: sizes and lookups. Assumption: 2,000,000 users and 300 bytes of features for each. The online store takes 2,000,000 x 300 bytes = 600 MB. If the offline store keeps a full copy every day, that is 600 MB x 365 = 219 GB a year. This is an upper bound: storing only changed values takes less. Each request reads 1 user and 200 restaurants, which is 201 keys, in one batch call. At the peak of 69.6 requests a second (lesson 163), that is 69.6 x 201 = 13,990 keys a second. Assumption: a call takes 8 ms at p99, which fits the 20 ms that lesson 163 gave to features and network.

Step 4: batch against stream. Another feature is "orders at this restaurant in the last 30 minutes", which shows if the kitchen is busy. A daily batch is useless, because the value changes every minute. Assumptions: a stream job needs 3 workers all day. A nightly batch job needs 10 workers for 1 hour. A worker costs Rs 20 an hour. Stream: 3 x 24 = 72 worker-hours = Rs 1,440 a day. Batch: 10 x 1 = 10 worker-hours = Rs 200 a day. The stream costs 1,440 / 200 = 7.2 times more. It is worth it only if the fresh number changes the prediction.

Step 5: logging and checking. Logging 40 features of 4 bytes for every shown restaurant takes 30,000,000 x 40 x 4 bytes = 4.8 GB a day. A daily check takes 10,000 logged rows and recomputes them offline. Suppose 1,800 differ: 1,800 / 10,000 = 18%, against an alert limit of 1% (100 rows, an assumption). If 1,750 of them are in orders_7d, you know where to look.

Real-world note (Uber, 5 Sep 2017). Uber described its ML platform, Michelangelo. The same feature expressions run when a model trains and when it predicts. Features that need history are computed ahead of time, in batch or from a stream, and kept in a key-value store. Source: https://www.uber.com/blog/michelangelo-machine-learning-platform/ (opens in a new tab) (checked 26 Oct 2026).

Real-world note (Razorpay, 26 Jun 2026). Razorpay described customer segments that scheduled batch jobs compute and write as files in object storage. The files are loaded into a key-value store for lookups by key. The post says a new serving store can be filled from the files without computing again. These are marketing segments, not model features, but the pattern is similar. Source: https://engineering.razorpay.com/turning-scattered-data-into-queryable-segments-at-scale-how-razorpay-built-its-customer-data-3937c4b012de (opens in a new tab) (text read from the company's blog feed, 26 Oct 2026).

A feature store solves skew and repeated work, because each feature has one definition and one job. It costs one more system to run, two stores to keep in step, and a rule that all teams use it. I would use a shared code library and serving logs instead when there is one model, few features and one team.

Fresh features solve stale inputs, and cost a stream job (7.2 times the nightly bill here). I would keep a feature in batch when an old value does not change the prediction.

"A model that is good in tests is good live." One user had 5 in training, 3 online and 4 in truth.

"A feature store is a fast database." The fast lookup is one half. The other half is one definition, history with times, and the point-in-time join.

"A model that answers live requests needs streaming features." Each feature can have its own refresh time. A 7-day order count can be a day old. The kitchen-busy count cannot.

1. What is training-serving skew? What is leakage? Which store does training read, and which does serving read?
2. A user ordered 3, 20, 50 and 150 hours before the prediction, and 1 hour after it. The feature is orders in the last 168 hours. (a) What is the exact value? (b) What does a calendar-date SQL that counts to the end of the day give? (c) The online store was refreshed 6 hours before the prediction, with a 168-hour window. What does it give? (d) 500,000 users have 200 bytes each. How big is the online store, and how big are 30 daily copies?
3. A model scored well in tests, but live orders did not rise. What do you do?