Is the model any good? Offline metrics, A/B tests and feedback loops

The interviewer says: "The data scientist says the new model is 3% better in the team's tests. Do you ship it? How do you know that it helps users? How do you know, six months on, that it still works?"

A weak answer says "ship it, the number is better", or "run an A/B test" and stops. A strong answer separates two questions: is the model good on past data, and is it good for the business with live users? It names a number for each, and the numbers that must not fall.

At SDE-2 level you should explain the metrics and an A/B test with a sample size. At SDE-3 level you should also explain feedback loops and slow falls in quality.

Think of a cricket batter. In the nets the batter faces balls that the coach chooses. In a match the bowlers try to get the batter out. Nets are cheap and fast; a match is slow, but it tells the truth. A model needs both.

Judging a model on past data (offline)

Held-out data is a set of past examples that training did not use. A score of the model on it is an offline metric. Use a later period, because the real model predicts the future.

The simplest score is accuracy: the share of right answers. It fails when the case you look for is rare. If 1 order in 1,000 is fake, a "model" that always says "not fake" has an accuracy of 99.9% and catches nothing.

For a yes-or-no model, count three kinds of case. A fake order that the model flags is a true positive. A real order that it flags is a false positive. A fake order that it misses is a false negative. Two scores follow:

• Precision = true positives / (true positives + false positives). Of the orders flagged, how many are really fake?
• Recall = true positives / (true positives + false negatives). Of all the fake orders, how many did we flag?

The model returns a score between 0 and 1. The threshold is the score from which we flag an order. A lower threshold flags more orders: recall goes up and precision usually goes down. A higher one does the opposite. The cost of each error picks the threshold.

You will also hear AUC. It is the chance that the model scores a random fake order above a random real one: 0.5 is a coin toss and 1.0 is perfect. It uses no single threshold, and it is less useful when positives are rare.

Why an offline gain is not a live gain

The held-out data came from the old system, so it shows only what users did with the old choices. Live features may differ (lesson 164), and the new model may be slower. So we also use an online metric: a business number measured on live users, for example the share of users who order in a week. We add guardrail metrics: numbers that must not get worse, such as p99 latency, errors, refunds and complaints.

A/B tests for a model

Lesson 127 explained the A/B test: split users by a hash, give group B the new model, and compare one business number. A model adds three things:

1. Sample size. A small difference needs many users. A rough formula for a yes-or-no rate is: users per group = 16 x p x (1 - p) / (d x d). Here p is the current rate and d is the smallest difference to see. It assumes equal groups, independent users, a 5% chance of a false alarm and an 80% chance of seeing a real difference of size d. It is a first estimate.
2. Whole weeks, and no peeking. People behave differently on weekdays and weekends, so stop only after full weeks. Fix the size and end date first. If you check the result every day and stop when it first looks good, false alarms become far more common than 5%. Watch guardrails daily; read the main number once.
3. The novelty effect. A new thing gets extra attention at first, and the gain fades. Kohavi and others suggest two weeks to look for this. They say it is uncommon, but they saw it in recommendations. When LinkedIn changed its "People You May Know" list, clicks rose at first, and the effect died down.

In shadow mode the new model gets a copy of each live request, but users see only the old model's answer. You compare predictions and speed. No user is at risk, but you learn nothing about user reaction. A canary (lesson 127) lets a small share of users see the new model while you watch the guardrails. A common order is shadow, canary, A/B test. The first two check safety; only an experiment shows value.

The feedback loop

The model decides what users see, and users can tap only what they saw. The next training data is made of those taps, so the model learns from its own choices. This is a feedback loop. Two biases follow. Position bias: an item near the top gets more taps for the same appeal. Popularity bias: a popular item gets more views, so more taps, so a better rank. Two ways out:

• Log what was shown, with its position, not only what was tapped. Then the data shows that an item was not tapped because nobody saw it.
• Show a small random share. For a few requests, order the list at random.

Drift

Drift means that the world changes and the model slowly gets worse. Watch the inputs (the spread of each feature, missing values) and the outcomes (precision on late labels, the online metric). Inputs usually warn first, because labels come late.

SwadKart (the food app of lesson 163) checks cash-on-delivery orders for fakes, and ranks restaurants.

Assumptions (made up to teach with): 1,000,000 orders a month, 1,000 fake. A missed fake costs Rs 400. A real order wrongly blocked costs Rs 50.

Step 1: the useless model (the predict box). It is right on 1,000,000 - 1,000 = 999,000 orders: accuracy 999,000 / 1,000,000 = 99.9%. It catches 0 fakes, and the loss is 1,000 x Rs 400 = Rs 400,000.

Step 2: two thresholds.

At 0.5 the model misses 1,000 - 800 = 200 fakes. Precision is 800 / 1,200 = 66.7%, recall is 800 / 1,000 = 80%, and accuracy is (800 + 998,600) / 1,000,000 = 99.94%. At 0.3 it misses 100. Precision is 900 / 2,500 = 36%, recall is 900 / 1,000 = 90%, and accuracy is 99.83%. Accuracy ranks the useless model (99.9%) above the 0.3 model.

Step 3: cost picks the threshold. At 0.5: 200 x Rs 400 + 400 x Rs 50 = Rs 80,000 + Rs 20,000 = Rs 100,000. At 0.3: 100 x Rs 400 + 1,600 x Rs 50 = Rs 40,000 + Rs 80,000 = Rs 120,000. Use 0.5. If a miss cost Rs 1,000, 0.3 would win.

Step 4: sample size. Online metric: the share of users who order in the test week. Assumption: it is 40% now, and we want to see 1 point (d = 0.01). Users per group = 16 x 0.4 x 0.6 / (0.01 x 0.01) = 3.84 / 0.0001 = 38,400. Assumption: 1,200,000 users open the app in a week. With 5% in each group, each group has 60,000 users, which is enough. A canary of 1% has 12,000 users: too few to see the gain, enough to check guardrails.

Step 5: whole weeks and novelty. Assumption: the gain is 0.5 points on weekdays and 2.5 on weekends. A test from Monday to Thursday shows 0.5, but the whole week gives (5 x 0.5 + 2 x 2.5) / 7 = 1.07. Assumption: the gain fades by week: 1.5, 1.0, then 0.6 points. A one-week test shows 1.5, which is 1.5 / 0.6 = 2.5 times the later level. Run two whole weeks or more, and compare week 1 with week 2.

Step 6: the feedback loop. Assumption: a user looks at position 1 with chance 60%, and at position 10 with chance 4%. Restaurants A (new) and B are equally good: a user who looks taps with chance 50%. B is at position 1, A at position 10. Tap rates: B 0.6 x 0.5 = 30%, A 0.04 x 0.5 = 2%. The data says B is 15 times better, so the next model ranks A lower still. Now show a random list in 5% of opens: 1,500,000 x 5% = 75,000 opens a day. A user can order from 200 restaurants, so A is at position 1 in 1 of 200 of them: 375 times a day. Assumption: a random list gives 85 orders for every 1,000 opens, against 100. Cost: 0.95 x 100 + 0.05 x 85 = 99.25, a 0.75% loss.

Step 7: drift. Assumption: a fake label is confirmed after 14 days, and weekly precision falls from 66.7% to 64%, 61% and 57%. An alert at 60% fires in week 4, 14 days late. An input alert is faster. Say the share of orders with a missing distance goes from 0.5% to 8%, against a limit of 2% (an assumption). You hear on day 1.

Real-world note (Netflix, 2026). Netflix wrote that the share of viewing from recommended titles is the wrong way to value recommendations, because titles are recommended to people who already want them. The team built a model to separate the two. Against its current system, random recommendations cut engagement by 16% and popularity ranking by 12% (the company's figures). Source: https://netflixtechblog.com/measuring-the-impact-of-personalized-recommendations-4c26be3a4d96 (opens in a new tab) (read from the company's blog feed on 26 Oct 2026, where it is dated 14 Oct 2026).

Offline metrics solve fast, cheap comparison of many models, but link weakly to the business. An A/B test solves that, and costs weeks and 38,400 or more users a group. I would use offline metrics to pick what to test, and an A/B test to decide.

A small random share solves a feedback loop at a small cost (0.75% here).

"The model has 99.9% accuracy, so it is good." A model that never flags anything has 99.9% too.

"It won the offline test, so ship it to everyone." The test data came from the old system. Use shadow mode, a canary and an A/B test.

"Taps show what users like." Two equally good restaurants had tap rates of 30% and 2%, only because of position.

1. What are precision and recall? Why can a useless model have high accuracy? What is a feedback loop?
2. Of 2,000,000 orders, 4,000 are fake. A model flags 5,000 orders, and 3,000 of them are fake. (a) What are the precision and recall of the model? (b) A miss costs Rs 300 and a wrong block costs Rs 40. What does the model cost, and what does no model cost? (c) An A/B test has a current rate of 20% and must see a difference of 1 point. How many users does each group need?
3. An A/B test of a new ranking model ran from Monday to Thursday and showed +2 points of orders. The team wants to ship to everyone. What do you say?