Search ranking basics: retrieve many, then rank a few

The interviewer says: "Your food app has 2 million dishes. A user types 'paneer butter' and 65,000 dishes match. How do you choose the 10 to show, in under 200 ms? How do you know the new order is better?"

A weak answer says "use Elasticsearch and sort by score". A strong answer uses two stages and names the features of the second. It gives a time budget for each stage, and says what the user sees when the second is slow.

At SDE-2 level you should explain the two stages and the features. At SDE-3 level you should also explain the labels, the metrics and the time budget.

Think of a library. A clerk first pulls 500 books that match your words. The clerk is fast and reads only titles. Then a slow expert reads the 500 covers and puts the best 10 on your desk. The expert cannot read all 2 million books.

• Retrieval is the first stage. It finds a few hundred or a few thousand results with a cheap method: the inverted index and the BM25 score of lesson 112. Each result is a candidate: a result that may be shown.
• Ranking is the second stage. A model, called the ranker, scores each candidate with more features and sorts the candidates by score. A feature is one input number (lesson 164).

Why two stages? Scoring one candidate takes time, and the request has a fixed budget (lesson 166). A heavy model cannot score 65,000 dishes in it. Elasticsearch has the same shape: its rescore feature reorders only the top results of the first stage.

Query understanding

Before retrieval, the system cleans the query. It fixes typos ("panner" becomes "paneer"). It adds synonyms ("aata" and "atta"). It turns words into filters ("under 300" becomes a price filter), which keep or remove dishes and do not change the score.

Features

Each case is a user, a query and a candidate. The ranker sees numbers from four groups.

As in lesson 164, use one definition of each feature for training and serving, and log the values that serving used.

Learning to rank

Learning to rank means that the ranker is a model learned from past examples, not a formula written by hand. One example is one search, one candidate, its features and a label: the answer we want the model to predict. Labels come from two places.

1. Logs of clicks and orders. An order is a strong yes. A click is a weak yes. A result that was shown high on the page and ignored is a weak no. This data is plentiful and cheap.
2. A judgement list. People rate pairs of a query and a result, for example from 0 (bad) to 3 (perfect). It is small and costs money, but it is clean. Use it to check the model and compute metrics.

There are three common ways to train the model.

• Pointwise. The model scores each candidate alone and tries to match its label, for example the chance of an order. It is simple, but it does not compare two candidates.
• Pairwise. The model sees two candidates of the same search, one better than the other, and learns to score the better one higher. It cares about order, which is the goal.
• Listwise. The model sees the whole list of one search and learns to raise a list score such as NDCG (defined below). It fits the goal best, and it is more work to build.

Click labels have a problem, position bias (lesson 165): a result at the top gets more clicks even if it is not better. Log the position of every result. Train with the position known and leave it out when serving, and show a small random share of searches in a random order.

Offline metrics

Say a judgement list gives each result a grade from 0 to 3. Grade 1 or more means relevant.

Precision at k is the share of the first k results that are relevant. It ignores the order inside the k.

NDCG (normalised discounted cumulative gain) rewards good results near the top. To compute it:

1. For each position i, divide the grade by log2(i + 1). The divisors for positions 1 to 5 are 1, 1.585, 2, 2.322 and 2.585.
2. Add the numbers. This is the DCG.
3. Compute the DCG of the best possible order, with the highest grades first. This is the ideal DCG.
4. NDCG = DCG / ideal DCG. It is between 0 and 1, and 1 means a perfect order.

Online metrics

An offline gain does not promise a live gain (lesson 165). Test live.

• Click-through rate (CTR): here, the share of searches with at least one click.
• Long click: a click after which the user does not come back to the results within a set time. A short click often means a wrong result.
• Also orders per search. Guardrails: p99 latency and errors.

Re-ranking for rules

After the ranker, a short step applies rules that change quickly. It removes sold-out dishes, puts one paid ad at position 3, and allows at most 2 dishes from one restaurant in the first 10. A rule change then needs no new model.

SwadKart searches its 2,000,000 dishes (lesson 112). A user types "paneer butter".

Assumptions (made up to teach with):

• "Paneer" is in 40,000 dishes and "butter" in 30,000 (lesson 112). Assumption: 5,000 have both. A dish with either word matches.
• The ranker needs 0.05 ms per dish, and one search uses one core (lesson 166). Load is 200 searches a second (lesson 112).
• Total p99 target: 200 ms.

Step 1: why two stages. Matches: 40,000 + 30,000 - 5,000 = 65,000. Scoring all of them takes 65,000 x 0.05 ms = 3,250 ms. The first 500 take 500 x 0.05 ms = 25 ms, which is 3,250 / 25 = 130 times less. At 200 searches a second, scoring all matches is 200 x 65,000 x 0.05 ms = 650,000 ms of work a second, so 650 cores. For 500 candidates it is 5 cores.

A funnel with four layers. The library has 2,000,000 dishes. The search words match 65,000 of them. Retrieval, the cheap first stage, keeps the best 500 as candidates. Ranking, the slow second stage, scores those 500 and keeps 10 to show.

Step 2: how deep to retrieve (the predict box). In 25 ms the ranker scores 25 / 0.05 = 500 dishes. Assumption: on old searches we measured how often the ranker's top 10 dishes are inside the first k results of BM25. Call it the retrieval recall.

1,000 candidates need 50 ms, over the 25 ms. We choose 500. The cost is the 7% of good dishes that do not reach the ranker, and no ranker can show a dish that retrieval did not find. Add a second source, such as the most popular dishes.

Step 3: features and a score. Take features between 0 and 1 and made-up weights: text 0.4, popularity 0.3, freshness 0.1, personal 0.2.

BM25 orders them A, B, C. The ranker orders them B, A, C. A learned model finds the weights from data.

Step 4: NDCG. For one query, people grade five dishes. The old order has grades 2, 0, 3, 1, 0. The new order has grades 3, 1, 2, 0, 0. The ideal order is 3, 2, 1, 0, 0.

• Ideal DCG = 3 / 1 + 2 / 1.585 + 1 / 2 + 0 + 0 = 3 + 1.262 + 0.5 = 4.762.
• Old DCG = 2 / 1 + 0 + 3 / 2 + 1 / 2.322 + 0 = 2 + 1.5 + 0.431 = 3.931. NDCG = 3.931 / 4.762 = 0.825.
• New DCG = 3 + 1 / 1.585 + 2 / 2 = 3 + 0.631 + 1 = 4.631. NDCG = 4.631 / 4.762 = 0.973.

Precision at 3 is 2 / 3 = 0.67 for the old order and 1.0 for the new order. The order 1, 2, 3, 0, 0 also has precision 1.0, but its NDCG is (1 + 1.262 + 1.5) / 4.762 = 0.79. Precision cannot see the order, NDCG can.

Step 5: online test. Assumption: 100,000 searches for each model. Model A: 40,000 searches with a click (CTR 40%) and 28,000 with a long click. Model B: 44,000 (44%) and 26,400 long clicks. B wins on CTR and loses on long clicks (26.4% against 28%). For a 1 point difference, the sample size of lesson 165 is 16 x 0.28 x 0.72 / (0.01 x 0.01) = 32,256 for each group. So 100,000 is enough. Do not ship B on this evidence. Look at orders per search first.

Step 6: time budget. The p99 times are:

• network and gateway 10 ms, query understanding 5 ms, retrieval 30 ms
• feature lookups (1 user key and 500 dish keys) 15 ms, ranker 25 ms
• re-ranking 5 ms, loading dish cards 20 ms, answer 10 ms

The sum is 10 + 5 + 30 + 15 + 25 + 5 + 20 + 10 = 120 ms, inside 200 ms. Set a timeout of 70 ms on the features and the ranker together (normal use is 15 + 25 = 40 ms). If it fires, show the BM25 order (lesson 82). The worst case is 10 + 5 + 30 + 70 + 5 + 20 + 10 = 150 ms, still inside 200 ms.

Real-world note (Airbnb, 22 Oct 2018). Airbnb described three ages of search ranking: a hand-made scoring function, gradient boosted trees, then neural networks. Its main offline metric was NDCG, with the booked listing graded 1 and the others 0. Source: https://arxiv.org/abs/1810.09591 (opens in a new tab).

Real-world note (Flipkart, Nov 2025). Flipkart wrote that search models need labels for query and product pairs (relevant, partly relevant, not relevant). People wrote them at high cost. The team built a language-model labeller that, by the post, works at far larger scale. Source: https://blog.flipkart.tech/llms-for-relevance-automating-high-quality-product-relevance-labeling-in-flipkart-search-ddd5ca50b584 (opens in a new tab) (the post names Nov 2025; the feed lists 3 Aug 2026).

Two stages solve a model that is too slow for all matches: 25 ms for 500 dishes, not 3,250 ms for 65,000. They cost a recall gap (7% of good dishes are lost at 500) and two systems to tune. I would add a second source when the gap hurts.

A learned ranker solves a hand-made formula that nobody can tune for many signals. It costs labels, a training pipeline and click biases. I would start with BM25 plus a few boosts, and train a ranker when the logs and the team exist.

"The ranker should score every matching result." That is 3,250 ms for one search here.

"The model with the higher click rate is better." B had 44% clicks and fewer long clicks. Clicks include wrong clicks.

"The top result gets the most clicks, so it is the best." Position alone raises clicks (lesson 165). Log the position.

1. What are retrieval and ranking? Why two stages? Name the four feature groups.
2. A judgement list grades four results in this order: 1, 3, 0, 2. The best order is 3, 2, 1, 0. (a) What is the NDCG of the shown order? (b) What is precision at 2? (c) A ranker needs 0.04 ms per candidate and has 30 ms. How many candidates can it score?
3. A new ranker raises the click-through rate from 38% to 41%, but orders per search do not move. What do you do?