RAG: letting the model read your documents (embeddings, vector search, chunking, freshness)

A company says: "Our help centre has hundreds of pages. Add a bot that answers from them." The interviewer asks: "How does the bot find the right page? What happens when a page changes? How big is the bill?"

A weak answer says "put all the documents in a special search tool" and gives no number. A strong answer first asks how big the documents are, how often they change, and who may read them. It then picks the simplest thing that works: the whole text in the prompt, a plain search for words, or a search by meaning. It says how the stored copy stays fresh when a page changes.

At SDE-1 level you should explain the steps. At SDE-2 level you should size the pieces and the cost. At SDE-3 level you should handle changes, permissions and a growing library.

Think of an open-book exam. The student cannot carry the whole library to the desk. So a helper finds the 4 pages that are closest to the question, and the student answers from those pages.

Retrieval-augmented generation (RAG) works the same way. Your system retrieves the pieces of your documents that are closest to the question. It adds them to the prompt, and the model writes the answer from them. We need it for two reasons from lesson 118. The model knows nothing about your private or new documents. And a whole library does not fit in the context window, and every token in it costs money on every call.

The steps are:

1. Once, before any question: cut each document into small pieces, called chunks. Turn each chunk into a list of numbers that stands for its meaning. Store the list with the chunk text. These stored lists and texts are the index.
2. For each question: turn the question into a list of numbers in the same way. Find the stored chunks whose lists are nearest. Build a prompt of instructions, those chunks and the question, and call the model.
3. Tell the model to answer only from the chunks, and to say "I do not know" when the answer is not there.

The next parts explain each piece.

Embeddings

An embedding is a list of numbers that stands for the meaning of a text. Texts with a similar meaning get lists that are near each other. "How do I get my money back?" and "Refund rules" share no word, but their lists are near. A special model, the embedding model, makes the lists. It is a call that you pay for by the token, like lesson 118. The list often has several hundred to a few thousand numbers. One provider offers 768, 1,536 or 3,072. Each model also has an input limit, and text past it may be cut.

Near means a small distance between two lists. With 2 numbers each, the distance is the square root of the sum of the squared differences. Real lists work the same way with more numbers. Tools can measure distance in other ways, and the idea is the same.

Vector search

A list of numbers is also called a vector. Vector search, a search by meaning, finds the stored vectors nearest to the question. The simplest way is exact search: compare the question with every stored vector. The work is chunks x numbers per vector. It is fine for a small library and too slow for a big one.

Approximate nearest neighbour search (ANN) finds vectors that are very near, and it may miss one. The share of the true nearest vectors that it finds is its recall. Hierarchical Navigable Small World (HNSW) is a common ANN method. Think of a road map. You take the highway first, then a main road, then a street. HNSW keeps several layers of links between vectors. The top layer has few vectors and long links. You start at the top, move to the nearest vector, drop one layer, and repeat. The bottom layer has all vectors. The authors say that the work grows slowly with the size of the library. The costs are memory for the links, a slow build, and recall below 100%. Use ANN only when exact search is too slow. One example tool is pgvector, an add-on for Postgres. It does exact search by default and offers HNSW.

Chunking

A whole document in one embedding gives a blurry list, and it may pass the input limit. A very small piece loses its context. So you cut the text into chunks.

• Chunk size: the tokens in one chunk. A common start is 400 to 512 tokens. There is no best size, so test on your own questions.
• Overlap: the tokens that two neighbouring chunks share, so that a sentence on the edge is not cut in two. A common start is 10 to 20 percent.
• chunks per document = (length - overlap) / (size - overlap), rounded up

Small chunks are precise and may lack context. Large chunks hold more, but the vector is blurry and each call carries more tokens. Cut at headings and paragraphs when you can.

Metadata filters

Metadata is facts about a chunk, stored beside it: the document, the language, the audience, the date. A metadata filter keeps only the chunks that match, for example audience = customer. Use a filter for permissions too. A chunk that a user may not read must never reach the prompt. Check that your tool filters during the search. Some tools filter after it and can return fewer chunks than you asked for.

Keeping the index fresh

A chunk that is out of date is a stale entry (lesson 34). Store a hash of each document, which is a short code that changes when the text changes. When the hash changes, delete all chunks of that document and make new ones. If you keep the old chunks, the model sees both the old and the new text.

When you change the embedding model, the old vectors are useless. Two models give lists that cannot be compared. So make new vectors for every chunk, in a new index next to the old one. Switch when it is ready, then delete the old one. This is the new-key idea of lesson 34. The question must use the new model too. This is common practice, and one provider documents it. We found no general standard.

When you do not need RAG

• The whole text in the prompt. If all the text fits in the window with room to spare, and the cost per call is small, skip RAG. Tradeoff's own tutor puts the whole lesson in the prompt, up to 48,000 characters, which is about 12,000 tokens.
• Plain keyword search. It finds documents that contain the words you typed. It is cheap and exact. It is better for codes and names, such as an error code. It fails when the user uses other words.

SwadKart has a help library of 400 documents.

Assumptions (made up to teach with):

• Each document has 1,500 tokens. Chunks are 500 tokens with 50 tokens of overlap.
• An embedding has 1,024 numbers of 4 bytes each. A server does 10 billion multiply-and-add steps a second.
• Input costs Rs 100 per million tokens (lesson 118). Embedding costs Rs 10 per million tokens. The context window is 128,000 tokens.
• A prompt has 800 tokens of fixed text, 4 chunks and a question of 50 tokens. Embedding the question takes 50 ms, and the search takes 5 ms.

Step 1: the whole library in the prompt? 400 x 1,500 = 600,000 tokens. That is above 128,000, so it does not fit. If it fitted, one question would cost 600,000 x Rs 100 / 1,000,000 = Rs 60.

Step 2: chunks. The step is 500 - 50 = 450. Chunks per document: (1,500 - 50) / 450 = 3.2, rounded up to 4. They start at tokens 0, 450, 900 and 1,350. The library has 400 x 4 = 1,600 chunks. Without overlap it would be 3 per document.

Step 3: exact search. The vectors take 1,600 x 1,024 x 4 = 6,553,600 bytes, about 6.6 MB. One search does 1,600 x 1,024 = 1,638,400 steps, which is 1,638,400 / 10,000,000,000 = 0.16 ms. Exact search is enough. HNSW would add work and no benefit.

Step 4: meaning against words (the predict box). Take 2 numbers as a toy. The refund chunk is (0.9, 0.1). The opening-hours chunk is (0.2, 0.8). The cancel chunk is (0.5, 0.5). The question is (0.8, 0.2). Distances: to refund, 0.14; to cancel, 0.42; to hours, 0.85. The refund chunk is nearest. A keyword search finds nothing, because "money back" shares no word with the page. A search by meaning finds it.

Step 5: the prompt. 800 + 4 x 500 + 50 = 2,850 tokens. Cost: 2,850 x Rs 100 / 1,000,000 = Rs 0.285. That is 60 / 0.285 = 210 times less than Step 1. The extra wait is 50 + 5 = 55 ms, about 11% of a 0.5 second start.

Step 6: a filter. 100 documents are for restaurants. A customer search uses audience = customer, so it covers 300 x 4 = 1,200 chunks. A restaurant page can never reach a customer.

Step 7: a change. The refund page changes from 7 days to 5 days. The hash changes. Delete its 4 chunks and embed 1,650 tokens (the chunks overlap): 1,650 x Rs 10 / 1,000,000 = Rs 0.0165. If we forgot to delete the old chunks, the answer could say 7 days or 5. A new embedding model needs all 660,000 tokens again: 660,000 x Rs 10 / 1,000,000 = Rs 6.6.

Step 8: growth. Say the library becomes 10,000,000 chunks. Vectors take 10,000,000 x 4,096 = 40.96 GB. Exact search does 10,000,000 x 1,024 = 10.24 billion steps, which is 1.02 seconds for one question. At 50 questions a second that is 50 x 1.024 = 51.2 seconds of work each second, so 52 servers. Assume that HNSW compares the question with 2,000 vectors: 2,000 x 1,024 = 2,048,000 steps, 5,000 times less. Now ANN is worth its costs. Re-embedding 10,000,000 x 500 = 5 billion tokens costs Rs 50,000.

RAG solves a library that is too big, too new or too private for the prompt (Rs 0.285 against Rs 60). It costs a pipeline to build and keep fresh, an extra 55 ms, and a risk that the search misses the right chunk. Then the model has nothing to answer from. I would drop RAG when the whole text fits and is cheap. I would use keyword search for codes, and add ANN only when exact search is too slow.

"RAG teaches the model our documents." Nothing is learned. The chunks are pasted into each prompt, and the model forgets them after the call (lesson 118).

"We always need a vector database and an ANN index." At 1,600 chunks, exact search takes 0.16 ms. ANN pays off at millions of chunks.

"Bigger chunks are safer because they hold more context." A large chunk makes a blurry vector. Four chunks of 2,000 tokens add 8,000 tokens to each prompt, not 2,000. That is 4 times the input cost.

1. What is an embedding? What does "near" mean? Why do we cut documents into chunks?
2. A library has 200 documents of 2,400 tokens. Chunks are 600 tokens with 60 tokens of overlap, and an embedding has 768 numbers of 4 bytes. (a) How many chunks per document, and in total? (b) How many bytes do the vectors take? (c) Does the whole library fit in a window of 128,000 tokens?
3. A team switches to a newer embedding model and embeds only the new documents. Search quality drops. Why, and what do you do?