Caching for AI: prompt cache and meaning cache

The help chat of lesson 118 pays for every call. The interviewer asks: "How do you cut the bill without making answers worse?" A weak answer says "cache the answers". A strong answer names two caches: one for the repeated start of the prompt, and one that reuses old answers, which is safe only for some questions.

Prompt cache. A print shop keeps the first pages of a booklet already set, and charges less for them.

A prompt often starts with the same text on every call. With a prompt cache, the provider stores that start and reads it at a lower price on the next call. On Anthropic's page (checked 20 Oct 2026), a cache read costs 0.1 times the normal input price on most models. A write (the first call) costs 1.25 times. It works only when:

• the start is byte-identical: exactly the same characters. One changed character breaks it from that point on. A time stamp in the first line breaks it every second;
• the start is long enough. The minimum is 512 to 4,096 tokens, by model. A shorter prompt is not cached, and you get no error;
• it is used again soon. The default lifetime is 5 minutes, and each use renews it.

So put fixed text first and changing text last.

Meaning cache. A clerk writes the answer to "When do you open?" on a card. She shows the card to anyone who asks "What are your opening hours?"

An embedding is a list of numbers. Similar questions get lists that are near each other. A meaning cache stores each question's embedding with its answer. For a new question, you find the nearest stored one. If the similarity is above a threshold, you return the stored answer and make no model call. If not, you call the model and store the new pair.

A hit can be wrong: two questions are near, but the answers differ. A higher threshold gives fewer wrong hits and fewer hits. It is safe for a public FAQ, where every user gets the same slow-changing answer. It is unsafe for a personal answer ("Where is my order?") or a fast-changing one ("Is the shop open now?"). Use a TTL (lesson 34) to keep entries fresh.

SwadKart's help chat makes 5,000 calls a day. Input costs Rs 100 per million tokens.

Assumptions (made up to teach with): the fixed text is 3,000 tokens. The chat history and question add 450 tokens. An answer is 150 tokens, and output costs Rs 500 per million tokens. The cache is empty 20 times a day, after gaps over 5 minutes. 2,000 calls a day are public FAQ questions.

Step 1: the prompt cache. The fixed part costs 3,000 x 100 / 1,000,000 = Rs 0.30. A read costs Rs 0.03 and a write Rs 0.375. Two uses cost 0.375 + 0.03 = Rs 0.405, against 0.30 x 2 = Rs 0.60. So yes. One use costs Rs 0.375, more than Rs 0.30. A day: 20 writes + 4,980 reads = 7.50 + 149.40 = Rs 156.90, against 5,000 x 0.30 = Rs 1,500. That is about 90% less. With the cache on, a call costs 0.03 (fixed read) + 0.045 (other input) + 0.075 (output) = Rs 0.15.

Step 2: the threshold. Assumption: at 0.90, the FAQ hit rate is 95% and 3% of hits are wrong. At 0.95 it is 80% and 0.5% are wrong.

• At 0.90: 1,900 hits x 0.15 = Rs 285 saved, and 1,900 x 3% = 57 wrong answers.
• At 0.95: 1,600 hits x 0.15 = Rs 240 saved, and 1,600 x 0.5% = 8 wrong answers.

The higher threshold gives up Rs 45 and avoids 49 wrong answers.

A prompt cache solves the cost of a long fixed start. It costs a rigid prompt order. A meaning cache solves repeated questions. It costs wrong hits and an embedding for every question. I would cache only public questions, and raise the threshold when a wrong answer costs more than money.

"Any prompt is cached automatically." A short prompt, a changed first line or a gap over 5 minutes means no hit.

"A similar question has the same answer." "Refund in 7 days" and "refund in 70 days" are near in meaning, but the answers differ.

1. What makes a prompt cache hit? Why is a meaning cache unsafe for "Where is my order?"
2. Fixed text is 2,000 tokens at Rs 100 per million. What do they cost (a) with no cache, (b) as a write, (c) as a read? (d) From which use is the cache cheaper?
3. A FAQ question arrives every 100 seconds. The TTL is 600 seconds. Use lesson 31. What is the hit rate?