How an LLM looks to a system designer: tokens, speed, cost, context limit

Many products now add a chat helper or a summary button. The interviewer asks: "What will this cost each month for 1,000 users? How long does a user wait? What happens when the document is too long?"

A weak answer says "use the best model" and gives no number. Another weak answer treats the model as free, or as a normal database call. A strong answer turns the feature into the pieces of text that the provider charges for. It counts the text that goes in and the text that comes out, and works out money and time with the arithmetic shown. It also names the size limit.

At SDE-1 level you should know that you pay for the amount of text. At SDE-2 level you should do the sums. At SDE-3 level you should see that a chat sends its history again on every turn, and that other languages cost more.

Think of a translator who charges by the word. Reading your text costs a little. Writing the translation costs more per word, because the translator writes one word at a time. A long translation takes a long time. The translator also has a desk of a fixed size: your text and the translation must fit on it together. This lesson has the same four parts: tokens, price, speed and the size limit.

The model as a remote service

A large language model (LLM) is software that reads text and writes text. Your system sends a call, which is one request. The call carries the prompt, which is all the text you send: instructions, data and the question. The model sends back the output, which is the text it writes.

An LLM remembers nothing between calls. This is like the stateless servers of lesson 26. A turn is one question and its answer. To continue a chat, your system sends the earlier turns again in the next call.

Tokens

The model does not read letters or words. It reads tokens: pieces of text from a fixed list. A common word is often one token. A rare word is split into several pieces. Numbers and punctuation are tokens too.

OpenAI's documentation says that one token is about 4 characters, or 0.75 words, of English text. This is an average for English only. So 1,000 English words are about 1,000 / 0.75 = 1,333 tokens.

Other kinds of text need more tokens for the same meaning. Hindi and other Indian scripts usually do, and so does code. How many more depends on the model. A tokenizer is the part that cuts text into tokens. One company says that older models need 4 to 8 tokens per word for Indian scripts, against about 1.4 for English. The same company says that its own newer tokenizer needs 1.4 to 2.1 tokens per word across 10 Indian languages. Do not guess. Measure your own text with the provider's token counter.

Do not mix up tokens and bytes. In lesson 14, a Hindi letter was 3 bytes. That is about storage. Tokens are about what the model reads and what you pay for.

Price: input tokens and output tokens

Providers charge per million tokens, with one price for input tokens (the prompt) and a higher price for output tokens. On Anthropic's pricing page, checked on 20 Oct 2026, the output price was 5 times the input price for every model listed. Other providers may differ. A likely reason is that the model writes output one token at a time. Some models also write hidden thinking before they answer. That thinking counts as output.

• cost of one call = input tokens x input price + output tokens x output price

Prices change often. Always read the provider's page before you quote one.

Speed: output length decides most of the wait

A call has two parts. First there is a wait before the model starts to write. Then the model writes one token after another. A simple formula for the time is:

• total time = start wait + output tokens x time per token

The model reads the whole prompt at the same time, so each input token adds much less time than an output token. A very long prompt still makes the start wait longer. So a long answer is slow. A shorter answer, or a limit on output length, is the best way to cut the wait. Answers differ in length, so use the p95 output length (lesson 4), and not the average, to choose a time limit.

The context window

The context window is the most tokens that one call can hold. The prompt and the output together must fit in it (OpenAI and Anthropic documentation). Window sizes differ by model. Anthropic's page lists 200,000 tokens for some models and 1,000,000 for others. Check the provider's page for the model you use.

If the prompt alone is too long, the call fails with an error. Anthropic's documentation also says that a longer prompt can make the model less accurate. A bigger window is a limit, not a goal.

Plan for it: count tokens before you send. The providers give a counter. Keep fixed text short. Drop the oldest turns of a chat, or cut a document, when the total gets near the limit.

The recipe for a cost estimate

1. Tokens in per call: fixed text + earlier turns + the new question.
2. Tokens out per call.
3. Calls per user per day, and the number of users.
4. Prices for input and output.
5. Time: start wait + output tokens x time per token.

SwadKart, a food app, adds a help chat about an order.

Assumptions (made up to teach with, not any provider's numbers):

• 1,000 users a day, one chat each, 5 turns per chat.
• Fixed text per call: 1,000 tokens (600 for instructions, 400 for the order data). A question is 50 tokens. An answer is 150 tokens.
• Input costs Rs 100 per million tokens. Output costs Rs 500 per million tokens. Output is 5 times the input price.
• The start wait is 0.5 seconds. The model writes 50 tokens per second, which is 0.02 seconds per token.
• The context window is 128,000 tokens.
• A page holds 3,000 characters.

Step 1: input tokens (the predict box). Each turn adds 50 + 150 = 200 tokens of history. The input of turn k is 1,000 + 200 x (k - 1) + 50.

• Turn 1: 1,050. Turn 2: 1,250. Turn 3: 1,450. Turn 4: 1,650. Turn 5: 1,850.
• Total: 1,050 + 1,250 + 1,450 + 1,650 + 1,850 = 7,250 tokens. The answer is more than 7,000.

Step 2: output tokens. 5 x 150 = 750 tokens.

Step 3: cost.

• Input: 7,250 x Rs 100 / 1,000,000 = Rs 0.725.
• Output: 750 x Rs 500 / 1,000,000 = Rs 0.375.
• One chat: Rs 1.10. One day: 1,000 x Rs 1.10 = Rs 1,100. A 30-day month: Rs 33,000.

The tokens in total are 7,250 + 750 = 8,000. Input is 7,250 / 8,000 = 91% of the tokens but only 0.725 / 1.10 = 66% of the cost. Output is 9% of the tokens and 34% of the cost.

Step 4: time. One answer takes 0.5 + 150 x 0.02 = 3.5 seconds. If we limit the answer to 100 tokens, it takes 0.5 + 100 x 0.02 = 2.5 seconds. Suppose the p95 answer is 400 tokens: 0.5 + 400 x 0.02 = 8.5 seconds. A time limit must cover that, and the average of 3.5 seconds would hide it.

Step 5: load. 1,000 chats x 5 calls = 5,000 calls a day. Assume half arrive in the busiest 4 hours (lesson 14). Peak = 0.5 x 5,000 / 14,400 = 0.17 calls per second. Each lasts 3.5 seconds, so 0.17 x 3.5 = 0.6 calls are running at any moment. The load is small.

Step 6: Hindi. Assumption: Hindi needs twice as many tokens for the same meaning. This is not a measurement. The fixed text stays in English. A question is 100 tokens and an answer is 300. History grows by 400 per turn.

• Inputs: 1,100, 1,500, 1,900, 2,300, 2,700. Total: 9,500. Output: 5 x 300 = 1,500.
• Cost: 9,500 x 100 / 1,000,000 = Rs 0.95, plus 1,500 x 500 / 1,000,000 = Rs 0.75. One chat is Rs 1.70, which is 1.70 / 1.10 = 1.55 times the English chat.
• Time per answer: 0.5 + 300 x 0.02 = 6.5 seconds, against 3.5. That is (6.5 - 3.5) / 3.5 = 86% longer.

Step 7: the context window. A chat of 200 turns has an input of 1,000 + 199 x 200 + 50 = 40,850 tokens. That is below 128,000. Now a user adds a 30-page document. It has 30 x 3,000 = 90,000 characters, which is 90,000 / 4 = 22,500 tokens. Every turn sends it again: 5 x 22,500 = 112,500 more input tokens. That costs 112,500 x Rs 100 / 1,000,000 = Rs 11.25. The chat now costs Rs 1.10 + Rs 11.25 = Rs 12.35, which is 12.35 / 1.10 = 11 times as much (rounded). A 400-page book is 400 x 3,000 / 4 = 300,000 tokens. That is more than 128,000, and the call fails.

• Paying per token solves running your own machines, and the bill starts at zero. It costs a bill that grows with prompt size and chat length. One document turned Rs 1.10 into Rs 12.35. It also costs a wait that you do not control. I would re-do the estimate whenever the prompt, the price or the model changes.
• A limit on output length solves both cost and wait (3.5 seconds became 2.5 seconds). It costs answers that may be cut short. I would set it from the p95 length of good answers.
• A big context window solves long inputs. It costs money on every call, and the model may be less accurate on very long prompts. I would send only what the question needs.

"A token is a word." It is a piece of text. In the example, 1,000 English words are about 1,333 tokens, and Hindi needs more.

"The cost is the answer the user reads." Input is the larger part. In the example, input was 91% of the tokens and 66% of the cost, because the chat sends its history again each turn.

"A bigger window means I can send everything." The window is a limit. Every token in it is paid for on every call. The output also uses space in the window.

1. How big is one English token? Why is the cost of a 5-turn chat not 5 times the cost of one call? Why does an output token cost more than an input token?
2. A support chat has 4 turns. The fixed text is 800 tokens, a question is 40 tokens and an answer is 120 tokens. Input costs Rs 100 and output costs Rs 500 per million tokens. The start wait is 0.4 seconds and each output token takes 0.025 seconds. (a) How many input tokens does the chat send? (b) What does one chat cost? (c) How long does one answer take?
3. A founder wants users to paste any document and ask questions. The window is 128,000 tokens, a page is 3,000 characters, and some users paste 300 pages. What do you say?