Skip to content

AI systems

Module 19 of the System Design Masterclass. Every lesson here is free to read, with no account.

  • 8 lessons
  • 2 h 49 min of reading
  1. 118How an LLM looks to a system designer: tokens, speed, cost, context limitWhat a token is, why you pay for input and output tokens at different prices, why output length decides most of the wait, what the context window limits, and how to work out the cost of an AI feature for 1,000 users.23 minCore
  2. 119Serving an AI feature: streaming, timeouts, fallbacks and limitsHow to show an AI answer fast (streaming), which three time limits to set, what to do when the model service fails (a second model, then an honest error), and how to live inside the provider's limits with a waiting line.24 minCore
  3. 120RAG: letting the model read your documents (embeddings, vector search, chunking, freshness)How an app lets a model answer from your own documents: embeddings, finding the nearest chunks, the HNSW idea, chunk size and overlap, metadata filters, re-indexing, and when keyword search or the whole text in the prompt is enough.22 minCore
  4. 121Caching for AI: prompt cache and meaning cacheTwo ways to pay less for AI calls: the provider's prompt cache for the unchanging start of a prompt, and your own meaning cache that reuses an old answer for a similar question, with the rules that keep both safe.12 minDepth
  5. 122Is the AI answer good and safe? Evals, prompt injection and output checksHow to test an AI feature with a fixed set of cases scored by code, a second model and people, what prompt injection is, and four defences written in code that still work when the model is fooled.24 minDepth
  6. 123Keeping the AI bill under control: routing, budgets and per-user limitsFive ways to keep an AI bill small and safe: send easy work to a small model, cap output length, batch work that can wait, give each user a daily budget with a token bucket, and alert on spend.12 minDepth
  7. 124AI agents and tools: loops, safety and idempotent toolsWhat an AI agent is (a model that calls tools in a loop), when a fixed workflow is better, and the controls that keep an agent safe: limits on steps, time and money, least privilege, approval, idempotent tools, a log of each step, and care with tool results.24 minDepth
  8. 125Case study: AI interview coach (Tradeoff itself)A real AI feature, taken apart: the server keeps the rules and the model only proposes, the model's JSON is checked, every quote in a report is checked, the fixed prompt start is cached, left sessions are finished lazily, and the cost and fair-use limits are worked out.28 minDepth · Case study

The AI tutor is the part you sign up for

Reading is free for everyone. A free account adds the tutor, quizzes, notes and your progress.

Sign up free