Skip to content

Counting, pipelines and media

Module 14 of the System Design Masterclass. Every lesson here is free to read, with no account.

  • 6 lessons
  • 2 h 38 min of reading
  1. 82Counting at scale: top-K, trending, leaderboardsHow to count when one counter is too busy or the items are too many: sharded counters, top-K with a count-min sketch, unique counts with HyperLogLog, trending, and leaderboards.24 minDepth
  2. 83Case study: Search autocompleteHow a shopping app shows 8 suggestions while you type: a trie with the best 8 phrases on every node, scores from search logs, a trending layer, safe refresh, and serving from memory.25 minDepth · Case study
  3. 84Case study: Ad-click aggregationHow to count billions of ad clicks: a fast stream for live numbers, a nightly batch for the exact bill, one-minute windows by event time, late clicks, and counting each click exactly once.27 minDepth · Case study
  4. 85Case study: Web crawlerHow a crawler fetches 100 million pages a day without hurting any site: the frontier with a queue per host, politeness and robots.txt, URL clean-up, and a Bloom filter that skips URLs it has already seen.27 minDepth · Case study
  5. 86Case study: File storage and sync (Dropbox, Drive)How a file app keeps your files on every device: chunks named by their hash, uploading only what is new, a metadata database, sync with a change journal, and conflicts kept as copies.27 minDepth · Case study
  6. 87Case study: Video streaming: on demand and live (Hotstar IPL)How to design a video app: make several sizes of each video, cut them into small files, let the phone choose the size, serve the files from a CDN, and prepare for a live match that millions join together.28 minDepth · Case study

The AI tutor is the part you sign up for

Reading is free for everyone. A free account adds the tutor, quizzes, notes and your progress.

Sign up free