Big systems in depth
Module 23 of the System Design Masterclass. Every lesson here is free to read. No account needed.
- 9 lessons
- 3 h 18 min of reading
- 154Stream processing: windows, late data, exactly-onceHow a program that reads a log all the time keeps state: window kinds, the watermark and late events, checkpoints, exactly-once to the outside world, joining two streams, and replaying the log to fix a bug.23 minDepth
- 155Batch processing and data warehouses: files, columns and star schemasHow big reports are built: batch jobs that split work and run a failed piece again, columnar files, a warehouse kept apart from the main database, star schemas, loading, date partitions and safe reruns.23 minDepth
- 156How a log-based queue stores data: segments, offsets, retention, compaction, replicationWhat a log-based queue does on disk: segment files, why appending is fast, how a read finds an offset, retention, compaction with tombstones, and how copies and the acknowledgement setting decide what a crash can lose.23 minDepth
- 157Collaborative editing: operational transformation or CRDTHow many people edit one text at once: operational transformation with one server, CRDTs for text, what each costs, and the parts around them (one server for each document, cursors, undo, history). A third option settles conflicts for each property.23 minSenior
- 158Distributed transactions in depth: two-phase commit, sagas, outboxOne action that changes data in several databases: two-phase commit with its failure points and in-doubt state, how groups of copies repair it, sagas in depth, an outbox with an idempotent consumer, and how to choose.23 minSenior
- 159Multi-tenancy: shared tables, schema per tenant or database per tenantHow a product that serves many customer companies keeps their data apart: three layouts and the mix, how to stop a leak, the noisy neighbour, finding a tenant's data, and moving a large tenant.24 minDepth
- 160Cell-based architecture: limiting the blast radiusSplit a service into independent cells, each serving a fixed group of customers: the blast radius of 1 in N, the cell router and control plane, cell size, deploying one cell first, draining, and what stays outside the cells.24 minSenior
- 161Cost engineering: cost per request as a design inputCost per request and cost per 1,000: the bill split into a fixed part and a part for each request, comparing two designs, the three levers, alerts on unit cost, and when saving is not worth the effort.12 minDepth
- 162Large data migrations: moving a live system's data with no downtimeMove the data of a live system to a new database, shard layout or region: backfill, following changes, checking, moving reads and writes a few percent at a time, a way back, and planning with numbers.23 minSenior
Want help while you learn?
Reading is free for everyone. A free account adds the AI tutor, quizzes, notes and your progress.