Cell-based architecture: limiting the blast radius

The interviewer says: "Last month one bad deploy made your payment service fail for every merchant. How do you make sure the next failure hurts only a small part of your customers?"

A weak answer says "add more servers" or "use a second zone". Those help when a machine dies. They do not help when the code, the data or one customer is the problem, because every server has the same code and the same shared parts. A strong answer splits the service into independent copies. It says what share of customers one failure can reach, what that costs, and what must still be shared.

This is an SDE-3 question.

A ship has walls inside its hull. If one room floods, the walls keep the water in that room. The ship floats. A cell is such a room for a service: a complete, independent copy of the service, with its own servers, data and queues. A cell serves a fixed group of customers.

In lesson 83 a bulkhead gave each dependency its own pool of workers inside one program. A cell applies the same idea to the whole service. In lesson 69 a shard splits only the data. A cell splits servers, queues and data together.

Blast radius is the share of users that one failure can hurt (lesson 127). With N equal cells, one failed cell hurts 1 / N of the users. With 10 cells that is 10%. You can quote this number.

Four parts make a cell design.

• The partition key. A field in every request that says which cell serves it, for example the merchant ID. A good key is one that few requests cross.
• The cell router. It reads the key, finds the cell, and sends the request there. It does only that. Keep it the thinnest layer you can build, because every request passes it. It is like the load balancer of lesson 54, with one difference: it must send a customer to the same cell every time, because that cell holds the customer's data. It keeps the table of "merchant to cell" in memory, so it still works for a while if the store of that table is down.
• The control plane. It creates cells, removes them, and moves customers between cells. It is not in the path of a request.
• The cell itself. It shares no state with other cells.

The router and the cells handle requests: together they are the data plane. The control plane only manages them.

A cell router reads the key of each request, such as the merchant ID, and sends it to the cell that serves that customer. There are four cells, and each is a whole copy of the service with its own servers and data. The third cell is down, so only its customers are hurt. The other three cells keep working.

Choosing the cell size

Three forces push on the size.

1. The largest customer must fit in one cell.
2. A cell must be small enough to test at full size. All cells have the same size, so one test covers all.
3. A cell must be big enough to save money, because each cell has its own fixed parts and its own spare servers.

Smaller cells shrink the blast radius. They also mean more things to run: deploys, alarms and data checks for every cell. If each cell keeps a spare server (lesson 31), they mean more spare capacity too. AWS lists idle capacity both ways, so check your own numbers.

Deploying and draining

Deploy to one cell first. This is a canary where the unit is a cell (lesson 127). If the new version fails, only that cell is hurt. Watch it, then go on in waves.

To drain a cell means to stop sending requests to it, for a bad release or for repair. Its work must go somewhere. If every cell can serve every customer, the others take its load: with N cells, each of the others gets N / (N - 1) times as much. If a cell owns its customers' data, they cannot move until the control plane has copied their data. After that, the same factor applies.

Moving a customer to another cell

The control plane moves a customer when a cell fills up, the customer grows, or a cell is retired. It uses the four steps of lesson 69: copy the customer's data to the new cell, keep sending new changes, switch the customer's entry in the router's table, delete the old copy. Move one customer at a time, slowly, so the new cell is not overloaded.

What stays outside the cells

• The router and the control plane. If they fail, every cell is cut off, so keep them small and run many copies.
• Data that all customers share, such as the table of "merchant to cell", the login system and billing across customers.

Each of these is a shared part again. Keep the list short, and design so that the data plane keeps serving when the control plane is down (our advice).

Work that crosses cells

Some actions involve two customers, for example a payment split between two merchants. If the two are in different cells, the action crosses cells. Choose a key that keeps such pairs in one cell, or send a message from one cell to the other and let the second cell act on it (an outbox, lesson 92).

Cells, a bulkhead and a shared fleet

ShopPay sends payment links for merchants. Assumptions (made up to teach with): the peak is 14,400 requests per second. One server handles 350 requests per second at 70% busy (lesson 31). A server costs Rs 10 an hour, which is 10 x 720 = Rs 7,200 a month. Each cell has its own database, queue and alarms, which cost Rs 25,000 a month. The largest merchant sends 9% of the traffic. The key is the merchant ID. A bad release fails every request in its cell, and the team stops it after 5 minutes.

Step 1: blast radius. One cell of 4 hurts 1 / 4 = 25% of the traffic. One cell of 10 hurts 10%.

Step 2: the price. One shared fleet needs 14,400 / 350 = 41.1, so 42 servers, plus 1 spare = 43. With N cells, each cell has 14,400 / N requests per second and its own spare.

Example for 4 cells: 14,400 / 4 = 3,600. Then 3,600 / 350 = 10.3, so 11, plus 1 spare = 12. The answer to the predict box: 4 cells hurt 25% and cost Rs 111,000 more a month (33%). 10 cells hurt 10% and cost Rs 347,400 more (104%).

Step 3: the largest merchant. A merchant with 9% of the traffic needs a cell that holds at least 9%. 100 / 9 = 11.1, so at most 11 cells. 20 cells of 5% each cannot hold this merchant. Even 11 cells would leave it almost a whole cell, so we choose 4 cells for now.

Step 4: deploy one cell first. The bad release fails every request in its cell for 5 minutes (300 seconds).

• All servers at once: 14,400 x 300 = 4,320,000 failed requests.
• One of 4 cells first: 3,600 x 300 = 1,080,000.
• One of 10 cells first: 1,440 x 300 = 432,000.

A 5% canary on one shared fleet (lesson 127) fails 14,400 x 5% x 300 = 216,000 requests, which is even fewer. So for a bad release alone, a canary can be enough. Cells add more when the failure is not a release: one bad request that crashes servers, one customer's bad data, or a change to the data layer, which a server canary cannot test alone.

Step 5: drain. Assumption: the control plane has copied the customers of one drained cell to the others, who share them equally. Each of the other 3 cells gets 4 / 3 = 1.33 times its load. With a 70% target, a cell must normally run at no more than 70 x 3 / 4 = 52.5% so that it can take this. With 10 cells the number is 70 x 9 / 10 = 63%.

Step 6: crossing cells. Assumption: 20% of actions involve two merchants, and merchants are placed at random. The chance that two merchants are in different cells is (N - 1) / N. With 4 cells: 0.2 x 3 / 4 = 15% of all actions cross cells. With 10 cells: 0.2 x 9 / 10 = 18%. A key that puts partners together lowers this.

Step 7: the router. Every request passes it. Assumption: its availability is 99.99%, so it can be down 43,200 minutes x 0.0001 = 4.3 minutes a month, however well the cells work.

What companies wrote

• Shopify wrote on 2 Mar 2018 that a pod is a set of shops on a fully isolated set of datastores. Job workers, app servers and load balancers are shared, so a pod is only part of the cell in this lesson. A router in the load balancers sends each request to its pod. The post says the aim was that a failure in one pod would not become an outage of the whole platform. Link: https://shopify.engineering/a-pods-architecture-to-allow-shopify-to-scale (opens in a new tab)
• Slack wrote on 22 Aug 2023 that after a network fault in one availability zone on 30 Jun 2021, it treated each zone as a cell. This is a different model from the rest of this lesson: a Slack cell serves every customer, so it limits a zone failure, not a failure of one customer group. The goal was to remove traffic from a zone within 5 minutes. Moving its most critical user-facing services took about 1.5 years, and some services cannot stay inside one zone (company figures). Link: https://slack.engineering/slacks-migration-to-a-cellular-architecture/ (opens in a new tab)

Cells solve a failure that reaches every customer. A bad release, a bad request or a heavy customer hurts 1 / N of the users: 25% with 4 cells, not 100%. Cells cost more servers and parts (Rs 445,600 against Rs 334,600), a router that every request passes, and work that crosses cells. I would add cells when one failure hurting all customers is costly and a canary does not catch the cause, when a customer needs isolation, or when a size limit forces a split. I would keep one shared fleet when the team is small or most actions touch many customers.

"A second zone gives the same protection as cells." A second zone covers a dead machine or room. The same bad code runs in both zones. A cell contains a bad release, because it reaches one cell first.

"The smallest cell is the best cell." In step 2, 20 cells cost 3.2 times the shared fleet, and in step 3 they cannot hold the largest merchant.

"The router can also check permissions and rules." Every cell depends on it. A bug there fails all cells. It maps the key to a cell and nothing else.

1. Define cell, cell router and control plane. What is the blast radius with N equal cells?
2. A service peaks at 7,200 requests per second. A server handles 300 per second at 70% busy, plus one spare server in each cell. How many servers in total for (a) one fleet and (b) 6 cells? What is the blast radius for 6 cells?
3. Your largest customer sends 15% of the traffic, and a team proposes 10 cells of equal size. What do you say?