Scale up or scale out, and why servers must be stateless

After you know the peak load (lesson 14), the next question is: "The load grows 10 times. What do you do?" A second question follows soon. A user logs in on one server, and the next request goes to another server. Is the user still logged in?

A weak answer is "add more servers" or "buy a bigger machine", with no reason. A strong answer compares the two, with numbers, and says what must be true for more servers to work. That condition is that no server keeps data that the next request needs.

At SDE-1 level you should name both ways and one cost of each. At SDE-2 level you should explain sessions and where to keep them. At SDE-3 level you should also say what stays hard when you add servers.

Think about a tailor with one shop. Orders grow. The tailor has two choices. One is a bigger shop with faster machines. The other is to open more shops with the same sign and the same skills.

There is a trap. The tailor writes each customer's measurements in a notebook at shop A. A customer who visits shop B is a stranger there, and must be measured again. The fix is to keep one register that every shop can read. Then any shop can serve any customer.

Scale up and scale out

Scaling up (also called vertical scaling) means moving to a bigger machine: more cores and more memory. Scaling out (also called horizontal scaling) means adding more machines of the same kind. A core does 1,000 ms of work in a second (lesson 15).

Scaling up is simple. Your code does not change, and no machine has to talk to another. Its limits are:

• A ceiling. The biggest machine on sale is the limit.
• One machine. If it stops, everything stops.
• Downtime to resize. In many clouds, you must stop a machine to change its size. One provider says so in its documentation (see Sources).

Scaling out has no fixed ceiling, and one broken machine takes away only a small part of the capacity. It also lets you add machines for a busy day and remove them after, as in lesson 13. Its costs are:

• Something must share the requests between the servers (the part in the lesson 2 drawing).
• More machines to set up, update and watch.
• Calls between machines are slower than work inside one machine.
• Data that one server keeps in its own memory is not visible to the other servers. This is the problem of state.
• The database is one shared place. More servers do not make the database bigger.

State and sessions

State is data that a server remembers from one request to the next. The common case is a session. A session is a record that says who a logged-in user is. In lesson 7, the cookie holds only an ID, and the server looks up that ID in its own records.

Suppose those records are in the memory of server A. The user logs in on server A. The next request goes to server B, which has no record for that ID. The user looks logged out. Requests are shared evenly at random over N servers. The chance that the next request goes to a different server is (N - 1) / N.

A server is stateless when it keeps nothing between two requests that it needs for the next one. Then any server can answer any request. A test: "If this server stopped now, and a new empty one took its place, would any user notice?" If yes, the server holds state.

Where state can live

1. In the memory of one server. This is the problem above. Avoid it.
2. Pinned to one server. The part in front sends each user to the same server every time, for example by a cookie. This is called a sticky session. It works, but the load can become uneven, and if that server stops, its users lose their sessions.
3. In the cookie. The server puts the session data in the cookie, with a signature (lesson 24) so the user cannot change it. The user can still read the data, so keep secrets out of it. Then no server needs to remember anything. The costs: the cookie travels with every request. It can safely hold only about 4,096 bytes (see Sources). You cannot easily cancel it before it expires.
4. In a shared store. All servers read and write the sessions in a key-value store (lesson 17), found by the session ID. The cost is one lookup for each request, and one more part to run. Files go to object storage (lesson 24). Other data goes to the database.

Most designs use option 4. They name the choice aloud and its cost.

The database is the hardest part to scale out, because it holds the data (lesson 17). Its first step is often a bigger machine.

We size the API servers of the food app from lesson 15 as its load grows.

Assumptions (made up to teach with):

• A request takes 8 ms of CPU time. We plan for 70% busy (lesson 15).
• One machine with c cores does c x 1,000 / 8 = 125c requests per second, and 87.5c at 70%. A 4-core server does 350.
• The biggest machine on sale has 64 cores. It does 87.5 x 64 = 5,600 requests per second at 70%.
• A server costs Rs 10 per hour (lesson 13).
• To keep the table simple, the server counts below do not include the spare server of lesson 15. The check questions do include it.
• 500,000 users are logged in at the peak. A session record is 1 KB.

Step 1: grow the load.

Scaling up works until 5,600 requests per second. After that, you have more than one machine, and you have the problems of scaling out anyway.

Step 2: a failure. With one machine, one failure takes away 100% of the capacity. With 58 servers, one failure takes away 1 / 58 = 1.7%.

Step 3: the session problem.

• With 4 servers, the chance of a different server is (4 - 1) / 4 = 3 / 4 = 75%. This answers the predict box.
• With 58 servers it is 57 / 58 = 98.3%. Adding servers makes the problem worse.

Step 4: three fixes.

• Sticky. If one of 58 servers stops, 500,000 / 58 = 8,621 users lose their session. With 4 servers it would be 500,000 / 4 = 125,000.
• Cookie. Each request carries 1 KB more. At 20,000 requests per second, that is 20,000 x 1 KB = 20,000 KB = 20 MB per second. It is 160 megabits per second more data coming in.
• Shared store. Assumption: one store does 100,000 simple operations per second. We need 20,000 lookups per second, which is 20% of it. The sessions are 500,000 x 1 KB = 500 MB, which fits in the memory of one machine. Each lookup adds about 0.5 ms for the round trip inside one data centre (lesson 14).

Step 5: why stateless pays. Assume the peak of 20,000 requests per second lasts 4 hours a day. The other 20 hours need 2,000 (6 servers).

• Scaling out, adding and removing servers: 6 x 20 + 58 x 4 = 120 + 232 = 352 server-hours = Rs 3,520 a day.
• 58 servers all day: 58 x 24 = 1,392 server-hours = Rs 13,920 a day. This is 13,920 / 3,520 = 3.95, so about 4 times more.

Removing a server is safe only when it holds no state. Otherwise its users lose their sessions.

• Scaling up solves the extra work of many machines: nothing changes in the code. It costs a ceiling (5,600 requests per second here), downtime to resize, and one machine that can stop everything. I would use it first for a small app or for the database. I would change when the ceiling is near or one failure costs too much.
• Scaling out solves the ceiling and the failure of one machine: 1.7% lost instead of 100%. It costs a part that shares requests, more machines to run, and the state problem. I would change to it when one machine cannot take the peak.
• A shared store solves state for any number of servers. It costs one lookup (about 0.5 ms) for each request, and a part that must not fail. I would keep a very small session in a signed cookie instead, if I did not want to run the store.

"Scale out is always better." At 300 requests per second, one 4-core server is enough. Five servers would add a part that shares requests, more updates, and the session problem, for no gain.

"Stateless means the app has no data." The state still exists. It lives outside the servers: in a shared store, in the database, in object storage, or in the cookie. The servers are stateless, not the system.

"More servers always means more capacity." Only if the servers are the limit. Assume each request makes 2 database queries, and the database does 10,000 queries per second. Then the system is limited to 10,000 / 2 = 5,000 requests per second. A 64-core machine already does 5,600, so more servers add nothing.

1. What is the difference between scaling up and scaling out? What does stateless mean? Name three places where state can live.
2. The peak is 12,000 requests per second. A request takes 8 ms of CPU. Each server has 16 cores. We plan for 70% busy and one spare. (a) How many servers? (b) Sessions live in the memory of each server. What share of a user's next requests goes to another server?
3. A team saves profile photos in a folder on the server's disk. There are 3 servers, and requests are shared evenly. What share of photo views fail, and what do you change?