Vertical vs horizontal scaling

The decision

Vertical scaling (scaling up) means moving to a bigger machine with more cores and memory. Horizontal scaling (scaling out) means adding more machines of the same kind, with a load balancer in front that shares the requests. Scaling up is simple, because your code does not change. Scaling out has no practical ceiling and survives the loss of one machine, but your servers must keep no user data in their own memory.

Choose vertical scaling when

• Your peak load fits on one machine with room to spare.
• The part you scale is a database. It holds the data in one place, so the first step is usually a bigger machine.
• Your code keeps state in memory, and you cannot change that soon.
• The team is small and cannot run many servers yet.

Choose horizontal scaling when

• Your peak is more than half of the biggest machine you are willing to run (Assumption: this leaves room to grow).
• One machine failing must not stop the service.
• Traffic has a sharp daily peak, so you want to add servers for a few hours and remove them after.
• Servers are stateless: any server can answer any request.

The numbers that change the answer

Assumptions: a request takes 8 ms of CPU. We plan for 70% busy. One core does 1,000 / 8 = 125 requests per second, which is 87.5 at 70%. A 4-core server does 350 at 70%. In this exercise the biggest machine we use has 64 cores, so it does 87.5 x 64 = 5,600. (Real clouds sell larger machines, up to 192 vCPUs in common families, but a service rarely speeds up in a straight line on that many cores, because locks and shared memory slow it down. The method is the same for any limit.) A 4-core server costs Rs 10 per hour. The counts below leave out the spare server.

• In this example, scaling up works until 5,600 requests per second. After that you have many machines anyway, with all the problems of scaling out.
• One failure: one machine takes 100% of the capacity with it. With 58 servers, one failure takes 1 / 58 = 1.7%.
• Time lost to resizing: many clouds need a stop to change a machine size (see Sources). Assumption: a goal of 99.9% uptime allows 0.001 x 30 x 24 x 60 = 43.2 minutes per month. A 5-minute stop fits once, not every week.

Daily cost. Assumption: the peak of 20,000 requests per second lasts 4 hours a day. The other 20 hours need 2,000, which is 6 servers.

• Scale out with automatic add and remove: 6 x 20 + 58 x 4 = 352 server-hours, which is Rs 3,520 per day.
• 58 servers all day: 58 x 24 = 1,392 server-hours = Rs 13,920 per day. That is 13,920 / 3,520 = 3.95 times more.
• Scale up cannot follow the curve, because each resize needs a stop. You pay for the peak size all day.

What breaks with each choice

• Vertical: a ceiling (the biggest machine), one point of failure, and a stop to resize. Fix: a standby machine, and a plan to scale out when the peak passes half of the biggest machine.
• Horizontal: user data kept in one server's memory. With N servers, the next request lands on a different server with chance (N - 1) / N. That is 75% for 4 servers and 98.3% for 58. The user looks logged out. The database is still one shared place, so more servers do not make it bigger. Fix: keep sessions in a shared store, and scale the database separately.

What it costs

• Scale up: little work. You pay a larger monthly bill for the machine, and a standby doubles it.
• Scale out: a load balancer, health checks, automatic server counts, and a shared store for sessions. Assumption: 2 engineers for 6 weeks to make a typical service stateless.
• Going back is easy in one direction. You can scale up now and scale out later, but only if you kept servers stateless in the code you write today.

How each level answers

SDE-1: Scaling up means a bigger machine. Scaling out means more machines. Scaling out needs stateless servers.

SDE-2: I scale up first, because it needs no code change. I scale out when one machine cannot hold the peak, or when one failure must not stop us. I keep sessions in a shared store so that any server can answer.

Senior: At 5,000 requests per second one 64-core machine still fits. It is at 89% of its planned limit of 5,600, with no room to grow. At 20,000 I need 58 servers, and one failure takes only 1.7%. A 4-hour peak costs Rs 3,520 per day with automatic scaling, against Rs 13,920 for a fixed fleet. The failure with scale out is lost sessions, and the fix is a shared store.

Staff: I want the choice to be reversible. I scale up now, because it takes hours, and I keep the code stateless so that scaling out stays cheap. The trigger to build the fleet is the peak passing half of the biggest machine, or an uptime target that one machine cannot meet. The platform team owns the fleet and the load balancer. I measure peak CPU, cost per 1,000 requests, and the share of capacity lost in one failure.

Say it in the interview

"I start by scaling up, because it needs no code change, and I keep servers stateless so I can scale out later. At 5,000 requests per second one 64-core machine is already 89% full, and at 20,000 I need about 58 small servers, where one failure costs only 1.7% of capacity. For the database I scale up first, because the data lives in one place."

Common mistakes

• Saying "scale out" with no number. One bigger machine may cost less and be simpler for months.
• Adding servers while sessions sit in server memory. With 58 servers, 98.3% of requests land on the wrong server.
• Forgetting the database. More app servers do not make it bigger, and they can overload it with connections.

Sources

• Resizing needs a stop: "You must stop your instance before you can change its instance type", Amazon EC2 documentation, https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/change-instance-type-of-ebs-backed-instance.html (opens in a new tab) (checked 26 Oct 2026). Other providers differ.
• Machine sizes: the AWS C7a page lists c7a.48xlarge with 192 vCPUs, so 64 cores is a choice for this exercise, not a vendor limit: https://aws.amazon.com/ec2/instance-types/c7a/ (opens in a new tab) (checked 27 Oct 2026).
• The CPU time, the 64-core limit, the price of Rs 10 per hour, the peak hours and the effort are assumptions made for teaching. The arithmetic is calculated from them.