When too much arrives: slow down, say no, or do less

Sale days, match nights and result days bring far more people than normal, all at once. This sudden jump in traffic is called a spike. The interviewer asks: "Traffic becomes ten times larger for five minutes. What do you do?"

A weak answer says "add more servers" or "put everything in a queue". New servers take minutes to start, and a long queue only moves the delay. A strong answer decides who waits, who is turned away, and what the service stops doing. It keeps the most important work, such as payment, safe.

At SDE-1 level you should say that the service must refuse or slow some work. At SDE-2 level you should do the arithmetic. At SDE-3 level you should also stop the refusals from turning into a retry storm.

Think about a small restaurant on a festival night. It has 20 seats, and 100 people arrive at once. The owner has three choices. She can ask people to wait outside and let them in as seats free up. She can say at the door: "We are full, please come back at 9." Or the kitchen can cook only the quick dishes tonight. A bad owner lets everyone in, and nobody gets food.

What goes wrong with no plan

Capacity is the most work per second that a part can finish. Overload means that work arrives faster than capacity for some time. From lesson 58:

• backlog growth per second = arrival rate - capacity
• wait for the next request = backlog / capacity

A user gives up after a timeout (lesson 54). The server often does not know this. It still does the work for a request that nobody waits for. That is wasted work. Once the wait is longer than the timeout, every request the server finishes is for a user who has left. The server is fully busy, and the useful work is almost zero. Adding servers helps only after minutes. So we need a plan that keeps useful work close to capacity.

Way 1: slow down (back-pressure)

A bounded queue is a queue (lesson 58) with a maximum size. A queue with no limit is unbounded. Back-pressure means that a full part tells the part before it to slow down. When the bounded queue is full, the sender must wait, or it gets a "busy" answer. The pressure moves back, step by step, to the first sender.

This works when the sender can wait: a nightly job, a file upload, or another internal service. It does not work for customers who arrive on their own. You cannot slow them. They leave or retry. Choose the size of the queue like this: capacity x the longest wait you accept. This is our own advice.

Way 2: say no (load shedding)

Load shedding means refusing some requests on purpose, so that the others are served well. Rules:

1. Refuse early and cheaply, before the slow work starts.
2. Decide from the server's own state: for example, more than a set number of requests in progress, or a queue that is full.
3. Refuse the less important first. This is a priority order. Keep checkout and payment. Refuse browsing first.
4. Refuse the old. Drop a request whose deadline (lesson 54) has passed.
5. Say how to come back. Answer 503 Service Unavailable, which means "too busy for now", with a Retry-After header (lesson 54).
6. Control the retries. Callers use backoff, jitter and a retry budget (lesson 54). Otherwise the refusals come back as a storm.

Load shedding is not rate limiting (lesson 56). A rate limit is a rule for each client. It holds even when the server is idle, and it is about fairness. Shedding protects the server. It starts only when the server is overloaded, and it works for all clients. A service often has both.

Way 3: do less (graceful degradation)

Graceful degradation means that an overloaded service still answers, but does less work for each request. Options:

• Turn off optional parts, such as recommendations.
• Serve a saved copy that may be old (a cache, lesson 31).
• Show simpler results, such as 10 items and not 50.
• Delay slow work, and tell the user ("we will email you", lesson 58).

Plan this before the spike. Write the list of what to turn off, in order, with a switch for each. Never turn off the promise to the user, such as a saved order (lesson 58).

Which one to use

Use them together. Do less first, because nobody is refused. Then refuse what is still too much, and refuse checkout last. Use back-pressure between internal parts.

A shopping app on sale night.

Assumptions (made up to teach with, not real data):

• The checkout service can finish 4,000 full requests a second.
• The sale opens, and 30,000 requests a second arrive for 300 seconds (5 minutes). 10% are checkout (3,000 a second) and 90% are browsing (27,000 a second).
• Users give up after 10 seconds. New servers need 3 minutes (180 seconds) to start.

Step 1: a queue with no limit. The backlog grows by 30,000 - 4,000 = 26,000 a second. A wait of 10 seconds is a backlog of 4,000 x 10 = 40,000. That happens after 40,000 / 26,000 = 1.5 seconds. This answers the predict box. The first 30,000 x 1.5 = 45,000 requests still get a good answer. After that, every finished request is for a user who has left. After 300 seconds the backlog is 26,000 x 300 = 7,800,000. The last request waits 7,800,000 / 4,000 = 1,950 seconds, which is 32.5 minutes.

Step 2: a bounded queue and shedding. We accept a wait of 2 seconds. The queue size is 4,000 x 2 = 8,000. When it is full, the server answers 503 at once with Retry-After: 5. Each second 4,000 requests finish within about 2 seconds, and 26,000 are refused. That is 26,000 / 30,000 = 86.7% refused. In 300 seconds, 4,000 x 300 = 1,200,000 requests get a good answer. In Step 1, only about 45,000 did.

Step 3: priority. Refusing at random keeps 4,000 / 30,000 = 13.3% of each kind. Checkout then gets 3,000 x 0.133 = 400 a second, so 86.7% of the checkouts fail. With priority, all 3,000 checkouts a second go first. The remaining 4,000 - 3,000 = 1,000 a second go to browsing, which is 1,000 / 27,000 = 3.7% of it.

Step 4: do less. A product page that comes from a saved copy, with recommendations off, costs 0.1 of a full request. Assumption: the copy may be up to 60 seconds old. The 1,000 spare full requests a second now serve 1,000 / 0.1 = 10,000 pages a second. That is 10,000 / 27,000 = 37% of browsing, and checkout is unchanged.

Step 5: retries. Assumption: a client retries up to 3 times, at once. That makes up to 1 + 3 = 4 tries. The load can reach 30,000 x 4 = 120,000 a second. Assumption: a refusal costs 0.01 of a full request. Refusing 116,000 a second costs 116,000 x 0.01 = 1,160 full requests a second, which is 29% of the capacity. With Retry-After, jitter and a retry budget of 10%, the load is about 33,000 a second. Refusing 29,000 costs 29,000 x 0.01 = 290, which is 7% of the capacity.

Step 6: back-pressure. A nightly job sends 1,000,000 emails. The workers send 40 a second. Assumption: a message is 1 KB. With no limit, the job fills the queue with 1,000,000 x 1 KB = 1 GB. With a bounded queue of 4,000, the job waits when the queue is full, and the queue holds 4,000 x 1 KB = 4 MB. Both take 1,000,000 / 40 = 25,000 seconds (6.9 hours). Back-pressure does not make the work faster. It keeps the queue small and makes the sender slow down.

Why not add servers? 30,000 / 4,000 = 7.5 times the capacity. New servers need 180 seconds. The spike lasts 300 seconds, so 180 / 300 = 60% of it is over before they help.

• Back-pressure solves a queue with no limit and a sender that is too fast. It costs a sender that waits, so it fails for customers who cannot wait. I would change to shedding when the senders are customers.
• Load shedding solves a service that collapses (1.2 million good answers instead of about 45,000). It costs refused users, and refusals that bring retries. I would change the priorities when the business changes.
• Graceful degradation solves refusing people. It costs a worse page, old data, and a plan to write and test. I would use it first, and keep the promise to the user.

"A bigger queue means no request is lost." The wait grows without limit. In the example, the last request waits 32.5 minutes, and a user waits only 10 seconds. The work is wasted.

"Load shedding is the same as rate limiting." A rate limit is per client and applies when the server is idle. Shedding starts when the server is overloaded, and it works for all clients.

"Add more servers and the spike is solved." The spike needs 7.5 times the capacity, and new servers take 180 seconds. Sixty percent of the spike is over before they help.

1. Name the three ways to handle too much work and say when each one fits.
2. A service can finish 2,000 requests a second. 12,000 arrive each second, with no limit on the queue. Users give up after 5 seconds. (a) After how many seconds is every request too late? (b) What queue size gives a wait of 1 second? (c) What share is refused with that queue?
3. A food app at lunch is overloaded. Checkout is 20% of the requests, and browsing is 80%. What do you do, and in what order?