One server, many requests: threads, event loops and why a busy server slows down

After you add servers (lesson 51), interviewers look inside one server. "How does your server handle 10,000 connections?" "Why is p99 bad at peak when the CPU is only half used?"

A weak answer is "add more servers". A strong answer separates the time a request works from the time it waits. It works out how many requests are inside the server at the same time, using a simple formula. It explains how one program can watch thousands of connections. And it shows why waiting time grows fast when a server is nearly full.

At SDE-1 level you should explain work and waiting. At SDE-2 level you should work out how many requests one server can finish and find its limit. At SDE-3 level you should explain the waiting line with numbers and read the signs in a live system.

A bank has tellers. A customer asks for a document that is kept in a back office, and the answer takes a while. A teller can stand and wait for the document, doing nothing. Or the teller can serve the next customer and come back when the document arrives. The first teller is wasted during the wait. The second one stays busy. Below, the first teller is like a thread that waits, and the second is like an event loop. Both are explained step by step.

What a server does for one request

For one request a server does four things: it reads the request, it computes, it waits, and it writes the answer. CPU time is the time a core really works (lesson 31). Wait time is the time the request waits for something else, such as a database, a disk or another service. Assumption: 10 ms of CPU time and 90 ms of waiting. So the request is inside the server for 100 ms. During the 90 ms, the core has nothing to do for this request.

Threads

A thread is one line of work inside a program. It has its own stack, a block of memory where it keeps its place and its local values. One core runs one thread at a time. The operating system switches between threads, and every switch takes a little time.

A thread costs memory. The system sets aside a stack for each thread. Assumption: 1 MB is set aside, which is close to the Java default on Linux (1,024 KB). A thread uses only a part of it. Assumption: a thread really uses about 100 KB. So memory has two numbers: 1 MB set aside, and 100 KB really used.

Thread per request

The simplest design gives each request its own thread. The code is easy to read, because the thread just waits when it needs to. It is wasteful when most of the time is waiting.

For 10,000 open connections we need 10,000 threads. Set aside: 10,000 x 1 MB = 10,000 MB = 10 GB. Really used: 10,000 x 100 KB = 1,000,000 KB = 1 GB. There is also a second cost. The system must switch between 10,000 threads, and it has a limit on how many threads it allows.

The thread pool

A thread pool has a fixed number of threads. Requests that arrive when every thread is busy wait in a line (also called a queue). The pool size is a limit on how many requests are in progress. The line has a limit too. When it is full, the server must refuse new requests or make callers wait.

Little's law

Little's law (Little, 1961) says this. Over a long time, the average number of requests inside a system equals the arrival rate times the average time each one spends inside. The arrival rate is how many requests arrive each second. The law holds when the server can keep up with the arrivals.

• requests inside = arrival rate x time inside
• threads needed = requests per second x seconds per request

Each request holds a thread for its whole time inside, including the waiting. So 400 requests per second that take 0.1 seconds each hold 400 x 0.1 = 40 threads.

Blocking, non-blocking and the event loop

A blocking call stops the thread until the result is ready. A non-blocking call returns at once, and the program checks later if the result is ready.

An event loop is one thread that watches many connections. It runs a small step when one of them is ready. On Linux, a call named epoll lets one thread watch a very large number of connections. When a step must wait for the database, it starts the call and moves on to the next ready connection. No thread sits idle.

Two bars drawn to the same scale of 100 ms. With thread per request, one thread works for 10 ms on request A and then waits 90 ms with nothing to do. With an event loop, the same single thread does the 10 ms of work of request A, then request B, and so on up to request J. In 100 ms it serves 10 requests, while each of them waits.

There is one rule: never block the loop. If one step does a long calculation, every other client waits. A loop uses one core, so a server with 4 cores runs 4 loops.

The words async and await are how many languages write this style. An await means "start this wait and let the loop run other work until it is done".

Some languages also offer light threads. These are many cheap threads that run on a few real ones. You write thread-per-request code, and the cost stays small.

The C10K problem

C10K means 10,000 clients at the same time. Dan Kegel wrote a well-known page about it. He named two main ways to solve it. One is a thread per client, which costs a whole stack for each client. The other is non-blocking input and output, with a call that tells the program which connections are ready, such as epoll.

With an event loop, a connection costs only its small state, not a whole stack. Assumption: 10 KB for each open connection. So 10,000 x 10 KB = 100,000 KB = 100 MB. Compare this with 1 GB really used by 10,000 threads: 10 times less.

The operating system needs a setting too. Each open connection uses a file descriptor, a number the system gives to an open file or connection. The system limits how many a program may hold. A server that wants many connections must have this limit set high enough.

The capacity of one server

Two limits decide how many requests per second a server can finish:

• CPU limit = cores x (1,000 ms / CPU ms per request) (lesson 31)
• Thread limit = pool size / seconds per request

The server's capacity is the smaller one. An event loop has no thread limit, so only the CPU limit is left.

Why latency grows fast near full load

Think about a shop with one cashier. If the cashier is busy 50% of the time, a customer seldom waits. If she is busy 95% of the time, customers arrive in groups by chance and a line forms. It takes a long time to clear, because an idle minute cannot be saved for later.

Some words first. A worker is a core or a thread that does the jobs. Service time is the time one worker needs for one job. Response time is the whole time from arrival to finish, waiting plus service. Utilisation tells how busy a worker is: the share of time it works.

A simple model has one worker. Jobs arrive at random times and have different lengths. The average response time in this model is:

• response time = service time / (1 - utilisation)

With a service time of 10 ms, the table shows the average response time. The last column is for 4 workers that share one line.

Look at the shape. The time rises slowly, then very fast. For one worker, p99 (lesson 4) is about 4.6 times the average. At 80% the average is 50 ms, so p99 is about 50 x 4.6 = 230 ms.

Column chart for one worker with a service time of 10 ms. The average response time is 20 ms at 50 percent utilisation, 33 ms at 70 percent, 50 ms at 80 percent, 100 ms at 90 percent, 200 ms at 95 percent and 1,000 ms at 99 percent. The columns rise slowly at first and then very fast.

This is a teaching model. If all jobs had the same length, the waiting part would be about half as long. So for that case the table is a high estimate. Real servers also have bursts, so real p99 is often worse. The names of the formulas are in Sources.

A line that grows without limit

If requests arrive faster than the server finishes them, the line grows without end. Callers stop waiting after some time. The server still works on a request after its caller has left, so it works for nobody. A growing line is the sign of a server over its limit.

Reading the signs

• High latency, low CPU: requests are waiting, not computing. Look at the thread pool, the line and the slow call they wait for.
• High CPU: the CPU limit is near. Less work per request or more cores is the fix.
• A growing line, or growing open connections: arrivals are above capacity.
• One event loop at 100% while other cores are idle: the server runs too few loops, or one step takes too long.

A design habit

Keep utilisation at peak below about 70 percent (lesson 31). The table shows why: the step from 70% to 90% costs much more than the step from 50% to 70%.

Assumptions (made up to teach with): 4 cores; 10 ms of CPU and 90 ms of waiting for each request; a sale day with 400 requests per second at the peak.

Step 1: the CPU limit. 4 x (1,000 / 10) = 4 x 100 = 400 requests per second.

Step 2: threads needed. Each request is inside for 10 + 90 = 100 ms = 0.1 s. Requests inside at 400 per second = 400 x 0.1 = 40 threads busy.

Step 3: the pool of 20. The pool can finish 20 / 0.1 = 200 requests per second. At 400 arrivals the line grows by 400 - 200 = 200 requests every second. This answers the predict box: the pool, not the CPU, is the limit. The CPU is used for 200 x 10 ms = 2,000 ms of 4,000 ms, which is 50%. This is "high latency, low CPU".

Step 4: the line and the caller. After 10 seconds, 2,000 requests are in the line. The last one waits 2,000 / 200 = 10 seconds. Assumption: callers give up after 3 seconds. A line of 3 x 200 = 600 requests is already too long, and it takes 600 / 200 = 3 seconds to reach. After that, the server works on requests whose callers have left.

Step 5: fix the limit. A pool of 40 threads finishes 40 / 0.1 = 400 requests per second, which equals the CPU limit. An event loop has no pool limit, only the CPU limit of 400. Both are at 100% CPU, so latency would explode (see the table).

Step 6: plan for 70%. 0.7 x 400 = 280 requests per second is the planned peak for this server. Threads needed at 280: 280 x 0.1 = 28. For 400 requests per second we need 400 / 280 = 1.43, so 2 servers (lesson 51). Each takes 200 requests per second, which needs 200 x 0.1 = 20 threads. A pool of 40 leaves room.

Run it. This code makes 1,000 requests that each wait 100 ms. Once with a pool of threads, and once with an event loop.

import asyncio, time
from concurrent.futures import ThreadPoolExecutor
N, WAIT = 1000, 0.1            # 1,000 requests, each waits 100 ms

def blocking(_):
    time.sleep(WAIT)           # the thread waits and does nothing else

async def non_blocking(_):
    await asyncio.sleep(WAIT)  # the loop runs other requests meanwhile

async def run_loop():
    await asyncio.gather(*(non_blocking(i) for i in range(N)))
start = time.perf_counter()
with ThreadPoolExecutor(max_workers=20) as pool:   # try 100 and 1000 too
    list(pool.map(blocking, range(N)))
print("20 threads:", round(time.perf_counter() - start, 2), "s")
start = time.perf_counter()
asyncio.run(run_loop())
print("event loop:", round(time.perf_counter() - start, 2), "s")

We ran it on one laptop on 29 Oct 2026 (Python 3.14). Your times will differ. With 20, 100 and 1,000 threads it printed 5.03 s, 1.03 s and 0.26 s. The event loop took about 0.1 s each time. The first two match the formula: 1,000 x 0.1 / 20 = 5 s and 1,000 x 0.1 / 100 = 1 s.

• Thread per request solves the problem of readable code: the thread just waits. It costs memory and switching: 1 GB really used by 10,000 threads in our example, and 10 GB set aside. I would change it when connections are many and mostly idle.
• An event loop solves that cost: 100 MB for 10,000 connections. It costs one rule, never block the loop, and code that must be written in the async style. I would move long calculations away from the loop, to other threads.
• Keeping utilisation low solves the jump in latency. It costs idle capacity: at 70% we use 280 of 400 requests per second. I would raise the target only when jobs are all the same size and traffic is smooth.

"A server at 90% CPU is healthy, because it is not at 100%." The table shows 100 ms with one worker and 30 ms with 4 workers sharing a line. The service time is 10 ms. So the time is 3 to 10 times higher, and rising fast. Real p99 is worse.

"If CPU is low, the server has room." In the example the CPU was 50% while the pool was 100% busy and the line grew. The limit was the 20 threads.

"Async makes each request faster." One request still takes 100 ms. Async lets one thread hold many waiting requests, so more of them fit in the server. A long calculation on the loop makes everything slower.

1. What is a thread pool? What does an event loop do when a request must wait for the database? What does Little's law say?
2. A service gets 250 requests per second. Each request takes 20 ms of CPU and waits 60 ms. (a) How many threads are busy? (b) How many requests per second can 4 cores do? (c) How many cores do we need to stay at 70% busy?
3. At peak, p99 is 3 seconds, CPU is 35%, and the pool of 20 threads is fully busy with a growing line. What do you check, and what do you change?