Polling, long polling, SSE and WebSockets

Some screens must change without the user doing anything. A cricket score changes after each ball. A delivery map shows the rider moving. The interviewer asks: "How does the phone learn that something changed?"

A weak answer names one technology and stops there. A strong answer starts from the numbers: how often the data changes, how many users watch, and whether the phone also sends data. It then compares the cost of two or three ways.

At SDE-1 level you should name the ways and say how they differ. At SDE-2 level you should work out the requests per second and the number of open connections. At SDE-3 level you should also plan for servers that restart, and for phones that lose the network.

Think about waiting for a train. You can walk to the enquiry desk every 5 minutes and ask. Or you can ask the clerk to answer only when the train comes, and you wait at the desk. Or you can listen to a radio that tells you each news item as it happens. Or you can stay on a phone call, where both of you can talk at any time.

These are four ways for a server to give news to a client. In lesson 7 you saw that HTTP works one way round: the client sends a request, and the server answers. The server cannot start a message by itself. So each way below is a trick on top of that rule.

Polling. The client sends a normal request on a timer, for example every 5 seconds. The request asks: "Is there a new score?" The server answers at once, even when nothing changed.

• requests per second = viewers / seconds between polls
• delay: on average half of the gap between polls, at most the full gap

Long polling. The client sends a request, and the server does not answer at once. It holds the request open until there is news, or until a time limit (such as 30 or 60 seconds). Then it answers, and the client sends the next request at once. There are far fewer empty answers. Each news item still costs a full request and answer, with all the headers.

Server-Sent Events (SSE). The client opens one HTTP request. The server answers, and keeps the answer open. It writes each news item into this answer as a line of text, whenever it likes. A news item is called an event. The data goes one way, from server to client. A browser has a built-in feature for SSE. It reconnects by itself. It also tells the server the ID of the last event it got, so the server can send what was missed.

WebSocket. The client sends an HTTP request that asks to switch protocol, with the header Upgrade: websocket. A protocol is a set of rules for sending messages, and HTTP is one. The server answers with status 101, which means "I agree to switch". From then on, the same TCP connection (lesson 7) carries messages in both directions, and either side can send at any time. A WebSocket does not reconnect by itself. Your code must do that.

What an open connection costs

With the last three methods, many clients hold a connection open for a long time. This costs:

• Memory. Each open connection uses memory on the server.
• Heartbeats. A heartbeat is a tiny message sent on a timer to show that the connection is alive. Some network devices close a connection that is idle, that is, one with no data for a while. The SSE standard suggests a comment line about every 15 seconds. WebSocket has built-in ping messages.
• Restarts. When a server restarts, all its connections break. All its clients reconnect at the same moment.
• The load balancer (lesson 28). It must keep connections open and must allow the upgrade to WebSocket. A layer 7 balancer reads the upgrade request. A layer 4 balancer passes the bytes. Least connections works better than round robin. Round robin does not see how many connections a server already holds, and here connections last long.

A live cricket score. The score changes after about every ball.

Assumptions (made up to teach with, not real traffic):

• 500,000 (5 lakh) people watch at the same time.
• The score changes once every 30 seconds.
• Polling asks every 5 seconds. Long polling holds a request for up to 60 seconds.
• One poll moves 800 bytes in total (request and answer, with headers). One SSE event is 120 bytes.
• A poll answered from memory costs 1 ms of CPU. A server has 4 cores and we plan for 70% busy (lesson 15).
• A server holds 50,000 open connections at our 70% target, using about 20 KB each.

Step 1: polling.

• Requests = 500,000 / 5 = 100,000 per second. This answers the predict box.
• A poll gets news only if the score changed since the last poll. That happens for 5 / 30 = 1/6 of the polls. So 5/6, which is 83%, are empty: 83,333 empty answers per second.
• Delay: 5 / 2 = 2.5 seconds on average, 5 seconds at most.
• One server: 4 x 1,000 / 1 = 4,000 per second. At 70%: 2,800. Servers = 100,000 / 2,800 = 35.7, so 36. Plus 1 spare = 37 servers.
• Data: 100,000 x 800 bytes = 80 MB per second = 640 megabits per second.

Step 2: long polling. Every viewer holds one request open: 500,000 held requests. Each news item ends the request, and a new one starts. New requests = 500,000 / 30 = 16,667 per second. That is 100,000 / 16,667 = 6 times fewer. Data: 16,667 x 800 = 13.3 MB per second, about 107 megabits per second. The delay is only the short gap before the next request. But the 500,000 held requests are open connections too. They need 500,000 / 50,000 = 10 servers, plus 1 spare = 11, the same as SSE. The 16,667 new requests per second need only 16,667 / 2,800 = 6 servers.

Step 3: SSE.

• 500,000 open connections. One server holds 50,000, so 500,000 / 50,000 = 10 servers. Plus 1 spare = 11 servers. Memory per server = 50,000 x 20 KB = 1 GB.
• Each news item goes to every viewer: 500,000 x 120 bytes = 60 MB. Over 30 seconds this is 60 / 30 = 2 MB per second, which is 16 megabits per second. That is 80 / 2 = 40 times less than polling.
• But the 60 MB leaves right after the score changes. In 1 second that is 60 x 8 = 480 megabits per second, or 48 megabits per second for each of 10 servers. The average hides this peak.
• Heartbeats: a comment every 15 seconds means 500,000 / 15 = 33,333 tiny messages per second. They are twice as many as real news items (16,667 per second), but each has only a few bytes.
• The delay is small, if we assume each server gets the new score within a second.

Step 4: a server restarts. With 11 servers, each holds 500,000 / 11 = 45,455 clients. When one restarts, its 45,455 clients reconnect. All at once, that is 45,455 new connections in about a second. If each client waits a random 0 to 10 seconds first, it is 45,455 / 10 = 4,545 per second. The other 10 servers get 455 per second each. Least connections then fills the restarted server. A phone that loses its network does the same: it reconnects and waits a random time. With SSE the browser sends the ID of the last event it got, so the server can send what was missed. With a WebSocket your own code must ask for the missed messages.

Step 5: choose. The score goes only from server to phone, so SSE fits. If a delay of up to 10 seconds is fine, polling every 10 seconds is simple: 500,000 / 10 = 50,000 per second. For a live auction where bidders send bids and see the top bid, both sides talk often, so use a WebSocket.

• Polling solves simplicity: ordinary requests, which work everywhere. It costs wasted requests (83% empty here) and delay. I would change when requests per second are high and the delay matters.
• Long polling solves most of the empty answers. It costs a held request for every viewer and full headers on each message. I would change to SSE when the data is one way and the platform allows open connections.
• SSE solves one-way news, with automatic reconnect. It costs open connections, and the client cannot send on it. On HTTP/1.1 (lesson 7), a browser allows only 6 SSE connections to one site across all tabs. I would change to WebSocket if the client must also send often.
• WebSocket solves two-way talk with low delay. It costs open connections, our own reconnect and heartbeat code, and a balancer that supports it. I would change to polling if messages are rare and users are many.

"WebSockets are always better than polling." Not always. Polling every 5 seconds needs 37 servers here, and open connections need 11. But if a fresh score once a minute is enough, polling needs 500,000 / 60 = 8,333 requests per second. That is 8,333 / 2,800 = 3 servers, plus 1 spare = 4. This is fewer than 11.

"An idle open connection costs nothing." Each one holds memory (1 GB per server here) and needs heartbeats. 500,000 connections send 33,333 heartbeats per second even when the score does not change.

"SSE and WebSocket are the same." SSE goes one way and the browser reconnects by itself. A WebSocket goes both ways, and you write the reconnect and the recovery of missed messages.

1. Which of polling, long polling, SSE and WebSocket send data both ways? What is a heartbeat for?
2. 200,000 people watch a score. Each phone polls every 4 seconds. The score changes every 20 seconds. (a) How many requests per second? (b) What share of the polls are empty?
3. A delivery app shows the rider on a map. 500,000 orders are active. The rider app sends its place every 4 seconds, and customers watch it. A teammate says "use WebSockets for everything". What do you say?