CDN: serving files from near the user

Your servers are in one city. Your users are in the whole country. A user in a far corner waits longer for every image and script, because each message travels a long road. When a photo becomes popular, thousands of users ask for the same file, and your servers must send it thousands of times.

Interviewers ask: "Where do images, scripts and videos come from? What happens when one photo goes viral?"

A weak answer says "we keep copies near the users" and stops. A strong answer says which files go there and which do not. It also says how long they stay, how a changed file reaches users, and what the servers see during a crowd.

At SDE-1 level you should explain why files are served from near the user. At SDE-2 level you should choose what to put there, and give a TTL. At SDE-3 level you should also explain how to change a file safely. You should say how to protect your servers when many nearby copies are empty at once.

Think about a newspaper. The press is in one city. Sending every reader's copy from that press would be slow. So the publisher prints once and sends copies to newsstands in every city. A reader buys at the nearest stand. If a stand has no copy of today's issue, it asks the press once, keeps the copy, and sells it to every next reader.

A web system works in the same way. The origin is your own servers, or your object storage (lesson 24), where the real files live. A content delivery network (CDN) is a company's network of cache servers in many cities, placed in front of your origin. Each cache server is an edge location, or simply an edge. The CDN sends each user to a nearby edge. Often it does this when the browser looks up the host name (lesson 7).

An edge is the cache from lesson 31, placed near the user:

1. The browser asks the edge for a file.
2. If the edge has the file, that is a hit. The edge answers at once.
3. If not, that is a miss. The edge asks the origin, answers the user, and keeps the file for its TTL. This is the pull way: the edge fetches a file when a user first asks for it. Some CDNs also let you upload a file to the edges before any user asks (push).

Each edge has its own cache. So the hit rate of lesson 31 is counted for each edge: reads per second at that edge x TTL.

A CDN does two jobs:

• It shortens the road. A new HTTPS connection needs about 3 round trips before the first byte (lesson 7). One is for the TCP link, one for the lock of HTTPS, and one for the request. Lesson 7 says the lock takes one or two; we assume one.
• It protects the origin. Most requests never reach the origin. Your servers send far fewer bytes and do less work.

A CDN charges for the data it sends and for each request (check the provider's price page). The gain is speed and a calm origin.

What to put at the edge

The test: "Is the answer the same for every user, and does it change rarely?"

A wrong choice here is not slow. It is a leak: if a personal response is stored at the edge, the next user can receive it.

How long: the headers

The origin tells the edge how long to keep a file in a header (lesson 7). The header is Cache-Control. Three values matter (see Sources):

• max-age=3600 means the copy is fresh for 3,600 seconds.
• private means only the user's own browser may keep it. A shared cache such as a CDN must not.
• no-store means nobody may keep it.

The cache key

The cache key is what the edge uses to decide that two requests ask for the same file. Often it is the whole URL, with its query string (check your CDN). Two details cause trouble:

• Query strings. A tracking part such as ?utm_source=ad7 is different for each campaign. If it is in the key, one picture becomes many copies. Leave it out of the key.
• Cookies. If the cookie is in the key, every user has their own copy of every file. The hit rate falls to almost nothing. Never put a login cookie in the key. If a response depends on the user, do not store it at the edge at all.

Changing a file

The same three ideas as in lesson 34:

1. Wait for the TTL. Simple. Users see the old file until it ends.
2. Purge. You ask the CDN to remove a file from all edges. It takes some time to reach every edge, and some providers charge for it. A purge of everything empties every edge at once (see "A crowd" below).
3. A new name. Put a version in the file name: app.8f3a1.js. When the file changes, the name changes. Each edge fetches the new name once, and nobody asks for the old name. Then the TTL can be very long. Provider documentation advises this for files that change often (see Sources). The page that points to the file must have a short TTL, so users find the new name.

A crowd: empty edges ask the origin together

When a file is new, or after a purge, every edge is empty. All edges then miss at the same time, and each one asks the origin. The origin gets one request from every edge, and more: while an edge waits for the file, more users ask the same edge. Two fixes:

• Request coalescing at the edge: the first request goes to the origin, and the others at that edge wait for its answer. This is the idea of lesson 35.
• An origin shield: one more cache layer between all edges and the origin. Edges ask the shield. The shield joins the requests for one file into one request to the origin (see Sources). The cost is one more stop on the road for a miss, and an extra charge.

Private files

A private file can still use the edge. The app checks the login and gives a signed link. It is a URL with a short expiry and a signature, as for pre-signed URLs in lesson 24. The edge checks the signature, then sends its copy.

A photo app has its servers in Mumbai. A user in the far north-east opens a photo.

Assumptions (made up to teach with, not real traffic):

• Round trip: user to Mumbai 120 ms. User to the nearest edge 15 ms. Edge to the origin 40 ms, on a link that is already open.
• A new connection needs 3 round trips before the first byte. The edge takes 5 ms to answer. The origin takes 20 ms.
• One origin server has a 1 Gbps link and we plan for 70% (lesson 15).
• The viral photo is 200 KB. 5,000 requests per second ask for it.
• There are 50 edges, and they share the requests equally.
• In Step 5, the origin is slow because of the crowd: its first answer takes 1 second to come back. So every request in that first second is a miss.

Step 1: the time to the first byte.

• Without a CDN: 3 x 120 + 20 = 380 ms.
• Edge hit: 3 x 15 + 5 = 50 ms. That is 380 / 50 = 7.6 times faster.
• Edge miss: 50 + 40 + 20 = 110 ms. Even a miss is faster, because the edge uses a link to the origin that is already open.

Step 2: the origin without a CDN. 5,000 x 200 KB = 1,000,000 KB per second. That is 1,000 MB per second, or 8,000 megabits per second, which is 8 Gbps. Servers = 8 / (1 x 0.7) = 11.4, so 12, plus 1 spare = 13 servers, only to send one photo.

Step 3: the origin with a CDN. Each edge gets 100 requests per second for this photo. The TTL is 1 hour = 3,600 seconds. Reads x TTL = 100 x 3,600 = 360,000. Hit rate = 360,000 / 360,001, which is more than 99.999%. Each edge asks the origin once an hour: 50 requests per hour, not 5,000 per second.

Step 4: a photo that few people open. Assumption: a photo gets 0.005 requests per second in the whole country. Per edge: 0.005 / 50 = 0.0001 per second.

• TTL 1 hour: 0.0001 x 3,600 = 0.36. Hit rate = 0.36 / 1.36 = 26.5%.
• TTL 1 day: 0.0001 x 86,400 = 8.64. Hit rate = 8.64 / 9.64 = 89.6%.

Rare files hit badly, unless the TTL is long. A long TTL is safe when the file name changes with the file.

Step 5: the crowd. All 50 edges are empty, and each gets 100 requests in the first second.

• No joining: every request goes to the origin: 50 x 100 = 5,000 requests. That is the full load of Step 2, at the worst moment.
• Each edge joins its requests: 1 request from each edge = 50 requests.
• With an origin shield: the shield joins the 50 requests into 1.

This answers the predict box: 5,000, then 50, then 1.

Step 6: a bad cookie. Suppose the cookie is in the key, and 1,000,000 users open the same logo. Each user has a different cookie, so each one is a miss: 1,000,000 requests reach the origin. With the cookie out of the key, only the 50 edges ask. That is 1,000,000 / 50 = 20,000 times fewer.

• A CDN solves distance and origin load: 50 ms instead of 380 ms, and 50 requests an hour instead of 5,000 a second. It costs a charge for data and for each request, and old files until the TTL ends. The hit rate is low for rare files (26.5%). I would add one when users are far from the origin, or when the same files are sent many times.
• A new file name solves old files and the cost of purges. It costs a page that must point to the new name, and a build step that makes names. I would use it for scripts, styles and images that change.
• An origin shield solves the crowd: 1 request instead of 50. It costs one more stop on the road for a miss, and an extra charge. I would add it when many edges miss together, for example after a purge.

"A CDN makes every request faster." Only files that the edge keeps. A cart, an order or a payment goes to the origin each time. In the example, a rare photo had a hit rate of only 26.5% at a TTL of 1 hour.

"Put the cookie in the cache key to keep users apart." It does keep them apart, but it makes the cache useless: 1,000,000 requests reach the origin, not 50. Do not store personal responses at the edge. Keep the cookie out of the key.

"When a file changes, purge everything on each deploy." It works, and then every edge is empty. Without joining requests, the origin gets 5,000 requests per second at once. Use new names for changed files, and purge only the paths that must change.

1. What are the origin and an edge? Name two kinds of response that must not be stored at the edge.
2. A photo is 150 KB and gets 2,000 requests per second. One origin server has a 1 Gbps link, and we plan for 70% busy plus 1 spare. (a) How many servers send it without a CDN? (b) There are 20 edges, all empty after a purge. Each gets 100 requests in the first second, and the origin needs 1 second to answer. How many requests reach the origin with no joining, with joining at each edge, and with an origin shield?
3. A shop puts /api/my-cart behind a CDN with a TTL of 60 seconds. Customers see each other's carts. What went wrong, and what do you do?