Case study: Video streaming: on demand and live (Hotstar IPL)

"Design a video streaming app" is a common question. A video is a very big file. People watch it on phones with fast and slow networks. For a live cricket final, millions of people press play in the same few minutes.

The interviewer asks: "A user taps play. How does the video start fast? What happens when 3 crore people watch the final together?"

A weak answer says "store the video and send it from the server". One server cannot send that many bytes: lesson 16 worked out 60 Tbps for 3 crore viewers. It also ignores a network that becomes slow during the video. A strong answer keeps several sizes of every video and cuts each one into small files. It sends them from a CDN (lesson 36) and lets the phone choose the size. It also says what is different for a live match, where the files do not exist yet.

At SDE-1 level you should follow one video from upload to play. At SDE-2 level you should add the numbers for storage, workers and bandwidth. At SDE-3 level you should also explain the delay of a live match, and what to do when the CDN cannot carry the load.

Think about a canteen in a big office. Food comes in small boxes. Before lunch the cook packs each dish in three box sizes. A hungry person takes the large box, and a person in a hurry takes the small box. The next box can be a different size. At a live event the cook is still cooking while you eat, so a box cannot be packed before the food exists. A video app works in the same way.

What we build

On demand means that the video already exists: a creator uploads it, and viewers watch it later. Live means that the video is made now, for example a match. Viewers see it a short time after it happens.

Requirements (the numbers are assumptions):

• A video starts within 3 seconds for 95% of the plays.
• A live match reaches the viewer at most 30 seconds after it happens.
• At the peak, 30,000,000 people watch together, and each needs 2 Mbps (megabits per second), as in lesson 16.

We leave out payments, recommendations, advertisements and copy protection.

Make several versions (transcoding)

A video is a list of pictures, called frames, shown many times a second. A resolution is the number of rows of dots in a frame. "1080p" means 1,080 rows. A bitrate (lesson 16) is the bits per second that the video needs. A bigger picture needs a higher bitrate.

The uploaded file may be too big for a phone. Transcoding reads the video and writes it again with new settings, such as a smaller resolution or a lower bitrate. It uses a lot of CPU.

One rendition is one version of the video at one resolution and bitrate. The ladder is the list of renditions that we make. Our ladder, an assumption, has five renditions: 240p at 0.4 Mbps, 360p at 0.8, 480p at 1.5, 720p at 3 and 1080p at 5.

Cut into segments

A keyframe is a frame that is stored whole. Other frames store only what changed since an earlier frame. A player can start playing only at a keyframe.

So we cut each rendition into segments: small video files of a few seconds, each starting with a keyframe. Our segments are 6 seconds long (an assumption). A segment never changes after it is written. It is an ordinary file, and the phone gets it with a normal HTTP request (lesson 7). So a CDN can keep it (lesson 36).

Cutting also helps the workers. We cut the original into pieces of 6 seconds. Each worker makes all five renditions of one piece, and the pieces go to the workers through a queue (lesson 58). Many workers run at the same time.

The manifest

How does the phone know which files to ask for? It reads a small text file called a manifest. In the HLS format (HTTP Live Streaming, RFC 8216) there are two kinds:

• The master playlist lists the renditions and the bitrate of each.
• A media playlist lists the segments of one rendition, in order.

#EXTM3U
#EXT-X-STREAM-INF:BANDWIDTH=800000,RESOLUTION=640x360
360p/playlist.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=3000000,RESOLUTION=1280x720
720p/playlist.m3u8

Another format, MPEG-DASH, has the same idea with other names. In HLS, BANDWIDTH is the peak bit rate of that rendition.

The player chooses (adaptive bitrate)

The player keeps a buffer: the seconds of video that it has downloaded but not yet shown. If the buffer becomes empty, the video stops. This is a stall.

A segment of 6 seconds plays for 6 seconds. To keep up, the phone must download it in less than 6 seconds:

• download time = segment bitrate x 6 seconds / download speed

Adaptive bitrate (ABR) means that the player measures its download speed and picks the rendition for each next segment. It can change at every segment, because we cut every rendition at the same times. Here is a simple rule, our own, to teach with:

1. Pick the highest rendition whose bitrate is at most 80% of the measured speed. The other 20% is spare, in case the speed drops.
2. Start with a low rendition, so the first segment arrives fast. Then climb.
3. If the buffer is almost empty, go down.

Serve the files from a CDN

From lesson 36: the origin holds the real files, and edges keep copies near the users. For on-demand video the origin is object storage (lesson 24). Segments never change, so each gets a numbered name and a long time to live (TTL). The origin sends a segment to an edge once. An origin shield (lesson 36) joins the requests of all edges into one.

On demand: one video from upload to play

1. The creator uploads the original file to object storage, with a pre-signed URL (lesson 24).
2. A message goes into a queue (lesson 58). Workers transcode the pieces and write the segments and playlists to object storage. If a worker fails, the queue gives its piece to another worker.
3. The player reads the master playlist, the media playlist and then the segments, through the CDN.

Live

For a live match, the steps are:

1. The stadium sends one stream to an ingest server, the first server of our system.
2. A live transcoder makes the renditions as the video arrives.
3. A packager cuts each rendition into segments and updates the playlist after each segment.
4. The segments go to the origin, then through the shield and the edges to the players.

A live media playlist lists only the newest segments, and it changes every 6 seconds. The target duration is the longest segment length: 6 seconds here. After it loads a playlist that changed, a player must wait at least one target duration before it reloads (RFC 8216). So the playlist has a short TTL, for example 2 seconds. The segments keep a long TTL.

Live delay is how far behind the stadium the viewer is. A segment is ready only when all 6 seconds are done. Also, a player should not start nearer than three target durations to the end of the playlist, because it could stall (RFC 8216). So, in our simple model:

• live delay = 3 x segment length + the time to encode, pack and carry (4 seconds, an assumption)

Shorter segments give less delay and more requests.

Before the match (pre-warming)

Pre-warming means getting the system ready before the crowd arrives. There are three parts:

1. Warm the files that exist. Fetch each app file, picture and pre-show video once through every edge. A live segment cannot be warmed, because it does not exist yet.
2. Warm the path. Send a test stream well before the toss (the coin flip that starts a cricket match). Then every connection between the origin, the shield and the edges is open and tested.
3. Warm the capacity. Start the servers whose answers the CDN cannot keep, such as login. New servers take minutes (lesson 63). Tell the CDN provider the expected bandwidth ahead of time.

Assumptions (made up to teach with, not real traffic):

• 1,000 hours of new video are uploaded each day.
• Making all five renditions needs 4 CPU core-seconds for each second of video. A server has 16 cores.
• Segments are 6 seconds. The viewers are spread like this: 240p 5%, 360p 10%, 480p 50%, 720p 30%, 1080p 5%.
• 30,000,000 viewers at the peak. There are 50 edges, sharing the viewers equally (as in lesson 36).
• The round trip from a phone to its edge is 40 ms. A phone has a download speed of 4 Mbps at the start.

Step 1: the storage.

• The five renditions together have 0.4 + 0.8 + 1.5 + 3 + 5 = 10.7 Mbps.
• One hour of video: 10.7 Mbps x 3,600 seconds = 38,520 megabits = 4,815 MB, about 4.8 GB.
• One day: 1,000 x 4.815 GB = 4,815 GB, about 4.8 TB.

Step 2: the transcoding workers.

• Video seconds per day: 1,000 x 3,600 = 3,600,000. CPU: 3,600,000 x 4 = 14,400,000 core-seconds = 4,000 core-hours.
• Average busy cores: 4,000 / 24 = 167. Uploads arrive in bursts, so we plan for 3 times the average (lesson 14): 500 cores. 500 / 16 = 31.25, so 32 servers, plus 1 spare = 33.
• A film of 2 hours is 7,200 seconds. As one job: 7,200 x 4 = 28,800 core-seconds = 8 hours on one core.
• Cut into 6-second pieces: 7,200 / 6 = 1,200 jobs of 24 core-seconds each. With 200 cores working together: 28,800 / 200 = 144 seconds, about 2.4 minutes. We ignore the small cost of joining the results.

Step 3: the player. This answers the predict box.

• The 80% rule: 0.8 x 2.2 = 1.76 Mbps. The highest rendition at or below it is 1.5 Mbps (480p).
• A 480p segment takes 1.5 x 6 / 2.2 = 4.09 seconds, less than 6. The buffer grows by 6 - 4.09 = 1.91 seconds with each segment.
• If the player picks 720p (3 Mbps): 3 x 6 / 2.2 = 8.18 seconds. The segment plays for 6 seconds but needs 8.18 to arrive, so the buffer shrinks by 2.18 seconds with each segment.
• Assume the buffer starts with 12 seconds of video. 12 / 2.18 = 5.5 segments, and 5.5 x 8.18 = 45 seconds. After about 45 seconds the video stops.

The start. Start on 360p. Its segment is 0.8 x 6 / 8 = 0.6 MB. At 4 Mbps it needs 0.6 x 8 / 4 = 1.2 seconds. The first request on a new connection needs 3 round trips (lesson 36). We use it for the master playlist. Then 1 more for the media playlist and 1 for the segment: 5 round trips. Start time = 5 x 40 ms + 1,200 ms = 1,400 ms. A 1080p first segment is 5 x 6 / 8 = 3.75 MB and needs 3.75 x 8 / 4 = 7.5 seconds. The start would take 200 + 7,500 = 7,700 ms, which breaks our goal of 3 seconds.

Step 4: the live bandwidth.

• Average bitrate: 0.05 x 0.4 + 0.1 x 0.8 + 0.5 x 1.5 + 0.3 x 3 + 0.05 x 5 = 0.02 + 0.08 + 0.75 + 0.9 + 0.25 = 2.0 Mbps.
• Data out: 30,000,000 x 2 Mbps = 60,000,000 Mbps = 60 Tbps. Only the edges can send this: 60 / 50 = 1.2 Tbps each.
• Segment requests at the edges: one segment per viewer every 6 seconds, so 30,000,000 / 6 = 5,000,000 per second. Playlist reloads add the same number.
• The origin is asked once for each new segment. The five segments of 6 seconds are 0.3 + 0.6 + 1.125 + 2.25 + 3.75 = 8.025 MB. That is 8.025 x 8 / 6 = 10.7 Mbps, the ladder total. The origin sends 10.7 Mbps for any number of viewers.

Step 5: the crowd at each segment. A new segment is new to every edge, every 6 seconds. Assume the origin needs 1 second to answer, so all requests in that second are misses. Assume viewers ask evenly (real viewers are less even): 5,000,000 requests.

• Without joining: 5,000,000 reach the origin.
• Each edge joins its requests (lesson 35): 50 edges x 5 renditions = 250.
• With an origin shield: 5.

Step 6: the live delay. 3 x 6 + 4 = 22 seconds, under our limit of 30. With 2-second segments: 3 x 2 + 4 = 10 seconds, but 30,000,000 / 2 = 15,000,000 requests per second, 3 times more.

Step 7: the toss. Assumption (lesson 16): 80% of the viewers join in the 10 minutes before the toss. Joins = 0.8 x 30,000,000 / 600 = 40,000 per second. Each join sends 3 requests that the CDN cannot keep, such as login: 120,000 a second. One server does 350 per second at 70% (lesson 16): 120,000 / 350 = 343 servers, and 344 with a spare. They must run before the toss.

Step 8: the twist. The CDN says it can carry only 48 Tbps for us. We do less, as in lesson 63: we cap the highest rendition.

• Cap at 720p: the 5% on 1080p move to 3 Mbps. Average = 2.0 - 0.05 x 2 = 1.9 Mbps = 57 Tbps. It is too much.
• Cap at 480p: the 35% on 720p and 1080p move to 1.5 Mbps, so 85% are at 1.5 Mbps. Average = 0.02 + 0.08 + 0.85 x 1.5 = 1.375 Mbps = 41.25 Tbps. It fits, with 6.75 Tbps spare.

• Many renditions solve one size for every network. They cost storage (4.8 GB an hour) and CPU. I would start with three renditions and add higher ones when a video becomes popular.
• Short segments solve delay: 10 seconds, not 22. They cost more requests (15,000,000 per second, not 5,000,000). I would use them only for a match.
• A CDN with a shield solves the origin load: 10.7 Mbps, not 60 Tbps. It costs a bill for each GB sent. I would add a second CDN when one cannot carry the peak.
• Capping the highest rendition solves a CDN limit (41.25 Tbps, not 60). It costs picture quality for 35% of the viewers. I would do it only for the peak.

"Streaming means the server sends one continuous video to each viewer." The phone downloads small files with normal HTTP, and the CDN keeps them. The origin sends 10.7 Mbps; the viewers receive 60 Tbps.

"The player should pick the rendition that matches its speed." Then a small drop in speed empties the buffer. Leave some spare speed.

"We can warm the CDN before the match, so no edge is ever empty." A live segment does not exist before it is made. It is new at every edge every 6 seconds. The shield and joined requests handle this: 5 requests, not 5,000,000.

1. What are a segment, a manifest and adaptive bitrate? Why can a CDN keep segments?
2. A film of 90 minutes has a ladder of 0.5, 1.5 and 4 Mbps. Making all three needs 3 core-seconds for each second of video. (a) How many MB do the three renditions take? (b) How long does one core need, and how long do 100 cores need? (c) A phone has 2.5 Mbps, and segments are 4 seconds. Which rendition does the 80% rule pick, and how long does one segment take to download?
3. A live match has 10,000,000 viewers. The extra time to encode, pack and carry is 4 seconds. The business wants a delay of at most 15 seconds. What segment length do you choose, and what does it cost in requests compared with 4-second segments?