Object storage and pre-signed uploads

Many apps let users upload a photo, a video or a document. So interviewers ask: "Where do you store the files? How does the upload work on a slow phone network?"

A weak answer is "I will save the file in the database". Another weak answer is "my server receives the file and saves it". A strong answer keeps the bytes in a store made for files, and only a small record in the database. The phone sends the file straight to that store. The answer also says what happens when the connection breaks in the middle of a video.

At SDE-1 level you should know why files stay out of the database. At SDE-2 level you should explain how a phone can send a file straight to the store with a short-lived permission link. At SDE-3 level you should also explain how a big upload is cut into parts and resumed, and the cost of each choice.

Think about a cloakroom at a wedding hall. You hand over a heavy bag and get a token with a number. Later, you show the token and get the same bag. The hall also keeps a small notebook that says whose token is whose. The big bags stay in the cloakroom. Only the small notes go in the notebook.

Files work in the same way. A blob (binary large object) is a file stored as raw bytes, such as a photo or a video. A database row is about 1 KB (lesson 14). A 3 MB photo is 3,000 times bigger, and a 50 MB video is 50,000 times bigger. Blobs in rows make the database large and slow to copy for safety.

Object storage

Object storage is a service made to keep blobs. You rent it, as in lesson 13. It has these parts:

• An object is the file bytes, plus a few facts about it, such as its size and its content type. A content type is a label like image/jpeg that says what kind of file it is.
• A bucket is a named container for objects.
• An object key is the name of one object inside its bucket, for example uploads/user-81/p-4f2a.jpg. The slashes are only part of the name. In most object stores there are no real folders.
• You use it over HTTP (lesson 7). PUT means "store this object". GET means "give me this object". DELETE removes it.

Three more facts matter. First, you write an object whole. In most object stores you cannot change a few bytes in the middle. A new object with the same key replaces the old one. Second, you cannot ask "find the photos of Pune". You can only ask for a key, or for all keys with the same beginning. Third, the provider keeps copies for you (lesson 15) and charges for space, requests and data sent out (lesson 13).

So the app keeps two things. The bytes go in the object store. The metadata (facts about the file) goes in the database: the owner, the object key, the size, the content type and a status.

Two ways to upload

Way 1: through the app server. The phone sends the file to your server, and your server sends it on to the object store. Every byte passes through your server, which holds a connection open for as long as the slow phone takes.

Way 2: a pre-signed URL. A pre-signed URL is a web address. It gives permission to do one thing to one object for a short time. The steps are:

1. The phone asks your app: "I want to upload a 3 MB image/jpeg." The phone sends its login cookie (lesson 7).
2. The app checks the login, the size and the type. It chooses the object key and saves a row with the status "uploading". It makes a pre-signed URL for PUT, for that one key, valid for a short time such as 15 minutes.
3. The app sends the URL back to the phone.
4. The phone sends the bytes straight to the object store, using that URL.
5. The phone tells the app "done". The app asks the store for the object's size and content type, checks them, and sets the status to "ready".

The object store must trust the URL without asking your app. So the URL carries a signature. This is a short code made from the details in the URL and a secret key. The details are the key, the action and the expiry time. Only your app and the object store know the secret key. The store makes the same code again. If anyone changes the details, the codes differ and the store refuses. Use HTTPS (lesson 7), so nobody can read the URL on the way.

Two warnings. A pre-signed URL works for anyone who holds it until it expires, so keep it short-lived and out of logs. Also, the app never sees the bytes, so it checks the file only after it arrives (step 5). Do not trust the size or type that the phone claims.

The same idea works for private downloads: the app checks the login, then gives a GET URL for a few minutes.

Multipart and resumable uploads

On a phone network, the connection breaks often. If one PUT carries a 50 MB video and the link breaks at 35 MB, all 35 MB are lost.

A multipart upload cuts the file into parts and sends each part as its own request. The steps are:

1. Start. The store returns an upload id.
2. Send the parts. Each part has a number. The store returns a receipt code for each part. The phone keeps the list of part numbers and receipt codes. Parts can go at the same time.
3. Complete. The phone sends the list. The store joins the parts, in order, into one object.

With pre-signed URLs, the app starts the multipart upload and signs one URL for each part. At the end, the phone sends its list to the app, and the app completes the upload. If a part fails, the phone sends only that part again. To resume after a long break, the phone looks at its list and sends the missing parts. If the URLs have expired, the app signs new ones for the missing parts only. The store can list the parts it holds, but keep your own list.

Some stores offer a resumable upload instead. It has no parts that you number. The steps are:

1. The phone asks the store to start a session. The store returns one session address for this one object.
2. The phone sends the bytes to that address, in one go or in chunks.
3. After a break, the phone asks the address how many bytes arrived. Then it sends only the rest.

A resumable upload is simpler for the phone, because the store keeps the count.

Part size. Small parts lose little when one fails but need more requests. Large parts do the opposite. Amazon S3 (see Sources) wants parts of at least 5 MiB (a little more than 5 MB), except the last part. It allows at most 10,000 parts. So for a 100 GB file, each part must be at least 10 MB.

If you start a multipart upload and never finish it, the store keeps the parts and you pay for them. So cancel unfinished uploads, or set a rule that cancels them after some days.

A photo and video app lets users upload. We compare the two ways.

Assumptions (made up to teach with, not real traffic):

• 20,000,000 photos of 3 MB each per day, and 2,000,000 videos of 50 MB each per day.
• A phone sends at 4 megabits per second = 4 / 8 = 0.5 MB per second.
• The peak is 3 times the average (lesson 14). A server has a 1 Gbps link, used up to 70% (lesson 15).
• A metadata row is 1 KB. An app server needs 2 ms of CPU time to make a pre-signed URL or to check an upload. It has 4 cores.
• Pre-signed URLs last 15 minutes. Parts are 8 MB.

Step 1: why not the database.

• Photos: 20,000,000 x 3 MB = 60,000,000 MB = 60 TB a day. Videos: 2,000,000 x 50 MB = 100,000,000 MB = 100 TB a day. Total 160 TB a day.
• Metadata: 22,000,000 rows x 1 KB = 22,000,000 KB = 22 GB a day. That is 22 / 160,022 = 0.014% of the bytes. The database keeps these 22 GB, and the object store keeps the 160 TB.

Step 2: way 1, through the app servers.

• Data in = 160,000,000 MB / 86,400 = 1,852 MB per second = 14,815 megabits per second = 14.8 Gbps. At peak: 14.8 x 3 = 44.4 Gbps.
• Servers for the links: 44.4 / (1 x 0.7) = 63.5, so 64, plus 1 spare = 65 servers.

Step 3: way 2, a pre-signed URL.

• Uploads at peak: photos (20,000,000 / 86,400) x 3 = 694 per second. Videos (2,000,000 / 86,400) x 3 = 69 per second. Total 764. Each upload makes 2 calls to the app: ask for the URLs (one answer holds all of them), and say "done". That is 764 x 2 = 1,528 calls per second.
• One server: 4 x 1,000 / 2 = 2,000 calls per second, and 1,400 at 70%. Servers = 1,528 / 1,400 = 1.09, so 2, plus 1 spare = 3.
• 65 / 3 = 21.7, so about 22 times fewer servers.

Step 4: one video that breaks. The video is 50 MB, so it takes 100 seconds. The link breaks after 35 MB. That is 35 / 0.5 = 70 seconds in.

• One upload. All 35 MB are lost. The phone starts again: 70 + 100 = 170 seconds.
• 8 MB parts. 50 / 8 = 6.25, so 7 parts (six of 8 MB and one of 2 MB). Parts 1 to 4 are done: 32 MB. Part 5 was 3 MB in, so 3 MB are lost. The phone sends from part 5: 50 - 32 = 18 MB = 36 seconds. Total = 70 + 36 = 106 seconds.
• Saved: 170 - 106 = 64 seconds (38%).

This answers the predict box: (a) 35 MB, (b) 3 MB.

The parts cost 1 start, 7 part requests and 1 complete: 9 requests to the store instead of 1. Part 5 is sent twice, so there is one more.

• Object storage solves the size, cost and copies of big files. It costs edits in place, searching by content, and a fee for each request. I would change it when files are tiny and very many, because request fees can then cost more than the space.
• A pre-signed URL solves the load on the app: 3 servers instead of 65. It costs a URL that works for anyone who holds it, two more calls, and checks only after the upload. I would upload through the app server when the app must look at every byte first.
• Parts solve the loss when a link breaks: 106 seconds instead of 170. They cost more requests (at least 9 instead of 1), a list that the phone keeps, and billed storage for unfinished uploads. I would use one upload for a small file, such as a 3 MB photo.

"Saving the file in a database column is simpler." In the example, 160 TB a day would sit in the database. It should hold only 22 GB of facts.

"A pre-signed URL is secure, so a long expiry is fine." Anyone can forward a 7-day link and use it for 7 days. That is 7 x 24 x 60 = 10,080 minutes, 672 times longer than 15 minutes.

"Parts make an upload faster." Not by themselves. The phone link of 0.5 MB per second is the limit. Parts help when a break happens (106 instead of 170 seconds).

1. Why do files stay out of database rows? What is a pre-signed URL? Name the three steps of a multipart upload.
2. A phone sends a 120 MB video over a link of 0.75 MB per second. The link breaks after 80% of the file. Parts are 10 MB. (a) How long does a clean upload take? (b) With one single upload, how many MB are lost, and what is the total time? (c) With parts, how many MB are lost, and what is the total time?
3. A teammate creates a pre-signed download link for a private photo with a 7-day expiry, and shows it in a public group chat. What is the risk? What do you change?