Serving an AI feature: streaming, timeouts, fallbacks and limits

You add a chat helper to an app. The interviewer asks: "The model takes 4 seconds to answer. What does the user see? What if the model service is down? What if 10,000 users arrive on a sale day?"

A weak answer says "call the API and show the result". A strong answer shows the first words early, and gives each wait its own time limit. It names a second model for the day the first one fails. It also says what the user sees when both fail, and how the system stays inside the provider's limits.

At SDE-1 level you should know that showing the answer piece by piece makes it feel faster. At SDE-2 level you should set time limits and a fallback. At SDE-3 level you should count the provider limits and size a waiting line with arithmetic.

A friend tells you a long story on the phone. You hear the first word at once, and the story takes as long as it takes. Now think of a friend who writes the whole story in a letter. You hear nothing until the letter arrives. The story is the same. The first way feels much faster.

An AI feature calls a model that you do not own, over the network. So it has the problems of lesson 54: the call is slow, or it fails. Lesson 118 added two more: the answer is long, and the provider limits how much you may use.

Streaming

A call has two times (lesson 118). The time to first token is the wait before the first piece of the answer arrives. The total time is the time until the last piece arrives.

• total time = time to first token + output tokens x time per token

Streaming means that the provider sends the answer in small pieces while the model writes it. Your server passes each piece to the browser at once. The usual way is Server-Sent Events (SSE), the one-way channel of lesson 75: the server writes, and the browser reads. The user sees words after the time to first token, not after the total time. Streaming does not make the total time shorter. It makes the wait feel shorter.

Streaming has three costs:

1. A connection stays open for the whole answer (lesson 75). Count them with the formula of lesson 54: requests in progress = requests per second x seconds each one takes.
2. An error can come after the answer has started. The status code is already 200 ("fine") when the first piece goes out. The provider documents that an error can still come after a 200 in a stream. So your code must watch for a broken stream.
3. You cannot check the whole answer before the user sees the first part.

Three time limits

A single timeout is not enough for a stream. Use three. This is design advice, not a standard.

Each limit catches a failure that the others miss. A stream that sends one token every few seconds never reaches the stall timeout. Only the deadline stops it. Measure the time to first token for the model you use.

When a call fails

The provider may answer 429 (too many requests) or a busy error such as 503 or 529. One of your time limits may also end. Use the rules of lesson 54: wait for the Retry-After time if there is one, use backoff and jitter, and retry at one layer only.

Before the first token, nothing has reached the user, so you can try again or switch to another model. After the first token, the user has read part of an answer. Do not join the text of two models. Show what you have and tell the user that the answer stopped, with a button to try again.

A fallback, and an honest error

A fallback is a second route that you use when the first one fails. It can be a second model, or a second provider. Four points:

• Try the first route once. If it fails before the first token, go to the second at once. Do not spend three tries on a route that is down.
• Test your prompt on both. The second model may answer in a different style, and it may count tokens in a different way (lesson 118).
• Two routes help only if they fail at different times. If both run in the same data centre, one outage can stop both.
• If both fail, tell the truth: "The helper is busy. Please try again in a minute." Show something that does not need the model, such as the order status from your database. Never make up an answer. A guessed delivery time is a false promise.

The provider's limits

A provider limits how much you may use. One provider measures requests per minute (RPM), input tokens per minute (ITPM) and output tokens per minute (OTPM). If you go over any one of them, it answers 429, and it may send a Retry-After header. The limits cover all your users together. The provider says it uses a token bucket (lesson 56), so capacity comes back little by little. A limit of 60 per minute may also be enforced as 1 per second. So a short burst can fail even when the minute total is fine.

Because the limits are shared, one user who sends too much can use the capacity of everyone. Use a rate limiter (lesson 56) with the user ID as the key.

Queueing

When demand is above the limit, you can put calls in a queue: a waiting line. The queue lets calls out at the rate that the limit allows, like the leaky bucket of lesson 56.

• longest wait = calls in the queue / calls let out per second

Set a longest wait that a user accepts. The queue may hold only rate x longest wait calls. When it is full, answer "busy" at once. A queue that grows without end gives every user a long wait and a useless answer (lesson 54: a deadline). A queue holds back a short burst. It does not fix demand that stays above the limit.

SwadKart, the food app of lesson 118, puts its help chat on a sale day.

Assumptions (made up to teach with, not any provider's numbers):

• An answer has 150 tokens. The time to first token is 0.5 seconds. Each token takes 0.02 seconds.
• On average a call sends 1,450 input tokens (lesson 118: 7,250 tokens over 5 turns).
• At the peak, 12 calls arrive each second. That is 12 x 60 = 720 calls a minute.
• The provider limits SwadKart to 1,000 requests and 400,000 input tokens a minute.
• A second provider allows 300,000 input tokens a minute.
• Each provider is down 0.5% of the time, and they fail independently (one failure does not cause the other).
• The API server has 200 workers (lesson 54).

Step 1: streaming (the predict box). The total time is 0.5 + 150 x 0.02 = 3.5 seconds. Without streaming, the first word appears at 3.5 seconds. With streaming, it appears at 0.5 seconds, which is 3.5 / 0.5 = 7 times sooner. The answer still ends at 3.5 seconds. Each chat holds a connection for 3.5 seconds, so 12 x 3.5 = 42 connections are open.

Step 2: the time limits. Assume the p99 time to first token is 1.5 seconds and the p99 gap between pieces is 0.3 seconds. A p99 answer has 600 tokens, so its total time is 0.5 + 600 x 0.02 = 12.5 seconds.

• First-token timeout: 3 seconds, which is 2 x 1.5.
• Stall timeout: 2 seconds.
• Total deadline: 20 seconds.

Now the provider stops answering. With no limit, the 200 workers are all stuck after about 200 / 12 = 16.7 seconds. With the 3-second limit, the connections in progress are 12 x 3 = 36. Next, a stream sends one token every 1.9 seconds. It never reaches the 2-second stall timeout. At that speed, 150 tokens would take 150 x 1.9 = 285 seconds. The 20-second deadline stops it after 20 / 1.9 = 10 tokens.

Step 3: the fallback. The first provider fails before the first token. The first-token timeout ends at 3 seconds. The second provider needs 0.5 seconds to start. The user sees the first word at 3 + 0.5 = 3.5 seconds. Compare a second try on the first provider before the switch. A backoff wait (assume 0.2 seconds) and a second 3-second timeout come first: 3 + 0.2 + 3 + 0.5 = 6.7 seconds.

Step 4: the benefit. One provider is down 0.5% of 43,200 minutes in a 30-day month, which is 216 minutes. Both are down together for 0.005 x 0.005 = 0.000025 of the time, which is 0.000025 x 43,200 = 1.08 minutes. This holds only if they fail independently, which real outages often do not. If both fail, the user sees the honest message and the order status page.

Step 5: the limits. Each call sends 1,450 tokens, so 720 calls a minute send 720 x 1,450 = 1,044,000 tokens. The request limit is fine, because 720 is below 1,000. The token limit is 400,000, so it stops us first. The most calls a minute are 400,000 / 1,450 = 275 (rounded down). That is 275 / 60 = 4.58 calls a second. Without a queue, 720 - 275 = 445 calls get 429. That is 445 / 720 = 62%.

Step 6: the queue. Say that a user accepts a 10-second wait. The queue may hold 4.58 x 10 = 45.8 calls, which we round down to 45. It grows by 12 - 4.58 = 7.42 calls a second, so it is full after 45 / 7.42 = 6 seconds. The last call waits 45 / 4.58 = 9.8 seconds. After 6 seconds, new calls get the honest "busy" message. The queue helped only for a short burst.

Step 7: the second provider. It adds 300,000 / 1,450 = 206 calls a minute (rounded down). Together: 275 + 206 = 481 calls a minute, which is 8 a second. Demand is 12 a second, so 720 - 481 = 239 calls a minute still fail, which is 33%. A second provider helps with limits, but here it does not remove them. We would also ask for higher limits.

• Streaming solves a long wait before anything shows (3.5 seconds became 0.5 seconds). It costs open connections, errors after the answer has started, and no check of the whole answer. I would stream any answer that takes over a second.
• Three time limits solve stuck workers and endless streams. They cost some false failures if a limit is too short. I would set each one from the p99 of its own part.
• A fallback solves a provider outage (216 minutes became 1.08 minutes). It costs a second prompt to test and answers in a different style. I would add it when the feature is on the main path of the product.
• A queue solves a short burst above the limit. It costs a wait for each user. I would size it from rate x longest wait, and refuse when it is full.

"Streaming makes the model faster." It does not. The total time was 3.5 seconds with and without it. Only the first word arrives sooner.

"One timeout on the whole call is enough." A stream that sends one token every 1.9 seconds never reaches a stall timeout. A call that never starts needs a first-token timeout. You need all three limits.

"If the model fails, show a default answer so the page is not empty." A made-up answer is worse than an honest one. Say that the helper is busy.

1. What is the time to first token? Why does streaming not shorten the total time? Name the three time limits.
2. An answer has 200 tokens. The time to first token is 0.4 seconds and each token takes 0.025 seconds. A call sends 2,000 input tokens. The provider allows 600,000 input tokens a minute. (a) When does the first word appear with streaming, and without? (b) How many calls a minute does the token limit allow? (c) Demand is 8 calls a second. The longest wait is 10 seconds. How many calls wait after 10 seconds, and when is the queue full?
3. The first model fails. A team shows a ready-made answer, "Your order will arrive in 30 minutes", so the page does not look empty. What do you say?