A long LLM response can take many seconds to finish. If your app shows a spinner for that whole time, the feature feels broken even when it works. Streaming fixes the perception problem: the user sees the first words quickly and reads along while the rest arrives.

It also adds real complexity. You now have a long-lived connection, partial states, and failures that happen halfway through an answer. This post walks through the pieces a small app needs, in the order you will hit them.

Decide whether to stream at all

Streaming is worth it when the output is prose the user will read as it arrives. It is usually the wrong choice when:

  • The output is machine-readable. If your app parses JSON from the model, you need the whole thing before you can validate it. See getting structured output your app can trust for that path.
  • The output is short. A one-word classification or a short title gains nothing from streaming.
  • You must check the output before showing it. Once text is on screen, you cannot take it back cleanly. If you run a filter or validation pass on the result, streaming means showing text before that pass has run.

Before you build anything, measure two numbers for your real prompts: time to first token and total response time. If the gap between them is small, streaming will not change how the feature feels.

The shape of the pipeline

The app should not call the model provider directly, because that would mean shipping your API key in the binary. The usual shape is:

  1. The app sends a request to your own small backend.
  2. The backend calls the provider with streaming enabled.
  3. The backend relays chunks to the app as they arrive, typically as server-sent events (SSE).

Do not pass the provider's raw event format through to the app. Translate it into a small format you own, so that a provider change or a model swap does not require an app update. Three event types cover most features:

data: {"type":"delta","text":"Hel"}

data: {"type":"delta","text":"lo there"}

data: {"type":"done","reason":"stop"}

data: {"type":"error","code":"upstream_timeout"}

The explicit done event matters. Without it, the client cannot tell a finished answer from a dropped connection. Include the finish reason, so the app knows when the answer was cut off by the output token limit and did not end naturally.

Stop your infrastructure from buffering

The most common streaming bug is a stream that works locally and arrives in one lump in production. Something between your backend and the phone is buffering. Check these:

  • Reverse proxy buffering. Nginx buffers proxied responses by default.
  • Compression. Gzip on a streaming response can hold chunks back until a buffer fills. Disable it for the streaming route.
  • Timeouts. Idle and read timeouts tuned for quick JSON endpoints will cut long generations short.
  • Your framework. Some server frameworks buffer response writes unless you flush explicitly.

For Nginx, a dedicated location for the streaming route keeps the changes contained:

location /v1/stream {
    proxy_pass http://app_backend;
    proxy_http_version 1.1;
    proxy_buffering off;
    gzip off;
    proxy_read_timeout 120s;
}

Set the response Content-Type to text/event-stream and add Cache-Control: no-cache. Then test with curl -N against production, not just localhost, and watch whether the chunks arrive one at a time.

Parse the stream on the client

On iOS, URLSession can iterate over a response line by line, which is enough for a simple SSE format like the one above:

let (bytes, response) = try await URLSession.shared.bytes(for: request)
guard (response as? HTTPURLResponse)?.statusCode == 200 else {
    throw StreamError.badStatus
}

for try await line in bytes.lines {
    guard line.hasPrefix("data: ") else { continue }
    let payload = Data(line.dropFirst(6).utf8)
    let event = try JSONDecoder().decode(StreamEvent.self, from: payload)

    switch event.type {
    case .delta: buffer.append(event.text ?? "")
    case .done:  finish(reason: event.reason)
    case .error: fail(code: event.code)
    }
}

On Android, OkHttp has an SSE module, or you can read the response body line by line in a coroutine. Either way, keep the parser in one small class that emits typed events. The UI layer should never see raw lines.

If the loop ends without a done event, treat the answer as incomplete, even though no error was thrown.

Make the UI calm

Tokens can arrive faster than the screen needs to redraw. Updating a text view on every chunk wastes work and can make scrolling stutter.

  • Throttle rendering. Append incoming text to a buffer and flush it to the UI on a short timer or once per frame. Profile on an older device to pick the interval.
  • Autoscroll only when the user is at the bottom. If they scrolled up to reread something, do not pull them back down.
  • Handle partial formatting. If you render Markdown, a half-arrived list or code fence will flicker between states. Either render plain text until the stream is done, or use a renderer that tolerates unclosed syntax.
  • Do not announce every chunk to screen readers. Announce that generation started, then announce the finished text. A live region that fires on every token is unusable.
  • Show a stop button in place of the send button for as long as the stream is running.

Cancellation has to reach the provider

A stop button that only hides the text is not cancellation. If the user stops, navigates away, or the app goes to the background, the cancel should travel the whole chain:

  1. The app cancels the network task, which closes the connection.
  2. Your backend detects the client disconnect.
  3. Your backend aborts the upstream request to the provider.

Step three is the one that gets skipped. If your backend keeps reading the upstream response after the client has gone, you may keep paying for output nobody will see. How providers bill aborted requests varies, so check your provider's documentation and confirm against your usage dashboard. Test it directly: start a long generation, kill the app, and check your backend logs to see when the upstream request actually ended.

Plan for mid-stream failures

With a non-streaming call, a request either succeeds or fails. With streaming there is a third outcome: half an answer followed by an error. Mobile networks make this routine.

  • Keep the partial text. Deleting what the user was already reading is worse than leaving it.
  • Mark it as incomplete with a visible note and a retry action.
  • Retry replaces the partial answer. A retry generates a fresh answer, so appending it to the old partial text produces duplicated or contradictory content.
  • Do not auto-retry silently. Each retry is a new paid request, and a retry loop on a flaky connection adds up.
  • Do not persist incomplete answers as complete. If you save conversation history, store the completion state with it.

These are the streaming-specific cases of a broader design problem, covered in designing LLM features that fail gracefully.

What to measure

Log a few numbers per request on your backend so you can tell whether streaming is doing its job:

  • Time to first token, measured from when the request arrives.
  • Total stream duration.
  • How the stream ended: done, client cancel, upstream error, or timeout.
  • Output tokens generated, including for cancelled requests.

A high cancel rate early in responses can mean answers are too long or start too slowly. A cluster of timeouts at the same duration usually points to a proxy or load balancer limit you have not raised yet.

Conclusion

Streaming is mostly plumbing, and each piece is small: an event format you own, a proxy that does not buffer, a client parser that emits typed events, a throttled UI, cancellation that reaches the provider, and an honest incomplete state. Build them in that order and test each against a real device on a poor connection. The feature will then feel fast without costing you correctness or money.