Most LLM features get built for the happy path. The prompt works, the response looks good, and the demo goes well. Then real users arrive with slow connections, long inputs and odd edge cases, and the provider has a bad afternoon. The spinner keeps spinning, or the screen shows half-parsed JSON.

A model call is the least reliable dependency most small apps have. It is slower than your own API, its output isn't deterministic, and it can be wrong without raising an error. That doesn't mean you shouldn't ship it. It means you should decide in advance what the app does when the call goes badly. This post covers how to do that without overbuilding.

Start by naming the failure modes

Before you write any handling code, list the ways a model call can fail. For most features the list looks like this:

  • Slow: the response arrives, but long after the user has stopped waiting.
  • No response: the network drops, the request hangs, or the device is offline.
  • Error: rate limits, quota exhaustion, server errors, or an expired API key.
  • Malformed: you asked for JSON and got prose, truncated JSON, or valid JSON with missing fields.
  • Refused: the model declines, often on harmless input that looks sensitive.
  • Plausible but wrong: well-formed output that is factually or contextually incorrect.

The first five are detectable at runtime. The last one usually isn't, which is why you need an offline evaluation step as well as runtime handling. If you haven't set one up, building a small eval set before you ship is the companion to everything below.

Decide what the feature does without the model

For every LLM feature, answer one question: if the model disappeared for a day, what would this screen do? The answer is your fallback. In practice there are a few options:

  1. A non-AI version of the same feature. Smart search falls back to plain keyword search. Auto-tagging falls back to manual tags. Summaries fall back to the first few lines of the original text.
  2. A cached earlier result. If the user already generated something for this input, show it again, labelled as such.
  3. Hide the entry point. If the feature is an optional extra, just don't show the button while it's unavailable.
  4. An honest error with a retry. This is the last resort, not the default.

Features with a clear non-AI fallback are much easier to ship. If you can't think of one, ask whether the model sits on a path the user can't avoid. If it does, the reliability bar is far higher.

Set timeouts from measured latency

The default timeout in many HTTP clients is long, sometimes a minute or more. No user waits that long for a suggestion. Set your own timeout, and base it on data rather than a guess:

  1. Log the latency of every model call during beta testing, with input size alongside it.
  2. Look at the slow tail (for example the 95th percentile), not the average.
  3. Set the timeout a bit above that tail, then check it against how long the UI can reasonably make someone wait.

If the slow tail is longer than users will tolerate, a timeout won't fix it. Shorten the input, ask for shorter output, stream the response, or pick a faster model.

Wrap the call so the rest of your code gets a small, explicit set of outcomes and never sees raw exceptions:

type SummaryResult =
  | { kind: 'ok'; value: Summary }
  | { kind: 'timeout' }
  | { kind: 'busy' }      // rate limited or overloaded
  | { kind: 'invalid' }   // response failed validation
  | { kind: 'error' };

async function summarize(text: string, timeoutMs: number): Promise<SummaryResult> {
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), timeoutMs);
  try {
    const raw = await callModel(text, { signal: controller.signal });
    const parsed = parseSummary(raw);
    return parsed ? { kind: 'ok', value: parsed } : { kind: 'invalid' };
  } catch (err) {
    if (controller.signal.aborted) return { kind: 'timeout' };
    if (isRateLimit(err)) return { kind: 'busy' };
    return { kind: 'error' };
  } finally {
    clearTimeout(timer);
  }
}

The UI then switches on kind, and every case has a defined behaviour. When a new failure shows up in production, you add a case and the type checker shows you every screen that has to handle it.

Retry carefully

Retries help with brief network errors and occasional server errors. They hurt in other cases:

  • Rate limits: retrying immediately makes things worse. Respect any retry-after signal the provider sends, or back off with jitter.
  • Timeouts: a retry doubles the wait. Usually it's better to show the fallback and offer a manual retry.
  • Invalid output: one retry is reasonable, since the same prompt can produce valid output on a second try. If it keeps failing, the prompt or the input is the problem, and more retries just add cost.

Cap the total number of attempts. Every retry is another billed call, so failures can quietly increase spend. Keeping LLM feature costs predictable covers how to budget for that.

Validate output before it reaches the screen

Treat model output as untrusted input. Even with structured output modes, check it before you render it:

  • Parse it against a schema, and reject missing or extra fields.
  • Check lengths. A title that runs to 400 characters is a failure even if it is valid JSON.
  • Check enums. If you asked for one of five categories, reject anything else instead of displaying it.
  • Check that references point to real things. If the model returns an item ID, confirm the item exists.

When validation fails, return invalid and fall back. Don't try to repair broken output with string manipulation. That code grows into something nobody wants to maintain.

Design honest UI states

Each outcome needs a UI state, and the copy matters more than you might expect. Some guidelines:

  • Loading: show progress that matches what's happening. If you stream, render tokens as they arrive. If you don't, a short message beats a bare spinner.
  • Partial: if a stream breaks midway, either keep the partial text with a clear marker that it's incomplete, or discard it. Never present it as finished.
  • Degraded: when the fallback is active, say so briefly, for example: Smart suggestions are unavailable right now. Showing recent items instead.
  • Failed: say what happened in plain words and give one action, usually a retry. Leave out provider error messages and status codes.

Never let a failed model call block the rest of the screen. Load the AI part on its own, so the core content shows up even if that call never finishes.

Add a remote kill switch

When a provider has an outage, or you find a bad prompt in production, you don't want to wait for an app review to switch the feature off. Put the key parameters in remote config that the app fetches on launch:

{
  "ai_summary": {
    "enabled": true,
    "timeout_ms": 8000,
    "max_attempts": 2,
    "prompt_version": "summary-v3"
  }
}

The values here are placeholders. Set them from your own measurements. Make the app's built-in defaults safe, so that if the config fetch fails the feature falls back gracefully and nothing crashes. Test the enabled: false path before launch, not during an incident.

Log failures without logging users

To improve your handling you need to know which failures actually happen and how often. Log the outcome kind, latency, input length, prompt version and model name. Don't log the prompt or response content by default. It's rarely needed for this kind of analysis, and it creates a privacy obligation. A privacy checklist for AI features goes into more detail.

A simple weekly view is enough: the share of calls by outcome, broken down by app version and prompt version. A rise in invalid after a prompt change tells you more than any user report will.

Test the failures on purpose

Failure paths that never run during development will break in production. Add a debug setting that forces each outcome:

  • Force a timeout by pointing at an endpoint that never responds, or by setting the timeout to a few milliseconds.
  • Force invalid by returning a canned malformed response.
  • Force busy by returning a rate-limit error from a stub.
  • Turn on airplane mode and go through the whole flow.

Walk through every state on a real device at least once before release, and again whenever you change the prompt or the model.

A short pre-ship checklist

  • Every failure mode is listed, and each one maps to a UI state.
  • The feature has a fallback that doesn't need the model.
  • The timeout is based on measured latency.
  • Retries are capped and skip rate limits and long timeouts.
  • Output is validated against a schema and sensible limits.
  • A remote kill switch exists, and you have tested it.
  • Logs record outcomes and latency, not user content.
  • Every failure state has been forced and checked on a device.

Wrapping up

Reliability for an LLM feature doesn't come from the model. It comes from the code around it: a clear list of outcomes, a fallback you can live with, limits on waiting and retrying, and a way to turn the feature off quickly. None of this takes long to build, and it's much easier before launch than in the middle of an incident. If the app still works when the model doesn't, you can afford to try more ambitious ideas with it.