Your LLM feature works. Users rely on it, your prompt is tuned, and your parser handles what comes back. Then the provider announces a new model, or a retirement date for the one you use. Switching looks like a one-line change. Often it isn't.

What a feature does depends on the prompt, the model and the generation parameters together. Change the model and the feature can change too, even if the new model is better overall. This post covers how to handle a model change the way you would handle any other dependency upgrade: pin the version, compare both models, roll out gradually and keep a way back.

Why a better model can still break your feature

Model upgrades rarely fail in obvious ways. The common problems are small changes that your code or your users depend on without anyone noticing:

  • Formatting drift. The new model wraps JSON in a code fence, adds a preamble, or uses a different style of bullet points.
  • Length changes. Answers get longer or shorter. That affects your layout, your truncation logic and your token bill.
  • Different refusal behavior. Requests the old model handled may now get hedged or declined, and the reverse can also happen.
  • Tool-calling differences. The model may call tools more or less often, call them in a different order, or fill in optional arguments differently.
  • Latency and cost. Per-token prices, tokenization and time to first token can all change, sometimes in opposite directions.
  • Tone. Users who are used to the feature's voice will notice when it changes, even if no single answer is worse.

Generic benchmarks catch none of these. Only your own inputs, checked against your own expectations, will show them.

Step 1: Pin the model and make it configuration

If your code uses an alias like latest, the provider can change your feature without telling you. Use a dated or versioned model identifier when the provider offers one, and keep it in configuration, not scattered through your code. Keep the prompt version and parameters in the same place, because the three only work as a set.

features:
  summarize_note:
    model: provider-model-2026-06-01
    prompt_version: v7
    max_output_tokens: 400
    temperature: 0.2
    candidate:
      model: provider-model-2026-10-15
      prompt_version: v7
      percent: 0

The candidate block is where the next model waits. With percent: 0 it does nothing in production, but your evaluation scripts can still read it. If your app talks to the provider through a server, as described in A Thin Proxy Between Your App and the LLM API, this config lives on the server and you can change it without shipping an app update. That matters a lot here.

Step 2: Run your eval set against both models

This is where an existing eval set pays off. If you don't have one yet, Build a Small Eval Set Before You Ship an LLM Feature explains how to put one together. Run every case through both the current model and the candidate, using the same prompt and parameters, and save the outputs next to each other.

Don't stop at pass/fail counts. Produce a diff report you can actually read:

  1. Cases that passed before and fail now. These block the upgrade until you understand them.
  2. Cases that failed before and pass now. These are useful, but check that they aren't passing by accident.
  3. Cases that pass on both models but produce visibly different output. Read a sample of these yourself. Tone and formatting drift tend to show up here.

Run each model more than once if your temperature is above zero. A difference that shows up in only one run out of several is noise. A difference that shows up in every run is real.

Step 3: Measure what the eval doesn't score

Quality is only part of the picture. For each model, log these numbers across the whole eval run:

  • Output tokens per request: median and the high end. Use the token counts the provider reports in its responses, not your own estimate.
  • Input tokens per request. The same prompt can tokenize differently on a new model.
  • Latency: time to first token and total time, measured from your server.
  • Parse failure rate. How often your structured-output validator rejects a response.
  • Refusal or fallback rate. How often your code ends up on its graceful-failure path.

Multiply the token counts by the provider's current published prices to get the cost per request for each model. Check the pricing page on the day you do this instead of relying on what you remember. A small-looking change multiplied across your daily request volume can be enough to change your decision.

Step 4: Adjust the prompt, and version it with the model

A prompt tuned for one model often carries workarounds the next model doesn't need, or misses instructions it now does need. If the diff report shows formatting drift, try fixing it in the prompt first: add an explicit output example, tighten the instruction about preambles, or move the format rule to the end.

When you change the prompt, give it a new version and tie it to the candidate model. Don't edit the prompt the current model uses. Mixing an old prompt with a new model, or the reverse, is how you end up with a combination nobody tested.

candidate:
  model: provider-model-2026-10-15
  prompt_version: v8   # v8 drops the old JSON-fence workaround
  percent: 0

Then run the eval again. Repeat until the candidate is at least as good on the cases that matter and the cost and latency numbers are acceptable.

Step 5: Roll out in stages from the server

An eval set covers what you thought to test. Real users will send things you didn't. Send a small, stable share of traffic to the candidate first. Bucket by user, not by request, so each person gets consistent behavior:

function pickConfig(userId: string, f: FeatureConfig) {
  const bucket = stableHash(userId) % 100;
  return bucket < f.candidate.percent ? f.candidate : f.current;
}

Record the model identifier and prompt version on every request log. Without them you can't tell which configuration produced a bad answer a user reports. Over the next few days, compare the two groups on the metrics from Step 3, plus any product signals you already collect, such as retry taps, thumbs-down ratings or how often people edit the generated text.

Increase the percentage in steps you're comfortable with, and wait long enough at each step to see a normal mix of usage, including weekends if your traffic changes then.

Step 6: Keep the way back open

Rolling back should mean setting the percentage to zero, not shipping a build. Keep the old model and its prompt version in config until the new one has handled full traffic for a while. Remove the old entry only after that, or when the provider retires the old model, whichever comes first.

Also decide in advance what triggers a rollback. A short written rule, such as "parse failures noticeably above the old model's rate" or "any spike in refusals on a core flow," saves you from deciding under pressure.

Stay ahead of retirement dates

Providers publish deprecation schedules, but they're easy to miss. A few habits help:

  • Subscribe to your provider's changelog or deprecation notices, and send them somewhere you actually read.
  • Put the retirement date of each pinned model in your calendar, with a reminder several weeks before it.
  • Begin evaluating a successor well before the deadline, so nothing pushes you into rushing the rollout.

A short checklist

  • Model pinned to a specific version and stored in server-side config
  • Prompt version stored alongside it
  • Eval set run on both models, with a diff report someone has read
  • Token counts, latency, parse failures and refusals compared
  • Per-request cost checked against current published prices
  • Model and prompt version recorded on every request log
  • Staged rollout bucketed by user
  • Rollback trigger written down, and old config kept

Wrapping up

A model upgrade is one of the few changes that can alter what your feature does without touching your code. If you treat it like any other dependency, with a pinned version, a side-by-side comparison, a gradual rollout and a quick rollback, it becomes a routine task. Most of the work happens once: after the config, the eval set and the per-request logging are in place, each later upgrade mostly means running the same steps again.