Running a model directly on the phone sounds like the best of all worlds: no server bill per request, no network round trip, and user data that never leaves the device. For some features it really is the right call. For others, it adds weeks of work and a support burden you did not plan for. This post covers how to decide, then how to integrate an on-device model so the feature works on the phones your users actually have.
Step 1: Decide whether the feature belongs on the device
Look at the task before you look at the technology. On-device models are good at narrow, well-defined jobs: classifying an image, pulling fields out of text, transcribing speech, suggesting tags, summarizing a short note, or rewriting a sentence. They are weaker at open-ended reasoning, long documents, and anything that needs current world knowledge.
Run through these questions:
- Does the input contain sensitive data? Health notes, private messages and photos are strong candidates for local processing. Keeping them on the device simplifies a lot of the work in our privacy checklist for AI features.
- Does the feature need to work offline? If users rely on it on planes, in basements or on poor connections, local inference is often the only good answer.
- Is the output short and checkable? A tag, a category or a one-line suggestion is easy for a small model to get right and easy for the user to correct.
- How often will it run? A feature triggered on every keystroke or every photo gets expensive fast on a server. Locally, the cost shows up as battery and heat instead, so it still needs a budget.
- Can you accept lower quality on some devices? Coverage is uneven. If the feature must behave the same everywhere, a server model may be simpler.
If most answers point toward local, keep going. If the task needs broad knowledge or long-form generation, a hosted model with a tight cost policy is usually the more honest choice.
Step 2: Choose between a system model and a bundled model
There are two broad routes, and they have very different trade-offs.
System-provided models
Both major mobile platforms now expose on-device language models through OS-level APIs: Apple's Foundation Models framework on supported Apple Intelligence devices, and Gemini Nano through ML Kit's GenAI APIs on supported Android devices. Both platforms also offer older, task-specific frameworks such as Vision, Speech and ML Kit for text recognition, barcode scanning and similar jobs.
- Pros: nothing added to your download size, the OS manages memory and updates, and the vendor has already done the hardware tuning.
- Cons: only available on specific devices and OS versions, the model can change under you with an OS update, and the user may need to enable a setting or wait for a model download.
Bundled models
You ship your own model file, converted to a format like Core ML on iOS or LiteRT (formerly TensorFlow Lite) on Android, or run a small open-weight model through a runtime library.
- Pros: you control the exact version, behavior is the same across OS updates, and it can reach older devices.
- Cons: app size grows, often a lot. You own the conversion, quantization and performance work, and you have to check the model's license yourself.
A common pattern for small studios is to use the system model when it's available and fall back to a server or to a simpler non-AI path when it isn't. For classic vision and speech tasks, the task-specific frameworks are often good enough that you don't need a generative model at all.
Step 3: Design for the devices that cannot run it
This is the step that most often goes wrong. Some share of your users will be on devices that can't run the model, have the feature turned off, or are still downloading it. Your app has to behave well for all of them.
Put the feature behind a small interface so the rest of the app never knows which backend is answering:
protocol TagSuggester {
var isAvailable: Bool { get }
func suggestTags(for text: String) async throws -> [String]
}
final class TagService {
private let backends: [TagSuggester] // ordered: on-device, server, keyword fallback
init(backends: [TagSuggester]) { self.backends = backends }
func suggest(for text: String) async -> [String] {
for backend in backends where backend.isAvailable {
if let tags = try? await backend.suggestTags(for: text), !tags.isEmpty {
return tags
}
}
return []
}
}Things to decide up front:
- What does the UI show when no backend is available? Usually the answer is to hide the feature rather than show a disabled button with no explanation.
- Is a server fallback allowed? If you promised "processed on your device", silently falling back to a server breaks that promise. Either ask, or don't fall back.
- What happens while the model is downloading or warming up? Check availability at the moment of use, not just at launch, because it can change during a session.
Step 4: Measure performance on real hardware
Simulators and emulators tell you almost nothing about inference speed, memory pressure or heat. Test on the oldest supported device you can get hold of, plus one recent device.
For each device, record:
- Cold start time: from the first call to the first result after a fresh launch. Model loading is often the slowest part.
- Warm latency: the median and worst case over a batch of repeated calls with realistic inputs.
- Peak memory: watch it in Instruments or Android Studio's profiler. Large spikes can get your app killed in the background.
- Sustained behavior: run the feature in a loop for several minutes and see whether latency climbs as the device heats up.
Practical habits that help:
- Load the model lazily, the first time the feature is likely to be used, not at app launch. Model loading can quietly undo startup time work.
- Run inference off the main thread, and make requests cancellable so a user who moves on doesn't leave the model working.
- Debounce triggers. Suggestions on every keystroke rarely beat suggestions after a short pause.
- Cap input length. Small models have small context windows, and long inputs are slow even when they fit.
Step 5: Evaluate quality per backend, not once
If you have more than one backend, you have more than one feature. A prompt that works well with a large hosted model can produce noticeably worse output from a small local one. Keep a small set of realistic inputs with expected outputs and run it against each backend separately.
Small models also need tighter prompts and output constraints. Where the platform supports structured or guided output, use it so you get a list of tags back rather than a paragraph you have to parse. Always validate what comes back: trim it, check its length, and drop anything that doesn't fit the expected shape before it reaches the UI.
Re-run the evaluation after each major OS update if you rely on a system model. The model may have changed even if your code didn't.
Step 6: Ship with escape hatches
- Remote kill switch: a config flag that turns the feature off, or switches backends, without an app update.
- Coarse telemetry: log which backend served each request, whether it succeeded, and rough latency buckets. Don't log the inputs or outputs themselves.
- User correction: make suggestions easy to dismiss or edit. Corrections are also the best signal you have that quality is slipping.
- Honest copy: if you say "on-device" in your marketing or privacy label, make sure it's true for every path the feature can take.
Wrapping up
On-device models are worth it when the task is narrow, the data is personal, or the feature has to work offline. The model call is the easy part. Most of the work is in the decisions around it: which devices get the feature, what everyone else sees, how you measure speed on real hardware, and how you check quality for each backend. Start with one small, checkable feature, put it behind an interface with a sensible fallback, and test it on the oldest phone you support before you promise it to anyone.