Speech-to-text is often the first AI feature a small app adds, and it is easy to underestimate. The demo takes an afternoon. Shipping it takes longer, because you have to decide where recognition runs, what happens when it fails, how you ask for microphone access, and how you know the transcripts are good enough. This post walks through those decisions in the order that worked for us.
Start with what the transcript is for
Before you pick an API, write one sentence about what the text does once you have it. The answer changes everything downstream:
- Dictation into a text field: the user sees the text and can fix it. Errors are cheap. Latency matters most.
- Voice commands ("add milk to the list"): the vocabulary is small, and a wrong word triggers a wrong action. Accuracy on a few phrases matters more than accuracy in general.
- Transcribing recordings (voice notes, interviews): nobody is waiting on each word, so you can process in batch and use a heavier model.
- Input to an LLM: the transcript becomes a prompt. Recognition errors stack up with model errors, so you need to test the whole chain together.
If you can't write that sentence yet, hold off on adding voice.
On-device or cloud: the real trade-offs
Both major mobile platforms ship system speech recognition, and many cloud providers offer transcription APIs. The choice is less about raw accuracy than about these factors:
On-device recognition
- Privacy: audio never leaves the phone, which makes your privacy policy and store disclosures simpler.
- Cost: no per-minute bill, which matters if usage grows without matching revenue.
- Offline: works on a plane or in a basement.
- Catches: which languages are supported depends on the OS version and the device. Some languages need a model download first. On older or low-end devices you may get worse quality, or no on-device option at all.
Cloud transcription
- Consistency: every user gets the same model, whatever their device.
- Features: depending on the provider, you may get punctuation, speaker labels, timestamps or custom vocabulary.
- Catches: you pay per unit of audio, you need a network connection, and you are now sending voice data to a third party. That has to show up in your disclosures and consent flow. You also need a backend or token broker so your API key never ships in the app binary.
A sensible default for a small app is on-device first, with cloud as an option for the specific jobs that need it, such as long recordings. If you go hybrid, tell the user which mode is active. Don't quietly upload audio because the local recognizer was unavailable.
Platform starting points
On iOS, the Speech framework's SFSpeechRecognizer can be told to stay on the device. Newer OS releases add newer speech APIs, so check the current documentation for your deployment target. The basic pattern looks like this:
let recognizer = SFSpeechRecognizer(locale: Locale(identifier: "en-US"))
guard let recognizer, recognizer.isAvailable else { showTypingFallback(); return }
let request = SFSpeechAudioBufferRecognitionRequest()
if recognizer.supportsOnDeviceRecognition {
request.requiresOnDeviceRecognition = true
} else {
// Decide explicitly: fall back to server recognition, or disable the feature.
}
request.shouldReportPartialResults = trueYou need two Info.plist strings: NSMicrophoneUsageDescription and NSSpeechRecognitionUsageDescription. Write them in plain language and say what the audio is used for.
On Android, SpeechRecognizer is the system entry point. Recent API levels add an explicit on-device recognizer:
val recognizer = if (Build.VERSION.SDK_INT >= 31 &&
SpeechRecognizer.isOnDeviceRecognitionAvailable(context)) {
SpeechRecognizer.createOnDeviceSpeechRecognizer(context)
} else if (SpeechRecognizer.isRecognitionAvailable(context)) {
SpeechRecognizer.createSpeechRecognizer(context) // may use a network service
} else {
null // show typing fallback
}You need the RECORD_AUDIO runtime permission. Note that the default recognizer comes from whichever recognition service the device has installed, so behavior can differ between manufacturers. Test on more than one brand of phone.
If you plan to bundle your own open-weights speech model instead, it needs the same work as any other bundled model: download size, memory, warm-up time and thermals. Our on-device model checklist covers those steps.
Streaming or batch
Streaming recognition returns partial results as the user speaks. It feels responsive, but it adds UI state: words change as the recognizer revises them, and you have to decide when an utterance is finished. Batch recognition (record first, then transcribe) is simpler and often more accurate, but the user waits at the end.
Rules of thumb:
- For dictation, stream. Show partial text in a lighter style and commit it once it's final.
- For commands, stream, but act only on the final result. Never on a partial one.
- For recordings, use batch. Save the audio file first so a failed transcription can be retried without asking the user to speak again.
Design the unhappy paths first
Most voice bugs come from situations the demo never hit. List them and decide on a behavior for each one before you polish the happy path:
- Permission denied: show the typing input and a short note about where to turn voice back on. Don't keep re-asking.
- Recognizer unavailable (unsupported language, model not downloaded, service missing): hide or disable the mic button instead of letting it fail when tapped.
- Silence or no speech detected: stop after a timeout you choose, and say so clearly.
- Interruption (phone call, another app takes the audio session, app goes to the background): stop cleanly and keep whatever was already transcribed.
- Low-confidence or garbled result: for commands, confirm before acting. For dictation, let the user edit.
- Network loss in cloud mode: keep the audio and queue a retry, or offer on-device recognition if it's available.
The general pattern of degrading to a manual path is covered in designing LLM features that fail gracefully. Voice input needs the same discipline.
Ask for the microphone at the right moment
Request microphone and speech permissions when the user first taps the mic button, not at launch. A short screen before the system prompt ("Voice input turns what you say into text. Audio stays on your device.") helps, but only if it's true. If you use cloud transcription, say so plainly, and make sure your store privacy labels and data safety form match.
Measure quality with your own audio
Published accuracy figures for a model tell you little about your users' accents, background noise and vocabulary. Build a small test set:
- Record 30 to 50 short clips that match real use: quiet room, street noise, fast speech, your app's domain terms and product names.
- Write the correct transcript for each one by hand.
- Run every clip through each candidate recognizer and compute word error rate: substitutions, deletions and insertions divided by the number of words in the reference. Open-source WER calculators are easy to find, or you can write one with an edit-distance function.
- For commands, also count how often the parsed action was correct. That's the number users actually experience.
Keep the clips and rerun them when you change recognizer, OS target or prompt. This is the same idea as the text evals in build a small eval set before you ship, applied to audio. Get consent from anyone whose voice you record, and keep the files out of public repos.
Small details that matter
- Locale: by default, use the user's preferred language if the recognizer supports it. Don't hardcode
en-US. - Custom vocabulary: if your app has unusual words, check whether your recognizer accepts contextual hints. It's often the cheapest accuracy gain available.
- Audio session: on iOS, set the audio session category before recording, and restore it afterwards so you don't break the user's music playback.
- Battery: stop the recognizer as soon as you have a final result. A forgotten live audio engine drains battery quietly.
- Logging: log error codes and durations, not transcripts or audio, unless the user has explicitly opted in.
Wrapping up
A good speech-to-text feature is mostly decisions, not code: what the transcript is for, where recognition runs, how each failure looks to the user, and how you measure quality on audio that sounds like your users. Start on-device, add cloud only where it clearly pays off, keep a typing fallback, and keep a small set of recorded clips you can rerun whenever something changes.