Many LLM APIs now accept images as well as text. That makes features like "snap a photo of the receipt" or "what plant is this" look like a weekend job: send the picture, ask a question, show the answer. A demo really does take that long. The work that matters comes after: deciding exactly what the model should do with the photo, keeping file sizes and costs in check, and handling the blurry, dark and off-topic photos real users will send.
This post walks through that work in the order we'd do it in a small app.
1. Shrink the task before you write any code
"Understand the image" is not a task. The narrower the job, the easier it is to test and the less the model can get wrong. Before you touch an API, write down:
- The input: what the photo should show. For example, a single printed receipt, roughly flat, in decent light.
- The output: the exact fields your app needs. For example, merchant name, date, total and currency. Leave out anything you won't use.
- The fallback: what the user can do when the model can't answer, such as typing the fields in by hand with the photo still on screen.
If you can't fill in all three, the feature isn't ready to build. Some photo tasks also don't need an LLM. Barcode scanning, text recognition and face detection all have mature on-device APIs that are fast, free to call and private. A common pattern is to run on-device text recognition first and send text, not the image, to the LLM. That's cheaper, and the photo never leaves the phone.
2. Prepare the image on the device
A raw camera photo is usually much larger than a vision model needs. Most providers resize large images on their side anyway, and many bill image input by resolution. Uploading the full-size file wastes bandwidth and battery and gains you nothing.
Do three things before upload:
- Resize. Cap the longest edge at a size you've tested (see step 5). Start from your provider's documented recommended size rather than guessing.
- Re-encode. JPEG at moderate quality is fine for most recognition tasks. Use PNG only when you need sharp edges, such as screenshots with small text.
- Strip metadata. Camera files often carry GPS coordinates, device details and timestamps. Re-encoding from pixel data usually drops them, but check the bytes you actually send instead of assuming.
On iOS, a minimal version looks like this:
func preparedJPEG(from image: UIImage, maxEdge: CGFloat) -> Data? {
let scale = min(1, maxEdge / max(image.size.width, image.size.height))
let size = CGSize(width: image.size.width * scale,
height: image.size.height * scale)
let format = UIGraphicsImageRendererFormat()
format.scale = 1
let resized = UIGraphicsImageRenderer(size: size, format: format)
.image { _ in image.draw(in: CGRect(origin: .zero, size: size)) }
// Redrawing from pixels drops the original EXIF block
return resized.jpegData(compressionQuality: 0.7)
}On Android, use Bitmap.createScaledBitmap and then compress(Bitmap.CompressFormat.JPEG, 70, stream). Read the EXIF orientation tag before you scale. Otherwise sideways photos reach the model sideways, which hurts accuracy without any visible error.
Photos are personal data, and they often include people and places nobody meant to share. If the feature sends images to a server, follow the steps in our privacy checklist for AI features: tell users before upload, check the provider's data retention terms, and don't keep copies on your own server unless you have a clear reason.
3. Constrain what comes back
A vision model will happily describe a photo in a paragraph. Your app needs fields. Ask for structured output and give the model an explicit way to say it couldn't do the job:
{
"status": "ok | not_a_receipt | unreadable",
"merchant": "string or null",
"date": "YYYY-MM-DD or null",
"total": "decimal string or null",
"currency": "ISO 4217 code or null"
}The status field matters most. Without it, a photo of a cat gets a confidently invented merchant name, because the model tries to fill every field you asked for. With it, you get a clear signal you can build UI around. In the prompt, say that null is the correct answer for any field the model can't read, and that guessing is worse than leaving a field empty.
Validate everything on your side anyway. Parse the date, check that the total is a number, and confirm the currency code is real. We covered this in more depth in getting structured output your app can trust. Images make it even more important, because you can't easily see why the model read a smudge as a 7.
4. Build a photo test set from real conditions
Tidy sample images make a vision feature look great. Real photos look different. Before shipping, collect test photos that cover how people will actually use the feature, with consent if they come from other people:
- Good light and bad light, including flash glare on glossy paper
- Tilted, partly cropped and crumpled subjects
- Several subjects in one frame, such as two receipts on a table
- Photos that have nothing to do with the feature
- Screenshots and photos of screens, if users might send those
- Different languages, scripts, number formats and date formats, if you ship in more than one locale
For each photo, write down the expected output, including the expected status. Run the whole set whenever you change the prompt, the image size or the model, and track two numbers: field accuracy on valid photos, and how often the model correctly refuses bad ones. A change that improves the first number while quietly making the second worse is a regression.
5. Find the smallest image size that still works
Image size affects cost, upload time and accuracy, so measure it instead of guessing. With your test set in place:
- Run the full set at three or four maximum edge sizes, from small to your provider's recommended maximum.
- For each size, record accuracy, average upload size and the input token count from the API response's usage field. Most providers report it per request.
- Choose the smallest size where accuracy stops improving in a meaningful way.
Small printed text is usually what forces you to a larger size. If accuracy only drops on fine print, cropping to the region of interest can work better than sending more pixels. Let the user adjust a crop box, or use an on-device document detector to find the edges.
Log the token count per request in production as well. That gives you a real cost per scan to plan with.
6. Design the screen for being wrong
Even with a good prompt and a sensible image size, the model will sometimes misread things. The UI should make mistakes cheap to fix:
- Show the photo next to the extracted fields so the user can check them at a glance.
- Make every field editable, and pre-fill fields rather than saving them straight away for anything that affects money or records.
- Give each status its own message. "This doesn't look like a receipt" and "We couldn't read this one, try more light" tell the user what to do next. A generic error doesn't.
- Offer manual entry in one tap from every failure state.
- Show progress. Uploading and analysing an image takes noticeable time, so show the user that something is happening.
If you can, count how often users edit each field after extraction. That's the most honest accuracy number you'll get, and it tells you where to focus the next round of prompt or crop work.
7. A short pre-launch checklist
- The task, output fields and fallback are written down
- On-device alternatives have been considered for part or all of the task
- Images are resized, re-encoded and stripped of metadata, and the stripping has been checked
- EXIF orientation is applied before upload
- Users are told that photos are uploaded, and provider retention terms have been checked
- Output follows a schema with a status field and is validated on your side
- The test set includes bad, off-topic and multi-subject photos
- Image size was chosen by measurement, and tokens per request are logged
- Every failure state has a clear message and a path to manual entry
Wrapping up
Vision models make features possible that small apps couldn't build a few years ago, but sending a photo to an API is only the first step. Keep the task narrow, make images small and clean before they leave the device, give the model a structured way to say "I can't tell", and test with the messy photos people actually take. Do that, and the feature can hold up beyond the demo.