Most LLM features start out cheap. You write a prompt, call an API, and the bill during development is close to nothing. Then you ship, real people use the feature in ways you didn't expect, and the cost turns into a number you have to explain. For a small studio, that number can decide whether a feature stays.
This post covers a practical way to keep LLM costs predictable: measure first, cut waste, cache what repeats, pick the right model, and put limits in place before you need them. None of it requires special infrastructure. A logging table and some discipline will do.
Step 1: Measure cost per user action, not per month
A monthly invoice tells you that you spent money. It doesn't tell you why. The unit that matters is cost per user action: one summary, one rewrite, one chat turn. Once you know it, you can compare it to what that action is worth to you, whether that's subscription revenue, ad impressions or retention.
Almost every LLM API returns token usage with each response. Log it with enough context to group by feature:
async function callModel(feature, userId, request) {
const started = Date.now();
const res = await llm.generate(request);
await db.insert('llm_usage', {
feature, // e.g. 'note_summary'
user_id: userId,
model: request.model,
input_tokens: res.usage.input_tokens,
output_tokens: res.usage.output_tokens,
cached: false,
latency_ms: Date.now() - started,
created_at: new Date()
});
return res;
}Keep a small config file with your provider's current per-token rates for each model, and compute cost at query time rather than storing it. Rates change, and you'll want to recompute history when they do.
After a week of real traffic, run three queries:
- Average and 95th-percentile cost per action, by feature.
- Cost per active user per day, sorted descending. Look at the top of that list.
- Input tokens versus output tokens. This tells you which side to optimize.
The 95th percentile and the top users usually tell a different story from the average. A feature that is cheap on average can still have a long tail driven by huge inputs or users who script it.
Step 2: Cut tokens you're paying for but not using
Input tokens are often the larger share, because prompts grow quietly. Typical sources of waste:
- Bloated system prompts. Instructions get added after every bug report and never removed. Re-read yours and delete anything your evals show has no effect.
- Too many few-shot examples. Try two instead of six and check whether quality drops on your eval set.
- Unbounded context. Sending a user's entire note history when the feature only needs the current note. Set a hard limit on input length and truncate or summarize anything beyond it.
- Full chat history on every turn. For conversational features, keep the last few turns verbatim and replace older ones with a short running summary.
- Verbose structured data. Pretty-printed JSON with long key names costs more than compact formats. Strip fields the model doesn't need.
On the output side, set a maximum output token limit on every call. Ask for the format you actually render: if the UI shows three bullet points, ask for three bullet points, not "a detailed summary." A model asked to explain its reasoning before answering will do so, at your expense, so only ask for it when you've measured that it improves results.
Step 3: Cache what repeats
Many in-app features see the same inputs more often than you'd guess: the same onboarding question, the same product description being translated, the same document being summarized by several people on a shared workspace. Caching has two layers.
Response caching in your own backend
If the same prompt with the same input should produce an acceptable answer every time, store the result keyed by a hash of everything that affects the output:
const key = sha256(JSON.stringify({
feature, model, promptVersion, input: normalize(userInput)
}));
const hit = await cache.get(key);
if (hit) return hit;
const res = await callModel(feature, userId, buildRequest(userInput));
await cache.set(key, res.text, { ttlSeconds: 60 * 60 * 24 * 7 });
return res.text;Include promptVersion in the key so editing your prompt invalidates old entries automatically. Normalize input carefully (trim whitespace, unify casing where it doesn't change meaning) to raise the hit rate. Log cache hits in your usage table too, so you can see the hit rate per feature.
Don't cache anything personal across users, and think about whether cached text could leak one user's content to another. Keying by user for private data is the safe default.
Provider-side prompt caching
Several providers offer discounted pricing when a long prompt prefix repeats across calls. To benefit, put the stable parts first (system instructions, examples, reference documents) and the variable user input last. Check your provider's documentation for minimum prefix lengths and how long cached prefixes live, then confirm in your usage logs that cached tokens actually show up.
Step 4: Use the smallest model that passes your evals
The most capable model is rarely necessary for every task. Classification, extraction, short rewrites and tagging often work fine on smaller, cheaper models. The way to find out is not to guess but to run your eval set against each candidate and compare pass rates alongside cost per action.
A common pattern is routing:
- Send the request to a small model first.
- If the output fails a cheap check (invalid JSON, missing required fields, a low self-reported confidence you've validated against your evals), retry with a larger model.
- Log how often escalation happens. If it's frequent, the small model is the wrong choice for that feature.
For some features, an on-device model or a non-LLM approach beats both. Spell-check-style fixes, simple keyword tagging or language detection may not need a hosted model at all. Each request you move off the API is a request with no per-call cost.
Step 5: Set budgets and limits before launch
Optimization lowers the average. Limits protect you from the tail. Put these in place before release, not after the first scary invoice:
- Per-user daily quota. Count actions or tokens per user per day and refuse politely past a threshold. Free and paid tiers can have different quotas.
- Per-request input cap. Reject or truncate inputs over a fixed size on the server, regardless of what the client sends.
- Rate limiting. Limit requests per minute per user and per IP to blunt scripted abuse.
- Global kill switch. A remote config flag that disables or degrades the feature instantly if spend spikes.
- Provider spend limits and alerts. Most providers let you set usage alerts or hard caps on the account. Turn them on.
A simple quota check looks like this:
const used = await db.sumTokensToday(userId, feature);
const limit = user.isPaid ? PAID_DAILY_TOKENS : FREE_DAILY_TOKENS;
if (used >= limit) {
return { error: 'daily_limit', message: 'You have reached today\'s limit for this feature.' };
}How should you pick the thresholds? Use your measurements from Step 1. Look at the distribution of daily usage per user and set the free limit above what normal users reach, so only outliers hit it. Revisit the number as you learn more.
Design the limit-reached state as part of the feature. Tell the user what happened and when it resets, and offer an upgrade if you have one. A silent failure looks like a bug.
Step 6: Watch it after launch
Costs drift. A prompt edit adds tokens, a new user segment uses the feature differently, a provider changes pricing. A short weekly review keeps surprises small:
- Cost per action by feature, compared to last week.
- Cache hit rate by feature.
- Escalation rate if you route between models.
- Number of users hitting quotas, and whether they're normal users or abusers.
Set an automated alert when daily spend exceeds a multiple of your recent average. It doesn't need to be clever; it needs to wake you up before the month ends.
A pre-launch checklist
- Token usage logged per call, with feature and user.
- Max output tokens set on every request.
- Server-side input length cap.
- Prompt reviewed and trimmed against your eval set.
- Response cache for repeatable requests, keyed with prompt version.
- Stable prompt prefix placed first for provider caching.
- Smallest passing model chosen per feature.
- Per-user quota, rate limits and a kill switch in place.
- Provider spend alerts enabled.
Wrapping up
LLM costs become a problem when nobody is measuring them. Log usage per action from day one, cut the tokens you don't need, cache the work that repeats, and choose models based on your evals rather than habit. Then put limits in place so the worst case is a polite message to one user, not a bill that wipes out the month. None of this is glamorous, but it's what lets a small team keep an AI feature running.