Not every AI feature has to be a chatbot. One of the most useful things a language model can do in a small app is quiet: help people find things. A notes app where searching "car stuff" turns up the note titled "Insurance renewal and MOT". A recipe app where "something quick with leftover rice" works. A help center where "can't log in" finds the article called "Resetting your password".
This is what embeddings are for. This post covers adding embedding-based semantic search to a small app without a lot of infrastructure, and how to check that it beats the keyword search you already have.
What an embedding actually gives you
An embedding model takes a piece of text and returns a fixed-length list of numbers, a vector. Texts with similar meanings end up with vectors that point in similar directions. To search, you embed the query, compare it to the stored vectors, and return the closest matches.
Some things follow from that:
- No text is generated. The model only maps text to numbers, so there is no hallucinated answer to guard against. The worst outcome is a bad ranking.
- Vectors from different models can't be compared. If you switch models, you re-embed everything. Plan for this from day one.
- Similarity is not relevance. The closest vector is not always the right result, especially for short queries, names, codes and exact phrases.
Step 1: Decide what a searchable unit is
Before choosing a model, decide what you are embedding. This choice affects quality more than most others.
- Short items (titles, product names, short notes): embed each item whole.
- Long items (articles, documents, long notes): split them into chunks, often by paragraph or heading, and embed each chunk. Store which item each chunk belongs to so you can group results.
- Structured records: build a small text representation on purpose instead of dumping the JSON. For a recipe, that could be the title, main ingredients and tags on separate lines.
Keep the function that builds the embedded text in one place and give it a version number. When you change it, you know which rows to re-embed.
Step 2: Choose where embedding happens
You have two broad options, with the same trade-offs as most AI features:
- Cloud embedding API. Easy to start with, usually good quality, and cheap per call compared with text generation. But user content leaves the device, and search depends on the network.
- On-device embedding model. Small sentence-embedding models can run on phones. Content stays local and search works offline. In exchange you get a bigger app, extra work to integrate, and usually somewhat lower quality.
If you lean toward on-device, the checklist in Adding an On-Device Model to Your Mobile App applies directly. Whichever you pick, embed queries and documents with the same model and the same preprocessing. Some models expect a different prefix or task type for queries than for documents, so check the model's documentation.
Step 3: Store vectors somewhere boring
For a small app you probably don't need a dedicated vector database. Consider these in order:
- Brute force in memory. Load the vectors and compare the query against all of them. For a personal notes app or a help center with a few hundred articles, this is often fast enough. Measure it on your slowest supported device with a realistic worst-case data size before deciding you need more.
- SQLite with a vector extension, or Postgres with a vector extension on the server. You keep one database and get indexed nearest-neighbour search when brute force gets slow.
- A dedicated vector store, only when you have measured a need the first two can't meet.
A minimal schema that leaves room for re-indexing:
CREATE TABLE embeddings (
item_id TEXT NOT NULL,
chunk_index INTEGER NOT NULL,
model TEXT NOT NULL, -- e.g. the model identifier
text_version INTEGER NOT NULL, -- version of your text-building function
content_hash TEXT NOT NULL, -- hash of the exact text embedded
vector BLOB NOT NULL, -- float32 array
PRIMARY KEY (item_id, chunk_index, model)
);The content_hash lets you skip re-embedding unchanged items. The model column lets you build a new index next to the old one and switch over once it is complete.
Step 4: Rank with more than one signal
Pure vector search has known weak spots. It can do badly on exact identifiers, rare names and very short queries. Your existing keyword search is good at exactly those. Combining the two, usually called hybrid search, tends to beat either one alone.
A simple, robust way to combine them is reciprocal rank fusion. You don't need to normalize scores across systems, only ranks:
def hybrid(query, k=60, limit=20):
kw = keyword_search(query, limit=50) # list of item_ids, best first
vec = vector_search(query, limit=50) # list of item_ids, best first
scores = {}
for results in (kw, vec):
for rank, item_id in enumerate(results):
scores[item_id] = scores.get(item_id, 0) + 1 / (k + rank + 1)
return sorted(scores, key=scores.get, reverse=True)[:limit]And the brute-force vector search itself, assuming vectors are normalized to unit length when stored:
import numpy as np
def vector_search(query, limit=50):
q = embed(query)
q = q / np.linalg.norm(q)
sims = MATRIX @ q # MATRIX: (n_chunks, dim), rows normalized
top = np.argsort(-sims)[:limit * 3]
seen, out = set(), []
for i in top: # collapse chunks to items
item = CHUNK_ITEM_IDS[i]
if item not in seen:
seen.add(item)
out.append(item)
if len(out) == limit:
break
return outThe constant k in the fusion function damps how much the top few ranks dominate. The commonly used default is a reasonable place to start. Tune it against your eval set, not by feel.
Step 5: Handle the edges
- Empty and tiny queries. For one- or two-character queries, skip vector search and use prefix matching.
- No good match. Vector search always returns something. If even the best similarity is low, show a "no close matches" state instead of confident-looking noise. Pick that threshold from your eval data, because similarity scales differ between models.
- Freshly edited items. Embed on save in the background, and fall back to keyword search for items still waiting to be embedded. A search that can't find the note you just wrote feels broken.
- Model migration. Build the new index in the background, compare it with the old one on your eval set, then switch. Delete old rows only after the switch has held up.
Step 6: Prove it is better
Semantic search can feel impressive in a demo and still be worse in daily use. Before shipping, put together a small set of real queries with the results you expect. Search logs (if you have them and are allowed to use them), support tickets and your own use are good sources. Make sure it includes:
- Queries phrased differently from the target content (where semantic search should win)
- Exact names, codes and quoted phrases (where keyword search should win)
- Queries that should return nothing
For each one, record whether the expected item appears in the top few results, under keyword-only, vector-only and hybrid search. Ship the configuration that wins overall, not the one that wins on your favourite example. The method in Build a Small Eval Set Before You Ship an LLM Feature works for this with almost no changes. Search is actually easier to score than generated text because each result is either in the top few or it isn't.
Privacy and cost notes
An embedding is derived from the user's text, so treat it like the text itself. If you use a cloud API, user content is sent to a third party at index time and at query time, and your privacy disclosures should say so. Vectors stored on a server need the same access controls and deletion handling as the content they came from. When a user deletes a note, delete its embeddings too.
On cost, the expensive moments are the first full index and every model migration. Day-to-day cost comes from new content plus queries. Estimate it by counting the tokens in your actual content and multiplying by your provider's current rate. Then cache query embeddings for repeated searches, and debounce search-as-you-type so you don't embed every keystroke.
Wrapping up
Semantic search is one of the lower-risk ways to add a language model to a small app. Nothing is generated, failures look like ordinary bad search results, and the infrastructure can be as simple as a table and a loop. The work is mostly in the unglamorous parts: choosing what to embed, versioning it so you can re-index, combining it with keyword search, and checking against real queries that it actually helps. Start with brute force and hybrid ranking, measure, and add more machinery only when the numbers say you need it.