Hybrid Search: Vector Search Alone Isn't Enough

Search a vector database for the error code "E-4021" and you might get back a tidy list of documents about error handling, logging, and troubleshooting philosophy. Everything except the one page that actually contains "E-4021."

That's not a bug. It's what pure semantic search does. It matches meaning, and "E-4021" doesn't mean anything, it just is a string that has to appear. This is the blind spot nobody mentions when they tell you to "just use a vector database" for RAG. The fix is hybrid search, and it's less complicated than it sounds.

Semantic search works by embedding. An embedding model maps every document and every query to a point in high-dimensional space, positioned so that similar meanings land close together. Retrieval is then just finding the document vectors nearest the query vector.

This is genuinely powerful. Ask "how do I get my money back" and semantic search will surface a document titled "Refund Policy" even though the two share no words. It handles synonyms, paraphrasing, and intent that keyword matching would sail right past.

The blind spot is exact tokens. Product codes, error codes, function names, acronyms, rare surnames. Embedding models blur these into a neighborhood of similar-looking strings, so the precise identifier you needed can rank below a pile of vaguely related text. When the exact string is the whole point, meaning-based search is the wrong instrument.

Keyword search still matters: BM25 in a minute

Keyword search is the old approach everyone assumes got replaced. It didn't. It matches documents that literally share words with the query, word order ignored, and it's very good at exactly the thing semantic search fails at.

The classic version is TF-IDF: score a document by how often the query's words appear in it, weighted by how rare those words are across the whole collection. Rare words count for more, common ones for less, and you normalize by document length so a long document doesn't win just for being long.

BM25 (Best Matching 25) is the refined version most retrievers actually use, and it fixes two things TF-IDF gets wrong:

Term frequency saturates. A document with your keyword 20 times isn't twice as relevant as one with it 10 times, so BM25 gives repeated matches diminishing returns instead of letting them pile up linearly.
Length penalties diminish too, and BM25 exposes two knobs so you can tune how aggressively it saturates term frequency and corrects for length.

The payoff is practical reliability: if your index preserves the term correctly, BM25 gives you a much better chance of surfacing the exact document than semantic search alone.

Hybrid search: run both, then fuse

Hybrid search is the obvious move once you stop treating the two techniques as competitors. You run keyword search and semantic search in parallel. Each returns its own candidate list, maybe 20 to 50 documents. You can trim both with metadata filters (date, author, document type) if you need to. Then you merge the two lists into one final ranking and return the top results.

The catch is the merge. BM25 produces relevance scores on one scale; semantic search gives you distances or cosine similarities on a completely different one. You can't just add them together. A BM25 score of 14 and a cosine similarity of 0.82 aren't measuring the same thing in the same units, and naively combining them lets whichever number happens to be larger dominate for no principled reason.

Reciprocal Rank Fusion: the merge that just works

The clean solution is to ignore the raw scores entirely and combine by rank instead. Reciprocal Rank Fusion (RRF) does exactly that.

For each result list, RRF gives a document points based on its position: a document ranked r earns 1 / (k + r) points, where k is a small constant. Sit at the top of a list and you earn a lot. Sit near the bottom and you earn a little. Add up each document's points across both lists, sort by the total, and that's your final ranking.

Why it works so well: it never looks at BM25 scores or cosine distances, only at positions, so the incompatible-scale problem simply disappears. A document that ranks high in both lists rises to the top. A document that ranks first in one list and is missing from the other still scores respectably. There is essentially one main parameter to start with, and the default is often a good baseline — if you have labeled queries, tune it instead of treating it as magic.

It isn't the fanciest fusion method out there, but it's robust and cheap, and it's very hard to break, which is why it's the default in so many production stacks.

My default hybrid retrieval setup

For most RAG systems, I start with:

Vector search for semantic intent.
BM25 for exact terms, names, codes, and acronyms.
Metadata filters before ranking where possible.
RRF as a simple first fusion method.
Reranking only when the first-stage retrieval is already decent.

Hybrid search is not fancy. It is just harder to fool.

Default to hybrid

If you're standing up retrieval for RAG, reach for hybrid search from the start instead of bolting it on later once the complaints roll in. Semantic search alone quietly drops exact terms. Keyword search alone misses meaning. RRF stitches the two together for almost no cost and none of the scale-matching headaches.

Pure vector search makes a great demo and a shaky product. The upgrade is cheap, so take it early.

If your retrieval keeps returning almost-right answers and you can't tell why, this is one of the first places I'd look. If you'd like a hand with it, get in touch.