Hybrid search: combining keyword and semantic retrieval

Why pure semantic search drops the exact terms that matter most

Here is the failure we get called in to fix more than any other. A team builds a RAG system on a vector database, demos beautifully on broad questions, then a user types 'error E-4042' and the system returns three paragraphs about general troubleshooting and nothing about E-4042. The exact code the user cared about is sitting right there in the docs, and the retriever walked past it.

This is not a bug in your embedding model. It is how embeddings work. A vector model compresses meaning into a few hundred numbers, and in doing so it deliberately throws away surface form. 'E-4042' and 'E-4043' look almost identical to an embedding even though they point at completely different problems. Part numbers, customer IDs, function names, legal clause references, drug names, ticker symbols: these are tokens where the literal string is the meaning, and that is exactly the information embeddings smooth over.

The pattern we see across client codebases is that semantic-only retrieval looks great in the demo and quietly fails in the long tail. Broad conceptual questions work. Then real users show up with their account number, a stack trace, a model SKU, or a specific person's surname, and recall falls off a cliff right where confidence should be highest.

• Identifiers: order numbers, SKUs, error codes, case IDs, version strings like v2.3.1
• Proper nouns: people, product names, internal project codenames that never appeared in training data
• Negations and rare tokens: a single 'not' or an uncommon acronym that an embedding averages away
• Exact-phrase policy language where 'within 30 days' must not become 'about a month'
• Anything a user copy-pastes verbatim expecting an exact match

Why pure keyword search misses the meaning

The obvious fix is to go back to keyword search, and for the cases above it works. Keyword retrieval (these days almost always BM25, the algorithm behind Elasticsearch, OpenSearch, and Postgres full-text) matches literal tokens and ranks by how rare and frequent they are. Type 'E-4042' and BM25 finds the document with 'E-4042' in it. Done.

But keyword search has the opposite blind spot. It only knows the words you typed. Ask 'how do I stop my subscription' and a BM25 index full of documents that say 'cancel your plan' returns nothing useful, because 'stop' is not 'cancel' and 'subscription' is not 'plan'. No shared tokens, no match. Synonyms, paraphrases, and the gap between how users talk and how your docs are written all defeat it.

So you are stuck between two retrievers with mirror-image weaknesses. Semantic understands intent but fumbles exact strings. Keyword nails exact strings but is deaf to meaning. The right move is not to pick one. It is to run both and combine them.

How hybrid search combines the two

Hybrid search is simpler than the name suggests. You keep your vector index and you add a keyword index over the same chunks. At query time you send the query to both retrievers, each returns its own ranked list of candidates, and a fusion step merges those two lists into one final ranking that goes to your model.

The win is that each retriever covers the other's failure mode. When a user pastes 'E-4042', the keyword side surfaces the exact document and the fusion step floats it to the top even if the semantic side ranked it low. When a user asks a fuzzy conceptual question, the semantic side carries it and keyword contributes little. You do not have to guess in advance which kind of query you are getting, which matters because real users mix both inside a single sentence ('why is my v2.3.1 install throwing a timeout').

A practical detail teams miss: pull more candidates than you think you need from each retriever before fusing. We typically grab the top 20 to 50 from each side, fuse, then keep the top 5 to 10 for the model. Fusing only the top 3 from each retriever throws away exactly the borderline result that the other retriever would have rescued.

Reciprocal Rank Fusion in plain English

The hard part of combining two lists is that the scores are not comparable. Cosine similarity from a vector search might land between 0.7 and 0.9. BM25 scores are unbounded and depend on term rarity. Averaging a 0.82 against a BM25 score of 14.3 is meaningless. You would spend weeks tuning normalization and still get it wrong.

Reciprocal Rank Fusion (RRF) sidesteps this entirely by ignoring scores and using rank position instead. The recipe: for each document, take 1 divided by (k plus its rank) in each list, then add those values across lists. The constant k (commonly 60) softens the gap between the first result and the second so one retriever cannot completely dominate. A document ranked first in both lists gets the highest combined score; a document ranked first in one list and absent from the other still gets a healthy boost.

Why we reach for RRF on almost every build: it needs no calibration, it does not care that your two retrievers produce numbers on different scales, and it is a few lines of code. You can hand-tune weighted score blending later if you have a labelled eval set, but RRF is the right default and it is usually within a hair of anything fancier.

• Each retriever returns a ranked list; only the position matters, not the raw score
• Score per document = sum over lists of 1 / (k + rank), with k around 60
• A result that both retrievers rank highly wins; a result strong in just one still places well
• No score normalization, no per-corpus tuning, works the day you turn it on
• Add per-retriever weights only if a labelled eval set tells you to, not by feel

When hybrid matters most (and when it does not)

Hybrid is not free. It adds a second index to build and keep in sync, and a fusion step in the hot path. So be honest about whether your corpus actually needs it. The deciding factor is how often exact strings carry the meaning of a query.

If your knowledge base is technical documentation, support tickets, legal contracts, product catalogs, codebases, or medical and financial content, hybrid is close to mandatory because those domains are dense with identifiers and precise terminology. If your corpus is purely conversational or narrative (marketing blog posts, general FAQs phrased in plain language with no codes), semantic alone often gets you 90 percent of the way and hybrid buys you a few points at most.

• Strong fit: docs with error codes, APIs, part numbers, version strings, statute references
• Strong fit: enterprise search where users query by customer name, ticket ID, or account number
• Strong fit: any domain where a wrong-but-similar answer is worse than no answer
• Weaker fit: short narrative corpora with no identifiers and lots of paraphrase
• Always worth it when you can measure a recall gap on identifier-style queries in your own eval set

Tuning the balance without guessing

The first question every team asks is how to weight keyword against semantic. Our answer is to resist tuning anything until you can measure it. Start with plain RRF, no weights, equal footing. It is a genuinely strong baseline and you should know how far it gets you before you touch a dial.

To go further you need a small evaluation set: 30 to 100 real queries with the documents that should be retrieved marked as correct. Mix query types deliberately so the set includes identifier lookups, conceptual questions, and the messy hybrids in between. Then measure recall at k and mean reciprocal rank as you vary the knobs. Without this set you are tuning by vibes, and vibes optimize for the three queries you happened to test by hand.

When you do tune, change one thing at a time: the k constant in RRF, how many candidates you pull from each retriever, the chunk size, and only last, per-retriever weights. We have watched teams burn a week reweighting retrievers when the real problem was chunks too large to ever match an exact code cleanly.

• Build a 30 to 100 query eval set with known-correct documents before tuning anything
• Deliberately mix identifier queries, conceptual queries, and hybrids in that set
• Track recall at k and mean reciprocal rank, not just whether a demo felt right
• Tune in order: candidate counts, then k, then chunk size, then weights last
• Re-run the eval on every change so you can prove a tweak helped rather than hoped it did

A practical path: add keyword to your vector search

You do not need to rebuild your stack to get most of this value. If you already have a working vector search, adding a keyword retriever beside it is usually an afternoon of work, and it is the single highest-return retrieval change we ship for clients. Here is the path we actually follow.

First, check whether your existing tools already do hybrid so you skip the plumbing. Several vector databases (Weaviate, Qdrant, Pinecone, Elasticsearch, pgvector paired with Postgres full-text) now expose a hybrid mode with RRF built in, and turning it on is sometimes a single parameter. If yours does not, you build the keyword index yourself, which is still small: a BM25 index over the same chunks you already embedded.

Then wire the fusion, evaluate honestly, and ship. Do not skip the eval set step out of impatience. The whole point of hybrid is that it fixes the failures your demo never showed you, so the only way to know it worked is to test the queries your demo never tried.

• Check if your vector DB has a native hybrid or RRF mode before building anything custom
• Index the exact same chunks for keyword that you already embedded, so results align
• Pull 20 to 50 candidates per retriever, fuse with RRF (k=60), keep the top 5 to 10
• Validate against an eval set heavy on identifiers, the queries semantic-only flubs
• Ship behind a flag and compare retrieval quality side by side before making it the default