Where dense-only retrieval breaks
Embedding-only retrieval finds semantically similar documents well, but it is fragile on queries that must match literally: product SKUs, error codes, statute section numbers, proper nouns. In enterprise RAG traffic, these exact-match queries account for 20-40% of volume and show up as the single most common failure pattern in dense-only setups. Running a lexical retriever like BM25 in parallel is what defends this segment.
Hybrid retrieves candidates; reranking orders them
The key is to separate the roles. BM25 and embeddings own recall - getting the correct document into the candidate set - while a cross-encoder reranker owns precision, reordering that set. A cross-encoder feeds query and document together and computes their interaction, so its relevance judgment is sharper than the independent encodings a bi-encoder produces. On the BRIGHT Biology benchmark, adding reranking alone lifted nDCG@10 from 0.13 to roughly 0.40, about a 3x improvement.
The established 2026 standard
The baseline for production search is now unambiguous: a two-stage pipeline that retrieves 20-50 candidates with BM25 and embeddings, then reranks them with a cross-encoder. Retrieve too few and the answer never enters the set, making reranking pointless; retrieve too many and reranker latency balloons. The 20-50 band is the practical balance point between recall and latency.
A detailed guide: from planning to operations
In planning, pin the target metrics to numbers. For example, set release gates of nDCG@10 at or above 0.35, p95 latency including reranking at or below 400ms, and a top-3 hit rate for exact-match queries at or above 95%. Build the evaluation set from real logs, labeling at least 200 queries split into exact-match, semantic, and mixed types, and measure the lift of hybrid+reranking over dense-only separately per query type.
Define failure patterns and their recovery branches up front. First, reranker timeout: if the p95 latency budget is exceeded, skip reranking and safely degrade to hybrid raw-score ordering. Second, candidate miss: if the answer is not among the 20-50, reranking cannot recover it, so add a retry branch that raises the BM25 weight or the candidate count. Third, low reranker confidence: if the spread between top scores falls below threshold, route to a human-review queue. Fourth, reranker outage: trip a circuit breaker to fall back to hybrid results immediately and fire an alert.
The operations checklist starts with observability. For every query, emit a standard log capturing query type, candidate count, top BM25 and embedding scores, rank movement before and after reranking, per-stage latency, and whether a fallback fired. Because the log contains the user's raw query, apply PII masking rules to emails, phone numbers, and national IDs before storage. Version-tag the reranker model and candidate-count parameters in the log so regressions stay traceable.
Quality control watches latency and accuracy together. Surface the daily reranking fallback rate, p95 latency, and per-type nDCG on a dashboard, and alert when the fallback rate exceeds 5% or p95 breaks the budget. Tune the reranker batch size and the number of candidates you truncate to keep latency within budget, but roll back if the accuracy drop crosses threshold.
The continuous-improvement loop treats failure logs as assets. Periodically label the queries that went to the human-review queue and those that triggered fallbacks, fold them into the evaluation set, and retune the reranker threshold and candidate count. Always A/B any new reranker or embedding model against the existing evaluation set and promote it only after confirming no per-type nDCG regression.
Executive summary
Dense-only is guaranteed to fail on the 20-40% exact-match segment, so make the two-stage pipeline - retrieve 20-50 candidates with BM25+embeddings, then rerank with a cross-encoder - your baseline. The payoff is large, as BRIGHT Biology showed with nDCG@10 tripling from 0.13 to 0.40 on reranking alone, but stable production operation requires a p95 latency budget, fallback branches, PII masking, and a per-type evaluation set alongside it.