Keyword, vector, hybrid, or reranking for RAG? Choose with a matched trial

Choosing a RAG search method is a retrieval design decision, not a contest between four fashionable labels. A question containing a part number needs the right record, while a person describing a fault in everyday language may use none of the words in the repair note. A hybrid query can collect both kinds of candidate. A reranker can reorder candidates already collected, but it cannot search the collection again for evidence that never reached it.

This page starts after the RAG investment decision. The retrieval evaluation diagnoses what an existing system returned for each question; this trial instead selects a retrieval configuration from matched alternatives. Hold document extraction and chunk boundaries steady during the comparison so a changed splitter does not masquerade as a better search algorithm.

Give each stage its own job

Route to test	What it can add	What it cannot guarantee
A: lexical search	BM25 ranks matching terms in searchable fields. A separately indexed exact-value field or deterministic lookup can protect an identifier that an analyzer might split.	Shared meaning without shared vocabulary may be missed; an ordinary BM25 match is not proof that the whole clause supports the answer.
B: dense vector search	An embedding can retrieve conceptually related descriptions when the query uses different wording, within the embedding model’s tested language and domain.	Similar wording or meaning can still point to the wrong model, date, or exception. An approximate neighbor list is not an exact-ID guarantee.
C: hybrid candidate search	Run lexical and vector branches against the same eligible collection, then fuse their ranked candidate lists. Azure AI Search and Elasticsearch document reciprocal rank fusion (RRF) as an implementation.	Fusion cannot repair a source absent from both branch lists. Branch depths, field choices, and filters still decide what is available.
D: reranking	Score an initial candidate set using the query and readable candidate text, then place stronger candidates earlier in the final context.	The reranker sees only its input set. Azure’s semantic ranker, for example, processes at most the first 50 initial results and adds usage cost.

These are routes, not a required stack. Test a reranker on the simplest first-stage route that already covers the necessary evidence; hybrid plus reranking is only one possible candidate. Raw BM25, vector similarity, RRF, and reranker scores have different scales. Do not compare their numbers as if they were one calibrated probability, or set one universal “confidence” threshold across them.

Freeze the evidence and divide the questions

Use one approved collection snapshot. Record document IDs and versions, the verified extracted text, the chunk configuration, and the same access and effective-date conditions for every route. Keep the answer-stage context budget and citation target fixed. Use the same embedding model and vector index for routes B and C. Record each route’s tokenizer, searchable fields, identifier treatment, branch candidate depth, fusion settings, reranker input limit, and final number of passages.

Build a labeled question set from the actual workflow, with a held-out portion for the final choice. A reviewer marks the source span or set of spans required to answer, or marks that no eligible evidence exists. Include distinct slices: exact IDs or quoted terms; everyday paraphrases; a code plus a natural-language symptom; aliases or a second language only if the chosen analyzer and embedding model claim to support them; plausible but wrong nearby records; and queries under restrictive filters. Give severe mistakes, such as returning another device’s safety step, a blocking rule before seeing the results. The evaluation-set guide explains how to maintain the cases after launch.

Keep authorization ahead of materialization on every branch. A protected passage must not enter the hybrid union, a reranker request, the model context, or a citation merely because a final filter would hide it. Vector filter timing also changes recall: Azure documents that post-filtering with a selective filter or a small neighbor count can omit an eligible match; its prefilter mode searches within the filtered set but may use more CPU and time. Test the behavior of the actual backend and carry the document-access trace into this trial. Ranking cannot repair a missing eligible candidate caused by late filtering.

Compare candidate coverage before final rank

For every query, save the ordered IDs from A and B, the IDs and ranks after C’s fusion, and the input and output IDs for D. Mark whether all required source spans appeared in each first-stage candidate set. Then inspect the exact passages that fit the common final context budget. A route can have good broad recall and still fail if its only useful passage falls below the cutoff sent to the generator.

Report results by query slice, not only as one average: complete evidence among the first-stage candidates, complete evidence actually delivered, a wrong near-match treated as support on a no-evidence case, and any wrong identifier, version, or access result. Measure retrieval p50 and p95 under a representative request mix. Include query embedding, both search branches, fusion, reranking, and network time in the route’s measured latency. Record per-query serving charges, indexed storage, and update work instead of counting only the final search call. Use a holdout set that did not determine branch weights, depths, or reranker choice.

A useful control compares routes under the same final passage and token budget. Candidate depths can differ because their cost is part of the proposed design; record those depths so a larger pool does not look like a free gain. If the target span never appears in D’s input, investigate first-stage recall, indexing, extraction, query processing, or filters. If it appears there but is displaced from the delivered context, reranking or a better first-stage ranking may be worth its additional latency.

Choose the least complex route that passes the gates

Start with a lexical route that treats exact IDs as exact values where the task requires that behavior. Keep it if it passes the paraphrase and mixed-question gates too. Add a vector route when the holdout shows that genuinely different language reaches evidence the lexical route misses, with no new blocking mistakes. Choose hybrid when the lexical and vector lists recover complementary eligible passages and their fused final list improves the task enough to justify another index, embedding step, and operational owner. Add reranking only when useful evidence is already in the candidate pool but the final budget cuts it off.

For every candidate, write a route-specific failure and the smallest next test. A code collision suggests an exact field or authoritative record lookup, not a larger cosine threshold. A missed paraphrase may need better vocabulary coverage, query processing, or a different embedding model. A source present in one branch but lost after fusion calls for inspecting branch depth and rank-fusion settings. A no-answer query with a plausible hit needs a separate evidence sufficiency rule; similarity alone is not permission to assert an answer. After selecting a route, run the citation and freshness test on the generated response.

Example: controller alarm versus symptom wording

Imagine a fictional maintenance library for rooftop irrigation controllers. Model HX-217 has a bulletin about a valve that only responds after a power cycle. A nearby HX-271 note uses many of the same terms but specifies a different reset sequence. The source reviewer identifies the approved HX-217 span and marks the other model’s instructions ineligible for that question.

Test three separate queries: the exact alarm code, “the valve wakes only after restarting the controller,” and a mixed query naming HX-217 with the symptom. The lexical route should be inspected for code handling; the vector route for finding the paraphrase without promoting the wrong model; and the hybrid route for retaining both constraints. Record the actual branch IDs and delivered passages in the worksheet. There is no claimed winner until the real collection, filters, and held-out queries have been run.

Example: a buried facilities guide

A fictional campus help desk receives “The washroom drain makes a pulsing gurgle when the exhaust fan starts.” Its approved guide uses the technical phrase “air-admittance valve oscillation during pressure change.” Many general plumbing notes could seem close to that question. A reviewer marks the narrow guide section as required and a general odor-control note as insufficient.

This case asks two different questions. Did any first-stage route place the guide inside the reranker’s actual input set? If so, did reranking move it into the fixed final context without pushing out another necessary condition? If the guide never entered that set, reranking cannot help; change retrieval or the source representation first. Repeat with a no-evidence question about a repair chemical absent from the library, so a broadly related passage is not treated as authorization to recommend it. The ranks and pass/fail cells remain blank until the test is executed.

Use the strategy worksheet

The workbench below records the frozen collection, query slices, four route configurations, first-stage and final evidence, timing, cost, and the decision. Fill it with sanitized IDs and measured runs; it saves a draft in this browser and can be copied or exported. Set blocking and performance thresholds before filling result cells. Reopen the choice after a new document shape, analyzer, embedding model, access rule, or query mix changes the experiment.

Boundary

This trial chooses retrieval and ranking for a collection that already needs request-time evidence. If the user names one short approved document, supplying it directly can be simpler. An authoritative database lookup may be safer for a structured key than any relevance search. This page does not establish source truth, replace access enforcement, repair stale indexing, or prove a generated answer is faithful; those require separate controls. No search route is universally best without task-specific measurements.