Evaluate RAG retrieval before judging the answer

A polished answer can hide a broken search step. The model may have guessed correctly despite receiving the wrong passage, or it may fail because the required clause never entered its context. Evaluate retrieval as its own operation: given a query, a caller, and a fixed corpus state, which passages were eligible, which were returned, and where did the useful ones rank?

Microsoft’s RAG evaluation guidance separates retrieval measures from answer measures and compares retrieved documents with human relevance labels. Use that distinction to locate the failure before changing a prompt or model. This lesson produces a small, repeatable retrieval trial; answer support belongs in the separate citation audit.

Fix the query and the evidence snapshot

Start with 15–30 real or realistic questions from one workflow. For each, preserve the exact query text, caller role, jurisdiction or product scope, and the corpus and index versions used in the run. A result cannot be reproduced if the search collection changes between the labeling session and the test.

Have a domain reviewer identify the smallest passage or set of passages that would allow a correct answer. Give each passage a stable document ID, passage ID, and source version. Label a passage as required, helpful but insufficient, or ineligible. Ineligible includes revoked access and superseded content even when the wording looks highly relevant. A second reviewer should resolve ambiguous labels before a search configuration is tuned against them.

Some queries have no eligible supporting passage. Mark their gold set as empty and record why: the material does not exist, has not reached the index, is outside the caller’s permission, or is not yet effective. An empty gold set is a deliberate test case, not a missing label.

Inspect the ranked list before the generated answer

Run the same questions through the actual retrieval endpoint. Capture the ordered top-k passage IDs, scores if available, applied filters, index timestamp, and snippets actually sent downstream. For answerable queries, record whether a required passage appeared at rank 1, within the first three, or only beyond the context budget. For empty-gold queries, inspect whether a plausible but unrelated hit was incorrectly treated as evidence.

Do not collapse every failure into one average. Keep separate counts for missing passage, poor rank, wrong version, permission leak, and false evidence on a no-answer case. A required clause at position eight is different from a clause absent from the index; increasing k might help the first case while making the prompt noisier. Microsoft’s document retrieval evaluator illustrates this separation with relevance labels, ranking measures, and missing-judgment checks. A score is useful only when the labels and the search configuration are recorded with it.

Example: an old policy outranks the current clause

A support assistant answers questions about returns. The query asks whether an item purchased after a policy change can be returned without a receipt. The old policy says yes; the current effective policy requires an order number. A reviewer labels the current clause required and the old clause ineligible for this date.

The old text ranks first because it matches the question more closely, while the current clause appears fourth. The diagnosis is version filtering or ranking, not answer wording. The team records the index state, adds an effective-date filter or version preference, and reruns the same query. It checks that the current clause reaches the available context and the retired one no longer masquerades as a valid source. Index updates are part of the test: source edits do not prove that the searchable copy has changed. When the source itself was revised or withdrawn, use an index refresh acceptance trial to verify propagation before rerunning this fixed-corpus retrieval comparison.

Example: no authorized answer remains

An employee asks whether contractors receive a discretionary bonus. The only document that discusses the bonus is restricted to payroll staff; the employee’s visible handbook says nothing about contractors. For this caller, the gold set is empty. A run under a payroll identity can be used as a separate positive control, but it cannot substitute for the employee’s test.

If the restricted passage appears anywhere in the employee’s returned context, the trial fails even when the final answer refuses to quote it. If the engine returns a general compensation page, a reviewer marks it insufficient rather than inventing a supporting passage. The retrieval handoff should report that no eligible support was found; the no-evidence response gate then decides what the user may be told and whether a person must take over. Recheck after permission changes: Azure AI Search documentation makes clear that query-time decisions depend on permission metadata already synchronized to the index.

Change one retrieval variable at a time

For each failed query, choose a candidate fix tied to the observed cause: ingestion for a missing document, a chunking trial for a split clause, the document-access trace for identity, filter, or citation failures, a retrieval-strategy comparison for a buried gold passage, or an evidence threshold for empty-gold cases. Repeat the trial on the same frozen questions, then add a small holdout set that was not used to tune the change. Record both improvements and regressions by case type.

The handoff record should name the query, caller, corpus snapshot, gold passage IDs or empty-gold reason, returned top-k IDs, first required rank, failure category, fix, and retest result. These fields make a disagreement inspectable. They also stop a high average retrieval score from concealing one unauthorized result or one obsolete passage that changes a user’s decision.

Boundary

This trial measures whether eligible evidence is retrieved and passed forward. It does not prove that the source is true, that the model faithfully uses it, or that a cited sentence supports the final claim. Check the answer and run a citation and freshness audit after retrieval passes its own gate.