How do you verify that a RAG document update or withdrawal reached the index?

A knowledge-base owner corrects a document, but the assistant still quotes the old clause. Or the owner withdraws a file, yet a vector hit and a cached answer keep it alive. A successful source edit, indexing request, or scheduled job is one event; the user-visible search and answer are later events. Treat “the index is current” as a claim to verify for a named source revision and withdrawal, not as a setting to assume.

This lesson tests the source-to-index-to-answer propagation path. The retrieval trial asks whether a fixed corpus returns the right passages, and the claim-level source audit asks whether an answer’s claims are supported by current evidence. The document-access trace checks a caller’s authorization, including revocation. Here the source document itself changes or is withdrawn: when do old searchable copies and cached outputs stop appearing, and how can a release owner prove that the replacement is visible?

Give each change an identity and a deadline

Create one record per source event: stable document key, source revision or content hash, event ID, update or withdrawal, source owner, effective time, and time the pipeline observed it. The source’s publication state is authoritative; a recently edited draft does not automatically replace a still-effective policy. Name the exact old revision that must disappear and the new revision, if any, that must become searchable. For a future-effective revision, state when the switch should occur.

Set an acceptable propagation window before testing. Record the clock used for the deadline and the customer-facing action while the change is pending. For a consequential withdrawn document, that action may be to suppress the document by authoritative source state or pause the affected answer route until removal is proven. A time limit is an operating requirement, not a promise from a search engine. If it expires, the owner needs an alert and an explicit fallback.

Layer	Evidence to collect	Why a previous layer is insufficient
Source	Event ID, revision, effective and withdrawal state	Editing a file says nothing about ingestion.
Ingestion	Run ID, cursor or high-water mark, per-item result and error	A completed run may skip a change or fail one item.
Search index	Parent and child IDs, source revision, lexical/vector visibility	A write acknowledgment can precede search visibility.
Delivery	Retrieval, direct citation fetch, result and answer caches	A clean index can still feed an old cached answer.

Match the ingestion route to its real change signal

A push pipeline should check the response for each document, preserve the event or operation ID, and retry failed items without losing an update or deletion. An Azure AI Search push update is processed in the background; a successful indexing response does not by itself prove that the next query sees the new document. A pull indexer has a different clock. Azure’s indexer overview says that after its initial run it processes detected changes, but change detection depends on the data source and configuration. Scheduling documentation sets a minimum interval for scheduled runs; a source edit does not wake a scheduled indexer instantly. Keep the run history, input revision, and the last processed cursor next to the source event.

Deletion needs an explicit signal. Azure’s Blob guidance says deletion detection is not automatic: use its supported native or custom soft-delete policy, configured from the first indexer run. A soft-deleted blob must remain long enough for the indexer to observe it; hard-deleting it first can leave a searchable orphan. Crucially, that guidance excludes one-to-many indexing from both Blob soft-delete options: child index documents need explicit deletion through the index API. Other pipelines may use a tombstone, an outbox event, or an explicit delete by stable ID. Name the mechanism for the actual parent-child layout and test it. Do not infer that “the source file is gone” means every search copy is gone.

If the schema, chunking rule, or embedding model changes, decide whether this is an incremental update or a full replacement build. A new embedding must be made for changed text; a new chunking rule can change the number and IDs of children. Record the old and new index or pipeline versions, stage a replacement when needed, and switch reads only after a complete probe. A reset or rebuild request is a start condition, not acceptance evidence.

Reconcile the parent and every old child

For an updated document, compare the source revision with every searchable child chunk and its embedding. When a revision shrinks from five chunks to three, two old children must be retired; overwriting three matching IDs is not enough. Keep a parent-to-child ledger or another enumerable relation so the verifier can query all prior child IDs, not just the currently expected ones. Check keyword and vector paths, and inspect the exact snippets delivered to the model. A hit with an old revision is still stale even when the new document also appears.

For a withdrawal, the expected replacement set is empty. Check that the old parent, all child text and vectors, extracted summaries, and direct citation references are absent or blocked. Include a positive control: an unrelated current document must remain searchable, so an empty search result does not falsely count as a successful cleanup. If the system cannot enumerate children or reconcile a lost deletion event, stop the claim of completion and assign repair to the ingestion owner.

Test search visibility and caches separately

Elasticsearch documents a near-real-time boundary: acknowledged indexing is not necessarily visible to search until a refresh. Its refresh=wait_for option waits for a refresh, while refresh=true forces one and can impose a throughput cost. If automatic refresh is disabled with index.refresh_interval=-1, wait_for can wait indefinitely until another operation triggers a refresh; it is not a business deadline guarantee. Neither refresh mode repairs a missed source event, stale embedding, or application cache. Do not add an expensive forced refresh to every write just to conceal a missing acceptance probe. Record when a search query first passes under the actual production read path.

Probe at least three inputs against the live retrieval stack: a phrase unique to the new revision, a paraphrase that should find its vector representation, and a phrase unique to the retired revision that must not return it. For a withdrawal, search by exact document and child IDs as well as ordinary user questions. Repeat with application retrieval caches and generated-answer caches enabled; invalidate by affected document or index version according to the system’s cache contract. Test a saved citation or source-preview link separately. The citation audit then checks the final claim against the version that actually reached the answer.

Keep the first passing observation and last failing observation, not a fabricated “instant update” timestamp. Search may become consistent at different times across routes, replicas, and caches. Define pass as all required probes passing within the chosen window; one correct answer does not prove the old version is gone. Keep negative probes in the suite after a later reindex or cache change.

Example: a lab booking rule changes on Monday

A fictional university knowledge base has a room-booking rule, revision R4. A reviewed R5 changes the maximum reservation length but takes effect next Monday. The assistant must continue using R4 until the effective boundary, then use R5. The source event records both the approved revision and effective time. The indexer may ingest R5 early, so retrieval needs a version or effective-state rule rather than “newest timestamp wins.”

The acceptance trial asks a question containing the new limit, a paraphrase of the same condition, and a question matching wording unique to R4. It checks the query path before and after the effective boundary, the chunk revision delivered to the model, and the cached answer. The case is illustrative; no latency or pass result is asserted. Those cells stay empty until the team’s own run.

Example: a supplier sheet is withdrawn

A fictional support team withdraws an inaccurate supplier specification sheet. Its earlier ingestion produced several chunks, vectors, a summary, and a citation preview. The source owner marks revision S2 withdrawn in a publication register, but storage cleanup physically removes the Blob without an explicit deletion of the child index records. A one-to-many index cannot rely on Azure Blob soft-delete policies to remove those children. The verifier treats the missing child-deletion evidence as an open risk, not as proof of removal.

The team checks the prior child-ID inventory and searches for an S2-only phrase. It also opens the saved citation and asks the original customer question through the answer cache. If any old surface remains, the event is not accepted: issue a targeted delete or rebuild according to the backend, invalidate affected caches, and rerun all probes. Keep a different, current supplier sheet as a positive control. The story does not claim the repair has been measured.

Fill the index refresh acceptance trial

The workbench below captures one source event, expected old/new revisions, ingestion evidence, parent and child cleanup, keyword/vector and cached-answer probes, first passing times, the deadline, and a named fallback. Entries remain in this browser; you can copy the sheet or download it as TXT. Use synthetic IDs and sanitized result summaries here; keep restricted source text, real queries, credentials, and internal index endpoints in authorized systems.

Boundary

This trial proves a particular revision’s propagation across the routes you actually tested. It does not prove the source’s factual truth, answer support, or document authorization. A permission-only change belongs in the document-access trace; query ranking on an unchanged corpus belongs in the retrieval trial. Rehearse the update and withdrawal path again after changing the source connector, deletion policy, index schema, chunker, embedding model, cache, or read route.