Decide whether RAG is worth building

Retrieval-augmented generation is valuable when the answer depends on information that must be selected at request time. It is unnecessary when a small, stable reference can be supplied directly or when the task does not need external facts.

The decision is not whether RAG is modern. It is whether retrieval removes an observed source-selection bottleneck in this workflow. Prove the bottleneck with real questions before accepting the indexing, authorization, evaluation, and update machinery.

Ask what changes between requests

Start with five dimensions:

• Volatility: how often the relevant facts change.
• Volume: how much material could plausibly contain the answer.
• Traceability: whether important claims must show their source.
• Access: whether different users may see different documents.
• Maintenance: who will ingest, remove, label, and test content over time.

RAG becomes more defensible as volatility, volume, traceability, and access complexity rise—provided the organization can own the maintenance.

Compare with smaller alternatives

Before building an index, test three simpler routes: attach the current document, place a short approved reference in the prompt, or use deterministic lookup for structured fields. The smallest route that meets the acceptance contract is usually easier to observe.

Retrieval is not a quality upgrade by itself. It adds a new decision: which material deserves to enter the model’s context.

Apply an investment gate

Write evidence for each row rather than assigning an impressionistic score.

Decision factor	Evidence for building	Signal to wait
Request-time selection	Different questions genuinely need different sources	Every request uses one short reference
Source range	People cannot reliably locate the needed passage	A small set has an obvious owner and selector
Change rate	Retired material must leave answers promptly	Content is stable and can ship with a version
Citation need	Decisions must be checked against original text	Work only rewrites or transforms format
Access variation	Visible sources depend on requester identity	Every authorized user sees the same material
Operating capacity	Ingestion, deletion, testing, and incidents have owners	Sources have no steward or lifecycle state

Pause if authorization cannot run before retrieval or if sources lack effective and retired states. Faster search does not compensate for exposing material that should never enter model context.

Run an offline retrieval trial before implementation

Sample questions from the actual work. For each, label the permitted sources, passage that must be found, confusing near-match that should be excluded, and the correct “no answer in the sources” result. Test retrieval alone before generating prose. Record whether the target appears, its rank, why incorrect material was selected, and whether authorization precedes similarity ranking.

Then pass the retrieved material to answer generation and inspect evidence support, preserved qualifications, and citation location. A correct retrieval followed by a wrong answer is different from never finding the source. The complete approach earns a release case only when both stages meet their respective acceptance rules.

Finish the trial with one of three decisions: build, repair source governance first, or continue with a smaller alternative. “The demo looked good” is not a decision.

Example: regional policy library

A company has hundreds of policies that change by country, employee type, and effective date. Answers must cite the exact clause visible to the requester. Sending the whole library is impossible, and a static summary would become stale.

RAG has a clear job here: filter by access and jurisdiction, find candidate passages, and preserve citations. The evaluation must test old versions, similarly named policies, and users with different permissions.

Counterexample: a stable writing guide

A small team uses a two-page style guide updated twice a year. Every writing request needs the same rules. Adding a retrieval service creates indexing, monitoring, and failure modes without reducing much context.

The guide can be versioned and included directly. Retrieval may become useful later if the source set grows, access splits, or the task needs examples from a large archive.

Calculate the operating promise

Write down who owns source approval, deletion, access changes, freshness, retrieval evaluation, and incident review. If those responsibilities have no owner, the system cannot promise grounded answers for long.

Draw the user path for failure as well. Decide whether empty retrieval produces a transparent no-evidence response, a clarifying question, or a human handoff. Decide how conflicting sources remain visible and whether indexing delay can fall back to a known source version. Put these behaviors in acceptance cases before launch.

Use a one-page RAG decision record

Capture the target task, observed source-selection failure, results from the three smaller alternatives, source owner, authorization point, offline retrieval cases, answer gate, expected update rhythm, stop condition, and review date. Reopen the decision when ownership or source structure changes; an old build approval is not permanent permission.

Common questions

Does a large document collection automatically require RAG? No. If the user explicitly selects one file for each task, total collection size is irrelevant; document parsing may be the actual bottleneck.

Can we connect a vector database before evaluating? A disposable engineering probe is fine, but connection is not evidence for release. Without fixed questions, correct sources, and no-answer cases, no one can tell whether retrieval helped.

How should RAG be compared with long context? Compare source selection, version replacement, access enforcement, citation location, and maintenance—not advertised capacity alone. Direct context is often simpler for short approved references; request-dependent collections create a stronger retrieval case.

Boundary

RAG cannot make an unsupported source true, resolve contradictory policy, or guarantee that the model uses a passage correctly. It improves access to evidence; citation audits and answer evaluation remain separate work.