How should you chunk RAG documents? Keep the full evidence intact

The point of chunking a document for RAG is to deliver enough evidence for a complete decision. A retrieved fragment that says an expense is covered but omits the adjacent approval condition can lead to a wrong answer even when the model faithfully follows that fragment.

This lesson addresses ingestion boundaries. Use the RAG investment gate to decide whether retrieval belongs in the workflow. Use the retrieval trial to diagnose an existing index’s results. Here you will compare chunk designs against the same questions and leave with a reproducible configuration record, rather than a universal number of tokens.

Mark the full evidence before choosing a size

Choose 12–20 real or realistic questions from one frozen source collection. Include clauses whose exceptions cross paragraphs, nested headings, tables, footnotes, short queries, and questions with no eligible answer. Ask a domain reviewer to mark the smallest source spans that jointly support each answer: document ID, version, page or section, and start and end location. A question can require two spans, such as a general rule and its exception. Mark the gold set as empty for a true no-answer case.

Do not define gold evidence by the chunk ID produced by one splitter. IDs change when boundaries change, so they cannot support a fair comparison. Instead, map every candidate chunk back to its stable source spans. The evidence reaching the answer stage can be one chunk or a safely assembled group, but it must include every qualification that changes the decision.

Check extraction separately from chunking

Before splitting a PDF or spreadsheet, inspect the extracted text against the source. Sample the reading order, headings, row and column labels, footnotes, page locations, and content that crosses a page. Use the extraction acceptance sheet to make this a separate original-to-parser gate before comparing chunkers. A missing battery-warranty footnote cannot be restored by increasing overlap. Record that as a parser failure and fix the extraction or supply a reviewed structured source first.

Azure’s layout and content-understanding documentation describes structure-aware processing of paragraphs, headings, and tables. Its semantic chunking option is still preview at the source check date. These capabilities explain why layout deserves inspection; they do not prove that automatic boundaries beat a task-specific rule. Hold the same verified extraction constant when comparing chunkers. Changing the parser and splitter together makes the result hard to explain.

Compare three small candidates

Start with a simple fixed-window baseline. The Azure AI Search chunking guide suggests 512 tokens and 25% overlap as a starting point for fixed-size experiments. Treat those values as candidate settings, not a universal optimum for Chinese text, tables, or every embedding model. Measure length with the tokenizer used by your actual model; a character count is not a token count.

Candidate	Boundary and overlap	Question it tests
A: fixed baseline	About 512 tokens, no overlap, avoiding mid-sentence cuts where possible	Which required spans are split by a simple window?
B: overlap only	Same window as A, initially 25% overlap	Does duplication rescue boundary conditions, and what does it cost?
C: structure first	Split at headings, paragraphs, or table units; cap long units at the same size, initially without overlap	Do document boundaries keep conditions together with less unrelated text?

If a meaningful source unit is larger than the model’s limit, do not force several pages into one chunk. A smaller child can locate the material while a neighboring passage or parent section supplies complete context. Amazon Bedrock documents a hierarchical approach that retrieves child chunks and passes broader parent chunks to the model. Account for the extra context tokens and broader citation range if you try that route.

Preserve provenance on every child

Store the parent document ID, source version, heading path, page or source span, and access metadata with each chunk. Decide whether heading text belongs in searchable content or a separate field, then keep that choice consistent during the comparison. If access depends on the caller, each child must inherit the fields needed for query-time filtering. Azure’s index-projection guidance specifically maps parent permission fields onto child documents; correct boundaries are useless if a restricted child is returned to the wrong person.

A citation should lead back to the narrow location that supports the claim. Returning only a document title makes verification difficult. If you assemble neighboring fragments for the model, do not imply that any one sentence in that assembled context supports every claim. Use the separate citation and freshness test to audit the final answer.

Run the same questions and measure complete evidence

Keep the source snapshot, extracted text, embedding model, retrieval algorithm, ranking settings, top-k value, access filters, and downstream context budget fixed across candidates. Capture both the ranked hits and the exact passages actually handed to the answer stage. Microsoft Foundry separates retrieval evaluation from response evaluation; this experiment asks whether the necessary material arrived before judging the prose.

For each query, record whether all required spans reached the delivered context, the first rank at which evidence became complete, false support on no-answer cases, stale or unauthorized results, citation location, and retrieved tokens. Summarize by case type: complete-evidence coverage, duplicate content, number of indexed chunks and embedding tokens, and retrieval latency. Keep a small holdout set outside tuning so the choice is not fitted only to the examples used to select settings.

Set blocking rules before running the comparison. A missing approval exception that reverses an expense decision, an unauthorized chunk, or a citation that cannot be traced may block a candidate regardless of its average score. If a complete gold passage is already in context but ranks poorly, investigate ranking. More overlap may only make duplicates more prominent.

Example: the expense clause crosses a paragraph

A travel policy first says that international rail fares are eligible expenses. The next paragraph requires a manager’s approval before departure. An employee asks, “Can I claim a cross-border train ticket without advance approval?” The reviewer marks both paragraphs as required source spans; the eligibility sentence alone is insufficient.

If candidate A ends between the two paragraphs and its top-k context includes only “eligible,” it fails the completeness check. Candidate B may duplicate enough text to recover the condition, or it may merely return the same partial rule twice. Candidate C might keep the clause under one heading, but its length, rank, and citation still need checking. No candidate wins by definition; the observed results belong in the worksheet.

Example: a table footnote changes the warranty

A device-warranty table lists model B’s main-unit warranty as 24 months. The battery cell points to a footnote below the table: battery coverage lasts six months. A customer asks, “How long is model B’s battery warranty?” The gold evidence is the model B row with its column relationship and the footnote. The 24-month main-unit value is a plausible distraction.

First inspect the extracted text. If model B’s row has been paired with model A or the footnote is absent, the failure belongs to extraction. Once extraction is correct, compare whether a fixed window drops the note, overlap restores it, or a table-aware unit retains the row, headers, and note. For a very long table, a targeted row can be retrieved together with its headers and related note; a whole-page chunk is not automatically a precise source.

Record the decision in a reusable trial

In the workbench below, name the frozen source and gold spans first. Give A, B, and C one result row each: settings, sample chunk, top-k hits, complete evidence, citation location, and cost. Then record the selected configuration, failures, owner, and trigger for retesting after a change in document shape or model. “Fix extraction first” and “fix ranking first” are valid conclusions when no splitter addresses the observed fault.

Common mistakes

Does more overlap always help? It can rescue text near a boundary, but it also creates more indexed and retrieved duplication. It cannot repair missing text or a broken table relationship.

Are smaller chunks always more accurate? They may locate a narrow phrase more easily while losing its condition, heading, or footnote. Parent or neighboring context may be necessary.

Can the final answer stand in for this test? A fluent answer does not show which evidence was delivered. Check complete source support in the actual context first; then evaluate the answer and its citations separately.

Boundary

This trial suits teams that have chosen their source collection and need to design its search index. It does not establish source truth, access policy, index freshness, answer faithfulness, or production load capacity. For a task that reads one short named file, passing that file directly may be simpler than building a chunked index.