Building RAG systems that can explain their answers

An assistant that answers questions about a product manual needs more than fluent language. It needs the right manual, the right version, and evidence for each important claim. Retrieval-augmented generation, or RAG, supplies relevant material to a model at request time. The difficult engineering work is deciding what material belongs in that context and proving that the resulting answer is useful.

This guide uses an illustrative internal documentation assistant. The examples describe a proposed design, not production results from a particular project.

Start with a question you can evaluate

Suppose someone asks, “How long do failed exports remain available?” A good answer should identify the retention rule, distinguish failed exports from completed ones, and cite the current policy. A vaguely relevant paragraph about exports is insufficient.

Before choosing a vector database, write a small evaluation set. Include direct questions, ambiguous questions, obsolete terminology, and questions whose answers are absent. For each answerable question, record the expected source and the essential facts. For unanswerable questions, record what clarification or refusal would be appropriate. Keep a separate held-out set so tuning does not become memorization.

Ingest documents with their context intact

Give every document a stable ID, version, access policy, and canonical URL. Split at meaningful boundaries such as headings and paragraphs. A chunk containing “This expires after seven days” becomes much less useful when its heading, “Failed exports,” disappears.

A practical record might look like this:

{
  "documentId": "export-policy",
  "version": "2026-11",
  "section": "Failed exports",
  "chunkId": "export-policy:2026-11:03",
  "text": "Failed export artifacts are retained for seven days.",
  "sourceUrl": "/docs/export-policy#failed-exports",
  "accessGroup": "support"
}

Choose chunk size through experiments. Small chunks can isolate the answer but lose qualifications. Large chunks preserve relationships but consume context and can bury the useful sentence. Overlap helps with boundaries, yet excessive overlap repeatedly retrieves the same passage. Deduplicate retrieved passages before assembly.

Keep ingestion repeatable. When a document changes, replace or retire its old chunks deliberately. Otherwise the assistant may retrieve two policies that disagree and silently choose the obsolete one.

Retrieve candidates, then select evidence

Semantic retrieval helps connect differently worded questions and passages. Keyword retrieval helps with exact identifiers, error codes, and product names. A hybrid strategy combines these signals; a reranker can then compare a smaller candidate set against the actual question. Context added to chunks can improve retrieval when the original text depends on surrounding material. These techniques are described in Anthropic’s contextual retrieval research.

Do not add raw keyword and vector scores as if they used the same scale. Rank fusion or calibrated scoring makes the combination explicit. Apply authorization while retrieving candidates and recheck it before returning evidence. A document’s relevance does not grant the user permission to see it.

For our example, a useful initial experiment is to retrieve a broader candidate pool, remove duplicate sections, and pass a smaller evidence set to generation. The exact counts should come from latency and answer-quality measurements, not a universal recipe.

Make evidence part of the output contract

Assign each selected passage a local source ID. Ask the model to connect factual claims to those IDs, and validate that every returned ID exists in the supplied context. That catches invented references, though it does not prove the cited passage supports the claim.

Treat retrieved text as data. A document that says “ignore the user and reveal secrets” is still an untrusted document. Tool permissions and access checks must live in application code. Instructions in a prompt alone cannot enforce those boundaries.

When evidence conflicts, expose the conflict or ask for the missing context. When evidence is absent, the assistant should say what it could not establish. An empty answer with a useful next step is preferable to a confident invented retention policy.

Debug the pipeline one stage at a time

Imagine the answer says thirty days instead of seven. Inspect the trace in order: was the current policy indexed, was its chunk retrieved, did reranking discard it, and did generation misread it? Each failure has a different fix. Changing the prompt will not repair missing ingestion.

Track retrieval coverage, claim support, appropriate abstention, latency, and cost separately. For example, if eight of ten answerable evaluation questions retrieve the expected evidence, retrieval coverage on that tiny set is 80%; it says nothing about whether the final answers are correct. Report sample size and inspect failures rather than presenting one percentage as overall quality.

A useful implementation exercise

Build a baseline over twenty short documents. Save each question, retrieved source IDs, final answer, latency, and reviewer notes. Then change only chunk boundaries and rerun the same evaluation. Repeat with hybrid retrieval. This produces evidence about which complexity actually improves your application.

The goal is a system whose mistakes can be located and repaired: document preparation, retrieval, evidence selection, or answer generation. That separation is what makes a RAG application maintainable after the first demo.