RAG Tutorial: Build a Reliable Retrieval-Augmented Generation Pipeline

Overview

Retrieval-augmented generation, commonly called RAG, combines two separate capabilities: retrieving relevant information from a controlled collection and asking a language model to generate an answer using that information. Instead of relying only on what the model learned during training, a RAG system supplies task-specific context at request time.

A dependable pipeline usually contains these stages:

1. Ingestion: collect documents and extract their text, structure, and metadata.
2. Chunking: divide documents into passages that can be searched and supplied to the model.
3. Embedding: convert passages and user queries into numerical representations.
4. Indexing: store embeddings, text, and metadata in a searchable vector database or equivalent index.
5. Retrieval: find candidate passages using semantic search, keyword search, filters, or a combination.
6. Answer generation: place the selected context into a controlled prompt and request a grounded response.
7. Evaluation: measure retrieval quality, answer quality, citations, latency, cost, and failure behavior.

The most important design principle is to treat retrieval and generation as different problems. A fluent answer does not prove that the correct passages were retrieved. Likewise, strong search results do not guarantee that the model will use them accurately. Test each stage independently before judging the complete application. For broader system design, see How to Build an AI Agent: Architecture, Tools, and Deployment Checklist.

Checklist by scenario

When starting a new RAG pipeline

• Define the answer boundary: decide which questions the system should answer and which it should decline or route elsewhere.
• Identify the source of truth: list the document collections, owners, formats, update cadence, and access rules.
• Preserve document structure: retain titles, headings, section paths, page numbers, URLs, timestamps, and document identifiers where available.
• Choose a chunking strategy: begin with chunks that preserve a complete idea, then test size and overlap against real questions.
• Attach useful metadata: include product, department, language, version, audience, effective date, and permissions when those fields affect relevance.
• Plan retrieval before generation: decide whether the first version needs semantic search, keyword search, metadata filters, reranking, or a hybrid approach.
• Specify refusal behavior: instruct the model to say when the supplied context does not support an answer rather than filling gaps from unsupported assumptions.

When improving poor recall

Recall is weak when relevant information exists in the source collection but does not appear among the retrieved results. Work through this checklist:

• Test the exact user question against the index and inspect the top results manually.
• Compare short, long, conversational, and keyword-heavy versions of the query.
• Check whether important terms were lost during parsing, OCR, normalization, or chunking.
• Try smaller or larger chunks while keeping related headings and definitions together.
• Use metadata filters only when their values are reliable and the query clearly supplies the corresponding constraint.
• Consider query rewriting, multiple search queries, hybrid retrieval, or a reranking stage for difficult searches.
• Separate “not retrieved” from “retrieved but ignored.” The remedy is different for each failure.

When answers are relevant but unreliable

• Show the retrieved passages to reviewers before changing the generation prompt.
• Require citations or source references that map to specific retrieved chunks.
• Tell the model to distinguish direct evidence, reasonable interpretation, and missing information.
• Place instructions outside the retrieved text so document content cannot silently override application rules.
• Use a structured output format when downstream software needs claims, evidence, confidence notes, or source identifiers.
• Test contradictory documents, outdated versions, incomplete procedures, and questions with no answer in the collection.

When the source collection changes

• Record an ingestion version or content snapshot so an answer can be traced to the indexed data.
• Remove or deactivate deleted documents instead of leaving stale chunks searchable.
• Re-embed content when the embedding configuration changes, and keep the old index available until the replacement is checked.
• Verify that document permissions and metadata remain aligned after updates.
• Run a regression set containing questions affected by the changed documents.

What to double-check

Ingestion quality: Tables, scanned PDFs, repeated headers, footers, code samples, and page breaks can produce damaged text. Inspect representative documents rather than assuming that successful file processing means successful extraction.

Chunk boundaries: A chunk should contain enough context to stand on its own. Splitting a definition from its conditions, or a procedure from its prerequisites, can create misleading matches. Overlap may help continuity, but excessive overlap can increase index size and return near-duplicates.

Metadata filters: Filters are valuable for tenant isolation, document versions, departments, and dates. They are also a common source of empty or incomplete results. Log the filter values used for each request and test missing, malformed, and conflicting metadata.

Prompt construction: Make the context delimiters clear. A practical generation instruction should state the task, identify the available evidence, define what to do when evidence is insufficient, and specify the expected response format. Prompt design principles can be reviewed in Prompt Engineering Best Practices: A Framework for Reliable LLM Outputs.

Citations: A citation should identify the actual supporting source, not merely the collection name. Check that every important claim has support and that citations are not generated for passages the model did not receive.

Evaluation: Build a small, representative test set before optimizing. Include answerable questions, unanswerable questions, questions requiring multiple chunks, similar-but-wrong documents, and questions affected by version or date. Track retrieval recall, relevance, groundedness, citation correctness, refusal behavior, latency, and cost as separate signals. The LLM Evaluation Guide provides a useful framework for test sets and regression checks.

Common mistakes

Another frequent mistake is optimizing only for a single successful demonstration. A RAG application serves a distribution of questions, not one carefully selected example. Keep examples that expose ambiguity, missing information, conflicting sources, and permission boundaries. If the RAG pipeline is part of an AI agent, also test tool selection, memory, and guardrails using the guidance in How to Build an AI Agent with Tools, Memory, and Guardrails.

When to revisit

Revisit this checklist before seasonal planning cycles, major content updates, model or embedding changes, new tenant or department launches, and changes to source-system permissions. It is also worth reviewing after a noticeable increase in unanswered questions, irrelevant citations, user corrections, latency, or operating cost.

For a practical maintenance cycle, schedule four checks:

1. Content check: confirm that indexed documents are current, complete, deduplicated, and correctly permissioned.
2. Retrieval check: sample real queries and inspect chunks, filters, scores, and missing-result cases.
3. Generation check: verify grounded answers, refusal behavior, citation accuracy, and output structure.
4. Regression check: run a stable evaluation set after changes and compare results with the previous baseline.

Keep logs that connect each response to the query, retrieved document identifiers, prompt version, model configuration, and evaluation outcome. This makes maintenance actionable instead of anecdotal. For production operations, pair these checks with the logging and feedback practices in AI Agent Observability: Logs, Traces, and Feedback Loops That Matter.

Finally, change one major variable at a time when diagnosing quality. If chunking, retrieval settings, prompt instructions, and the model all change together, you may improve the result without learning which component caused it. A reliable RAG system is not finished when it produces a good answer; it is maintainable when you can explain why that answer was produced, detect when it stops being reliable, and correct the responsible stage.