RAG (Retrieval-Augmented Generation) Patterns
RAG (Retrieval-Augmented Generation) Patterns
Section titled “RAG (Retrieval-Augmented Generation) Patterns”Retrieval-Augmented Generation grounds an LLM’s answer in documents you retrieve at query time, instead of relying only on what the model memorized in training. It’s the standard way to give a model private, current, or domain-specific knowledge without fine-tuning — and to get citations.
The pipeline is simple to describe and surprisingly easy to get wrong: chunk → embed → retrieve → (rerank) → generate. Each stage has decisions that dominate quality.
Chunking strategies
Section titled “Chunking strategies”How you split documents determines what can be retrieved. Too large and you bury the relevant sentence in noise (and waste context); too small and you lose the surrounding meaning.
- Fixed-size with overlap — simple baseline (e.g. ~500 tokens, 10–15% overlap so a fact split across a boundary still appears whole somewhere).
- Structure-aware — split on headings/paragraphs/code blocks so chunks are semantically coherent. Usually beats blind fixed-size on real documents.
- Contextual chunking — prepend chunk-specific context to each chunk so an isolated paragraph still carries meaning. Anthropic’s Contextual Retrieval generates a short situating blurb per chunk with an LLM (reporting a large drop in retrieval-failure rate, more so combined with reranking) — distinct from prepending one document-level summary to every chunk. Reduces “orphaned chunk” retrieval misses.
Embedding models
Section titled “Embedding models”Embeddings turn text into vectors so semantically similar text lands nearby. Pick a model by your needs (quality, dimension/cost, max input length, multilingual, self-hostable) and benchmark on your own data — leaderboard rank rarely transfers perfectly. The MTEB leaderboard is the living reference for retrieval embedding quality. Critically: the same model must embed both documents and queries, or the vectors aren’t comparable.
Retrieval: BM25 vs vector vs hybrid
Section titled “Retrieval: BM25 vs vector vs hybrid”| Method | Strength | Weakness |
|---|---|---|
| BM25 (keyword/lexical) | Exact terms, names, codes, rare tokens | No synonym/paraphrase understanding |
| Vector (semantic) | Meaning, paraphrase, fuzzy match | Misses exact identifiers; opaque |
| Hybrid (both, fused) | Best of both | More moving parts |
In practice hybrid wins for most corpora: run BM25 and vector search in parallel and fuse with Reciprocal Rank Fusion (RRF). Pure vector search famously fails on things like part numbers and error codes that BM25 nails. See the Prompt Engineering Taxonomy for how to present the retrieved context to the model.
Reranking
Section titled “Reranking”First-stage retrieval favors recall (get the candidates); a reranker then favors precision. A cross-encoder reads the query and each candidate together and rescores them — much more accurate than the bi-encoder similarity used for retrieval, but too slow to run over the whole corpus. The standard pattern: retrieve top ~50, rerank to top ~5, send those to the model.
Evaluation
Section titled “Evaluation”“It looks good” is not evaluation. Measure the two halves separately:
- Retrieval quality — does the right chunk get retrieved? Use recall@k / precision@k / MRR against a labeled query set.
- Generation quality — is the answer faithful to the retrieved context (no hallucination) and relevant to the question? Frameworks like RAGAS score faithfulness, answer relevance, and context precision/recall.
Build a small golden set of question→expected-source pairs early; it’s what lets you change chunking or models without flying blind.
Common failure modes
Section titled “Common failure modes”- Retrieval miss — the answer isn’t in the top-k. Usually chunking or embedding mismatch; hybrid + reranking helps.
- Lost in the middle — models attend less to the middle of a long context; put the most relevant chunks first/last and keep context tight.
- Hallucination despite context — instruct the model to answer only from the provided context and to say “I don’t know” otherwise; verify with faithfulness metrics.
- Stale index — re-embed when documents change; a RAG system is only as fresh as its index.
Related
Section titled “Related”- Agentic AI Patterns — agents that call retrieval as a tool
- Prompt Engineering Taxonomy — presenting context and constraining output
- AI Prompting Guide — community prompting practices
- AI/ML Learning — foundational concepts