RAG Best Practices
RAG grounds LLM in external knowledge - answering Qs by retrieving relevant context from your own docs => sends it to LLM to generate accurate answers. This prevents hallucinations or incorrect info, provides references. Deploying requires extensive experimentation to tune / optimize many params:
Evaluation - component-wise (retrieval and generation evaluated separately) or end-to-end eval. Retrieval metrics: recall@k, precision@k, MRR, NDCG (need labeled relevant chunks). Generation metrics (RAG triad, reference-free via LLM-judge): retrieval quality (recall@k etc.), faithfulness/groundedness (a. supported by chunks), answer correctness vs. GT (RAGAS). End-to-end: answer correctness vs. golden answers from a reference dataset, human scores. Build the eval set early (50–200 Q&A pairs) = evaluation-driven development.
Data quality (if inaccurate, biased d.) - data prep, diverse d., knowledge graph, addit. context / metadata
Ingestion: Most real-world RAG failures start before chunking - parse PDFs/HTML/Office w/layout awareness — tables, headers, figures (tools: Unstructured, Docling, LlamaParse; OCR for scans). Extract metadata (title, date, author, source, permissions) — needed for filtering & citations. Freshness: incremental indexing on doc change (upsert by doc ID + content hash), handle deletes; schedule syncs from sources (Confluence, SharePoint, S3). Garbage in → garbage retrieved.
Chunking - heavily affects RAG quality (more’n embeddings). Effective chunk sizes of 100-700, larger chunks - noise. # chunks - diminishing returns beyond 7 due to LLM's context length. Strategies include using smaller chunks or chunk overlapping (retrieving adjacent chunk content), specifically: fixed-size w/10–20% overlap (baseline), recursive splitting (paragraph → sentence → word boundaries), semantic chunking (split where embedding similarity between adjacent sentences drops), structure-aware (Markdown/HTML headers, code functions, PDF layout). Small-to-big / parent-document retrieval: embed small chunks for precise matching, but pass the larger parent chunk to the LLM for context.
Embeddings - off-the-shelf models work well often, but fine-tune embed model on the domain if needed. Smaller models may outperform large ones. Select via MTEB leaderboard (Massive Text Embedding Benchmark); typical dims 384–3072; cosine or dot-product similarity (c. measures angle only, dp - angle + vector magnitude; if vectors are L2-normalized - no difference. Magnitude = longer docs = more confidence in info, short docs are more ambiguous.) Check what loss func embed model was trained with: dot / inner product loss or cosine (normalized contrastive) loss and use that one - mismatching degrades retrieval quality. Matryoshka-trained models allow truncating embed dims w/little loss (training at different dims 32, 64, 128, 256, 512, 768 - forces model to pack the most important info into earliest dims => first 64 dims capture coarse semantics, 128 adds finer distinctions, 256 gets you most of full quality. Fine-tuning uses contrastive pairs (query, positive passage, hard negatives). Asymmetric models embed short queries and long passages w/different prefixes (‘query:’ / ‘passage:’) - getting dual-encoder-like behavior from a single model.
Retrieval – a) term-based (e.g. BM25) wins on exact terms (product names), IDs, jargon, b) vector similarity (embeddings) wins on paraphrases/synonyms, c) hybrid retrieval is the robust default. Add metadata filtering (date, source, tenant), applied pre- or post-ANN search.
Reranking - more robust than retriever, e.g. binary embed quantiz. for retrieval, but int8 for reranking
LLM – experiment for your app's unique demands - accuracy, latency, cost. Larger models better for reasoning, use efficient Mixture of Experts (Mistral outperforms Llama 2 70B), intent classifiers help map user query to predefined canonical forms.
Tune and explore configurations, optimize hyperparameters.
RAG Benefits
Up-to-date Info: LLMs - fixed knowledge cutoff dates; RAG easily incorporates current info into LLM’s output, bypassing the limitations of LLM finetuning - overfitting or catastrophic forgetting.
Data Privacy & Security: safer alternative vs. training LLM on proprietary data (extraction attacks).
Reducing Hallucinations (grounded in real data) + verifiable references.
Simpler and more cost-effective to implement (no MLE expertise) vs. finetuning, RAG enhances retrieval model quality without the need to train the LLM. BUT COST TRADE-OFF: RAG shifts cost from one-time training to per-query inference - retrieved chunks are re-sent on every call. Mitigate w/prompt caching, smaller k, reranking down to fewer better chunks, and semantic caching of frequent answers.
RAG vs. Fine-Tuning
Fine-tuning increases accu by 6%. RAG additionally increases accu by 5% more. Choice of approach depends on specific app, nature and size of data, available resources for model development. Use either or both:
Fine-tuning = model adaptation - changing the LLM’s behavior (structure=weights), vocabulary, writing style, customizing model's tone or jargon for a niche application, refine safety and helpfulness - knowledge mostly unchanged (Google example);
RAG relies on updated external data to generate outputs grounded to custom knowledge while the LLM’s vocab and writing style are unchanged. RAG is for knowledge, fine-tune is inefficient for this.
if your app needs both custom knowledge & LLM adaptation - use a hybrid approach (RAG + fine-tuning),
if you don't need either, prompt engineering is the way to go.
Types of RAG: 1) GraphRAG (RAG w/knowledge graphs (KGs) for augmenting context w/structured domain-specific knowledge; automated KG construction (triple extraction); best for multi-hop questions (facts across docs) & global/corpus-level Qs (‘main themes across all reports’) where chunk-level vector RAG fails, 2) LightRAG - simple & fast, graph-enhanced, near-GraphRAG quality at a fraction of cost/latency: extracts entity–relation graph from docs; dual-level retrieval (low-level: specific entities/relations; high-level: broader themes) = graph traversal + vector search; supports incremental index updates (no full rebuild): Article + GitHub, 3) RAFT trains special Q&A model w/CoT responses - robust in ignoring irrelevant distractor docs, 4) SELF-RAG - self-reflection using fine-tuned LLM to predict if retrieval is needed & then evaluate relevance of retrieved info, 5) Corrective RAG: uses lightweight T5 model to assess quality of retrieved docs (classify as Correct, Ambig. - supplement w/web search, Incorrect - replace w/web) - more flexible, easy to implement than Self-RAG. 6) RAT = RAG+COT: CoT reason → retrieve CoT steps → refine → answer.
Embedding Quantization & Truncation: substantial cost and latency reductions in retrieval and similarity search + fewer bits for storage. Example: [12, 1, -100, 0.3,] => [1,1,0,1,] (0 if negative):
Binary Quantization: 45x lower latency, 32x less memory, 96% of retrieval performance.
Scalar (int8) Quantization: 4x lower latency, 4x less memory, 99.6% of retrieval performance.
Combine both: binary search for min. latency & memory + scalar rescore for high performance = save costs.
Embedding Truncation: w/min. performance loss, faster retrieval, clustering, etc. Train model on domain. Matryoshka Representation Learning (MRL) - leading dims carry most information; truncate e.g. 1024→256 dims for ~4x cheaper search w/small accuracy loss; optionally rescore top candidates w/full-dim vectors.
Errors in RAG Systems
Chunking
E1 Overchunking: small or disjointed chunks => incomplete coverage of topics.
E2 Underchunking: chunks too large, multiple topics, irrelevant info dilutes meaning.
E3 Context Mismatch: arbitrary splitting => breaks context, separates definitions & info they support.
Retrieval
E4 Missed Retrieval: Relevant chunks not retrieved => generator errors (incomplete, hallucin.).
E5 Low Relevance: Retrieved chunks loosely related to query.
E6 Semantic Drift: reliance on keywords hurts semantic relevance => query’s intent mismatched.
E7 Low Recall: necessary chunks retrieved, but reranked too low and not sent to generation model.
E8 Low Precision: irrelevant chunks ranked highly and sent to generation model => noise.
Generation
E9 Abstention Failure: model should have abstained, but answers incorrectly.
E10 Fabricated Content: response includes unverifiable info not grounded in external knowledge.
E11 Parametric Overreliance: LLM relies on its internal (parametric) knowledge, not retrieved docs.
E12 Incomplete Answer: response is correct, but misses critical details.
E13 Misinterpretation: generator misrepresents retrieved content.
E14 Contextual Misalignment: response factual + from related info, but doesn’t address query.
Agentic RAG System - using AI agents to optimize information retrieval and synthesis.
Single-Agent Agentic RAG for simpler tasks (resource efficient) - single agent manages query eval, knowledge source selection, data integration, and response generation. Diverse retrieval methods: structured DBs, unstruct. semantic search, web search, and recommender systems.
Multi-Agent Agentic RAG - scalability & task specialization for more complex applications – coordinator agent orchestrates multiple retrieval agents optimized for the above 4 retrieval methods. Parallel processing enhances efficiency, accu, and retrieval relevance. Retrieved data is sent to LLM to generate comprehensive responses.
Core agentic behaviors on top of static RAG: routing (decide whether to retrieve at all and from which source), per-source query rewriting, and iterative retrieval — retrieve → judge sufficiency → re-retrieve w/refined query until evidence suffices. Fixes single-shot retrieval failures at the cost of latency and tokens.
AI Agents perform tasks autonomously. Benefits: efficiency & cost reduction, scalability & adaptability, enhanced decision making (increasing workloads, respond quickly to market changes).