RAG (Retrieval-Augmented Generation)

RAG fetches relevant documents from a database before answering – so an AI can use knowledge that wasn't part of its training.

The problem

An LLM only knows what ended up in its weights during training. It doesn't know current events, internal company documents, or your private knowledge base – unless you paste them into the prompt. But with thousands of documents, that no longer fits into a context window.

The solution: Retrieval-Augmented Generation

RAG combines search with text generation. Instead of cramming all knowledge into the model, this process runs automatically on every request: your question is turned into an embedding vector, the most similar text snippets are found in a database using that vector (retrieval), and only those few relevant snippets are added to the context window together with your question. The LLM then formulates its answer based on this supplied material.

Why this is useful

Makes current and private knowledge usable without retraining the model
Less hallucination, because the answer is based on real, supplied sources instead of pure memorization
Answers can be backed up with source citations

A refinement

Anthropic describes a technique called "Contextual Retrieval," where extra context from the overall document is added to each text snippet before embedding – this significantly improves how reliably the right snippets get found.