Building a Simple RAG System Yourself

Chunking, embeddings, vector search, context in the prompt – the four steps behind every RAG system, plus when a long context window is simply enough.

The four steps

RAG (Retrieval-Augmented Generation) pulls relevant text snippets from your own knowledge base before each answer and feeds them to the model as part of the prompt. The flow has four steps:

Chunking: Break the knowledge base into small text pieces, usually a few hundred tokens each.
Embeddings: Convert each chunk into a vector with an embedding model that captures its meaning.
Vector search: For a query, find the chunks whose vectors are most similar to it.
Context into the prompt: Insert the retrieved chunks into the prompt before the model answers.

RAG vs. a long context window

If your entire knowledge base fits comfortably in the context window, RAG is often unnecessary overhead – just putting everything in the prompt is simpler and less error-prone. RAG pays off when the knowledge base is bigger than the context window, changes frequently, or only a small, shifting slice is relevant per query.

Common failure points

Chunks too big: irrelevant filler crowds out what matters. Chunks too small: context gets lost. Pure vector search misses exact-term hits (technical terms, product codes) – combining it with classic keyword search catches those.

EXAMPLE

Prompt template after retrieval: "Answer the following question using only the excerpts below. If the answer isn't there, say so. EXCERPTS: [chunk 1] [chunk 2] [chunk 3] QUESTION: How do I cancel my subscription?"

🛠️ EXERCISE — TRY IT YOURSELF

Build a tiny RAG system for a handful of your own text files (e.g. 5-10 notes) and test whether it finds the right source.

Split 5-10 short text files into chunks of roughly 200-300 tokens.
Generate embeddings for each chunk and for 3 test questions whose answers you already know.
Compare cosine similarity and check whether the chunk with the correct answer actually comes out on top.

✅ SELF-CHECK

☐ Did vector search find the right chunk for all 3 test questions?
☐ Was there a case where a wrong but semantically similar chunk won instead?
☐ What would smaller or larger chunking have changed about this result?

QUICK QUIZ

When is a simple RAG system the better choice over just putting the entire knowledge base into the context window?