What Is RAG? Retrieval-Augmented Generation, Explained Simply

Ask a raw LLM about your company's return policy, and it will confidently make something up. That's not a bug — the model simply doesn't know anything it wasn't trained on.

RAG — Retrieval-Augmented Generation — fixes this. It's the technique behind almost every "chat with your documents" product you've seen, and it's one of the most useful things a business can build with AI today.

The one-sentence version

Instead of asking the model to answer from memory, you first retrieve the relevant pieces of your own documents, paste them into the prompt, and ask the model to answer from those. The model stops guessing and starts quoting.

How it works, step by step

There are two phases: indexing (done once, ahead of time) and querying (done per question).

Indexing: preparing your knowledge

1. Collect your documents — PDFs, Notion pages, support tickets, contracts, whatever holds your knowledge.
2. Chunk them into small passages, usually a few hundred words each. Too big and retrieval gets sloppy; too small and chunks lose context.
3. Embed each chunk — an embedding model converts the text into a vector (a list of numbers) that captures its meaning. "Refund policy" and "money-back guarantee" end up as nearby vectors, even with zero shared words.
4. Store the vectors in a vector database — Qdrant, Pinecone, ChromaDB, or pgvector if you already run PostgreSQL.

Querying: answering a question

1. Embed the user's question with the same embedding model.
2. Find the chunks whose vectors are closest to the question vector — this is semantic search.
3. Stuff the top chunks into the prompt: "Answer using only the context below."
4. The LLM writes the answer, grounded in your actual documents.

A concrete example

A law firm I worked with had 3,000+ pages of contracts and case notes. Associates burned hours searching for precedents. With RAG:

• Every document was chunked, embedded, and stored in Qdrant
• An associate now asks: "Which of our past agreements limited liability to fees paid?"
• The system retrieves the relevant clauses and the LLM answers with citations to the exact documents

Search time dropped from hours to seconds — and because answers cite sources, trust in the system is high.

RAG vs. fine-tuning: which one do you need?

This confusion costs people real money, so let's settle it:

Most businesses asking about fine-tuning actually need RAG. If you want the model to know things, use RAG. If you want it to behave a certain way, consider fine-tuning.

Where RAG falls apart (and how to fix it)

RAG is powerful but not magic. The common failure modes:

• Bad chunks. Splitting mid-sentence or mixing topics in one chunk confuses retrieval. Split by structure — headings, sections — not by raw character count.
• Wrong retrievals. If the retrieved chunks are irrelevant, the answer will be too. Hybrid search (vector + keyword) fixes most of this.
• No guardrails. Always instruct the model to say "I don't know" when the context doesn't contain the answer. A system that admits gaps beats one that hallucinates with confidence.

Should you build one?

If your team repeatedly searches documents, answers the same questions, or onboards people into piles of tribal knowledge — yes. A focused RAG system is one of the highest-ROI AI projects a business can run, and it's typically a matter of weeks, not months.

I build these as part of my custom software and AI development service — including private deployments where your documents never leave your own infrastructure. And if you want to see RAG built from scratch, there's a full walkthrough on my YouTube channel.