RAG explained: how to build an AI chatbot that knows your business

Ask a general AI chatbot "What's your return policy?" on a shop's website and it will happily invent one. It sounds confident, it's polite, and it's wrong. That's the number-one reason businesses don't trust AI assistants.

Retrieval-Augmented Generation (RAG) is how you fix that. It's the technique behind most useful company chatbots today, and the idea is simpler than the name suggests.

The idea: an open-book exam

A plain language model answers from memory, like a student in a closed-book exam. If it never learned your return policy, it guesses.

RAG turns it into an open-book exam. Before answering, the system looks up the most relevant pages from your documents and hands them to the model with an instruction: "answer using only this." Now the answer is grounded in your real policy, and it can even show which page it came from.

How it works, step by step

1. Collect and clean your knowledge

PDFs, FAQs, product pages, Notion docs, support tickets. Garbage in, garbage out, so remove duplicates and outdated versions first. This step matters more than the choice of AI model.

2. Split into chunks

Documents are cut into passages of a few hundred words, with a little overlap so sentences aren't cut in half. Each chunk keeps its metadata: title, URL, date.

3. Turn chunks into embeddings

An embedding model converts each chunk into a list of numbers (a vector) that captures its meaning. "Refund window" and "how long do I have to return an item" end up close together even though they share no words.

4. Store them in a vector database

As a Django developer, my favourite option is pgvector: it adds vector search to PostgreSQL, so your AI data lives next to your normal app data. No extra service to run.

5. Retrieve, then generate

When a question arrives, it's embedded the same way, the closest chunks are fetched, and those chunks plus the question go to the language model with clear rules.

A minimal Django + pgvector example

This is a stripped-down version of the pattern I use. embed() and llm() stand in for whichever embedding and chat model you choose.

# models.py
from django.db import models
from pgvector.django import VectorField

class Chunk(models.Model):
    source = models.CharField(max_length=255)      # e.g. "returns-policy.pdf"
    text = models.TextField()
    embedding = VectorField(dimensions=1536)


# rag.py
from pgvector.django import CosineDistance
from .models import Chunk

def retrieve(question, k=5):
    q = embed(question)
    return (Chunk.objects
            .annotate(distance=CosineDistance("embedding", q))
            .order_by("distance")[:k])

def answer(question):
    chunks = retrieve(question)
    context = "\n\n".join(f"[{c.source}] {c.text}" for c in chunks)
    prompt = (
        "Answer using ONLY the context below. Cite sources in [brackets]. "
        "If the answer isn't in the context, say you don't know.\n\n"
        f"Context:\n{context}\n\nQuestion: {question}"
    )
    return llm(prompt)

That's genuinely the core of it: about 25 lines. Everything else is about making it good.

Where RAG projects go wrong

• Bad chunking. Chunks that are too big bury the answer; too small and they lose context. Split on headings and paragraphs, not every N characters.
• Search that only matches meaning. Vector search can miss exact things like order numbers, SKUs and error codes. Combining it with keyword search ("hybrid search") fixes most of this.
• No "I don't know". If retrieval finds nothing relevant, the bot must say so and offer a human, not improvise. Add a minimum similarity threshold.
• Stale knowledge. Update a policy and forget to re-index, and the bot quotes the old one. Re-embed changed documents automatically (a nightly job works fine).
• Never measuring. Keep a list of 30–50 real customer questions with correct answers and re-test after every change. It's the cheapest quality control there is.

RAG or fine-tuning?

For a business chatbot, start with RAG almost every time. Fine-tuning is a later optimisation, and you can combine the two.

Where it pays off

• Customer support: answers shipping, returns and product questions 24/7, with links to the source page.
• Internal knowledge: new staff ask "how do we process a refund?" instead of digging through drives.
• E-commerce search: "a waterproof jacket under $100 for hiking" finds the right products even when the words don't match.

Shubham Nirmal

Python & Django developer in Lalitpur, Nepal. I build fast backends, automation and AI tools, and mentor people starting out in IT.