A RAG chatbot answers from your own documents instead of from the model's memory. Gather and clean your content, split it into chunks, embed and store them, retrieve the most relevant chunks for each question, give them to the model with instructions to answer only from them, then test the answers before you publish.
What RAG changes about a chatbot
A plain chatbot answers from whatever the model learned in training. That's fine for general chat and useless for questions about your returns policy, your product range or last week's update. Retrieval-augmented generation, or RAG, fixes that by fetching passages from your own documents for each question and handing them to the model with the question. The model then answers from your sources, not from memory.
The reason you can't just paste everything into the prompt is size. A model doesn't have your documents, and it can't fit them all in its context window. So the system searches first, picks the few chunks that matter, and sends only those.
Which route fits you: a hosted builder or a framework
You don't write code and want a bot on your site
A hosted builder is the shorter road. You add files, web pages and text as sources, and the platform handles chunking, storage and retrieval. Chatbase works this way: it takes files, websites, text, Q and A pairs and Notion pages, and its docs say a new source is picked up within seconds, with no retrain button.
You have a developer and want control
A framework lets you choose every part: how documents are split, which embedding model, which vector store, which language model, how answers are checked. LangChain, LlamaIndex and Haystack are the common starting points, and a vector store such as Qdrant sits behind any of them.
You want an API without assembling the parts
A hosted platform in between gives developers retrieval and generation through one API. Vectara is built this way, with docs covering ingestion, search and retrieval, and evaluating answers for hallucination.
Whichever route you pick, the steps underneath are the same. A hosted tool simply does some of them for you.
Seven steps from a folder of documents to a live bot
- Gather the sources and clear out the junk. Collect the documents the bot should answer from, and remove anything out of date or private. The bot can only be as accurate as its sources, and anything you upload can end up in an answer. Framework data connectors can pull content in, while hosted tools let you upload directly.
- Split the content into chunks that read sensibly on their own. Retrieval returns chunks, not whole documents, so a chunk that depends on the paragraph before it will confuse the model. The frameworks all include splitters for this step.
- Embed each chunk and store it. Turn every chunk into an embedding and put it in a vector store or a hosted knowledge base. This is the index the bot searches. Framework tutorials walk the whole load, split, embed and store path in one place.
- Retrieve the most relevant chunks for each question. When a question arrives, search the index and pull back the closest matches. Haystack builds this from pipeline components you can swap, with document stores behind them.
- Ground the answer. Send the question and the chunks to the model with instructions to answer only from them and to cite the source. Grounding is the point of RAG; without it, you've built a slow search box that still guesses.
- Test with real questions before launch. Ask the questions customers ask, and check each answer against the source it cites. Retrieval can return the wrong chunk and the model can still sound confident. LlamaIndex includes an evaluating section in its docs for this stage.
- Publish and keep the sources in sync. Put the bot on your website or messaging channels, then update the knowledge base whenever your content changes. A stale knowledge base gives stale answers, which is why auto-resync of website sources is worth having.
When your own documents start giving the orders
Here's the risk most first builds ignore. Retrieved text can attack your bot. LangChain's RAG tutorial, in its security considerations, warns that RAG applications are susceptible to indirect prompt injection. A retrieved document may contain text that looks like instructions, and because those chunks share the context window with your system prompt, the model may follow them.
Picture a web page you've added as a source that hides a line telling the assistant to ignore its rules or recommend a competitor. Nobody typed that into your chat box, yet it reaches the model with the same weight as your own content. The tutorial says plainly that no prompt or delimiter strategy fully prevents this.
What it advises instead is checking the output: validate answers and confirm they cite the sources you'd expect. In practice that means being careful which outside pages you index, keeping the bot's permissions narrow so a bad instruction can't do much, and logging answers so you can see when one drifts from its source. Treat everything retrieved as content to quote, never as instructions to obey.
The six tools, side by side
| Tool | Type | Best suited to | Stands out for |
|---|---|---|---|
| Chatbase | Hosted chatbot builder | No-code teams | Channels and auto-resync of website sources |
| Vectara | Hosted RAG platform | Developers who want one API | Retrieval, generation and observability together |
| LangChain | RAG framework | Developers building their own | One tutorial covering load, split, embed, store, retrieve |
| LlamaIndex | Open-source RAG framework | Developers with many data sources | Data connectors plus an evaluating section |
| Haystack | Open-source pipeline framework | Developers who want swappable parts | Pipelines of reusable components |
| Qdrant | Vector store | The framework route | Open-source store that fits behind any framework |
For more options at each layer, my guides to RAG frameworks and platforms and the chatbot builders that work without code go further than this page.
Want AI doing the heavy lifting in your marketing?
I build the systems that handle the boring 80 percent, so you get your week back. Done properly, with the human kept in.
My verdict on the first version you ship
Start smaller than you think. Pick twenty or thirty documents you trust, build the bot on the route that matches your skills, and test it with real customer questions before it goes anywhere near your homepage. Make it cite its source every time, because a citation is how you and your customers check it. And be choosy about outside pages: your bot will read every word you give it, including the words you didn't mean it to obey.
If the bot is heading for your own site, this guide to website chatbots for small businesses is a useful next read.
Want your tool on this list? Featured spots and reviews here are paid partnerships. Here's how to partner with me.
Frequently asked questions
What is a RAG chatbot?
A chatbot that retrieves passages from your own documents and gives them to the language model, so answers come from your sources and not only from training data.
Do I need to code to build one?
No. A hosted builder lets you add files, websites and text as sources without code. The code frameworks are for developers who want control over each part.
Do I need a vector database?
The framework route stores chunks in a vector store such as Qdrant. Hosted builders and RAG platforms manage retrieval for you.
Does a RAG chatbot need retraining when my content changes?
Chatbase's docs say there's no retrain button: a new source is picked up within seconds, and website sources can be resynced.
Can a RAG chatbot still make things up?
Yes. Retrieval can return the wrong chunk and the model can still sound confident, so validate outputs against the retrieved sources and test with real questions.
