Ask a general AI chatbot about your company's refund policy and it will produce a confident, plausible and probably wrong answer. It has never seen your policy. It is filling the gap with patterns from everything else it has read.
Retrieval-augmented generation, usually shortened to RAG, is the standard way to fix that. It is behind most genuinely useful business AI tools: support assistants, internal knowledge search, sales helpers and document Q&A.
The idea in one paragraph
Instead of asking the AI model to answer from memory, you first retrieve the most relevant pieces of your own information, then give them to the model along with the question, and ask it to answer using only that material. It is the difference between asking someone to recall a document from memory and handing them the document open at the right page.
How it works, step by step
1. Collect your knowledge
Gather the sources the AI should answer from: help articles, policies, product sheets, contracts, SOPs, past support replies, and database records like products or orders.
2. Split it into chunks
Long documents are broken into smaller passages, usually a few paragraphs each, so the system can find precisely the part that answers a question rather than an entire 40-page manual.
3. Turn chunks into embeddings
Each chunk is converted into an embedding: a list of numbers that represents its meaning. Passages about similar topics end up with similar numbers, even if they use different words. "How do I get my money back?" and "refund process" land close together.
4. Search by meaning
When a question comes in, it is converted the same way, and the system finds the chunks closest in meaning. Many production systems combine this with traditional keyword search, because exact terms like product codes and names matter too.
5. Generate a grounded answer
The top results are passed to the language model with instructions such as: answer using these sources, cite them, and say you do not know if the answer is not there. The model's job becomes reading and summarising, which it is very good at, rather than remembering, which is where it fails.
Why not just train a model on your data?
Fine-tuning a model on your documents sounds intuitive, but for factual Q&A it is usually the wrong tool:
- Freshness: when a policy changes, RAG only needs the document updated. A fine-tuned model needs retraining.
- Traceability: RAG can show which source an answer came from. A fine-tuned model cannot.
- Access control: RAG can filter sources by who is asking, so staff see internal documents while customers only see public ones.
- Cost: RAG works with off-the-shelf models and is cheaper to start and maintain.
Fine-tuning has its place, for example in teaching a specific style or format, but it is rarely the first step.
Where RAG helps a business
- Customer support: answering questions from your help centre and policies, with handoff to humans.
- Internal knowledge: letting staff ask "what is our process for X?" instead of searching shared drives.
- Sales enablement: pulling the right case study, spec or pricing rule during a conversation.
- Document-heavy work: contracts, compliance documents and technical manuals.
What makes RAG work well
Building a demo takes an afternoon. Building something reliable takes care in a few areas:
- Content quality. If your documents contradict each other or are out of date, the AI will faithfully repeat the confusion. Cleaning up sources is often the most valuable part of the project.
- Chunking strategy. Chunks that are too small lose context. Chunks that are too large dilute relevance. Structure-aware splitting, by headings and sections, usually beats fixed sizes.
- Retrieval quality. Most bad answers come from retrieving the wrong passages, not from the model. Hybrid search and re-ranking help a lot.
- Clear refusal behaviour. The assistant must say "I'm not sure" rather than invent an answer.
- Evaluation. Keep a set of real questions with known correct answers and test against it every time you change anything.
- Permissions. Make sure users can only retrieve what they are allowed to see.
Is RAG right for you?
If people in or around your business repeatedly ask questions whose answers already exist somewhere in your documents or data, RAG is likely a strong fit. If the answers do not exist anywhere yet, start by writing them down. AI can only retrieve what you have recorded.
We design and build RAG systems with retrieval, evaluation and guardrails, connected to the tools your team already uses. Tell us what questions you want AI to answer.