RAG explained: Why the promise of more accurate AI still comes with sharp edges

For much of the modern artificial intelligence boom, one flaw has dominated public debate. Large language models sound authoritative, but they are not reliably truthful. They can invent sources, misstate facts and blur speculation with certainty. As these systems have crept into workplaces, newsrooms and public services, that weakness has become harder to excuse.

Retrieval-augmented generation, usually shortened to RAG, has emerged as the industry’s favoured answer. The idea is simple and appealing. Instead of asking a model to respond from its internal training alone, you require it to retrieve information from a defined set of documents first, then generate an answer grounded in those sources. In theory, the model stops guessing and starts citing.

The appeal is obvious

Organisations can connect models to policy manuals, legal guidance, product documentation or internal knowledge bases. Answers can be kept up to date without retraining a model. Citations offer the comforting appearance of accountability. For vendors, RAG has become shorthand for “enterprise-ready AI”.

Yet beneath the promise lies a more complicated reality. RAG does not remove uncertainty. It redistributes it. Accuracy becomes dependent not just on the model, but on how documents are prepared, retrieved, secured and interpreted. In other words, the problem shifts from artificial intelligence to systems engineering.

At its core, RAG is an architectural pattern rather than a single technology. Documents are first cleaned and broken into chunks. Those chunks are transformed into numerical representations known as embeddings, which capture semantic similarity. The embeddings are stored in a vector database, a system designed to retrieve items that are “close” in meaning rather than identical in wording.

Model instructions

When a user asks a question, the system converts the query into an embedding, searches the database for the most relevant chunks and feeds them to the language model. The model is instructed to answer using only that material, often with explicit citations back to the source text.

This is where the story usually ends in marketing decks. In practice, every step is a potential point of failure.

Chunking, for example, is often treated as a technical detail. It is anything but. Break documents into pieces that are too small and you strip sentences of their conditions, caveats and exceptions. Break them into pieces that are too large and retrieval becomes imprecise, pulling in irrelevant material that muddies the answer. A single missing paragraph can flip the meaning of a policy from permissive to prohibitive.

Retrieval itself is probabilistic

Vector search finds text that is similar, not necessarily text that is correct. If the system retrieves the wrong passage, the model will still produce an answer, because that is its function. The result can look authoritative and be entirely misplaced. The presence of a citation can make the error more persuasive, not less.

This is one of the uncomfortable truths about RAG. It often fails silently. When retrieval misses the best source, users rarely see an error message. They see a fluent response with footnotes. The risk is not wild hallucination, but subtle misrepresentation.

There is also the problem of staleness. Many organisations maintain multiple versions of the same document. Unless metadata such as version numbers, dates and approval status are handled carefully, retrieval can surface outdated guidance. In fast-moving domains, such as regulation or healthcare, that can have real-world consequences.

These weaknesses explain why RAG should not be confused with fine-tuning. Fine-tuning adapts a model’s behaviour by training it on examples. It is well suited to tone, format and classification tasks. RAG, by contrast, supplies facts at query time. Expecting fine-tuning to “store” an organisation’s entire knowledge base is unrealistic. Expecting RAG to solve behavioural issues is equally misguided. Each tool addresses a different layer of the problem.

Security complicates matters further

RAG systems ingest external text and pass it directly to a model. That text is not inert. It can contain instructions as well as information. This creates an opening for prompt injection, a class of attacks in which malicious text tells the model to ignore previous rules, reveal internal data or perform unauthorised actions.

In a traditional software system, data and code are clearly separated. In a language-model system, the boundary is porous. If retrieved content is not clearly treated as data, the model may follow instructions embedded within it. A hostile document inside a knowledge base can become an attack vector.

Data leakage is a related concern. Retrieval systems must enforce access controls with absolute precision. If a user’s query can trigger retrieval from documents they should not be allowed to see, the model may summarise or paraphrase restricted material even if it never displays the raw text. Logging and monitoring can make the situation worse if prompts and retrieved content are stored without adequate protection.

Risks are not theoretical

Security researchers have repeatedly demonstrated systems that can be coerced into revealing internal prompts, training data or confidential documents. RAG increases the surface area for such attacks because it deliberately broadens what the model can “see”.

For this reason, some of the most important work around RAG happens far from the model itself. Mature teams treat retrieval as a security-sensitive component. They enforce permissions at query time, not just at the interface. They log which documents were retrieved and why. They test systems with deliberately malicious documents to see whether the model obeys them.

Evaluation is another area where the hype tends to fade. Many organisations assess RAG systems by asking whether the answers “look right”. That is not enough. Proper evaluation involves measuring retrieval quality explicitly. Did the correct document appear in the top results? How often does retrieval miss relevant material? How often does the model answer confidently when it should refuse?

Refusal behaviour is fundamental and often neglected

A well-designed RAG system should be able to say, clearly, that the answer is not present in the available documents. In practice, this is hard to achieve. Models are biased towards answering. Without careful prompting and testing, they will fill gaps with plausible text rather than admit uncertainty.

The need for such discipline undercuts one of the more seductive narratives around RAG, the idea that it is a simple add-on that turns a general model into a reliable expert. In reality, RAG pushes responsibility onto the system builder. The model vendor provides a component. The organisation assembles the machinery that determines whether the output is trustworthy.

This shift has broader implications. As artificial intelligence moves into areas such as customer service, healthcare triage and public administration, the line between assistance and authority blurs. A system that cites internal documents can appear official even when it is wrong. Errors can become institutional facts, repeated and reinforced by the very systems designed to prevent them.

There is also a cultural dimension

RAG encourages a document-centric view of knowledge. If something is not written down, it is invisible. If it is written down badly, the system will faithfully reproduce that badness. In this sense, RAG inherits all the biases, omissions and ambiguities of organisational paperwork.

None of this makes RAG useless. On the contrary, it is one of the most promising techniques for making AI systems more accountable. It allows users to see what a model is relying on. It enables rapid updates without retraining. It offers a path away from pure statistical guesswork.

But it does demand realism. RAG is not a truth machine. It is a retrieval and synthesis pipeline with failure modes that are easier to miss precisely because the outputs look polished.

The most responsible deployments acknowledge this openly. They expose citations clearly. They allow users to inspect sources. They invest in testing, monitoring and red-teaming. They treat silence, the ability to say “I do not know”, as a success condition rather than a defect.

In that sense, RAG is less a breakthrough than a mirror. It reflects the quality of the systems and processes built around it. Where documents are well maintained, permissions are clear and evaluation is rigorous, it can meaningfully reduce error. Where those foundations are weak, it can amplify confusion while projecting confidence.

Deeper lessons

The deeper lesson is that accuracy in artificial intelligence is not something you buy off the shelf. It is something you engineer, maintain and govern. RAG does not change that. It simply makes the responsibility harder to ignore.