RAG can make AI models riskier and less reliable, new research shows

Retrieval-Augmented Generation (RAG) is rapidly emerging as a robust framework for organizations seeking to harness the full power of generative AI with their business data. As enterprises seek to move beyond generic AI responses and leverage their unique knowledge bases, RAG bridges general AI capabilities and domain-specific expertise. 

Hundreds, perhaps thousands, of companies are already using RAG AI services, with adoption accelerating as the technology matures.

Also: I tested 10 AI content detectors, and these 5 correctly identified AI text every time

That’s the good news. The bad news: According to Bloomberg Research, RAG can also vastly increase the chances of getting dangerous answers. 

Before diving into the dangers, let’s review what RAG is and its benefits. 

What is RAG?

RAG is an AI architecture that combines the strengths of generative AI models — such as OpenAI’s GPT-4, Meta’s LLaMA 3, or Google’s Gemma — with information from your company’s records. RAG enables large language models (LLMs) to access and reason over external knowledge stored in databases, documents, and live in-house data streams, rather than relying solely on the LLMs’ pre-trained “world knowledge.”

When a user submits a query, a RAG system first retrieves the most relevant information from a curated knowledge base. It then feeds this information, along with the original query, into the LLM. 

Maxime Vermeir, senior director of AI strategy at ABBYY, describes RAG as a system that enables you to “generate responses not just from its training data, but also from the specific, up-to-date knowledge you provide. This results in answers that are more accurate, relevant, and tailored to your business context.”

Why use RAG?

The advantages of using RAG are clear. While LLMs are powerful, they lack the information specific to your business’s products, services, and plans. For example, if your company operates in a niche industry, your internal documents and proprietary knowledge are far more valuable for answers than what can be found in public datasets.

By letting the LLM access your actual business data — be these PDFs, Word documents, or Frequently Asked Questions (FAQ) — at query time, you get much more accurate and on-point answers to your questions. 

In addition, RAG reduces hallucinations. It does this by grounding AI answers to reliable, external, or internal data sources. When a user submits a query, the RAG system retrieves relevant information from curated databases or documents. It provides this factual context to the language model, which then generates a response based on both its training and the retrieved evidence. This process makes it less likely for the AI to fabricate information, as its answers can be traced back to your own in-house sources. 

Also: 60% of AI agents work in IT departments – here’s what they do every day

As Pablo Arredondo, a Thomson Reuters vice president, told WIRED, “Rather than just answering based on the memories encoded during the initial training of the model, you utilize the search engine to pull in real documents — whether it’s case law, articles, or whatever you want — and then anchor the response of the model to those documents.”

RAG-empowered AI engines can still create hallucinations, but it’s less likely to happen. 

Another RAG advantage is that it enables you to extract useful information from your years of unorganized data sources that would otherwise be difficult to access. 

Previous RAG problems

While RAG offers significant advantages, it is not a magic bullet. If your data is, uhm, bad, the phrase “garbage-in, garbage out” comes to mind.  

A related problem: If you have out-of-date data in your files, RAG will pull this information out and treat it as the gospel truth. That will quickly lead to all kinds of headaches. 

Also: Want generative AI LLMs integrated with your business data? You need RAG

Finally, AI isn’t smart enough to clean up all your data for you. You’ll need to organize your files, manage RAG’s vector databases, and integrate them with your LLMs before a RAG-enabled LLM will be productive. 

The newly discovered dangers of RAG

Here’s what Bloomberg’s researchers discovered: RAG can actually make models less “safe” and their outputs less reliable. 

Bloomberg tested 11 leading LLMs, including GPT-4o, Claude-3.5-Sonnet, and Llama-3-8 B, using over 5,000 harmful prompts. Models that rejected unsafe queries in standard (non-RAG) settings generated problematic responses when the LLMs were RAG-enabled.

They found that even “safe” models exhibited a 15–30% increase in unsafe outputs with RAG. Moreover, longer retrieved documents correlated with higher risk, as LLMs struggled to prioritize safety. In particular, Bloomberg reported that even very safe models, “which refused to answer nearly all harmful queries in the non-RAG setting, become more vulnerable in the RAG setting.”

Also: Why neglecting AI ethics is such risky business – and how to do AI right

What kind of “problematic” results? Bloomberg, as you’d expect, was examining financial results. They saw the AI leaking sensitive client data, creating misleading market analyses, and producing biased investment advice. 

Besides that, the RAG-enabled models were more likely to produce dangerous answers that could be used with malware and political campaigning. 

In short, as Amanda Stent, Bloomberg’s head of AI strategy & research in the office of the CTO, explained, “This counterintuitive finding has far-reaching implications given how ubiquitously RAG is used in gen AI applications such as customer support agents and question-answering systems. The average internet user interacts with RAG-based systems daily. AI practitioners need to be thoughtful about how to use RAG responsibly, and what guardrails are in place to ensure outputs are appropriate.”

Sebastian Gehrmann, Bloomberg’s head of responsible AI, added, “RAG’s inherent design-pulling of external data dynamically creates unpredictable attack surfaces. Mitigation requires layered safeguards, not just relying on model providers’ claims.”

What can you do?

Bloomberg suggests creating new classification systems for domain-specific hazards. Companies deploying RAG should also improve their guardrails by combining business logic checks, fact-validation layers, and red-team testing. For the financial sector, Bloomberg advises examining and testing your RAG AIs for potential confidential disclosure, counterfactual narrative, impartiality issues, and financial services misconduct problems. 

Also: A few secretive AI companies could crush free society, researchers warn

You must take these issues seriously. As regulators in the US and EU intensify scrutiny of AI in finance, RAG, while powerful, demands rigorous, domain-specific safety protocols. Last, but not least, I can easily see companies being sued if their AI systems provide clients with not merely poor, but downright wrong answers and advice. 

Want more stories about AI? Sign up for Innovation, our weekly newsletter.