What AI pioneer Yoshua Bengio is doing next to make AI safer

As AI companies double down on AI agents as the future of work, one founding thinker of the technology is moving in reverse, advocating for simpler systems. 

On Tuesday, award-winning deep learning scientist Yoshua Bengio launched LawZero, an AI nonprofit “committed to advancing research and developing technical solutions for safe-by-design AI systems,” according to a press release. One of the nonprofit’s first objectives will be to create Scientist AI, a “non-agentic AI system” intended to function as a guardrail to keep other AI systems in check. Other AI labs have explored similar concepts to monitor autonomous agents and reduce hallucinations. 

Also: Tech prophet Mary Meeker just dropped a massive report on AI trends – here’s your TL;DR

“This system is designed to explain the world from observations, as opposed to taking actions in it to imitate or please humans,” a paper on Scientist AIs said. Generating theories about world data, the system is designed to operate with uncertainty to protect against overconfidence — an issue that often plagues chatbots, for example. 

Bengio and his co-authors advise that a system with this design could help humans advance scientific breakthroughs, including AI safety efforts. 

“Ultimately, focusing on non-agentic AI may enable the benefits of AI innovation while avoiding the risks associated with the current trajectory,” the paper said. “We hope these arguments will motivate researchers, developers, and policymakers to favor this safer path.”

Also: How much energy does AI really use? The answer is surprising – and a little complicated

Currently a professor at the University of Montreal, Bengio is known for his work in deep learning, a foundational component of generative AI, for which he earned an A.M. Turing Award in 2018. Regarded as one of the godfathers of the technology and one of the most regularly cited experts in the area, he has also long expressed concern over how AI’s abilities will impact society if left unchecked. 

His fears appear to be coming true, especially recently. “Today’s frontier AI models are developing dangerous capabilities and behaviours, including deception, self-preservation, and goal misalignment,” the LawZero release noted, referencing recent studies and red-teaming results from several AI firms. In April, OpenAI recalled a model update for being too sycophantic, a trait that can have dangerous consequences if users abuse it; that same month, Anthropic found Claude was being successfully misused to generate malware and launch disinformation campaigns. Models from Chinese AI startup DeepSeek have proven easy to jailbreak.

(Disclosure: Ziff Davis, ZDNET’s parent company, filed an April 2025 lawsuit against OpenAI, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.)

Anthropic, known for its focus on safety compared to other major AI companies, recently upped security measures for its Claude 4 Opus model “due to continued improvements in CBRN-related knowledge and capabilities,“ referring to chemical, biological, radiological, and nuclear (CBRN) weapons. The model’s predecessor, Claude 3 Opus, demonstrated an ability to fake its values and intentions in order to undermine commands from its creators, a process known as alignment faking.

Also: You can use OpenAI’s super powerful AI coding agent Codex for just $20 now

“Currently, AI is developed to maximize profit,” Bengio said in a video accompanying the announcement, a direction LawZero hopes to reverse, or at least diversify away from. According to the release, LawZero will focus on designing AI for safety over commercial applications — a distinct shift from major AI initiatives, which are geared increasingly towards military applications, agentic AI for enterprises, and consumer AI tools. 

Want more stories about AI? Sign up for Innovation, our weekly newsletter.

LawZero said it hopes its nonprofit status will keep it “insulated from market and government pressures, which risk compromising AI safety.” However, OpenAI was famously started as a nonprofit, too, and is technically still run by one. Just last month, OpenAI settled an ongoing debate over its future by becoming a Public Benefit Corporation (PBC) under its nonprofit arm, OpenAI Inc. At the same time, the lab is reportedly shrinking its testing timelines due to market pressures while asking that the US government loosen potential regulation in exchange for early access to new models. 

Also: OpenAI wants ChatGPT to be your ‘super assistant’ – what that means

Still, LawZero isn’t alone in positioning itself toward the greater good; in the PBC announcement, OpenAI CEO Sam Altman emphasized that OpenAI would focus on healthcare and scientific initiatives, launching OpenAI for Countries, a global democratic AI program, the same day.

As for artificial general intelligence (AGI), Bengio isn’t convinced the tech industry should be racing toward it. As he told Axios: “If we continue on this path, that means we’re going to be creating entities — like us — that don’t want to die, and that may be smarter than us and that we’re not sure if they’re going to behave according to our norms and our instructions.” 

Also: Microsoft is an AGI skeptic, but is there tension with OpenAI?

His comments refute the primary tone of leadership at dominant AI companies like OpenAI and Meta, which have been bullish on AGI and its impending realization. It’s unclear how fighting for AI that does less than what investors and AI evangelists are currently envisioning will impact industry priorities. With the Trump administration’s AI Action Plan still in development before its July 19 due date, it’s similarly uncertain whether public policy will reflect Bengio’s concerns. 

Get the morning’s top stories in your inbox each day with our Tech Today newsletter.