Nvidia's 70+ projects at ICLR show how raw chip power is central to AI's acceleration

One of the most important annual events in the field of artificial intelligence kicks off this week in Singapore: the International Conference on Learning Representations. As usual, chip giant Nvidia had a major presence at the conference, presenting over 70 research papers from its team.

Also: Nvidia launches NeMo software tools to help enterprises build custom AI agents

The papers cover topics ranging from generating music to creating 3D-realistic videos, robot training tasks, and the ability to generate multiple large language models at the push of a button.

Not just a chip company

“People often think of Nvidia as a chip company that makes awesome chips, and of course, we’re really proud of that,” said Bryan Catanzaro, Nvidia’s head of applied deep learning research, in an interview with ZDNET. “But the story that I think matters the most is that in order for us to make those awesome chips, we have to do research like this, because this teaches us how to make all of those systems.”

Also: Nvidia dominates in gen AI benchmarks, clobbering 2 rival AI chips

The papers presented this week, most of which were published over the past year or so on the arXiv preprint server, range from pure research to programs that offer immediately usable tools.

In the former category, for example, a project called LLaMaFlex improves the task of generating many large language models from a single parent. It is commonplace today to “distill” a single, very large LLM into “student” LLMs that inherit the capability of the “teacher” but take up less memory storage.

Nvidia researchers Ruisi Cai and his team observed that the method of distillation could be improved by using what they call “elastic pretraining.” Taking a single, large pre-trained LLM — in this case, Meta Platforms’s Llama 3.18B — they put it through a single additional training phase with 60 billion new training tokens. The result is an algorithm called a “router” that can automatically output any number of differently sized offspring LLMs at virtually the push of a button.

Fugatto foundation model

In the category of more tangible programs, the Fugatto 1 program is a “foundation model” for audio synthesis, an AI model that can handle any combination of text instructions and sound clips and transform the clip based on the instructions. “I’m really excited about Fugatto,” Catanzaro told ZDNET.

Also: I’ve tried lots of AI image generators, and Nvidia and MIT’s is the one to beat for speed

For example, Fugatto can produce a sound upon request, such as a cat’s meow. It can pick apart a song sample to reproduce each separate vocalist. It can merge the sound of rippling water with the sound of a classical guitar to create a hybrid sound that is an admixture of the two.

The neural net of Fugatto is one developed at Google in 2022 that can operate on “spectrograms,” sounds as wave patterns. The original contribution of Nvidia’s Rafael Valle and his team is a new dataset and a training regimen that teaches the model to handle complex textual commands.

Also: The rise of AI PCs: How businesses are reshaping their tech to keep up

Nvidia projects such as Fugatto build upon many prior innovations, as does any research lab. One of the important aspects that sets apart Nvidia’s research papers is that they tend to offer more technical details about the hardware implementations used in the research, such as, for example, the number of GPU chips used, whereas other labs often leave that data out.

AI informing chip development

Research projects like LLaMaFlex and Fugatto serve many functions. They highlight the many ways Nvidia’s chips can be used, which is always a great way to promote the capabilities of those parts. They also keep Nvidia involved in the state of the art for AI, which can inform the company’s chip development. They help the company attract talent by showcasing projects that can win awards and peer recognition.

Also: Google’s latest chip is all about reducing one huge hidden cost in AI

And they show off how the raw power of Nvidia chips plays a large part in AI as a field. The “acceleration” of AI is a story that isn’t known as much as it should be, said Catanzaro.

“It’s my belief that a lot of the progress in AI over the past 15 years has actually come from acceleration,” said Catanzaro.

Check out all the Nvidia research publications on the main research website.

Get the morning’s top stories in your inbox each day with our Tech Today newsletter.