Malicious AI Models on Hugging Face Exploit Novel Attack Technique

Researchers at Reversing Labs have discovered two malicious machine learning (ML) models available on Hugging Face, the leading hub for sharing AI models and applications.

While these models contain malicious code, they were not flagged as “unsafe” by Hugging Face’s security scanning mechanisms.

The Reversing Labs researchers saw that these malicious models exploit a novel malware distribution technique by abusing Pickle file serialization.

Background on Pickle

ML models use algorithms to learn patterns and make predictions based on data.

These models are stored in various formats to be shared and reused without retraining.

One common format is Pickle, a Python module that serializes and deserializes ML model data – an action known as Pickling by the Hugging Face community.

However, Pickle is unsafe because it allows Python code to run during deserialization.

This risk means Pickle files should only be used with trusted sources. However, on open ML platforms like Hugging Face, where collaboration is key, Pickle files are widely used.

Many ML developers prioritize ease of use and productivity over security, leading to a growing problem of malicious Pickling.

The nullifAI Technique

While malicious Pickling is not a new method, the way these two models managed to bypass Hugging Face’s malicious Pickle files detection tool, Picklescan, is new.

The two malicious models the Reversing Labs researchers detected are stored in PyTorch format, a compressed Pickle file. However, they are compressed using the 7z format instead of the ZIP format PyTorch traditionally uses. This means they cannot be loaded using PyTorch’s default function, torch.load().

The Reversing Lab researchers believe this trick is likely the reason why Picklescan did not flag these models as unsafe.

However, they added that the two malicious models used broken Pickle files, suggesting they were proof-of-concept models for testing a novel attack method rather than actual malicious ones.

Shortcomings in Hugging Face’s Malicious File Detection

The findings of the Reversing Labs report, published on February 6, revealed new shortcomings in how Hugging Face’s Picklescan detects malicious models.

Recently, Checkmarx criticized Picklescan’s reliance on a blacklist of “dangerous” functions to detect malicious models, which are “basic security features, but not scalable or adaptable as known threats morph – and new threats emerge,” said the Reversing Labs report.

Additionally, the Reversing Labs researchers now showed an inability for Picklescan to properly scan broken Pickle files.

The researchers explained that Pickle file deserialization works in a different way from Pickle security scanning tools.

“Picklescan, for example, first validates Pickle files and, if they are validated, performs security scanning. Pickle deserialization, however, works like an interpreter, interpreting opcodes as they are read — but without first conducting a comprehensive scan to determine if the file is valid, or whether it is corrupted at some later point in the stream,” Reversing Labs said in a report.

This allows attackers to integrate malicious code in broken Pickle files and avoid having their models flagged as “dangerous.”

“The explanation for this behavior is that the object deserialization is performed on Pickle files sequentially. Pickle opcodes are executed as they are encountered and until all opcodes are executed or a broken instruction is encountered. In the case of the discovered model, since the malicious payload is inserted at the beginning of the Pickle stream, execution of the model wouldn’t be detected as unsafe by Hugging Face’s existing security scanning tools,” Reversing Labs concluded.

Upon discovering the malicious Pickle files, Reversing Labs informed the Hugging Face security team about the vulnerability.

Hugging Face removed the malicious models within 24 hours. Additionally, they updated the Picklescan tool to detect threats in “broken” Pickle files.

Photo credit: Tada Images/Robert Way/Shutterstock