The Unsolvable Problem: XZ and Modern Infrastructure

The ongoing prevalence (and rise) of software supply chain attacks is enough to keep any software developer or security analyst up at night. The recent XZ backdoor attack is finally behind us, and luckily there was no widespread reach of the backdoored library. If you hadn’t heard, this software supply chain attack was a malicious effort that targeted Linux systems, and this attack had been years in the making.

There’s no denying that an event like XZ will happen again, and we may not be so lucky next time. But what hasn’t been discussed is how what happened with XZ isn’t a problem we can solve with best practices today. So, if we can’t solve this problem of backdoor supply chain attacks, how do we chart a safe route forward?

The Unsolvable Problem

Sometimes reality can be harsh, but the painful truth about this sort of backdoor attack is that there is no solution, we simply don’t know how to solve this one. Many projects and organizations are happy to explain how they keep you safe, or how you can prevent software supply chain attacks, by doing this one simple thing. However, the industry as it stands today lacks the ability to prevent an attack created by a motivated and resourced threat actor. In fact, the Anchore 2022 Software Supply Chain Security Report shows that the security of open source software containers is ranked as the number one challenge by 24% of respondents, so this is not an isolated business concern. The same survey also reports that more than half of respondents say that securing the software supply chain is a top or significant focus. This indicates that recent, high-profile attacks like the XZ attack have put software supply chain security on the radar for the majority of organizations.

If there is a malicious open source maintainer, we (as an industry) lack the tools and knowledge to prevent this sort of attack, as you can’t actually stop such behavior until after it happens. When we use open source software, there is so much of it, we can’t possibly vet it. We rely on the community to help find and fix problems, which is exactly what happened with the XZ backdoor attack.

HOWEVER, that doesn’t mean we are helpless. We can take a page out of the playbook of the observability industry. Sometimes we’re able to see problems as they happen or after they happen, then use that knowledge from the past to improve the future, that is a problem we can solve. And it’s a solution that we can measure. If you have a solid inventory of your software, past, present, and future, then looking for affected versions of XZ becomes simple and effective.

Today and Tomorrow

Looking for a vulnerable version of XZ, specifically versions 5.6.0 and 5.6.1, sounds like it should be an easy task, but trying to solve a problem like this at scale is always a challenge. We don’t know what we will need to quickly search for in the future. Will it be a binary file, a python package, or maybe just a checksum. We don’t know what the next attack will be, an accurate inventory will be important.

The industry is currently putting a focus on using a software bill of materials, or SBOM, as the way to track the contents of software. We see a focus on these inventories in new development standards such as the secure software development framework, or SSDF. By using an SBOM to track software inventory, we have a standardized way to not only track our own software, but to also share those inventories with our customers and partners, and to receive an SBOM from our suppliers. SBOMs aren’t perfect, but they are the first step to having software inventories we can use in the future.

What Now?

Anyone who has been following industry news is probably wondering what supply chain story will happen next. The size and complexity of open source software is enormous and growing more complex every day. Open source is so embedded in our products and services now there’s no way we can stop using it, it’s here to stay, so what responsibilities do we have? If it’s too big to fail, and too big to fix, we have to figure out how we can use open source in ways that make sense. We have technologies now to help keep track of your open source software components, but just keeping track is the first step. It’s just as important to move quickly when the next XZ shows up. If we’re going to use open source, we have to move at the speed of open source. We can’t solve the problem that brought us to XZ, but we can make sure when the next one happens, we can start responding in minutes instead of days.

About the Author

Josh Bressers is the Vice President of Security at Anchore, a modern software composition analysis company that automates software compliance to save time and reduce risk. At Anchore he guides security feature development for the company’s commercial and open source solutions. He is a co-lead of the OpenSSF SBOM Everywhere project, and is a co-founder of the Global Security Database project at the Cloud Security Alliance.

Bressers can be reached on LinkedIn or by visiting https://www.anchore.com/.