The new CIO security priority: Your software supply chain

One reason open source is popular in the enterprise is that it provides well-tested building blocks that can speed up the creation of sophisticated applications and services. But third-party software components and the convenience of packages and containers bring risks along with the benefits because the applications you build are only as secure as those dependencies.

Software supply chain attacks are becoming so widespread that Gartner listed them as the second biggest threat on for 2022. By 2025, the research firm predicts 45% of organizations globally will have experienced one or more software supply chain attacks — and 82% of CIOs think they will be vulnerable to them. These include attacks via vulnerabilities in widely used software components such as Log4j, attacks against the build pipeline (c.f., SolarWinds, Kaseya, and Codecov hacks), or hackers compromising package repositories themselves.

“Attackers have shifted priority from production environments to software supply chains because software supply chains are the weakest link,” explains Lior Levy, CEO of Cycode. “As long as software supply chains remain relatively easy targets, software supply chain attacks will increase.”

Recent high-profile incidents have been a wake-up call for the software development industry, says Rani Osnat, senior vice president of strategy at Aqua Security. “We’ve uncovered decades of opacity and lack of transparency and that’s why it’s such a big deal.”

Studies of codebases that use open source code shows that vulnerabilities and out-of-date or abandoned components are common: 81% of codebases had at least one vulnerability, 50% had more than one high-risk vulnerability, and 88% used components that weren’t the latest version or had no new development in two years.

These issues are unlikely to dent the popularity of open source though — and commercial software and services are also vulnerable. When LastPass was attacked it didn’t lose customer data, but an unauthorized party was able to view and download some of its source code, which might make it easier to attack users of the password manager in the future, and the Twilio breach enabled attackers to launch supply-chain attacks on downstream organizations.

The ‘shadow code’ threat

Just as security teams defend their networks as if already breached, CIOs must assume all code, internal or external, and even the development environments and tools their developers use have already been compromised and put policies in place to protect against and minimize the impact of attacks against their software supply chains.

In fact, Osnat suggests CIOs think about this “shadow code” the way they do about shadow IT. “This needs to be looked at as something that is not just a security problem, but really something that goes deep into how you obtain software, whether it’s open source or commercial: how you bring it into your environment, how you update it, what kind of controls you want to have in place and what kind of controls you want to demand from your suppliers,” he says.

Transparency: Toward a software bill of materials

Physical supply chains already use labels, ingredient lists, safety data sheets, and bills of materials so regulators and consumers know what ends up in products. New initiatives aim to apply similar approaches to software, helping organizations understand the web of dependencies and the attack surface of their software development process.

White House executive order 14028 on software supply chain security requires software vendors supplying the federal government to provide a software bill of materials (SBOM) and use the supply chain levels for software artifacts (SLSA) security checklist to prevent tampering. Because of this, “we’re seeing a lot of enterprises take a much more serious look at their software supply chain,” says senior Forrester analyst Janet Worthington. “All companies today both produce and consume software and we’re seeing more of the producers come to us and say, ‘How do I produce software that is secure and that I can attest to with a software bill of materials.’”

There are numerous cross-industry projects, including NIST’s National Initiative for Improving Cybersecurity in Supply Chains (NIICS), the Supply Chain Integrity, Transparency, and Trust (SCITT) initiative from Microsoft and other IETF members, as well as the OpenSSF Supply Chain Integrity Working Group.

“Everybody is taking a more holistic approach and saying, wait a minute, I need to know what I’m bringing into my supply chain that I’m creating the software with,” Worthington says.

A recent Linux Foundation survey found that SBOM awareness is high, with 47% of IT vendors, service providers, and regulated industries using SBOMs today and 88% expecting to use them in 2023.

SBOMs will be most useful to organizations that already have asset management for software components and APIs. “People who have robust software development processes today find it easier to slot in tools that can generate a software bill of materials,” Worthington says.

SBOMs can be created by the build system, or they can be generated by software composition analysis tools after the fact. Many tools can integrate into CI/CD pipelines and run as part of a build, or even when you pull down libraries, she says. “It can warn you: ‘Hey, you have this component in your pipeline and it’s got a critical issue, do you want to continue?’”

For that to be useful, you need clear policies on how developer teams acquire open-source software, says Chainguard CEO Dan Lorenc. “How do developers know what their company’s policies are for what’s considered ‘secure’ and how do they know that the open source they are acquiring, which constitutes the great majority of all software being used by developers these days, is indeed untampered with?”

He points at the open-source Sigstore project that JavaScript, Java, Kubernetes, and Python use to establish provenance for software packages. “Sigstore is to software integrity sort of what certs are to websites; they basically establish a chain of custody and trust verification system,” he says.

“I think a CIO should start by indoctrinating their developer teams in these fundamental steps of using emerging industry standard approaches for one, locking down build systems, and two, creating a repeatable method to verify trustworthiness of software artifacts before bringing them into the environment,” Lorenc says.

Making the contribution

Whether it’s components, APIs, or serverless functions, most organizations underestimate what they’re using by an order of magnitude unless they run routine inventories, Worthington points out. “They find out that some of these APIs aren’t using proper authentication methods or are maybe not written in a way that they expected them to be and maybe some of them are even deprecated,” she says.

Beyond vulnerabilities, evaluating the community support behind a package is as important as understanding what the code does because not all maintainers want the burden of having their code treated as a critical resource. “Not all open source is made the same,” she warns.

“Open source may be free to download but certainly the use of it is not free. Your use of it means that you as are responsible for understanding the security posture behind it, because it’s in your supply chain. You need to contribute back to it. Your developers need to participate in fixing vulnerabilities,” says Worthington, who suggests organizations should also be prepared to contribute monetarily, either directly to open-source projects or to initiatives that support them with resources and funds. “When you create an open-source strategy, part of that is understanding the budget and implications.”

Don’t think of that as just an expense, but as an opportunity to better understand the components you depend on. “It even helps retain developers because they feel like they’re part of the community. They’re being able to contribute their skills. They can use this on their resume,” she adds.

Remember that vulnerabilities can be found anywhere in your technology stack, including mainframes, which increasingly run Linux and open source as part of the workload but often lack the security processes and frameworks that have become common in other environments.

Protecting your pipeline

Protecting your software delivery pipeline is also important. NIST’s Secure Software Development Framework (SSDF) and SLSA is a good place to start: This covers best practices at various maturity levels starting with a simple build system, then using logs and metadata for audit and incident response through to a fully-secured build pipeline. The CNCF’s Software Supply Chain Best Practices white paper, Gartner’s guidance on mitigating software supply chain security risks, and Microsoft’s OSS Secure Supply Chain Framework, which includes both processes and tools, are also helpful.

It’s important to note, however, that simply turning on automated scanning tools intended to find malicious code can produce too many false positives to be helpful. And although version control systems such as BitBucket, GitHub, GitLab, and others include security and access protection features (including increasingly granular access policy controls, branch protection, code signing, requiring MFA for all contributors, and scanning for secrets and credentials), they often have to be explicitly enabled.

Also, projects such as Factory for Repeatable Secure Creation of Artifacts (FRSCA) that aim to secure build pipelines by implementing SLSA in a single stack aren’t yet ready for production, but CIOs should expect build systems to include more of these practices in future.

Indeed, while SBOMs are only part of the answer, the tools to create and work with them are also still maturing, as are the processes for requesting and consuming them. Contracts need to specify not only that you want SBOMs but how often you expect them to be updated and whether they will include vulnerability reports and notifications, Worthington advises. “If a new important vulnerability like Log4j is found, is the vendor going to tell me or am I going to have to search myself in the SBOM to see if I’m affected?”

Organizations will also need tools to read SBOMs and put in place processes to take actions on what these tools find. “I need a tool that can tell me what are the known vulnerabilities [in the SBOM], what are the licence implications, and does that happen continuously,” Worthington says.

CIOs should keep in mind that an SBOM “is an enabler but it doesn’t actually solve anything in terms of securing your supply chain. It helps you cope with incidents that might come your way,” says Osnat, who is optimistic about both the speed of industry response and the broad collaboration that’s going on around standards for SBOMs  and code attestation that will help make tools interoperable (something organizations raised as a particular concern in the Linux Foundation research). That could lead to the same improvements in the standards of transparency and reporting across the industry that SOC 2 delivered.

That said, CIOs don’t have to wait for new standards or tools to begin making security as much a part of the developer role as quality has become in recent years, Osnat says. His suggestion: “Start by getting your CISO and lead engineer in a room together to figure out what the right model is to make that work for your organization and how that transformation will occur.”