Lesson Learned: How SolarWinds Strengthened its Security Post-Incident

While the past two years have been busy for most security professionals, very few can match the level of tension Tim Brown has gone through. As CISO of the software firm SolarWinds since 2017, he was in charge of security there when several US federal government services suffered from severe data breaches in 2020, following an attack in which Russian-backed threat actors exploited different software vulnerabilities, including in Orion, SolarWinds’ IT monitoring system.

Brown shared his experience of running an incident response and remediation plan following the high-profile security incident during one of Mandiant Worldwide Information Security Exchange‘s (mWISE) opening keynotes on October 18, 2022.

“First off, you have to build a real hard shell – nobody in the world says anything nice about you for four months, at least,” Brown stated.

The first action the SolarWinds’ security team took after the attack was to get help from the legal firm DLA Piper, who advised them in 2018 during the software company’s initial public offering (IPO).

“We were very direct in our disclosure and shared as much as possible, especially with our customers, who were our first focus. However, with so much fake news going around, we had to ignore the press for a little bit,” Brown admitted.

Implementing a New Secure-By-Design Program

Although he didn’t share elements of the investigation and forensics, the CISO revealed that he told his team of around 400 engineers not to build any products for the first six months, and instead focus exclusively on securing the existing ones.

This was done by introducing a new secure-by-design program. “In our instance, the source code control system was not changed but the end result was changed. The attackers broke through a virtual machine, which meant that the first step of this new program was making sure the source code matched what we produced: we get a product, decompile it and then check the source code – and repeat for all of our 50 products,” he recalled.

Then, SolarWinds engineers had to create a new build system, an automated process of compiling computer source code into binary code, external to their own environment and ephemeral, as well as a new repository for all the products.

“Then, we had to establish a staging pipeline and a production pipeline, with fewer people granted access within each, to the build system. We open-sourced all of this,” Brown added.

In the beginning, motivation from the engineers was “easy to get,” Brown said. “Someone broke into their house and changed their code, so they were mad. But after six months, it started waning a little bit, and we started shifting to working on new features again.”

Overall, Brown said this process “worked pretty well for us: we had about 93% renewal rate prior to the incident, then it went down to around 80% post-incident, and it came back up over 90% now. We did all the remediation necessary, and our inspection partners and threat hunt partners have been checking everything for two years. We are now the safest bet in town.”

Establishing a Security Committee Within the Board

The cyber-attack also enticed the CISO to strengthen both his company’s defensive and offensive capabilities.  

“Before my incident, I ran my own security operation center (SOC); now I have three: a CrowdStrike SOC, a SecureWorks SOC and my own, as well as access to forensic technology services from KPMG. We also went from a part-time red team to a full-time one,” Brown said.

Another major change at SolarWinds was the creation of a technology and cybersecurity committee on its board of directors – “something that is not common,” noted Charles Carmakal, consulting CTO at Mandiant, who was hosting the mWISE keynote.

“Usually, cyber skillsets are either not represented or merely secondary within boards, but we thought it was important to establish a separate cybersecurity committee. We meet regularly – our meetings are scheduled quarterly, but they usually end up being more frequent than that. In those meetings, we brief the board members on what risks we face as a company. It helps the board support our initiatives and additional investment into security,” Brown shared.

Finally, when asked to give the last word of the keynote, the CISO, now also VP of security at SolarWinds, offered words of hope. “Be prepared for long days and long nights, but you will get through it, and you’ll be better for it,” he concluded.