Break glass in case of emergency: How a SASE approach can solve remote network outages

As much as we’d love to write an article on how Cisco IT has perfected a network that never breaks, we’re not there yet. It’s a fact of life that networks and systems do — and will continue to — break. In fact, my home ISP went down as I was writing this piece.

You may be thinking, ‘networks breaking is nothing new’ — and that is true. However, the new complexities introduced by hybrid work means networks now break in exciting new ways. With the rise of hybrid work, Cisco IT has run into a unique circular dependency issue, which happens when a network outage prevents network operators from accessing the network they need to repair and troubleshoot. Specifically, if our VPN goes down, how do we access the network to troubleshoot and triage the VPN issue? How do we get into our out-of-band network in this situation?

While rare, these kinds of outages have a big impact on the business. The time to repair increases significantly when our network operators cannot remotely access an out-of-band network and must physically travel to the site.

To avoid being locked out of our own network, we started studying the newest tools and solutions, considering their ability to create a secure, accessible network. Our team landed on a Secure Access Service Edge (SASE) approach to avoid a network operator being locked out of the out-of-band network when the in-band network is down.

The solution

Our network engineers named the solution ‘Break Glass,’ as in: Break Glass in Case of Emergency. We have protections in place for security measures, but we can break them in an urgent situation.

For the solution, we leveraged Cisco+ Secure Connect alongside Azure Active Directory. Cisco+ Secure Connect is Cisco’s simple and easy SASE offering, which provides VPN-as-a-Service – ultimately allowing for secure connectivity and private access to our out-of-band network. Azure Active Directory allowed us to decouple out-of-band access from the on-prem network and its dependencies. With a cloud-based VPN, it is now completely decoupled from our network and its dependencies, and our network operators can get into the out-of-band network when the in-band network is down.

Joseph Bradley was the lead engineer on the project and describes the reasons we selected a SASE solution.

“We were tasked with creating a secure way to access the Cisco network and repair issues while off premises — and we had a list of constraints. Specifically, we had to administer the solution with a small team, and the solution had to be completely decoupled from our AD and IAM systems during an event. Finally, we had to provide access to only a certain set of Hosts which would be used as jump hosts, and the internal solution had to be available in L3 connectivity/routing. Using the constraints above, our internal remote access team decided the best solution would be to use a combination of products that already exist, yet in a novel way. We partnered with our information security team, Cisco Umbrella, and Azure teams to create this unique solution, then improved and added features.”

To meet these constraints, the solution was set up using accounts that are only allowed to connect to the Cisco+ Secure Connect service and the out-of-band network. It leverages a site-to-site VPN tunnel from the Cisco+ Secure Connect service to our network team’s out-of-band infrastructure, and provides access to our jump host infrastructure that allows our network engineers to access out-of-band ports on our network devices.

A thought-starter for further applications of SASE

Project ‘Break Glass’ has sparked curiosity of how we can use SASE to solve additional challenges. One application of SASE that the Cisco IT team is considering is acquisition integration. Cisco has many tenants that need access to their own cloud resources. Historically, we built bespoke network or security stacks on-premises. However, with SASE, we could facilitate this connectivity securely, without building on-prem network stacks.

Another application of SASE is for secure internet access for Cloud Desktop (DaaS). A SASE-based internet edge for cloud virtual machines would allow us to avoid backhauling to the enterprise network. For the user, this increases performance with no compromise in security, as the user’s machine has direct access to the cloud. IT teams no longer have to deal with managed hardware devices — the completely control the virtual machine and can apply updates or security patches no matter where the user is, with no dependency on hardware.

Stay tuned as we continue experimenting with applications of SASE and Cisco+ Secure Connect in future blogs. And, we’re curious how you’re using SASE within your enterprise networks. Let us know on the @CiscoIT Twitter, or in the comments.

Inside Cisco IT Blog

Share: