How IOS XE Developers at Cisco Work Remotely and Cohesively on a 190-million-line Code Base – Cisco Blogs

The option to work remotely predated the pandemic at Cisco. I’ve worked at least partially as a remote worker most of my 20+ years as an IOS developer. As a major vendor of VPN, teleconferencing, and messaging solutions, we already had the infrastructure, expertise, and experience to work remotely. When the company officially closed its offices in March of last year to 75,900 employees, IOS XE  developers were well prepared.

The development environment within Enterprise Networking operates at tremendous scale while maintaining an extremely high bar for quality. How? It’s due in no small part to two important factors: the ability to scale IOS XE development through our unique tooling and the inherent ability to run our code modularly and independently.

Remote Working: Benefits and Pitfalls

We’ve seen a lot of technologies we’ve taken for granted at Cisco being used more often and by more people during the pandemic. For example, volumes on the Webex platform tripled in April.

I’m based in Sydney, Australia, and have never felt any less effective than if I was a developer living closer to our main offices in San Jose, Bangalore, or Ottawa. Our extensive use of video, WebEx, IM and other collaboration technologies keep us very connected.

Nothing in my workday had changed when COVID hit and we were told to go home,  The code and people and tools were all in remote locations. I occasionally went into the Cisco office in Sydney but not having to go saves me an hour a day of commute time.  Yes, there are those hankering to get back to the office (and I do miss the yearly trip to San Jose). But we’ve been working very efficiently during the shelter at home period.  Great team cohesion is one of the reasons that the transition to remote working has been less disruptive to our productivity and morale.

The main challenge I’ve seen during the office closures is how to bind the teams together. Letting everybody know what others are doing is one strategy. I send out a weekly email newsletter to let developers know what teams and individuals are working on. Similarly, Chuck Robbins and other Cisco leaders started holding weekly 75-minute check-ins to listen to employees and address questions. Surveys of employees about their workplace experience went out to make sure everyone was okay.

A major benefit of a remote workforce is the ability to hire the best software engineers with the best skills. We have an incredible team in China, for example. It shouldn’t matter where a good person is located.

Remote Working: More Lessons Learned

• Make sure that your equipment works. If sound is a problem, invest in headphones, professional microphones, or whatever works. If people cannot understand what you are saying due to garbled audio, no one is going to listen to you.
• Be disciplined with your time. There is no boss or colleague looking over your shoulder. That means putting in the effort required and not putting in so much that it consumes you. Managers have a responsibility to set the appropriate expectations for employees; to manage what is expected of them while balancing their work/life balance.
• Beware of hazy thinking. The isolation of remote work can result in engineers looking at a problem the wrong way or with flawed logic. One computer science department I heard of had a stuffed bear called Teddy. Teddy was their best tech support guy. Before you could talk to a tutor, you had to describe your problem verbally to Teddy. The act of verbalizing issues allowed the person to work through the logical argument and identify their own issue more clearly. instead of trying to resolve difficult problems on our own.
• Make the time to connect with colleagues. It’s important to get things off your chest, gossip about other people, talk about what you’re excited about and challenged by, and generally get the pulse of what is happening beyond your remote world.

Development at Scale

There’s a perception among some in the industry that Cisco is stuck in the old way of developing features for our OS. It couldn’t be farther from the truth. We have reworked an array of internal processes (e.g., how we do in-service software upgrades, the addition of programmatic interfaces, breaking down functional areas into independent processes for greater resiliency).

Only a few companies in the world have our development scale – Google and Amazon, for example. With thousands of engineers writing code around the world daily, any error makes the entire system screech to a halt. In a small organization, an email to everybody can fix a software problem.  But with a large organization like Cisco, spread across different countries and time zones, that doesn’t always work.

Two features in the IOS XE development environment – via tooling and architecture – ensure that our engineers can keep doing their work, despite any errors made by others along the way.

Scale Through Tooling

IOS XE developers can validate fixes and changes independently, in isolated modules, before their work goes into the common code base. Small teams can address large software requirements modularly. That means that any breakages or other errors don’t impact everyone else.

It used to take three to four days to validate code. Now its down to three and a half hours. Developers can focus on running their tests and performing their builds. Any breakages are limited to the individual, functional unit. As the code moves through the process, it is re-tested. By the time it gets into the code base it’s clean.

How is this done? We enable the faster validation of commits by making as much of the code as possible common across the platforms. This drastically reduces the time required to compile code. We organize the tests into a hierarchy. The most critical tests run on every changeset. Those tests are most likely to be broken after a large number of developers use them. More localized tests run in a targeted fashion.

The Ability to Run Code Separately

The ability to run code separately, in modules, is inherent in the design of IOS XE. We can have teams working on apps and others working on infrastructure. The app developers can continue writing code independently of everyone else, even if there’s a breakage in their modules.

We just finished integrating the Viptela SD-WAN technology into our offering. Their code structure was different than ours and couldn’t handle the kind of development scale and velocity that our largest customers expect. We had to rework it to be more modular and allow people to work on it independently.

In upcoming blogs, I’ll discuss how the IOS XE team has added upgradability, resiliency, and programmability to IOS XE.

Check out our Intent-Based Networking video channel.

Subscribe to the Networking blog

Share: