Don’t let automation break change management

The drive to automate more and more network operations is a good thing, but it exposes a need for network teams to ensure their change-management processes are in order.

Networks are doing more, becoming integral to zero-trust security architectures, for example, and to end-to-end enterprise optimization endeavors. Networks are also connecting more things than ever: Mobile devices and IoT nodes continue to proliferate outside data centers and IaaS environments, while inside the enterprise, VMs and containers and separate environments segregating groups of them from each other for security purposes continue to proliferate.

Unfortunately, network teams aren’t growing as fast as the network’s footprint or scope of functions. Automation at every level, from deeply embedded “autonomous networking” functionality all the way up through the ad-hoc automation network teams write themselves, offers network professionals the lifeline they need to keep up.

In this context, where automation is essential and unavoidable, network teams need to make sure all the good they can do with automation is not done at the expense of or in conflict with one of the other pillars of enterprise IT: change management. They need to make sure automation is controlled by change management, and that they are keeping change management processes in step with their increasing reliance on automation.

One aspect is to implement change management on the automation, including the scripts, config files, and playbooks, used to manage the network. The use of code management tools helps with this: check-out and check-in events help staff remember to follow other parts of proper process. Applying change management at this level means describing the intended modifications to the automation, testing them, planning deployment, having a fallback plan to the previous known-good code where that is applicable, and determining specific criteria by which to judge whether the change succeeded or needs to be rolled back. (Read more: How to manage scripts that manage network automation)

Another aspect, sometimes overlooked, is to make sure that the presence and operation of automation is accounted for in the change management process.

Consider an example. A network engineer brings a planned configuration change through the change management process, to allow a new kind of HVAC sensor the facilities team is trialing to communicate with its management platform. During an approved change window, a network tech successfully makes that change, manually. The control panel sees the sensor, huzzah! But a few minutes after that testing, the control panel loses contact with the sensor again. The tech revisits the switch and finds the configuration has reverted to its previous “gold” template, re-executes the change, and the controller sees the sensor again. Rinse and repeat, until the tech and the engineer realize that an automated “sweep-and-correct script” meant to prevent configuration drift is the problem.

Failure to ensure that the configuration change got properly wired into the automation resulted in a failed change event and indicates a gap in the process.

There are multiple implications:

Putting in place automation to lock-in a network state is a change management event, and in a sense, a change to the architecture; creating it and putting it into production needs to go through the whole approval and deployment process, and all future changes need to be made with its presence and operation in mind—considering it has to be part of future change management evaluations.

In addition, changing anything in the environment means either changing the script to make and maintain that change itself, or building into it a mechanism for tagging a deviation from normal configurations as being known and approved.

This consideration of change management is equally important whether your environment is built around a central network controller, SDN style, or around staff-written imperative automations in whatever language, or declarative automation, or (as is increasingly common) a mix of all these mechanisms. Following proper process is the only way to build a reasonably complete and accurate picture of what is on the network (they’re never going to be 100% complete and accurate for most folks). And, it is the only way to keep staff and software all going in the same direction, working together instead of working at odds with each other.