Water-authority network upgrade spots problems faster

The Albuquerque water authority says recent network upgrades give it greater visibility and control over its remote sites and makes for faster responses to leaks and other problems.

The Albuquerque Bernalillo County Water Utility Authority manages more than 3,000 miles of water-supply pipeline covering more than 650,000 users. The authority manages 135 remote locations, which include well sites, tanks, and pump stations, all of which have programmable logic controllers (PLC) connected to a dedicated, fixed-wireless network running at 900MHz back to the core network.

“The [main treatment] plant was built [about] 15 years ago,” said Kristen Sanders, the authority’s chief information security officer. “So if a piece of equipment went out, replacing it would be about shopping on eBay.” Also the authority’s fiber backbone that connects the sites with the main plant was past it’s service life and had to be replaced.

Moreover, the remote sites weren’t attached to the IP network itself, so the previous management vendor Televent would have to remotely VPN into remote sites to diagnose problems, which generally weren’t apparent until something stopped working.

When completed, the upgrade will add ruggedized Cisco IE3400 switches with embedded Cisco Cyber Vision software to connect the remote PLCs to the underlying SCADA network. That enables the in-house IT team to manage and monitor them remotely rather than relying on a third-party to provide after-the-fact diagnoses of problems. The upgrade is underway but not all of the authority’s remote sites have been connected yet.

The new switches and software let the IT workers see leaks and other maintenance problems as they begin rather than letting them run until a component actually fails. “[Previously,] we didn’t have any notice. Something would stop working, and someone would have to go out and look at it,” said Sanders. “If there was some sort of network anomaly in the past, you’d have to get someone to do a Wireshark packet capture.”

Before the upgrade, the main purification plant and remote sites were connected via basic, multi-mode fiber, which was a good decade past its listed service life, said Sanders. Swapping in single-mode fiber allows signals to propagate better thanks to its smaller optical-core diameter, and the new fiber can carry more bandwidth.

The authority was already a Cisco shop and stuck with the vendor, according to network manager Jerry Monjaras. who performed much of the upgrade with the in-house staff and some remote assistance from Cisco. After the upgrade, the core switches are Cisco 9500s with 10G uplinks to a stack of IE5000s in the server room. The previous iteration of the network ran on ASA 5520 security appliances and 2955 switches, both of which reached end-of-support in 2018.

“My initial goal was just to upgrade the backbone to gig fiber, but with Cisco’s offering, they were able to add the Cyber Vision, so it was way more than I expected,” Monjaras said.

Between the upgraded data center switches and the new fiber, ruggedized switches for field equipment and the new software which runs in the authority’s data center, the overhaul provides multiple benefits.  “Everything’s managed almost completely in-house … which saves tons of downtime and tons of money,” said Monjaras.