8 big IT failures of 2023
Starting and stopping digital processes at the exact same time every day is in fact something computers are fairly good at, and that people tend to screw up now and then, so it was perhaps inevitable that one of these days a crisis would arise. And arise it did on January 24, when a Chicago employee failed to turn the backup server off at the appropriate time. As a result, when trading began in New York at 9:30 a.m., the NYSE computers thought they were continuing the previous day’s trading session and ignored the day’s opening auctions, which are supposed to set initial prices for many stocks. The outcome was a series of violent market swings and numerous transactions at incorrect prices that had to be cancelled at great expense. The lesson: Never send a human to do a computer’s job, especially if that computer’s job is pretty simple.
In space, no-one can cancel your software license
NASA is a scientific marvel that does all sorts of cool and inspiring space stuff; it’s also a sprawling government bureaucracy with thousands of employees and computer systems under its umbrella. Unfortunately, the agency is having a harder time keeping track of all those computers than it is various bits of space debris. A report this year from the OIG focused on numerous licenses NASA purchased for Oracle products to support the Space Shuttle program, which wrapped up more than a decade ago; not only is the agency locked into Oracle tech as a result, but poor documentation processes means that NASA isn’t really sure how many of those Oracle systems they’re actually using. As a result, the agency spent $15 million over the past three years on software it may not be using, but didn’t want to risk a software audit from Oracle that might end in a fine that’s even more costly.
The solution to a problems like this is to implement a software asset management program that can help you understand exactly what software you’re using and what license you need and don’t need. The good news is that the US federal government has mandated that agencies like NASA implement such programs; the bad news is that, according to the OIG report, “efforts to implement an enterprise-wide software asset management program have been hindered by both budget and staffing issues and the complexity and volume of the agency’s software licensing agreements.”
Software licensing situation cloudy
If NASA serves as an example of an overly cautious government agency paying for software it may not be using just in case, cloud service provider Nutanix was rocked by a scandal this May when it emerged that the company was taking the opposite approach to software licensing. Specifically, Nutanix used third-party software in a “noncompliant manner,” which is a euphemistic way of saying “without paying for it, even though they were supposed to pay for it.”
The company used software from two different vendors for the purposes of “interoperability testing, validation and customer proofs of concept, training and customer support.” Unfortunately, they did all that using versions of the software that were marked for evaluation purposes only, an “evaluation” process that lasted for years. The issue was discovered by an internal review, and because the vendors needed to be paid for the noncompliant use, Nutanix was unable to file its quarterly earnings report to the SEC on time because it was trying to get a handle on what it owed. The screwup resulted in the CIO leaving the company, with the lesson perhaps being that the only thing worse than paying for software you don’t use is not paying for software you do.
Turn off the lights, the party’s over
This next story is, technically, an IT fail that dates back to 2021, but we’ll include it this year’s roundup because it was in 2023 when it was finally resolved. For nearly 10 years, Minnechaug Regional High School in Massachusetts had been happily running a “green lighting” system installed by 5th Light that automatically adjusted the lights inside and outside the school as needed. But in August 2021, teachers and students noticed that the lights were staying on at full brightness continuously. It turns out the system had been hit by malware, and had gone into a fallback mode in which the lights never turned off.
