The Perils of Patching

The process of keeping systems updated shouldn't be treated as the mindless that we mistake it for; no strategy is an invitation to disaster.

• By Roberta Bragg
• February 09, 2004

In recent columns I've been advocating automatic updating for home users and small businesses. As expected, several of you relayed tales concerning automatic patching and how it can cause problems. I won't belabor the point, but I will repeat it one more time: Determine the patching strategy that makes sense for your business. This strategy is based on your risk assessment. In a very small business or home network, the risk of disaster from not applying patches overwhelms the risk that a patch may break some functionality.

However, Kevin Perry, Director of Information Technology for Southwest Power Pool (SPP), www.spp.org, reminds us that in at least one industry, there's greater risk in a patch breaking functionality than there is in a patch not getting deployed. SPP includes as members 14 investor-owned utilities, six municipal systems, eight generation and transmission cooperatives, three state authorities and one federal government agency, three independent power producers, and 13 power marketers. SPP coordinates, promotes and communicates information about reliability in all aspects of the energy its members serve to its more than 4 million customers. It covers a geographic area of 400,000 square miles, with a population of more than 18 million people.

Mr. Perry has this to say about automatic updates:

"Roberta: While in many cases, and especially in the case of home users, automatically updating the Windows operating system and applications is desirable, we must be careful to not make it a mandatory standard across the board as Microsoft is supposedly considering. Within the electricity sector, we have critical systems that operate the grid and keep the lights on. Some of these systems are Windows based, some are Unix based, and all need to be patched as security upgrades are announced. And within the industry, we are generally very good about applying the patches as quickly as we can, as well as mitigating risk by other means. The issue is that we must first carefully test the installation of the patch in an offline environment to ensure that it does not break the critical third-party applications we are running.

"The last thing I want to read by candlelight is that the next massive power outage occurred when patches were automatically installed to our critical systems…

"I am not suggesting that automatic updates applied to a critical system will result in the next great blackout. Within the electricity industry, we have recently implemented a cyber security standard that requires utilities operating control areas to test updates to their critical operations control center systems in an isolated environment, i.e. an offline system. That does not mean that the critical production system won't crash when the update is applied, but it minimizes the risk. And that is one of the reasons our critical systems are redundant as well. We have to be able to control when the update is applied as part of our risk mitigation strategies."

This is an important statement for a couple of reasons. First, it's a good reminder for all of us. Decide what your risk factors are and act accordingly. For many of you, that means a structured patch implementation process that includes testing.

Second, there are vulnerable computer systems in places you never imagined. Control systems that used to run on proprietary processors and use proprietary communication protocols now run on Windows, Unix and Ethernet. This means administrative consoles, control systems and other components that are part of a utility or manufacturing plant must be protected in much the same way we protect ordinary networks; in some cases, more so. These control systems may be just as vulnerable as ordinary networks to attacks by individuals and by worms and viruses.

No, I don't think that utility companies are ignoring common sense and hooking their control systems directly up to the Internet so that operators can surf the Web. But they might be connecting them to office networks that are connected to the Internet. They might also be in danger if remote connections are provided for monitoring and maintenance purposes. In fact, these computers and their networks are subject to all the vulnerabilities that the ordinary network has.

Perry's e-mail is significant for a number of reasons. The director of IT for an organization of power companies is taking issue with me for advocating automatic updating, and has laid out the possible horrific consequences of a thoughtlessly applied patch. And his organization has given approval to having his e-mail quoted. This tells me that the risk, while present, isn't being ignored. It tells me that work's being done to secure these systems. The people responsible for security there are being proactive. They're doing risk evaluation, testing patches and applying them.

Please don't take this to mean that every utility company and manufacturing plant is doing all they can; I don't know that.

Here's what I do know, however: If you're part of an organization that uses similar Windows-based control systems, and your organization hasn't been addressing this issue, it's time to do so. Traditionally, process people and IT people have their differences. I have some thoughts on how to help them work together to secure their systems and would be happy to talk to you. If you're addressing security issues having to do with control systems, I'd also be happy to hear from you.