The Changing Face of Microsoft Patch Management

Four years ago last month, SQL Slammer shot from anonymous malware larvae to Public Exploit Number One, infecting as many as 75,000 hosts between the time it was identified and stamped out. Then, just months later, another super worm -- this time dubbed MS Blaster -- once again wrought havoc, this time on both consumer and enterprise systems.

In the months after SQL Slammer and MS Blaster, Microsoft Corp. pledged to ramp up its security responsiveness, acknowledging that -- even though both worms targeted previously identified (and already patched) vulnerabilities -- the vagaries of patch management on Windows systems helped contribute to the destructiveness of the attacks.

In the case of SQL Slammer, for example, a lot of organizations weren't even aware that SQL Server was installed (much less running) in their environments: thanks to Microsoft's SQL Server Desktop Engine, which is installed with Visual Studio,Veritas Backup Exec and a host of other programs, a host of non-SQL Server systems were slammed by Slammer.

In several important respects, the situation is much improved, users say. In place of its once-chaotic patch delivery process, for example, Microsoft now has “Patch Tuesday,” whereby it issues software patches once a month on the second Tuesday. In addition, the software giant now provides an advance notification on the Thursday preceding Patch Tuesday to alert customers to what should be coming down the pike.

Similarly, Redmond's patch-delivery strategy, which once began and ended with Systems Management Server (SMS), has been enhanced, too. Today, the software giant augments SMS with Windows Update Services (WUS), the "Automatic Updates" feature that's built into both its desktop and server operating environments; Software Update Services (SUS); and Windows Server Update Services (WSUS), the successor to SUS.

But how does Microsoft's patching strategy rate with IT pros? After all, it was partly due to the hue and cry IT pros raised in the aftermath of first SQL Slammer and then MS Blaster that Microsoft introduced drastic changes in its security responsiveness. The good news, IT pros say, is that Microsoft (in the words of the White Album-era Beatles) has come a long, long, long way over the last four years. "We've not had a 'wormy' thing in a long time.  Most issues these days are browser [and/or] malware [attacks] or Word zero days, so that my 'must patch now' has been a bit lessened," says Small Business Server (SBS) MVP  and IT pro (E-MAILED STEVE FOR FORMAL TITLE/LOCATION/COMPANY) Susan Bradley.  "I still patch IE faster when there is a patch for that, and look at 'from remote' [code execution] for higher risk factors. Even when the item is labeled less critical by Microsoft, I'm looking at social engineering factors as a higher criteria and assigning more risk."

Bradley also avails herself of other Microsoft security resources, including Redmond's pre-Patch Tuesday advance notifications, e-mail security bulletins and, on occasion, its post-Patch Tuesday in-depth Webcast, too. "I'm signed up for the Advance Notice and the direct e-mails. I don't always listen to each Webcast," Bradley continues, but adds that she does listen to many of them.

Bryan Lucas, director of technical services with Texas Christian University (TCU), says he's similarly encouraged by Microsoft's security turnaround.

"[W]e think it is a great model.  At first [Microsoft] was somewhat mocked for how many patches they were releasing, but after a while the attitude became one of gratitude for fixing problems and respect for having a consistent procedure. In fact, now you see other vendors following the same model," Lucas indicates.

Lucas says TCU usually pushes out patches shortly after they become available via WSUS. "We get the e-mail notification, push them out to a pilot group via WSUS and then roll them out into production only a few days later," he confirms. For the most part, Lucas says, this arrangement has worked more or less perfectly. Although -- as with any change management practice in a large organization --  there was the occasional bump. "Seems somewhere -- it's hard to recall now -- around October there was a bad patch and we had issues with the svchost crashing, as did many other institutions."

As far as IT pro Dale Roberts is concerned, Microsoft's standardized patching policy is a huge improvement over the way things used to be. "It seemed like they just quietly released patches every now and then -- usually in response to some new exploit. Obviously, they were releasing patches whenever they identified vulnerabilities, either on their own [via internal testing] or from outside researchers, but you usually only heard about them when something went horrendously wrong, like with SQL Slammer," says Roberts, a network administrator with American University (AU) in Washington D.C.

That being said, Roberts points out, he only patches his non-Microsoft systems a few times a year, at most; he can't imagine doing that in an all-Windows environment: "I generally only have to update [Solaris] twice a year -- although if there's an exploit, like with the recent Solaris 10 TELNET flaw, I'll patch right away," Roberts says.

Microsoft has something of a checkered patching history. Its recent updates (e.g., Windows XP Professional Service Pack 2; Windows Server 2003 Service Pack 1) have been fairly reliable, but that wasn't always the case. Microsoft issued several bug-ridden Windows NT service packs (including a notoriously unstable Windows NT 4.0 Service Pack 2 update); several problematic Office updates (such as its notorious Service Release 1 updates for both Office 97 and Office 2000 ); and a handful of troublesome Back Office server updates, including no less than two flubbed releases of the same Exchange 2000 Server patch. 

The lesson many users drew, in any case, was to go slow and steady on the Windows patching front. That seems to be changing in some environments. Perhaps that's a reflection of the heightened importance of staying up to date on patches, at least in the post-Slammer and post-Blaster milieu. Perhaps it's an indication that Microsoft has vastly improved its patch development, patch review and patch regression testing practices. Most likely, it's some combination of both factors. In any event, users like Lucas push out their patches fairly rapidly, while Bradley says she patches IE flaws and potential remote code execution exploits as soon as patches become available. 

"We used to wait two weeks, [but] now we roll them out to production generally within five working days," Lucas indicates. "[U]ntil recently we never had any significant trouble even with that kind of fast pace.  We test to a group of about 50 power and internal users (we have more than 2500 faculty and staff computers we patch) and rely on their feedback  We have been considering a formal change control procedure for our desktop images."

No one seems to think Microsoft should rest on its laurels, however. "They aren't mature yet, but they are getting there and they continue a good faith effort and their tools are free. I'm not certain other platforms even have a free enterprise tool that you can control patching," Lucas concludes, noting that he's still waiting to see what Microsoft delivers in its forthcoming WSUS 3.0 release, which is currently in beta. "[I]n general, [I'd like to see] reporting and the ability to deploy non-[Microsoft] patches through WSUS. Granted that would overlap with SMS, but WSUS works very well comparatively."

SBS MVP Bradley, for her part, thinks Microsoft still hasn't quite made good on the vision outlined by CEO Steve Ballmer . "I don't feel that we've met those goals yet.  Yes, Vista has better patching APIs to majorly reduce reboots, and I'm probably reacting to the DST patches and how much sneaker net I still had to do to deal with those, but I don't think we're where we should be," she concludes. "We still need help in deploying systems securely; we're still running way too much as non-admin."