So, another Patch Tuesday has passed – and it was a big one. But the news late Thursday 02/11 was a bit less nice: it seems that one of the patches included causes that dreaded BSOD on certain Windows XP boxes.
Microsoft is aware of the problem, which involves the MS10-015 bulletin (aka the 17-year-old Windows flaw), and is investigating; see their discussion here. It seems that it’s not a universal problem, based on anecdotal evidence collected by SANS. The always dependable Brian Krebs, who broke the story, suggests that there is a work-around to eliminate the vulnerability without installing the patch; and earlier this morning Microsoft wrote: Customers can disable the NTVDM subsystem as a workaround and we have provided an automated method of doing that with a Microsoft Fix It that you can find here: http://support.microsoft.com/kb/979682. While this does not help netbook users using XP, it should ameliorate the situation for most of those impacted.
So, if you have not yet installed this update, please consider holding off until Microsoft is able to understand / resolve the issue. And, while the situation is still unclear, let’s take a moment to reflect on some lessons learned …
- Complexity. No one, not even the mighty Microsoft, can test all conceivable combination of hardware and software in the PC universe (yes, smug Apple fanboys, I’m leaving you out of this one) to understand how a match might negatively impact a particular system. Might be a good thing to remember next time you read complaints about how long it takes Microsoft to release a patch for some problem or other.
- Process. It’s good to have a patching process in place, which includes testing and phased roll-out; someday we should discuss this further (altho it’s outside my normal bailiwick) but in the meantime check this out.
- Tools. This is also why it’s important to have a good patching tool; if you run a sizable IT network, you’ll want to know who has already implemented the patch so you can concentrate on them instead of running around looking at *all* the boxes on your system.
- Simplification. And finally (for now), this should also get you thinking about simplifying the standard image on your network endpoints; the fewer unknown apps running on your boxes, the fewer interactions you have to worry about. This is probably as useful in normal day-to-day operations (think: fewer help desk calls) as it in “crisis” moments like this.
Overall, I do not feel that it’s fair nor constructive to fault Microsoft for this problem just yet – I’m sure the responsible engineering team at Microsoft is working 24×7 to root cause the issue and develop a fix. At this juncture, the facts are not yet clear – in fact, as I write this, Brian Krebs, is now reporting that the BSOD may be caused by malware (specifically, a rootkit). I’m sure we’re going to hear a lot more in the coming days – lots of armchair quarterbacking and second guessing – but I prefer to withhold judgment until all the facts are in. All I can say is: stay calm and carry on – have a plan, have the tools and have the understanding to implement both correctly.