Microsoft Promises Better Communication After Azure Leap Day Outage
It was bad enough that Microsoft's Windows Azure cloud service was unavailable for much of the day on Feb. 29 thanks to the so-called Leap Day bug. But customers struggled to find out what was going on and when service would be restored.
That's because the Windows Azure Dashboard itself wasn't fully available, noted Bill Laing, corporate VP of Microsoft's Server and Cloud division, in a blog post Friday, where he provided an in-depth post-mortem with extensive technical details outlining what went wrong. In very simple terms, it was the result of a coding error that led the system to calculate a future date that didn't exist.
But others may be less interested in what went wrong than in how reliable Windows Azure and public cloud services will be over the long haul. On that front, Laing was pretty candid, saying, "The three truths of cloud computing are: hardware fails, software has bugs and people make mistakes. Our job is to mitigate all of these unpredictable issues to provide a robust service for our customers."
Did Microsoft do enough to mitigate this issue? Laing admits Microsoft could have done better to prevent, detect and respond to the problems. In terms of prevention, Microsoft said it will improve testing to discover time-related bugs by upgrading its code analysis tools to uncover those and similar types of coding issues. The problem took too long -- 75 minutes -- to detect, Laing added, noting the specific issue regarding detecting fault with the guest agent where the bug was found.
Exacerbating the whole matter was the breakdown in communication. The Windows Azure Service Dashboard failed to "provide the granularity of detail and transparency our customers need and expect," Laing said. Hourly updates failed to appear and information on the dashboard lacked helpful insight, he acknowledged.
"Customers have asked that we provide more details and new information on the specific work taking place to resolve the issue," he said. "We are committed to providing more detail and transparency on steps we're taking to resolve an outage as well as details on progress and setbacks along the way."
Noting that customer service telephone lines were jammed due to the lack of information on the dashboard, Laing promised users will not be kept in the dark. "We are reevaluating our customer support staffing needs and taking steps to provide more transparent communication through a broader set of channels," he said. Those channels will include Facebook and Twitter, among other forums.
Windows Azure customers affected by the outage will receive a 33 percent credit, which will automatically be applied to their bills. However, such credits, while welcome, rarely make up for the cost associated with downtime. But if Microsoft delivers on Laing's commitments, perhaps the next outage will be less painful.
See Also:
Posted by Jeffrey Schwartz on March 14, 2012