RCP Update

Sign up for our newsletter.

I agree to this site's Privacy Policy.


Microsoft BPOS Suffers Multiple Outages (UPDATED)

Customers of Microsoft's Business Productivity Online Services (BPOS) have experienced e-mail outages over the past few days.

The outages were first reported on Thursday afternoon by ZDNet upon receiving complaints from angry customers. ZDNet pointed to this Microsoft Online Services TechCenter forum where a number of customers using BPOS reported suffering service interruptions.

"Cause of #bpos mail delays mitigated; message queues are draining; watch the dashboard for updates," Microsoft said in a post on Twitter this afternoon. A Microsoft spokeswoman confirmed the outages, saying the company will issue a formal statement. 

The outages appear to have impacted Exchange Online, with some customers reporting message delays as well as lost e-mails. One Microsoft partner said in a phone interview that messages were delayed up to six hours.

"It's been resolved in that there are not currently delays but it's been in and out since Tuesday afternoon," the partner said. "I'm hoping that it [the outages] won't come back up again but I don't know for certain that it is permanently resolved."

Users of Exchange Online fumed on the online forum. "We're a worldwide corporation using this. If it doesn't improve, we may have to go back to in-house Exchange," said one poster.

Said another: "I migrated our company to Exchange Online from in-house Exchange 2003 last October, and I'm sorry to say that I regret everything I ever said about how this would be better. It has been far worse in terms of both performance and reliability. I hate to be so harsh, but I am deeply frustrated...We're actively looking at migration paths back to in-house e-mail."

The outages come as Microsoft is looking to promote the next generation of Exchange Online, Office 365, which it released for beta testing last month. The Microsoft spokeswoman said the outages "only affected BPOS and did not affect Office 365 whatsoever."

UPDATE: Late on Thursday afternoon, Dave Thompson, corporate vice president of Microsoft Online Services, posted an explanation for the BPOS outages on the Microsoft Online Services Team Blog. According to Thompson, the intermittent access reported by BPOS users was the result of issues relating to "malformed email traffic" that occurred on Tuesday and Thursday.

Thompson wrote:

"On Tuesday at 9:30am PDT, the BPOS-S Exchange service experienced an issue with one of the hub components due to malformed email traffic on the service. Exchange has the built-in capability to handle such traffic, but encountered an obscure case where that capability did not work correctly. The result was a growing backlog of email. By 12:00am PDT, the malformed traffic was isolated and the mail queues cleared. The delays encountered by customers varied, on the order of 6-9 hours. Short term mitigation was implemented and a fix was under development.

"At 9:10am PDT today, service monitoring again detected malformed email traffic on the service. The problem was resolved at 10:03am, but users experienced up to 45 minute email delays during this time. A second, but related issue was detected via monitoring at 11:35am PDT, resulting in email stuck in some end users' outboxes. The issue was remediated at 12:04pm PDT. During this time, more than 1.5 million messages had queued on the service awaiting delivery. The backlog was 90% clear by 4:12 PM, but because of this large backlog of email, customers may have experienced delays of as long as 3 hours. We are implementing a comprehensive fix to both problems."  

Thompson's post also noted that an unrelated DNS issue on Thursday "prevented users from accessing Outlook Web Access hosted in the Americas, and partially impacted some functionality of Microsoft Outlook and Microsoft Exchange ActiveSync devices." That issue was resolved a few hours later, Thompson said.

About the Author

Jeffrey Schwartz is editor of Redmond magazine and also covers cloud computing for Virtualization Review's Cloud Report. In addition, he writes the Channeling the Cloud column for Redmond Channel Partner. Follow him on Twitter @JeffreySchwartz.

comments powered by Disqus

Reader Comments

Wed, May 18, 2011 Jim CA

For 100 users or less, the cost savings is undeniable. Today, Exchange should be in house for any demanding environment with 500+ users(but this will likely change over the next few years).

Fri, May 13, 2011 JohnK US

Peter's comments are worth repeating "Whilst any outage be it on-premise or hosted is extremely frustrating and potentially costly it is always a possibility; what I would say is that Microsoft operate a financially backed SLA to cover customers subscription costs in the case of an outage - this SLA states 99.9% uptime (99.95% actual) . . . ." An in house network admin that claims he can provide that level of service is deluding only him/herself. When events like this happen on the 'local' or 'in-house' level, there's no forum to post the complaints and damnations, but that does not mean the same problems don't exist.

Fri, May 13, 2011 Lígia Bueno São Paulo

The paradigm shift offered by cloud computing has raised several arguments regarding the benefits and dangers. By purchasing a solution in Cloud Computing, it should be borne in mind that the tool should enable mobility, via the major browsers to access email, documents, calendar and chat, access anywhere, at any time for any browser without having to worry about bkp to email, documents, spreadsheets, etc.. It should also allow management of multiple user profiles and email accounts in one place, offer checking Anti-Spam and Anti-virus software out of equipment PC, Laptop or Server; large area of e-mail to the user (Over 20 GB per user), and of course, guarantee contract, high availability and data security. The platform Google Apps was born in this concept since 2008 becoming a tool for high availability and confidentiality. I think it's worth, now that you seek a new solution that meets business needs, giving attention to the Google model, before returning to contemplate the purchase of an internal structure for management of the e-mail.

Fri, May 13, 2011 Chir

These outages over the past year have actually been positive. Not only has it opened managements eyes to the PR song and dance lies, but it also has them seeing that in-house IT has value. On a side note my former "move everything to the cloud" supervisor has been given the heave ho, with luck we should have our mission critical mail back in house in 3 to 4 months. Thanks for the promotion MS!!

Thu, May 12, 2011 Peter Kempton United Kingdom

I have been working with Microsoft Online Services since it's launch in 2009 (BPOS) and have been working with the IT reseller community since 2010 to support and drive adoption of these services as a specialist cloud coach. Whilst any outage be it on-premise or hosted is extremely frustrating and potentially costly it is always a possibility; what I would say is that Microsoft operate a financially backed SLA to cover customers subscription costs in the case of an outage - this SLA states 99.9% uptime (99.95% actual). If we consider that in one year 99.95% equates to just under 4.5 hours of downtime a year we as businesses need to understand what that downtime means to us and plan for it's eventuality. In my experience working with Microsoft BPOS and Office 365 both in the SME and enterprise space a customers experience and understanding of the services is greatly improved when they engage with specialist IT companies who work with this technology and Microsoft day in and day out; as a company managing your own on-premise infrastructure any outage resolution can be classed only as best effort using the resources available to you - when opting for a cloud solution hosted by a third party you can be assured that their resources are dedicated to these solutions 100% of the time and that any "fix" will be handled promptly and professionally. Whilst I am a great advocate of cloud technology I often advise that it is not for everyone and that any adoption of it should be planned in detail with experienced IT partners who can advise customers as to best practices and preparations for any downtime. 

Thu, May 12, 2011 uz

Seriously!?! I've spent countless hours and months preparing a move to BPOS. In fact we just sent out a detailed plan yesterday for our sharepoint managers to review all the sharepoint data before migration. I really don't want to continue supporting and managing exchange in-house.. are there other vendors out there that will utilize our rackspace and manage an exchange server for us?! Being in IT.. I know technology can fail. But try explaining that to board members and execs. Any tips or suggestions would be greatly appreciated!

Thu, May 12, 2011 Juliano Adolfo Fenólio Sao Paulo, Brazil

I would say to the customers that intend to migrate back to in-house e-mail to not do such a thing. Keep on the cloud and move onto Google Apps.

Thu, May 12, 2011 scot

Our large organization went to BPOS last fall. We have had multiple unresolved issues which have prevented a complete migration. Now, we have had all day outages two days this week on top of all that. I can't believe what kind of PR Microsoft must have had to sell this thing to management.

Add Your Comments Now:

Your Name:(optional)
Your Email:(optional)
Your Location:(optional)
Please type the letters/numbers you see above