Why Microsoft 365 Went Down on November 12: The Root Cause, Explained

Rows of servers in a data center

Microsoft has published its post-incident report for the November 12 outage that took down Outlook and Exchange Online across the Americas. The short version: a bug Microsoft shipped in July sat quietly in the servers that front Microsoft 365 for more than three months. On November 12, extra traffic and a chunk of servers already out of service pushed those servers into the bug, and the systems built to route around failures spread it across the whole fleet. Email was down longest because Exchange Online, unlike Teams or OneDrive, could not be moved off the failing gateway. Nothing a customer did caused it or could have prevented it, which is why the useful lessons are about planning.

Key takeaways

  • The root cause was a latent defect in a Windows component built specifically for Microsoft’s networking gateway servers. It does not affect Windows PCs or servers in your office.
  • Three things lined up: the July bug, extra load from Teams failing over to the same gateway, and about 14% of gateway capacity already out for an unrelated issue.
  • Automatic failover made it worse. Each server that stopped responding pushed its traffic onto the rest, and within two hours more than 80% of gateway servers were unresponsive.
  • Teams, OneDrive, SharePoint and Office apps recovered earlier because they could switch to a second gateway. Exchange Online could not, so email stayed down.
  • For a small business, the lesson is to plan for a day without email specifically, and to keep your own copy of your mail.

What caused the November 12 outage?

According to Microsoft’s post-incident report, shared with Microsoft 365 administrators (a copy was posted on r/sysadmin), the outage hit the networking gateway that sits in front of many Microsoft 365 services. Three factors combined.

A bug from July. A fix prepared specifically for the gateway servers introduced lock contention inside a Windows process. Under certain load, it used up the server’s CPU and memory until the machine stopped responding. The change rolled out through Microsoft’s normal staged deployment starting in July and ran worldwide for more than three months with no problems. Microsoft says its load testing missed it because the lab traffic and hardware didn’t match production: the bug is worse on newer machines with more cores, and testing focused on smaller hardware.

More load than usual. Microsoft runs two separate gateway systems so services can move between them during trouble. About 15 hours before the outage, Teams moved from the other gateway onto the main one that also carries Exchange Online. That is routine, but it added load and changed the traffic pattern.

Less capacity than usual. Nearly 14% of gateway servers were out of service because of a separate issue from the day before. On its own that was fine. Combined with the extra Teams traffic during peak US hours, it was enough to trigger the July bug.

Why did it spread so far so fast?

When a server stops responding, Microsoft’s traffic management automatically sends users to healthy servers. Normally that is what keeps an outage small. Here it did the opposite. The same load landed on fewer servers, which pushed each of them into the same bug, and the failure moved from server to server. In under two hours more than 80% of the gateway servers were unresponsive. The walls between regions that should have contained it also failed, because of yet another separate issue, so the problem spread globally.

Why was email down longer than Teams?

Teams, OneDrive, SharePoint and the Office apps were able to fail over to Microsoft’s second gateway system, so they came back earlier and Teams chats and meetings mostly worked during the outage. Microsoft says Exchange Online cannot move off this gateway for scale reasons, so email had to wait for the main gateway to be fixed.

Why did recovery take so long?

Two reasons. As servers came back, the waiting traffic hit them all at once and knocked them over again, and Microsoft’s automated recovery wasn’t built for a fleet that was almost entirely unresponsive. Also, because the bug came from a change made months earlier, the usual quick fixes (rolling back, turning off a feature flag) weren’t available. Engineers built an update that returned the code to its earlier state, then took groups of servers out of service, patched and restarted them, and brought them back a region at a time.

What should a small business take from this?

This is a good example of what we wrote about recently in Software Is Breaking Down: a failure caused by several ordinary things lining up inside a system too large for any one test to cover. You can’t change how Microsoft runs its gateways. You can change how much a day like November 12 costs you.

  1. Plan for email being the thing that’s down. Teams kept working for most people. Decide in advance that Teams, a phone tree or text messages are how your staff and key clients reach each other when Outlook is out, and keep a contact list stored outside Microsoft 365.
  2. Don’t count on redundancy you can’t see. Microsoft had two gateway systems and failover between regions. Email still went down. Your own plans (a second internet line, a backup server, a second vendor) are worth testing, not just owning.
  3. Keep your own backup of mail and files. Microsoft keeps the service available. A separate backup of Exchange Online, OneDrive and SharePoint means you can still find an important message or file when the service is having a bad day.
  4. Write down what changed. Microsoft’s bug took months to show up. In a small office, the same pattern looks like a setting changed in the summer that causes trouble in the fall. A simple change log makes those much faster to trace.
  5. Watch for follow-up scams. Outages are followed by fake “verify your mailbox” emails. Microsoft does not ask you to sign in through a link to restore service.

Our write-up from the day of the outage has a short checklist for what to check after mail starts flowing again.

Frequently asked questions

Does this bug affect the Windows computers in our office?

No. Microsoft says both the defect and the fix that caused it were specific to its Microsoft 365 networking gateway servers and do not apply to other Windows deployments.

Could we have done anything to avoid the outage?

No. The problem was entirely inside Microsoft’s infrastructure. What you can control is how your business keeps working during an outage and how quickly it recovers afterward.

Should we move email away from Microsoft 365?

For most small businesses, no. Microsoft 365 is still more reliable than running your own mail server. Plan for the rare outage, keep a separate backup, and make sure staff know what to use when email is down.

Want a plan for the next outage?

An outage plan for email, files and phones is part of our free 30-minute IT assessment. We go through it with you and give you the findings in writing. Book a free assessment or call us at (617) 322-5155.

— Boston Managed IT

Picture of Nicholas Salem

Nicholas Salem

Nick Salem founded Boston Managed IT in 2017. Before that he spent six years as IT director at WCI, a Boston nonprofit, where he built the IT department from scratch and learned budgeting and forecasting alongside systems administration. He works with small and mid-sized businesses and growing companies on IT infrastructure, security and policy, and writes about Microsoft 365, security and running IT for small businesses.

Free assessment

Not sure where you stand?

Thirty minutes with a senior engineer on your security, backups, and Microsoft 365. Findings in writing.

Keep reading

Tangled network cables in a server rack

Software Is Breaking Down. What That Means for Your Small Business

Outages, PHP2Shell and supply chain attacks point to a systemic software...
Small business owner reviewing a cybersecurity checklist on a laptop at a desk

NIST’s New Small Business Cybersecurity Guide: What to Do First

The National Institute of Standards and Technology just published a small...

Person at a laptop waiting for email to load during a Microsoft 365 outage

Microsoft 365 Outage on November 12: What Happened and What to Do Next

Outlook, Exchange Online and Teams went down across the Americas on...

Free tools

Check where you stand, no call needed.

Email security check

Can someone send email as your company? Grade your SPF, DKIM and DMARC in about ten seconds. Public DNS only.

7 cybersecurity mistakes

A short cheat sheet of the mistakes small businesses make most, and how to fix each one.

Free technical assessment

Thirty minutes with a senior engineer.

  • Security posture, backups, and Microsoft 365, reviewed live
  • The three fixes that matter most, ranked, with rough effort
  • Findings in writing, and an honest answer on whether you need us

Book your assessment

Next, you pick a time on our calendar. No sales deck.

Prefer the phone? (617) 322-5155