Tech

Microsoft attributes global Outlook outage to latent defect and capacity issues

The outage was triggered by the convergence of three distinct factors. First, a software defect was embedded in a fix for Microsoft 365 networking gateway servers. Microsoft noted that this defect remained undetected during standard load testing because the company’s test labs used “weaker hardware SKUs with fewer cores,” whereas the defect was “worsened on machines with more cores and newer hardware.” Second, a surge in load on the networking gateways occurred on Wednesday, Nov. 11, following a minor issue with Microsoft Teams that prompted a failover to an alternative gateway. While this failover is part of standard resiliency operations, it altered overall traffic patterns. Third, an unrelated issue the previous day left 14% of Microsoft’s server capacity for the networking gateway out of service. Lamanna clarified that this capacity reduction “had no impact on its own,” but when combined with the increased load, it forced the network gateway to encounter the latent bug during the peak traffic window in the United States.

As requests built up, servers stopped responding, leading to a cascading failure. Microsoft’s automated systems attempted to reroute traffic, which caused the defect to recur across the network, eventually affecting more than 80% of its servers. Recovery was slower than typical outages because the automated recovery processes were not designed to handle such a broad unresponsive state. Engineers were required to take servers offline individually to apply a fix before carefully reactivating them. When servers were brought back online, they were immediately overwhelmed by the volume of waiting traffic, causing them to become unresponsive again, which extended the duration of the service interruption.

Photo by panumas nikhomkhai on Pexels

Microsoft has apologized for the disruption, with Lamanna stating, “We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations.” The company emphasized that the defect had been “saturated globally over several weeks” without prior impact, highlighting a gap between their production hardware environments and their testing protocols. The incident underscores the complexity of modern cloud infrastructure, where minor configuration changes and standard failover mechanisms can interact to create systemic failures. No further regulatory steps or patch deadlines were announced beyond the internal fixes applied to restore service.

Thomas Reed

Thomas Reed writes about the technology industry with a focus on AI, digital services, cybersecurity, software, and emerging technologies. He follows company announcements, product changes, technical developments, and regulatory issues shaping the sector. Thomas aims to translate technical developments into clear reporting without oversimplifying important details or presenting early claims as established facts.

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button