The outage was triggered by the convergence of three distinct factors. First, a software defect was embedded in a fix for Microsoft 365 networking gateway servers. Microsoft noted that this defect remained undetected during standard load testing because the company’s test labs used “weaker hardware SKUs with fewer cores,” whereas the defect was “worsened on machines with more cores and newer hardware.” Second, a surge in load on the networking gateways occurred on Wednesday, Nov. 11, following a minor issue with Microsoft Teams that prompted a failover to an alternative gateway. While this failover is part of standard resiliency operations, it altered overall traffic patterns. Third, an unrelated issue the previous day left 14% of Microsoft’s server capacity for the networking gateway out of service. Lamanna clarified that this capacity reduction “had no impact on its own,” but when combined with the increased load, it forced the network gateway to encounter the latent bug during the peak traffic window in the United States.
As requests built up, servers stopped responding, leading to a cascading failure. Microsoft’s automated systems attempted to reroute traffic, which caused the defect to recur across the network, eventually affecting more than 80% of its servers. Recovery was slower than typical outages because the automated recovery processes were not designed to handle such a broad unresponsive state. Engineers were required to take servers offline individually to apply a fix before carefully reactivating them. When servers were brought back online, they were immediately overwhelmed by the volume of waiting traffic, causing them to become unresponsive again, which extended the duration of the service interruption.

Microsoft has apologized for the disruption, with Lamanna stating, “We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations.” The company emphasized that the defect had been “saturated globally over several weeks” without prior impact, highlighting a gap between their production hardware environments and their testing protocols. The incident underscores the complexity of modern cloud infrastructure, where minor configuration changes and standard failover mechanisms can interact to create systemic failures. No further regulatory steps or patch deadlines were announced beyond the internal fixes applied to restore service.




One Comment