Update About the October 4 Outage

UPDATE on October 18, 2021 at 1:25PM PT: We acknowledge how disruptive the outages earlier this month were to businesses and apologize for the impact and frustration they caused. To help address some questions that we’ve been hearing following the outages, we’re sharing an update about what happened and what we’re doing to address the issues.

What happened

The October 4 outage occurred due to a command issued by an engineer during a routine maintenance which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers from the Internet globally. Our Facebook Engineering blog provides a more detailed explanation about what caused this outage and why it took time to restore our services. Ads did not deliver during the time our systems were offline, and advertisers were not and will not be billed for ads during the outage. However, we’ve heard from customers that their campaigns experienced volatility as our services came back online.

Separately, on October 8, a configuration change caused some people and businesses to have trouble accessing some of our apps and products for a shorter period of time. Following discovery of the issue, our teams were able to quickly resolve it and restore access to our services.

What we’re doing

We have been working to understand how these outages impacted our customers and developing plans for how to rectify any issues that we identify, which includes compensation for impacted customers.

We recognize stability and reliability are critically important for the people and businesses who use our services around the world. That’s why we will continue our efforts to prevent similar disruptions from occurring in the future. As Santosh Janardhan, VP of Infrastructure, shared: “Every failure like this is an opportunity to learn and get better, and there’s plenty for us to learn from this one. After every issue, small and large, we do an extensive review process to understand how we can make our systems more resilient. That process is already underway... From here on out, our job is to strengthen our testing, drills, and overall resilience to make sure events like this happen as rarely as possible.”