Facebook and its sister properties Instagram and Whatsapp suffer from persistent global failures. We don’t yet know why this happened, but the how is clear: this morning, Facebook prompted the company to revoke vital digital records that were telling computers and other internet-enabled devices how to find these targets online.
Doug Madory is Director of Internet Analysis at Kentik, a San Francisco-based network monitoring company. Madory said around 11:39 a.m. ET (3:39 PM UTC) today that someone on Facebook had initiated an update to the company’s Border Gateway Protocol (BGP) records. BGP is a mechanism by which Internet service providers worldwide exchange information about which providers are responsible for forwarding Internet traffic to which specific groups of Internet addresses.
In simpler terms, sometime this morning Facebook took away the map that tells the world’s computers how to find their various properties online. Therefore, when you type Facebook.com into a web browser, the browser has no idea where to find Facebook.com and returns an error page.
The Facebook outage not only stranded billions of users, it also prevented its employees from communicating with one another through their internal Facebook tools. That’s because Facebook’s email and tools are all managed internally and through the same domains that are now stranded.
“Not only are Facebook’s services and apps no longer available to the public, but its internal tools and communication platforms, including Workplace, are also out,” the tech reporter told the New York Times Ryan Mac tweeted. “Nobody can do any work. Several people I spoke to said it was like a ‘snow day’ at the company. “
The failures come just hours later 60 minutes from CBS has broadcast an eagerly awaited interview with Frances Haugen, the Facebook whistleblower who recently leaked a series of internal Facebook investigations that showed the company knew its products were causing mass harm and that it profits from taking bolder steps to curb abuse on its platform – including disinformation and hate speech – prioritized.
We don’t know how or why the outages at Facebook and its other facilities continue, but the changes must have come from inside the company as Facebook manages these records internally. It is currently unclear whether the changes were made maliciously or accidentally.
Madory said it could be that someone screwed it up on Facebook.
“Over the past year or so we’ve seen a lot of these big outages where some sort of update to their global network configuration went wrong,” Madory said. “Of course we can’t rule out someone hacking you, but you could have done that to yourself too.”
Update, 4:37 p.m. ET: Sheera Frenkel with the New York Times tweeted that Facebook employees told her they were having trouble accessing Facebook buildings because their employee badges were no longer working. This could be one of the reasons this outage has been going on for so long: Facebook engineers may have trouble physically accessing the computer servers required to upload new BGP records to the global internet.
Update, 6:16 p.m. ET: A trusted source who spoke to someone about Facebook’s recovery efforts was told that the outage was caused by a failed routine BGP update. The source said the buggy update prevented Facebook employees – most of whom work remotely – from undoing the changes. Meanwhile, those with physical access to Facebook’s buildings could not access Facebook’s internal tools as they were all tied to the company’s stranded domains.
Meanwhile, several different domain registration companies have put the Facebook.com domain up for sale. It did this thanks to automated systems that search for registered domains that appear to have expired, abandoned, or recently vacated. There’s no reason to believe that this will actually sell Facebook.com, but it’s fun to think about how many billions of dollars it could make in the open market.
This is a developing story and will likely be updated later in the day.