Major Cloudflare Outage Was Down to a Botched Update File

The recent Cloudflare outage that took half the internet with it wasn't caused by some major new DDoS attack or a socially engineered hack of the company servers. Instead, it was triggered by a botched update. Designed to improve the efficiency of Cloudflare's server infrastructure, the update instead caused a file to double in size unexpectedly, causing website after website to fall flat on its face.

For tech writers like myself, it was mad to see so many of the usual publications I haunt go down during the outage, but major mainstream services were affected, too. ChatGPT went down, as did Uber, McDonald's, Twitter/X, and long-time esports king, League of Legends.

And it was all because of one file. Initially, Cloudflare analysts and internet commentators claimed the snafu was likely caused by a DDoS attack—potentially because Micorosft's Azure had been hit by one mere days before—but that proved to not be the case.

"The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind," Cloudflare wrote in bold text on Nov. 18.

Credit: Clourflare

Cloudflare CEO Matthew Prince explained what happened in an apology blog post. When adjusting permissions for files in a specific database, a file in Cloudflare's bot manager tool inadvertently doubled in size. This was designed with very specific file size limitations in mind, when one of them breached it, it completely broke apart, breaking key functionality for Cloudflare's protective system—and it took huge swaths of the internet with it.

It took a few hours for Cloudflare to track this problem down, but once it did, the fix was trivial. The service simply updated the database with an older version, and the bot was fixed. Therefore, so was the internet.

Prince said this was the worst outage Cloudflare has faced since its major 2019 downtime, and pledged to adjust Cloudflare policies to prevent it from happening again.