Cloudflare’s Major Outage: A Deep Dive into the Technical Breakdown
On November 19, 2025, Cloudflare, a leading internet infrastructure provider, experienced a significant network failure that disrupted global internet traffic for several hours. This incident affected millions of users and various online services, highlighting the vulnerabilities inherent even in robust cloud infrastructures.
Incident Overview
The outage commenced at 11:20 UTC and was traced back to an internal configuration error, not a cyberattack. This event underscores the potential for routine internal processes to inadvertently cause widespread service disruptions.
Root Cause Analysis
The disruption originated from a routine update to permissions within Cloudflare’s ClickHouse database cluster, aimed at enhancing security for distributed queries. At 11:05 UTC, this change inadvertently made underlying table metadata in the ‘r0’ database visible to users. Consequently, a Bot Management query failed to account for this visibility, pulling duplicate column data and expanding a critical feature file to double its expected size.
This feature file, refreshed every five minutes to combat evolving bot threats via machine learning, exceeded the software’s hardcoded limit of 200 features. This breach triggered panics in the core proxy system known as FL.
Impact on Services
Initially, the fluctuating failures were mistaken for a massive Distributed Denial of Service (DDoS) attack, coinciding with the downtime of Cloudflare’s external status page. The Bot Management module, essential for scoring automated traffic, halted request processing, causing cascading errors throughout the network.
In the newer FL2 proxy, this led to outright 5xx HTTP errors; older FL versions defaulted bot scores to zero, potentially blocking legitimate traffic for customers using bot-blocking rules. Core services were severely impacted, delivering error pages to users accessing Cloudflare-protected sites and increasing latency due to resource-intensive debugging.
The Turnstile CAPTCHA system failed entirely, blocking logins; Workers KV experienced elevated errors, indirectly crippling dashboard access and authentication via Cloudflare Access. Email Security temporarily lost some spam detection capabilities, though no major customer data was compromised, and configuration updates lagged.
Recovery Efforts
By 17:06 UTC, full recovery was achieved after halting the propagation of the faulty file, rolling back to a known-good version, and restarting the proxies. Cloudflare’s CEO, Matthew Prince, expressed sincere apologies, describing the incident as deeply painful and unacceptable for a major internet service provider. The company identified this as its worst core traffic outage since 2019.
Contextualizing the Incident
This incident is part of a concerning trend of failures related to configuration issues among major cloud providers. Just weeks prior, on October 29, 2025, Azure suffered a global outage due to a buggy tenant change in its Front Door CDN, disrupting Microsoft 365, Teams, and Xbox for hours and affecting airlines like Alaska. Similarly, AWS endured a 15-hour blackout on October 20 in its US-East-1 region, where DNS issues in DynamoDB affected EC2, S3, and services like Snapchat and Roblox.
These incidents highlight the over-dependence on centralized providers, where single missteps can break the internet repeatedly in 2025.
Preventive Measures and Future Steps
To prevent future incidents, Cloudflare is strengthening its file ingestion processes to guard against malformed inputs. They are also implementing global kill switches, reducing the overload of error reports, and reviewing proxy failure modes. Although the outage was not caused by malicious intent, it serves as a clear reminder that as cloud ecosystems expand, the importance of operational precision also increases.
Conclusion
The November 19, 2025, outage at Cloudflare underscores the critical need for meticulous internal processes and robust safeguards within cloud infrastructures. As the digital world becomes increasingly reliant on these services, ensuring their stability and resilience is paramount.
