Press "Enter" to skip to content

Fast.ly broke the Internet for an hour this morning


Enlarge / The United Kingdom government’s official website was one of those affected by this morning’s outage. The cryptic “Guru Mediation” message shown is an untrapped, unskinned error returned from the Varnish cache server powering the Fastly CDN.
Leon Neal via Getty Images

For roughly an hour this morning—6 am to 7 am EDT, give or take a few minutes—enormous swathes of the Internet were down or interestingly broken. Sites taken down included CNN, The Guardian, The New York Times, PayPal, and Spotify, among many more—including The Verge, which resorted to reporting via Google Docs during the duration of the outage.

The underlying problem was an outage at Fastly, one of the world’s largest Content Delivery Network providers—the entire service went down due to a misconfiguration which it had deployed to all of its Points Of Presence (POPs) globally. As a result, sites using Fastly for content delivery came up with various errors dependent on the local site configuration. Some sites delivered relatively uninformative, plain HTTP 503 (Service Unavailable) pages, while others returned errors such as “Fastly error: unknown domain.”

The “unknown domain” error gives us some tantalizing hints to the nature of the problem, which is more than Fastly’s own status updates have so far. This tells us that Fastly’s network was up and its Varnish cache servers were answering requests, but its cache configuration—the Varnish Control Language files which point the cache server to the back-end servers supplying the original content—was almost certainly either missing or garbled.

Just came in:  Galaxy S22 Camera Specs Leak Tips Improved Hardware

Fastly’s own status page acknowledged the issue at 5:58 am EDT, then declared it identified at 6:44 am and fixed (with “increased origin load,” which in many cases may effectively have meant sites still unavailable for a while) at 6:57 am. The status page classified the issue as a “global CDN disruption” but gave no technical details. A tweet from the Fastly engineering team gave slightly more detail:

Although the outage was mercifully brief—Director of Internet Analysis Doug Madory told CNN that his firm Kentik saw Fastly traffic disappear at 5:49 am and begin reappearing at 6:39 am—the financial impact can be enormous. Media measurement firm Kantar guesstimated $29 million dollars in ad revenue lost worldwide during the hour-long outage.

Fastly’s brief snafu doesn’t appear to have bothered investors too much. FSLY dipped $0.81 in pre-market trading at the New York Stock Exchange—likely due to the outage itself—but bounced up $3.24 by 10 am. At closing, it was up to $56.20, a 10.8 percent increase from yesterday’s $50.70 closing price.