Any update on this outage? It's > 24 hours now.

I'm a bit amazed at the size of this outage. Any updates? Much of our product has been down and we have lots of customer support inquiries asking about when this will be up. Nothing on twitter about the outage.
19 Replies
HuskyNZ
HuskyNZ9mo ago
Cloudflare Status
Welcome to Cloudflare's home for real-time and historical data on system performance.
ianh
ianh9mo ago
can you share whats going on? we need to blast our entire userbase at this point so we'd love to give them some info on whats going on
Chaika
Chaika9mo ago
Specifically: https://www.cloudflarestatus.com/incidents/hm7491k53ppg Which parts are down though? For the most part, Edge Traffic is unaffected
ianh
ianh9mo ago
Images are down for us Do we know why this is happening? This is a pretty major outage, I'd expect a cloudflare twitter announcement there are dozens of services down but nothing on the public threads
HuskyNZ
HuskyNZ9mo ago
Images are showing as Degraded Performance so im not surprised Not the full extent At least from what i can gather
ianh
ianh9mo ago
wow, good luck restoring this. It's crazy how massive of an outage
Chaika
Chaika9mo ago
Power to Cloudflare’s core North America data center has been partially restored. Cloudflare has failed over some core services to a backup data center, which has partially remediated impact. Cloudflare is currently working to restore the remaining affected services and bring the core North America data center back online.
The closest offical explaination they gave given, datacenter lost power, only partial restored for a while
HuskyNZ
HuskyNZ9mo ago
Its instreasting that it's taken down alot more of the core services,
ianh
ianh9mo ago
im blown away by how large this is and the length its been down
Chaika
Chaika9mo ago
the core indeed has most of their core services, only a few like tunnel api failed over instantly
HuskyNZ
HuskyNZ9mo ago
Yea
Chaika
Chaika9mo ago
yea haven't been too many updates on the dc itself, apparently it was Flexential pdx02 which CF colocates in, rumor mill says the dc lost utility power, and gens failed to fire up, and they got partial power back only after a few hours. I think(?) they have full power now, but guessing other complications are still a factor. That DC as well is the landing for a few submarine cables and other critical infrastructure
HuskyNZ
HuskyNZ9mo ago
This is what i have heard
Hitesh Nalamwar (Outgoer)
when streaming service will go live?
HuskyNZ
HuskyNZ9mo ago
Cloudflare Stream?
Hitesh Nalamwar (Outgoer)
yeah We can't upload video ( it big setback and I am afraid if this can happen in future(
HuskyNZ
HuskyNZ9mo ago
#stream Is the place to go with this it seems its back with degraded perfomece
NicoJuicy
NicoJuicy9mo ago
It's going to be an interesting post-mortem...