I m writing here in distress as all of
I'm writing here in distress as all of our infrastructure is down and have been so for the past 19 hours. Apparently we had a Worker that was serving images which is against the TOS (2.8). We were notified about this yesterday (sunday) at around 21.00 CEST. We solved the issue within the hour, and notified support. However, instead of throttling the worker in question, Cloudflare has throttled all traffic (which includes our API) meaning that all payloads above 5kb are slowed down (resulting in response times up to 30 seconds). This effectively means our whole service is down. We're a very early stage startup, however, with +22k monthly active users, so this is very expensive. We have been in contact with support all morning, assuring them that the issue is resolved, however, our service is still down, and no replies have been received in a long time. Anybody here who can poke the right person?
21 Replies
@rita / @krimby perhaps?
Ticket no. #2534438 if that can help in any way.
Unknown Userâ˘3y ago
Message Not Public
Sign In & Join Server To View
great, thanks @rita
Unknown Userâ˘3y ago
Message Not Public
Sign In & Join Server To View
That is too early to tell unfortunately.. It sounds like @rita is on it and it is hard to imagine more capable hands to be in to get to the bottom of that.
Indeed @geg - Rita fixed it extremely fast, so thanks for that. Just in time for our peak traffic hours.
Well, I think to improve these checks, and the whole situation in general, I would recommend:
1. Give a notice prior to closing down the service, 24 hours, 3 days, 1 week, etc.
2. Give that notice during working-hours (and not sunday night đ )
3. Check periodically if the issue has been resolved. We solved the issue within the hour late last night, however, it took 16 hours (and some creative ways to get in contact) for it to be resolved.
It was, to be fair, a rather painful morning. Just happy it's resolved now.
Hmm - that's interesting. However, as can be read here: https://www.cloudflare.com/en-gb/terms/#28-limitation-on-serving-non-html-content it seems like we wasn't allowed to service images?
However, now I notice it does state:
Unless explicitly included as part of a Paid Service purchased by you [...]We did purchase and pay for Cloudflare Workers for a few months.
Pedantically, thatâs only referring to storage offerings imo - not if you can use Workers to serve it from other sources that youâd grab with something like
fetch
They need amending to clarify that
The impression I have is that it's fine - from blogs, social media, posts here, etc
But I'd be too wary of building on that w/o the 'legalese' backing that the terms currently lack. They need an amendment and definitely some clarity, since this topic comes up very often and there's no real guarantee for you to point to.
If it's fine but the Trust & Safety đ¤ says no, that's just a ticking time bomb
Yeah. I'm just saying that as it stands currently, the terms are ambiguous and you run the risk of being hammered by the evil robot.
Both things that need changing, one way or another.
Can I store images on my origin if I'm using IR to serve them? Does that only extend to Workers using IR? What if I'm using 1 Worker to expose R2 publicly and another to fetch(R2Worker, { cf: { image: blah } }, is Mr Robot smart enough to know that I'm using R2 or does it matter at all since I'm paying for the Worker & IR?
There's edge cases where there's no clarification in the terms and I suspect 'someone in sales told me this' isn't going to appease someone's boss when it goes down.
Image Resizing
i.e like https://discord.com/channels/595317990191398933/779390076219686943/1011299280180482278 uses
Do I want to make a cost-effective Images alternative that works for low-quality high-quantity use-cases with Image Resizing/Cache API/R2? Yes.
Would I use it in production with the current terms? Never.
Getting clarity in the terms would benefit everyone - so it's just a waiting game
It's ~3x more expensive than a per-GB pricing model for our use-case right now. My plan was using IR/R2 with the Cache API - or even replicating Cache Reserve by storing the resized variants in R2 as a perma-cache of sorts.@kiannh So you recommend we don't revert back to using Cloudflare Workers to cache our images from our Google Bucket, and wait for a bit?
If you was caching them as well as using Image Resizing, it wouldn't surprise me if the cache usage upset some automated detection.
I am paying for Workers @zeblote so I beleive not
Hmm - well something must be off with the check then, because we are paying:
The automated check for video specifically references the free plan - wouldnât surprise me if non-video throttling impacts everyone
Still not sure if we can re-enable the "proxy" to cache images from our GCS bucket or not. Or if we need to enable the
cf.image property in the request to bypass the throttle.I would say you probably should not. My personal understanding of the terms is that if you want to use cache for non-html content you must be storing the data in KV/DO/R2. But I could be completely wrong 𤡠the only way I would suggest you enable it is given explicit confirmation by someone at cloudflare that your case is allowed via a support ticket.
Unknown Userâ˘3y ago
Message Not Public
Sign In & Join Server To View
Serving images is fine yeah we are talking about cache here specifically
Unknown Userâ˘3y ago
Message Not Public
Sign In & Join Server To View
I don't really think disabling is much a great idea because the user could rely on cache and not have enough servers so disabling could effectively take down their service also (or if they have auto scaling could burn a hole in their pockets so). IMO a better course of action is to contact the user first especially if they are on a paid plan. But I don't have the power to make that how it works so đ¤ˇ
Unknown Userâ˘3y ago
Message Not Public
Sign In & Join Server To View
We were talking specifically about kv and R2 using the cache API in that conversation so that's what the response was to. And if you look at the supplemental terms it specifically excludes workers from where it says non html content.
https://www.cloudflare.com/supplemental-terms/
The Cloudflare Developer Platform consists of the following Services: (i) Cloudflare Workers, a Service that permits developers to deploy and run encapsulated versions of their proprietary software source code (each a âWorkers Scriptâ) on Cloudflareâs edge servers; (ii) Cloudflare Pages, a JAMstack platform for frontend developers to collaborate and deploy websites; and (iii) Workers KV, Durable Objects, and R2, storage offerings used to serve HTML and non-HTML content.
Cloudflare
Supplemental Terms | Cloudflare
This page is for those who are interested in our Supplemental Terms.
I can't tell you there actual intent for what they want to allow cached but from my (non professional) opinion there's a reason why they partitioned the points in that statement I copied from the terms that I believe you have to be using kv/do/R2 to cache non html
And from the post you link we were specifically talking about cloudflare storage offerings so in my opinion it was also about that