R2 Access Via Domain is Down?

Howdy! I got a report from some of my users that R2 is no longer serving content. Everything from my custom domain is returning a 200 with 0 bytes (even if the file does not exist). For example: https://cdn.btst.io/test.html
curl -i https://cdn.btst.io/test.html
HTTP/2 200
date: Thu, 11 Jan 2024 02:34:45 GMT
content-length: 0
access-control-allow-origin: *
vary: Origin
access-control-allow-credentials: true
access-control-allow-headers: Content-Type, Vary, Origin, Referer, User-Agent, Authorization
access-control-allow-methods: GET, POST, PUT, DELETE, OPTIONS, PATCH, HEAD
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=8OA78wCnvcRLW8CC5ljmtce%2BUhoPFbwNekVtEOQFJR2pMlYT5PVokesuCbId8U2u05wrqi3%2FkxP3kUy3K92R%2BvEg3w7XGcMk5vQraiHBgoKxbYMbUfBYchbrM4woyQ%3D%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 8439c3330d605cae-RDU
alt-svc: h3=":443"; ma=86400
curl -i https://cdn.btst.io/test.html
HTTP/2 200
date: Thu, 11 Jan 2024 02:34:45 GMT
content-length: 0
access-control-allow-origin: *
vary: Origin
access-control-allow-credentials: true
access-control-allow-headers: Content-Type, Vary, Origin, Referer, User-Agent, Authorization
access-control-allow-methods: GET, POST, PUT, DELETE, OPTIONS, PATCH, HEAD
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=8OA78wCnvcRLW8CC5ljmtce%2BUhoPFbwNekVtEOQFJR2pMlYT5PVokesuCbId8U2u05wrqi3%2FkxP3kUy3K92R%2BvEg3w7XGcMk5vQraiHBgoKxbYMbUfBYchbrM4woyQ%3D%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 8439c3330d605cae-RDU
alt-svc: h3=":443"; ma=86400
I don't know how long this has been happening, but I don't see any evidence reported anywhere? Is this an issue on y'all's side?
No description
23 Replies
cole
coleOP•16mo ago
For what it's worth, workers fetching from R2 seem to be working fine. I would hate to rearchitect during downtime, but that seems like a viable path at this point.
kian
kian•16mo ago
I can't reproduce this with a custom domain, and the fact it also serves a 200 OK on / despite / having never worked on custom domains is weird. What is the CORS config on that bucket? I suspect this isn't actually hitting R2, especially since it's serving CORS headers without me passing an Origin header.
cole
coleOP•16mo ago
I just swapped cdn.btst.io over to a worker, so it's working now. cdn2.btst.io is pointed at the bucket now and still showing the weird behavior
cole
coleOP•16mo ago
No description
cole
coleOP•16mo ago
āœ— curl -i cdn.btst.io/test.html
HTTP/1.1 200 OK
Date: Thu, 11 Jan 2024 03:01:17 GMT
Content-Type: text/html
Content-Length: 25
Connection: keep-alive
ETag: "68219c28daba0ca0e67295f1d71aaf37"
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=x4xcZfVXl5sGxXIIlP3uehMNGl7SfVRPkBu%2BF9WPODYldFIi%2FaGDLfUra4%2B6WjiF6nL%2F3B184A3k9pUj2cacA6nFS1mKa%2BkZ84y8jfpkg8%2Fx8tGRsMbTsn7hwyUn0w%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 8439ea1278115cb1-RDU
alt-svc: h3=":443"; ma=86400

<html>
Success!!
</html>
āœ— curl -i cdn2.btst.io/test.html
HTTP/1.1 200 OK
Date: Thu, 11 Jan 2024 03:01:22 GMT
Content-Length: 0
Connection: keep-alive
Access-Control-Allow-Origin: *
Vary: Origin
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type, Vary, Origin, Referer, User-Agent, Authorization
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS, PATCH, HEAD
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=ioFpioSLw6%2B4yTuBiRiId5XFaKbmupwDEngFurqI00Psx6r77PrA68uWDly8vNZSbBuqTfIZpRsTaoOZttQMCoqP0REN3ytXbH4w1KKsckwtB791dE8qq3QuQ6Z0PGg%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 8439ea2cc8ed5ca1-RDU
alt-svc: h3=":443"; ma=86400
āœ— curl -i cdn.btst.io/test.html
HTTP/1.1 200 OK
Date: Thu, 11 Jan 2024 03:01:17 GMT
Content-Type: text/html
Content-Length: 25
Connection: keep-alive
ETag: "68219c28daba0ca0e67295f1d71aaf37"
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=x4xcZfVXl5sGxXIIlP3uehMNGl7SfVRPkBu%2BF9WPODYldFIi%2FaGDLfUra4%2B6WjiF6nL%2F3B184A3k9pUj2cacA6nFS1mKa%2BkZ84y8jfpkg8%2Fx8tGRsMbTsn7hwyUn0w%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 8439ea1278115cb1-RDU
alt-svc: h3=":443"; ma=86400

<html>
Success!!
</html>
āœ— curl -i cdn2.btst.io/test.html
HTTP/1.1 200 OK
Date: Thu, 11 Jan 2024 03:01:22 GMT
Content-Length: 0
Connection: keep-alive
Access-Control-Allow-Origin: *
Vary: Origin
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type, Vary, Origin, Referer, User-Agent, Authorization
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS, PATCH, HEAD
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=ioFpioSLw6%2B4yTuBiRiId5XFaKbmupwDEngFurqI00Psx6r77PrA68uWDly8vNZSbBuqTfIZpRsTaoOZttQMCoqP0REN3ytXbH4w1KKsckwtB791dE8qq3QuQ6Z0PGg%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 8439ea2cc8ed5ca1-RDU
alt-svc: h3=":443"; ma=86400
Yeah, super weird issue haha. It was working up until tonight though haha
kian
kian•16mo ago
See what https://developers.cloudflare.com/fundamentals/basic-tasks/trace-request/ says, you'll have the trace tool on the sidebar in the dashboard.
cole
coleOP•16mo ago
That's a cool little service!
cole
coleOP•16mo ago
No description
cole
coleOP•16mo ago
Same response for the working one with a worker, FWIW This still seems to be affected. I just realized that my less popular sites have broken R2 bucket CDNs as well šŸ˜ž
kian
kian•16mo ago
@Sid | R2 might be able to help - I don't think the traffic is hitting R2, but I also don't have the ability to figure out why it isn't going to R2.
cole
coleOP•16mo ago
Thanks! That'd be helpful!! Am I asking in the right place? Or should I escalate / push somewhere else / wait for Sid? I am not super confident that I found the proper channel šŸ˜… I'm also quite surprised that nobody else has reported.
kian
kian•16mo ago
I would be surprised if it's affecting anyone else but you. I can't repro it and there's no other reports on the community forums or here You'd probably be better off posting in #r2
cole
coleOP•16mo ago
Ah good to know! I wonder if a newer worker route is matching the cdn urls
cole
coleOP•16mo ago
Ugh. Indeed. I had a worker route added that started trapping requests before they made it to R2 😭 It's so interesting to me the way that worker routes are global across all subdomains.
No description
cole
coleOP•16mo ago
Anyways, thanks for your help @kian !!
kian
kian•16mo ago
*/* would be expected to match everything You might want to scope it down to a more specific hostname
cole
coleOP•16mo ago
Oh yeah, I knocked it down. That originally came from some experiments with the "SaaS custom domain routes" for workers
kian
kian•16mo ago
Unfortunately the trace tool you used earlier doesn't yet tell you if a Worker route was matched
cole
coleOP•16mo ago
You kinda need a wildcard to catch those routes for whatever domains a SaaS user puts in and you want to run your worker on Which basically means you need a dedicated TLD per function to run a SaaS w/ custom domains on. Unless I'm missing something or you add a route for each domain
Chaika
Chaika•16mo ago
You can do *btst.io/*, Service: none to stop it from taking over subdomains for that domain, the more specific one wins docs: https://developers.cloudflare.com/cloudflare-for-platforms/cloudflare-for-saas/start/advanced-settings/worker-as-origin/
Workers as your fallback origin Ā· Cloudflare for Platforms docs
Learn how to use a Worker as the fallback origin for your SaaS zone.
cole
coleOP•16mo ago
Thanks @Chaika ! Is there something like a "not match" blocker, so I can do a * that matches SaaS domains, without matching domains on my TLD? In my testing SaaS domains (i.e. user provided domains), I needed a matching rule defined in order for my worker to serve traffic to them
Chaika
Chaika•16mo ago
Right, like the tutorial there says, you can make one more specific like *.<zonename>.com/* service none along with the */* for SaaS, and the more specific one wins (so service none will apply to your own domain)
cole
coleOP•16mo ago
Ohhh I missed that nuance! That "None" is super helpful, thanks!

Did you find this page helpful?