fetch behaviour doesn't match curl or fetch in node/browser
Sending the following signed request fails with 403 from a Worker but succeeds in the browser and node repl. (Note the example is time sensitive and will soon time out for all platforms.)
Why? How do I fix it?
The server has a very permissive CORS policy (enabling all domains).
Is there a way to debug which headers Cloudflare actually sends in the
fetch request? The network panel in the Workers code console shows that the correct headers should be sent, but then again there is a big notice saying "Provisional headers are shown.", so it's hard to say for sure.22 Replies
403 likely means it just being blocked as being from a bot or because they're maybe blocking Workers
What's the response body? May give more indication
403 in this case signifies an incorrect signature – at least, I've seen a lot of 403s developing this feature and it's always been an incorrect signature.
I can't seem to access any response body, which again makes me think it's something CORS related (on the Workers end) but it's hard to tell
I've deployed the above Worker to test it and I get the following CURL response from it:
so there is no body in this case (which is different to the normal 403 case from this service – there I'd get some encoded XML with an error message, which is hinted at by the application/xml mime type)
It's possible
fetch is truncating the body because it's a HEAD request
I tested this again with a GET request. There the signature succeeds in Workers the same as it does on other platforms. This means the server is not blocking Workers generally.
It also means I can't test the theory that the error is being truncated only in HEAD requests. Maybe GET request errors are not truncated, but they just work.
Is it possible Workers does not pass all headers for HEAD requests?Nah we don't discriminate on method
It'll just always pass all headers
I wonder how it is possible that only HEAD requests don't work in Workers but if I copy/paste the same fetch statement (including
method) to another platform it just worksInteresting it works for GET and not HEAD
Can you give me any tips on how to debug this further? I've already been at it for about 4 hours and I'm out of ideas.
I can't seem to get any more information out of the Workers dev environment – the Network tab doesn't show me any details for the failing requests. Logging the result there doesn't seem to help either – I get the same nondescript information I pasted above.
error is
though that may be due to it beign expired at this point
You probably get that locally now because the signature has expired
I'll give you a new one that will work for another 10-15mins
fetch("https://flowkey.oss-cn-shanghai.aliyuncs.com/1051-j-cole-mini_2017-03-24_10-03-72.jpg" , { headers: Object.fromEntries([["authorization","OSS4-HMAC-SHA256 Credential=LTAI5tHz4CPSsjRiV9qzEfZb/20241129/cn-shanghai/oss/aliyun_v4_request,Signature=574119a4c1a33ffc373d2d6fa4901f4c923430fa6406704e51ff18f28ba9771d"],["x-oss-content-sha256","UNSIGNED-PAYLOAD"],["x-oss-date","20241129T014427Z"]]), method: "HEAD" })
that works in the node repl for example200 locally, 403 deployed
looking at deployed logs
Hmm so no error being returned from the API this time
since this works locally and that also runs the runtime, I think this may be something within the cf network but i'm not sure what :thonk:
Everything but HEAD works. I tried a PUT request and was able to upload something to the bucket and then GET it again: all via the Worker.
Can you check again that HEAD doesn't have something strange going on regarding the headers it sends?
i can confirm the 403 is coming from the origin, we arent changing 200 -> 403 so that's at least a good sign
local dev works and that also runs the workers runtime
I'm also pretty sure it's coming from the origin due to the content-type which is set to application/xml – that only happens in the case of an API error
that's interesting. is it a "close enough" runtime or is it really identical?
it is literally the runtime, we have an internal layer that just does the cf platform part but all the actual behaviour of apis is all public, open source and used in local/remote
https://github.com/cloudflare/workerd for ref
cool ok
So should I tag this issue differently or are you able to mention somebody who could help debug at the network level?
I tried something else: If I deliberately mess with the string used to sign the request I get a proper error message back:
It's weird that the 403 is happening at all, but it's even weirder that it seems unsure itself why. Maybe something in the network is confused because a GET request is triggering a HEAD request? Not sure why that would trigger 403 though
Ok something really weird is happening. If the requested file is a .jpg (existent or not) I get the 403, but if I deliberately put a non-existent filename with a different extension I get 404 as expected. If I put an existent .txt file I get 200, including via CF
Is cloudflare trying to optimize the image file somehow?
Same issue with .png files, and .mp3 🤔
It's not the mime type. setting a
.txt file to image/jpeg returns a 200 with the "correct" Content-Type, not a 403. So I guess it's the file extension
Still through CF: Uploading a new ".jpg" file with text contents "works" (I can GET it) but HEAD fails with 403.
The same weirdness applies for .mp4 and .js file extensions but not for .py.
I have the feeling this is something to do with some optimization cloudflare is trying to apply to files with certain extensions. It's quite telling that .js is problematic but .py is not for example.
That said: .exe and .zip (I'm running out of extensions to try) suffer from the same issue, which seems more surprising to me.Again though, it's the origin returning 403, optimisation would come after the origin returned
(though, you aren't doing any optimisation anyway)
Right, the origin is returning 403. But only in the CF Worker. So CF must be altering the request to the origin somehow for HEAD requests with certain file extensions 🤔
No file extension is also fine for example. So from the extensions I've tried, only
.py , .txt, and no extension at all have worked.
You can try it yourself easily:
curl -I https://test.flowkey.workers.dev/test
curl -I https://test.flowkey.workers.dev/test.mp2
curl -I https://test.flowkey.workers.dev/test.mp3
curl -I https://test.flowkey.workers.dev/test.mp4
curl -I https://test.flowkey.workers.dev/test.mp5
all of those should return with 404, but the "common" filetypes (mp3, mp4) return with 403 instead.
Again, this does not happen with the same requests being made from another platform (presumably also not from a local cloudflare dev environment)
I have to go to bed. Here's hoping for a miracle or at least some minor enlightenment by the time I get up 👼so this is the error: https://api.aliyun.com/troubleshoot?q=0002-00000201
The request you initiated uses a V4 signature, but the signature provided in the request does not match the signature calculated by OSS.which is interesting
OpenAPI自助诊断-阿里云OpenAPI开发者门户
OpenAPI自助诊断工具是阿里云OpenAPI开发者门户提供的OpenAPI报错排查工具，根据返回的错误信息、requestid智能匹配诊断方案，助力开发者自闭环阿里云OpenAPI报错问题。
Can you share how you found that out? That might help debug
Normally they tell you what the signature should have been based on (headers etc). If something is altering the original request (the signature is already correct for the unaltered request), that message should indicate what has changed
the error code header is set still there's just no error code body for this error
< x-oss-ec: 0002-00000201
This error message means the
Authorization header is being sent (otherwise how would OSS know that it's a v4 signature). But maybe the other required headers (specifically x-oss-date and x-oss-content-sha256) are being swallowed somehow. I am going to try to set up a local server that prints the incoming requests and point the worker to it via a tunnel.
An update: I set up netcat locally with a tunnel and sent fetch requests to it from a Worker in the same way I was sending them to OSS. All headers arrived as they should, regardless of the url path. Even if I took the exact request coming in from the worker, with all CF headers, and sent it to OSS, it worked. So I still don’t know what the issue is.
That said, I also found a built-in OSS feature that does what I need (the custom functionality I was building), so I’ve canned the project I was working on here entirely.
I don’t consider this issue closed (it’s real), but for my part, it has been abandoned. Thanks for your support!