Cloudflare Developers•2y ago

Has there been any R2 releases within

Has there been any R2 releases within the past week that change the SigV4 side? I don't actually think aws4fetch's presigned URLs work anymore.

35 Replies

Unknown User•2y ago

Message Not Public

kianOP•2y ago

Just tested it - so the issue is when aws4fetch is specifying a custom expiry. Which, is what the docs example does. Doing the equivalent with aws-sdk-js-v3 is just fine - so I suspect there's a discrepancy in what aws4fetch is doing which used to work and no-longer does. It seems like R2 is wanting me to send the header, X-Amz-Expires, in my request to the presigned URL - which of course shouldn't be the case as presigned URLs do everything in the URL with query parameters. But, aws4fetch adds x-amz-expires into the x-amz-signedheaders I'd have to test if aws-sdk-js-v3 does that Okay, yeah - it doesn't. so the tl;dr is... aws4fetch, when providing an expiry, adds it into x-amz-signedheaders aws-sdk-js-v3, when providing an expiry, does not add it into x-amz-signedheaders The question is that if this is an issue with aws4fetch (do not add it into x-amz-signedheaders if signquery is true) or an issue with R2 (should x-amz-signedheaders values be plucked from query params if not in the request headers?) That's something only the R2 team can answer - but I can test it against S3 if you'd prefer to just maintain compatibility with them

import { AwsClient } from "aws4fetch";

export default <ExportedHandler>{
  async fetch() {
    const ACCOUNT_ID = "";
    const ACCESS_KEY_ID = "";
    const SECRET_ACCESS_KEY = "";
    const BUCKET_NAME = "";

    const R2_URL = `https://${ACCOUNT_ID}.r2.cloudflarestorage.com`;

    const client = new AwsClient({
      accessKeyId: ACCESS_KEY_ID,
      secretAccessKey: SECRET_ACCESS_KEY,
    });

    const request = new Request(`${R2_URL}/${BUCKET_NAME}/dog.png`, {
      method: "PUT",
    });

    const signed = await client.sign(request, {
      aws: { signQuery: true },
      headers: {
        "X-Amz-Expires": 3600,
      },
    });

    const response = await fetch(signed.url, {
      method: "PUT",
      body: "123",
      // Uncomment me and it'll work!
      // headers: {
      //   "X-Amz-Expires": "3600",
      // },
    });

    console.log(signed.url);
    console.log(response.status, await response.text());

    return new Response("lol");
  },
};

import { AwsClient } from "aws4fetch";

export default <ExportedHandler>{
  async fetch() {
    const ACCOUNT_ID = "";
    const ACCESS_KEY_ID = "";
    const SECRET_ACCESS_KEY = "";
    const BUCKET_NAME = "";

    const R2_URL = `https://${ACCOUNT_ID}.r2.cloudflarestorage.com`;

    const client = new AwsClient({
      accessKeyId: ACCESS_KEY_ID,
      secretAccessKey: SECRET_ACCESS_KEY,
    });

    const request = new Request(`${R2_URL}/${BUCKET_NAME}/dog.png`, {
      method: "PUT",
    });

    const signed = await client.sign(request, {
      aws: { signQuery: true },
      headers: {
        "X-Amz-Expires": 3600,
      },
    });

    const response = await fetch(signed.url, {
      method: "PUT",
      body: "123",
      // Uncomment me and it'll work!
      // headers: {
      //   "X-Amz-Expires": "3600",
      // },
    });

    console.log(signed.url);
    console.log(response.status, await response.text());

    return new Response("lol");
  },
};

That's the repro

JustinNoel•2y ago

I was having the same problem and switched to the S3 sdk. https://discord.com/channels/595317990191398933/940663374377783388/1066149813919416383

kianOP•2y ago

Your report is what made me test it haha That code used to work perfectly fine little more than a week ago, and is the documented code, so regardless of if it should(n't) have worked it does seem like there was something in R2 that caused it to no-longer be happy If the fix is in R2 or in aws4fetch though - geblobcatshrug

JustinNoel•2y ago

I thought I tried commenting out the X-Amz-Expires and it still failed. However, maybe I'm mistaken. Any pros/cons to using awsfetch vs the S3 SDK?

kianOP•2y ago

Probably bundle size? aws4fetch is extraordinarily small & lightweight dev/publish should give you a Total Upload: output that'll tell you what the uploaded bundle size is My repro minifies down to 2.97 KiB gzip'd I suspect the aws-sdk-js-v3 is... bigger but it's likely more ergonomic since everything is a function made for a specific command - whereas aws4fetch is more or less just SigV4 for Fetch

JustinNoel•2y ago

Dang, that is tiny. My build: • Without S3 SDK: public/_worker.js 403.9kb • With S3: public/_worker.js 1.2mb ⚠️ 😧 I haven't tried publishing this so I might be in trouble.

kianOP•2y ago

Is that the actual output from Wrangler or just using something like stat/ls

JustinNoel•2y ago

That's my build process with Hono:

"build": "run-p build:*",
"build:css": "tailwindcss -i src/tailwind.css -o ./public/assets/tailwind.css",
"build:hono": "esbuild --bundle src/index.tsx --format=esm --outfile=public/_worker.js",

"build": "run-p build:*",
"build:css": "tailwindcss -i src/tailwind.css -o ./public/assets/tailwind.css",
"build:hono": "esbuild --bundle src/index.tsx --format=esm --outfile=public/_worker.js",

However, the public/_worker.js is not minified locally so maybe won't be a problem

kianOP•2y ago

minify should help since it isn't on by default w/ Wrangler and whatnot but also, naturally, don't forget it's gzip that matters

kianOP•2y ago

JustinNoel•2y ago

When gzipped my worker.js goes down to 218K. Thanks for bringing this up. I dropped the ball on it as I needed to get something working and then I was in bed sick for a few days.

Unknown User•2y ago

Message Not Public

kianOP•2y ago

console.log(
    await getSignedUrl(S3, new PutObjectCommand({Bucket: 'sdk-example', Key: 'dog.png'}), { expiresIn: 3600 })
)

console.log(
    await getSignedUrl(S3, new PutObjectCommand({Bucket: 'sdk-example', Key: 'dog.png'}), { expiresIn: 3600 })
)

kianOP•2y ago

Unknown User•2y ago

Message Not Public

kianOP•2y ago

Expires sounds like object lifecycle TTLs expiresIn is X-Amz-Expires

Unknown User•2y ago

Message Not Public

kianOP•2y ago

It populates the param and adds it into SignedHeaders Then it's rejected since, well, presigned URLs don't send any custom request headers The question is just do we need to try it against S3 & see if S3 are plucking the values for SignedHeaders from query params - in which case, R2 should do the same, or if it's a bug in aws4fetch that needs fixing

Unknown User•2y ago

Message Not Public

kianOP•2y ago

So S3 also rejects ameowbongotap

Unknown User•2y ago

Message Not Public

kianOP•2y ago

I'd guess it's a simple if (this.signQuery) { // only add 'host' to signableHeaders } but it's AWS so maybe there's another pitfall

Unknown User•2y ago

Message Not Public

kianOP•2y ago

https://github.com/mhart/aws4fetch/issues/46

GitHub

Providing custom headers when using signQuery results in incorrect ...

Issue When using signQuery: true to create presigned URLs, and also providing the X-Amz-Expires header to control expiry, aws4fetch is adding x-amz-expires into the x-amz-signedheaders parameter. S...

kianOP•2y ago

made a GH issue to track it in any case It might just be a failing of the docs - Michael replied on the GitHub issue and, with a second look, adding the header doesn't actually populate x-amz-expires anyways

kianOP•2y ago

It's done in https://developers.cloudflare.com/r2/data-access/s3-api/presigned-urls/#presigned-url-alternative-with-workers - but that doesn't seem to do anything more than add it into SignedHeaders

kianOP•2y ago

Yeah - this just seems to be the docs throwing me off. A few people have used the code as-is & not reported any issues and I only recently saw Justin said that it wasn't working, so assumed it was a recent change with R2 that upset it. Turns out that code doesn't appear like it ever should have worked

Unknown User•2y ago

Message Not Public

kianOP•2y ago

Made a draft at https://github.com/cloudflare/cloudflare-docs/pull/7417 Need to write the PR description & test it a bit

Unknown User•2y ago

Message Not Public

kianOP•2y ago

Not first time I’ve focused on why something isn’t working rather than stepping back & wondering if it should work that way in the first place - coincidentally the two instances were both R2 related notlikemeow

shocked no-one reported it, but it’s also possible that they were just at the building stage of their apps & never had to test the URL it spits out https://discord.com/channels/595317990191398933/940663374377783388/1067770224507637790 & https://discord.com/channels/595317990191398933/1064338146898214952/1064338146898214952 both used the headers - so it sounded like it was supposed to work Guess not cc @justinnoel - take a look at this PR for the amended example

Unknown User•2y ago

Message Not Public

JustinNoel•2y ago

Thanks so much for pursuing this Kian and @sdnts . This works perfectly now 💥!

Gaming

Programming

Has there been any R2 releases within