Cloudflare Developers•12mo ago

`CopyObject` S3 compatibility

CopyObject S3 compatibility S3 clients will send the x-amz-tagging-directive which is unsupported as per https://developers.cloudflare.com/r2/api/s3/api/. Has anyone actually been able to use CopyObject with R2 in that case? The S3 API will either use a value of REPLACE or COPY and defaults to COPY and I can definitely see that COPY is unsupported at the very least. Can someone confirm that R2 works without this header being passed through, and that internally instead of defaulting to COPY it's then no-op? If so, it shouldn’t take too long to just at least not throw an error to allow the clients to copy objects. Otherwise, I will have to go extremely low level and send an API request manually to bypass client behaviour of sending this header. It's not really dev friendly for CloudFlare to not adhere to default client implementation when trying to provide S3 compatibility...

24 Replies

Erisa•12mo ago

I don't understand the question. Tags are not supported, so any options or endpoints relating to them will fail or be no-op. Every effort is taken to ensure compatability with library implementations when deciding how to handle these unsupported properties. Are you having any specific errors and if so can you share an example with e.g. the AWS SDK code that reproduces the issue?

ermiyaOP•12mo ago

They’re not, but the SDK will send the header And CloudFlare only does a no-op if it’s not there But SDK default behaviour is to send it

using Amazon.S3;
using Amazon.S3.Model;

var client = new AmazonS3Client(config);

var request = new CopyObjectRequest
{
    SourceBucket = originBucket,
    SourceKey = fileIdString,
    DestinationBucket = destinationBucket,
    DestinationKey = fileIdString
};

await client.CopyObjectAsync(request);

using Amazon.S3;
using Amazon.S3.Model;

var client = new AmazonS3Client(config);

var request = new CopyObjectRequest
{
    SourceBucket = originBucket,
    SourceKey = fileIdString,
    DestinationBucket = destinationBucket,
    DestinationKey = fileIdString
};

await client.CopyObjectAsync(request);

The point is that while it’s not supported, the copy request should still go through Above code yields:

Amazon.S3.AmazonS3Exception: Header 'x-amz-tagging-directive' with value 'COPY' not implemented

Amazon.S3.AmazonS3Exception: Header 'x-amz-tagging-directive' with value 'COPY' not implemented

ermiyaOP•12mo ago

I’ve pointed to the SDK code here: https://stackoverflow.com/a/77633341/4800344

Stack Overflow

Why doesn't CopyObject for CloudFlare R2 not work with the AWS SDK ...

I'm trying to move my object between two buckets. I execute a CopyRequest, then a DeleteRequest. But when I'm trying to copy the object, I get the following error: Amazon.S3.AmazonS3Exception: Hea...

ermiyaOP•12mo ago

The same happens using the JS, Python and Java SDKs too @Erisa | Support Engineer Is that clearer? Happy to provide any more info as needed

Valerii M•11mo ago

Hey @ermiya , did you have a chance to find the solution for your problem? Looks like we have the same problem when using the .NET SDK and R2 CopyObject method.

ermiyaOP•11mo ago

No, problem still persists I’m disappointed that I’ve provided full reproduction steps and nothing from CloudFlare It’d be solved with under 10 lines of code; I’d do a PR if it was open source but alas not

Valerii M•11mo ago

Yeah, I've found that we may try to solve this problem with JS AWS SDK: https://developers.cloudflare.com/r2/examples/aws/custom-header/#set-a-header-for-all-requests-with-aws-sdk-js-v3 , as I understand AWS allow us to execute middleware after the request is formed by their Marshall. But it sounds insanely disproportionate to deploy separate app just to copy objects between buckets... So @Erisa | Support Engineer , @Brandon | Support Engineer , we'd really appreciate your help on guidance how this problem should be solved.

Configure custom headers · Cloudflare R2 docs

Some of R2’s extensions require setting a specific header when using them in the S3 compatible API. For some functionality you may want to set a …

Erisa•11mo ago

Sorry your last ping slipped through the cracks as help on here is best effort, but I've now raised this internally with the R2 team and will see what they can do to help with this

ermiyaOP•11mo ago

Thank you, let me know please if they can’t reproduce or at least have and now is filled as a bug Appreciate it

Valerii M•11mo ago

If you need any help from my side as well – let me know, it's reproducible for our team

Erisa•11mo ago

Yeah I'll keep this thread updated, the reproduction you gave seems pretty clear so don't think we'll have problems but will let you know The R2 team have filed an internal ticket to track getting this fixed cc @Valerii Maslenykov

ermiyaOP•11mo ago

@Erisa | Support Engineer Legend thank you 🎉🎉🎉

Valerii M•11mo ago

@Erisa | Support Engineer , thanks for the update. Do you have any ETA for this ticket to be completed? Knowing the size of organization tickets like these can be postponed for a very long time, so I’d like to know whether you plan to do this in the short term 🙂 Hello, @Erisa | Support Engineer . Just a kind reminder about my question 🙂

Erisa•11mo ago

Hi, sorry, I'm not able to give an ETA on this, but I'll keep here updated if I have more to share. I just know its on their radar now and will be tackled when they can

Valerii M•11mo ago

@Erisa | Support Engineer , thank you

Valerii M•11mo ago

@ermiya , if you're still interested in whether CopyObject works without AWS SDK, the answer is yes. I was able to make it work using CloudFlare's Postman collection https://developers.cloudflare.com/r2/examples/postman/ Planning to use AWS SDK or something like this: https://github.com/tsibelman/aws-signer-v4-dot-net

Postman · Cloudflare R2 docs

Postman is an API platform that makes interacting with APIs easier. This guide will explain how to use Postman to make authenticated R2 requests to …

GitHub

GitHub - tsibelman/aws-signer-v4-dot-net: Sign HttpRequestMessage u...

Sign HttpRequestMessage using AWS Signature v4 using request information and credentials. - GitHub - tsibelman/aws-signer-v4-dot-net: Sign HttpRequestMessage using AWS Signature v4 using request i...

Erisa•11mo ago

Are you also able to try alternate libraries such as https://github.com/minio/minio-dotnet ?

GitHub

GitHub - minio/minio-dotnet: MinIO Client SDK for .NET

MinIO Client SDK for .NET. Contribute to minio/minio-dotnet development by creating an account on GitHub.

Valerii M•11mo ago

@Erisa | Support Engineer , thanks for your suggestion, that may be one of many solutions for our problem. It sounds like it should work properly, as far as in their code they set this header only when tags are provided. https://github.com/minio/minio-dotnet/blob/4c4a8917e758a1cb6de20d1bd4f7e5fcb7f24606/Minio/DataModel/Args/CopyObjectRequestArgs.cs#L125

GitHub

minio-dotnet/Minio/DataModel/Args/CopyObjectRequestArgs.cs at 4c4a8...

MinIO Client SDK for .NET. Contribute to minio/minio-dotnet development by creating an account on GitHub.

Erisa•11mo ago

Yeah

SeanK•11mo ago

I tried Min.io .net client which works if you have no nesting in your copy: var srcArgs = new CopySourceObjectArgs().WithBucket("sourceBucket").WithObject("file.txt"); var copyArgs = new CopyObjectArgs().WithBucket("targetBucket").WithObject("file.txt").WithCopyObjectSource(srcArgs); await minioClient.CopyObjectAsync(copyArgs); But this fails: var srcArgs = new CopySourceObjectArgs().WithBucket("sourceBucket").WithObject("folder/file.txt"); var copyArgs = new CopyObjectArgs().WithBucket("targetBucket").WithObject("folder/file.txt").WithCopyObjectSource(srcArgs); await minioClient.CopyObjectAsync(copyArgs); As an alternative, what I ended up doing is running rclone inside a process: var psi = new ProcessStartInfo(); psi.FileName = "C:\pathToRClone\rclone.exe"; psi.Arguments = "connection:sourceBucket/folder/file.txt connection:targetbucket/folder"; psi.RedirectStandardError = true; psi.UseShellExecute = false; var process = Process.Start(psi); var standardErrorTask = process.StandardError.ReadToEndAsync(); var internalTimeOut = TimeSpan.FromHours(1); var timeoutSignal = new CancellationTokenSource(internalTimeOut); try { await process.WaitForExitAsync(timeoutSignal.Token); var standardError = await standardErrorTask; if (process.ExitCode != 0) { throw new Exception($"Proccess failed. Exit Code {process.ExitCode}. StdError {standardError}"); } } catch { process.Kill(); process.WaitForExit(); throw; }

Erisa•11mo ago

But this fails:

Whats the error you get back?

SeanK•11mo ago

I did another test and it's not the nesting that's the problem, it's the type of file (.ts which I'm sure I could sort out by setting the right content type somewhere). I can transfer a m3u8 text file nested without issues so Min.io is a solid workaround as suggested.

ermiyaOP•10mo ago

Running rclone in a process like this is incredibly dangerous and extremely non practical for serverless environments Other SDK implementations may work but the main AWS SDK should be the one that is used for verify compatibility - it is S3-compatible not Minio-Compatible

Valerii M•10mo ago

@Erisa | Support Engineer , are there any updates on the issue? We decided to deprioritize features where the CopyObject is used, to avoid using multiple SDK clients on our project (cause a lot of stuff was built on AWS SDK already).

Gaming

Programming

`CopyObject` S3 compatibility