S3 to R2 Migration
Hello @Tanmay Kapoor
I am trying to migrate AWS CloudFront S3 Bucket data to CloudFlare R2 Bucket.
While migrating process I am using AWS bucket name, Access Key and Secret Key
But I'm getting error I don't know why?
If I have invalid credentials then what is the correct way to get right credentials for migrate to R2
Error is :
API Request Failed: PUT /api/v4/accounts/55f6dd6c3ab4261cdae76c27ec96734c/slurper/source/connectivity-precheck (403)
The connection cannot be verified. Check the following and try again.
Please help me in this case.
Thanks
Tushar M
56 Replies
Hey @tusharmoradiya, what tool/sdk are you using? I'm not sure I recognize it
Oh wait
I'm dumb, it's slurper
I am just migrating throuh website https://developers.cloudflare.com/r2/data-migration/super-slurper/
Cloudflare Docs
Super Slurper · Cloudflare R2 docs
Super Slurper allows you to quickly and easily copy objects from other cloud providers to an R2 bucket of your choice.
Yeah, idk why I thought it was something else, sorry!
It's okay. would you please help me what is an issue?
Are you also providing an Access Key for the R2 side?
from where i can get Access Key?
Actually, to confirm, which page are you seeing the 403 error on? In Page #1, 2, or 3?
okay let me send you screenshot
yes this page
Ok, so that means that for whatever reason it isn't able to verify your credentials are valid. If you try to perform a
ListObjects with an AWS SDK/the AWS CLI, does it suceed?I have already used ListObjects in policy
still it's give me error
We are using the AWS CLI
With the same credentials? And no special endpoint?
Ye with the same credentials
Hm... I'm not too sure then. I'm not proficient with how Slurper works, but I'll try to rope in someone who is more familiar when they come back online
Okay, I'm waiting
@Tanmay Kapoor Would you please take care of this?
@Tanmay Kapoor I would be very grateful if you can help me out over the issue! its bit of urgent for us to migrate from S3 to R2.
Hey @tusharmoradiya, I would suggest to wait for the other SME for help. Some of the SMEs are in diff timezones.
Ok, Thanks Let me know once SME is available for help
Actually i don't know rclone and how to work with rclone that's why i am using Slurper
Approx 100GB data
would you please guide me rclone?
Actually i tried rclone but i don't know anwers of that questions that's why i didn't do it because if i do anything wrong it will break data i think
Let me try once again
which option should i select?
i didn't config yet
what should be name?
thay give me 61 options
to select
what whould be here?
they are asking for access_key_id
the id which i am trying to enter while migrating through slurper
now they are asking for region
mumbai
ap south 1
now asking for endpoint
now asking for location constraint
enter mean value write or hit return?
now?
what should i write after :
rclone ls s3:
i have written this now what i do?
hit return?
it's doing some process
like this
i need to wait till process complete?
ok
done ctrl + c
name r2
which option should i select for r2?
same for R2 and S3 ?
ok
asking for credentials
i need to add credentials of R2?
where i can get credentials for R2?
yes asking for accesskey
where i can get accesskey for R2
from where would you please give me location?
what would be region?
for endpoint hit return?
i am asking because i have endpoint
"https://55f6dd6c3a245b4261cdae76546c27ec96khd734c.r2.cloudflarestorage.com"
ok
config done
now what i should do?
rclone ls s2:
i wrote this
getting this error
sorry
rclone ls r2:
getting this
now wait for complete the process
right?
okay after complete should i need to do anything or it is last process?
thank you so much
if i will have any problem during this process. i will ping here
@Space I have one question. if i close terminal or if internet will be gone. process will be terminated or continoue running?
It will stop running, but you should be able to run the command again to get it to continue
will it continue from where i stopped?
Yep
Ok Thanks
rclone is fastest method to migrate the data? or any other method can you suggest?
@Space can i do it using incremental migration?
I have done it in staging environment. Now i want to do it in production environment. now i have one question. if i start copy and in between process some media added in S3 then that media will be copied or not?
I mean i can't upload to r2 from application before copied all media to r2. after shifting completely to r2. i will give the update of application to users. hope you understand my point
after complete migration to r2. i will stop use s3 bucket. till complete i have to use s3 because application is live and live user uploading data on daily basis
Yes, I have done transfer in staging. which we have started yesterday
yes
yes, but my concern is while transfering the data. if new data is coming in s3 then what happend? that data also will transer or not?
it will copy whole data or only copy missed data?
okay
let me do this
they are uploading posts, stories, send media to chat all the things
I have Android and iOS application and i use AWSs3transferutility through the swift package in iOS
Don't worry i have added security. and every users have not their own bucket
We are uploading to aws using poolid
and it's stored in server not in application
Hello
I have shifted all media to production now. I have connected domain for Public access --> Domain is : vnt4r8t7zh9wq.tepnot.com. I can access the media which are shifted S3 to R2 publicly. but when i upload new media and trying to access publicly. i am getting error 404 not found but media is already available in R2.
Error I am getting is :
404 Not Found
Code: NoSuchKey
Message: The specified key does not exist.
Key: uploads/posts/image/C806CBEF-1512-45FA-B005-587F158069F9.jpg
RequestId: XM20S3HGPDJY1WFN
HostId: ZnGOve4/gwOLuJrR0BGP5QjxYMjfh/7ChAqJROfFhquBV94kLqhVrjy9TlbPW7CiIOzo4hpWMxw=
An Error Occurred While Attempting to Retrieve a Custom Error Document
Code: NoSuchKey
Message: The specified key does not exist.
Key: error.html
URL which i am accessing is :
https://vnt4r8t7zh9wq.tepnot.com/uploads/posts/image/C806CBEF-1512-45FA-B005-587F158069F9.jpg
@Space