Transfering 600k images from google cloud storage to cloudflare images
I have just signed up for cloudflares creator bundle for images. I have around 600k of images on gcp which is roughly 1.1tb of data. Whats the most effecient way to transfer these images into cloudflare? Thanks!
2 Replies
Have you had a look at the Sourcing Kit (https://developers.cloudflare.com/images/cloudflare-images/sourcing-kit/) yet? That seems perfect for your usecase!
Actually, nevermind. This only supports Amazon S3
From looking at the docs I think the most efficient way will either be to
- Use URL upload for each image. The upside of this is that you don't need to download it locally, the downside is that you're limited to 1200 per account per 5 minutes. You can get more by making more CF accounts, inviting them to your account and having them make API keys to upload images. Each account will be limited to 1200/5min, so in total it'll be more.
- You could also download them all locally, then upload them using the batch API (https://developers.cloudflare.com/images/cloudflare-images/upload-images/images-batch/). This will probably be quicker, but might be more costly as now you have to pay for bandwidth out of GCP and into your server, instead of just out of GCP.
Neither of them are great options, but since the sourcing kit only supports AWS, I think those are your best bets.
thanks @zegevlier!! ill look into this