s3 slow upload
I'm currently working on uploading a dataset from S3 to my Pod using cloud sync. The dataset I'm uploading from S3 is about 1TB in size, so I set the volume size to 2TB. However, when I check the progress bar, it shows as follows:
314.897 GIB / 316.270 GIB 100% 13.045 MIB/S ETA 1M47S
The upload seems to be progressing extremely slowly, and I can't estimate how long it will take. Could I be doing something wrong? I'm using 7 H100 GPUs, and the billing is adding up even though I haven't started working on the project yet. I would appreciate any quick help.
6 Replies
https://rclone.org/ may be better if you want to view the progress. If you're using secure cloud, I also recommend using a network volume in future, because then you can attach a cheap pod to it for downloading your data from S3 instead of having to download using the expensive GPU types.
Rclone
Rclone syncs your files to cloud storage: Google Drive, S3, Swift, Dropbox, Google Cloud Storage, Azure, Box and many more.
Thanks for your reply. I also have a questiont that is it possible to upload 1TB size of data from s3 to Pod? Have you ever tried it before? I'm curious it is really possible thing or not.
Its likely possible but will be extremely slow
how usually long does it take?
Impossible to say, too many variables
Thanks for your kind reply!