Very slow upload speeds from serverless workers
I'm uploading files to Supabase from within the serverless workers and I noticed the process is extremly slow. I understand there's some latency because most workers I'm getting are in Europe and my Supabase instance is on US east, but still, almost 20 seconds to upload a 8MB file is bad.
I've check it's not a Supabase issue as I'm based in Europe and my upload speeds are just fine.
11 Replies
I'm experiencing the same issue with downloading. Almost 2 minutes to download a file!!
How are you downloading the files? I am downloading & uploading images using presigned urls with just requests and I am getting around 2 secs for both
Same, using pre-signed URLs.
I understand there might be networks slowdowns from time to time and that there's an extra latency between EU and US/CA, but 2 minutes for a 8 MB file is not an acceptable slowdown. Especially because I'm paying GPU time for this.
Is there some kind of storage I can upload the files I need to process beforehand so that when the worker runs, can retrieve them quickly? Can volumes be used for this purpose?
In which region in Europe are your workers?
As far as I know, the volumes from runpod aren't the best solution for such case. They work best for storing "static" files like checkpoints or bigger models. I am personally using storj for storing processing results for a small period of time
The only idea I am having right now would be to check if your workers have slow network for all requests or only for your buckets. Or maybe your docker image has some issues? I am using "nvidia/cuda:11.7.1-cudnn8-runtime-ubuntu22.04" with no issues
I'm sorry, but I keep getting this replies that tries to pin the cause of the issue to anything except RunPod's network. Of course there's nothing wrong with the Docker image, it's the default nvidia/cuda image. These extremely slow downs are not all the time, it happens some time, but when it happens it's crazy. Again, 2 minutes to transfer a single 8 MB file.
I'm not usually this mad when these things happen, but here I'm being charged GPU time just to have a worker transferring files at 65 kB/second...
Try to avoid EUR-IS-1 which is a datacenter in Island. You can measure upload/download speeds using a pod in this region and verify that the network to this datacenter is super slow.
Uncheck EUR-IS-1 and EUR-IS-2 in the Advanced settings
It's not the EUR-IS. It's mostly the EU-RO-1, but if I uncheck it, there's no availability of the 16 GB GPU...
Europe-to-US latencies should not cause this kind of slowdowns...
1st off this is just my observations and opinion, nothing official! Watching RunPod develop and grow I wouldn't be surprised if at some point 16GB GPU are phased out completely. What I do is just mark my endpoint to use 16GB as priority 1 and 24GB as priority 2. Most of the time, I still get 16GB but if I get a 24GB I just eat the 0.0003 per second difference. Thinking of pointing it out to my customers, i.e. Congrats FREE boost to 24GB!