R
RunPod3mo ago
EMPZ

Very slow upload speeds from serverless workers

I'm uploading files to Supabase from within the serverless workers and I noticed the process is extremly slow. I understand there's some latency because most workers I'm getting are in Europe and my Supabase instance is on US east, but still, almost 20 seconds to upload a 8MB file is bad. I've check it's not a Supabase issue as I'm based in Europe and my upload speeds are just fine.
11 Replies
EMPZ
EMPZOP3mo ago
I'm experiencing the same issue with downloading. Almost 2 minutes to download a file!!
agentpietrucha
agentpietrucha3mo ago
How are you downloading the files? I am downloading & uploading images using presigned urls with just requests and I am getting around 2 secs for both
EMPZ
EMPZOP3mo ago
Same, using pre-signed URLs. I understand there might be networks slowdowns from time to time and that there's an extra latency between EU and US/CA, but 2 minutes for a 8 MB file is not an acceptable slowdown. Especially because I'm paying GPU time for this. Is there some kind of storage I can upload the files I need to process beforehand so that when the worker runs, can retrieve them quickly? Can volumes be used for this purpose?
Coderik
Coderik3mo ago
In which region in Europe are your workers?
agentpietrucha
agentpietrucha3mo ago
As far as I know, the volumes from runpod aren't the best solution for such case. They work best for storing "static" files like checkpoints or bigger models. I am personally using storj for storing processing results for a small period of time The only idea I am having right now would be to check if your workers have slow network for all requests or only for your buckets. Or maybe your docker image has some issues? I am using "nvidia/cuda:11.7.1-cudnn8-runtime-ubuntu22.04" with no issues
EMPZ
EMPZOP3mo ago
No description
EMPZ
EMPZOP3mo ago
I'm sorry, but I keep getting this replies that tries to pin the cause of the issue to anything except RunPod's network. Of course there's nothing wrong with the Docker image, it's the default nvidia/cuda image. These extremely slow downs are not all the time, it happens some time, but when it happens it's crazy. Again, 2 minutes to transfer a single 8 MB file. I'm not usually this mad when these things happen, but here I'm being charged GPU time just to have a worker transferring files at 65 kB/second...
Coderik
Coderik3mo ago
Try to avoid EUR-IS-1 which is a datacenter in Island. You can measure upload/download speeds using a pod in this region and verify that the network to this datacenter is super slow.
Coderik
Coderik3mo ago
Uncheck EUR-IS-1 and EUR-IS-2 in the Advanced settings
No description
EMPZ
EMPZOP3mo ago
It's not the EUR-IS. It's mostly the EU-RO-1, but if I uncheck it, there's no availability of the 16 GB GPU... Europe-to-US latencies should not cause this kind of slowdowns...
Encyrption
Encyrption3mo ago
1st off this is just my observations and opinion, nothing official! Watching RunPod develop and grow I wouldn't be surprised if at some point 16GB GPU are phased out completely. What I do is just mark my endpoint to use 16GB as priority 1 and 24GB as priority 2. Most of the time, I still get 16GB but if I get a 24GB I just eat the 0.0003 per second difference. Thinking of pointing it out to my customers, i.e. Congrats FREE boost to 24GB!
Want results from more Discord servers?
Add your server