Slowish Downloads
I'm trying to setup a pod running ComfyUI for Flux at the moment, and it's going to take 30-40 mins just to download the models with the speed it's running at.
Download speed has been much faster in the past.
Pod ID: v4kbmqzmcyr3yf
32 Replies
yeah should b 3-4x that speed , what command did u use to download ?
the script is just using wget
it was running this provisioning script: https://raw.githubusercontent.com/ai-dock/comfyui/main/config/provisioning/flux.sh
wget --header="Authorization: Bearer <HF_Token>" url
^^ command i use replace <HF_Token> with your hf token
no brackets
Experiencing the same issue again, we transferred from EU-is-1 because it was terribly slow, but now the RO Secure cloud is slow as well. These slowdows are started like a month ago, and we cannot continue using Runpod if this issue persist any longer. We are losing prestige because of this.
2024-09-03T10:02:48Z 79d44721116c Downloading [===========> ] 987.1MB/4.481GB
2024-09-03T10:02:48Z 79d44721116c Downloading [===========> ] 987.6MB/4.481GB
2024-09-03T10:02:48Z 79d44721116c Downloading [===========> ] 988.2MB/4.481GB
2024-09-03T10:02:49Z 79d44721116c Downloading [===========> ] 988.7MB/4.481GB
2024-09-03T10:02:49Z 79d44721116c Downloading [===========> ] 989.3MB/4.481GB
2024-09-03T10:02:49Z 79d44721116c Downloading [===========> ] 989.8MB/4.481GB
Yeah I gave up waiting earlier, not keen on paying for an hour of waiting on downloads. Hopefully something can be fixed
Yep, we are getting calls from paying customers nonstop
with that command i download flux1.dev in about 9mins
Okay you are probably using another datacenter or another machine in the datacenter.
EU-SE-1
normally on a A40
Okay we are experiencing the slowdown in the RO datacenter
if you download it with the command i posted how fast is it ?
We are not even there, currently the image is still pulling
what image ?
Docker image
the pod ?
which template ?
Custom
ahhh
idk then dude , mayb put in a support ticket
takes me about 30-40 tops to restart from scratch when i trash a pod
where are you pulling your docker container from ?
fuck that server is painfully slow
The image was publicly hosted on github, but it was insanely slow so I rehosted it on docker hub, it helped somewhat.
yeah i was going to say put it on docker hub , I just spun up a pod and downloaded the template i normally use and its slow af
might be an idea to change servers
Also slow for serverless image downloading from Docker Hub. On a CPU machine (cpu3c-32-64)
I'm experiencing the same issue downloading from dockerhub on EU-RO-1
We can't use runpod with this type of performance, we need to be able to scale up and down faster than this. If this doesn't get resolved we'll be forced to look for an alternative.
@slack
Escalated To Zendesk
The thread has been escalated to Zendesk!
👌
Create another one 🙂
Aye, I heard ya the first time 😛
https://discord.com/channels/912829806415085598/1280808639778525280
EU-RO-1 region had recently network being worked one that should now make connections faster.
EU-RO-1 is huge demand region so it's higher network usage.
Being able to tell demand per region would be great so we can make a more educated choice when it comes to which region to pick.
Should be able to gleam that info from GraphQL, which provides data about availability.
Where in spec can I find it? It's a bit hard to navigate haha
Was thinking about GpuAvailability https://graphql-spec.runpod.io/#definition-GpuAvailability
That wouldn't show overal demand right? Just how many GPUs are used etc?
Yeah, was thinking could do that query per region but not seeing that option. 😦