Throttled download speed from container registry while still being billed
Hello, Im trying to run a custom image from ghcr on a pod, which ive done many times. Its taking a long time, likely going to be 45 mins total just to download the image. I'm also noticing that im being billed while I wait for this very slow download speed.
Is this intentional? If so it seems like a conflict of interests between runpod and the users.
10 Replies
It might be worth trying a different container registry or switching to a different data center to see if you get better speeds.
What is the region chosen? I encountered a similar problem using EU-RO-1
The network is complex, multiple factors from source to destination, the load on data center links, the location of your image registry and its download speed capabilities, and the routing path between your image registry and our data centers. Since our data centers are located in different countries, the routing path’s congestion and other variables also play a significant role.🥲
I'm getting the same, this "still fetching" while being billed - CA-MTL-1 datacenter.
@yhlong00000 In fact, the speed to download model from huggingface and civitai is acceptable... I wonder, can we pay extra storage fee to hold docker image in the network volume.
How long it stuck like that , try to create a new pod
if its all like that, try reporting this in a ticket
thats kind of like docker registry, currently not possible, but i think they are planning to make something like this
Mb Runpod needs to host their own registry? If you ask me, we would pay an extra per hour just to launch pods faster (or mb buy some premium for an extra XX$). Other providers give such opportunity, and it seems Runpod experiences lack of this feature 😦
Yeah I guess they do, yes runpod hasnt supported something like this I guess but if you wanna try to do your own I guess you can launch a cpu pods in the same region then run your own registry
I'm not sure it going to work well with that method tho
Interesting idea. The user may first start a CPU pod to provide the image and shut it down when the GPU pod is started. (I think Runpod will not be happy with this situation.)
However, can it set a fixed domain name for the CPU pod? If not, then the user needs to check the domain name of the CPU pod and replace the Docker image URL for the GPU pod.
Yes you can set a fixed domain name by using like cloudflare tunnels