My pod is taking forever to download the image
1 x RTX 2000 Ada
6 vCPU 31 GB RAM
Image size is around 18gb
Solution:Jump to solution
@jojje runpod team suggested to keep images in docked registry instead of GitHub.
6 Replies
Same issue here. My images are hosted on gcr.io (github). This has been a problem in the RO region for the past few days I tried. In US regions with H100 etc, the transfer is blazingly (~300-400 MB/s) quick
But in RO it's slow as molasses. Was getting around 2-3 MB/s for 2000-Ada. Switched to 4000-Ada hoping that machine had better connectivity, but zero difference.
At this rate It'll take forever to download the 14GB image (with all the ML libs needed for the workload). The NVIDIA CUDA layers alone are > 10 GB.
Please someone from Runpod, can you shed some light on the docker layer-pull issue in general?
In particular for my issue, Is the network link / path between RO and Github limited in some way? If so, what EU regions would you recommend instead?
PS. I'm using "secure cloud" if that wasn't self-evident.
Solution
@jojje runpod team suggested to keep images in docked registry instead of GitHub.
Thanks. Did they say why?
I'm asking because it's faster than dockerhub for me in the US.
Edit: I followed your suggestion @rahul and pushed a copy to dockerhub. The download from there was almost instant. Thanks for this tidbit!
So it seem there is some serious issue between GH and Runpod, and it would be good to learn why.
But my problem is at least resolved for now. Thx again.
have you opened a ticket too for this?
give them the pod id's
They said they experience slower connection when it's gh registry.
Maybe docker hub has more cdn I'm guessing, and gh only places registry in us?