Slow network volume
Some people reported, that loading models from network-volumes is very slow compared to baking the model into the image itself.
40 Replies
@Encyrption would you mind sharing your experience / tests on this topic again?
@briefPeach would you mind sharing your experience / tests on this topic?
With identical payloads on identical images with only difference was one had network volume and the other had the models baked into the image. While I would see no discernable difference between executionTime I would consistently see an additional 30 - 60 seconds in delayTime when using network volume. I only tested this in EU-RO.
And all of this was happening a month ago right?
yes
@NERDDISCO When I tried network volume with mine it was EU-RO it wouldnt leave queue
@Karlas this sounds strange, not sure if this is related to the network volume. Did it resolve in the end?
Nope wasnt able to resolve it
I removed network volume and it was back to working fine
@NERDDISCO How much network volume do you think I need for 8b model?
Also it was on EU-SE 1
@Karlas you should be good with around 20 GB, because the summary of all files in https://huggingface.co/meta-llama/Meta-Llama-3.1-8B/tree/main roughly is 18 GB. Maybe this was the issue with your worker? That the size was not large enough?
alright
should i try again with the 8b model with a network volume?
Yeah I would try, because maybe this was the issue that it stuck. As you can totally create situations, when something is breaking, for example if the storage is not big enough.
So if you have some time and energy, I would appreciate it if you could test this again
and can i test on any region?
Would you mind creating a new post, so we can talk about all the things llama 3.1 8B? I want to keep the info here about the network volumes 🙏
Works now
Not getting stuck in queue
perfect!
But again I am not trying with a 70b model like I was before, just with a 8b model
When I used 70B I gave the network volume 150gb