Serverless vllm running but still downloading?
Title says it all and this shouldn't happen
13 Replies
still
anyways, how to replicate: create a vllm endpoint, then create a network storage, then attach ONLY after its created and all ready
Then send requests when 1 worker is ready in the latest workers, to openAI endpoint
@nerdylive
I encountered this issue as well, particularly with endpoints created after the update. Has anyone filed a support ticket or found a solution for this?
@yhlong00000
@nerdylive
Escalated To Zendesk
The thread has been escalated to Zendesk!
i'll try to
Hi there. This is not optimized behavior, but it is expected behavior., Since when you switch to using a network volume, all the running containers need to be removed, and need to be changed to be from the same data center the network volume is in
I'll speak to the team about optimizing this behavior, but this is an edge case do we likely will take some time to do this.
Nonetheless, the downloading new data occurs because we need to change all the Workers to be only from the data center connected to the network volume.
Thank you for response.
I noticed that when this issue occurs, the Docker pull process runs again each time, resulting in delay that take significantly longer than expected.
As a workaround, would switching all the workers to the same data center reduce the frequency of this issue?
Yup that should work, Keep it in the same data centre as uh the network volume
I tried your suggested approach of consolidating the serverless workers in a single data center for one day, but the issue still occurred. It also happens even when we rely solely on container disks instead of a network volume. When the issue occurs, we see a “still fetching image” message.
Is there a behavior where the worker’s Docker image is periodically removed after it remains inactive for a certain period of time?
If this, too, is expected behavior, I apologize .
would you just be able to always use the network volume hre?
Understood. I'll try it while keeping the Active Worker running. Thank you!