zeeb0t
RRunPod
•Created by zeeb0t on 11/12/2024 in #⚡｜serverless
Mounting network storage at runtime - serverless
I am running my own docker container and at the moment, I’m using the runpod interface to select network storage which then presents at /runpod-volume
This is OK, however, what I am hoping to do (instead) is mount the volume at runtime programmatically.
Is this in anyway possible through libraries or API? Basically I would want to list the available volumes, and where the volume exists within the same region as the container / worker, it will mount it.
I’m wanting to do this as I plan to make a volume in every region and then by not selecting the volume at the serverless create interface, and instead mounting at runtime, it would in theory be able to then use ANY available GPU in all regions, whilst still having access to that regions volume.
If not, I need to create a serverless cluster in every region, and then I may be routing requests to a cluster that has no available GPU at that point in time. It is far from ideal.
11 replies
RRunPod
•Created by zeeb0t on 11/10/2024 in #⚡｜serverless
Rope scaling JSON not working
When I try to use rope scaling, with the JSON that works fine in my own vLLM... it errors out on serverless. I tried setting it to just 'type' also but this produces the same error.
{"factor":4,"original_max_position_embeddings":32768,"rope_type":"yarn"}
Here is the log:
2024-11-10T10:54:29.062414854Z engine.py :114 2024-11-10 10:54:29,062 Error initializing vLLM engine: string indices must be integers
....
2024-11-10T10:54:29.064682309Z rope_type = rope_scaling["type"]
2024-11-10T10:54:29.064683918Z TypeError: string indices must be integers
2 replies