Serverless vllm - lora
Is there a way to set the lora-modules (for the vllm docker container --lora-modules lora_adapter1=abc/efg) in the Template, or do i need to use the "standard" vllm container for it?
7 Replies
Is there an option in the ui?
there are only the options for enabling it and so on, the option for adding the url is missing
Do you have the docs for the args
In vllm
Docs
For the runpod vllm serverless worker, these are the ENVs, which are available: https://docs.runpod.io/serverless/workers/vllm/environment-variables
Environment variables | RunPod Documentation
Configure your vLLM Worker with environment variables to control model selection, access credentials, and operational parameters for optimal performance. This guide provides a reference for CUDA versions, image tags, and environment variable settings for model-specific configurations.
No the one you wanna use, in vllm's docs
Make an issue in the github of runpod vllm-worker I guess
Thanks Sven for making this
crazy enable_lora flag was added but not the arg to actually add lora adapters
Hoping this gets merged soon
The Pull Request was merged. Does someone know when the new Versions appears for Runpod? (also the vllm worker with the 1.7.4 SDK?)