nalak Posts - Answer Overflow

nalak

•Created by nalak on 8/20/2024 in #⚡｜serverless

Running a specific Model Revision on Serverless Worker VLLM

How do I specify the model revision on serverless? I was looking through the readme in https://github.com/runpod-workers/worker-vllm and I see I can build a docker image with the revision I want, but is that the only way to go about this? Specifically, I wanna setup this huggingface model: https://huggingface.co/anthracite-org/magnum-v2-123b-exl2 edit: fixed the model link

48 replies