Run Mixtral 8x22B Instruct on vLLM worker
Hello everybody, is it possible to run mixtral 8x22B on vLLM worker i tried to run it on the default configuration with 48 gb GPU A6000, A40 but its taking too long, what are the requirements for running mixtral 8x22B successfully ? this is the model that im trying to run https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1
81 Replies
Sorry im new to using GPU's for LLM models
oh it actually needs a bunch of vram to run
you can try using half for running it in lower vrams
all good
thanks for the reply what do you mean by half
in the quantization part
use any to run it in lower vrams
let me check which GPU would be suitable to run this btw ?
oh wait
the DTYPE i mean*
also this too
try experimenting with those in the env variables of your endpoint
sure
Environment variables | RunPod Documentation
Environment variables configure your vLLM Worker by providing control over model selection, access credentials, and operational parameters necessary for optimal Worker performance.
i think this would also be a good option to set right ? since it will divide the memory
never tried vllm worker yet tbh, sure if you want to try it go ahead
yeah seems like a good option to try
ah i see, would there be any substantial decrease in quality if i ran the model in half memory ?
maybe try browsing around on quant, etc
cool thanks
Because I don't know about them but yeah i think it will if you use the lower dtype
yeah makes sense
@nerdylive looks like mixtral 8x2b requires up to 300 gb of vram, and the highest available gpu is 80gb of vram if it uses half amount of memory which would be 150 gb it should be possible to divide 50gb of vram between 3 workers. idk if thats possible do you know somebody from the team that can help me out here ? my company actually wants to deploy this model for our product.
Wew where did you get that estimate from
Yeah it's a huge model
from the mistral discord
Nah, it's not possible yet to divide them onto 3 workers I think
I c
Pods is possible to use multiple gpu
Or maybe explore accelerate for this ( not sure )
i see let me search, though could somebody from the runpod team confirm this ?
Maybe contact support for that
Like from the website
ah ok, i misunderstood this is a community server
sorry
Yeah there are some staffs here but they are easier to manage support request via website support
It's fine
Yeah i will contact through official channels thanks for all the help appreciate it
how do i mark this post as solved ?
Up to you did you find a way to run that model yet?
Without the worker splitting idea
I'd like to know your updates too haha, I think dont mark it first
the only way i have right now is to use a vm with 300gb of vram but it would be costly and not sure if i can find a vm like that, i opted for runpod because it had cheap pricing and easy deployments
sure i will post updates here
one guy in mistral discord also wanted to split memory in order to run the model between 4x gpus
they suggested vLLM for this which is what runpod workers are using i think
Vllm supports that?
i havent looked into it yet but they suggested it and TGI
try asking which feature is it or how
yeah
Thanks
@nerdylive this is what i got
https://docs.mistral.ai/deployment/self-deployment/vllm/ in this guide they set the tensor parallel size to 4 i wonder if runpod does it as well
vLLM | Mistral AI Large Language Models
vLLM can be deployed using a docker image we provide, or directly from the python package.
Oh I think it's configurable
Check the vllm docs on runpod
let me check
oh this was the option lol
i set it to 3 but still ran out of memory
i used 2 gpu per worker as well actually 80gb
Wow it works?
no i ran out of memory
even with the above config
I'm trying to do the same thing as you right now lol will update if I figure something out
Thanks alot
@Alpay Ariyak mayby you could help here π
Hi, you need at least 2x80Gb GPUs afaik
Hey, yes I used 2x 80 GB GPU per worker with 3 workers but I got an error
torch.cuda ran out of memory while trying to allocateWait what there is 2x 80gb?
I thought it was for 48 gbs only
How did you get that
Oof still need more memory huh, try sending the full logs
yeah i will try soon
i just selected the option 2 gpu per worker and 80gb H100
Oh? I can only do 2 GPUs per worker with 48GB GPUs, not 80GB GPUs. Are you sure?
Unless you're doing a pod instead of serverless
In which case ignore me π
My apologies, you actually need 4x80gb for 8x22B
is that actually possible in serverless
Not with the current limits, no
ah that sucks
alright
btw, what are streams transported in? sse? how do i retrieve it on python and iterate the responses async
With OpenAI compatibility?
No the stream endpoint
default one
{{URL}}/stream/:id
Not sse, regular get request
Will return yielded outputs from the worker since last /stream call
wait so i poll the stream endpoint?
What goal do you have in mind?
Like websockets probably? i was hoping the stream endpoint to be alike
OpenAI compatibility streaming is through SSE
Hey, sorry to hijack the thread, I'm also looking into deploying vLLM on RunPod serverless. The landing page indicates that it should be possible to bring your own container, not pay for any idle time, and have <250ms cold boot. Is this true? It sounds too good to be true.
Oh what about the stream im talking about?
Yes, through flash boot
That one is strictly polled
Oh alright
Does this 250ms cold boot time really include everything? Or does it only contain some things, such that the actual cold boot time might be 30 seconds or something? For example, the time to load LLM weights into memory typically takes more than 10 seconds.
Everything, due to not needing to reload weights
That's just insane if it really works
Haha try it out!
Yeah, reading the docs right now to figure out what is everything i need to do to try it... I currently have a Docker image that spins up a fork of oobabooga web ui, I'm thinking about setting that up for the serverless experiment.
Yeah your actually right, i confused it as 80gb
my bad guys
even with using dtype half ? we need 4x80 gb ?
8x22B = 176B parameters. At 16bit, 2 bytes per parameter, that's 352GB just for the model parameters
At 8bit (1 byte per parameter) it's still 176GB
I could be mistaken around this, I'm not an expert on this for sure
But my understanding is that you can just fit 8x22B on 4x80GB with 8bit quantization
I see yeah that makes sense
i will revisit this in the future
316gb for 16bit according to https://huggingface.co/spaces/Vokturz/can-it-run-llm
Weβre raising the serverless gpu count limits around next week
I believe even up to 10x A40 per worker
WOOOO
on other gpus too?
Yes, 2x of everything at the very least iirc
yay~!
nice this will be useful thanks alot
Hey @Alpay Ariyak just wondering is it really normal for vllm to load big models everytime very sloow
Like every request is 100secs ++
Like this mixtral or the llama 3 70b
Any solutions to make that loading faster?