Can you now run gemma 3 in the vllm container?
In the serverless, its seems im getting an error, any help on this
18 Replies
can you send the error
I deleted it but it seems bc gemma3 is a new model so transformer is relatively outdated afaik?
hmm did you use vllm? or only transformer?
yeah maybe its a good thing to check first for compatibility
I used the preset vllm, llama 3.2b worked but the new gemma 3 didnt
vLLM needs to publish an update first unfortunately
You can use vLLM directly from the main branch, but that's not super easy if you're using our vLLM template iirc
I think we can update and build our own vllm template from that vllm-worker repo in github easily
Just update the requirement.txt or wherever installs the vllm
Looks like vllm v0.8.0 added gemma3 support will the serverless vllm be updated soon?
usually its delayed, so probably a few days/ weeks late
Hi, i have the same issue , have you resolved it? , if then please help me out with it too
I used ollana
Ollama
Okay
yes i think vllm is updated
already
I deployed an end point to try to call gemma3:4b but nothing is happening when I call it, anybody managed?
Yes it works
just now i've tried for you!
you need access from hf + hf token to access it
use vllm to configure + check the allow remote code options in the config ( in runpod menu when configuring vllm )
second image is the next page
