Guide to deploy Llama 405B on Serverless?
Hi, can any experts on Serverless advice on how to deploy Llama 405B on Serverless?
33 Replies
@octopus - you need to attach a network volume to the end point. The volume should have at least 1 TB space to hold the 405 B model (unless you are using quantized models). Then increase the number of workers to match the model gpu requirement (like 10 48 GB GPUs)
I tried several 405 B models in HF but get error related to rope_scaling. Looks like we need to modify it to null and try. To do this I need to download all files and upload again.
does the vllm worker supports this yet?
@nerdylive not sure about this, do we have a document or page that lists vllm's support for a model?
on the docs of vllm, not on runpod
look at the right versions, maybe current vllm is outdated
yes that one is for the latest version
hope vllm-worker now is the latest
looks like it supports
LlamaForCausalLM
Llama 3.1, Llama 3, Llama 2, LLaMA, Yi
meta-llama/Meta-Llama-3.1-405B-Instruct, meta-llama/Meta-Llama-3.1-70B, meta-llama/Meta-Llama-3-70B-Instruct, meta-llama/Llama-2-70b-hf, 01-ai/Yi-34B, etc.
okay, again..
check the current vllm worker's vllm version
i think last time it hasn't been updated yet
I am using runpod/worker-vllm:stable-cuda12.1.0
since I am using serverless I am unable to run any command
and does it working with llama3.1 now?
yeah ofc, leme check the repo one sec
No I get error related to rope_scaling
llama 3.1 's config.json has lots of params under rope_scaling
rope scaling huh, i think you're unable to set that too for current vllm worker version
but the current vllm accepts only two params
this is current's vllm-worker docs:
https://docs.vllm.ai/en/v0.3.2/models/supported_models.html
2024-07-24T04:42:22.063990694Z engine.py :110 2024-07-24 04:42:22,063 Error initializing vLLM engine:
rope_scaling must be a dictionary with two fields, type and factor, got {'factor': 8.0, 'low_freq_factor': 1.0, 'high_freq_factor': 4.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'}