RunPod•4w ago

Cannot run Cmdr+ on serverless, CohereForCausalLM not supported

I'm getting this error for all Cmdr+ models on serverless:

Error initializing vLLM engine: Model architectures ['CohereForCausalLM'] are not supported for now.

Error initializing vLLM engine: Model architectures ['CohereForCausalLM'] are not supported for now.

Although in vLLM issues we see that CohereForCausalLM is supported

6 Replies

haris•4w ago

cc: @Alpay Ariyak

Alpay Ariyak•4w ago

Try dev image instead of stable

octopus•4w ago

tried that this is the error we get:

return future.result()/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 370, in _load_chat_template
2024-06-12T04:16:52.930112985Z [rank0]:     with open(chat_template, "r") as f:
2024-06-12T04:16:52.930128535Z [rank0]: TypeError: expected str, bytes or os.PathLike object, not dict

return future.result()/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 370, in _load_chat_template
2024-06-12T04:16:52.930112985Z [rank0]:     with open(chat_template, "r") as f:
2024-06-12T04:16:52.930128535Z [rank0]: TypeError: expected str, bytes or os.PathLike object, not dict

this is the model we tried: https://huggingface.co/alpindale/c4ai-command-r-plus-GPTQ

Alpay Ariyak•4w ago

Can you share your env vars

haris•3w ago

cc: @octopus

octopus•3w ago

@Alpay Ariyak

Gaming

Programming

Cannot run Cmdr+ on serverless, CohereForCausalLM not supported