octopus Comments - Answer Overflow

octopus

Posts Comments

RRunPod

•Created by octopus on 6/11/2024 in #⚡｜serverless

Cannot run Cmdr+ on serverless, CohereForCausalLM not supported

8 replies

RRunPod

•Created by octopus on 6/11/2024 in #⚡｜serverless

Cannot run Cmdr+ on serverless, CohereForCausalLM not supported

this is the model we tried: https://huggingface.co/alpindale/c4ai-command-r-plus-GPTQ

8 replies

RRunPod

•Created by octopus on 6/11/2024 in #⚡｜serverless

Cannot run Cmdr+ on serverless, CohereForCausalLM not supported

tried that this is the error we get:

return future.result()/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 370, in _load_chat_template
2024-06-12T04:16:52.930112985Z [rank0]:     with open(chat_template, "r") as f:
2024-06-12T04:16:52.930128535Z [rank0]: TypeError: expected str, bytes or os.PathLike object, not dict

return future.result()/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 370, in _load_chat_template
2024-06-12T04:16:52.930112985Z [rank0]:     with open(chat_template, "r") as f:
2024-06-12T04:16:52.930128535Z [rank0]: TypeError: expected str, bytes or os.PathLike object, not dict

8 replies

RRunPod

•Created by octopus on 6/10/2024 in #⚡｜serverless

What quantization for Cmdr+ using vLLM worker?

@digigoblin can I use the original CohereForAI/c4ai-command-r-plus then? what parameter values should I input and vRAM GPU is needed to run it? alternately I tried this alpindale/c4ai-command-r-plus-GPTQ but it seems to give some error saying ' CohereForCausalLM is not supported'

12 replies

RRunPod

•Created by octopus on 6/10/2024 in #⚡｜serverless

What quantization for Cmdr+ using vLLM worker?

@aikitoria said here that vllm was supporting cmdr+ https://discord.com/channels/912829806415085598/948767517332107274/1230643876763537478

12 replies

RRunPod

•Created by octopus on 6/10/2024 in #⚡｜serverless

What quantization for Cmdr+ using vLLM worker?

@aikitoria

12 replies

RRunPod

•Created by octopus on 2/29/2024 in #⚡｜serverless

Serverless calculating capacity & ideal request count vs. queue delay values

@flash-singh any idea?

4 replies

RRunPod

•Created by ashleyk on 2/26/2024 in #⚡｜serverless

Unacceptably high failed jobs suddenly

Gotta give @ashleyk a job at this point, he helps everyone

46 replies

RRunPod

•Created by octopus on 2/26/2024 in #⚡｜serverless