Rope scaling JSON not working
When I try to use rope scaling, with the JSON that works fine in my own vLLM... it errors out on serverless. I tried setting it to just 'type' also but this produces the same error.
{"factor":4,"original_max_position_embeddings":32768,"rope_type":"yarn"}
Here is the log:
2024-11-10T10:54:29.062414854Z engine.py :114 2024-11-10 10:54:29,062 Error initializing vLLM engine: string indices must be integers
....
2024-11-10T10:54:29.064682309Z rope_type = rope_scaling["type"]
2024-11-10T10:54:29.064683918Z TypeError: string indices must be integers
1 Reply
can you also create an issue here
https://github.com/runpod-workers/worker-vllm/issues
GitHub
Issues · runpod-workers/worker-vllm
The RunPod worker template for serving our large language model endpoints. Powered by vLLM. - Issues · runpod-workers/worker-vllm