RunPod•3mo ago

Chat template error for mistral-7b

2024-10-14T10:19:42.283509829Z --- Starting Serverless Worker |  Version 1.7.0 ---
2024-10-14T10:19:42.283511520Z ERROR 10-14 10:19:42 serving_chat.py:155] Error in applying chat template from request: As of transformers v4.44, default chat template is no longer allowed, so you must provide a chat template if the tokenizer does not define one.
2024-10-14T10:19:42.283814574Z /src/engine.py:183: RuntimeWarning: coroutine 'AsyncMultiModalItemTracker.all_mm_data' was never awaited
2024-10-14T10:19:42.283849707Z   response_generator = await generator_function(request, raw_request=dummy_request)
2024-10-14T10:19:42.283854767Z RuntimeWarning: Enable tracemalloc to get the object allocation traceback

2024-10-14T10:19:42.283509829Z --- Starting Serverless Worker |  Version 1.7.0 ---
2024-10-14T10:19:42.283511520Z ERROR 10-14 10:19:42 serving_chat.py:155] Error in applying chat template from request: As of transformers v4.44, default chat template is no longer allowed, so you must provide a chat template if the tokenizer does not define one.
2024-10-14T10:19:42.283814574Z /src/engine.py:183: RuntimeWarning: coroutine 'AsyncMultiModalItemTracker.all_mm_data' was never awaited
2024-10-14T10:19:42.283849707Z   response_generator = await generator_function(request, raw_request=dummy_request)
2024-10-14T10:19:42.283854767Z RuntimeWarning: Enable tracemalloc to get the object allocation traceback

I am beginner to this things and need some help to resolve this issue. I am using a fine tuned version of mistralai/Mistral-7B-Instruct-v0.3 in a 16 bit float precision and doing the inference using openai interface for which i receiving this error "TypeError: 'NoneType' object is not subscriptable". Additionally I was also facing kv cache 26k and model length 32k mismatch error. I am doubting whether the serverless configurations the i have set is right or wrong. A sample guide would be helpful.

4 Replies

CR-TZM•3mo ago

I have the same issue

yhlong00000•3mo ago

some docs might help: https://github.com/vllm-project/vllm/blob/main/docs/source/serving/distributed_serving.rst https://docs.vllm.ai/en/latest/models/engine_args.html

GitHub

vllm/docs/source/serving/distributed_serving.rst at main · vllm-pro...

A high-throughput and memory-efficient inference and serving engine for LLMs - vllm-project/vllm

DaColorIsBlue•3mo ago

whats the fix?

nerdylive•2mo ago

Maybe you should change some of the arguments when creating the endpoint For the kv cache and model len mismatch

Gaming

Programming

Chat template error for mistral-7b