R
RunPod•5d ago
rmnvc

Troubles with answers

I use MistralAI 7B Instruct model and i use standart settings in serverless and im getting strange answers. I tried different temperament values, but it didn't help. Please tell me how to fix it
No description
No description
1 Reply
3WaD
3WaD•5d ago
Have you tried providing a list of messages instead of a prompt, which I believe is used for standard text completion even in non-openai requests? Many chat-finetuned models might not be the best in that. Chat completion uses conversations like: 
"messages": [
    {"role": "system", "content": "You're helpful assistant"},
    {"role": "user", "content": "How are you?"}
]
"messages": [
    {"role": "system", "content": "You're helpful assistant"},
    {"role": "user", "content": "How are you?"}
]
Output: 
"choices": [{
    "message": {
      "role": "assistant",
      "content": "I am feeling good, how may I assist you today?",
    }
  }]
"choices": [{
    "message": {
      "role": "assistant",
      "content": "I am feeling good, how may I assist you today?",
    }
  }]
Standard text completion looks like: 
"prompt": "How are you? I am"
"prompt": "How are you? I am"
Output: 
"choices": [
    {
      "text": " feeling good today."
    }
  ]
"choices": [
    {
      "text": " feeling good today."
    }
  ]
You can also use openAI compatible endpoint and requests at api.runpod.ai/v2/{RUNPOD_ENDPOINT_ID}/openai/v1/chat/completions.

Did you find this page helpful?