max tokens openAI assistant

Hey,

First and foremost thank you for the amazing tool Baptiste!

Quick question, is there any way to set a max_token in the OpenAI block?

My assistant is using 19k tokens per query which is a bit too expensive…

Baptiste•5/21/24, 10:02 AM

Tracking it here: https://github.com/baptisteArno/typebot.io/issues/1524

GitHub

Support for max tokens option in AI blocks · Issue #1524 · baptiste...

᲼

᲼ ᲼ Paul.᲼ ᲼OP•5/24/24, 6:47 PM

Thank you

(On a similar topic I think that there is an issue with the total tokens in save response)

Baptiste•6/6/24, 7:31 AM

I am not sure about how is helpful here

Baptiste•6/6/24, 7:32 AM

because it could greatly impact the user experience

Baptiste•6/6/24, 7:32 AM

It means the assistant reply could be cut off if max tokens limit is reach

Baptiste•6/6/24, 7:32 AM

I would advise you to instruct the model to not generate long answers instead

᲼

᲼ ᲼ Paul.᲼ ᲼OP•6/6/24, 8:30 AM

Isn’t the max token affecting the input memory instead of the output?
Also conversely, I know it’s possible to make a check for a long message sent by the user and ask them to write a shorter message but is it on your radar to add a limit within the send message block? Otherwise a user has to retype their message if they send a message that is too long

Baptiste•6/6/24, 8:58 AM

https://platform.openai.com/docs/api-reference/chat/create#chat-create-max_tokens

Baptiste•6/6/24, 8:58 AM

The maximum number of tokens that can be generated in the chat completion.

᲼᲼ ᲼ Paul.᲼ ᲼Isn’t the max token affecting the input memory instead of the output? Also conv...

Baptiste•6/6/24, 8:59 AM

Yeah I could add such feature

BBaptiste https://platform.openai.com/docs/api-reference/chat/create#chat-create-max_token...

᲼

᲼ ᲼ Paul.᲼ ᲼OP•6/6/24, 11:18 AM

Got you, my bad!