How do I implement text streaming from the openai api

13 Replies

MEE6•6mo ago

Wohooo @haarismian, you just became a Waspeteer level 2!

Hi @haarismian. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave: ---------------------- I'm sorry, but the provided knowledge sources do not contain specific information on how to implement text streaming from the OpenAI API using the Wasp framework. However, you can refer to the OpenAI API documentation for more details on how to use their API for text streaming. In the context of Wasp, you would typically interact with external APIs within your actions or operations. For example, in one of the Discord threads, a user was trying to use the OpenAI API within an action in their Wasp project. They imported the OpenAI module and attempted to create an assistant using openai.beta.assistants.create(). However, they encountered an issue because the 'beta' property did not exist on the imported 'openai' module. This suggests that you would need to correctly import and use the OpenAI module within your Wasp actions or operations. Remember to add the OpenAI module to your dependencies in your main.wasp file. Here is the relevant extract for reference: Discord support thread: openai gpt Assistant api, or embeddings template? How to select version when git cloning? > Answers Part 3 (https://discord.com/channels/686873244791210014/1196842660938399744) For more information on how to define actions and operations in Wasp, you can refer to the Wasp documentation here: Wasp Data Model Operations Overview (https://wasp-lang.dev/docs/data-model/operations/overview) ---------------------- Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: ---------------------- Sorry @haarismian, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist: Perhaps try again in 15mins?

haarismianOP•6mo ago

@martinsos any guidance here would be appreciated. I want to stream the results from open ai to my frontend thanks!

Emil•6mo ago

can you give more context on your setup?.. here's how i would usually do it>>

// after fetch, here's how to consume the stream

 const reader = res.body?.getReader();
    while (true || !stopStreamref.current) {
      const { done, value } = (await reader?.read()) as ReadResult;
      if (done || stopStreamref.current) {
        setOnStream(false);
        break;
      }
      yield new TextDecoder("utf-8").decode(value);
    }

// after fetch, here's how to consume the stream

 const reader = res.body?.getReader();
    while (true || !stopStreamref.current) {
      const { done, value } = (await reader?.read()) as ReadResult;
      if (done || stopStreamref.current) {
        setOnStream(false);
        break;
      }
      yield new TextDecoder("utf-8").decode(value);
    }

and then do for loops on set state until the stream done

haarismianOP•6mo ago

thanks @Emil , so what does that look like on your backend? my struggle is that in actions I don't know how to repeatedly get it to return the stream chunks. do I just put a return in the for await?

haarismianOP•6mo ago

I am basically I guess trying to implement server sent events?

Emil•6mo ago

if you return. you only get once. if you yield. you will return multiple times. try change return to yield

martinsos•6mo ago

Here is an example by @miho on how to do streaming in Wasp, check it out! https://github.com/wasp-lang/wasp/tree/main/examples/streaming

haarismianOP•6mo ago

Thank you very much! @martinsos any way I can make that a post call?

martinsos•6mo ago

@haarismian you should be able to, not sure how it vibes with streaming, but you can make it POST in Wasp, check it out here, the docs on API: https://wasp-lang.dev/docs/advanced/apis

Custom HTTP API Endpoints | Wasp

In Wasp, the default client-server interaction mechanism is through Operations. However, if you need a specific URL method/path, or a specific response, Operations may not be suitable for you. For these cases, you can use an api. Best of all, they should look and feel very familiar.

haarismianOP•6mo ago

thank you @martinsos , wanted to report it did indeed work with streaming. In case you need it for future reference here's what I did 1. Followed the example and created a custom API and middleware but with http request as a post 2. passed the body object I wanted into the usetext stream functon in the react component as so: useTextStream('/api/streaming-test', {stateVars}); 3. updated the use textstream function to accept the state 4. updated fetchstream with the variable Let me know if I can help with knowledge transfer in any way. Thanks for yoru help

martinsos•6mo ago

Thanks @haarismian that is awesome! This alreayd helps, these questions are searcheable by others and also published online so it will enable others to find your solution!

Rcts•4w ago

This was exactly what I was looking for. Thanks The only missing things are types but anyway I'll make it work 🙂

Gaming

Programming

How do I implement text streaming from the openai api