siggmus Posts - Answer Overflow

siggmus

•Created by siggmus on 9/17/2023 in #workers-help

How to make a Worker stream (an OpenAI) response back to the front-end?

I am trying to make a worker return chunks of data to the front-end instead of waiting for the full OpenAI result before responding. Currently, it does not return any data before the full response is received from OpenAI. Any idea of how to solve this? Worker code: (I copied the code from the Cloudflare docs (https://developers.cloudflare.com/workers/examples/openai-sdk-streaming/) and made minor adjustments to the header configuration to avoid CORS errors during development.)

export default {
    async fetch(request, env, ctx) {
        const openai = new OpenAI({
            apiKey: "##-#######"
        })

        const stream = await openai.chat.completions.create({
            model: "gpt-3.5-turbo",
            messages: [{ role: "user", content: "Tell me a story using 900 chars." }],
            stream: true
        })

        let { readable, writable } = new TransformStream()

        let writer = writable.getWriter()
        const textEncoder = new TextEncoder()

        for await (const part of stream) {
            console.log(part.choices[0]?.delta?.content || "")
            writer.write(textEncoder.encode(part.choices[0]?.delta?.content || ""))
        }

        writer.close()

       return new Response(readable, {
            status: 200,
            headers: {
                'Access-Control-Allow-Origin': '*',
                'Access-Control-Allow-Headers': 'Origin, X-Requested-With, Content-Type, Accept',
                'Content-Type': 'text/plain; charset=utf-8'
            }
        })
    }
}

Front-end code:

const onSubmit = async (data) => {
            const response = await fetch('CLOUDFLARE_END_POINT');

            const reader = response.body.getReader();

            while (true) {
                const { done, value } = await reader.read();

                if (done) {
                    break;
                }
                console.log(new TextDecoder().decode(value));
            }
}

4 replies

Gaming

Programming