anurag Comments - Answer Overflow

anurag

•Created by siggmus on 9/17/2023 in #workers-help

How to make a Worker stream (an OpenAI) response back to the front-end?

@bkyerv @siggmus do you have the link? I'm facing this issue and couldn't find a solution. https://discord.com/channels/595317990191398933/1213091192946827284/1213098116224655370

4 replies

CDCloudflare Developers

•Created by anurag on 3/1/2024 in #workers-and-pages-discussions

Cloudflare Worker is waiting for the

I removed the context overide code and now I have context.waitUntiil the router but I don't understand why waitUntil would work in this case. Because I'm not sending any response at the end of router v1/streaming it's going into the global catch-all route of 404 so the client says 404 but I'm getting the streamed data in the console not the frontend.

.post("/v1/streaming", async (request, { OPENAI_API }, context) => {
    console.log("🚀 - .post - context:", context);
    console.log("🚀 - .post - context:", context.waitUntil);
    let { readable, writable } = new TransformStream();
    const client = new ChatOpenAI({
      openAIApiKey: OPENAI_API,
    });

    const stream = await client.stream(
      "Write a 2 sentence poem about a sunset."
    );
    let writer = writable.getWriter();
    const textEncoder = new TextEncoder();

    context.waitUntil(
      new Promise(async (resolve) => {
        for await (const part of stream) {
          console.log("🚀 - forawait - part:", part);
          if (part.content.length) {
            // Assuming part.content is a string containing the text
            const words = part.content.split(" "); // Split the text into words
            for (const word of words) {
              const sseFormattedData = `data: ${word}\n\n`; // Format each word as SSE
              await writer.write(textEncoder.encode(sseFormattedData));
              console.log(
                "🚀 - forawait - sseFormattedData:",
                sseFormattedData
              );
              await new Promise((r) => setTimeout(r, 1000)); // Wait for 1 second before sending the next word
            }
          }
        }

        writer.close();

        resolve("ok");
      })
    );

  })

.post("/v1/streaming", async (request, { OPENAI_API }, context) => {
    console.log("🚀 - .post - context:", context);
    console.log("🚀 - .post - context:", context.waitUntil);
    let { readable, writable } = new TransformStream();
    const client = new ChatOpenAI({
      openAIApiKey: OPENAI_API,
    });

    const stream = await client.stream(
      "Write a 2 sentence poem about a sunset."
    );
    let writer = writable.getWriter();
    const textEncoder = new TextEncoder();

    context.waitUntil(
      new Promise(async (resolve) => {
        for await (const part of stream) {
          console.log("🚀 - forawait - part:", part);
          if (part.content.length) {
            // Assuming part.content is a string containing the text
            const words = part.content.split(" "); // Split the text into words
            for (const word of words) {
              const sseFormattedData = `data: ${word}\n\n`; // Format each word as SSE
              await writer.write(textEncoder.encode(sseFormattedData));
              console.log(
                "🚀 - forawait - sseFormattedData:",
                sseFormattedData
              );
              await new Promise((r) => setTimeout(r, 1000)); // Wait for 1 second before sending the next word
            }
          }
        }

        writer.close();

        resolve("ok");
      })
    );

  })

3 replies

Gaming

Programming