How to make a Worker stream (an OpenAI) response back to the front-end?
I am trying to make a worker return chunks of data to the front-end instead of waiting for the full OpenAI result before responding. Currently, it does not return any data before the full response is received from OpenAI.
Any idea of how to solve this?
Worker code: (I copied the code from the Cloudflare docs (https://developers.cloudflare.com/workers/examples/openai-sdk-streaming/) and made minor adjustments to the header configuration to avoid CORS errors during development.)
export default {
async fetch(request, env, ctx) {
const openai = new OpenAI({
apiKey: "##-#######"
})
const stream = await openai.chat.completions.create({
model: "gpt-3.5-turbo",
messages: [{ role: "user", content: "Tell me a story using 900 chars." }],
stream: true
})
let { readable, writable } = new TransformStream()
let writer = writable.getWriter()
const textEncoder = new TextEncoder()
for await (const part of stream) {
console.log(part.choices[0]?.delta?.content || "")
writer.write(textEncoder.encode(part.choices[0]?.delta?.content || ""))
}
writer.close()
return new Response(readable, {
status: 200,
headers: {
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Headers': 'Origin, X-Requested-With, Content-Type, Accept',
'Content-Type': 'text/plain; charset=utf-8'
}
})
}
}
Front-end code:
const onSubmit = async (data) => {
const response = await fetch('CLOUDFLARE_END_POINT');
const reader = response.body.getReader();
while (true) {
const { done, value } = await reader.read();
if (done) {
break;
}
console.log(new TextDecoder().decode(value));
}
}Stream OpenAI API Responses · Cloudflare Workers docs
Documentation for Cloudflare Workers, a serverless execution environment that allows you to create entirely new applications or augment existing ones …
3 Replies
have a look at the workers-discussions channel. this has been resolved. issue is related to location of the return response statement
@bkyerv Thank you. It would have taken me some time to figure that out. That solved it.
For others who have the same issue, here is the modified workers code:
const { OpenAI } = require("openai");
export default {
async fetch(request, env, ctx) {
const openai = new OpenAI({
apiKey: "##-#####"
})
// make our request to the OpenAI API
const stream = await openai.chat.completions.create({
model: "gpt-3.5-turbo",
messages: [{ role: "user", content: "Tell me a story using 1000 chars." }],
stream: true
})
const headerCode = {
'Access-Control-Allow-Origin': '*',
'Access-Control-Allow-Headers': 'Origin, X-Requested-With, Content-Type, Accept',
}
// Using our readable and writable to handle streaming data
let { readable, writable } = new TransformStream()
const writer = writable.getWriter();
const encoder = new TextEncoder();
const response = new Response(readable, {
headers: headerCode
});
(async () => {
try {
for await (const chunk of stream) {
const content = chunk['choices'][0].delta.content;
const encodedMessage = encoder.encode(content);
writer.write(encodedMessage);
}
writer.close();
} catch (e) {
console.error(e);
// writer.abort(e);
return new Response('bad bad error', {
headers: headerCode
});
}
})();
return response;
}
}@bkyerv @siggmus do you have the link? I'm facing this issue and couldn't find a solution.
https://discord.com/channels/595317990191398933/1213091192946827284/1213098116224655370