R
RunPod•5mo ago
Fimbul

Stream using ReadableStream (SSE - Server Sent Events)

How can I stream results out of my runpod serverless worker? Some background: I am NOT using VLLM I am using yield in my handler Hitting the v2/{endpoint_id}/stream/{job_id} endpoint gives me back a JSON containing the partial response chunks and "consumes" the "queue". Am I expected to poll the /stream endpoint? Because if so I'm going to poll it every 50ms since it doesn't appear to be rate limited. What I was expecting was a ReadableStream... did I misconfigure something? Should I mark my generator handler as async? Does setting return_aggregate_stream: True return a ReadableStream under /run?
Solution:
Answering my own question for posterity: No, this is not supported, you have to poll the /stream endpoint, but this is not so bad, because it is blocking, meaning it will wait until a chunk arrives and reply immediately...
Jump to solution
3 Replies
nerdylive
nerdylive•5mo ago
No check general 👌
Solution
Fimbul
Fimbul•5mo ago
Answering my own question for posterity: No, this is not supported, you have to poll the /stream endpoint, but this is not so bad, because it is blocking, meaning it will wait until a chunk arrives and reply immediately
nerdylive
nerdylive•5mo ago
Yay I'll try to write a piece of feedback about this I was looking for that readablestream forgot if that exists 😂
Want results from more Discord servers?
Add your server