R
RunPod2mo ago
dhe128

How to get `/stream` serverless endpoint to "stream"?

Example from official documentation: https://docs.runpod.io/sdks/javascript/endpoints#stream
from time import sleep
import runpod


def handler(job):
job_input = job["input"]["prompt"]

for i in job_input:
sleep(1) # sleep for 1 second for effect
yield i


runpod.serverless.start(
{
"handler": handler,
"return_aggregate_stream": True, # Ensures aggregated results are streamed back
}
)
from time import sleep
import runpod


def handler(job):
job_input = job["input"]["prompt"]

for i in job_input:
sleep(1) # sleep for 1 second for effect
yield i


runpod.serverless.start(
{
"handler": handler,
"return_aggregate_stream": True, # Ensures aggregated results are streamed back
}
)
import runpodSdk from "runpod-sdk";

async function main() {
const runpod = runpodSdk(RUNPOD_API_KEY);
const endpoint = runpod.endpoint(ENDPOINT_ID);
const result = await endpoint.run({
input: {
prompt: "Hello, World!",
},
});

console.log(result);

const { id } = result;
for await (const result of endpoint.stream(id)) {
console.log(`${JSON.stringify(result, null, 2)}`);
}
console.log("done streaming");
}
import runpodSdk from "runpod-sdk";

async function main() {
const runpod = runpodSdk(RUNPOD_API_KEY);
const endpoint = runpod.endpoint(ENDPOINT_ID);
const result = await endpoint.run({
input: {
prompt: "Hello, World!",
},
});

console.log(result);

const { id } = result;
for await (const result of endpoint.stream(id)) {
console.log(`${JSON.stringify(result, null, 2)}`);
}
console.log("done streaming");
}
The example suggests that the /stream endpoint should return the entire input, one letter at a time. However, when I try it it only sends the first letter back, followed by the http connection closing. Is /stream intended to be streamed from start to finish, or polled repeatedly?
Endpoints | RunPod Documentation
Learn how to interact with RunPod's endpoints using the JavaScript SDK, including synchronous and asynchronous execution methods, status checks, and job cancellation. Discover how to set timeouts, execute policies, and purge queues.
6 Replies
dhe128
dhe128OP2mo ago
* Looking at js-sdk source: https://github.com/runpod/js-sdk/blob/main/src/index.ts#L203 stream() doesn't pass {responseType: 'stream'} to xior.get so it can't be streaming * Running the runpod UI with network tab open: (https://www.runpod.io/console/serverless/user/endpoint/$endpointId) * It polls /stream every few seconds, not doing streaming.
nerdylive
nerdylive2mo ago
yes it doesn't actually "stream" its mimicking stream like behaviour if you have generator handler (view docs), and it is like pull based queue system so once you hit that, the data got removed and you can poll that many times during the job processing to retrieve new messages
rinarakaki
rinarakaki4w ago
Is it true only for the js client or the serverless runtime entirely?
it doesn't actually "stream"
If so, it really must be documented because to all of us 'stream' means 'chunked transfer encoding', not the general category of usage encompassing polling. I was suspicious why they don't have -N/--no-buffer flag in their curl example for the stream endpoint but now have figured out why.. https://docs.runpod.io/serverless/endpoints/job-operations#stream-results
Job operations | RunPod Documentation
Learn how to use the Runpod Endpoint to manage job operations, including running, checking status, purging queues, and streaming results, with cURL and SDK examples.
nerdylive
nerdylive4w ago
Yes it must use polling to update
dhe128
dhe128OP4w ago
thank you! I got it working with stream via polling :scarlettHeart:
nerdylive
nerdylive4w ago
Your welcome!
Want results from more Discord servers?
Add your server