Proper way to listen stream
If I understood correctly, the only way to get stream updates is making request to stream endpoint like it showen in docs here https://docs.runpod.io/reference/llama2-13b-chat.
for i in range(10):
time.sleep(1)
get_status = requests.get(status_url, headers=headers)
print(get_status.text)
Is there any other way to get hands on updates from async generator handler?
Would be nice to have something like this https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events, like it's done in openai api https://platform.openai.com/docs/api-reference/streaming
Or maybe websockets will do too.
2 Replies
Solution
we dont have SSE support yet, will plan to look at that
for now looping is the only way, you don't need the sleep, the request is blocking itself