nikolai
RRunPod
•Created by nikolai on 12/15/2024 in #⚡｜serverless
Consistently timing out after 90 seconds
No, wasn't cold starts, it was with hot instances. It definitely was the Cloudflare Proxy.
If anybody is facing this problem in the future: Support suggested switching to an TCP port. Haven't tried it as I'll be moving away from Serverless.
27 replies
RRunPod
•Created by nikolai on 12/16/2024 in #⚡｜serverless
Almost no 48GB Workers available in the EU
Actually EU-SE-1 is the only region that still has some 48 GB's left, but if I'm not mistaken it recently had all A6000's removed. I'm not even able to spin up a new worker there.
2 replies
RRunPod
•Created by nikolai on 12/15/2024 in #⚡｜serverless
Consistently timing out after 90 seconds
Because I require tool calls, it's not possible for me to use the async API, which can be only exposed through the OpenAI endpoint. :/
https://github.com/runpod-workers/worker-vllm/blob/main/src/handler.py#L11
27 replies
RRunPod
•Created by nikolai on 12/15/2024 in #⚡｜serverless
Consistently timing out after 90 seconds
Thanks for the swift response 🙂
Yes, it's the vllm worker. I'm hitting the OpenAI endpoint. In this case, it probably is the Cloudflare Proxy, yes.
Is there any way to circumvent it?
27 replies