Coderik
RRunPod
•Created by kazuph（かずふ）🍙 on 12/4/2024 in #⚡｜serverless
Has anyone experienced issues with serverless /run callbacks since December?
Same here with two clients of mine simultaneously. Status request immediately after the job's completion returns the correct result. But no webhook
88 replies
RRunPod
•Created by EMPZ on 9/12/2024 in #⚡｜serverless
Very slow upload speeds from serverless workers
13 replies
RRunPod
•Created by EMPZ on 9/12/2024 in #⚡｜serverless
Very slow upload speeds from serverless workers
Try to avoid EUR-IS-1 which is a datacenter in Island. You can measure upload/download speeds using a pod in this region and verify that the network to this datacenter is super slow.
13 replies
RRunPod
•Created by Coderik on 9/12/2024 in #⚡｜serverless
TTL for vLLM endpoint
After digging, I think it cannot be done even by modifying the vllm worker's code. I've reached out to the support to clarify.
13 replies
RRunPod
•Created by EMPZ on 9/12/2024 in #⚡｜serverless
Very slow upload speeds from serverless workers
In which region in Europe are your workers?
13 replies
RRunPod
•Created by Coderik on 9/12/2024 in #⚡｜serverless
TTL for vLLM endpoint
Sure, I'm using the runpod's vLLM worker: https://github.com/runpod-workers/worker-vllm
13 replies
RRunPod
•Created by Coderik on 9/12/2024 in #⚡｜serverless
TTL for vLLM endpoint
Yes, I'm getting 400 status and validation errors
13 replies
RRunPod
•Created by Coderik on 9/12/2024 in #⚡｜serverless
TTL for vLLM endpoint
When I use
https://api.runpod.ai/v2/{ID}/openai/v1 endpoint, the OpenAI's input format is enforced, so I cannot pass policy there. Based on the worker-vllm code, it seems that at some moment the (OpenAI-compatible) payload is wrapped in the input field, so that the rest of the scheduling and handling can happen. I assume that the capability to handle TTL is there, I just cannot figure out, how to pass the config. Am I missing something?13 replies
RRunPod
•Created by Coderik on 9/12/2024 in #⚡｜serverless
TTL for vLLM endpoint
There is
policy.ttl parameter for regular tasks (https://docs.runpod.io/serverless/endpoints/send-requests#execution-policies), but not for OpenAI-compatible API powered by vLLM (https://github.com/runpod-workers/worker-vllm).13 replies
RRunPod
•Created by Coderik on 9/12/2024 in #⚡｜serverless
TTL for vLLM endpoint
But this is execution timeout. The time spent waiting in the queue does not count as far as I can tell. What I'd like to achieve is discard a task that was sitting in the queue longer than it's TTL. In my case there is a timeout on the caller's side, so the response from such a task will not be received anyways.
13 replies