Jobs Stays in In-Progress for forever
Sometimes I never get response when I make a request. It stays in progress and doesn't even show execution time.
How to Get the Progress of the Processing job in serverless ?
When I use status/id, it only return like {delayTime: 873, id: 3e9eb0e4-c11d-4778-8c94-4d045baa99c1-e1, status: IN_PROGRESS, workerId: eluw70apx442ph}, no progress data.
I want progress data just like screenshot on serverless console log。Please tell me how to get it in app client....
Worker Keeps running after idle timeout
Hi!
I have observed that my worker keeps running even there is no request and idle time(60s) has been reached.
Also when I make a new request in such a moment my request fails....
May I deploy template ComfyUI with Flux.1 dev one-click to serverless ?emplate
When I click deploy, I only see 'Deploy GPU Pod' , no serverless .
Incredibly long startup time when running 70b models via vllm
I have been trying to deploy 70b models as a serverless endpoint and observe start up times of almost 1 hour, if the endpoint becomes available at all. The attached screenshot shows an example of an endpoint that deploys
cognitivecomputations/dolphin-2.9.1-llama-3-70b . I find it even weirder that the request ultimately succeeds. Logs and screenshot of the endpoint and template config are attached - if anyone can spot an issue or knows how to deploy 70b models such that they reliably work I would greatly appreciate it.
Some other observations:
- in support, someone told me that I need to manually set the env BASE_PATH=/workspace, which I am now always doing
- I sometimes but not always see this in the logs: AsyncEngineArgs(model='facebook/opt-125m', served_model_name=None, tokenizer='facebook/opt-125m'..., even though I am deploying a completely different model...