Not using cached worker
I've been running into this problem for several days now. I have a endpoint that runs a forge webui worker with a network volume attached. And as you know forge takes some time to start and only then generates the image. So generally when I send a request to a worker it takes some delay for the start process then generates images.
But recently I've run into an issue where there is already a worker running with webui forge started and ready to accept requests but when I submit a new request it completely starts a new worker, which results in huge delay times. My question is, why isn't it using the already available worker which has forge loaded?
And no, the requests weren't submitted one after the other so there is no reason to start a new worker...
Jobs Stays in In-Progress for forever
Sometimes I never get response when I make a request. It stays in progress and doesn't even show execution time.
How to Get the Progress of the Processing job in serverless ?
When I use status/id, it only return like {delayTime: 873, id: 3e9eb0e4-c11d-4778-8c94-4d045baa99c1-e1, status: IN_PROGRESS, workerId: eluw70apx442ph}, no progress data.
I want progress data just like screenshot on serverless console log。Please tell me how to get it in app client....
Worker Keeps running after idle timeout
Hi!
I have observed that my worker keeps running even there is no request and idle time(60s) has been reached.
Also when I make a new request in such a moment my request fails....
May I deploy template ComfyUI with Flux.1 dev one-click to serverless ?emplate
When I click deploy, I only see 'Deploy GPU Pod' , no serverless .
Incredibly long startup time when running 70b models via vllm
I have been trying to deploy 70b models as a serverless endpoint and observe start up times of almost 1 hour, if the endpoint becomes available at all. The attached screenshot shows an example of an endpoint that deploys
cognitivecomputations/dolphin-2.9.1-llama-3-70b . I find it even weirder that the request ultimately succeeds. Logs and screenshot of the endpoint and template config are attached - if anyone can spot an issue or knows how to deploy 70b models such that they reliably work I would greatly appreciate it.
Some other observations:
- in support, someone told me that I need to manually set the env BASE_PATH=/workspace, which I am now always doing
- I sometimes but not always see this in the logs: AsyncEngineArgs(model='facebook/opt-125m', served_model_name=None, tokenizer='facebook/opt-125m'..., even though I am deploying a completely different model...