retrieving queue position for a specific task in RunPod serverless API
Hello, I'm using the RunPod serverless API and I have a question about the following:
Assuming there are 10 jobs in the QUEUE for the serverless setup, is there a way to retrieve, using the RunPod API, the position in the queue for a specific taskId that is waiting? In my app service, I need to fetch the order in which a user's diffusion model will be trained so that I can provide the correct waiting time to the user.
6 Replies
No i don't think there's a way to see position for jobs
If you have 10 jobs in the the QUEUE you should consider increasing the number of max workers you have set. If you set to 30 max workers you won't end up with 10 in queue. It doesn't cost any more to run 10 jobs in parallel (with more workers) than to QUEUE jobs to a single worker.
Yeah some difference only for cold starts and the timeouts at the end of run
We have over 50 inference operations coming in per second. So, even if there are 30 workers, it is not enough.
That's impressive! How many active workers do you use?
Are you using queue delay or request count in your advanced settings? I’ve found that queue delay, with a short delay time, works great for handling bursts of traffic since it adds workers based on how long requests are waiting. On the other hand, request count is better for steady or predictable loads because it scales workers in proportion to the total number of requests.