r1
RRunPod
•Created by r1 on 1/3/2024 in #⚡｜serverless
How to retire a worker and retry its job?
flash mentioned it was a limitation of the current scheduler implementation, but i'm unclear if it's changed recently.
yeah, just hitting the serverless endpoint
14 replies
RRunPod
•Created by r1 on 1/3/2024 in #⚡｜serverless
How to retire a worker and retry its job?
so we schedule 1000 jobs all at once, the scheduler spawns workers to take them, and then once those workers refresh, they can't pick up additional jobs since no new jobs have come in.
14 replies
RRunPod
•Created by r1 on 1/3/2024 in #⚡｜serverless
How to retire a worker and retry its job?
it's because our jobs are super bursty, and requests are scheduled when they come in, not dynamically as jobs complete (i dont know if this has changed)
14 replies
RRunPod
•Created by r1 on 1/3/2024 in #⚡｜serverless
How to retire a worker and retry its job?
We used to have the refresh_worker flag, but that seemed to break autoscaling for us (we'd have 3000 jobs in the queue and one worker plugging away on all of them, even though we had several idle workers)
it would definitely fix the issue though
14 replies