stuck in "stale worker", new release with new image tag not deploying "latest worker"
I've noticed this problem several times. Sometimes, I pushed a new containter image version, I click "New release" and input my new container with new version tag, but it won't fire "latest workers", instead, it just shows "stale workers 2" and stuck in there forever.
as you can see in the screenshot, i already bumped the image to "weixuanf/runpod-worker-comfy:nslog15" but it still stuck in stale workers using "weixuanf/runpod-worker-comfy:nslog13"
To work around this, if I change the number of max workers in "Edit endpoint", it may fire new "latest workers" being initializing sometimes ( not always)
12 Replies
Now I changed the "max worker" to 1 and it fired the "latest worker" being initializing again. Is it related to the workers quota?
but I only have 2 endpoints, and 1 having only 1 worker, the other I usually set it to 2 or 3 workers ^^
these are my settings, it has flash boot enabled
I think if only 1 workers it cant move some into latest workers after you deploy a new config
i've tried 2 workers as well. Either 1 or 2 Max Workers this problem happens occasionally. I can't deploy new docker image. I have to change the max workers number to something else to force it to refresh image
Why don't you just leave it at 3? 3 is the default for a very good reason.
Setting 1 max worker is the worst idea in history.
Max workers 2, or 3 should be better. It's a weird thing that Runpod doesn't really talk about that 1 just doesn't work well from my 2 cents.
Is weird though that your workers are going stale :/
In my opinion, setting it to 3 by default implies that its not a good idea to set it to anything less than 3.
Ok I’ll try 3 then!
Oh ok I can only have 5 workers in total..and i have more than one endpoints, so thats why i kept the workers number low 🤔
If you have more than one endpoint, its better to upgrade your workers to increase your total worker count. You will run into constant problems trying to run 1 or 2 workers per endpoint.
I also have multiple endpoints but I upgraded the workers to get more. You can upgrade all the way to 30 by yourself, then after that, you can contact RunPod to give you more.
You will run into constant problems trying to run 1 or 2 workers per endpoint.I see! that's interesting. Thanks for the info! 👍 🤔yeah it happens occasionally, but not everytime, kinda weird