Worker logs say this to all requests: API: unreachable.. retrying in 100ms
Logs of one of my workers say "API: unreachable.. retrying in 100ms" to all requests. See screenshot attached.
Then Runpod's API replies the following:
{ "delayTime": 10776,
"executionTime": 10206,
"id": "03fec117-c135-435b-bed1-577883d280d9-e1",
"output": "Error queuing workflow: <urlopen error [Errno 111] Connection refused>",
"status": "COMPLETED"}
The last item in the logs (other than "API: unreachable.. retrying in 100ms") says:
S[runpod-worker-comfy] WebSocket connection closed: no close frame received or sent
After this log the worker returned only "API: unreachable.. retrying in 100ms" and response as seen above.
Any ideas what may have caused this behavior and how to avoid it?
Other workers with the same docker image work fine.
7 Replies
What code do you think caused this?
It says the API is unreachable meaning your image usage might be wrong
I guess the comfyui cant be connected to.
That's sad, but sounds right 😦
Do you think there is a way to handle this error? For example, force restart the worker somehow?
Just set the worker to 0 first
Which image are you using currently? are you making it yourself?
If yes, check your code. If no, check the readme.md on the template/image you're using
Yeah - it's a custom image.
Manually setting max workers to 0 and then back works.
Maybe there is a way to reload workers programmatically?
Oh now its working? I'm not sure what causes it until now btw
Like refresh workers Check on the docs
Or by graphql you can set the endpoint to 0 and back to x amount
Yes - if I set max workers to 0 and then back to some number - the problem goes away for some time.
Will investigate further.
Thanks for the advise about graphql. At least I can reload all workers programmatically.
Hmmm
Alright
Use refresh workers if you want the easy way