RunPod•2w ago

Pod stuck when starting container

Yesterday I updated my serverless endpoint with "New release" button. However when the new request came the worker stuck when trying to start container and sucked the remaining funds from my account. In the logs I see multiple worker exited with exit code 0 errors Probably something wrong with my container, but would be nice if after multiple failed attempts to start container the worker stopped automatically and didn't drain money.

11 Replies

nerdylive•2w ago

maybe its on your program ( not sure ) how does your handler file looks like, or describe how it works maybe

Encyrption•2w ago

Returning 0 is reporting success. If you want it to count as a failure it will need to return a number other than 0. Suggest you open a ticket with RunPod (under help/contact on their site) they are generally pretty good about those kinds of things.

vitalik•2w ago

It is just starting ComfyUI and executing prompts from incoming requests

if __name__ == "__main__":
    wait_for_service(url="http://{}/prompt".format(COMFYUI_ADDRESS))
    print("ComfyUI  is ready. Starting RunPod...")
    runpod.serverless.start({"handler": handler})

Ok, thanks. The weird part is that it kept trying to execute single request for many hours, logs look like this 2024-10-09 06:55:45.855[n5bwvtgs4p6x3t][error]worker exited with exit code 0 2024-10-09 06:55:29.577[n5bwvtgs4p6x3t][error]worker exited with exit code 0 2024-10-09 06:55:13.271[n5bwvtgs4p6x3t][error]worker exited with exit code 0 ... and so on

Encyrption•2w ago

I think I see the problem, You should NOT be querying ComfyUI in main. That should happen in your handler function. In main this is before you have any request. Does your handler function have similar code or does your handler function call another function with similar code as below?

# The top level element "prompt" is required by ComfyUI
data = json.dumps({"prompt": workflow}).encode("utf-8")

req = urllib.request.Request(f"http://{COMFY_HOST}/prompt", data=data)

# The top level element "prompt" is required by ComfyUI
data = json.dumps({"prompt": workflow}).encode("utf-8")

req = urllib.request.Request(f"http://{COMFY_HOST}/prompt", data=data)

vitalik•2w ago

yes, it has exact same code and some other stuff like uploading generated images to s3

Encyrption•2w ago

Are you using this one? https://github.com/blib-la/runpod-worker-comfy

GitHub

GitHub - blib-la/runpod-worker-comfy: ComfyUI as a serverless API o...

ComfyUI as a serverless API on RunPod. Contribute to blib-la/runpod-worker-comfy development by creating an account on GitHub.

Encyrption•2w ago

I use that one and this is my complete main:

# Start the handler only if this script is run directly
if __name__ == "__main__":
    runpod.serverless.start({"handler": handler})

# Start the handler only if this script is run directly
if __name__ == "__main__":
    runpod.serverless.start({"handler": handler})

vitalik•2w ago

nice, i'll have a look at it, thank you

Encyrption•2w ago

wait... that is for Serverless Worker, not pod.

vitalik•2w ago

yes, i'm using serverless

Encyrption•2w ago

🙂 oh ok yeah then that shold work for you

Gaming

Programming

Pod stuck when starting container