ofer
RRunPod
•Created by ofer on 1/16/2025 in #⚡｜serverless
"Throttled" and re-"Initializing" workers everywhere today
Is there some incident going on with serverless today? I have 30 workers that are all "Throttled", other workers just disapear and others initialize instead of them all the time.
every request that normally takes 10 seconds is taking minutes...
This is true in multiple locations too. Most of my workers ended up in CA-MTL-1 but others in EU-* are displaying the same problems
9 replies
RRunPod
•Created by ofer on 11/10/2024 in #⚡｜serverless
First attempt at serverless endpoint - "Initializing" for a long time
Hi.
New to RunPod, trying to run a serverless endpoint with a worker based on https://github.com/blib-la/runpod-worker-comfy and not able to get it past the "Initializing" status.
There are NO logs anywhere in the console
Here's what I did:
- cloned the github repo and modified the Dockerfile to include the model tensorfiles required
- Built the docker successfully, and pushed it to dockerhub as a public container (at least for now - to avoid authentication problems). On dockerhub it is: "ofertal/runpod-worker-comfy:pixelwave" and is about 9.5G in size
- Created a template that uses the container image "ofertal/runpod-worker-comfy:pixelwave"
- Deployed a serverless GPU Endpoint that uses that template with RTX3090 GPUs, 0 active workers, 4 max workers
- sent a run request to the Endpoint url
Here's what I observed:
- The worker has been "Initializing" for a very long time so far
- There are no logs in the logs tab ("No Logs Found. Try changing the date range or query.")
- I can see that my request was queued on the requests tab, but its status is IN_QUEUE and has been for a long time
- The workers tab has no workers available.
Any idea what I might be doing wrong here?
3 replies