Problem with RunPod cuda base image. Jobs stuck in queue forever
Hello, I'm trying to do a request to a serverless endpoint that uses this base image on its Dockerfile
FROM runpod/base:0.4.0-cuda11.8.0
I want the serverside to run the input_fn function when I do the request. This is part of the server side code:
```model = model_fn('/app/src/tapnet/checkpoints/')...Solution:
Hmm yeah I guess python 3.11 is missing from that runpod base image..
Why no gpu in canada data center today?
My network volume is in ca-mtl-1, there is no any gpu now.
Solution:
Hey y'all, we disable the creation of new pods four days before a maintenance to stop further issues (this was not something I was personally aware of until now otherwise it would have been posted in #🚨｜incidents). However, I talked with the team and you should be able to create new pods again, let me know if you're running into any issues.
delay time
I have a serverless worker, which is configured to have 15 max workers. However, I notice that only about three of them are actually usable. My workload is configured to timeout if it takes longer than a minute to process.
The other workers randomly have issues such as timing out when attempting to return job data or completely failing to run and having to be retried on a different worker, leading to a delay/execution time of over 2-3 minutes
Executing 6 different jobs all have very different delay times. Some worker ids are consistenly low delay time but some randomly take forever. Is there anything I can do to lower this randomness? Additionally can I delete/blacklist these workers that perform poorly...
