R
RunPod•4mo ago
ssssteven

GPU memory usage is at 99% when starting the task.

I started to notice some GPU OOM failure today, and it's specific to this instance: A40 - 44adfw5inhfp98 When the job starts, it says the GPU utilization is at 99%. Did something change on RP?
1 Reply
ssssteven
ssssteven•4mo ago
triple checking my code I don't think my code has changed...
Want results from more Discord servers?
Add your server
More Posts
Should i wait for the worker to pull my imageI have a large image (100 GB), should i wait for worker to pull the image before starting any inferePossible memory leak on ServerlessWe're testing different mistral models (cognitivecomputations/dolphin-2.6-mistral-7b and TheBloke/doare we able to run DinD image for GPU pods?Hi, anyone tried running DinD in GPU pods?Runpod error starting container2024-03-07T14:40:19Z error starting container: Error response from daemon: failed to create task forRunpod SD ComfyUI Template missing??Where did the "Runpod SD ComfyUI" template go? Can anyone help? I've been using it extensively for aDockerless dev and deploy, async handler need to use async ?handler.py in HelloWorld project, there is not 'async' before def handler(job): . But in serverlesSomething broken at 1am UTCSomething was broken at 1am UTC which caused a HUGE spike in my cold start and delay times.Should I use Data Centers or Network Volume when confige serverless endpoint ?My project is an AI portrait app targeting global users. The advantage of using data centers is the