echoSplice
Explore posts from serversRRunPod
•Created by fireice on 7/23/2024 in #⚡｜serverless
Why "CUDA out of memory" Today ? Same image to generate portrait, yesterday is ok , today in not.
Thanks for that tipp
47 replies
RRunPod
•Created by fireice on 7/23/2024 in #⚡｜serverless
Why "CUDA out of memory" Today ? Same image to generate portrait, yesterday is ok , today in not.
Ill try this in a new deployment. Just thought it was odd that just this one worker failed
47 replies
RRunPod
•Created by fireice on 7/23/2024 in #⚡｜serverless
Why "CUDA out of memory" Today ? Same image to generate portrait, yesterday is ok , today in not.
How? I dont see a file option in DM
47 replies
RRunPod
•Created by fireice on 7/23/2024 in #⚡｜serverless
Why "CUDA out of memory" Today ? Same image to generate portrait, yesterday is ok , today in not.
I'll send you the start.sh and handler script.
47 replies
RRunPod
•Created by fireice on 7/23/2024 in #⚡｜serverless
Why "CUDA out of memory" Today ? Same image to generate portrait, yesterday is ok , today in not.
This specific worker had a 100% fail rate though.
47 replies
RRunPod
•Created by fireice on 7/23/2024 in #⚡｜serverless
Why "CUDA out of memory" Today ? Same image to generate portrait, yesterday is ok , today in not.
we don have that functionality in our code.
They should all load the very same way
47 replies
RRunPod
•Created by fireice on 7/23/2024 in #⚡｜serverless
Why "CUDA out of memory" Today ? Same image to generate portrait, yesterday is ok , today in not.
yes
47 replies
RRunPod
•Created by fireice on 7/23/2024 in #⚡｜serverless
Why "CUDA out of memory" Today ? Same image to generate portrait, yesterday is ok , today in not.
we run stable diffusion with automatic1111
47 replies
RRunPod
•Created by fireice on 7/23/2024 in #⚡｜serverless
Why "CUDA out of memory" Today ? Same image to generate portrait, yesterday is ok , today in not.
This?
{5 items
"endpointId":"6oe3safoiwidj3"
"workerId":"m07jdb658oetph"
"level":"info"
"message":"Compile with TORCH_USE_CUDA_DSA to enable device-side assertions. "
"dt":"2024-08-03 18:27:11.64919904"
}47 replies
RRunPod
•Created by fireice on 7/23/2024 in #⚡｜serverless
Why "CUDA out of memory" Today ? Same image to generate portrait, yesterday is ok , today in not.
sorry, not sure how I would get a stacktrace. I just downloaded the logs directly from runpod
47 replies
RRunPod
•Created by fireice on 7/23/2024 in #⚡｜serverless
Why "CUDA out of memory" Today ? Same image to generate portrait, yesterday is ok , today in not.
47 replies
RRunPod
•Created by fireice on 7/23/2024 in #⚡｜serverless
Why "CUDA out of memory" Today ? Same image to generate portrait, yesterday is ok , today in not.
I have not switched it back on. But I can give you the logs from the weekend when it happened
47 replies
RRunPod
•Created by fireice on 7/23/2024 in #⚡｜serverless
Why "CUDA out of memory" Today ? Same image to generate portrait, yesterday is ok , today in not.
thats why not all of the generations failed and my other endpoint runs fine
47 replies
RRunPod
•Created by fireice on 7/23/2024 in #⚡｜serverless
Why "CUDA out of memory" Today ? Same image to generate portrait, yesterday is ok , today in not.
I have just realised this only happend on one specific worker:
m07jdb658oetph47 replies
RRunPod
•Created by fireice on 7/23/2024 in #⚡｜serverless
Why "CUDA out of memory" Today ? Same image to generate portrait, yesterday is ok , today in not.
I have a second serverless endpoint running that uses the same template. that one is running fine
47 replies
RRunPod
•Created by fireice on 7/23/2024 in #⚡｜serverless
Why "CUDA out of memory" Today ? Same image to generate portrait, yesterday is ok , today in not.
47 replies
RRunPod
•Created by echoSplice on 2/13/2024 in #⚡｜serverless
Serverless errors in the logs
yes. that was built with runpod==0.9.4
6 replies