Pod seems to behave like 'Spot' even though it's configured for 'On Demand'
I'm using the stable diffusion template, and haven't had any issues for a few days. Now image generation is taking up to 3 mins before it even starts to generate. Nothing in the logs suggests any sort of error or unusual behavior. How can I diagnose what's wrong? I don't want to terminate and try a new pod because I'll have to redownload all my models and loras.
7 Replies
How is this behaving like spot? The difference between spot and on-demand is that you can lose your spot pod if someone outbids you on price.
I will click 'generate' and the pod does nothing for 3 minutes. The logs show nothing happening either.
Thats not spot behavior
I see. Do you have any idea what might be wrong?
Maybe the host power limited the GPU, what is the output of
nvidia-smi ?278W / 450W | 8847MiB / 24564MiB |
well it looks like its somehow resolved itself now lol. I ended the pod rather than just restarting. Thank you
So you switched pods?