RunPod•12mo ago

Cuda error: illegal memory access encountered

I'm running stable diffusion webui in api mode on an A4500 (not serverless). Recently, I'm running into this issue a lot more frequently:

2023-12-29T00:37:16.100138705Z *** API error: POST: http://[redacted]/sdapi/v1/txt2img {'error': 'RuntimeError', 'detail': '', 'body': '', 'errors': 'CUDA error: an illegal memory access was encountered\nCUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\n'}
2023-12-29T00:37:16.122794716Z     Traceback (most recent call last):
2023-12-29T00:37:16.122830027Z       File "/workspace/venv/lib/python3.10/site-packages/anyio/streams/memory.py", line 98, in receive
2023-12-29T00:37:16.122834167Z         return self.receive_nowait()
2023-12-29T00:37:16.122837317Z       File "/workspace/venv/lib/python3.10/site-packages/anyio/streams/memory.py", line 93, in receive_nowait
2023-12-29T00:37:16.122840327Z         raise WouldBlock
2023-12-29T00:37:16.122843587Z     anyio.WouldBlock

2023-12-29T00:37:16.100138705Z *** API error: POST: http://[redacted]/sdapi/v1/txt2img {'error': 'RuntimeError', 'detail': '', 'body': '', 'errors': 'CUDA error: an illegal memory access was encountered\nCUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\n'}
2023-12-29T00:37:16.122794716Z     Traceback (most recent call last):
2023-12-29T00:37:16.122830027Z       File "/workspace/venv/lib/python3.10/site-packages/anyio/streams/memory.py", line 98, in receive
2023-12-29T00:37:16.122834167Z         return self.receive_nowait()
2023-12-29T00:37:16.122837317Z       File "/workspace/venv/lib/python3.10/site-packages/anyio/streams/memory.py", line 93, in receive_nowait
2023-12-29T00:37:16.122840327Z         raise WouldBlock
2023-12-29T00:37:16.122843587Z     anyio.WouldBlock

This wasn't happening before. Any idea what may be the issue here?

0 Replies

No replies yetBe the first to reply to this messageJoin

Gaming

Programming

Cuda error: illegal memory access encountered