No device found for buffer type CPU for async uploads
Trying to deploy a pod using KoboldCpp template. The model downloads and all the layers go onto the GPU (A6000, 70B q4ks, 12288 context), but then it just sits there with this and I can't connect to it
2 Replies
@Henky!!
Currently at work so ill be able to respond better in a few hours but let me take a quick look
I don't yet see anything out of the ordinary it should move past that though. Its possible the context didnt fit and it crashed without runpod showing that part in the file. I recommend testing first at 4K context and upscaling if that works. If it doesnt work I can help better once I am home
Its fine if it takes a minute or two to move past that