Liringlas
RRunPod
•Created by Liringlas on 11/3/2024 in #⚡｜serverless
Issue with KoboldCPP - official template
I tried with two models (103b Midnight Miqu v1.0 and 123b Behemoth v1.1) in Q4 GGUF on a pod with the https://www.runpod.io/console/explore/2peen7lpau template. In both cases the models download successfully (2 files in both cases)
When launching Kobold CPP the following error:
Something possibly went wrong, stalling for 3 minutes before exiting so you can check for errors.
The full logs are included.
- The pod had 2x A40 48GB gpu with the default 125GB temporary container disk, and the default environment variables except for the model address.
The KCPP args (default) should allow 2 GPUs if I understand correctly: --usecublas mmq --gpulayers 999 --contextsize 4096 --multiuser 20 --flashattention --ignoremissing
Thanks a lot!
24 replies