Henky!!
RRunPod
•Created by Liringlas on 11/3/2024 in #⚡｜serverless
Issue with KoboldCPP - official template
Nice that you got it working, if you want to hang out with the other koboldcpp users https://koboldai.org/discord
24 replies
RRunPod
•Created by Liringlas on 11/3/2024 in #⚡｜serverless
Issue with KoboldCPP - official template
The odd part is all of them were listed as 100GB ram for me so I'd expect that to fit even without the new optimization
24 replies
RRunPod
•Created by Liringlas on 11/3/2024 in #⚡｜serverless
Issue with KoboldCPP - official template
I can't reproduce it anymore so I suspect it was regular ram related, my latest change should make system ram irrelevant
24 replies
RRunPod
•Created by Liringlas on 11/3/2024 in #⚡｜serverless
Issue with KoboldCPP - official template
@nerdylive Is there a way for me to know in hindsight how much ram that instance had? I wonder if its being task killed
24 replies
RRunPod
•Created by Liringlas on 11/3/2024 in #⚡｜serverless
Issue with KoboldCPP - official template
Updated the error message to give that hint in the future
24 replies
RRunPod
•Created by Liringlas on 11/3/2024 in #⚡｜serverless
Issue with KoboldCPP - official template
If you do go for split GPU deleting the image gen model after the fact can help since that adds a couple of gigabytes to the first GPU, runpod does not allow deleting it before making the pod due to a runpod bug
24 replies
RRunPod
•Created by Liringlas on 11/3/2024 in #⚡｜serverless
Issue with KoboldCPP - official template
This one launches for me succesfully on 1xA100 :
24 replies
RRunPod
•Created by Liringlas on 11/3/2024 in #⚡｜serverless
Issue with KoboldCPP - official template
Although that specific upload may also be broken
24 replies
RRunPod
•Created by Liringlas on 11/3/2024 in #⚡｜serverless
Issue with KoboldCPP - official template
But people who try it on 2x48GB have been reporting it dooesn't fit especially if they use Q4_K_M
24 replies
RRunPod
•Created by Liringlas on 11/3/2024 in #⚡｜serverless
Issue with KoboldCPP - official template
That model Q4_K_S I have succesfully tested on an A100
24 replies
RRunPod
•Created by Liringlas on 11/3/2024 in #⚡｜serverless
Issue with KoboldCPP - official template
Or use context that doesn't fit
24 replies
RRunPod
•Created by Liringlas on 11/3/2024 in #⚡｜serverless
Issue with KoboldCPP - official template
The issue is that people try to fit models that don't fit
24 replies
RRunPod
•Created by Liringlas on 11/3/2024 in #⚡｜serverless
Issue with KoboldCPP - official template
Ah since we assisted in our discord but I can help
24 replies
RRunPod
•Created by Netrve on 7/24/2024 in #⚡｜serverless
How does the vLLM template provide an OAI route?
No matter if its a exl2 webserver, kcpp webserver or ollama webserver it would work best that way
8 replies
RRunPod
•Created by Netrve on 7/24/2024 in #⚡｜serverless
How does the vLLM template provide an OAI route?
All we need is a 1:1 mapping of the webserver
8 replies
RRunPod
•Created by Netrve on 7/24/2024 in #⚡｜serverless
How does the vLLM template provide an OAI route?
We all need the same thing
8 replies
RRunPod
•Created by Netrve on 7/24/2024 in #⚡｜serverless
How does the vLLM template provide an OAI route?
Alright because the actual worker part i need would be insanely simplistic
8 replies
RRunPod
•Created by Netrve on 7/24/2024 in #⚡｜serverless
How does the vLLM template provide an OAI route?
+1 making a template for KCPP is pointless without this
8 replies