Kobold.cpp - Remote tunnel loads before the model, causing confusion (possible off-product issue)
Here's the log piece:
Not sure if it affects smaller text models, I tested official Kobold, seems to have the same issue.
Did they move remote tunnel to async?
Main issue with this - the link shows
everything functions properly once the model loads.
10 Replies
maybe its more because of your app / library that serves your model
if you create cloudflare tunnels in your pod it should work great
We are aware of this, its because we changed how those tunnels work and have a duplicate output by accident
Will be fixed in a future version
But moving the tunnels to async is exactly what it is, we added a feature that allows remote switching between model configs (Not exposed yet on runpod) and for that we need to keep the tunnels in their own process
Oh didn't see it had kobold in it lol nc
Solution
Should be fixed
Fixed, no longer reproducible, thank you.
Knew it! Moving those to async, tho, is really good.
Got reversed for people not using the new mode since they don't need it
If you want KoboldAI support I also recommend https://koboldai.org/discord where we hang out, I see that much sooner
Discord
Join the KoboldAI Discord Server!
This community is dedicated to the usage and development of KoboldAI's software, as well as broader text generation AI. | 12833 members
Or #KoboldCpp - Chatbots - Instruct - Story Writing - Adventure Games - OpenAI/Ollama API - Fast Bootup! since I do check that channel
Joined, ty.