thanatos121.
Security issue: Attackers Scanning Runpod pods?
It is either that, or there is some internal runpod test happening.
Has anyone else experienced this?
What I am also confused by, is I thought that you had to pass your runpod API key to be able to access these servers but that definitely isn't the case. I can easily run inference on my servers without passing my key with an empty bearer token.
7 replies
RRunPod
•Created by Casper. on 2/5/2024 in #⚡｜serverless
SGLang worker (similar to worker-vllm)
@Alpay Ariyak The vLLM support does not support LLaVa Next 1.6 as it doesn't support the multiple image sizes. SGlang is still the only platform that is officially supported
11 replies
RRunPod
•Created by Casper. on 2/5/2024 in #⚡｜serverless
SGLang worker (similar to worker-vllm)
For reference, it does look like there might be an existing solution with "LMscript" but I am unfamiliar with it so not sure how well it will work. An official worker sanctioned by runpod would be preferable in my opinion: https://github.com/sgl-project/sglang/issues/274
11 replies
RRunPod
•Created by Casper. on 2/5/2024 in #⚡｜serverless
SGLang worker (similar to worker-vllm)
@Alpay Ariyak This would be a top priority for me as well. Currently, LLaVa multimodal models aren't supported by vLLM or TGI so SGLang seems like the only easy way to deploy. I will hack something together in the meantime but having a natively supported worker would be appreciated.
11 replies