thanatos121.
RRunPod
•Created by Casper. on 2/5/2024 in #⚡｜serverless
SGLang worker (similar to worker-vllm)
@Alpay Ariyak The vLLM support does not support LLaVa Next 1.6 as it doesn't support the multiple image sizes. SGlang is still the only platform that is officially supported
11 replies
RRunPod
•Created by Casper. on 2/5/2024 in #⚡｜serverless
SGLang worker (similar to worker-vllm)
For reference, it does look like there might be an existing solution with "LMscript" but I am unfamiliar with it so not sure how well it will work. An official worker sanctioned by runpod would be preferable in my opinion: https://github.com/sgl-project/sglang/issues/274
11 replies
RRunPod
•Created by Casper. on 2/5/2024 in #⚡｜serverless
SGLang worker (similar to worker-vllm)
@Alpay Ariyak This would be a top priority for me as well. Currently, LLaVa multimodal models aren't supported by vLLM or TGI so SGLang seems like the only easy way to deploy. I will hack something together in the meantime but having a natively supported worker would be appreciated.
11 replies