worker-vllm list of strings
Hey,
I have fine-tuned model that i want to deploy in serverless. I tried the vLLM prompts approach with list of strings (as attached) on T4 Colab and it works really well - response in 0.5 secs. And here is my question - do i need to create my own worker to post input as a list of strings or you handle this in your vllm-worker? -> https://github.com/runpod-workers/worker-vllm
Thanks for you reply 😉
...
Deploying H2O LLM Studio /w auth using Ngrok
I have been working most of the day to get this container deployed to runpod. Here's the trick though. I included nginx in the mix and am using it as a proxy_pass. This way I can use some sort of auth.
Here is the nginx config.
events {
worker_connections 1024;
}...
Wrong GPUs being assigned
I'm paying for the "7 x H100 80GB SXM5", but when I run nvidia-smi I get that there's actually 7 NVIDIA RTX A4000, a much inferior graphics card. What gives?
