NERDDISCO
RRunPod
•Created by Thibaud on 8/20/2024 in #⚡｜serverless
SGLang
We are totally still working on this, I made sure that you receive an update via the issue.
120 replies
RRunPod
•Created by naaviii on 8/29/2024 in #⚡｜serverless
Urgent: Issue with Runpod vllm Serverless Endpoint
this one is not available in the UI yet when using quick deploy, so you have to change the docker image yourself in the endpoint
22 replies
RRunPod
•Created by naaviii on 8/29/2024 in #⚡｜serverless
Urgent: Issue with Runpod vllm Serverless Endpoint
@naaviii we have created a fix, can you please check the latest version:
runpod/worker-v1-vllm:v1.3.1dev-cuda12.1.022 replies
RRunPod
•Created by naaviii on 8/29/2024 in #⚡｜serverless
Urgent: Issue with Runpod vllm Serverless Endpoint
@naaviii nope, I'm very sorry for this situation 😦
22 replies
RRunPod
•Created by naaviii on 8/29/2024 in #⚡｜serverless
Urgent: Issue with Runpod vllm Serverless Endpoint
I talked with the team and try to help them resolve the issue
22 replies
RRunPod
•Created by naaviii on 8/29/2024 in #⚡｜serverless
Urgent: Issue with Runpod vllm Serverless Endpoint
I just saw that there is already an issue for the problem: https://github.com/runpod-workers/worker-vllm/issues/104
22 replies
RRunPod
•Created by naaviii on 8/29/2024 in #⚡｜serverless
Urgent: Issue with Runpod vllm Serverless Endpoint
Hi @naaviii, I have no update for you yet, but I will ping you the second I have something.
22 replies
RRunPod
•Created by naaviii on 8/29/2024 in #⚡｜serverless
Urgent: Issue with Runpod vllm Serverless Endpoint
I hope we can get this sorted out quickly
22 replies
RRunPod
•Created by naaviii on 8/29/2024 in #⚡｜serverless
Urgent: Issue with Runpod vllm Serverless Endpoint
you are super welcome!
22 replies
RRunPod
•Created by naaviii on 8/29/2024 in #⚡｜serverless
Urgent: Issue with Runpod vllm Serverless Endpoint
I got the ID via DM, it looks like a problem in the worker-vllm. I asked our team to take a look at this. Will report back once I hear something.
22 replies
RRunPod
•Created by naaviii on 8/29/2024 in #⚡｜serverless
Urgent: Issue with Runpod vllm Serverless Endpoint
can you also please provide the docker image version of the worker-vllm that you are using? And the endpoint id (which is ok to share, as an API key is still needed to access it), but you can also DM me the endpoint ID if you want!
22 replies
RRunPod
•Created by naaviii on 8/29/2024 in #⚡｜serverless
Urgent: Issue with Runpod vllm Serverless Endpoint
@naaviii thanks for reporting this. So you are saying that once you connect a network volume to your vllm endpoint, then
client.completions.create stops working?22 replies
RRunPod
•Created by Thibaud on 8/20/2024 in #⚡｜serverless
SGLang
@Thibaud is still under investigation! Possible first result this week.
120 replies
RRunPod
•Created by Thibaud on 8/20/2024 in #⚡｜serverless
SGLang
that would be really nice indeed.
120 replies
RRunPod
•Created by Thibaud on 8/20/2024 in #⚡｜serverless
SGLang
Thank you very very much!!!
120 replies
RRunPod
•Created by Thibaud on 8/20/2024 in #⚡｜serverless
SGLang
@Thibaud so we were talking about this internally and we are already working on some things to make this happen. Would you mind opening a new issue on the worker-sglang repo with your findings ? Because we would love to keep track on where this came from. And you did a great job on debugging already and we would love to keep you in the loop on that.
120 replies
RRunPod
•Created by Thibaud on 8/20/2024 in #⚡｜serverless
SGLang
Thanks for pointing this out, I will check this out myself and see if we can somehow get around doing this. I remember that I heard something about this at some point, but I need to dig deeper
120 replies