Alpay Ariyak
RRunPod
•Created by yarcat on 6/24/2024 in #⚡｜serverless
Stuck in the initialization
Of course! Did that solve it?
79 replies
RRunPod
•Created by yarcat on 6/24/2024 in #⚡｜serverless
Stuck in the initialization
Also all machines are now 12.1+ so you can use that worker
79 replies
RRunPod
•Created by yarcat on 6/24/2024 in #⚡｜serverless
Stuck in the initialization
Container disk size is super small, make it at least 50
79 replies
RRunPod
•Created by nerdylive on 7/1/2024 in #⚡｜serverless
VLLM WORKER ERRROR
Only h100s and L40s support fp8
24 replies
RRunPod
•Created by octopus on 6/25/2024 in #⚡｜serverless
Distributing model across multiple GPUs using vLLM
Yeah that’s a vllm issue, it doesn’t allow 6 or 10
10 replies
RRunPod
•Created by octopus on 6/25/2024 in #⚡｜serverless
Distributing model across multiple GPUs using vLLM
You don't need it, as it's automatically set to the number of GPUs of the worker
10 replies
RRunPod
•Created by shensmobile on 6/13/2024 in #⚡｜serverless
vLLM streaming ends prematurely
Sorry for the delay
45 replies
RRunPod
•Created by shensmobile on 6/13/2024 in #⚡｜serverless
vLLM streaming ends prematurely
This was fixed!
45 replies
RRunPod
•Created by shensmobile on 6/13/2024 in #⚡｜serverless
vLLM streaming ends prematurely
All others are good
45 replies
RRunPod
•Created by shensmobile on 6/13/2024 in #⚡｜serverless
vLLM streaming ends prematurely
This seems isolated to that and US-OR
45 replies
RRunPod
•Created by shensmobile on 6/13/2024 in #⚡｜serverless
vLLM streaming ends prematurely
Is CA-MTL-1 a requirement for you?
45 replies
RRunPod
•Created by shensmobile on 6/13/2024 in #⚡｜serverless
vLLM streaming ends prematurely
Please do for now, I don’t have access atm to the settings
45 replies
RRunPod
•Created by shensmobile on 6/13/2024 in #⚡｜serverless
vLLM streaming ends prematurely
Of course!
45 replies
RRunPod
•Created by shensmobile on 6/13/2024 in #⚡｜serverless
vLLM streaming ends prematurely
The second one, I agree its confusing to tell which is the id haha
45 replies
RRunPod
•Created by shensmobile on 6/13/2024 in #⚡｜serverless
vLLM streaming ends prematurely
And your endpoint id please
45 replies
RRunPod
•Created by shensmobile on 6/13/2024 in #⚡｜serverless
vLLM streaming ends prematurely
Can you share your entire endpoint configuration
45 replies
RRunPod
•Created by shensmobile on 6/13/2024 in #⚡｜serverless
vLLM streaming ends prematurely
we're still looking into this
45 replies
RRunPod
•Created by shensmobile on 6/13/2024 in #⚡｜serverless
vLLM streaming ends prematurely
Okay, that's great to know, so issue is outside of worker
45 replies
RRunPod
•Created by shensmobile on 6/13/2024 in #⚡｜serverless
vLLM streaming ends prematurely
After you send the streaming request and it finishes, can you go to the console and check status of that request, it should show full output from worker, need to see if it’s also cut off there
45 replies
RRunPod
•Created by shensmobile on 6/13/2024 in #⚡｜serverless
vLLM streaming ends prematurely
Were you streaming w openai compatibility or not?
45 replies