Ryan
RRunPod
•Created by Bj9000 on 1/27/2025 in #⚡｜serverless
Serveless quants
Can anyone help on this?
4 replies
RRunPod
•Created by DEVIL_EGOX on 12/5/2024 in #⚡｜serverless
vllm +openwebui
@Aung Nanda Oo you connection URL in openwebui should be set to this:
https://api.runpod.ai/v2/YourServerlessEndpointIDhere/openai/v1
44 replies
RRunPod
•Created by DEVIL_EGOX on 12/5/2024 in #⚡｜serverless
vllm +openwebui
actually it may be an issue when the GPU im trying to use is unavailable... if openwebui doesnt get a response the side wont load for about a minute until the request times out
44 replies
RRunPod
•Created by DEVIL_EGOX on 12/5/2024 in #⚡｜serverless
vllm +openwebui
actually seems like its not a big issue, its in the running status for milliseconds
44 replies
RRunPod
•Created by DEVIL_EGOX on 12/5/2024 in #⚡｜serverless
vllm +openwebui
only problem is everytime i reload or change pages in my openwebui site it spins up a worker because the endpoint gets triggered when it looks for available models
44 replies
RRunPod
•Created by DEVIL_EGOX on 12/5/2024 in #⚡｜serverless
vllm +openwebui
i got it working
44 replies
RRunPod
•Created by DEVIL_EGOX on 12/5/2024 in #⚡｜serverless
vllm +openwebui
right..... i guess i left out the last part https://api.runpod.ai/v2/{RUNPOD_ENDPOINT_ID}/openai/v1
44 replies
RRunPod
•Created by DEVIL_EGOX on 12/5/2024 in #⚡｜serverless
vllm +openwebui
You got it working or never?
44 replies
RRunPod
•Created by DEVIL_EGOX on 12/5/2024 in #⚡｜serverless
vllm +openwebui
Dang, it's something I really want to be able to do too
44 replies
RRunPod
•Created by DEVIL_EGOX on 12/5/2024 in #⚡｜serverless
vllm +openwebui
@DEVIL_EGOX did you ever get this working?
44 replies