Aung Nanda Oo
RRunPod
•Created by DEVIL_EGOX on 12/5/2024 in #⚡｜serverless
vllm +openwebui
Thanks I got it!
44 replies
RRunPod
•Created by DEVIL_EGOX on 12/5/2024 in #⚡｜serverless
vllm +openwebui
Guys I am facing issue while using run pod
RUNPOD_CHATBOT_URL = "https://api.runpod.ai/v2/vllm-runpod-endpoint-id/openai/v1"
vllm- should be hard coded since it does not have it anymore ?
response = client.chat.completions.create(
model=model_name,
messages=[{"role": "user", "content": "What is the capital of Germany"}],
temperature=0,
top_p=0.8,
max_tokens=2000,
)
err
44 replies