will_t
RRunPod
•Created by will_t on 12/6/2024 in #⚡｜serverless
New vllm Serverless interface issue
it's like this in my script Runpod_endpoint = "vllm-xxx", previously I only replaced the xxx part
15 replies
RRunPod
•Created by will_t on 12/6/2024 in #⚡｜serverless
New vllm Serverless interface issue
15 replies
RRunPod
•Created by will_t on 12/6/2024 in #⚡｜serverless
New vllm Serverless interface issue
havent tried other models with different prompt format yet like mistral
15 replies
RRunPod
•Created by will_t on 12/6/2024 in #⚡｜serverless
New vllm Serverless interface issue
yeah, this prompt format only works for llama models, which is what I'm currently using
15 replies
RRunPod
•Created by will_t on 12/6/2024 in #⚡｜serverless
New vllm Serverless interface issue
yeah everything's the same. I tried to run it several times and was able to get it working again. I guess I missed out on rerun a code chunk.
15 replies