R
RunPod•4w ago
Casper.

update worker-vllm to vllm 0.5.0

vLLM just got bumped to 0.5.0 with significant features being ready for production. @Alpay Ariyak FP8 is very significant but so is speculative decoding and prefix caching. - FP8 support is ready for testing. By quantizing the portion model weights to 8 bit precision float point, the inference speed gets 1.5x boost. - Add OpenAI Vision API support. Currently only LLaVA and LLaVA-NeXT are supported. - Speculative Decoding and Automatic Prefix Caching is also ready for testing, we plan to turn them on by default in upcoming releases.
Solution:
For sure, already in progress!
Jump to solution
2 Replies
Solution
Alpay Ariyak
Alpay Ariyak•4w ago
For sure, already in progress!
Casper.
Casper.•4w ago
Nice to hear it's already in progress! Let me know when it's ready and I would love to test it out
Want results from more Discord servers?
Add your server
More Posts
SDXL Quick Deploy through Runpod Doesn't workI sent a request in to test it such as below, and it threw an error. There are other alternatives, sVideo processingHey, What are your approaches and/or recommendations for processing videos in serverless workers?Pod GPU assign issueRecently I started noticing that sometimes any new pod initialising is stuck at this step. SometimesPod Unable to Start Docker ContainerI've tested this Docker image on my local computer and other servers, however on Runpod it seems to How can I install a Docker image on RunPod?I had a chat with the maintainer of aphrodite-engine and he said I shouldn't use the existing RunPodCPU Only Pods, Through RunpodctlHeyo! Is there a way to create cpu only pods through runpodctl? I don't see a flag for cpu type, butcan 3 different serverless workers running from same network volume?Hi @digigoblin I have checked your answer about symbol linking network volume dir to serverless dUnable to create template or pod with python sdk version 1.6.2```python import runpod import os runpod.api_key = os.getenv("RUNPOD_API_KEY") try: # Creatin