Can we run aphrodite-engine on Serverless?
aphrodite-engine is a fork from vLLM and also supports exl2 format, which gives it a huge advantage. Are there any plans to support aphrodite-engine in future on RunPod's serverless offering?
I believe currently aphrodite-engine is only supported as a single server on RunPod.
Thanks
8 Replies
I mean nothing stops your from building own worker 🙂
How? lol
all workers are open source so you can have look at code and build own. Some coding and then packaging docker image
Seriously, I would be interested building my own queue and host aphrodite-engine on a single pod instead.
You mean this worker: https://github.com/runpod-workers/worker-vllm
GitHub
GitHub - runpod-workers/worker-vllm: The RunPod worker template for...
The RunPod worker template for serving our large language model endpoints. Powered by vLLM. - runpod-workers/worker-vllm
Or any other you could recommend?
I do not know what aphrodite-engine is so cant tell but yes if it's fork ov vllm it will be good start point
Yes find your own source study them and that will help you understand, the triggers of runpod serverless is quite easy
They all are documented on runpod docs