Topics

RunPod•3mo ago

Can we run aphrodite-engine on Serverless?

aphrodite-engine is a fork from vLLM and also supports exl2 format, which gives it a huge advantage. Are there any plans to support aphrodite-engine in future on RunPod's serverless offering? I believe currently aphrodite-engine is only supported as a single server on RunPod. Thanks

8 Replies

Madiator2011•3mo ago

I mean nothing stops your from building own worker 🙂

houmie•3mo ago

How? lol

Madiator2011•3mo ago

all workers are open source so you can have look at code and build own. Some coding and then packaging docker image

houmie•3mo ago

Seriously, I would be interested building my own queue and host aphrodite-engine on a single pod instead.

houmie•3mo ago

You mean this worker: https://github.com/runpod-workers/worker-vllm

GitHub

GitHub - runpod-workers/worker-vllm: The RunPod worker template for...

The RunPod worker template for serving our large language model endpoints. Powered by vLLM. - runpod-workers/worker-vllm

houmie•3mo ago

Or any other you could recommend?

Madiator2011•3mo ago

I do not know what aphrodite-engine is so cant tell but yes if it's fork ov vllm it will be good start point

nerdylive•3mo ago

Yes find your own source study them and that will help you understand, the triggers of runpod serverless is quite easy They all are documented on runpod docs

We're a community of enthusiasts, engineers, and enterprises, all sharing insights on AI, Machine Learning and GPUs!

12KMembers

View on Discord

Want results from more Discord servers?

Add your server

More Posts

Template pytorch-1.13.1 lists cuda 11.7.1 version but is actually cuda 11.8?I tried running a model that requires pytorch-1.13.1 and 11.7 but it said the cuda version doesn't m Idle timeout not workingHi team. I'm setting my serverless endpoint with a Idle timeout for 180 seconds, but it's idleing o Can't connect to sfptHi, I can't access sftp. On my previous pod I could do it and I just swaped the ip and the port, but Unable to ssh onto my pod with the public key already on the runpod serverI am unable to ssh into the pod when using the command from runpod's site: ``` name .ssh % ssh root Is serverless cost per worker or per GPU?I'm looking at serverless GPU options and when looking at 48 GB GPU it costs $0.00048/s. But is tha openai compatible endpoint for custom serverless docker imagehow can I get openai compatible endpoint for my custom docker image in runpod serverless. I am tryi Securely using serverless endpoints on the client-side?I have a use case where I'm doing a client-server webapp that uses serverless endpoints. In order to I wanna use comfyUI for a img2vid workflow - can I do this via the serverless serviceI already tried setting up a pod yesterday where I uploaded all the needed models and stuff but toda Using network volume with serverlessI am running a stateless model within serverless to modify provided image. I am wondering if the net Python modules missing when pod is startingWhen starting a comfy-ui pod after some downtime, I get a lot of messages of the kind ``` Import t