houmie Comments - Answer Overflow

Topics

houmie

•Created by houmie on 6/28/2024 in #⚡｜serverless

vLLM serverless throws 502 errors

Unsure, still investigating. RunPod Serverless is extraordinary for what it does. But it has still quite a few bugs.

11 replies

•Created by houmie on 6/28/2024 in #⚡｜serverless

vLLM serverless throws 502 errors

Thanks, I will report this to support.

11 replies

•Created by Armyk on 5/30/2024 in #⚡｜serverless

GGUF in serverless vLLM

Hence I assume the latter is experimental.

58 replies

•Created by Armyk on 5/30/2024 in #⚡｜serverless

GGUF in serverless vLLM

The same isn’t true about Aphrodite as serverless

58 replies

•Created by Armyk on 5/30/2024 in #⚡｜serverless

GGUF in serverless vLLM

No, I mean right now I can create an vLLM serverless directly from run pod dashboard.

58 replies

•Created by Armyk on 5/30/2024 in #⚡｜serverless

GGUF in serverless vLLM

But it will be experimental right? I don't think it's that easy and I'm so busy as it is with coding. 🙂

58 replies

•Created by Armyk on 5/30/2024 in #⚡｜serverless

GGUF in serverless vLLM

Yes, but aphrodite runs only on classic Pods and it's very expensive to run. 🙂 This is why I love serverless, it's cheap to begin with (but gets 3 x more expensive if you have constant traffic). Serverless is great to start a project with minimal traffic, only if the project is a success and can generate money, then it's worth it to switch to a classic pod with aphrodite.

58 replies

•Created by Armyk on 5/30/2024 in #⚡｜serverless

GGUF in serverless vLLM

Yeah I just hope vLLM would one day support EXL2. It would open up so many new opportunities.

58 replies

•Created by Armyk on 5/30/2024 in #⚡｜serverless

GGUF in serverless vLLM

Yeah and It's not the fastest though.

58 replies

•Created by Armyk on 5/30/2024 in #⚡｜serverless

GGUF in serverless vLLM

GGUF is a format for offline use on your own computer. It's not meant to be for servers really. Use AWQ or GPTQ untill ex2 is supported on vLLM.

58 replies

•Created by houmie on 6/20/2024 in #⚡｜serverless

How to download models for Stable Diffusion XL on serverless?

Sorry, I worded it badly. Of course I could do 5% of 50 GB. What I meant is that having the actual number is more accurate and convenient.

32 replies

•Created by houmie on 6/20/2024 in #⚡｜serverless

How to download models for Stable Diffusion XL on serverless?

Ah yeah. It says 95%. Yeah would be good if it could give us an actual number instead of guess work.

32 replies

•Created by houmie on 6/20/2024 in #⚡｜serverless

How to download models for Stable Diffusion XL on serverless?

I don't think that's possible on network storage because it shows everything

32 replies

•Created by houmie on 6/20/2024 in #⚡｜serverless

How to download models for Stable Diffusion XL on serverless?

ok, after I attached it to a pod how could I do that? df -h ?

32 replies

•Created by houmie on 6/20/2024 in #⚡｜serverless

How to download models for Stable Diffusion XL on serverless?

Thanks, yes, I 'm making progress with runpod-worker-a1111 . Is there a way to check from the dashboard how much space is left on the network-storage?

32 replies

•Created by houmie on 6/20/2024 in #⚡｜serverless

How to download models for Stable Diffusion XL on serverless?

Ahh nice. But this repo is based on the classic SD, not SDXL, correct? In that case for SDXL, I will try to close it and change the files myself. Then I need to add the model to my docker image and push it to DockerHub, correct? Then in RunPod I would create a template based on the dockerHub image and build a new serverless endpoint? So far my plan makes sense? 🙂 And will the model that the docker downloads be added to the attached network storage? I have a feeling because there is no environment variable passed in, the docker image is loaded in local storage, instead of network storage. I hope I'm wrong, because that would take a very long time each time I would post to the endpoint.

32 replies

•Created by houmie on 6/20/2024 in #⚡｜serverless

How to download models for Stable Diffusion XL on serverless?

Ah so it's currently using the base model stable-diffusion-xl-base-1.0 ? So do I have to clone the https://github.com/runpod-workers/worker-sdxl and change the two files manually fromstabilityai/stable-diffusion-xl-base-1.0 to stablediffusionapi/dreamshaper-xl ? Is there no environment variable to inject in instead?

32 replies

•Created by houmie on 6/20/2024 in #⚡｜serverless

How to download models for Stable Diffusion XL on serverless?

Or do I need to clone https://github.com/runpod-workers/worker-sdxl and add the DreamShaper XL to it, push it to DockerHub and then pull it as serverless template?

32 replies

•Created by houmie on 6/20/2024 in #⚡｜serverless

How to download models for Stable Diffusion XL on serverless?

So if I wanted to use DreamShaper XL could I do this with that?

32 replies

•Created by houmie on 6/18/2024 in #⚡｜serverless

RUNPOD_API_KEY and MAX_CONTEXT_LEN_TO_CAPTURE

I see. Ok, so there is no way to set a custom key. Thanks

27 replies