riceboy26
RRunPod
•Created by riceboy26 on 9/25/2024 in #⚡｜serverless
Sharing a pod template
Yea, that’s surprising…. It was created using the desktop web app
9 replies
RRunPod
•Created by riceboy26 on 9/25/2024 in #⚡｜serverless
Sharing a pod template
But the template had to be created manually instead of cloning a serverless endpoint
9 replies
RRunPod
•Created by riceboy26 on 9/25/2024 in #⚡｜serverless
Sharing a pod template
Are u sure? I was able to create a shareable serverless template
9 replies
RRunPod
•Created by riceboy26 on 9/25/2024 in #⚡｜serverless
Sharing a pod template
NVM... there's an option to specify a servereless template instead of a pod template
9 replies
RRunPod
•Created by peteryoung2484 on 9/13/2024 in #⚡｜serverless
Is there a way to speed up the reading of external disks(network volume)?
Are you pretty glued to docker build cloud or are u open to GCP artifact registry?
I use it for non-api / typical backend docker images and apparently the limit is 5TB https://cloud.google.com/artifact-registry/docs/docker/pushing-and-pulling
49 replies
RRunPod
•Created by peteryoung2484 on 9/13/2024 in #⚡｜serverless
Is there a way to speed up the reading of external disks(network volume)?
Godly
49 replies
Minimize the startup time of ComfyUI on serverless/pod GPU
Just curious, what’s the size of your docker image?
You could also try a multi stage docker file to have an initial image download and install everything, and then only copy the files you care about for the runtime image which gets rid of all the cached and miscellaneous files to reduce your docker image size
15 replies
RRunPod
•Created by peteryoung2484 on 9/13/2024 in #⚡｜serverless
Is there a way to speed up the reading of external disks(network volume)?
Maybe Claude sonnet or Gemini could do it actually tho soon
49 replies
RRunPod
•Created by peteryoung2484 on 9/13/2024 in #⚡｜serverless
Is there a way to speed up the reading of external disks(network volume)?
I’m waiting for Google to develop their TPU stable diffusion ecosystem. Supposedly inference is ALOT faster and cheaper, but most of the open source projects (like controlnet, reactor, adetailer, etc) that are super useful to improve quality need to migrate over which is a huge project. It’s literally moving from PyTorch to tensorflow which means pretty much rebuilding everything from scratch
49 replies
RRunPod
•Created by peteryoung2484 on 9/13/2024 in #⚡｜serverless
Is there a way to speed up the reading of external disks(network volume)?
That might work. But then some stable diffusion models aren’t compatible with diffusers directly and you might run out of vram lol
49 replies
Minimize the startup time of ComfyUI on serverless/pod GPU
There’s also a “cache.py” file in runpod’s worker-a111 repo that you could take inspiration from that initializes models(interrogator/BLIP) by making sure they’re downloaded and loaded for subsequent requests, which really helped for my case (I’m using a111 still but there’s probably a similar thing you could do if ur using flux or comfyui)
15 replies
Minimize the startup time of ComfyUI on serverless/pod GPU
That’s probably why. I keep seeing around the serverless channel that network volumes are really slow. (I’m anticipating this issue too)
I’d suggest trying to fit the most common components(Loras, BLIP, and custom nodes) that are reasonably small directly in the docker image to help with loading time.
Network volumes have to load data through the physical wire which is always going to be slower than if your docker image had it available on RAM or local disk already.
The magnitude I saw other people have on loading times with network volume is 10-40 seconds. On my local Rtx 3070, loading a simple base 3-6GB model(like revanimated) takes under 3-6 seconds
15 replies
RRunPod
•Created by peteryoung2484 on 9/13/2024 in #⚡｜serverless
Is there a way to speed up the reading of external disks(network volume)?
Oo what i meant was, say you support sd1.5 and revanimated on your service.
Instead of having both sd1.5 and Reva images in the same docker image for a single runpod endpoint….
You have a dedicated runpod endpoint for sd1.5 with a docker image that has only sd1.5 baked in
And another separate runpod endpoint for revanimated with a different docker image with only revanimated baked in
And you’d do this for each model you support. And since each endpoint has its own workers(but ur not charged until the active request), you’d take advantage of flash bot when requests come in while minimizing model loading time
Generally speaking tho, how might this be different from replicate?
49 replies
RRunPod
•Created by NERDDISCO on 8/9/2024 in #⚡｜serverless
Slow network volume
What does it mean to compress a model when it’s a .pth or .safetensor file?
62 replies
Running a Dockerized Pytorch-based computer vision app
Deploy your docker image
Add your docker credentials to your runpod secrets if your docker image is private
Decide if you want to use a pod (available at all times and billed accordingly) or go serverless(has some delays for startup time and billed only for usage)
Follow the docs to set yourself up. For pod, I think you should be set up immediately
For serverless, you’ll need to have a custom rp_handler to call inference and return the payload to your own specs
2 replies
RRunPod
•Created by peteryoung2484 on 9/13/2024 in #⚡｜serverless
Is there a way to speed up the reading of external disks(network volume)?
In ur case, it might actually make sense programmatically build your docker images so then you have per-base-model-docker-images
I think this is how replicate was able to have cheap and fast inference time at the model level
Then, in your front end, u would basically map each model request to its respective model endpoint and won’t need to wait for start up time
49 replies
RRunPod
•Created by peteryoung2484 on 9/13/2024 in #⚡｜serverless
Is there a way to speed up the reading of external disks(network volume)?
The loras I use alone already take up 60Gb 😅
I also have about 100 other base models (most are between 3-6gb big)
49 replies