Managing multiple pod discovery
Hi, if I want to put a load balancer/queue system for multiple pods, is there some premade app I can use for that? I was thinking of something like Kubernetes but its not compatible with Podrun. Or is this not the use case of podrun?
5 Replies
What is Podrun? Do you mean RunPod? Kubernetes is not compatible with RunPod and you can't use a load balancer. If you want to scale up and handle many concurrent requests, you should use RunPod Serverless, not Pods.
Yeah RunPod. I want to use some open source abstraction without lock-in
https://www.anyscale.com/ray-open-source
There is this, might be a good approach but I see no Runpod compat
Anyscale
Ray Distributed Computing - Anyscale
Ray, the fastest growing open source framework in distributed machine learning, makes it radically simple for developers to scale their most compute-intesive workloads from their laptop to any cloud.
Its not compatible with RunPod, RunPod Pods aren't designed to be load balanced etc, they are meant to be stand-alone. If you want to do this, you will have to figure out a solution yourself, its not something RunPod offers or can support you with.
As I mentioned, you can use serverless, it has queing, load balancing, auto scaling etc all built in, so either use that or you are on your own.
Yeah, ty. I'll look into other providers for inference.