Can I select the GPU type based on the base model in python script ?
Some base models can run on a 24G GPU, while others require a 48G GPU.
7 Replies
Yes
What do you mean, can you explain more about what are you trying to do?
My requirement is to generate photos in different styles using different base models.
Some base models can complete the task with a 24G GPU, while others require a 48G GPU (24G would cause memory overflow).
I will pass the base model name as a parameter. Currently, the code only returns CPU, GPU, and XPU.
device = get_torch_device() generator = torch.Generator(device=device).manual_seed(seed)
Can I specify the GPU specifications?
For example, if my base model name is x, I want to add a condition in the code like if(base_model == x) gputype = 48G else gputype = 24G。
Nope, not possible, GPU configuration is at endpoint level, not request level
oh.. yea true, the other way around is you would hit another endpoint, so you will have 2 endpoints, 1 for 24, 1 for 48
Yep thats the only solution
I see， thanks
This is RunPod adjecent, but I came across this library:
https://github.com/anyscale/llm-router
This seems to be what you're looking for; however, you'd still have to do a lot of custom configuration to get this running and I believe the above guidance still applies.