serverless multi-gpu
Hi, in the serverless endpoint console I'm seeing that you can't have a serverless multi-gpu endpoint except for 2x A40? Is this correct? So essentially the serverless product is only for smaller models?
5 Replies
Its supported by the 48GB tier which includes A40 and A6000, not just A40.
Thanks. Why can I only select 2 GPU's per instance in the 48gb tier? There is no option to do 4x 48gb or whatever for serverless?
To prevent someone from using all available capacity and leaving no capacity for the other customers.
Solution
we are slowly allowing more as we get more available capacity