2x A100 / 3x 48 GB on Serverless
Hi @flash-singh, a while back we talked about having multiple GPUs on serverless and then you introduced 2x 48 GB. Now there are larger models out like Mixtral 8x7B which requires a minimum of 100GB, but ideally 120GB VRAM to serve.
Do you have any plans to expand capacity to allow for this in your serverless products? Perhaps, an easier route is to allow 3x 48 GB GPUs since that can serve models like Mixtral.
3 Replies
MI300 are coming in April, until then we might allow 2x A100 and 3x A6000
MI300 will be nice. At what price though?
likely similar to H100 price