More NVIDIA GPU Support coming?
Any plans to include RTX 4090 / 6000ADA or older devices?
DThe 4090 will probably not be officially supported since it's not a datacenter card. Modular is focusing on newer datacenter cards. However, it will probably work.
BAs Owen said, our current list of officially-supported NVIDIA GPUs are the ones that we run regular CI and benchmarks against: A10, A100, L4, and L40. Unofficially, Ampere and Lovelace architectures should be supported, and we've had a number of people run RTX 30XX and 40XX GPUs successfully with MAX as of the 24.6 release.
Make sure you're using at least NVIDIA driver version 555, and keep in mind that an 8 billion parameter model with bfloat16 weights will need a 24 GB RAM GPU.
Make sure you're using at least NVIDIA driver version 555, and keep in mind that an 8 billion parameter model with bfloat16 weights will need a 24 GB RAM GPU.