Overclocking Multiple Nvidia GPUs possible in Bazzite after X11 is abandoned in F40?
Hello all,
I want to take the plunge into immutability but made the mistake of going all in on Nvidia cards and worry about getting things to play nicely with Wayland after F40 abandons X11.
I currently run on OpenSUSE with 3 3090s using X11. My primary GPU is plugged into a pci-16x port and the other 2 are on pci-1x risers. I use a script to set persistence mode, max power draw, fan and clock speeds independently for each card on boot.
All cards use CUDA 100% of the time in the background (even while I am gaming).
Setting the power and clocks is important for efficiency and vital for my setup, so AFAIK I am bound to X11.
I understand that I can just choose X11, but I want to set things up in a way where I don't have to make many changes in the future.
When I migrate to Bazzite for my host machine, what is the best way to replicate this setup?
Does xwayland allow me to set clocks/power?
Is vfio with hardware pass-through for each card a viable solution? (I had this working on Arch a few years ago and liked the set up)
Can I use Wayland as my primary WM and open a distrobox with an X11 session to set the clocks while retaining GPU priority in the primary session?
Grateful for any advice, and excited to make the leap into immutability.
28 Replies
X11 should have nothing to do with hardware control, you should be able to control it even on Wayland in exactly the same way. That being said so long as X11 is a better experience for Nvidia we will make sure it continues to be included in future versions
If you have any specific issues with your hardware control under Wayland I would be glad to assist
Cheers,
My understanding is that Nvidia's drivers only make available the coolbits flag and max power/ clock settings to an xserver.
I understand that some people have had success running Wayland and then also running X11 headless in order to set their clocks.
I would be be interested in what the best way of achieving that would be in an immutable setup. I am hoping it is as simple as creating a distrobox to set the clocks and run nvidia-xconfig and it doesn't require layering or opening a second TTY
GitHub
fork of https://gitlab.com/leinardi/gwe. Contribute to dankamongmen/GreenWithEnvy development by creating an account on GitHub.
This is what we use for clock control, the Nvidia control panel is X only but the actual feature of overclocking doesn't care what your display server is
GWE is a great tool for single GPU setups. Unfortunately it does not support multi GPU setups (beyond allowing you to modify the primary GPU)
The arch wiki says that a coolbits value of 12 set in /etc/X11/xorg.conf is required in order to be able to modify fan+clocks (I undervolt so don't need value to add value 16)
https://wiki.archlinux.org/title/NVIDIA/Tips_and_tricks#Overclocking_and_cooling
Here's another article on it.
https://www.phoronix.com/news/MTY1OTM
Am I just totally misunderstanding this as a limitation of Wayland?
How To Overclock New NVIDIA GPUs On Linux
With this morning's release of the NVIDIA 337.12 Beta Linux driver there is finally GPU overclocking support for the NVIDIA 'Fermi' GPUs and newer, a.k.a
If I used vfio and passed each card through to a Bazzite-Arch VM maybe I could use GWE without needing X11?
Yes on the misunderstanding, but if that tool doesn't cover your use case it may as well be true that it only works on X11
A rootful distrobox may work in this case
Alternatively, you'll just have to remain on X for now and we'll likely continue to support it past F40
Would a rootful distrobox allow me to effectively run a headless X11 server in session so I could make changes?
Passing each card through to a VM could be a neat trick if it allowed me to use GWE without a need for X11.
I'll tinker around with some options and do more research on uBlue and distrobox.
Thanks for your help 🙂
The way the guided support for vfio in bazzite is set up, it will result in binding the GPU to the vfio drivers, meaning they can not be used by the host or containers, only VMs.
This is the easiest method for us to support as there are lots more caveats when you want to hotswap the card between host and vm.
I did however write the vfio stuff to not prevent you from doing something more exotic with it.
If you enable vfio but do not set the kargs, you can setup hotswap yourself, but you will most likely not be able to get support from us on it due to lack of us being able to set it up ourselves (not that many people here that use it plus it is pretty advanced functionality) and lack of hardware to test with.
Plus hot swapping the GPUs tend to be way more problematic (especially when it is more than one and they are using the same driver as the host card)
Hope that will help at least 🙂
vfio should not be needed for distrobox however
For my usecase I don't need to render on anything but my primary card.
Hotswapping is not important to me, as neat as it would be.
Does Bazzite's vfio allow passing cards by unique ID?
If GWE does indeed allow clocking and undervolting without coolbits the vfio setup you guys made should work on a pure Wayland setup provided each slave card has its own VM.
I'd have to find a near lossless way to consolidate the compute back to the host or a stratum running in a docker.
Rootful distrobox doesn't feel in the spirit of uBlue xD
I still need to understand how to use distrobox better in an immutable setting.
My only previous experience with VMs is KVM/QEMU
you bind them using vendorID:deviceID in kernel arguments
if all your cards share the same deviceID then you will need to add custom scripts to initramfs and bind manually using pciaddresses which is not a usecase we support (since the script has to be manually tailored to the system), plus it is not something i have done myself yet
Can I bind them based on their pcie slot?
no, pciaddress
kinda like pci slot but its an address assigned by the kernel and uefi
essentially you have a script that say "hey this specific card at this address space, do this with it before the system boots"
The script you are talking about is I think how I had it working before in Arch.
Could it be achieved using an overlay if needed?
im not very familiar with dracut, but i did write in the ability to add things to dracut without it clashing with what we do in bazzite
let me dig up the PR which has the documentation as it is fairly new
so if it is something you want to experiment with, i wont stop you 😄
Does Bazzite-Nvidia have support for the native nvidia-container-toolkit through distrobox?
GitHub
feat: add initramfs rebuilding with support for user defined initra...
This adds come needed flexibility to how bazzite-hardware-setup handles initramfs args.
Changes:
Added /etc/bazzite/initramfs/ as $INITRAMFS_DIR
Added /etc/bazzite/initramfs/args.d/ as $INITRAMFS_C...
I'll tinker for sure and report back what works.
awesome as i only have 2 gpus and one dedicated for VM use and the host one set up as hotswap, but i have also carefully selected my hardware to do this and i know i do not have things running in the background that could use the card when i do hotswapping so i can easly support my 1 usecase
making something more universal for everyone will be a lot harder, so for me it is off the table until the process gets simplified
also if you want a tool to make it easier to get the pciaddresses and ids and such, i have one written myself in my personal repo
https://github.com/HikariKnight/ls-iommu
GitHub
GitHub - HikariKnight/ls-iommu: A tool to list devices in iommu gro...
A tool to list devices in iommu groups, useful for setting up VFIO - GitHub - HikariKnight/ls-iommu: A tool to list devices in iommu groups, useful for setting up VFIO
makes it very easy to probe for info needed for vfio
I'm going to try to find the documentation on how hotswapping worked on the Arch config I had from a few years ago. Would be really cool if Bazzite could support it somehow.
Awesome, I'll take a look
hotswapping does work, it just gets exponentially more complicated to support when you dont know what the user will be running in the background 🤣
or they have multiple identical cards (deviceID wise)
worst case they have something running in a distrobox which uses the card they are hotswapping (lets say for simplicity this is their host card since most people have 1 gpu)
since the card is still used and you fire up a VM that triggers the hotswap, the VM will start, hotswap will fail because the card is being used, the VM will hang waiting for the card, the users screen will be black, waiting for the VM to fully start so it will have an image again
and the only way to easily debug it is with ssh, before the system hang 😄
If each vendorId is different, but the device IDs the same it would be easier?
if each vendorid or deviceid is different it will work
or at least different from the host card
I think this solution would work for my usecase then.
I will try the X11 route with rootful distrobox to set clocks through CLI and no passthrough first.
Whenever I go down the vfio rabbithole I am always sad that I own Nvidia cards 🤣. I'd love to pass a card some ram, an SSD and some core through to a HTPC but gamescope doesn't work with Nvidia.
yeah i use amd for my host gpu and nvidia for my VMs
I just got around to installing Bazzite-Nvidia on my machine.
GreenWithEnvy is useless in both the Wayland and X11 session as coolbits must be set to either 8 to allow overclocking or preferably 12 to enable fan control as well.
This is noted in the FAQ on the GWE page and at the bottom of clocking section of the program running in Bazzite.
Perhaps in a future version of Bazzite the Nvidia post install script could offer to set coolbits for the user while installing GWE so at least the program does something for those with a single card.
https://gitlab.com/leinardi/gwe
GitLab
Roberto Leinardi / GreenWithEnvy · GitLab
System utility designed to provide information, control the fans and overclock your NVIDIA card