VFIO with Nvidia dGPU and AMD iGPU
Helping a friend setup VFIO with an Nvidia GPU for a Windows VM, and the guide at https://docs.bazzite.gg/Advanced/dracut-and-initramfs/?h=vfio#if-you-just-plan-to-add-options-to-a-module-you-should-consider-using-a-karg is outdated, per discord chat. The goal is nothing special - just the iGPU for Bazzite, and Nvidia GPU for the VM. VM hangs on attempted boot.
How to Modify Initramfs and Dracut - Bazzite Documentation
Bazzite is a custom image built upon Fedora Atomic Desktops that brings the best of Linux gaming to all of your devices.
Solution:Jump to solution
it was User Error(tm) - adding the display spice server back caused the VM to boot, the NVIDIA app installed and started downloading the drivers, ...and then my VM abruptly ran out of its VHD
118 Replies
Hello, I'm the friend being helped here. Can provide results of any attempted fixes. Thanks again for your time.
well the "section" example in the docs for adding the drivers need to be updated to a different example instead of referring to the vfio drivers which we pre-include now
can i see the output of
ls-iommu -grk?certainly
also small tip. put text like that between
```
text here
```
will make it appear as
much easier to read 🙂
ok so it should be bound then to the driver
should be as simple as adding it to the VM, and start the VM while a monitor is attached to the nvidia gpu
it is able to add the device(s) to the VM, where it wasn't before doing the VFIO setup (it would hang when attempting to add or remove)
but the VM does not boot with those devices, not even a sign of life on the monitor connected to the dGPU
is the vm already up and running with windows when you dont have the cards added?
yes, but the 'basic display driver' is unusable for my use case
yeah i am just asking since we have had cases where people dont even have a working VM before starting this 😆
understandable. yes, the VM booted and was able to launch the application I need (Unity Hub/Unity 6) but the basic display driver would hard crash the system when any load was placed on the 'gpu'
install something like TightVNC server inside windows and verify you can connect to it using something like remmina
and see if that has any output when you do the passthrough
as the card might not show any output until you have the driver installed
hmmm. that might be difficult to work around, considering the NVIDIA driver installer won't proceed through installing if it doesn't detect the NVIDIA GPU
thats why youre setting up VNC on the windows machine
if you actually get to the desktop and it just doesnt show a display because passthrough breaks the spice display in virt-manager
you can get around it with vnc
and just install the driver then
once driver is installed the monitor should light up
this is assuming its not freezing up because the VM is setup as a bios vm instead of uefi
it does light up briefly but the VM does not appear to get even as far as the UEFI screen
I'd assume it wouldn't have booted at all if that was the case
setting up VNC, this is all new to me. steep learning curve over a very short time
thats why we dont really recommend vfio to people, its like learning mountain climbing but your first mountain is mt everest or something like that :clueless:
setting up windows to go for a bootable windows from an external ssd or a 2nd machine to stream from is easier 😆
well, my alternative is debugging the flatpak version of Unity 6 for my use case, which is probably much more involved
actually i would say the alternative is windows to go if you have an external ssd 😛
or debugging the flatpak
:clueless:
i havent heard great things of the unity flatpak tbh, cant remember if people got it working in distrobox though
it works near-flawlessly but generates a mountain of arcane errors when trying to export a model from the scene. @oishishou and I wrote that off as above our pay-grade
tbh i would do so too unless my adhd kicked in 🤣
my adhd has most definitely kicked in. I'm locked-in, Hikari.
well the only thing stopping you atm is actual hardware and firmware shenanigans
it is agonizing how close it seems like we are. IOMMU works and groups the NVIDIA GPU in isolation, VFIO works and confirms it's in that mode via SuperGFXctl, and the device(s) add to the VM without hanging it. it just won't boot
i have a usb controller (not gamepad) that should be able to be used for passthrough to my VM but it freezes up my host system if i send it through, no idea why :clueless:
thats a firmware shenanigan
luckily i can passthrough a single thunderbolt controller and use a dongle for my VM as dumb as it looks 😭
hey, if it's stupid and it works, it isn't stupid
if we cant get it working, i would recommend going the windows to go route though on an external ssd if you can
way easier to setup and you already have a VM to install it on the ssd with
essentially it gives you an SSD you can plug into your pc and boot from and it will boot kind of an isolated windows system
I don't have an external ssd, closest I can do is a USB3 mass storage (which is internally just an sdcard)
yeah you dont want to do that
I guarantee running Unity on that would be excruciating yeah
you can look at
sudo dmesg --follow and journalctl -e --follow when you start the vm and see if its screaming somewhere when you start the vm
you can also do sudo virsh start vm_domain_name and see if libvirt spits out anything
had to use that last one on one of my VMs today actually to just have it tell me "hey this folder doesnt exist moron" 😂
as i was mounting a folder as a disk inside that vm and the folder didnt exist anymore
so it would just start and then kill the vm without any error on my end 😂just to make sure: I don't want to remove the default Display Spice from the VM devices, right? just add the PCI devices for the NVIDIA GPU
nope you want to keep it added (especially if you plan to use looking-glass)
Solution
it was User Error(tm) - adding the display spice server back caused the VM to boot, the NVIDIA app installed and started downloading the drivers, ...and then my VM abruptly ran out of its VHD
looks like I'm doing the first time setup again gamers
then you can actually claim you did it on first try :clueless:
there will be no proof left to say otherwise
you want windows to have a bare minimum of 128gb
with apps its 256gb-512gb minimum
that is very nearly the entirety of that SSD. yikes
idk how much space unity takes
but my windows gpu passthrough vms are 256gb
~15GB for my use case
ok so that and whatever youre doing in unity + 64gb for windows+2-4gb of nvidia drivers (im highballing the drivers)
Ooh, I'm the one who said delete the video devices. Oops. Sorry.
also this is looking-glass
I'll make it 128GB but this definitely gives me a good reason to tackle debugging the flatpak Unity also
maintaining a windows VM for this one purpose does not spark joy
lets me utilize the windows VM with gpu passthrough as an application, sadly it has issues with 6.13 and they are bisecting the kernel to try find the cause of it not working on 6.13+ for some people
i sadly use a windows VM for 1 game due to some useful addons for that game that do run on linux, but the transparency is black so well... i dont feel like playing with a black screen over my game 🤣
luckily i only need the addon for some content
probably followed an ancient guide
I'm a Vtuber, with a known-working workflow for modifying my model with Unity shader features... on Windows. As far as I know, nobody else is trying to do what I'm trying to do on Linux, and several of the guides on the subject insist on using a specific 2019 version of Unity, which is not necessary
Yeah, Gentoo user, haven't done it in a few years. Containers and immutability are new to me, too.
maybe you will be the person to change that 😛
I AM the ancient guide lol
hey, I like the sound of that
💜
well you are a gentoo user so checks out
gentoo has good documentation though just like arch with its wiki
Just not quite a fit for Bazzite
I am getting much more comfortable with being a Bazzite user. it has had a learning curve but no worse than any other desktop Linux I've tried
will definitely rep it on my stream whether or not I can get everything working here
oh yeah i get you, my friend chris titus uses bazzite only for his steamdeck and htpc, he runs arch on his work pc as that fits his workflow more, the rest he just wants to work always when he turns them on
Chris is great, been following his Windows guides for years
im a twitch and youtube mod for him, good friend of mine
he just never streams when im awake anymore
ayyy 🤝 community crossover
I would like to find one other example of 'Linux Vtuber'. Surely that isn't just totally untrodden territory
there are a handful, im not into the whole vtuber space so i dont remember any names. but i know brodie robertson has interviewed at least 2 linux vtubers over the past year
btw @oishishou i never got into testing this on gentoo if you feel experimental one day 😆
https://github.com/HikariKnight/quickpassthrough
GitHub
GitHub - HikariKnight/quickpassthrough: A project to remove the com...
A project to remove the complexity of setting up GPU passthrough for qemu - HikariKnight/quickpassthrough
I might eventually, but I'm unfortunately now on only 1 GPU. Second one died out of warranty.
supports dracut, initramfs-tools and mkinitcpio though
and grub
so it should just work if those criterias are met
Ooh, I'm full custom initramfs by hand
It would be cool to see if it works, though
oh boy dont think i support that, i only have support for those initramfs handlers mentioned added for building the initramfs
Haha, yeah, totally unrealistic
and grub and kernelstub(systemd-boot) are the only bootloaders i support (if it finds neither it will spit out what you need to add to the bootloader though)
Of course, I'm using EFI stub
ok i am definitively not writing in support for you then :clueless:
take the kernel args needed and go do it yourself 🤣
Each system has an artisanal hand-crafted kernel with initramfs lol
I'm glad I jumped the fence into the Discord. I had some anxiety about it, and also the invite link 'discord.bazzite.gg' didn't launch in my Discord client by default - I had to send the discord.gg link to myself in a DM
not sure who to report that to but it was an obstacle
it would be cool however to see how it handles a system like yours, if you do get around to test it could you run it, not apply anything (ctrl+c when it asks you about applying and that it requires to run sudo) and send me the config folder it generated and the log file if it got through all the dialog prompts @oishishou , i would be very curious in how it handles that situation
Oh, no, hahaha... I don't even have sudo installed
think thats a browser/discord flatpak issue since the discord.bazzite.gg just redirects you
That explains why mine worked fine. No containers.
still, would be fun to see what the config it spits out would look like, assuming it doesnt crash when it finds no initramfs handler
could be. I was able to work around it just fine, but you might be able to claim discord.gg/bazzite
or /universalblue I suppose
think we would need to be a certain size for that or something, not something i control
yeah that might be one of the things unlocked with boosts, haven't checked in my own discord
no idea, guess its something to add to the bucket list 😂
you'll get there. bazzite already has traction, getting recommended by more than one cc I follow
yeah we have pretty big growth
we have existed for longer but we didnt start using fedora countme before 07.2024
before that date we were counted as "fedora kinoite" and "fedora silverblue" heavily skewing the fedora atomic numbers
who do I have to thank for the official Bazzite OS-tan, Ametrine? she makes me smile
that would be @adelia for the concept and her sister for the art
hi
hi, I like your character concept a lot, she makes me smile
ok, so we have signs of life from the windows VM. it did recognize the NVIDIA card, it did install the NVIDIA app and drivers, installing the drivers did cause the monitor the dGPU was plugged into to come to life. I believe that's where the issue that started this thread is closed.
Unity 6 is installing and I look forward to making sure it does what I want it to shortly
best of luck. if you do have vfio related questions feel free to @ me and i will respond when i can
Okay, I’ve got one. What do I need to do to get back to Bazzite running on the dGPU only?
I tried setting Hybrid mode in SuperGFXctl, and it appeared to work, but after a reboot it’s hanging on the BIOS logo. Still outputting video through the dGPU, but not booting.
rpm-ostree kargs --delete-if-present="vfio_pci.ids=blah1,blah2,etc"
or just EDITOR=nano rpm-ostree kargs --editor and remove the vfio_pci.ids bit yourself and then CTRL+X to save
it is just the bit with the ids you need to remove
all the other vfio stuff is safe to keepnot sure how to go about doing this when I’m unable to boot
press ESC when booting
if you get to a grub> prompt, type
normal and press enter
then press E to edit the entry for this boot only
then remove the vfio_pci.ids bit and press CTRL+X to boot once with it like that
i have no way to undo what supergfxctl did without you being able to boot into the system though (i also lack a system that needs supergfxctl so i know nothing of it)
but as long as you get into the grub boot menu i can helpit just simplifies switching from dGPU/hybrid mode and VFIO, and confirms whether VFIO is supported and enabled
it’s technically for hybrid laptops, so it may actually be unwise for me to use if I don’t need to do this GPU passthrough again
think their vfio part is only tested for laptops, no idea how they implement it tbh
it is very late here for me so i will be heading to bed soon
got work in the morning 😅
I’ve successfully booted once as directed, what do I need to do to safely reboot again?
this
if you dont remember the ids for your gpu just use the editor method
just make sure you only remove the
vfio_pci.ids=vid:did,vid:did part
dont remove anything before or after it 🙂rpm-ostree kargs --delete-if-present="vfio-pci.ids=10de:2488,10de:228b"
got it with the editor method, just waiting for ostree to finish
Thank you for your assistance. I wouldn’t have been able to get here from where I was, without some help. 💜
np!
next project, new pc with vfio or 2nd pc just for doing the unity stuff? :clueless:
or do I find out the completely undocumented way to do this on the Unity flatpak?
:clueless:
yep, whats your plan
there is also the windows to go option but that involves rebooting again 🤣
flatpak is the next attempt, it may be that all of the guides on doing what I want to do with shaders are obsolete now
in which case the flatpak should work
ideally flatpak, 2nd alternative is distrobox
one of those 2 has to work
surely. there’s gotta be some serious game dev who’s figured out their workflow in Unity 6 on Linux.
it’s not well supported, from what I’ve been reading, but I refuse to believe no one has done it before
if you find it, it might be a random github gist or on a random blog in the middle of nowhere
like for example information about changing the smbios info for your VM, you can find that in the official VRchat wiki of all places 🤣
I did find a VRChat creator asking how to get the specific ancient 2019 release that’s recommended by a couple guides working, but that might turn out to be pointless
yeah would be sick to get the newest version working
instead of relying on old cruft
big agree, would give me a very good reason to document this process and get a new guide out there
Hello, after spending awhile getting everything mostly working on Unity 6, I determined there was actually a feature that 100% relied on me running the 2019 build.
Retracing my steps here, it was trivial to get the Windows 10 VM set up with that build. I’m very pleased with the result.
I’ll go find the message where I realized this was user error and mark it as the solution. Thanks for your time Hikari 💜
yw!
ope I guess @oishishou would have to do that
whenever he returns: pls right click -> Apps -> Mark Solution on this message
and yeah setting up vfio the first time takes ages, replicating it on a system you have done it once on is a lot faster
one command + replug monitors + reboot is pretty painless yeah
and i made sure i made the host config as stupidly easy as possible :clueless:
now imagine if you had a hdmi switch for just 1 monitor
truuuue I could make this even easier. I have enough ports for that but not enough cables
i also have a software based kvm solution for moving my mouse and keyboard between VM and host
i press a key combo, it runs a script and it passes through my mouse and keyboard into the VM
i do the same in windows, it ssh's into the host as root and runs the script again to unplug the mouse and keyboard from the VM
in a perfect world, I would only need to do this ridiculous solution once. since it’s already been more attempts than that, we have determined this world is not perfect.
I’ve gotten the shader process down to a science, though. I’m pretty sure it’ll only take me like half an hour if I do need to make changes to the model down the line
might stream the process once I get new body preset in, since I’ll be compelled to redo it anyway
if you had space and pcie lanes for 2 gpus (and there wasnt a kernel bug killing looking-glass atm) you could have it pretty damn seamless without having to mess with the cables and such
GitHub
3. Quality of life tweaks
Repo containing the PDF for my detailed vfio configuration setup and performance tweaking. - HikariKnight/vfio-setup-docs
this was written before looking-glass had spice support
what I would really like to do is actually sort out what is making this setup rely on Unity 2019. my research suggests that is beyond not just my skill level, but beyond the skill levels of everyone who has ever documented this process.
I did actually get Unity 6 on the flatpak to create and export a model, it just doesn’t let me change the parameters of the shaders at all.
I could’ve chosen to be satisfied with that, but… if you see the result in motion you’ll understand why I put in the effort
guess that will become a side project :clueless:
just gotta find more creators that already have their own workflows working here. It’s seeming a lot more likely than I thought it was when I started.