VM GPU Passthrough
Hey @HikariKnight , I had a quick question before I got started with this whole rigamarole.
As discussed in another thread, single GPU Passthrough is not supported (or at the very least not easy to do), so it is recommended to have 2 GPUs available, whether that be two dGPUs or one iGPU and one dGPU. Am I able to set up a VM with the latter, then easily change it to the former later?
I ask because I am missing a power cable for my second dGPU (should have one by EOD) but I was looking to get started with setting everything up now
131 Replies
one igpu (for the host) and one dgpu (for the VM) is enough
i have read some people have success with passing through the igpu but the chances of that being in an IOMMU group with something else that shouldnt be passed through is higher.
you will need a working VM before starting as you will know stuff is working then.
you can check your iommu groups with this
https://github.com/HikariKnight/ls-iommu
GitHub
GitHub - HikariKnight/ls-iommu: A tool to list devices in iommu gro...
A tool to list devices in iommu groups, useful for setting up VFIO - HikariKnight/ls-iommu
also setting up singlegpu passthrough once you run the ujust to enable the vfio modules is the same as on a normal linux distro, it is just that it relies on scripts to accurately kill your display manager, clean up your session then unbind and rebind the gpu.
that mess we do not help setting up 😅 plus youre prone to crashing your system a few times while setting it up, having ssh up with a
journalctl -xe --follow and a dmesg --follow is advised if you try set it up.
for dual gpu you just run the ujust, make sure the card is is in its own IOMMU group and then just add a rpm-ostree karg to permanently bind the card to the vfio driver (until you remove the karg either through rpm-ostree or temporarily remove it during boot in grub)If I set it up with iGPU for host and dGPU for VM, will I still be able to use the dGPU for the host when I am not using the VM?
no
you would be able to maybe do it dynamic loading but at that point youre in the same area as single gpu passthrough since it does something very similar
that settles it then. 2 dGPU solution is the way I want to go because I want to game on linux and use windows vm exclusively for adobe software.
So first thing's first from what I am gathering, is make a working VM
the biggest lynchpin after having 2gpus is ideally having a motherboard that supports bifurcation
meaning you can split the x16 lanes of the first PCIe slot to x8 and send the other x8 to the 2nd or 3rd PCIe 16 slot
otherwise the 2nd gpu will run on usually x2 or x4 lanes and be severely bandwidth limited
x8 is a nice middleground since it does not affect performance much (single digit or less) and a lot of mid range cards these days even are x8 lane cards anyway 🙃
hmm I should look into what the 2080 Super is in that regard...
most likely x16 but the difference between x16 and x8 mode is usually not that high because gpus tend to not be able to maximize the bandwidth on PCIe x16 slots (unless its a new card in an ancient slot)
personally i would use the stronger card for gaming and the weaker card for the VM, especially since your 2nd gpu will be a 900 series gpu which support might vanish for at nvidias whim here
that was my plan 🙂
just out of curiosity, do you think there would be a noticeable decrease in power if I skip bifucation on the x16 slot and leave the 970 with x4?
the gpu would lose half its bandwidth but you should still be able to use it for tasks that are not graphically demanding
i forgot what you were planning to use the passthrough for, i vaguely remember photoshop
yeah it will be exclusive Adobe software. Photoshop, Illustrator, After Effects (<- thats the big one)
mostly JUST AE. I am using this experience to learn new alternative software for photoshop and illustrator on linux, there just isnt really a good alternative to AE
idk how demanding aftereffects is, i would say give it a shot and see how it works, knowing there is a possibility to be bandwidth limited if you have a motherboard that cant bifurcate the lanes
for reference the gpu performance in a VM should be within 1-2% of what you get on metal (non-vm) if you have enough lanes for the gpu
@ROG
ROG STRIX Z370-E GAMING | Motherboards | ROG Global
ASUS ROG Z370 ATX gaming motherboard with advanced cooling, Aura Sync, SupremeFX, dual M.2, Intel LAN, onboard Wi-Fi and USB 3.1 for 8th Gen Coffee Lake.
I will look into bifucation later, am I able to turn that on later and and have it work without issue?
well if it is the board i linked then it should support it
yes sorry, that IS the board I am using! Thank you for looking into that for me!
running in x8x8 will most likely disable the bottom slot
meaning you have to use the 2 reinforced slots as those are the ones being bifurcated
the manual will have more details about it
https://dlcdnets.asus.com/pub/ASUS/mb/LGA1151/ROG_STRIX_Z370-E_GAMING/E13238_ROG_STRIX_Z370-E_GAMING_UM_WEB_082417.pdf?model=ROG%20STRIX%20Z370-E%20GAMING
okay good, thats how I have it set up right now
this shouldnt affect the m.2 drive i have installed in there correct?
no idea, you would have to check the manual for it 🙂
also, Thank you so much for all this help. I am working from home right now and am splitting my attention between both of these tasks
np i need to run to the store
I will take a look!
alright I have the vm installed, which btw, was insanely easy to install once I knew that ujust was needed to install virt-manager and the other required files
LOVE that
Is bifurcation something I set up in the bios?
yes
something about pcie mode or something, just look for an option that mentions x16 mode or something
I have found a bunch of settings that say x16, but they only let me choose gen 1, 2, 3
Nothing that says split to x8 + x8
its on auto so it will auto detect
Right, but this is all I can chose from on both
unless should I keep is auto?
keep it on auto and it shoud auto go into x8 x8 when you have 2 gpus (in the reinforced slots)
if you connect a 3rd pci device it becomes x8 x4 x4
or someting similar
Hey so I am going through the quickpassthrough setup and I am getting this error at the write stage
Here is my debug file
dont use quickpassthrough on atomic, it literally says so in the project page
what i linked to you was ls-iommu which is the underlying tool for quickpassthrough which does work on atomic as all it does is list your hardware and the iommu groups
GitHub
GitHub - HikariKnight/ls-iommu: A tool to list devices in iommu gro...
A tool to list devices in iommu groups, useful for setting up VFIO - HikariKnight/ls-iommu
Oh sorry. I did read that but misunderstood it thinking that after doing those things in Bazzite, it would work. Stupid I know. Again, I am brand new to all of this command line stuff and its quite difficult for me to understand without visually being show how to do this step by step. I am trying my best to learn and do it on my own though without being a bother and appreciate your continued help.
I initially tried to build what you linked first in terminal, but got a "command not found" with the "go mod download" command. So I downloaded it and ran terminal from that folder and couldn't get the executable to run, continuously running into "command not found". I decided to run it in Konsole and it returned the what is in the image. Because the konsole window wasn't giving me any more options with how to move forward, I tried Quickpassthrough with my previously stated understanding and was seemingly getting somewhere (I know now that is not the case). At the very least, I know based on what Quickpassthrough mentioned, the two IOMMU Groups I am looking for are 10de:13c2, and 10de:0fbb (The audio and the video of the 970). I am unsure of what to do from here.
yeah you have a problem
both gpus are in group 1
any suggestions on how to fix that?
if your bios has an option for ACS (i know asrock tends to have that) enable it
other than that there is nothing you can do :/
there is a hacky kernel patch but we do not support it in bazzite since it is a hacky workaround that needs to be updated every time the kernel updates, and since our kernel moves so fast it will be impossible for us to maintain so we do not support it
damn. okay I will check that out in a bit. what does ACS stand for so I research that?
cant remember but its a mode that shuffles around the IOMMU groups a bit and possible make them more favorable for passthrough
the thing with IOMMU groups is that everything in 1 group has to be passed through to the VM together with everything else in the group
Ah gotcha
I will do some poking around and report back
Alright. I have news to report.
TLDR: Intel blows.
I am running an 8700K which supports iGPU and dGPU iommu separation, but NOT 2 dGPU separation.
Specifically looking at that second comment with the youtube link https://www.reddit.com/r/VFIO/comments/74h6xj/post_your_z370_board_iommu_groups/
So with this in mind, how would I go about setting the iGPU to the windows guest and leave the dGPU for the linux host?
if your board allows having the igpu active along with the dgpu and they are in separate iommu groups (and the igpu is not in a group with anything vital)
okay I think I have done that
igpu in group 0 and dgpu in group 2
cool you should be able to pass through the igpu then
you will need to have a dummy dongle for it or a monitor connected to it for windows to enable the gpu
but for testing a 2nd input or monitor is all you need
okay good to know. I have a second monitor to work with
okay I have set the iGPU to the VM following the instruction in ujust, how would I go about testing to see if it worked?
ls-iommu -gk -i 0
see if its bound to vfio-pci
most likely not since i havent accounted for igpus and i dont know the driver arc uses, but you can easily modify itI am getting ls-iommu command not found
you used it here
I just right clicked the ls-iommu executable in the downloads folder and clicked "run in konsole", I though because you added different arguments that it was different.
either way, this is what it looks like when I do those same steps
youre supposed to open the terminal and run the executable manually
because it is a cli application
if you are in the same folder as the binary then you need to use
./ls-iommu -gk -i 0
because ./ means "in this directory"thank you for that tip!
sorry for the late reply, you should be able to just add the igpu as a pci device for the VM and boot it now
I went to add hardware, and added the UHD Graphics, but came up with an error. I will post the error in a bit
also no need to apologize, I am just glad you are sticking with me here on this hair pulling "adventure"
always hair pulling the first time, after that you will go "i know this"
anyways i have to head off, got appointments in the morning
I dream of the day I am this good :KEKW:
but yes thank you so much for the help today
also here is the error I was talking about
this is your clue
cannot limit locked memory of process 209809 to 43016781824: Operation not permittedThanks to your hint I THINK I have fixed the locked memory issue... I changed the values in memlock.conf to the value it said in the error
but now I have this error
I am trying to change the permissions of that file but nothing is working
Are you running virt-manager as sudo? sudo or being in the libvirt group is required for vfio.
if you want to avoid running it as sudeo you can add yourself to the libvirt group
then relog
So I went into qemu.conf and changed the user to root following a suggestion I found online.
After doing the commands you suggested, I am getting the same result
none of that is needed
none of the edits you have done to files should be required
Alright I will go back and change them to the defaults
I have set everything back to their defaults and I am back to this error
I also re-entered these commands and restarted the whole computer
and I have no idea how to run virt-manager as sudo (I tried the "./" before virt-manager in the respective folder and it said command not found)
"./" Wont work unless the binary is on the same folder.
Running as sudo should be unnecessary if you are in the libvirt group. Check with the
id commandalso make sure you didnt set the vm up as a User Session when you added a connection
it has to be a System VM
the VM cannot use all your RAM either or all your cores
you need to leave enough to keep the host system running
this is basic VM 101
Oh... I think thats probably the issue then... it is totally setup as a user session
Is there any way to transfer over the vm I made into a System vm?
man... how is it problem after problem...
This is what I get when I try to add the connection
systemctl status libvirtdsystemctl restart libvirtdtry now
That seemed to have worked!
Gah it looks like I am going to have to setup a whole new VM under the system session aren't I?
you can copy the qcow2 file
copy it to
/var/lib/libvirt/images/ requires sudo
when you make a VM make sure you set it to "manual install"
also make sure the VM is EFI (as in it uses OVMF)
bios installs do not work well
kinda thought you had done this before when you talked about it originally 😛oh man I am sorry I gave you that impression, this is literally my first time make a VM in linux, and am about as beginner a linux user could be
you have no idea how much i appreciate the help
Its way too easy to get stuck, search the issue on DDG, implement a suggested solution from some random user, and then ultimately mess up the install
well as i said the ujust does 90% of the job for a dual card setup
the remaining 10% is binding the gpu (through rpm-ostree kargs), making the VM and passing through the card
and I bet if I had set up the VM properly the first time, it would be up and running by now...
Sorry to make you work so hard 😅 but again I really appreciate the help
its easy enough to move the disk file into the system qemu though
just copy it to the location i mentioned and manually select it and say you will manually install a system later
unless you also installed it as a bios install
Sorry to ask more questions, where can I find the disk file? I have opened the libvrt/images/ folder as sudo, i just need the file. I am not sure if I installed it as a bios install, but I dont think I did
look at the disk in the VM
mine says SATA Disk 1, is that okay?
yea because thats the bus used for the disk
mine is virtio because linux includes the virtio drivers usually
Hi everyone, I'd like to know if ublue uses a kernel with ACS patch. If not, if there's a way to install a custom kernel that has it like CachyOS one
this has been mentioned and explained inside this very thread...
the answer is still NO
we cannot support hacky kernel patches for workarounds to bad firmware (same goes with vendor-reset).
if you want to use a custom kernel, make a custom image and literally see how bazzite adds its fsync kernel instead of the normal fedora kernel
our scope isnt to be the VFIO OS, it is to be a gaming os similar to SteamOS but more up to date.
Thanks
me personally i try get motherboards from vendors that add the ACS patch in the bios instead (ex: Asrock) it is a way better solution 🙂
Yes, agree, but I couldn't pass the really good price found on mine
Fair. I'm currently saving up to get the board for my upgrade first. Everything else can wait
I got an Asus x670e ProArt for the price of a cheap b650e, an equivalent Asrock would have costed more than double
Oh I looked at that one. I didn't like the pcie lanes on it. 2nd GPU will only have 4 lanes so it will be kneecapped
just if you will share with a nvme slot, otherwise is 8 lanes
it has 4 nvme slots, if you use the 4th bottom one, it will do that
I have a lot of drives
Me too, but I don´t need a second discrete GPU. I use the iGPU for host and passthrough the dedicated one
I need 2 dgpus
So I need at minimum 2 x8 lanes
And those boards are expensive
yeah
MSI x670e carbon and ASRock x670e taichi being the cheapest for me
I'm not aware of a board that has 4+ nvme slots and don´t halve second pcie slot
I only need 2 nvme
Rest can be sata
oh, ok. but Asus ProArt wouldn't be x4 then for you, not sure it has enough SATA ports for you though, it only has 4
Yeah won't work for me think I got 8 drives total
Fair, you could use an nvme to sata adapter or a PCIe one, but I get avoiding using those
8 drives is messy enough
yeah, I have a NAS, so I don´t use SATA in my pc
I have both
I decided to just say screw it and reinstall the vm under the system session, then after it finished I added the igpu passthrough. How can I check to see if it worked?
check device manager and try install the driver
https://github.com/HikariKnight/vfio-setup-docs/wiki
references since you are new to vfio
GitHub
Home
Repo containing the PDF for my detailed vfio configuration setup and performance tweaking. - HikariKnight/vfio-setup-docs
omg THANK YOU for this resource!!!
this is so helpful
I am changing the bus type of Sata Disk 1 to VirtIO correct?
So I changed it, and followed the rest of the instructions. Does this mean it worked?
at the very least i can change the resolution now haha
yes you just need the intel driver for the igpu now
yes if you have the virtio driver installed and booted windows once with a virtio disk present and formated the dummy disk
oh my good god its done! it is installed and working! You sir, have the patience of a saint. Thank you so much for helping me out with this
I have made note, saved links, and will definitely be reviewing this all again to really commit this to memory
Hey @HikariKnight, I'm sorry to bother you with this. Another user said that you could help me with my problem/question. My goal is to be able to use my 3090 (the only gpu in my system beside my apu) in a virtual machine. I don't know much about gpu passtrough, I pretty much stumbled upon it on my research why my 3090 wasn't showing up in my VM. What do I have to do to be able to use gpu passtrough with my vm?
enable iommu and sr-iov(if available) in your bios, sometimes iommu is hidden under sr-iov
then do
ujust setup-virtualization and enable vfio and follow the instructions
this assumes you will use your APU on the host and reserve the rtx 3090 exclusively on your VMyou might also need this to check if the gpu is in its own iommu group
https://github.com/hikariknight/ls-iommu
GitHub
GitHub - HikariKnight/ls-iommu: A tool to list devices in iommu gro...
A tool to list devices in iommu groups, useful for setting up VFIO - HikariKnight/ls-iommu
this is the only setup we support. if you plan to do single gpu passthrough and use your nvidia card for both, then you need to enable iommu in the bios, enable vfio with ujust, ignore all instructions in the ujust and once rebooted you follow whatever single-gpu guide at your own risk, we will not help you.
Ok, under what category is sr-iov or iommu usually in the bios
i wish manufacturers could standardize
that is my response
Ok
Thank you, I will try it
@HikariKnight I am having issues with the openGL version in my VM. With my graphics (Intel UHD Graphics 630), I SHOULD have OpenGL 4.6, but as far as I can tell, the VM is only running 1.1. I have downloaded the latest drivers but it will not update OpenGL
I think on Windows its always opengl 1.1
Did a little digging and found this: https://askubuntu.com/questions/1238523/enabling-opengl-in-windows-10-guest-vm-in-qemu
Ask Ubuntu
Enabling OpenGL in Windows 10 Guest VM in QEMU
I'm trying to install Sketchup 2019 on a Windows 10 VM (Ubuntu 18.04, QEMU 3.1.0), however Sketchup requires OpenGL 3.1 or later in order to run. I had asked a similar question previously hoping by
after running the Command Prompt and selecting the first option in it. I now have acces to OpenGL 4.2
cool glad that works then
not something i have had to ever deal with