r/VFIO • • Mar 21 '21

Meta Help people help you: put some effort in

640 Upvotes

TL;DR: Put some effort into your support requests. If you already feel like reading this post takes too much time, you probably shouldn't join our little VFIO cult because ho boy are you in for a ride.

Okay. We get it.

A popular youtuber made a video showing everyone they can run Valorant in a VM and lots of people want to jump on the bandwagon without first carefully considering the pros and cons of VM gaming, and without wanting to read all the documentation out there on the Arch wiki and other written resources. You're one of those people. That's okay.

You go ahead and start setting up a VM, replicating the precise steps of some other youtuber and at some point hit an issue that you don't know how to resolve because you don't understand all the moving parts of this system. Even this is okay.

But then you come in here and you write a support request that contains as much information as the following sentence: "I don't understand any of this. Help." This is not okay. Online support communities burn out on this type of thing and we're not a large community. And the odds of anyone actually helping you when you do this are slim to none.

So there's a few things you should probably do:

  1. Bite the bullet and start reading. I'm sorry, but even though KVM/Qemu/Libvirt has come a long way since I started using it, it's still far from a turnkey solution that "just works" on everyone's systems. If it doesn't work, and you don't understand the system you're setting up, the odds of getting it to run are slim to none.

    Youtube tutorial videos inevitably skip some steps because the person making the video hasn't hit a certain problem, has different hardware, whatever. Written resources are the thing you're going to need. This shouldn't be hard to accept; after all, you're asking for help on a text-based medium. If you cannot accept this, you probably should give up on running Windows with GPU passthrough in a VM.

  2. Think a bit about the following question: If you're not already a bit familiar with how Linux works, do you feel like learning that and setting up a pretty complex VM system on top of it at the same time? This will take time and effort. If you've never actually used Linux before, start by running it in a VM on Windows, or dual-boot for a while, maybe a few months. Get acquainted with it, so that you understand at a basic level e.g. the permission system with different users, the audio system, etc.

    You're going to need a basic understanding of this to troubleshoot. And most people won't have the patience to teach you while trying to help you get a VM up and running. Consider this a "You must be this tall to ride"-sign.

  3. When asking for help, answer three questions in your post:

    • What exactly did you do?

    • What was the exact result?

    • What did you expect to happen?

    For the first, you can always start with a description of steps you took, from start to finish. Don't point us to a video and expect us to watch it; for one thing, that takes time, for another, we have no way of knowing whether you've actually followed all the steps the way we think you might have. Also provide the command line you're starting qemu with, your libvirt XML, etc. The config, basically.

    For the second, don't say something "doesn't work". Describe where in the boot sequence of the VM things go awry. Libvirt and Qemu give exact errors; give us the errors, pasted verbatim. Get them from your system log, or from libvirt's error dialog, whatever. Be extensive in your description and don't expect us to fish for the information.

    For the third, this may seem silly ("I expected a working VM!") but you should be a bit more detailed in this. Make clear what goal you have, what particular problem you're trying to address. To understand why, consider this problem description: "I put a banana in my car's exhaust, and now my car won't start." To anyone reading this the answer is obviously "Yeah duh, that's what happens when you put a banana in your exhaust." But why did they put a banana in their exhaust? What did they want to achieve? We can remove the banana from the exhaust but then they're no closer to the actual goal they had.

I'm not saying "don't join us".

I'm saying to consider and accept that the technology you want to use isn't "mature for mainstream". You're consciously stepping out of the mainstream, and you'll simply need to put some effort in. The choice you're making commits you to spending time on getting your system to work, and learning how it works. If you can accept that, welcome! If not, however, you probably should stick to dual-booting.


r/VFIO • • 9m ago

Running llama.cpp in a VM with no GPU passthrough: Qwen3.8 27B at 98% of native on one RTX 6000 Ada

2 Upvotes

My earlier post was done using Qwen2.5 and rightly so I was called out for it. So I reran it with Qwen3.8 27B, Qwen3.5 9B and Qwen3.6 35B-A3B with as much proof as I could think of. The VM has no NVIDIA device, driver or nvidia-smi. llama.cpp's CUDA calls go through VMAT to the card on the host. Qwen3.8 27B: 45.4 tokens/s on the host, 44.6 in the VM. All four models land at 98%. Three VMs at once get 45.7 together, against 45.0 for three programs on the host.
Once I get my hands on Qwen4, I’ll post another video based on it…

Full video with every step: https://youtu.be/r6yhOVWxP5Y?is=reuv6gZptnDNzU1D


r/VFIO • • 9h ago

I built a compatibility database for gaming in VMs (plus Proton reports) and the report coverage is thin. If you test games, I could use your results

Thumbnail
pictures.svr-host.de
3 Upvotes

vmdb.it is my site, so full disclosure right away. It collects compatibility reports for gaming in virtual machines, VMware, VirtualBox, QEMU/KVM, Hyper-V, Parallels, and it also takes regular Proton reports for the same games. The VM part exists because the question "will this game's anti-cheat ban me if I run it in a VM" comes up constantly and the answers are scattered across dead forum threads.

Reports are anonymous, no account needed, takes a few minutes. There's also a My Setup thing where you store your hardware/VM/Proton config in your own browser storage. Reports matching your setup get sorted first and the report form pre-fills from it, so the second report is mostly clicking submit.

Anyway, the actual reason for the post: coverage is thin and it's not growing on its own. Pulled these from the API a few minutes ago:

\- Crimson Desert Enhanced: 0 reports. It's also one of the most searched games on the site right now.

\- Bodycam: 0

\- Marathon: 0

\- CoD Modern Warfare 4: 1 VM report, 0 Proton

\- Forza Horizon 6: 3 VM reports, 0 Proton

If you run any of these in a VM or through Proton, a report takes like five minutes and the next person who searches the game gets an actual answer instead of an empty page. Same goes for anything else you play of course, those are just the gaps people keep hitting.

And if you ran a game in a VM and got banned by anti-cheat, please write that up too. That's data that basically doesn't exist anywhere else.

The API is public and read-only if you'd rather script against it: API docs

If you think the whole idea is dumb or something on the site is broken, I want to hear that too.


r/VFIO • • 2d ago

Support RX 9070 XT pass-through to CachyOS VM

3 Upvotes

Specs: OS: Proxmox CPU: Ryzen 9 5900x MOBO: Gigabyte X570 Aorus Master GPU1: Gigabyte Radeon RX 9070 XT GAMING OC 16G GPU2: RTX 3070 ti GPU3: RX 6400 (used for host) RAM: 64GB DDR4 (F4-3200C16D-64GVK)

I have yet to assemble this system but I have some of these parts on the way to complete a new build. I want to pass through the 9070 XT to a CachyOS VM. I have seem some people have successfully done this with a Windows VM but not Linux. Has anyone does this?


r/VFIO • • 2d ago

Resource Something some might somehow find somewhat useful sometimes

Thumbnail
github.com
5 Upvotes

A year ago I posted about my dynamic GPU passthrough setup for which I wrote a overengineered program to manage and streamline all that. At the time I didn't release it because I didn't think it was robust enough. After a year of using it I only had to add some extra logging or fix parsing. At one point I just started to forget it even existed because I never had any problems with it. So now I think it is somewhat robust enough to be used but I make no promises after all this was made only for one machine and never tested on anything else. The code is here.

So instead of moving scripts around you just add a bit of xml to your VM config. For example this bit of xml for the pci device 0000:0a:00.0. Which tells the hook to unbind from kwin, kill all the processes, unload the nvidia driver and load the vfio driver.

    <dyn:dynamic_unbind xmlns:dyn="0.0.0.0">
      <pci_device domain="0x0000" bus="0x0a" slot="0x00" function="0x0" driver_finder_on_shutdown_method="store"/>
    </dyn:dynamic_unbind>

r/VFIO • • 3d ago

Nvidia's ethics: driver gave a fake PCI BAR error in 2021

120 Upvotes

With Nvidia in the news this week, I want to tell an old story about how I believe this company treats its customers.

In 2021 I had a laptop with a GeForce GTX 1650 and wanted to pass the discrete GPU through to a Xen guest to run KataGo. Nothing exotic: my hardware, my hypervisor, Nvidia's own official Linux driver in the guest.

It didn't work. The driver failed with an error about PCI BARs. So I did what anyone would do and treated it as a real BAR problem. I spent between one and two weeks on firmware settings, kernel parameters and Xen configuration. None of it helped, because none of it was the problem.

Eventually I asked myself a different question: if Nvidia wanted to stop this from working on purpose, what is the laziest way they could do it? Answer: compare the hypervisor's CPUID signature against a hardcoded string.

So I ran the strings command on the driver binary, and there it was: XenVMMXenVMM, Xen's CPUID signature. I changed it to something random of the same size.

The GPU worked immediately. Same hardware, same BARs, same everything.

I never saw the source code (it was not publicly available), so this is inference, but the experiment is hard to read any other way: the driver detects Xen and refuses to run, and it reports the refusal as a hardware fault. Not "GeForce is not supported in virtual machines", which would at least have been honest and saved me over a week. A fake error that sends you debugging something that isn't broken.

I learned afterwards that this was the Linux/Xen face of a well-known error that people had been fighting when trying to pass Nvidia GPUs to guests. This points to driver code deliberately made to fail, with a misleading message. In my opinion, the reason was to push virtualisation users to Quadro cards with similar silicon at several times the price.

They dropped the check later, and from then on the GPU ran without patching their driver with a hex editor.

The moral, for me: a company that will lie to you in an error message has zero ethics. They didn't then, I see no reason to believe they do now, and nothing they ship should be trusted further than what you can verify yourself. Which, with a closed driver, is not very far.

Note: I was just a hobbyist Go player trying to take a break at the time; I did not keep any logs with the specific Xen, Linux and Nvidia driver versions.


r/VFIO • • 2d ago

Eve-NG on Apple Silicon

2 Upvotes

Has anyone been able to get EveNG to work on Apple Silicon M1 Pro? If so would you mind explaining how you got it to work?


r/VFIO • • 7d ago

Discussion GPU-PV on KVM update: Windows VMs now work on AMD too, looking for testers with other cards

Post image
96 Upvotes

Hey, a while back I posted about getting Hyper-V style GPU-PV working on KVM. The VM runs the real Windows driver and the host keeps the card, so my desktop stays on it the whole time.

Since then I've rewritten it all in Rust, made an app that installs Windows from an ISO by itself in about 9 minutes, and got AMD working. Windows guests run AMD's own driver now, and on my Ryzen's iGPU the desktop, D3D11, D3D12, Vulkan and OpenGL all work. Video decode doesn't yet. Linux guests on AMD work too.

The problem is I've only got my 5090 and that iGPU, and one tester has run it on a 4080. So I'm looking for people with:

  • AMD discrete cards, RX 6000 especially. RX 7000 and 9000 are less likely to work yet.

  • NVIDIA 40, 20 and 16 series

  • NVIDIA 30 series, which isn't supported yet, but I need a card to get it there

You need the open NVIDIA kernel modules or amdgpu on the host. If your card isn't supported, it still installs with plain VMs and tells you why.

Obviously this project is experimental. My point in working on it is to prove that Linux can run GPU VMs. Since no one had ever built a tool like this, I always thought it wasn't really possible, so here we are.

My goal is to support as many GPUs as I can so more people can try it, but I won't be able to manage this project solo.

Repo: https://github.com/xnixy/guesthouse
AMD reports go in #1 and NVIDIA ones in #9, or just reply here.


r/VFIO • • 6d ago

Support Need some help on decision to buy a second GPU

3 Upvotes

So my system config is 9900X, 5070ti, 32g ddr5.

I have managed to pass through the igpu of the 9900X and get it binded to a windows VM when I start via qemu/virt-manager. But thing is it seems that once it starts if i shut it down i cannot restart the VM with the gpu still being loaded. I need to reboot my host. (Code 43)

My main job in windows is dev related for windows native stuff. So it works out relatively fine. But i was wondering if I am missing something and this issue is resolved now already.

In case it cant be done was wondering what low powered gpu i should get, cgpt and claude gave mixed answers wrt amd vs nvidia, says rdna 1 cards also have similar issues rdna2 may be better but not sure. I was eyeing the RX 6500 as its cheap enough will solve my problem and can be used without changing my psu (corsair 850w) . So should i bite the bullet and get that 2nd gpu or are there options i can use?


r/VFIO • • 7d ago

macOS 26 running as a KVM guest on Omarchy (Asahi) on my M1 Max machine

14 Upvotes

It runs on KVM with two kernel patches (Apple's PAC VM-key registers, and one MMIO store XNU makes that KVM can't handle). omvm builds those as a second kernel and keeps the stock one in the boot menu. QEMU is aquarat's KVM port of the vmapple machine. Graphics go from macOS's paravirtual GPU driver through reims-vgpu to Vulkan on Honeykrisp.

Honeykrisp needed VK_EXT_fragment_shader_interlock exposed, because the macOS compositor uses it almost everywhere. That's in my Mesa branch. It still fails 64 MSAA CTS cases, so it isn't ready for an MR yet. aquarat's 16-bit flat varyings fix is in there too.

xcodebuild runs at 95-99% of native CPU. I use it in a 120 Hz window or over Moonlight from my phone. Apple ID and iMessage don't work, because Apple ties them to a macOS host's Secure Enclave.

Repo: https://github.com/sapoepsilon/omvm
Mesa branches: https://github.com/sapoepsilon/mesa
Follow me on twitter: x.com/killerisya


r/VFIO • • 8d ago

Resource [Update] geforce-sriov v0.2.0 beta: SR-IOV vGPU on consumer NVIDIA Ampere and Ada GPUs

Thumbnail
gallery
158 Upvotes

Hey everyone, I posted here a few days ago about getting SR-IOV vGPU working on consumer Ampere GPUs. I've made quite a bit of progress since then and just released geforce-sriov v0.2.0 beta.

To recap, geforce-sriov is a toolkit that lets multiple VMs share a consumer NVIDIA GPU through SR-IOV, rather than mdev used by many earlier vGPU unlock projects.

The biggest news is Ada support is now working! Thanks to a friend who generously lent me their RTX 4070 SUPER, I was able to experiment with Ada and figure out how to get SR-IOV vGPU working on it.

Unlike Ampere, most consumer Ada cards report only 1 VF in hardware, so getting 4 VFs working required additional patches to NVIDIA's host driver and vGPU plugin. The current version supports up to 4 VFs per card using the proprietary driver with GSP disabled.

What's new in v0.2.0:

  • Ada support: Tested on RTX 4070 SUPER, with Linux and Windows guests working. Ampere and Ada cards can also share the same host at the same time.

  • Simplified setup: The geforce-sriov setup command now handles prerequisite checks, installation, driver patching, and VF activation automatically. Run it once, cold power cycle the host, then run it again.

  • Improved documentation: Reorganized the README and expanded the guides with diagrams to make installation, configuration, troubleshooting, and recovery easier to follow.

  • Other improvements: MSI-X support for both generations, automatic VF setup at boot, readiness checks before VM startup, and a cleaner uninstall process.

It's still beta, and so far I've only tested it on my dual RTX 3080s and an RTX 4070 SUPER but other Ampere and Ada GPUs should work too.

GitHub: https://github.com/noon-at-cgn/geforce-sriov

Release: https://github.com/noon-at-cgn/geforce-sriov/releases/

If anyone has other Ampere or Ada GPUs and wants to try it, compatibility reports would be really helpful.


r/VFIO • • 7d ago

Support Inconsistent frametimes in Minecraft, other games work fine (QEMU w/ GPU passthrough)

Thumbnail reddit.com
1 Upvotes

r/VFIO • • 8d ago

Anyone using a Windows VM with GPU passthrough for professional CAD work on Linux?

3 Upvotes

I’ve been dual-booting Windows and Linux for a long time, but Linux has become my primary OS for pretty much everything except work.
I still regularly need Windows for AutoCAD, Revit, and ReCap. I’ve tried BricsCAD on Linux, and while it’s a great application, it doesn’t quite cover all my needs.
Lately, I’ve been considering switching to a Windows VM using QEMU/KVM with GPU passthrough instead of constantly rebooting into Windows.
I’m particularly interested in hearing from people who actually use a similar setup for daily work, especially with CAD or other GPU-intensive professional software.
A few questions:
How does performance compare to running Windows natively?
Have you experienced stability issues during long working sessions?
How seamless is the integration with the Linux desktop (clipboard, file sharing, multiple monitors, keyboard shortcuts)?
Does GPU passthrough cause any issues with Hyprland or Wayland?
Is the setup reliable enough for professional work, or does dual boot remain the more practical solution?
My current setup:
OS: CachyOS / Windows 11 dual boot
Desktop: Hyprland (Wayland)
CPU: Ryzen 7 5700X
GPU: Radeon RX 570
RAM: 16 GB
I’d really appreciate any real-world experiences, especially from people who have used this kind of setup long-term.


r/VFIO • • 11d ago

Success Story got hyper-v GPU-PV running on plain KVM, no passthrough

Thumbnail
gallery
146 Upvotes

No VFIO, no vGPU, no SR-IOV. The host keeps the card the whole time.

Windows boots under QEMU/KVM, thinks it's running on Hyper-V, and just loads the normal NVIDIA Windows driver. There's a ~600 line QEMU patch for the VMBus devices plus a Python daemon that pretends to be the Hyper-V host and talks to the NVIDIA Linux driver. Frames go straight from VRAM to a GTK viewer as dmabufs.

Right now D3D12, D3D11, Vulkan and OpenGL all run on the real card. Godot Forward+ works up to 4K (windowed or exclusive fullscreen) and sound goes through PipeWire. I also made a little app that does a fully unattended Windows install from an ISO in about 8 minutes, and the GPU already works when it finishes.

I'm also poking at a Linux guest with GPU-PV and CUDA, but that's still early.

Raw numbers on a 9950X3D + RTX 5090 on Fedora:

HostGuest
Compute115-119 TFLOPS
VRAM copy~1,070 GB/s
Submit round trip21 µs

Vulkan and OpenGL pixel tests match the host exactly.

Code isn't public yet. It still depends on a capture from my own machine and I've only tried it on a 5090. Mostly just wanted to show it off. Anyone else been messing with this on KVM? What would you want tested first?

Edit (forgot to mention): GPU isolation between processes, each Windows process gets its own GPU page tables. In a test where one process tried to read and overwrite another's GPU memory, it got 0 of 16,384 words and the data survived.

Edit 2: I started the public repo, where I’m moving the code in pieces. It will take some time, but at least for now you can review the QEMU patch and the general structure. https://github.com/xnixy/guesthouse/ and roadmap at https://github.com/xnixy/guesthouse/issues/2

Edit 3: almost all code is public, please see the roadmap.


r/VFIO • • 10d ago

Tired of Anti-Cheat issues and black screens? I built a GUI for safe GPU Passthrough with dynamic hardware checks.

Thumbnail
gallery
28 Upvotes

Hey everyone,

We all know the struggle in the Linux community with software support and Anti-Cheat games. Even VM and GPU passthrough solutions are overly complicated, and one wrong move can turn your whole host into a black screen.

That's why I built a tool called Wise Linux Studio.

I combined advanced backend tools like QEMU/KVM, VFIO-PCI (for hardware isolation), libvirt, and Looking Glass, and wrapped them in a simple Python (Tkinter) GUI. It passes through the actual hardware to the Windows VM, but with a strict safety system. It scans your hardware topology, detects if you have a dual-GPU or single-GPU setup, differentiates the iGPU from the dGPU, and dynamically adjusts how the tool operates based on that.

I just wanted to merge these powerful tools with a safety mechanism and a simple GUI that actually guides you.

It’s not a massive world-changing project (: and I can't guarantee every single program will work perfectly since I haven't tested everything yet (:

But I did test it on a laptop with only an integrated GPU. The safety system immediately detected it, locked the GRUB setup buttons, and blocked the VM creation to protect the host from crashing!

Currently, the tool only supports Fedora.

If anyone is interested or wants to mess around with the code, here is

https://github.com/t11man/Wise-Linux-studio.git


r/VFIO • • 10d ago

Support 7900xtx gpu passthrough stopped working

2 Upvotes

Hello everyone,

A few months ago, I had a psuedo single GPU passthrough setup working with my 7900xtx (I say psuedo because I am passing both a 7900xtx and 6600 to the vm.) I was using it one day when I accidentally rebooted the VM instead of shutting it down, and since then, I haven't been able to get a display signal. I can VNC into the VM from my laptop and as far as I can tell, the GPU is attached to the VM with no issues (atleast according to Windows). But in the displays, I see the Amd vDisplay instead of my physical monitor, so I'm not sure what do do next. Here's a list of what I've tried so far:

  • Enabling and disabling above 4g decoding and rebar in the bios

  • Forcing iommu on in the bios

  • Enabling and disabling rom bar in virt manager

  • Dumping the vbios in gpu-z and adding it to my virt manager xml

  • Using a dummy bios

  • adding the vendor id and hidden state xml tags

  • Adding grub arguments from the Arch wiki

  • Reinstalling drivers in the VM

  • Booting a fresh windows install

This is also the current virt manager xml that I have atm:

<domain type='kvm'>
  <name>gaming-7900xtx</name>
  <uuid>daf7ae8c-24f6-4c3f-9830-577b781f0a91</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/11"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>33554432</memory>
  <currentMemory unit='KiB'>33554432</currentMemory>
  <memoryBacking>
    <source type='memfd'/>
    <access mode='shared'/>
  </memoryBacking>
  <vcpu placement='static'>24</vcpu>
  <os firmware='efi'>
    <type arch='x86_64' machine='pc-q35-11.1'>hvm</type>
    <firmware>
      <feature enabled='no' name='enrolled-keys'/>
      <feature enabled='yes' name='secure-boot'/>
    </firmware>
    <loader readonly='yes' secure='yes' type='pflash' format='raw'>/usr/share/edk2/x64/OVMF_CODE.secboot.4m.fd</loader>
    <nvram template='/usr/share/edk2/x64/OVMF_VARS.4m.fd' templateFormat='raw' format='raw'>/var/lib/libvirt/qemu/nvram/gaming-7900xtx_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vpindex state='on'/>
      <runtime state='on'/>
      <synic state='on'/>
      <stimer state='on'/>
      <vendor_id state='on' value='9876543210ba'/>
      <frequencies state='on'/>
      <tlbflush state='on'/>
      <ipi state='on'/>
      <avic state='on'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
    <smm state='on'/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' clusters='1' cores='12' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/colin/VMs/gaming.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x17'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x18'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='10' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='10' port='0x19'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
    </controller>
    <controller type='pci' index='11' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='11' port='0x1a'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
    </controller>
    <controller type='pci' index='12' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='12' port='0x1b'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/>
    </controller>
    <controller type='pci' index='13' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='13' port='0x1c'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/>
    </controller>
    <controller type='pci' index='14' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='14' port='0x1d'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/>
    </controller>
    <controller type='pci' index='15' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='15' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='16' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='16' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='17' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='17' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='18' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='18' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='19' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <address type='pci' domain='0x0000' bus='0x0d' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <filesystem type='mount' accessmode='passthrough'>
      <driver type='virtiofs'/>
      <source dir='/home/colin/VMSHARE'/>
      <target dir='VMSHARE'/>
      <address type='pci' domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
    </filesystem>
    <filesystem type='mount' accessmode='passthrough'>
      <driver type='virtiofs'/>
      <source dir='/mnt/5a3a870e-a864-4373-b19f-6732830ebbbc/Games'/>
      <target dir='GamerSSD'/>
      <address type='pci' domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
    </filesystem>
    <filesystem type='mount' accessmode='passthrough'>
      <driver type='virtiofs'/>
      <source dir='/mnt/33eed47a-4105-468b-8f0d-ce491e8e7f21/Games'/>
      <target dir='GamerSSD2'/>
      <address type='pci' domain='0x0000' bus='0x0c' slot='0x00' function='0x0'/>
    </filesystem>
    <interface type='network'>
      <mac address='52:54:00:12:58:00'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <tpm model='tpm-crb'>
      <backend type='emulator' version='2.0'>
        <profile name='default-v1'/>
      </backend>
    </tpm>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' passwd='146064'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x13' slot='0x01' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <rom bar='off'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <rom bar='off'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </hostdev>
    <watchdog model='itco' action='reset'/>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
</domain>

And here are the relavent specs:

  • GPU 1: Amd rx7900xtx

  • GPU 2: Amd rx6600

  • CPU: Amd ryzen 9 7950x

  • Motherboard: Gigabyte B650 AORUS ELITE AX V2

  • Distro: Arch linux

I'm gonna keep trying to chip away at this, but I'll do anything if you have any insight or thoughts about this 🙏 Thank you!!


r/VFIO • • 11d ago

looking-glass-b6-linux-kit Public - Accelerated headless display hw

5 Upvotes

So, lately I’ve been building an AI development environment that includes a graphical interface, and I wanted everything to run as "smoothly" as possible. Since my onboard GPU was just sitting there unused, I figured... why not? Unfortunately, I ended up spending an entire night getting it configured. The documentation is sparse, and every system presents its own unique set of quirks. Anyway, I finally finished around 6:00 AM, and since it’s all set up, I thought others might be interested in this ready-to-use solution.

What’s included: A reproducible kit for an Ubuntu 26.04 virtual machine with an XFCE/Xorg desktop environment, an AMD GPU configured via VFIO PCI passthrough, and a "Looking Glass" client running on the host system (using a second GPU). It features bidirectional clipboard support via SPICE, an ICH9 audio system, LAN bridging, IVSHMEM/KVMFR, automatic startup, and a 4K SDR configuration.

It’s available on GitHub here: https://github.com/rav0rmat/looking-glass-b6-linux-kit

I hope you find it useful—and, of course, I’d appreciate any feedback!


r/VFIO • • 11d ago

[Fix] Blackwell GPU passthrough: "No devices were found" / "WPR2 already up" after a guest reboot. For headless Linux guests, blacklist nvidia_drm + nvidia_modeset (RTX PRO 4000, FreeBSD bhyve host)

3 Upvotes

TL;DR: the fix

**Symptom:** an NVIDIA Blackwell card passed through to a Linux VM works on the first VM boot after the host boots. Any **guest** reboot or shutdown then hangs. After that, the card is dead inside the VM (`nvidia-smi`: "No devices were found"; dmesg: `unexpected WPR2 already up, cannot proceed with booting GSP`) until the **whole host** is rebooted. FLR and PCIe link resets from the host don't bring it back.

**Cause (in our case):** NVIDIA's *display* modules (`nvidia_modeset`, `nvidia_drm`) hang while unloading in a VM that has no display (stuck in `nvRmVTSwitch`). So the driver never gets to shut the GPU's GSP firmware down. Its protected memory region (WPR2) stays up, and the next driver start refuses the card.

**Fix (headless / compute guests):** never load those two modules:

```

# /etc/modprobe.d/nvidia-headless.conf

blacklist nvidia_drm

blacklist nvidia_modeset

install nvidia_drm /bin/false

install nvidia_modeset /bin/false

```

```

sudo update-initramfs -u

```

Then reboot the **host** once, to get a clean card.

After that only `nvidia` and `nvidia_uvm` load (all that CUDA needs). Guest reboot, guest poweroff and an ACPI shutdown from the host all work, and the card comes back every time without a host reboot, including when the VM autostarts after a host reboot.

**What it does NOT fix:** hard-killing the VM while the driver is loaded (`vm poweroff -f`, `virsh destroy`, and so on) still leaves WPR2 up, and only a host reboot clears that. Shut the guest down properly.

---

## Setup

- **Host:** FreeBSD 15.1-RELEASE, **stock bhyve (no patches)**, vm-bhyve, UEFI guest firmware (`BHYVE_UEFI.fd`). Intel i7-7700T (Kaby Lake), B250 chipset, VT-d enabled, 64 GB RAM.

- **GPU:** NVIDIA RTX PRO 4000 Blackwell SFF Edition, 24 GB (`10de:2c33`, GB203GL), plus its audio function (`10de:22e9`), in the CPU's x16 slot.

- **Guest:** Ubuntu Server 24.04.5, kernel 6.8.0-147-generic, 4 vCPU, 32 GB (wired), headless.

- **Guest driver:** 615.71.09, open kernel module (`nvidia-headless-no-dkms-615-server-open`, `linux-modules-nvidia-615-server-open-generic`, `nvidia-utils-615-server`), CUDA 13.4.

## Passthrough config

Host, `/boot/loader.conf`:

```

vmm_load="YES"

pptdevs="1/0/0 1/0/1"

```

vm-bhyve guest config:

```

passthru0="1/0/0"

passthru1="1/0/1"

wired_memory="yes"

```

That gives bhyve `-S` (wired memory) and `-s 0:6:0,passthru,1/0/0 -s 0:6:1,passthru,1/0/1`.

Host after a reboot:

```

$ pciconf -l | grep ppt

ppt0@pci0:1:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x2c33 subvendor=0x10de subdevice=0x2053

ppt1@pci0:1:0:1: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x22e9 subvendor=0x10de subdevice=0x0000

```

Guest:

```

$ lspci | grep -i nvidia

00:06.0 VGA compatible controller: NVIDIA Corporation Device 2c33 (rev a1)

00:06.1 Audio device: NVIDIA Corporation Device 22e9 (rev a1)

```

Driver install (Blackwell needs the **open** driver):

```

sudo apt update

ubuntu-drivers list --gpgpu

sudo ubuntu-drivers install --gpgpu nvidia:615-server-open

sudo apt install nvidia-utils-615-server # --gpgpu leaves out nvidia-smi

```

The first boot with the driver works:

```

$ nvidia-smi

+-----------------------------------------------------------------------------------------+

| NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4 |

+-----------------------------------------+------------------------+----------------------+

| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |

| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |

|=========================================+========================+======================|

| 0 NVIDIA RTX PRO 4000 Blac... Off | 00000000:00:06.0 Off | Off |

| 30% 36C P8 9W / 70W | 1MiB / 24467MiB | 0% Default |

+-----------------------------------------+------------------------+----------------------+

```

## The problem: what we saw

  1. **`sudo reboot` inside the guest:** the guest never finishes going down. The bhyve process stays alive, there's no network, and vm-bhyve logs no exit.

  2. **ACPI shutdown from the host (`vm stop`):** the same. Still "Running" after several minutes.

  3. A guest reboot **before the NVIDIA driver had ever loaded** (right after installing it) worked fine. That was the first hint that the driver was involved.

  4. **`vm poweroff -f`, then `vm start` right away:** `bhyve exited with status 4`. This one turned out to be a red herring: `vm list` still showed "Locked" because the old instance was still being torn down. Wait for "Stopped" before starting again.

  5. **`vm poweroff -f`, wait for "Stopped", `vm start`:** the guest boots, but

```

$ nvidia-smi

No devices were found

```

```

$ sudo dmesg | grep -i -e nvrm -e wpr -e gsp | head -15

NVRM: GPU0 objClInitPcieChipset: *** Unable to get PCI port handles

NVRM: Xid (PCI:0000:00:06): 62, 324f892d 0002e740 00000000 2067ede2 2067f180 2067f0c2 20680598 20680a06

NVRM: GPU0 _kgspProcessRpcEvent: Attempted to process RPC event from GPU0: 0x1022 (RECOVERY_ACTION) during bootup without API lock

NVRM: GPU0 _kgspProcessRpcEvent: Attempted to process RPC event from GPU0: 0x1029 (PMU_HALTED) during bootup without API lock

NVRM: GPU0 kgspSendInitRpcs_IMPL: GMCAPI_CMD_GSP_INIT failed: 0x65

NVRM: GPU0 nvAssertOkFailedNoLog: Assertion failed: Call timed out [NV_ERR_TIMEOUT] (0x00000065) returned from kgspSendInitRpcs(pGpu, pKernelGsp) @ kernel_gsp_gh100.c:1091

NVRM: GPU0 _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP

NVRM: GPU0 _kgspBootGspRm: (the GPU is likely in a bad state and may need to be reset)

NVRM: GPU0 RmInitAdapter: Cannot initialize GSP firmware RM

NVRM: GPU 0000:00:06.0: RmInitAdapter failed! (0x62:0x40:2130)

NVRM: GPU 0000:00:06.0: rm_init_adapter failed, device minor number 0

NVRM: GPU 0000:00:06.0 init failed during probe (ret=-5).

```

Only a full host reboot brought the card back.

(`objClInitPcieChipset: *** Unable to get PCI port handles` shows on every boot, including the good ones. bhyve puts the GPU straight on the guest's bus 0, with no PCIe root port above it. It seems harmless.)

## Troubleshooting

**Does the card support FLR?** On the host:

```

$ pciconf -lc pci0:1:0:0

...

cap 10[60] = PCI-Express 2 legacy endpoint max data 256(256) FLR RO

max read 512

link x8(x16) speed 2.5(32.0) ASPM disabled(L1) ClockPM disabled

...

```

Yes, it does.

**FLR from the host, with the VM stopped:** `devctl reset ppt0` and `devctl reset ppt1` (both silent). Then `vm start`: still "No devices were found".

**A PCIe link reset from the host:** `devctl reset pci1`. That's the bus under the root port, and FreeBSD's bridge reset disables the link and retrains it (a hot reset), saving and restoring the devices' config around it. First we checked that bus 1 holds only the GPU's two functions:

```

$ pciconf -l | grep 'pci0:1:'

$ devinfo -p pci1

pci1 pcib1 pci0 pcib0 acpi0 nexus0

```

Result: no change. The guest dmesg showed `unexpected WPR2 already up` again.

**Research:** this matches the known Blackwell "reset bug" reported for the RTX 5090 and RTX PRO 6000 under Linux KVM/VFIO. GSP firmware sets up a protected region (WPR2) that survives FLR, so the next driver start refuses to boot GSP. Links at the end.

**The idea:** if the guest's driver unloaded properly, GSP would tear WPR2 down itself. On a fresh card (after a host reboot), inside the guest:

```

$ lsmod | grep nvidia

nvidia_uvm 2088960 0

nvidia_drm 155648 0

nvidia_modeset 1556480 1 nvidia_drm

nvidia 16875520 2 nvidia_uvm,nvidia_modeset

$ sudo systemctl stop nvidia-persistenced # fine

$ sudo rmmod nvidia_uvm nvidia_drm nvidia_modeset nvidia # hangs forever

```

From a second SSH session:

```

$ sudo dmesg | grep -i nvrm

NVRM: Xid (PCI:0000:00:06): 56, pid=..., name=(udev-worker), CMDre 00000000 00000000 00000400 00000001 00000000

...

nvRmApiControl+0x46/0x60 [nvidia_modeset]

? nvRmVTSwitch+0x2d/0x40 [nvidia_modeset]

```

So the unload hangs in **`nvidia_modeset`'s VT switch** (handing the console back), in a VM where nothing is attached to the GPU's outputs. The same thing happens on reboot or shutdown. GSP never shuts down, WPR2 stays up, and the next start fails. (The `Xid 56` is a display-engine error, logged as `udev-worker` loaded the display modules at boot.)

## The fix

```

f=/etc/modprobe.d/nvidia-headless.conf

echo 'blacklist nvidia_drm' | sudo tee $f

echo 'blacklist nvidia_modeset' | sudo tee -a $f

echo 'install nvidia_drm /bin/false' | sudo tee -a $f

echo 'install nvidia_modeset /bin/false' | sudo tee -a $f

sudo update-initramfs -u

sync

```

Then one host reboot for a clean card. After it:

```

$ lsmod | grep nvidia

nvidia_uvm 2088960 0

nvidia 16875520 1 nvidia_uvm

```

| Test | Before the fix | After the fix |

|---|---|---|

| `sudo poweroff` in the guest, then `vm start` | hang | card back, about 45 s from poweroff to `nvidia-smi` |

| `sudo reboot` in the guest | hang | card back, about 40 s |

| `vm stop` (ACPI) from the host, then `vm start` | hang | card back |

| host reboot, VM autostarts (`vm_list`) | n/a | card back |

## Notes and caveats

- **Headless/compute only.** No display from the passed-through GPU in the guest. If you need the display modules, this isn't your fix. We didn't test a guest with a monitor on the GPU, where the VT switch might behave differently.

- **Don't hard-kill the VM** while the driver is loaded. That still leaves WPR2 up.

- **After a guest driver upgrade,** check the modprobe file is still there and `lsmod` shows only `nvidia` and `nvidia_uvm`.

- So far this is tested with `nvidia-smi` and repeated restarts.

- **KVM/Proxmox users with Linux guests:** we haven't tested on KVM, but if your guest is headless and you hit the Blackwell reset bug, this might be the same mechanism, and it's cheap to try.

- **For FreeBSD folks:** on the same host, a Maxwell GTX 960 couldn't be initialized in a bhyve guest at all (`RmInitAdapter failed`), as others report for older NVIDIA cards without the community bhyve patches. This Blackwell card with the open/GSP driver needed **no** bhyve patches on 15.1.

## Bonus: the same card on the FreeBSD host itself

Before passing it through, we ran the card on the FreeBSD host (Vulkan, for whisper.cpp). Two gotchas:

  1. **`nvidia-driver-580` hung at the first open:** `nvidia-smi` sat in state `D`, Ctrl-C did nothing, and shutdowns hung after "All buffers synced". The port's pkg message says GSP firmware is **off by default**, and some GPUs don't work without it. We switched to `nvidia-driver` 595.104.02 and added `hw.nvidia.registry.EnableGpuFirmware="1"` to `/boot/loader.conf`, and `nvidia-smi` worked. (Both changed together; GSP is the likely one.)

  2. **Non-root processes couldn't use the card** (root could). The Vulkan loader said `Could not get 'vkCreateInstance' via 'vk_icdGetInstanceProcAddr' for ICD libGLX_nvidia.so.0`, and dmesg said `firmware_get_flags: insufficient privileges to load firmware image nvidia_gsp_ga10x_fw` and `RmInitAdapter failed! (0x61:0x56:2726)`. Without persistence, the driver powers the GPU down when the last client closes and loads the GSP firmware module again on the next open, and only root may load kernel modules. The fix is to keep it loaded: `sysrc kld_list+=nvidia_gsp_ga10x_fw`.

## References

- Tom's Hardware, RTX 5090 / PRO 6000 reset bug: https://www.tomshardware.com/pc-components/gpus/rtx-5090-pro-6000-bug-forces-host-reboot

- NVIDIA forums, RTX PRO 6000 Max-Q FLR failure after KVM guest shutdown: https://forums.developer.nvidia.com/t/rtx-pro-6000-blackwell-max-q-10de-2bb4-flr-fails-on-vfio-pci-release-after-kvm-guest-shutdown-gpu-inaccessible-until-cold-power-cycle/386790

- open-gpu-kernel-modules issue #1442 (stale WPR2): https://github.com/NVIDIA/open-gpu-kernel-modules/issues/1442

- Level1Techs, RTX 50 reset bug thread: https://forum.level1techs.com/t/do-your-rtx-5090-or-general-rtx-50-series-has-reset-bug-in-vm-passthrough/228549


r/VFIO • • 12d ago Bravo

Resource geforce-sriov: Enabling NVIDIA SR-IOV vGPU on consumer Ampere GPUs

Thumbnail
gallery
230 Upvotes

TLDR: 

  • I got NVIDIA SR-IOV vGPU working on consumer GeForce Ampere and Ada GPUs, tested with 8 concurrent Linux and Windows VMs.

  • Unlike previous projects that enabled vGPU on consumer NVIDIA cards using the older mdev approach, this project uses actual PCIe SR-IOV virtual functions to share a GPU between multiple VMs.

  • It's still experimental, currently verified only on my hardware, and requires NVIDIA's vGPU software and licensing.

Source code and documentation: https://github.com/noon-at-cgn/geforce-sriov

Update (Nov 7, 2026): Just released geforce-sriov v0.2.0 beta, now with Ada support! This release also brings a much simpler setup process, improved documentation, and several other fixes.

Update (Nov 4, 2026): Initial testing with Ada on a 4070 Super borrowed from my friend shows promising results, although the method requires quite a few changes and has more limitations. I'll update the post once I've done more testing.

Introduction

I've been experimenting with the hidden PCIe capabilities on NVIDIA GeForce cards, particularly SR-IOV, and managed to get NVIDIA's vGPU stack working on my RTX 3080s using actual PCIe virtual functions.

My goal was to share the GPUs between multiple VMs without dedicating an entire card to each guest. GPU passthrough works well, but it's not ideal when several Linux and Windows VMs only need a portion of the GPU's resources.

The result is 8 concurrent VMs sharing 2 memory-modded RTX 3080 20GB cards on Proxmox, with both Linux CUDA and Windows Direct3D workloads running.

I've open-sourced the implementation and wanted to share how it works here.

Background: SR-IOV vs the older vGPU unlock approach

Projects like DualCoder/vgpu_unlock and mbilker/vgpu_unlock-rs, along with polloloco/vgpu-proxmox, have helped enable NVIDIA vGPU on consumer Maxwell, Pascal, and Turing cards.

These approaches primarily rely on NVIDIA's older mediated device (mdev) architecture, where the host driver manages multiple virtual GPUs behind a single physical PCIe function.

Ampere and newer professional NVIDIA GPUs use a different architecture based on SR-IOV (Single Root I/O Virtualization).

With SR-IOV, the GPU exposes:

  • Physical Function (PF): The main PCIe function responsible for managing the GPU.

  • Virtual Functions (VFs): Additional PCIe functions that can be assigned to individual VMs.

  • vGPU profiles: NVIDIA's software associates a virtual GPU configuration with each VF, including its framebuffer allocation.

Unlike legacy mdev, these VFs are actual PCIe functions enumerated by Linux.

Consumer NVIDIA GPUs still contain SR-IOV-related capabilities in their PCIe configuration space, even though they aren't exposed through the normal capability discovery process.

How it works

The implementation has five main parts.

1. Exposing the hidden PCIe capabilities

PCIe devices have a configuration space, a region of registers that the operating system uses to discover and configure device capabilities. An offset is a location within that space, measured in bytes from its beginning.

Normally, Linux discovers capabilities by following a linked list of entries in PCIe configuration space.

On Ampere, the SR-IOV extended capability exists at offset 0xBCC, but isn't linked into that list, so Linux doesn't discover it.

So, I wrote a small Linux kernel livepatch module that makes the PCI subsystem discover this hidden capability on consumer GPUs. The module also handles related PCIe features:

  • SR-IOV: Exposes the capability at 0xBCC, allowing Linux to recognize the available VFs.

  • ARI: Exposes the hidden capability at 0xC14 and enables ARI forwarding on the upstream bridge when supported.

  • MSI-X: Reveals the hidden MSI-X capability, which allows the GPU to use multiple independently configurable interrupt vectors instead of relying on a single interrupt.

Since Linux discovers PCIe capabilities during device enumeration, geforce-sriov performs a controlled PCI remove/rescan before NVIDIA's driver binds to the GPU. After the rescan, Linux recognizes the GPU's SR-IOV capability.

Great, with SR-IOV exposed, the next step is getting NVIDIA's vGPU stack to use it.

2. Creating the virtual functions

Once Linux recognizes SR-IOV, geforce-sriov uses NVIDIA's own sriov-manage utility to enable the VFs.

On Ampere, the SR-IOV capability reports support for 16 VFs per GPU, but I can currently get only 4 VFs working. The limitation appears to be related to ARI (Alternative Routing-ID Interpretation), which I'll get into later.

For example:

03:00.0   Physical GPU (PF)
03:00.4   Virtual Function 1
03:00.5   Virtual Function 2
03:00.6   Virtual Function 3
03:00.7   Virtual Function 4

These are real SR-IOV virtual functions exposed by the GPU and enumerated as separate PCIe functions, rather than software-mediated devices managed through the legacy mdev approach.

3. Getting NVIDIA's vGPU software to recognize the card

Now we need to make NVIDIA's vGPU driver accept the card. By default, NVIDIA's vGPU software doesn't recognize consumer GeForce GPUs like my RTX 3080s as supported devices.

My solution uses a small userspace identity hook loaded into nvidia-vgpud. The hook presents my RTX 3080 as an RTX A6000, its professional counterpart based on the same GA102 die, allowing NVIDIA's daemon to recognize the card and register the corresponding vGPU profiles.

Once the profiles are registered, we can choose which vGPU type to assign to each VM. These profiles are defined by NVIDIA and determine the resources available to the guest, including its framebuffer allocation.

The RTX A6000 has 23 profiles in the tested driver version. A few examples:

  • RTXA6000-2Q - 2GB VRAM

  • RTXA6000-4Q - 4GB VRAM

  • RTXA6000-8Q - 8GB VRAM (7GB usable framebuffer)

  • RTXA6000-48Q - 48GB VRAM (45GB usable framebuffer)

For my setup, I used RTXA6000-4Q, allocating 4GB per VM. With 4 VFs per GPU, that's 16GB allocated across four guests on each 20GB card.

Note: Workstation and datacenter GPUs typically have more VRAM than their consumer counterparts. For example, the RTX A6000 has 48GB, while my modified RTX 3080 has 20GB. The available profiles aren't automatically restricted to the physical GPU's memory capacity. Four 4Q profiles fit within 20GB, but four 8Q profiles would require 32GB. You need to account for the combined allocations and driver overhead when choosing profiles.

4. Working around a driver scheduler issue

This was one of the biggest obstacles in the project and required quite a bit of reverse engineering of NVIDIA's proprietary vGPU host driver.

With SR-IOV exposed, the VFs created, and the vGPU profiles registered, the guest VMs could boot and recognize their vGPUs. However, the first actual CUDA workload would consistently hang, with no useful errors in the host logs.

After several hours of tracing through the driver with the help of AI, I narrowed the issue down to the GPU's graphics runlist scheduler being disabled. A callback handling a PMU (Power Management Unit) interrupt was repeatedly disabling the scheduler, which prevented guest workloads from executing. I still don't know exactly why NVIDIA's driver disables the scheduler in this situation.

The workaround is a two-byte driver patch that replaces a conditional jump with NOP instructions, making the callback skip the scheduler-disabling call.

geforce-sriov-patch can automatically apply the change to a copy of the user's own NVIDIA driver object. The patched object is then used to rebuild nvidia.ko. With the patch, both CUDA and D3D11 workloads ran successfully.

5. Driver and licensing requirements

A few things to clarify about geforce-sriov:

  • NVIDIA drivers and proprietary binaries are not distributed. The driver patch is applied to the user's own copy.

  • No vBIOS flashing or firmware modifications are required.

  • No vGPU licensing bypass is provided.

You still need to obtain NVIDIA's vGPU host and guest drivers and licensing yourself to use this project.

Tested configuration

ComponentConfiguration
HypervisorProxmox VE
Kernel7.0.14-16-pve
CPUIntel Xeon E5 v4
GPUs2x memory-modded RTX 3080 20GB (GA102)
PCI device ID10de:2206
NVIDIA host drivervGPU-KVM 580.159.01
Virtual functions4 per GPU, 8 total
Guests4 Linux + 4 Windows 11
vGPU profileRTXA6000-4Q (4GB per VM)

Each physical card runs two Linux guests and two Windows guests.

On Proxmox, a VF can be assigned to a VM using:

hostpci0: 0000:03:00.4,mdev=nvidia-527

Note: Proxmox still uses the mdev= parameter to select the vGPU profile, even though the underlying GPU uses SR-IOV virtual functions rather than legacy mdev sharing.

Initial testing

I ran all eight VMs simultaneously with:

  • CUDA workloads in the Linux VMs.

  • D3D11 graphics workloads in the Windows VMs.

  • Four active vGPUs on each physical card.

The test ran for 30 minutes without any observed failures. This isn't enough to establish long-term stability, but it verifies that multiple independent Linux and Windows GPU workloads can run concurrently.

Performance results

I ran the same PyTorch benchmarks in two configurations to measure virtualization overhead with a single active vGPU:

  • Physical GPU: Direct CUDA access from an LXC container on the Proxmox host.

  • vGPU: A VM using a 4GB RTXA6000-4Q profile on the same card.

Both configurations used the same Debian root filesystem, PyTorch environment, four pinned CPU cores, and 8GB of system RAM.

The workloads were FP32 and FP16 matrix multiplication using 4096 × 4096 matrices, plus a 1GB device memory copy.

BenchmarkPhysical GPU (LXC)vGPU (VM)
FP32 matmul20.47 TFLOPS20.54 TFLOPS
FP16 matmul (Tensor Cores)58.35 TFLOPS58.54 TFLOPS
GPU memory copy681.7 GB/s704.4 GB/s

The FP32 and FP16 results were within 1% of direct physical-GPU access, suggesting very little virtualization overhead for these particular workloads.

Some qualifications:

  • Results are medians of three measured runs per configuration.

  • The physical-GPU baseline used NVIDIA's vGPU host driver, not the regular GeForce driver.

  • Only one vGPU was actively using the card during the comparison.

The full methodology and screenshots are available here:

https://github.com/noon-at-cgn/geforce-sriov/blob/main/docs/showcase.md

The ARI problem

The biggest unresolved issue is ARI (Alternative Routing-ID Interpretation), which is currently limiting me to 4 VFs per GPU.

Normally, PCIe uses a 3-bit function number, allowing up to 2^3 = 8 functions (0–7) per device. ARI repurposes the 5-bit device number field to extend the function number to 8 bits, allowing up to 2^8 = 256 functions on a single device.

This matters for SR-IOV because a GPU may need to expose many VFs. On my RTX 3080s, the SR-IOV capability reports support for 16 VFs.

Without ARI, the function layout looks like this:

Function 0   GPU (Physical Function)
Function 1   GPU HDMI/DP Audio
Function 2   Unused
Function 3   Unused
Function 4   VF 1
Function 5   VF 2
Function 6   VF 3
Function 7   VF 4
------------- 3-bit function limit
Function 8   VF 5  (requires ARI)
Function 9   VF 6  (requires ARI)
...
Function 19  VF 16 (requires ARI)

So although conventional PCIe allows eight functions, only four are available for VFs because the GPU's SR-IOV configuration places the first VF at function 4. Functions 0–3 aren't part of the VF allocation.

To expose all 16 VFs on the same device, the GPU needs to support function numbers beyond 7, which needs ARI.

So far, I've managed to:

  • Expose the GPU's hidden ARI capability at 0xC14.

  • Enable ARI forwarding on the immediate upstream PCIe bridge.

  • Confirm that Linux recognizes ARI as enabled.

  • Successfully create four VFs at functions 4–7.

However, the GPU doesn't appear to correctly decode extended function numbers (8 and above), even with ARI enabled in software.

For now, the working configuration is limited to 4 VFs per card, despite the hardware reporting support for 16.

Current limitations

This is still an experimental project, not a general-purpose GeForce vGPU solution yet.

Verified so far:

  • Two memory-modded RTX 3080 20GB cards (10de:2206, subsystem 10de:146d).

  • Proxmox VE with kernel 7.0.14-16-pve.

  • NVIDIA vGPU host driver 580.159.01.

  • Four working SR-IOV VFs per GPU.

  • Eight concurrent Linux and Windows guests.

  • CUDA, D3D11, and single-vGPU compute benchmarks.

Not yet verified:

  • Other Ampere cards.

  • Ada and Blackwell GeForce GPUs are not expected to work with the current implementation (see below)

Other practical limitations:

  • vGPU profiles must fit within the GPU's physical VRAM.

  • GPUs used as the host's boot display require special preparation.

  • Kernel and driver updates may require rebuilding or reapplying the driver workaround.

  • NVIDIA vGPU software and licensing are required.

Ada and Blackwell compatibility

  • The current implementation is not expected to work with Ada and Blackwell GPUs because their vGPU drivers rely on GSP (GPU System Processor), an onboard processor that runs NVIDIA firmware and handles much of the GPU initialization and resource management.

  • Ampere also has GSP hardware, but my configuration uses the older host-driver path, where these operations are primarily handled by the CPU-side driver. On Ada and Blackwell, more of this functionality runs inside GSP firmware, making it harder to modify or intercept from the host.

  • The current driver workaround doesn't apply to that architecture, and Blackwell also requires NVIDIA's open kernel-module path, which the project doesn't yet support.

I would like to experiment more with both generations, but I don't currently have access to Ada or Blackwell GPUs for testing.

Try it out

Disclaimer: geforce-sriov is an experimental, unofficial project and is not affiliated with or endorsed by NVIDIA. Use it at your own risk, preferably on a non-production system.

The full source code, installation instructions, technical documentation, and benchmarks are available:

GitHub: https://github.com/noon-at-cgn/geforce-sriov

Compatibility reports are welcome as compatibility remains largely unverified beyond my tested setup.


r/VFIO • • 13d ago

Support VM IOMMU Passthrough Error

3 Upvotes

These are all my specs for any necessary reference:

  • AMD Ryzen 7 9800x3D

  • Gigabyte X870 GAMING X WIFI7

  • 32GB DDR5 6000

  • Nvidia 2080 8GB

  • Nvidia 1050 Ti (Passthrough Device)

Detailed below is the issue:

In my QEMU setup, I have my GPU passthroughed into the VM, but when I try and inizialize the VM, this error occours:

Error starting domain: internal error: QEMU unexpectedly closed the monitor (vm='win10'): 2026-11-02T22:38:58.440650Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:05:00.0","id":"hostdev0","bus":"pci.5","addr":"0x0"}: vfio 0000:05:00.0: failed to setup container for group 16: Failed to set group container: Invalid argument

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 67, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 101, in tmpcb
    callback(*args, **kwargs)
    ~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 57, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/domain.py", line 1446, in startup
    self._backend.create()
    ~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib64/python3.14/site-packages/libvirt.py", line 1395, in create
    raise libvirtError('virDomainCreate() failed')
libvirt.libvirtError: internal error: QEMU unexpectedly closed the monitor (vm='win10'): 2026-11-02T22:38:58.440650Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:05:00.0","id":"hostdev0","bus":"pci.5","addr":"0x0"}: vfio 0000:05:00.0: failed to setup container for group 16: Failed to set group container: Invalid argument

This is the GREP for IOMMU at boot/VM initialization:

[    0.379018] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    0.397334] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[   15.468791] amdgpu 0000:75:00.0: AMD-Vi: PerfOpt armed on IOMMU0
[   51.116905] pcieport 0000:04:00.0: Firmware has requested this device have a 1:1 IOMMU mapping, rejecting configuring the device without a 1:1 mapping
. Contact your platform vendor.

Detailed below is my setup fore GRUB, IOMMU and relevent systems to make this feature function:

GRUB Config:

GRUB_DEFAULT='6'
GRUB_DISABLE_RECOVERY='true'
GRUB_DISABLE_SUBMENU='true'
GRUB_ENABLE_BLSCFG='true'
GRUB_TERMINAL_OUTPUT='console'
GRUB_TIMEOUT='10'
GRUB_CMDLINE_LINUX_DEFAULT='splash amd_iommu=on iommu=pt amd_iommu=relax_unity'
GRUB_DISTRIBUTOR='Nobara Linux'
GRUB_CMDLINE_LINUX="amd_iommu=on iommu=pt amd_iommu=relax_unity"
GRUB_THEME="/usr/share/grub/themes/sleek/theme.txt"

This is my IOMMU Groups: (IOMMU Group 16 has the devices that I need to passthrough)

IOMMU Group 0:
       00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
IOMMU Group 1:
       00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge [1022:14db]
IOMMU Group 10:
       00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Internal GPP Bridge to Bus [C:A] [1022:14dd]
IOMMU Group 11:
       00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
       00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 12:
       00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 0 [1022:14e0]
       00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 1 [1022:14e1]
       00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 2 [1022:14e2]
       00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 3 [1022:14e3]
       00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 4 [1022:14e4]
       00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 5 [1022:14e5]
       00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 6 [1022:14e6]
       00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 7 [1022:14e7]
IOMMU Group 13:
       01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] [10de:1e87] (rev a1)
       01:00.1 Audio device [0403]: NVIDIA Corporation TU104 HD Audio Controller [10de:10f8] (rev a1)
       01:00.2 USB controller [0c03]: NVIDIA Corporation TU104 USB 3.1 Host Controller [10de:1ad8] (rev a1)
       01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller [10de:1ad9] (rev a1)
IOMMU Group 14:
       02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 [144d:a802] (rev 01)
IOMMU Group 15:
       03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Upstream Port [1022:43f4] (rev 01)
IOMMU Group 16:
       04:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
       05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] [10de:1c82] (rev a1)
       05:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
IOMMU Group 17:
       04:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 18:
       04:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
       07:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 0c)
IOMMU Group 19:
       04:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
       08:00.0 Network controller [0280]: Realtek Semiconductor Co., Ltd. RTL8851BE PCIe 802.11ax Wireless Network Controller [10ec:b851]
IOMMU Group 2:
       00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge [1022:14db]
IOMMU Group 20:
       04:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 21:
       04:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 22:
       04:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 23:
       04:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 24:
       04:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
IOMMU Group 25:
       04:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
       0e:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. OM3PDP3 PCIe 3 NVMe SSD (DRAM-less) [2646:500d] (rev 01)
IOMMU Group 26:
       04:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
       0f:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 800 Series Chipset USB 3.x XHCI Controller [1022:43fc] (rev 01)
IOMMU Group 27:
       04:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
       10:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller [1022:43f6] (rev 01)
IOMMU Group 28:
       11:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM4242 PCIe Switch Upstream Port [1b21:2421] (rev 01)
IOMMU Group 29:
       12:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM4242 PCIe Switch Downstream Port [1b21:2423] (rev 01)
IOMMU Group 3:
       00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
IOMMU Group 30:
       12:01.0 PCI bridge [0604]: ASMedia Technology Inc. ASM4242 PCIe Switch Downstream Port [1b21:2423] (rev 01)
IOMMU Group 31:
       12:02.0 PCI bridge [0604]: ASMedia Technology Inc. ASM4242 PCIe Switch Downstream Port [1b21:2423] (rev 01)
       73:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM4242 USB 3.2 xHCI Controller [1b21:2426] (rev 01)
IOMMU Group 32:
       12:03.0 PCI bridge [0604]: ASMedia Technology Inc. ASM4242 PCIe Switch Downstream Port [1b21:2423] (rev 01)
       74:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM4242 USB 4 / Thunderbolt 3 Host Router [1b21:2425] (rev 01)
IOMMU Group 33:
       75:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Granite Ridge [Radeon 610M] [1002:13c0] (rev cb)
IOMMU Group 34:
       75:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Radeon High Definition Audio Controller [1002:1640]
IOMMU Group 35:
       75:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 19h PSP/CCP [1022:1649]
IOMMU Group 36:
       75:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 3.1 xHCI [1022:15b6]
IOMMU Group 37:
       75:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 3.1 xHCI [1022:15b7]
IOMMU Group 38:
       75:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Ryzen HD Audio Controller [1022:15e3]
IOMMU Group 39:
       76:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 2.0 xHCI [1022:15b8]
IOMMU Group 4:
       00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge [1022:14db]
IOMMU Group 5:
       00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge [1022:14db]
IOMMU Group 6:
       00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
IOMMU Group 7:
       00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
IOMMU Group 8:
       00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge [1022:14da]
IOMMU Group 9:
       00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Internal GPP Bridge to Bus [C:A] [1022:14dd]

My VFIO Config file:

options vfio-pci ids=10de:1c82,10de:0fb9
softdep nvidia pre: vfio-pci
softdep nouveau pre: vfio-pci

———————————————————————————————————————————–

I have seen suggestions to turn of Pre-Kernel DMA Protection, which I would not like to do if possible because I would like to maintain security and I still want to be able to play certain anti-cheat games on Windows.

All suggestions are greatly appreciated.


r/VFIO • • 14d ago

Help! Secondary GPU acting weird after guest VM termination.

Post image
2 Upvotes

I need some help here!

I terminated my virtual machine while it was using PCIe passthrough for my second GPU, now, my second monitor (hooked up to my second GPU) is running at 1 fps.

SYSTEM INFO

HP z840 workstation.
Primary GPU (Slot 1): Nvidia GTX 980
Secondary GPU: Nvidia Quadro K4200

Ubuntu 26.04.1 LTS

$ uname -a
Linux HP-Z840-Workstation 7.0.0-42-generic #42-Ubuntu SMP PREEMPT_DYNAMIC Thu Oct 15 08:06:44 UTC 2026 x86_64 GNU/Linux
$

Software: KVM/QEMU, via virt-manager

Here's the story:

Recently, I spun up my first ever virtual machine, using virt-manager

I also had GPU passthrough going, using this guide from github.

I followed the guide fairly exactly, except for usb passthrough, SSD passthrough, and the ACS override patch.

GPU passthrough worked flawlessly.

I was quite impressed, except for one thing. my secondary monitor, plugged into the secondary GPU, was flickering.

I tried to shut the VM down because I needed to set up USB passthrough, and since I didn't have usb control over it, I force-terminated it. That's when things got yucky.

As far as I can GUESS, the VFIO hook script didnt run. Don't take my word on that, I really have no idea.

The VM terminated, but my secondary monitor never came back up.
It just said "no signal," so I rebooted my computer and still nothing. I finally thought to manually run the VFIO release script

/etc/libvirt/hooks/qemu.d/user/release/end/unbind_vfio.sh

#!/bin/bash

## Load the config file
source "/etc/libvirt/hooks/kvm.conf"

## Unbind gpu from vfio and bind to nvidia
virsh nodedev-reattach $VIRSH_GPU_VIDEO
virsh nodedev-reattach $VIRSH_GPU_AUDIO

## Unload vfio
modprobe -r vfio_pci
modprobe -r vfio_iommu_type1
modprobe -r vfio

I then rebooted and the secondary monitor did come back online, but at somewhere around 1 fps. Mind you, it didn't just refresh cleanly. I don't know how to describe what it was doing, so here is a link to a video.

The list of connected monitors is also showing something odd.
See the attached photo of my secondary monitor showing up as "Unknown 23" "
It used to show up as it's normal name (which I do not remember)

The secondary monitor "Unknown 23" " also stays there when I disconnect the screen from the computer.

This issue persists across displays, ubuntu operating systems, and live images.

I did get the issue to go away by removing the GTX 980 from the computer, not moving the K4200 from it's slot.

I ran dkms status, and it came back with virtualbox-generic something or other. I dont know exactly what dkms does, but from what I read, it sounds like that doesn't matter, as it went away when I uninstalled virtualbox.

nvidia-smi doesn't show my second gpu.
Hwever, lspci does, as well as lshw -c videocommand.

On that note, here is the output from that lshw command:

$ sudo lshw -c video
  *-display                 
       description: VGA compatible controller
       product: GM204 [GeForce GTX 980]
       vendor: NVIDIA Corporation
       physical id: 0
       bus info: pci@0000:04:00.0
       logical name: /dev/fb1
       version: a1
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress vga_controller bus_master cap_list rom fb
       configuration: depth=32 driver=nvidia latency=0 mode=1920x1080 resolution=1920,1080 visual=truecolor xres=1920 yres=1080
       resources: irq:82 memory:a2000000-a2ffffff memory:90000000-9fffffff memory:a0000000-a1ffffff ioport:2000(size=128) memory:a3080000-a30fffff
  *-display
       description: VGA compatible controller
       product: GK104GL [Quadro K4200]
       vendor: NVIDIA Corporation
       physical id: 0
       bus info: pci@0000:84:00.0
       logical name: /dev/fb0
       version: a1
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress vga_controller bus_master cap_list fb
       configuration: depth=32 latency=0 mode=1920x1080 resolution=1920,1080 visual=truecolor xres=1920 yres=1080
       resources: memory:f3000000-f3ffffff memory:d0000000-dfffffff memory:e0000000-e1ffffff ioport:e000(size=128) memory:f4080000-f40fffff
$

It looks like my primary GPU (GTX 980) has the nvidia drivers loaded, so it's acting fine, but the secondary GPU doesn't soI guess that's why it's running at 1 fps.

This is all the info I have; If you need more, don't hesitate to ask.

If this is the wrong subreddit, I would be grateful if someone pointed me to the right one.

Does anyone know what is going on, and how to fix the issue?

Any help would be very welcome, thank you.


r/VFIO • • 19d ago

SR-IOV on Iris Xe integrated graphics is surprisingly good

12 Upvotes

The machine: Dell Inspiron 16 7620 2-in-1, with 64GB DDR4. a 12th gen Alder Lake CPU + Iris Xe iGPU and a potato of a dGPU (2GB MX550). It has a "muxless" setup where the dGPU has no physical outputs, so Looking-Glass is the only way to go.

I mostly use a Windows VM for Fusion360, which requires "some" GPU but isn't particularly demanding. I have been running Virtual Display Driver on the MX550 + Looking Glass and this has been generally usable at 1080p but gets laggy and unstable if I go beyond that.

Recently I tried setting up the SR-IOV branch of the i960 kernel driver. It installed straightforwardly according to the instructions. The trickiest part for Windows setup was using a script to extract a firmware image from Dell's BIOS updater. Another thing to watch out for is blocking certain Linux apps (like Blender) from trying to enumerate and use all the virtualized iGPUs at once.

The surprising part is, the hands-on-the-keyboard feeling is that Windows seems to be way more responsive with the virtual display running on the iGPU. I can comfortably run Fusion at 1440p, whereas when running VDD on the MX550 it got pretty laggy and crashy above 1080p.

On the other hand, every in-VM benchmark I can run says the MX550 beats the virtualized iGPU. All I can figure is these benchmarks aren't capturing what's going on with Looking-Glass -- I certainly see frame drops that aren't being registered by the in-VM benchmarks. I suppose that when the virtual framebuffer lives in system memory, Looking-Glass is doing memory-to-memory copies and this works faster than going over the PCI bus. Handing both a virtualized iGPU and the MX550 over to the the VM does seem to be the most performant option.


r/VFIO • • 23d ago

Windows 11 with VirtIO-GPU 3D Acceleration on QEMU/KVM — No GPU Passthrough!

Post image
19 Upvotes

Built on triton,Neptune virtio gpu and native
Dxvk,d3d11 now work on Radeon 780M and RTX4060.with very good performance


r/VFIO • • 24d ago

Discussion Roast my build

Thumbnail reddit.com
1 Upvotes