Since a new video game was coming out, but Linux version wasn't coming until much later, and it wouldn't play in Wine, I've decided to use Windows to play it. That of course meant using VM GPU passthrough. My setup was good for that to start with - ASRock 99FX board and AMD FX CPU - so I didn't had to basically buy another computer to do it. However, along the way, I've encountered what seemed like every single possible problem.
VM dropped me into UEFI console
I've had this problem making me just scratch my head for a while. I was trying to follow the advice to use SCSI hard disk driver, and as it said, I would also mount the Virtio drivers disk - and then the UEFI would fail to pick up a boot drive. The solution was to simply install it like normal and do the fancy hardware tricks later.
VGA driver loaded before VFIO
I'm running Ubuntu and it has kernel modules load automatically, and more often than not you will not be able to change the order in which they're loaded through conventional means. The solution was to use soft dependency, I've created a "radeon.conf" file and put "softdep radeon pre: vfio-pci" into it.
Having anything other than literally "host-passthrough" as CPU model sometimes caused "CPU feature missing"
Yes I know that all the walkthroughs say to use that specifically, but I figured since the libvirt GUI had the "pass host model" checkbox, that would be fine. Looking up the error suggested to use one of QEMU virtual CPU models, that solved the problem but then it came back again, when I finally got to edit the XML file over virsh I'd see that it uses "host-model" as the CPU, setting that to "host-passthrough" fixed the problems.
Adding command line devices caused PCI slot collisions
I've only added Samba file sharing NIC through command line so only it caused problems: you need to pass a network adapter together with the SMB user network, because latter is outside of the normally created network adapters' pool and they cannot see it. A device added over command line automatically uses up a PCI slot, but then some other device tries to use the same slot. I've never found a good solution, but you can manually edit your VM XML settings file and change the PCI slot of the offending device to something else. Changing anything about the device in GUI will immediately undo it.
Sharing input via linux-input cases Permission Denied errors
There could be two reasons and they could be both causing problems, simultaneously. The first is simply the fact that kvm process doesn't have permission to access these /dev/input files. I edited the qemu.conf file and aded myself as a user and "kvm" as group, and added myself to the "input" group, and added these devices into the list. It still didn't work, turns out you need to re-login before these changes take effect. It still didn't work; I've checked input device permissions and turns out everything in /by-id/ and /by-path/ is owned by root, so I had to resolve the symlinks and put the /dev/input/eventXYZ in there instead. That still didn't work, because AppArmor was stopping libvirtd from accessing these; after setting it into "complain" mode it finally started working.
Audio stuttering and crackling
That was a super common problem as per my research, but it turns out that the root of the problem is that the buffer size is not sufficiently large for the low frequency it's being read at. Simply increasing the buffer size does the job. I'm using PulseAudio so I was still stuck with gigantic input lag, so I thought of buying a soundcard to bypass it. I could have used a custom build but for the time being I've decided against it.
The linux-input devices are not infallible
So I decided to play some games and few minutes in, the mouse stopped responding. It worked just fine on the host and I could switch back and forth with no problem, just by pressing both control keys. The mouse was stuck and required a complete VM reset to get it working again. It became clear that I'd had to buy another set of inputs to pass through, because having middleware sitting in between, increasing latency and reducing reliability for no good reason, didn't stuck me as a good idea.
Conclusion
At that point it was basically a second computer, parasiting on my main computer. So I just bit the bullet and bought an low end PC, plugged into it my host GPU and host hard drive, and now that's my Linux machine. The original machine, now lacking HDD but having an SSD in place, is now a Windows gaming machine.
This is a fascinating technology and a significant milestone in emulation, but as far as gaming goes, it's just impractical. I can live with small audio problems and with 5 FPS drop, but not with huge random stutter every so often, and not with the Linux machine is left in semi-usable state while VM is running, not to mention possible security problems. It's much more practical to just buy a second computer. It will only be a small fraction of the price of the gaming machine, and both computers are untainted by each other's presence.