Video virtualization via GPU pass-thru for mining cards

[Schol-R-LEA]

I have a series of topics which may call for someone to write one or more how-to wiki pages, should anyone here be a subject-matter expert on the topic.

From most general to most specific:

• What is non-CPU hardware virtualization in general, and what methods exist for virtualizing various common devices in a way that would allow a developer to test their drivers.
• What methods exist for virtualizing GPUs using multiple graphics cards (or an iGPU and a graphics card) driving separate displays
• Is it possible to use windowed pass-through from a guest GPU to display on a primary display connected to a host GPU? (this video demonstrates that pass-through can be done when using a headless Nvida graphics card as the primary GPU passed through an iGPU, but I don't know if the output of a secondary GPU used by a VM can be passed through the main display via an iGPU as a sub-window of the main display.)
• For each of these approaches, how does setting this up for a Intel CPU with an iGPU differ from setting it up for an AMD APU, if at all?
• How do Nvidia GPUs differ from AMD GPUs when used virtualized? Are their any further differences when used via pass-through (if possible)?
• How would this be configured using Windows? Using Linux?
• Assuming that a given motherboard has only one PCIe x16 or x8 slot, would most such graphics cards actually work on a x4 or x1 slot (they are supposed to support it according to the PCIe standard, but...) and would the performance still be sufficient for driver testing purposes?
• Are the GPUs from one or two generations prior (for Nvidia cards, AMD with both dedicated cards and APUs, and Intel iGPUs) similar enough to current-gen card that a minimal driver for the older GPUs would still function with the newer ones, or at least be usable as a stepping stone to supporting them?

What I have in mind follows on from the Linus Tech Tips video linked to above. I intend to do some research on the problem RSN myself, but it is a large enough set of problems that I thought I would pose them to others to see if anyone else thinks this is worth pursuing. It doesn't look like something a single dev would be able to crack, unless they already have some experience with video virtualization, and I am not sure who if anyone here has applied that.

My thought is that for driver testing, it should be possible to use a second GPU - whether a pair of cards, or a card and an iGPU/APU - as the GPU of a virtual machine, as has been occasionally been discussed before. There are a large number of older cards on the used market, both standard cards and dedicated mining cards, any of which ought to be suitable for this use, though not necessarily ideal. However, AFAICT this would usually require the dev to have a second monitor for the virtualized system (or a KVM to multiplex a single monitor, or something similar). This could present issues with cost for a lot of people here, and would in many instances be less convenient as well. Thus, I wanted to see if pass-thru would be an option for running the video of a VM.

Also, while I haven't checked recently, the last time I looked the headless mining cards were considerably less expensive than the the regular ones, following the steep decline in mining operations about a year and a half ago. If this is still true, then a headless card could be a great way to test video drivers on the cheap, if the pass-thru was a feasible method.

Any thoughts?

For reference, my own system as of today is:
Ryzen 3 2200g APU
16 GiB 2400 DDR4
MSI B350M Pro-VDH motherboard
250GiB NVMe SSD (currently disabled for annoying reasons, I mean to take care of that soon)
1TiB WD Blue HDD
Corsair CX 650W 80+ Bronze Modular PSU
Manjaro Linux, Kernel 5.5.13-1
XFCE 4

While I do intend to upgrade relatively soon (I bought this system in late 2018 with the intent of doing so as money became available, which it so far hasn't, though I did replace the PSU that came with the case),

[Octocontrabass]

What methods exist for virtualizing GPUs using multiple graphics cards (or an iGPU and a graphics card) driving separate displays

The only one I know of is IOMMU, which may be called Intel VT-d, AMD-Vi, or something else on different CPUs.

Is it possible to use windowed pass-through from a guest GPU to display on a primary display connected to a host GPU? (this video demonstrates that pass-through can be done when using a headless Nvida graphics card as the primary GPU passed through an iGPU, but I don't know if the output of a secondary GPU used by a VM can be passed through the main display via an iGPU as a sub-window of the main display.)

From what I understand, this form of passthrough is accomplished by copying the framebuffer from one GPU to the other, with no special hardware support. That makes it easy to generalize across different GPUs (just take the buffer and push it somewhere else), but difficult to virtualize (the guest owns the buffer).

Assuming that a given motherboard has only one PCIe x16 or x8 slot, would most such graphics cards actually work on a x4 or x1 slot (they are supposed to support it according to the PCIe standard, but...) and would the performance still be sufficient for driver testing purposes?

Generally yes, most GPUs will work that way, so you can develop a working driver. However, you may want to perform benchmarks that involve pushing a lot of data to or from the card, and for that you'll need the x16 slot.

[eekee]

Just for completeness:

• What is non-CPU hardware virtualization in general, and what methods exist for virtualizing various common devices in a way that would allow a developer to test their drivers.

The baseline method is just emulation. For instance, KVM uses Qemu to emulate IO hardware. Pass-through is, of course, the other option. (VirtIO is also available, but that won't help you develop drivers for hardware.)

I've never heard any complaints about Qemu's IO hardware emulation. I have heard complaints about Virtual Box's; some drivers developed on real hardware wouldn't work in VBox, and this changed with 'patchlevel'. It went something like this: at vbox 3.0.20 everything worked; at 3.0.34 something stopped working; at 3.0.52 that thing worked again but something else didn't. (Actual numbers inexact.) I don't know if vbox still has this problem, I suppose it depends on whether they test properly these days.

[ggodw000]

i work at amd now exactly in gpu virtualization. but being only 1 year there, just only scratching the surface. there are tons of thing involved in gpu virtualization and mind boggling.
thousands of registers, several dozens of clocks.
Good thing about AMD gpu virtualization over nvidia is AMD completely opensourced its virtualization driver where you can look there
https://github.com/GPUOpen-LibrariesAnd ... ualization
However unfortunately it looks more than a year old, not sure when they will update the repo.