OSDev.org
https://forum.osdev.org/

PCIe MSI Interrupts directly into Userspace with IOMMU
https://forum.osdev.org/viewtopic.php?f=1&t=57111		 Page 1 of 1
Author:  nxtv2 [ Sat Feb 10, 2024 8:37 pm ]
Post subject:  PCIe MSI Interrupts directly into Userspace with IOMMU
I am currently writing a userspace driver to interface with an Altera PCIe card. My goal is to reduce max interrupt latency as much as possible.
I am currently using VFIO_DEVICE_SET_IRQS ioctl to register an MSI interrupt with an Eventfd, then read to wait on it.

If my understanding is correct, this will syscall into the kernel and wait. When the Board sends the MSI, the IOAPIC will signal the correct IRQ which should jump to kernel code and signals the kernel to schedule the userpsace process to return from the read.

Is it possible to configure the system such that i halt the CPU core from the userspace process and the MSI is handled by that same CPU core, and continues execution or call a "handler"? Without going into the kernel and waiting on a select, poll or read.

I imagine this needs x2apic mode or Intel posted interrupts, but i am not sure how to do it. Any help would be appreciated.
Author:  nullplan [ Mon Feb 12, 2024 10:19 pm ]
Post subject:  Re: PCIe MSI Interrupts directly into Userspace with IOMMU
nxtv2 wrote:
I am currently writing a userspace driver to interface with an Altera PCIe card. My goal is to reduce max interrupt latency as much as possible.
Well, those two go together like oil and water. Userspace drivers are for things where latency doesn't really matter. Truthfully, the latency for the system call shouldn't matter all that much, but if speed is your goal, the fastest will be a direct interrrupt handler in the kernel.
nxtv2 wrote:
Is it possible to configure the system such that i halt the CPU core from the userspace process and the MSI is handled by that same CPU core, and continues execution or call a "handler"? Without going into the kernel and waiting on a select, poll or read.
No, and it would be pretty unsafe if it was possible. This is basically disabling the operating system during the wait for the interrupt. So if the interrupt never comes, you can now not get out of the program. You cannot go to another terminal and kill your process, because you've disabled the OS.

On a more technical level, hardware interrupts are always taken at CPL 0, so to run userspace code requires at least one transition. Unless you'd like to run userspace code at CPL 0, again disabling the OS.
Author:  nxtv2 [ Tue Feb 13, 2024 5:05 am ]
Post subject:  Re: PCIe MSI Interrupts directly into Userspace with IOMMU
Thanks for the help.

I am using a quad core CPU in an embedded real time Linux system, so worst-case latency is the important metric. The context switch between userspace to kernel and back takes a bit of time. Userspace on its own is more deterministic, since i use `no_hz` option and `isolcpus` to isolate that core from the kernel.

I have to write the driver in userspace since it uses libraries that would be too difficult to port to kernel.

Is there a way to execute hardware interrupts in CPL 3 possibly with the help of the IOMMU? My understanding is that the IRTE in the IOMMU allows a VM to service interrupts without doing a VM_EXIT, so the CPU never goes to CPL 0 for the host kernel. It stays in the guest OS.
Page 1 of 1 All times are UTC - 6 hours
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group
http://www.phpbb.com/