Bochs, KVM, mptable, and PCI

[PaulP]

A quick parsing of the mptable on both Bochs and KVM show that no PCI bus is listed. (Only 1 entry of type 1 and it is ISA.) Yet, on both machines, the PCI bus is in fact enabled. In Bochs for example, it was compiled with --enable-pci, and several pci devices are listed in the runtime configuration.

A scan over the PCI bus yields numerous PCI devices, and there are many interrupt (type 3) entries in the mptable, which seem to belong to these PCI devices.

I've read several posts on this forum that speak to this, basically saying this is how Bochs does it. It also makes sense that since KVM now uses the Bochs bios that KVM behaves the same way.

What I'm wondering about is if this is actually a a valid configuration, or is the mptable support simply broken. I've written some code to deal with the situation, but right now it is a simulation only occurence. Can a system be built in such a way that the PCI interrupts are all routed through the ISA bus, and listed as ISA interrupts in the mptable.

I'm fine residing this to a lack of Bochs support, but I want to make sure I'm not making improper assumptions about the two buses and mptables.

Thanks. And thank you all for the very informative wiki and forum. The information here has been invaluable.
-Paul

[geppyfx]

My bochs 2.4 precompiled for windows has 16 entries of type 3, all ISA; and 1 entry of type 0 - CPU. 1 entry - type 2. I believe that this is a bug because there is no ExtINT entry. You have no choice but to use PIC only. Maybe bochs has complete ACPI. I haven't deal with pci devices in acpi yet because it requires parsing AML.

But, I have EPOX EP-8KDA3J which has exactly the same MP table except it has ExtINT which makes possible to use mixed mode. I consider this config to be completely valid although limited. (the motherboard also has LINTs entries)

[PaulP]

Well I'm actually currently running code that is bypassing the PIC for IRQ0 and 1 (both ISA IRQ's). I directly handle these via the IOAPIC, based solely on the information in the mptable. So I know operating without the PIC is definitely possible.

What I'm not sure of yet is if the PCI devices can be used without the PIC. My gut tells me the entries in the mptable that claim of be of type ISA, are in fact of type ISA, and the IRQ field specifies the IRQ the PCI devices report. There is an ISA entry with the correct IRQ for each IRQ that a PCI device claims to own (according to the PCI bus). My guess (still untested) that that each of these interrupts will work through the IOAPIC based on the id and intin fields, as edge triggered active high (standard isa interrupt).

My main concern is that this isnt some Bochs strangeness, but rather an actual valid configuration where somehow pci devices are routed through isa interrupts.
-Paul

[geppyfx]

Well I'm actually currently running code that is bypassing the PIC for IRQ0 and 1 (both ISA IRQ's). I directly handle these via the IOAPIC, based solely on the information in the mptable. So I know operating without the PIC is definitely possible.

Certainly possible, you can redirect any ISA irq to IOAPIC because there are MP entries present for this .

What I'm not sure of yet is if the PCI devices can be used without the PIC.

No you can not. There are no entries for PCI devices. There are 2 choices: 1) use only ISA devices with IOPAIC and stay away from pci, or 2) use PCI & ISA devices with PIC only.
You can always use MSI if device + pci bus + motherboard (all 3) support msi.

My main concern is that this isnt some Bochs strangeness, but rather an actual valid configuration where somehow pci devices are routed through isa interrupts.
-Paul

This is bochs bug and its not valid config. You could try to make pci devices work with ioapic but only by probing each pci device (not recommended).

[PaulP]

No you can not. There are no entries for PCI devices. There are 2 choices: 1) use only ISA devices with IOPAIC and stay away from pci, or 2) use PCI & ISA devices with PIC only.

My point was while there are no entries in the mptable for the PCI devices, there are ISA entries that claim to share the same IRQ number as the PCI devices. I've also seen other posts from users that state the PCI interrupts are routed as ISA interrupts. The entries give enough IOAPIC information to allow their routing. I just haven't been able to generate a PCI based interrupt in bochs yet, to test this fully.

You could try to make pci devices work with ioapic but only by probing each pci device (not recommended).

I'm curious how this would work? Are you able to probe the PCI devices in some way to get the IOAPIC routing information? I thought the only way to find out how to get the PCI devices to work with the IOAPIC was via the mp or acpi tables.

Thanks.
-Paul

[geppyfx]

You could try to make pci devices work with ioapic but only by probing each pci device (not recommended).

I'm curious how this would work? Are you able to probe the PCI devices in some way to get the IOAPIC routing information? I thought the only way to find out how to get the PCI devices to work with the IOAPIC was via the mp or acpi tables.

You set polarity & trigger on ioapic input XX and do something with pci device that would trigger an interrupt. No interrupt - no device. You change polarity & trigger, try again, then change ioapic input # ... . Once again its not recommended but is possible in theory. I obviously have no desire to get into this mess.

[geppyfx]

there are ISA entries that claim to share the same IRQ number as the PCI devices

How did you figured out that they 'claim to share'?

The entries give enough IOAPIC information to allow their routing.

Would you not need different polarity & trigger for pci?

[PaulP]

How did you figured out that they 'claim to share'?

This was based off the mptable entry for that given ISA Interrupt saying its IRQ matched the IRQ of a PCI device, according to the PCI bus.

So I've spent some more time on this (for those who may be interested) and I've actually gotten PCI interrupts through the IOAPIC functioning in Bochs.

PCI interrupts are in fact routed to the IOAPIC, they just aren't reported in the mptable. Moreover, if a PCI device claims to operate on IRQ X, and there is an entry in the mptable for ISA IRQ X, that entry in the mptable is not valid, and the pci interrupt is NOT routed to that entry.

For example, I'm running on a system in which the mptables report that ISA IRQ 11 is routed to IOAPIC entry 11. However, my PCI device also reports IRQ 11. When I probe the IOAPIC, interrupts are actually being routed to IOAPIC entry 18 (and the mptables do not even report an entry 18).

If Bochs is compiled with BX_SUPPORT_APIC, PCI interrupts are routed to IOAPIC entries 16-19, relating to PIRQA-D (A->16, B->17, etc). There is some funky math in the Bochs code that takes a device number and pin (A-D), and manipulates this to form a PIRQ. I believe this particular form of modification is specific to the bochs PCI setup, not a given standard.

Another unfortunate situation is that although the PCI interrupts are going to the IOAPIC, they are being delivered as ISA (edge triggered) style.

I'm current writing some ugly hacky code for our OS that attempts to detect this ugly situation, and tries to compensate.

Does anyone know if the Bochs acpi tables provides better support for this? I haven't had a chance to pull that part of bochs apart yet. Does it tell you where the PIRQ's are routed, or what device maps to what PIRQ? Even if it did that, I fear the fact that the interrupts are delivered ISA instead of PCI style is a major issue for proper (clean) detection.
-Paul

[PaulP]

I've discovered some additional information that explains some strange pci / ioapic related behavior.

It appears that there is repeated code in the PCI / PIC IRQ code sections of Bochs. As of now, if you raise a PCI based IRQ X, it will route that IRQ to IOAPIC Entry X, as well as IOAPIC Entry 16 + PIRQ.

If you examine the code in iodev/pci2isa.cc and iodev/pic.cc, you'll see that both the pic raise_irq() function and the pci2isa set_irq() function both make the same call to the IOAPIC's set_irq function. The PIC code raises the IRQ on IOAPIC entry X, and the pci2isa set_irq code raises it on 16 + PIRQ. The code blocks (and comments) are identical, with the exception of a single change in an if condition. This may be a copy paste issue. This causes a single interrupt to arrive at multiple locations inside the IOAPIC.

This would appear to be a bug (this does not seem like a valid configuration), so I'm reporting it to Bochs.
-Paul