I/O-APIC init

Question about which tools to use, bugs, the best way to implement a function, etc should go here. Don't forget to see if your question is answered in the wiki first! When in doubt post here.
geppyfx
Member
Member
Posts: 87
Joined: Tue Apr 28, 2009 4:58 pm

Re: I/O-APIC init

Post by geppyfx »

There is a possiblity of 32 "devices" on each bus and each "device" can have 4 pins A,B,C,D. Combination of pci bus,device,pins (around 2baits) gives you unique pci device. Each unique pci device can be mapped into every single IOAPIC ID & into every single IOAPIC IRQ within that iopaic id. That's worst case in terms of memory usage that you should be able to handle.

I think think that this is minimum data structure to map pci device to ioapic int:

Code: Select all

struct ioapicPCIbus_t {
uint8t pciBus;
uint8t pciDevice;
uint8t ioapicID;
uint8t ioapiqIRQ;  //aka pin
};
You would have a dynamic size collection of these 4 bait blocks. You can arrange it by pciBus OR/AND by ioapicIRQ OR/AND by ... .

Also, ACPI maps each pci device into Global Interrupt Number which is a DWORD. I am not sure how many IOAPICs they intended to cover with this. More than 256? Any clarification?

Perosnally I decided to limit myself to 16 IOAPICs and I have datastructure similar to this:

Code: Select all

db pciBus
db pciDevice
dw ioapicMask
db ioapiqIRQ
erch bit in ioapicMask represent diff ioapic. If a device routed to the same ioapic irq on different ioapics I would have single entry. From my reading the forum I concluded that this can happen on servers. If device routed to different irqs then I unfortunately have multiple entries.
FlashBurn
Member
Member
Posts: 313
Joined: Fri Oct 20, 2006 10:14 am

Re: I/O-APIC init

Post by FlashBurn »

Now the things becoming really confusing.

I thought that every PCI bus is connected to only 1 IO-APIC and all device (which aren´t able to use MSIs) can only use this IO-APIC. As I mentioned earlier, you can update/change the IO-APIC input pin a PCI int is using (but if you do that in the PCI cfg space, then you have to do it for all device which use the same PCI int on the same bus). So I only need to know which PCI bus is connected to which IO-APIC and then I can read the input pin from the PCI cfg space.

Another problem occured as I was rewriting my code. I do support default configurations of the MPS, but there is anywhere mentioned to which input pin the PCI ints are connected (or did I miss this?). So my above assumption would make my life easier.
Cognition
Member
Member
Posts: 191
Joined: Tue Apr 15, 2008 6:37 pm
Location: Gotham, Batmanistan

Re: I/O-APIC init

Post by Cognition »

I think this kind of touches on what I was asking earlier, in the ACPI specification you have the ability to access the PCI routing table which essentially tells you the global interrupt vector each pin for a PCI slot can possibly be connected to. From that data you could determine the IOAPIC itself and which line on it is used. The MPS specification seems to do this through the I/O interrupt entries, which have a special format for them defined in section D.3. The _PRT routing table is explained in section 6.2.11 of the ACPI Specification as of version 3.0a, it's important to note that this is in the ACPI namespace, which would require you to have a functioning AML interpreter and ACPI namespace in your kernel.
Reserved for OEM use.
FlashBurn
Member
Member
Posts: 313
Joined: Fri Oct 20, 2006 10:14 am

Re: I/O-APIC init

Post by FlashBurn »

I would need an AML interpreter in my loader and I think this will never happen.

I finished all the init and dynamic allocation of irq handlers stuff, but maybe I come across a bug in qemu. I would need someone on Linux or better someone who compiled qemu for himself to prove it. On qemu the IO-APIC ID is "0" and this is wrong, according to the MPS and ACPI tables it should be (like it is in bochs) 2 for 2 dual cpu system. I think they forgot to init the id.

Another question would be, if I can assume that there are entries for all ISA irqs in the MPS? I know that bochs/qemu haven´t got any PCI routing info in their MPS tables, but on a real PC there should be?
User avatar
Brendan
Member
Member
Posts: 8561
Joined: Sat Jan 15, 2005 12:00 am
Location: At his keyboard!
Contact:

Re: I/O-APIC init

Post by Brendan »

Hi,
FlashBurn wrote:
Brendan wrote:Sounds like a good idea, just don't forget that the same IRQ can be shared by multiple devices (a simple "device_ID = interruptTable[IO_APIC_input_number]" array won't work) and the same device can have multiple interrupts (a simple "IO_APIC_input_number = interruptTable[device_ID]" array won't work either). ;)
Oh, then it could happen, that there are more than 1 entries in the MPS for 1 device?!
As far as I know, for a multi-function device you can have 8 "functions" and each "function" can use one of the PCI bus interrupt lines (so, each of the 4 PCI interrupts lines are used twice). Each "function" can only use one interrupt line. There's also MSI - each of those "functions" can have one (or more?) MSI interrupts too, but normally you'd use the PCI interrupt lines or MSI (and not both at the same time).

That only applies to PCI devices though. For a "master list of interrupts" you also need to consider legacy devices (e.g. some old sound cards use multiple ISA interrupts) and other "non-PCI" devices (e.g. HPET, which often supports 2 IRQs but may support more).

Mostly what I'm thinking is that you need a hierarchical tree of "device structures' which includes a list of interrupts that each device is using (if any); plus a linked list for each possible interrupt vector where each entry in the linked list has a pointer to the "device structure" for the device.

For example, you'd be able to do something like:

Code: Select all

    DEVICE_STRUCT *device;

    for(interrupt_number = 32; interrupt_number < 256; interrupt_number++) {
        device = global_interrupt_list[interrupt_number];
        while(device != NULL) {
             printf("Int %u = device %s\n", interrupt_number, device->name);
             device = device->next;
        }
    }
FlashBurn wrote:Could you tell me on which addresses are your 2 IO-APICs on the pc that has 2? I´m asking, because I need to know if I could assume that every IO-APIC is at an 4KB address and that 1 IO-APIC "uses" a whole 4KB page.
From memory, I think they are at 0x0FEC00000 and 0x0FEC01000. The actual address of each I/O APIC should be determined from the MP specification tables or the ACPI tables. As far as I know all I/O APICs always consume 4 KiB of space.
geppyfx wrote:Also, ACPI maps each pci device into Global Interrupt Number which is a DWORD. I am not sure how many IOAPICs they intended to cover with this. More than 256? Any clarification?
For 80x86, you can't have more than 256 interrupts. Other architectures may support more interrupts though, and both PCI and ACPI may be used on other architectures that do support more interrupts.

In theory, this is a "per CPU" limit. For example, you can have multiple PCI host controllers where the interrupts for each PCI host controller is sent to a specific CPU (or group of CPUs) and each CPU (or group of CPUs) use different IDTs. In this case you'd have up to 220 IRQs per PCI host controller. With 4 PCI host controllers you'd be able to have about 880 completely independent IRQs. However, ACPI will still only use global interrupt numbers from 32 to 255, and it'd be up to the OS to determine which CPUs receive which IRQs, etc.
FlashBurn wrote:Another problem occured as I was rewriting my code. I do support default configurations of the MPS, but there is anywhere mentioned to which input pin the PCI ints are connected (or did I miss this?). So my above assumption would make my life easier.
For some ancient computers, the PCI interrupts are connected to a "PCI interrupt router" and mapped to ISA interrupts (just like modern computers when they're using the PIC), and (unlike modern computers) only the ISA interrupts are connected to the I/O APIC. For default configurations you can assume this is the case (and that the "global interrupt number" for each PCI device is the same as the "PIC interrupt number" stored in the device's PCI configuration space by the BIOS).


Cheers,

Brendan
For all things; perfection is, and will always remain, impossible to achieve in practice. However; by striving for perfection we create things that are as perfect as practically possible. Let the pursuit of perfection be our guide.
FlashBurn
Member
Member
Posts: 313
Joined: Fri Oct 20, 2006 10:14 am

Re: I/O-APIC init

Post by FlashBurn »

As I just found out, by testing my code on my pc the real pc shows the same behavior like qemu. I would assume (I think most others too) that if I get an IO-APIC in the MPS/ACPI tables and there stands IO-APIC ID is e.g. 4 that the ID I get from the IO-APIC is the same. But this isn´t so. So I will try now to save the ID I get from the tables into the IO-APIC(s). This will hopefully make my code run on my pc and qemu.

It would be nice if others could also look if on their PCs the IO-APIC also has not the value which stands in the tables (IO-APIC ID= logical cores).

Edit::

So I don´t change the IO-APIC ID in the IO-APIC, because this only works if there are not more than 15 cpus (logical). So I let the ID as it was, but I save the ID which stands in the tables and now my code is working :D

I also found out that my pc´s bios says that the IO-APIC has the ID 4 (MPS tables) or 2 (ACPI tables). I have to investigate if the ACPI tables are wrong or if they use really this value.

Edit2::

If someone please could look if the IO-APIC IDs are different between MPS and ACPI tables it would be really cool, because ACPI tables don´t use the ID to address the IO-APIC, but they use the global int to do that. The problem I have here is that I use the ACPI tables for finding cpus and MPS tables for initing the IO-APIC, but if the IDs differ my code will fail or I will have to add code so that I search the IO-APICs in the MPS tables and update the ID in my structures.

Edit3::

Why are the ISA redirection tables of MPS and ACPI different? It seems in the MPS stand only these irq which are used (I mean where an ISA device is connected) and the ACPI tables tell you all redirections, although they are not in use by an ISA device. So maybe it would be better to only use the MPS for that, because so I know which input pins are free to use for PCI devices.
User avatar
Brendan
Member
Member
Posts: 8561
Joined: Sat Jan 15, 2005 12:00 am
Location: At his keyboard!
Contact:

Re: I/O-APIC init

Post by Brendan »

Hi,
FlashBurn wrote:As I just found out, by testing my code on my pc the real pc shows the same behavior like qemu. I would assume (I think most others too) that if I get an IO-APIC in the MPS/ACPI tables and there stands IO-APIC ID is e.g. 4 that the ID I get from the IO-APIC is the same. But this isn´t so. So I will try now to save the ID I get from the tables into the IO-APIC(s). This will hopefully make my code run on my pc and qemu.
That's annoying, but as far as I know the I/O APIC's "APIC ID" isn't actually used for anything (by software or by hardware). A few days ago I saw a comment in Intel's documentation for the 5520 chipset, which said "the I/O APIC ID doesn't do anything and exists due to FUD" (and yes, Intel did say "FUD" - I was quite surprised).
FlashBurn wrote:Why are the ISA redirection tables of MPS and ACPI different? It seems in the MPS stand only these irq which are used (I mean where an ISA device is connected) and the ACPI tables tell you all redirections, although they are not in use by an ISA device. So maybe it would be better to only use the MPS for that, because so I know which input pins are free to use for PCI devices.
For ACPI, there's normally an extra ISA interrupt listed that's used for ACPI's SCI. This interrupt won't be listed by the MP specification tables.

Underneath all the ACPI stuff there's a system controller in hardware. When the computer is running in "legacy mode" different events cause the system controller to generate an SMI ("System Management Interrupt"), which makes the CPU run the BIOS's SMM code. When the OS enables "ACPI mode" these same events cause the system controller to generate an SCI ("System Control Interrupt") instead, which is a (mostly) normal IRQ that the OS handles by using ACPI's AML code. On most of the computers I've looked at, the SCI is mapped to ISA IRQ 9 (but that's just coincedence).

As far as I know, except for the extra SCI, ACPI's tables should tell you the same as the MP specification tables for ISA IRQs (ignoring the fact that ACPI won't mention ISA IRQs that are mapped in a standard way - e.g. "IRQ n -> I/O APIC input n" as edge triggered).


Cheers,

Brendan
For all things; perfection is, and will always remain, impossible to achieve in practice. However; by striving for perfection we create things that are as perfect as practically possible. Let the pursuit of perfection be our guide.
Post Reply