IO-APIC info

[01000101]

Hey, I have been thinking about implementing smp-distributed interrupts through the IO-APIC to act as a load-balancer. Because my OS is tied to a small target hardware range, I have read that it would not be all that necessary to parse the MP/ACPI structures as I would not need to check for hardware differences. Is there any documentation that describes the programming of the IO-APIC to distrubute interrupts across multiple CPU's? Would the benefits of this cascading interrupt handling outweight the benefits of the latency reduction from polling?

I have already setup the L-APIC and can send the startup IPI's to the dormant AP's to wake them and set them up into LMode, but right now everything is polling based.

Does the IO-APIC still require the normal IDT/ISR setup? or is that what the LVT is for (routing interrupts)? I have found very little info on the IO-APIC in the Intel manuals, but quite a bit on the L-APIC, that is why I'm having issues with setting up the environment. Pseudo-code would really help me out with this as I am pretty clueless on how to get the basic environment setup for distributed interrupts.

Thanks.

[Laksen]

I tried to dig out some information about the IO apic. I'm about at the same position as you.

I parse the SMP table or ACPI table to find IO Apic entries
If you want to enable IO apics you have to set the correct wire mode. This was very hard information to find and I can't remember what the bits names were anymore. The following should be correct according to some specification I've long lost:

Code: Select all

   outportb(0x22, 0x70);
   outportb(0x23, 0x1);

I also read somewhere in a specification that you have to mask all interrupts on the PIC.

I have yet to find out how to set up the entries in the IO Apic itself so I haven't been able to test anything of this yet

Info:
Specification for Intel's 82093AA:
http://www.datasheetcatalog.org/datashe ... Xtwstt.pdf

[bewing]

I don't know how much help it will be, but Brendan put a heck of a lot of APIC info (including for IO APIC) on this thread:
http://forum.osdev.org/viewtopic.php?f= ... 08&start=0

[01000101]

I had already read through that thread, and while it has some very detailed explanations, it seems to dive to deep into things to quickly without giving a top-level overview of the basic setup. I think that once I understand the basic explanations, I will be able to pick up the details quickly.

What I have now is:
* sets up eth cards
* parses the MP tables
* enumerates all processors and io apics
* parses the first 3 offset registers in the (in my case 2) io apic(s)
* enable the Local APIC (both msr 0x1b and through the spurious-interrupt vector register)
* sends the INIT and SIPI to the AP's
* AP's go through the trampoline->ap_boot code and begin polling eth cards

I believe I have the basic environment for a (hopefully) pain-free way to setup interrupts. I can grab (preexisting) idt/isr code quickly, but I'm not sure how to connect that with the io-apic. I'd really like to get an overview of what I need to accomplish to meet my goal of distributed interrupts. If that can be explained with respect to my preexisting environment, that would be excellent as I am running into troubles finding good information on the steps needed to setup interrupts through the io-apic.

[Brendan]

Hi,

I'd really like to get an overview of what I need to accomplish to meet my goal of distributed interrupts.

Here's a list of things for you to do:

• Parse the MP specification tables and/or ACPI's "MADT" table to find out how many I/O APICs are present and how many inputs each of these I/O APICs have.
• Design a structure that describes an I/O APIC input (e.g. edge triggered/level triggered, active high/active low, etc) that also includes the source of the IRQ (e.g. ISA bus IRQ number, or PCI bus/device/function). Parse the MP specification tables and/or ACPI's "MADT" table to find out what is connected to each I/O APIC input and how each input should be programmed; and put this information into the structure you defined. Hopefully, you'll end up with an array of these structures (with one entry for each I/O APIC input).
• For each I/O APIC input there's a corresponding "I/O Redirection Table entry" (in the I/O APIC) that you need to program. To program the I/O Redirection Table entry you need the information you already got from the last step, but you also need to decide which CPU/s the interrupt will be sent to and which interrupt vector (which IDT entry) the IRQ should use.

That's mostly all there is to it.

To begin with, make it simple - for e.g. send all IRQs to one CPU (the BSP), and use sequential interrupt vectors (e.g. so that the first I/O APIC input causes interrupt 0x40, the second causes interrupt 0x41, the third causes interrupt 0x42, etc). You can make this a lot better (and a lot more complicated) by making more intelligent decisions about which CPU/s and which interrupt vector each IRQ is sent to, but don't worry about that to start with - I'd recommend getting something like the PIT or RTC IRQ working before you start improve things and making it more complicated.

I'd also recommend programming the I/O APIC completely, but leaving each IRQs masked/disabled in the "I/O Redirection Table entry". When a device driver initializes it can install an interrupt handler and then ask for the device's IRQ to be enabled.


Cheers,

Brendan

[01000101]

Thanks, that was a little 'higher-level'.
Have you thought about writing a tutorial/wiki-entry for the L-APIC/IO-APIC as I've seen you've discussed this before in far greater detail?

Right now I'm working on storing crucial info from the MP tables, and will proceed to the next step a little later.

But I do need an IDT setup correct? as from what I understand, the IO-APIC will just act in the PIC's place and still send interrupts through the system-bus (intel xeon) and to the CPU which will go to the appropriate IDT entry.

Also, a 'little' off topic, but what processors use the x2APIC?

[01000101]

OK, so I've been working a bit and here's my current progression.

* set up IDT/ISRs (they work, tested with PIC)
* mask all interrupts on the PIC
* parse the SM tables for IO-APICs and IO-Interrupts (and store into structures)
* enable xAPIC
* setup the wire-mode (using above code, hopefully it is correct)
* setup 5 IO-APIC redirection tables

No interrupts get received though.
I set the redirection registers to 0xFF000000000008xx.

Code: Select all

/* io_apic_write_64(io_apic, reg, val) */

    io_apic_write_64(0, 0x10, 0xFF00000000000810);  // vect: 0x10, fixed int, dst: logical bcast
    io_apic_write_64(0, 0x12, 0xFF00000000000811);  // vect: 0x11, fixed int, dst: logical bcast
    io_apic_write_64(0, 0x14, 0xFF00000000000820);  // vect: 0x20, fixed int, dst: logical bcast
    io_apic_write_64(0, 0x16, 0xFF00000000000821);  // vect: 0x21, fixed int, dst: logical bcast
    io_apic_write_64(0, 0x18, 0xFF00000000000840);  // vect: 0x40, fixed int, dst: logical bcast

Those also are set for edge-triggered and are unmasked. I'm apparently missing something as that seems pretty decent of a test, but not a single interrupt get received. I'll start inputing better values taken from the SM tables (such as trigger type and such), but for the moment (and those sum up the majority) I'll test with the broadcast method.

Should I be researching more about the wire-mode or is that a minor detail in this? Also, why must the vector start at 0x10? For example, if the register is 0x12 (IOREDTBL1) and the vector is 0x10, does that mean that interrupt 0x01 will get send to the handler for IDT[0]?

[Brendan]

Hi,

Have you thought about writing a tutorial/wiki-entry for the L-APIC/IO-APIC as I've seen you've discussed this before in far greater detail?

I probably should, but I tend write that sort of thing while I'm implementing associated code, and won't be writing APIC related code again for a while (still reworking boot code here ) ...

But I do need an IDT setup correct? as from what I understand, the IO-APIC will just act in the PIC's place and still send interrupts through the system-bus (intel xeon) and to the CPU which will go to the appropriate IDT entry.

Yes - you will need an IDT (and the I/O APIC does just act in the PIC's place, sort of).

Also, a 'little' off topic, but what processors use the x2APIC?

That's a good question - AFAIK Intel's new "core i7" (Nehalem) and chipsets designed for Intel's "Quickpath" interconnect will be the first to support x2APIC. However, for backward compatibility purposes these computers will (normally) look like xAPIC systems when they boot - an OS would need to detect support for x2APIC and enable it (if the OS supports x2APIC). Note: Intel's x2APIC document also says that for computers with more than 255 CPUs, the BIOS may enable "x2APIC" mode so there is no backward compatibility. I wouldn't worry about this (yet) - computers with that many CPUs will take at least a few years to arrive (and even then it'll be rare).

Also, AFAIK there aren't any OSs that support x2APIC yet. It's important to consider x2APIC in your OS design though - for a simple example, my previous OS design used a 256-bit "CPU affinity mask". Windows is worse - on 32-bit versions of Windows the "GetProcessAffinityMask()" API function (and all the other API functions that deal with CPU affinity) returns a 32-bit CPU affinity; and for 64-bit versions of Windows the API functions that deal with CPU affinity work on 64-bit masks. This means that supporting more than 256 CPUs (or 32/64 CPUs for Windows) will break the API and break all the applications that use CPU affinity.

OK, so I've been working a bit and here's my current progression.

* set up IDT/ISRs (they work, tested with PIC)
* mask all interrupts on the PIC
* parse the SM tables for IO-APICs and IO-Interrupts (and store into structures)
* enable xAPIC
* setup the wire-mode (using above code, hopefully it is correct)
* setup 5 IO-APIC redirection tables

No interrupts get received though.
I set the redirection registers to 0xFF000000000008xx.

Ok, 0xFF000000000008xx corresponds to: destination 0xFF (send to all CPUs), interrupt not masked, edge triggered, active high, physical destination mode, lowest priority delivery mode, interrupt vector "xx". That looks right to me (for ISA IRQs).

Some notes that may or may not help:

• Even though the I/O APIC documentation says interrupt vectors from 0x10 to 0xFE are valid, don't use interrupt vectors from 0x00 to 0x1F for IRQs - they're reserved for exceptions and the local APIC might not like it.
• Check that all reads and writes to the I/O APIC are 32-bit. If the access is any other size it'll be ignored.
• Check that the "Task Priority Register" in each CPU's local APIC is set correctly. If the TPR = 0xFF the CPU won't accept any interrupts, and if the TPR = 0x00 it'll accept all interrupts (the TPR is the CPU's priority threshold, where the CPU only accepts an interrupt if bits 4 to 7 of the interrupt vector are greater than bits 4 to 7 of the TPR).
• Check that the local APICs are enabled. If they're disabled they'll still send and receive IPIs, but won't receive IRQs from the I/O APIC.
• The EOI for APICs is different - you need to write 0x00000000 to the local APICs "end of interrupt" register (except for NMI, SMI, extINT, INIT or SIPI interrupt types) instead of doing the "out 0x20, 0x20" thing to send an EOI to the PIC/s. This includes IPIs.
• You can read the local APIC's IRR, ISR and TMR registers to see if the local APIC recieved the interrupt from the I/O APIC. You can also check the I/O APIC's "delivery status" bit in the corresponding "I/O Redirection Table entry" to see if it tried to send the IRQ. This might help to find out if the IRQ was sent from the I/O APIC to the local APIC (or not). For this sort of testing it'd be easier to send the IRQ to the BSP, so you don't need to guess which CPU/local APIC the IRQ may have been sent to.
• Don't assume "ISA IRQ n" is connected to "I/O APIC input n". For example, it's common for the PIT (ISA IRQ 0) to be connected to I/O APIC input 2 (or not connected to the I/O APIC at all in some cases).
• On some (mostly older) computers there's an IMCR that controls whether the PIC is being used or the I/O APIC is being used. If the hardware uses the IMCR then you'd need to write 0x70 to I/O port 0x22, then write 0x01 to I/O port 0x23 to make sure interrupt lines go through the I/O APIC. There's a flag (bit 7 in "MP Feature Byte 2") in the "MP Floating Pointer Structure" that tells you if there's an IMCR or not. IIRC for ACPI systems there is no IMCR (and also, no way to tell via. the ACPI tables if an IMCR is present or not).

Should I be researching more about the wire-mode or is that a minor detail in this? Also, why must the vector start at 0x10? For example, if the register is 0x12 (IOREDTBL1) and the vector is 0x10, does that mean that interrupt 0x01 will get send to the handler for IDT[0]?

IMHO you shouldn't need to know about virtual wire mode (unless you're using the PICs and the I/O APIC/s at the same time, which I wouldn't recommend - it complicates things for no reason).

If the interrupt vector (in the "I/O Redirection Table entry") is 0x10, then the I/O APIC will send interrupt 0x10 to the CPU when the IRQ occurs, and interrupt 0x10 is used for the FPU floating point error exception. Also note that interrupt vector 0xFF isn't valid - I use interrupt vector 0xFF for the local APIC's spurious interrupt because it can't be used for much else (and because the spurious interrupt vector bits 0 to 3 are hardwired to 1 in some CPUs, and vector 0xFF is the power-on default).


Cheers,

Brendan

[01000101]

That was an excellent explanation of things, and I followed most (if not all) of the advice given and have implemented them, but now I still don't get ISR interrupts, I only get an INT 8 (double fault). IIRC exceptions are a local interrupt set, and therefor do not even pass through the IO-APIC anyways, so no progress for the interrupts.

As I believe I am so close to actually figuring this out, I'm going to post some of the experiment code and hopefully we can figure this out. I understand your point on the wiki article, and believe that once this is worked out, I'd like to start a programming tutorial on it (once I've figure more about it out of course).

--update--
* I have switched the lowest interrupt vector to 0x20 (32) and scale up by 1 through the entire array (24 entries).
* All reads and writes are 32-bits, but my function uses a 64-bit integer and splits it up later (see io_apic_write_64(x,x);)
* I now set the TPR to 0
* I am almost 100% sure the Local APIC's are enabled (see the code + error tests)
* I send an EOI before enabling interrupts to make sure no hangs occur before interrupts can even happen.
* I now test for the IMCRP bit in the MP tables, and set those port values when appropriate.

[edit] BTW, this is post MP table parsing, so I've already checked for IO apics and IO interrupts.[/edit]

Code: Select all

// v = void

v Write_Local_APIC(uint32 reg, uint32 val)
{*(uint32*)((uint64)0xFEE00000 + reg) = val;}

uint32 Read_Local_APIC(uint32 reg)
{return (uint32)(*(uint32*)((uint64)0xFEE00000 + reg));}

static void io_apic_set_reg(uint8 io_apic, uint8 reg)   // set active reg
{
    if(io_apic >= io_apic_count){error("ioapic: (set) io_apic >= io_apic_count",0); return;}
    *(uint32*)((uint64)smp_ioae[io_apic].address) = (uint32)reg;
}

static uint64 io_apic_read_64(uint8 io_apic, uint8 reg)
{
    uint32 lo, hi;
    uint64 val;
    
    if(io_apic >= io_apic_count){error("ioapic: (read) io_apic >= io_apic_count",0); return 0;}
    
    io_apic_set_reg(io_apic, reg);
    hi = *(uint32*)((uint64)smp_ioae[io_apic].address + 0x10);
    io_apic_set_reg(io_apic, reg+1);
    lo = *(uint32*)((uint64)smp_ioae[io_apic].address + 0x10);
    val = hi;
    val <<= 32;
    val |= lo;
    return val;
}

static void io_apic_write_64(uint8 io_apic, uint8 reg, uint64 val)
{
    io_apic_set_reg(io_apic, reg);
    *(uint32*)((uint64)smp_ioae[io_apic].address + 0x10) = (uint32)(val >> 32);
    io_apic_set_reg(io_apic, reg+1);
    *(uint32*)((uint64)smp_ioae[io_apic].address + 0x10) = (uint32)(val & 0xFFFFFFFF);
}

v Send_INIT_IPI()
{
	printf("apic: sending INIT IPI (%x | %u)\n", Read_Local_APIC(0xF0), cpu_count);
    
    Write_Local_APIC(0x310, 0xFF000000);  // set the destination as 0xFF (broadcast)
    Write_Local_APIC(0x300, 0x000C4500);  // send a broadcast INIT IPI
    while(Read_Local_APIC(0x300) & 01000){wait(10);}    // wait until all the CPU's ACK
    Write_Local_APIC(0x310, 0xFF000000);  // set the destination as 0xFF (broadcast)
    Write_Local_APIC(0x300, 0x000C4610);  // send a broadcast SIPI IP  
}

v Enable_Local_APIC()
{
    Write_Local_APIC(0xF0, (Read_Local_APIC(0xF0) | 0x1FF));    // set the Enable bit (SVR)
    
    __asm__ __volatile__("movq $0x1b, %rcx; rdmsr; or $0x800, %rax; wrmsr;");   // set the Enable bit (MSR_1b)
    
    if(!(Read_Local_APIC(0xF0) & 0x100))    // if (SVR) does not have the Enable bit set
    {error("Local APIC will not enable (%x)", Read_Local_APIC(0xF0));}
    
    if((Read_Local_APIC(0x30) & 0xFF) != 0x14)  // if the APIC version is not 0x14
    {error("Local APIC has incorrect version (%x)", Read_Local_APIC(0x30));}
    
    Write_Local_APIC(0x80, 0); // set TPR to 0 (enable all interrupts)
    if(Read_Local_APIC(0x80) != 0)
    {error("Local APIC has incorrect TPR (%x)", Read_Local_APIC(0x80));}
}

v Detect_xAPIC()
{
    if(!(cpuid_features_rcx & (1 << 21)))
    {error("xAPIC not found",0);}
    
    printf("apic: cpuid_features_rcx: %x \n", cpuid_features_rcx);
}

v Setup_IOAPIC()
{
    uint16 i;
    // set vectors 0x20 -> (0x20 + 24) to enabled and unmasked
    for(i = 0; i < 24; i++)
    {io_apic_write_64(0, 0x10 + (i * 2), (0xFF00000000000800 | (0x20 + i)));} // fixed int, dst: logical bcast
    
    if(IMCRP_bit) 
    {
        printf("ioapic: IMCRP bit set, switching from PIC to IOAPIC mode\n");
        outportb(0x22, 0x70);   // make sure IOAPIC is in use and not the PIC
        outportb(0x23, 0x01);
    }
    
    Write_Local_APIC(0xB0, 0);      // send EOI to Local APIC to clear things out
    __asm__ __volatile__("sti;");     // enable interrupts
}

v init_APIC()
{
    Detect_xAPIC();     // detect an extended APIC
    Enable_Local_APIC(); 
    init_idt();  // setup the IDT, setup ISRs, setup timer and keyboard handler (int 0 and 1) (no PIC remapping)
    Send_INIT_IPI(); // wake up APs
    Setup_IOAPIC(); 
}

Hopefully the code was not too rough to read, I didn't take the time to make it look good haha.
There is no PIC remapping, and even when there was, the result was the same. I only get a double fault and no other interrupts (it halts after exception). The result is the same with or without the AP wakeup code as the AP's never enable interrupts and halt before the double fault even occurs.

Am I correct in my assumption that exceptions do not travel through the IO-APIC and that is why it can be recieved? I also find it relevant to remind that a double fault is caused by (says intel) NMI's and INTR's, so of all the exceptions to occur, I believe this one is actually relevant to this task at hand.

[01000101]

I just realized that if I remap the PIC's and leave them masked, it no longer double faults, but still no interrupts get through. That is beyond confusing to me as I check for the IMCRP bit and handle if necessary (which it isn't in BOCHS or my Xeon machine).

I've tried setting the logical destination in the IO Redirection Table(s) to 0,1,0xFF and the result is the same... nothing.

I've disabled, enabled, changed the interrupt type (fixed to extINT) and still nothing changes. The only thing that seems to have an impact is the PIC remapping (which is remapped far before I even init the IOAPIC).

[01000101]

haha, *kicks self*.

I figured out the issue. I can now receive interrupts from the IO-APIC.

I was writing the lower 32-bits to the high byte of the IO redirection tables, and the high 32-bits to the low byte. So I was writing them backwards. I realized this when I did a full debugging of the [IOAP] device in BOCHS and noticed that the redirection tables showed no destination and the masked bit set.

new function code:

Code: Select all

static void io_apic_write_64(uint8 io_apic, uint8 reg, uint64 val)
{
    io_apic_set_reg(io_apic, reg+1);
    *(uint32*)((uint64)smp_ioae[io_apic].address + 0x10) = (uint32)(val >> 32);
    io_apic_set_reg(io_apic, reg);
    *(uint32*)((uint64)smp_ioae[io_apic].address + 0x10) = (uint32)(val & 0xFFFFFFFF);
}

Now all seems to be well, I can receive and EOI them just fine. Now I just need to figure out how to tell what interrupt is what line on the IOAPIC (eg: which IO redirection table is for the timer, from what I can tell the MP tables don't tell me that).

Thanks for your tremendous help Brendan.

[Brendan]

Hi,

haha, *kicks self*.

I figured out the issue. I can now receive interrupts from the IO-APIC.

Congrats!

Now all seems to be well, I can receive and EOI them just fine. Now I just need to figure out how to tell what interrupt is what line on the IOAPIC (eg: which IO redirection table is for the timer, from what I can tell the MP tables don't tell me that).

The MP tables do tell you what's connected to each I/O APIC input - see section 4.3.4, "I/O Interrupt Assignment Entries" in the specification. It's a little messy (especially PCI devices, although "conforms to the specifications of the bus" can be a pain in the neck too) but it's all there...


Cheers,

Brendan

[01000101]

--update--
I have put the processors into logical destination mode, programmed the Logical Destination Register (LDR) and have cascaded clusters of interrupts across clusters of processors. Now I have been able to assign processors 0&1 to eth0, 2&3 to eth1, and all processors to the timer interrupt (they get reduced by lowest priority).

This is one of the coolest programming feats I have accomplished thus far as it is one of the few original ideas that actually made it from a cool idea in the creation of the design to an actual implementation exactly as I wanted it to be. With this programmed, and combined with my preexisting handlers/drivers, it won't be long until I have a stable extreme scalable design for the pressures that packet filtering exerts.

[jal]

Now I have been able to assign processors 0&1 to eth0, 2&3 to eth1, and all processors to the timer interrupt (they get reduced by lowest priority).

Just a quick question that I don't feel STFWing: if the interrupt gets sent to both processors, will they both start their ISR and is it up to the implementation to synchronize this?

This is one of the coolest programming feats I have accomplished thus far as it is one of the few original ideas that actually made it from a cool idea in the creation of the design to an actual implementation exactly as I wanted it to be. With this programmed, and combined with my preexisting handlers/drivers, it won't be long until I have a stable extreme scalable design for the pressures that packet filtering exerts.

Well, congrats. Your project seems one of the (very) few commerically viable ones.


JAL

[01000101]

Now I have been able to assign processors 0&1 to eth0, 2&3 to eth1, and all processors to the timer interrupt (they get reduced by lowest priority).

Just a quick question that I don't feel STFWing: if the interrupt gets sent to both processors, will they both start their ISR and is it up to the implementation to synchronize this?

I have it set to 'lowest priority' mode. In this mode, when multiple cpu's are allowed to respond to an interrupt, they choose the least used cpu at the time to take the interrupt. I'm still learning about all of this, so maybe that is wrong, but that is the idea that I have got from the specs.

This is one of the coolest programming feats I have accomplished thus far as it is one of the few original ideas that actually made it from a cool idea in the creation of the design to an actual implementation exactly as I wanted it to be. With this programmed, and combined with my preexisting handlers/drivers, it won't be long until I have a stable extreme scalable design for the pressures that packet filtering exerts.

Well, congrats. Your project seems one of the (very) few commerically viable ones.
JAL

Thanks!