IO-APIC

[FlashBurn]

There isn?t much material about the io-apic and how to programe it! So I hope here are some who know how do work with it.

I?ve taken the code from o3one and used it as a start point:

Code: Select all

;----------------------------
;   map irqs 0-15
align 4
.loop@irq0_15:
   ;---------------------------
   ;   mask irq
   mov [edi+ioapic_regs_t.ioregsel],edx
   mov [edi+ioapic_regs_t.iowin],IOAPIC_IRQ_MASK
   ;---------------------------
   ;   write upper half first
   add edx,1

   mov [edi+ioapic_regs_t.ioregsel],edx
   mov [edi+ioapic_regs_t.iowin],IOAPIC_IRQ_DEFAULT_HIGH
   ;---------------------------
   ;   select lower half
   sub edx,1

   mov [edi+ioapic_regs_t.ioregsel],edx
   ;---------------------------
   ;   get the int for the irq
   mov eax,[irq2int+4*ecx]
   ;---------------------------
   ;   set default bits and write lower half
   or eax,IOAPIC_IRQ_DEFAULT_LOW
   add edx,2
   mov [edi+ioapic_regs_t.iowin],eax
   ;---------------------------
   ;   next irq or we have finished
   add ecx,1

   cmp dl,30h
   jne .loop@irq0_15
;----------------------------
;   io-apic ignore loop, will be changed when I know from where I get the ints
   mov eax,IOAPIC_IRQ_MASK
align 4
.loop@irq_ignore:
   mov [edi+ioapic_regs_t.ioregsel],edx
   add edx,2
   mov [edi+ioapic_regs_t.iowin],eax

   cmp dl,40h
   jne .loop@irq_ignore
;----------------------------
;   deactivate the PIC
   mov al,70h
   out 22h,al
   mov al,01h
   out 23h,al

Now I get the following error from Bochs and I think that I?m missing something.

Code: Select all

00001824027i[IOAP ] vector 0xd0 stuck?

I get this error on and on! I finish my irq function with an EOI to the apic. I could also imagine that I forgot something when configuring the apic!

[JAAman]

all the information you need is provided by intel (at least for intel chipsets) vol.3 chapter 8 tells about the local apic, and comunication between them (most of the important stuff -- and everything that can be gaurenteed with 3rd-party chipsets)

if you need more specific information to the I/O APIC, try the intel chipset manuals (documentation is freely availible for almost every product intel has ever created)

for non-intel chipsets, information will be much harder to come by
for non-intel CPUs, AMDs manual may contain some differences (though they may be farely similar), but chipset references for AMD CPUs will be very hard to find (at least for current chipsets)

here is the Intel chipset homepage:
http://www.intel.com/products/chipsets/index.htm?iid=CorporateV3+Header_2_Product_Chip

[Rob]

I was under the impression that Bochs had incomplete (?) APIC support. I could be wrong on that though.

[FlashBurn]

So there isn?t anyone who knows what I could have forgotten? I looked at the manuals of the apic and the io-apic, but I haven?t found anything which would help

[Brendan]

Hi,

Now I get the following error from Bochs and I think that I?m missing something.

Code: Select all

00001824027i[IOAP ] vector 0xd0 stuck?

I get this error on and on! I finish my irq function with an EOI to the apic. I could also imagine that I forgot something when configuring the apic!

I think this means that the I/O APIC failed to deliver the interrupt to a local APIC/CPU.

Without knowing what values "IOAPIC_IRQ_DEFAULT_HIGH" and "[irq2int+4*ecx]" are, I can't figure out what is going wrong...

Also for Bochs (and most real machines), there is no IMCR and the PIC is disabled by masking all IRQs in the PIC. For some computers there is an IMCR and you do need to do this:

Code: Select all

;   deactivate the PIC
   mov al,70h
   out 22h,al
   mov al,01h
   out 23h,al

But when there's no IMCR you shouldn't do it.

To determine if there is an IMCR or not you'd need to parse Intel's MP specification tables. If you're getting IRQ and I/O APIC information by parsing ACPI APIC tables instead, then the computer is ACPI compatibile and has no IMCR.


Cheers,

Brendan

[FlashBurn]

Ok, I always forget to give the values of such constants ::)

Code: Select all

IOAPIC_IRQ_DEFAULT_LOW= 0x8900
IOAPIC_IRQ_DEFAULT_HIGH= 0xff000000
IOAPIC_IRQ_MASK= 0x10000

The [irq2int+4*ecx] is a table where are the ints for the irqs! I use a table, so that I can have ints which aren?t following each other. At the moment they are, because I only want to get it working. But here is the table:

Code: Select all

INT_SMP= 040h
irq2int:
                     dd INT_SMP
                     dd INT_SMP+1
                     dd INT_SMP+2
                     dd INT_SMP+3
                     dd INT_SMP+4
                     dd INT_SMP+5
                     dd INT_SMP+6
                     dd INT_SMP+7
                     dd INT_SMP+8
                     dd INT_SMP+9
                     dd INT_SMP+10
                     dd INT_SMP+11
                     dd INT_SMP+12
                     dd INT_SMP+13
                     dd INT_SMP+14
                     dd INT_SMP+15
                     dd 0
                     dd 0
                     dd 0
                     dd 0
                     dd 0
                     dd 0
                     dd 0
                     dd 0

Also now it is vector 0x40 and if I run Bochs in 1 cpu mode it is also vector 0x41 which stucks.

[Brendan]

Hi,

Ok, "IOAPIC_IRQ_DEFAULT_LOW" works out to a level sensitive interrupt with active high trigger mode, and a logical destination address in "IOAPIC_IRQ_DEFAULT_HIGH", using "send to lowest priority" mode.

Interrupts from a PCI bus are usually level sensitive active low, but EISA is normally level sensitive active high and ISA is usually edge sensitive active high. Motherboard manufacturers can use different signals (but almost all of them don't).

Normally, I'd expect most of these interrupts to come from ISA, possibly with some connected to PCI, perhaps one with a strange looking "SCI" connection to the motherboard, and perhaps one that is meant to be "extINT" and is connected to the PIC chips (used for "mixed mode" on older SMP motherboards).

On a real computer, an ISA device (like the PIT or keyboard controller, which is meant to be edge triggered active high) you could get the same IRQ several times because the I/O APIC could get the EOI before the "pulse" sent by the device goes back to it's inactive (low) state. Alternatively, the I/O APIC might miss the pulse entirely if it's fast enough (i.e. think it was noise) because the APIC is looking for a level trigger.

AFAIK this won't effect Bochs though - it's not as realistic as real hardware.

In general, you need to find out what type of signal is connected to each I/O APIC input. There's only 2 ways to do this reliably - MP specification tables and the ACPI APIC table. There's a few "tutorials" on the internet that are just plain wrong here.

The other thing I guess I should mention, is that some I/O APICs have 24 inputs, but some have 16 and some have 28. For each I/O APIC you can read the number of inputs it has from bits 16 to 23 of it's "I/O APIC version register". My Intel datasheet calls this the Maximum Redirection Entry, and says that this value can range from 0 through to 239 (from 1 to 240 inputs).

Also, some computers have more than one I/O APIC. For example, my server here has 2 I/O APICs with 16 inputs each (32 IRQ's total).

In logical destination mode, the I/O APIC will take the destination address (0xFF000000) and broadcast it to all CPUs.

Each CPU will get this value and do a "logical AND" with it's local APIC's destination format register. This determines which bits are important and which bits are ignored. Then it gets the result and does another "logical AND" with it's local APIC's logical destination register. After this, if any bits are still set the local APIC accepts the interrupt.

If more than one CPU accepts the interrupt, then the CPUs will work who is the lowest priority and any other CPUs will ignore the interrupt.

This makes me wonder if you've setup each CPU's local APIC properly.

For example, if the local APIC's logical destination register is left with it's default value (0x00000000) then this final "logical AND" will leave no bits set and the CPU won't accept the interrupt.

Also now it is vector 0x40 and if I run Bochs in 1 cpu mode it is also vector 0x41 which stucks.

That would be the PIT (IRQ0) which keeps on trying to interrupt and doesn't stop, and occasionally you'd get one from the keyboard (IRQ1) which only happens once because you don't get the scancode out of the keyboard controller chip...


Cheers,

Brendan

[FlashBurn]

Ok, I see I have to look at the manuals once more, because I haven?t set anything in the logical destination register.

[FlashBurn]

Ok, I have written the code for setting the logical destination register and now it works. But there is one thing I don?t understand, how the logical destination and the destination format are working! For what do I need these both? And what value should I write into it. I don?t use the logical destination mode, why have I to write something into the ldr?

[Brendan]

Hi,

Ok, I have written the code for setting the logical destination register and now it works. But there is one thing I don?t understand, how the logical destination and the destination format are working! For what do I need these both? And what value should I write into it. I don?t use the logical destination mode, why have I to write something into the ldr?

You are using logical destination mode because bit 11 in your "IOAPIC_IRQ_DEFAULT_LOW" is set. Logical destination mode allows the I/O APIC to send the interrupt to more than one CPUs at the same time - either so that all CPUs get interrupted, or so that the lowest priority CPU gets the interrupt (which is what you're doing).

You could clear this bit and use physical destination mode. In this case you have to put the local APIC ID for the target CPU into the destination field (the highest 8 bits of "IOAPIC_IRQ_DEFAULT_HIGH") so that the I/O APIC knows which CPU the interrupt is meant to go to. In this case only one CPU will ever receive the interrupt.

For normal SMP where all CPUs are equal, what you're doing is better. It means that if one CPU is already servicing an interrupt or something then it'll be a higher priority than other CPUs, and a new IRQ would be sent to a lower priority CPU. If you've got 8 CPUs and 8 IRQs happen at the same time, then all CPUs will get an IRQ each and all IRQs would be handled in parallel. If you used physical destination mode instead, then one CPU can only be running one IRQ handler at a time and it'd take much longer to service those 8 IRQs.

Another trick here is that a CPU's "task priority register" (in the local APIC) can be used to effect which CPU is the lowest priority (or which CPU will end up handling an IRQ). For example, when a CPU is doing nothing you could set the task priority register to "low" and when a CPU is doing important work you could set it to "not as low". In this case an IRQ would be handled by the CPU that was doing nothing rather than interrupting a CPU that was doing something important.

For the last version of my kernel, that is exactly what I did - the scheduler would set the local APIC's task priority register depending on the currently running thread's priority, so that CPUs running low priority threads end up handling IRQs much more often.

Now imagine a computer with 2 Opteron chips. Here the PCI bus is connected to one of the CPUs. This means things on the PCI bus can be accessed from one CPU faster than it can be accessed from the other CPU. This is a result of AMD's hyper-transport bus.

In this case, both CPUs aren't equal - one is better for I/O. Because of this you might get better performance by always sending IRQs to the same CPU, so that the IRQ handling for the device is done by the CPU that the device is connected to. This would also minimizes the traffic between CPUs.

Now imagine a computer with a pair of dual-core Opterons. The first two CPUs are connected directly to the "I/O controller hub", and last two CPUs aren't. It'd be good if IRQs were handled by any of the first two CPUs and if both these CPUs were treated equally.

To do this, you can set the "destination address" in the I/O APIC to "0xFF" and use "send to lowest priority mode". Then for the first 2 CPUs you'd set the local APICs logical destination register to 0xFF and for the last 2 CPUs you'd set the local APICs logical destination register to 0x00. Now, when an IRQ occurs it's sent to all CPUs, the last 2 CPUs ignore it and first 2 CPUs will decide who is the lowest priority, and only one of them will end up handling the IRQ.

[Continued]

[Brendan]

[Continued]

Now imagine you've got a large server with 8 dual core Opterons and 2 seperate PCI host controllers. You want interrupts from "PCI controller 0" to be handled by CPU 0 or CPU 1, and interrupts from "PCI controller 1" to be handled by CPU 14 and CPU 15. In this case, for interrupts from PCI controller 0 you could set the I/O APICs destination address to 0x01 and for interrupts from PCI controller 1 you could set the I/O APICs destination address to 0x02. For the local APIC's logical destination register, on CPU 0 and CPU 1 you'd use 0x01, on CPUs 2 to 13 you'd use 0x00 and on CPUs 14 and 15 you'd use 0x02. Now when an IRQ occurs the I/O APIC sends either 0x01 or 0x02 and all CPUs will "logically AND" this with their local APIC's logical destination register. For most CPUs this will mean all bits are clear and the interrupt is ignored, but for 2 of the CPUs this won't be the case and the lowest priority CPU out of these would handle the IRQ.

Of course the logical destination register in the local APIC is also used for IPIs (Inter-Processor Interrupts). For example, imagine you want to send an IPI to all CPUs that are running Tetris without annoying CPUs that aren't. When a CPU starts running Tetris it could set the highest bit in it's local APICs logical destination register (and clear this bit when it stops running Tetris). Then you could use send an IPI with a logical destination of 0x80 and only CPU running Tetris will receive it...

For an OS to handle any hardware combination perfectly, it needs to first figure out the system topology - the relationships between all CPUs and all I/O devices. Then it can plan the IRQ handling system to be as efficient as possible. It isn't easy.


Cheers,

Brendan

[FlashBurn]

Thanks for the detailed explanation! I think this should be added to the FAQ, shouldn?t it?!

[Pype.Clicker]

Thanks for the detailed explanation! I think this should be added to the FAQ, shouldn?t it?!

certainly, as soon as the FAQ has recovered