i/o apic, local apic and irqs

[Kicer]

HI.

I'm trying to uderstand how this stuff (apic) works.
Actually i have set up local and io apic for triggering irq0 to cpu0.
It works fine but i'm wondering how to send irq0 to all cpus?

i have 2 ideas:
1. after cpu0 retrivers irq0, it sends ipi to other cpus.
2. i think i could use logical mode in io apic and then use set of local apic ids in destination field, but i'm not quite sure how does it work.

the second idea seems to be better, but as i said i don't know how to use it.
Or maybe there are some other options ?

[Brendan]

Hi,

It works fine but i'm wondering how to send irq0 to all cpus?

It's good to be curious and experiment (but I wouldn't recommend broadcasting IRQ0 to all CPUs as part of any OS design)...

2. i think i could use logical mode in io apic and then use set of local apic ids in destination field, but i'm not quite sure how does it work.

the second idea seems to be better, but as i said i don't know how to use it.

First I should point out that there's a "cluster mode" and a "flat mode". The cluster mode is used for NUMA machines that have special APIC controllers between NUMA nodes, and isn't used for normal computers (including AMD's "ccNUMA" computers). I'll only be describing "flat mode" here - whether or not it's worth supporting cluster mode can be left for another discussion. There's also a new "x2APIC" that supersedes the "xAPIC" (which superseded the original APIC) - I won't be describing x2APIC either (I haven't had a chance to play with it yet ).

Basically (for logical destination mode), take the "Logical Destination Address" from where-ever the IRQ came from (e.g. from the I/O APIC redirection table entry's "Destination Field" in bits 56 to 63) and AND it with the local APIC's "Logical APIC ID". If any bits are still set, then the CPU can receive the IRQ.

This means that if you've got 8 CPUs you could have a different bit set in each local APIC's "Logical APIC ID", and you'd be able to select any number of CPUs by setting the corresponding bits in the "Logical Destination Address".

For example, if each CPU's "Logical APIC ID" is set like this:

• CPU0_logical_ID = 00000001b
  CPU1_logical_ID = 00000010b
  CPU2_logical_ID = 00000100b
  CPU3_logical_ID = 00001000b
  CPU4_logical_ID = 00010000b
  CPU5_logical_ID = 00100000b
  CPU6_logical_ID = 01000000b
  CPU7_logical_ID = 10000000b

Then:

• If logical_destination = 00000001b, then interrupt sent to CPU0
  If logical_destination = 00001000b, then interrupt sent to CPU3
  If logical_destination = 01100000b, then interrupt sent to CPU5 and CPU6
  If logical_destination = 00001111b, then interrupt sent to CPU0, CPU1, CPU2 and CPU3
  If logical_destination = 11111111b, then interrupt sent to all CPUs

However, this may or may not suit your purposes. The big problem here is that you run out of bits when there's more than 8 CPUs present.

For larger systems you need something smarter. To do something smarter you need to figure out why interrupts are being sent and where they're being sent; and based on that information decide what each bit in the "Logical Destination Address" will be used for.

The other thing I should mention is that the same stuff ("Logical APIC IDs" and "Logical Destination Address") is also used for "send to lowest priority" interrupts (it's not just for broadcast interrupts). The "send to lowest priority" delivery mode can be useful for IRQs - for e.g. so that an IRQ is sent to a CPU that is idle instead of being sent to a CPU that's doing important work (and maybe instead of being sent to a CPU that's in a power saving state).

Or maybe there are some other options ?

The other option (for hardware IRQs and IPIs) is to use physical destination mode with the destination APIC ID set to 0xFF, which will broadcast the interrupt to every CPU. Also, for IPIs (but not IRQs) there's a few shorthand modes - "send to self", "send to all including self" and "send to all excluding self".


Cheers,

Brendan

[Kicer]

ok thx this knowledge will be usefull.

actually i have problem with my code on real cpu (core 2 duo) but it works on bochs and qemu.
that's what i do:
1. i map io and local apics without cache (that's working for sure - tested with waking up aps)
2. i turn on io apic - *SVR|=ApicEnabled; where SVR is 0xfee000f0 and ApicEnabled=0x100
3. i setup and load idt with 0x20 entry - works for sure, tested previously with pic
4. i turn on symmetric io mode:
asm(
"movb $0x70,%al\n"
"outb %al,$0x22\n" //0x70 -> 0x22 = IMCR
"movb $1,%al\n"
"outb %al,$0x23\n" //0x1 -> IMCR, turn on Symmeteric I/O mode
);
5. i do:
*Register_Select=0x10
*Window=0x20;
*Register_Select=0x11;
*Window=target<<24;
where Register_Select is equal to 0xfec00000, Window=0xfec00010, target=value of local apic id of current (BSP in this case) cpu.

6. asm("sti\n");

then mu pc hangs - it's awaiting for irq0 to calculate it's speed (also chars on my monitor should be changing (as a part of irq0 procedure) but they aren't).
what important am i missing ?

[Brendan]

Hi,

actually i have problem with my code on real cpu (core 2 duo) but it works on bochs and qemu.
that's what i do:
1. i map io and local apics without cache (that's working for sure - tested with waking up aps)
2. i turn on io apic - *SVR|=ApicEnabled; where SVR is 0xfee000f0 and ApicEnabled=0x100

You may need to set the Task Priority Register in the local APIC too...

3. i setup and load idt with 0x20 entry - works for sure, tested previously with pic

You'd need to change the EOI, so that the EOI is sent to the APIC/s instead of the PICs (write zero to the local APIC EOI Register).

4. i turn on symmetric io mode:
asm(
"movb $0x70,%al\n"
"outb %al,$0x22\n" //0x70 -> 0x22 = IMCR
"movb $1,%al\n"
"outb %al,$0x23\n" //0x1 -> IMCR, turn on Symmeteric I/O mode
);

This may or may not be necessary. If it's not necessary it can cause problems (like writing a random value to a random I/O port - you never know if a device is using the I/O port or how it'll effect things). For most (all?) modern computers it's not necessary. Bochs/Qemu emulates an old Pentium chipset (but it's probably not necessary for Bochs/Qemu either). You need to parse MP tables to find out if this is necessary or not (also, IIRC if there's ACPI tables and no MP table then it's not necessary).

5. i do:
*Register_Select=0x10
*Window=0x20;
*Register_Select=0x11;
*Window=target<<24;
where Register_Select is equal to 0xfec00000, Window=0xfec00010, target=value of local apic id of current (BSP in this case) cpu.

This code sets up I/O APIC input #0 as a "fixed delivery, edge triggered, active high" interrupt. Unfortunately the PIT chip probably isn't connected to I/O APIC input #0 - quite a few computers connect the PIT to I/O APIC input #2 instead, but there's no reason why the PIT couldn't be connected to I/O APIC input #9 or I/O APIC input #13 or something else, and no reason why it could be connected to a completely different I/O APIC chip.

You're meant to parse the MP table and/or ACPI tables to find out how many I/O APICs are present and how they're configured. You're also meant to get the physical addresses of the I/O APICs and local APICs from these tables.

Lastly, make sure you're using 32-bit dwords when accessing I/O APICs and local APICs - if you try to write an byte or a 16-bit word they'll ignore you.


Cheers,

Brendan

[Kicer]

Hi,

actually i have problem with my code on real cpu (core 2 duo) but it works on bochs and qemu.
that's what i do:
1. i map io and local apics without cache (that's working for sure - tested with waking up aps)
2. i turn on io apic - *SVR|=ApicEnabled; where SVR is 0xfee000f0 and ApicEnabled=0x100

You may need to set the Task Priority Register in the local APIC too...

ok i'll read about it

3. i setup and load idt with 0x20 entry - works for sure, tested previously with pic

You'd need to change the EOI, so that the EOI is sent to the APIC/s instead of the PICs (write zero to the local APIC EOI Register).

yeah i knew it

4. i turn on symmetric io mode:
asm(
"movb $0x70,%al\n"
"outb %al,$0x22\n" //0x70 -> 0x22 = IMCR
"movb $1,%al\n"
"outb %al,$0x23\n" //0x1 -> IMCR, turn on Symmeteric I/O mode
);

This may or may not be necessary. If it's not necessary it can cause problems (like writing a random value to a random I/O port - you never know if a device is using the I/O port or how it'll effect things). For most (all?) modern computers it's not necessary. Bochs/Qemu emulates an old Pentium chipset (but it's probably not necessary for Bochs/Qemu either). You need to parse MP tables to find out if this is necessary or not (also, IIRC if there's ACPI tables and no MP table then it's not necessary).

another thing to check

5. i do:
*Register_Select=0x10
*Window=0x20;
*Register_Select=0x11;
*Window=target<<24;
where Register_Select is equal to 0xfec00000, Window=0xfec00010, target=value of local apic id of current (BSP in this case) cpu.

This code sets up I/O APIC input #0 as a "fixed delivery, edge triggered, active high" interrupt. Unfortunately the PIT chip probably isn't connected to I/O APIC input #0 - quite a few computers connect the PIT to I/O APIC input #2 instead, but there's no reason why the PIT couldn't be connected to I/O APIC input #9 or I/O APIC input #13 or something else, and no reason why it could be connected to a completely different I/O APIC chip.

You're meant to parse the MP table and/or ACPI tables to find out how many I/O APICs are present and how they're configured. You're also meant to get the physical addresses of the I/O APICs and local APICs from these tables.

now i know why irq entires in mp table are present

Lastly, make sure you're using 32-bit dwords when accessing I/O APICs and local APICs - if you try to write an byte or a 16-bit word they'll ignore you.

this works for suer, i alway analyze gcc's code with objdump

Cheers,

Brendan

thx for help