Enable local-apic interrupts?

[nooooooooos]

Following words i read in the intel-smp manual at B.3:

Then the operating system should enable its own local APIC, thereby allowing IPI communications
with other APIC-based processors. At this time, the APs’ local APICs have interrupts disabled.
Interrupts must remain disabled at the APs’ local APICs while the BSP is enabling the I/O APIC
and bringing the system to the normal operating state.

Actually, the AP doesn't execute my Timer-Interrupt Handler now. (Only the BSP)
But I don't know how to enable the interrupts on the local apic...In the part of the local Apic in the Intel Manual 3a, I didn't found anything.

Thanks
Noooooooooos

[Combuster]

It is there:
Intel 3, 7.5.11: "Local Vector Table" (especially, look at figure 7-8)

[nooooooooos]

I founded the local vector table in 8.5.1 and in figure 8.8...Is this the correct chapter??

And then ist the error-vector used for cpu-exeptions? Is there only necessary to demask the lint0 and lint1? Or must I set also the vectors?

Thx
Noooooooooos

[JAAman]

It is there:
Intel 3, 7.5.11: "Local Vector Table" (especially, look at figure 7-8)

uh... how old is your reference? there is no 3A:7.5.11

even my old revision 12 edition stops at 7.5.5

ya i think 3A:8.5.1 and 3A:fig8-8 is what he meant

[Combuster]

uh... how old is your reference? there is no 3A:7.5.11

I downloaded it less than a year ago from Intel. I just checked however, and the copyright is dated to "1999"

[JAAman]

dont know how that happened... you did download it from the proper intel site right? (i know a lot of other websites host old copies also... or maybe you just accidentally deleted the wrong file )

knew it had to be old when my oldest hard-copy didnt even have that section (dated 2003)

[nooooooooos]

Thank you...But how I have to set the LVT Registers?

(a)Must I only change the settings of the LINT0 and LINT1? Or are there other Registers necessary? (I dont want to use the apic-timer)
(b)And what settings I have to change? It's logical to clear the mask bit...but what else?
(c)Which value for the vector is needed? Must I set the vectors of both, the LINT0 and the LINT1, onto different values? (I have already redirected the ioapics to 0x20-0x37). I don't catch on why to use a second vector.

Nooooooooooos

[Brendan]

Hi,

(a)Must I only change the settings of the LINT0 and LINT1? Or are there other Registers necessary? (I dont want to use the apic-timer)
(b)And what settings I have to change? It's logical to clear the mask bit...but what else?
(c)Which value for the vector is needed? Must I set the vectors of both, the LINT0 and the LINT1, onto different values? (I have already redirected the ioapics to 0x20-0x37). I don't catch on why to use a second vector.

For initializing the local APIC, you'd want to clear the disable bit, setup LINT0 and LINT1, setup the spurious interrupt and install a spurious interrupt handler, set the "task priority register" and setup the "destination format register" (not necessarily in this order).

You should parse either ACPI tables or Intel's Multiprocessor tables to determine how each local APIC input and each I/O APIC input should be programmed. This information includes which PCI devices are connected where, if each input is level triggered or edge triggered, what type of interrupt (SMI, NMI, extINT, etc). Typically (but not necessarily always) these tables will tell you that one of the local APIC inputs (on each CPU) is used for NMI and the other input (on each CPU) is used for INTR. Please note that for SMI, NMI and INTR the interrupt vector is not used by the APIC (and should be set to zero IIRC).

For the other local APIC interrupts (timer, error interrupt, thermal monitoring interrupt, etc), your OS can either use them or disable them. They won't be listed in the ACPI tables or MPS tables as they're entirely inside the CPU (they're more like exceptions than IRQs IMHO).

I/O APICs should be configured dynamically based on what the ACPI or MPS tables say. Some systems have more than one I/O APIC (e.g. my server has 2 of them with 16 inputs per I/O APIC, but I've seen specs for larger servers with four I/O APICs). The highest 4 bits of the interrupt vector used for each IRQ determines that IRQ's priority, so it's usually best to spread the interrupt vectors out (e.g. int 0x22 for the PIT timer, int 0xF0 for the floppy controller, int 0x30 for the network card, etc).

Bunching them all together (e.g. from 0x20 to 0x37) means that you're only using 2 priorities - IRQs from 0x20 to 0x2F using one priority, and IRQs from 0x30 to 0x37 using another priority. Depending on your OS this can be a bad idea. For example, if a floppy drive IRQ and an ethernet card IRQ happen at the same time, then normally you'd want the ethernet card's IRQ to have higher priority to reduce it's latency (so that the ethernet card's IRQ doesn't wait until after you've finished servicing the floppy drive's IRQ).

The way I do it is have interrupt handlers for all possible interrupts. During boot I setup the first 32 interrupts for exceptions, then use 0x20 for the kernel API, 0x21 to 0x23 for IPIs and 0xFF for the local APIC's spurious IRQ. I install dummy IRQ handlers for interrupts 0x24 to 0xFE. Then I use the desired IRQ priority to search for the "best" free interrupt vector when each IRQ handler is installed. In this way a network card driver might tell the OS to use "priority 3" for it's IRQ, while a floppy driver might tell the OS to use "priority 10" for it's IRQ.

The other thing to consider is which CPU/s each IRQ is delivered to. In a NUMA machine you might want to deliver IRQs to CPUs that are "close" to the devices generating that IRQ to reduce bus traffic. In an SMP computer you might want to spread IRQs around, so that you don't have one CPU handling all/most IRQs (unbalanced CPU load). You might also want to send IRQs to CPUs that aren't in some sort of power management state. For example, if a CPU is in a sleep state then it's probably better to send the IRQ to a different CPU than to bring that CPU out of the sleep state; and if a CPU is hot (operating at reduced performance due to thermal throttling) then you might want to send the IRQs elsewhere to help bring that CPU back to a normal temperature. Lastly, if an IRQ occurs frequently then you might want to send it to the same CPU all the time, in the hope that the IRQ handler and it's data is still in that CPU's cache.


Cheers,

Brendan

[nooooooooos]

Thank you very much...

You should parse either ACPI tables or Intel's Multiprocessor tables to determine how each local APIC input and each I/O APIC input should be programmed. This information includes which PCI devices are connected where, if each input is level triggered or edge triggered, what type of interrupt (SMI, NMI, extINT, etc). Typically (but not necessarily always) these tables will tell you that one of the local APIC inputs (on each CPU) is used for NMI and the other input (on each CPU) is used for INTR. Please note that for SMI, NMI and INTR the interrupt vector is not used by the APIC (and should be set to zero IIRC).

You say, that the vector from INTR an NMI should be set to zero. But how the CPU gets these interrupts? Which entry of the IDT is activated?

An INTR is the signal of a pic. How I realize which interrupt is generated on the pic? Do I have to use the pic or it is possible to mask it? (You said that it is necessary to enable LINT1 and LINT0, then I dont have a chance to mask it?)


Noooooooooooos

[Brendan]

Hi,

You say, that the vector from INTR an NMI should be set to zero. But how the CPU gets these interrupts? Which entry of the IDT is activated?

For NMI, the CPU generates an exception instead of an IRQ (interrupt #2).

The INTR line originates from the PIC chips and is connected/routed to the local APIC's LINT0 input (possibly via. the I/O APICs). When the PIC chip wants to generate an IRQ it uses the INTR line to tell the local APIC to expect an interrupt vector, and if LINT0 is programmed as "extINT" the local APIC tells the PIC chips to proceed and the PIC chips send the interrupt vector to the local APIC (and the local APIC forward the interrupt vector to the CPU).

It is possible to use both the I/O APICs and the PIC chips at the same time. For example, you might use the PIC chips for ISA devices and the I/O APICs for PCI devices (and for some old motherboards this is the only way to get IRQs from the PIT chips while using I/O APICs).

Normally (for "I/O APIC only") you'd mask all IRQs in the PIC and leave LINT0 setup for "extINT". If you disable LINT0 without masking IRQs in the PIC then it might cause problems on some motherboards (e.g. messed up PIC state causing problems during reset), and there's not much point masking IRQs in the PIC and disabling LINT0.

This is possibly the easiest way to do things - e.g. mask all PIC iRQs before entering protected mode, then (when you're ready) either enable PIC IRQs again or enable I/O APIC IRQs. I never used "mixed mode" (where both the PICs and I/O APICs are used) - I just make sure the OS can work fine without the PIT on old motherboards (I use local APICs instead of the PIT if local APICs are present).

Also, don't forget about the IMCR (which is used by some old motherboards/chipsets to change IRQ routing) - see Intel's Multiprocessor Specification for details.


Cheers,

Brendan

[nooooooooos]

Thanks, now mostly all is clear...

But I wonder why the PIC is assigned as INT 0 to IO Apic. Is this Interrupt called when a PIC Interrupt is generated? How my ISR must look like? Or is this interrupt from PIC delivered with its own vecotor and I can leave the ISR empty?

EDIT: Strange, yesterday I searched in my ASM book about the NMI, but i didn't find everything...Now I see that the Exeption 2 is omitted.
EDIT2: Do you don't use both Pics together because it can be problematic?


Thanks
Noooooooooos

[Brendan]

Hi,

But I wonder why the PIC is assigned as INT 0 to IO Apic. Is this Interrupt called when a PIC Interrupt is generated? How my ISR must look like? Or is this interrupt from PIC delivered with its own vecotor and I can leave the ISR empty?

In this case, you'd program I/O APIC input #0 as "extINT" too. I'm not sure of the exact reason for this (I'd assume that in some/most modern systems they use the I/O APIC to route the PIC chip's INTR line to the local APIC instead of connecting it directly to the local APIC to make it easier to make motherboards - less wires, and reduced number of pins in the southbridge/chipset). You don't need an ISR for this (set the interrupt vector in the I/O APIC to zero) as the interrupt vector is supplied by the PIC chips, not the I/O APIC.

EDIT: Strange, yesterday I searched in my ASM book about the NMI, but i didn't find everything...Now I see that the Exeption 2 is omitted.

Information about NMI is hard to find (except for the details for the exception itself). AFAIK it was used for memory errors and other hardware failures, but now that's been mostly replaced by machine check exceptions and different hardware mechanisms (that allow the CPU/software to determine what caused the error, rather than just knowing an error occured).

Also, sometimes people use NMI for things it wasn't necessarily intended for. For an example, there's Linux patches that configure the PIT chip's IRQ (in the I/O APIC) so it generates an NMI that's broadcast to all CPUs, which is then used as a watchdog timer. In this way the watchdog timer can detect if the OS has locked up, even if all CPUs are in infinite loops with interrupts disabled (as NMIs can't be masked by "CLI").

EDIT2: Do you don't use both Pics together because it can be problematic?

Using the PIC chips and the I/O APICs at the same time ("mixed mode") should work without problems, but it makes your code more complicated. AFAIK there's only one reason to use PICs and I/O APICs at the same time - on some motherboards the PIT timer's IRQ isn't connected to the I/O APIC.

For me, it's much easier to simply not use the PIT chip in this case and avoid the extra complexity involved with "mixed mode" (I use the periodic IRQ from the CMOS/RTC to track real time, and the local APIC timer/s for the scheduler) so that my code doesn't need the PIT chip.

Of course this might change in future versions of my OS - I intend to make the OS dynamically select which timers to use for what (although this is mostly for HPET support).


Cheers,

Brendan

[exkor]

I believe theres is another reason to use PIC timer instead of APIC timer - APIC timer has larger error. That is, say, APIC will generate timer int every 10ms +/- 0.01ms when PIC will generate timer int every 10ms +/- 0.0001 ms. These values are not exact and only to show where the APIC problem is.

I am actually wondering if this is correct, and if its not then I would like page reference in PIC & APIC docs that explain this situation.

[Combuster]

Well, the maximum resolution of a timer depends on its frequency. The PIT runs on a fixed frequency of a few MHz, The APIC timer runs on the same frequency as the APIC bus, which seems to be the FSB speed (please correct me if wrong), which implies an error improvement of a hundredfold over the PIT. The critical problem is to determine the exact APIC frequency, since it isn't a fixed number. The actual accuracy is as high as the time sources used to sync it to - if you use more sources, the accuracy will statistically improve. On a computer, you can use the PIT and the RTC together to clock the APIC (in 50% of the cases the accuracy will be better than that of the PIT). If you use NTP, you can determine the frequency to near-perfection, but doing so requires time and a working network connection.

If you try the APIC in bochs, it runs on the exact frequency as the emulated CPU, allowing you to interrupt the processor at any time with an error margin of zero. Try that with the legacy PIT

[nooooooooos]

Thank you very much...

Now I have a new problem with the APIC. Namely I saw that for redirecting Ints in the IOApic i can use only 4 bit long LApic IDs. But whats if a system has more then 16 Apics?
Could it be that the extension of the LApic ID wasn't adapted to these spezifications and that I can simply use all 8bit?

EDIT: Which pin polarity do I have to programm for ExtInt?


Nooooooooooos