OSDev.org

After a long time using 8259 PIC and a single-processor environment, I going APIC programming. I've had a good introduction to the topic (read the manual from Intel chapter 10). First I want to work with APIC on single-processor environment. But there are some things I could not understand yet:

If I use APIC, I have to "abandon" PIC 8259?
There are master and slave IRQ or not?
I have to remap the master and slave IRQs as PIC?

And last (not so important) in an MP environment, where there is a LAPIC for each processor, I have to install an interrupt handler for all CPUs? And when an interrupt occurs, the LAPIC are "minimally smart" and directs the interrupt to handler in the CPU only (it would be chaos if should call the same handler for all CPUs ), or I have to program this feature?

Hi,

arabasso wrote:If I use APIC, I have to "abandon" PIC 8259?

In theory, it is possible to use both at the same time. In practice there's no reason to use both at the same time as all IRQs that are connected to the PIC chips are also connected to the IO APIC anyway, and it just causes unnecessary confusion.

arabasso wrote:There are master and slave IRQ or not?

IRQs are just IRQs. There's a master and slave PIC because a single PIC chip only has 8 inputs and that wasn't enough so IBM added a second PIC chip. There isn't a master and slave I/O APIC. Usually there's only one I/O APIC with 24 inputs and that's enough inputs for everything. However, especially for servers, it's entirely possible to have multiple I/O APICs (e.g. one of the computers I have here has two I/O APICs where each has 16 inputs, and I've seen datasheets for large servers with four I/O APICs). When there are multiple I/O APICs they are all independent - none are "master" and none are "slaves".

arabasso wrote:I have to remap the master and slave IRQs as PIC?

You'd mask all IRQs in the PIC so they can't occur, and use the I/O APICs instead. However, it is still technically possible for the PIC chips to generate spurious IRQs (which can't be masked or prevented in the PIC); so just in case you'd still need to remap the PICs even though you aren't using them, and have spurious IRQ handlers for the PICs. In theory it may be possible to prevent the PICs from sending any signal to the CPU (by breaking the connection between the master PIC's "INTR" line and the CPU) but there's at least 3 different ways this signal can be connected to the CPU (directly to a local APIC input, via. an I/O APIC input, or via. a special gate in the chipset) and it's a lot easier not to bother (and more fun to collect statistics, so the administrator can find out how many spurious IRQs from the PIC chips have occurred, as a high number of spurious IRQs might indicate noisy signals and motherboard problems).

arabasso wrote:And last (not so important) in an MP environment, where there is a LAPIC for each processor, I have to install an interrupt handler for all CPUs? And when an interrupt occurs, the LAPIC are "minimally smart" and directs the interrupt to handler in the CPU only (it would be chaos if should call the same handler for all CPUs ), or I have to program this feature?

For IRQs from the IO APIC/s, you have to program the IO APIC to tell it which CPU/s each IRQ should be sent to, and the IRQ can be sent to a specific CPU or broadcast to multiple CPUs. For the local APIC there's several sources of interrupts - interrupts that were sent by software on other CPUs (called Inter-Processor Interrupts, or IPIs), the local APIC's inputs (which are typically only used for NMI and maybe the connection from the PIC chips to the CPUs), plus some internal interrupt sources - the local APIC timer's IRQ, the thermal status IRQ, the error IRQ, etc. All interrupts from the local APIC go to the CPU connected to that local APIC (you shouldn't try to broadcast something like the local APIC timer's IRQ to several CPUs, for e.g.).

In all of these cases (IPIs, I/O APIC and local APICs), for normal interrupts (e.g. not including things like NMI) software determines which interrupt vector (in the IDT) should be used for that interrupt; and there's a priority mechanism that depends on the interrupt vector. The CPU itself also has a "current priority level" that works in conjunction with the interrupt vector priorities (and has nothing to do with anything else, like current privilege level/CPL). Receiving an interrupt causes the CPU's priority to be raised to the priority of that interrupt, where lower priority interrupts are temporarily blocked. This means you'll probably want things like IPIs and the local APIC timer to use higher priority interrupt vectors and things like the floppy controller (and spurious IRQs) to use lower priority interrupt vectors (so that the local APIC timer and IPIs can interrupt the floppy controller's interrupt handler, but the floppy controller's IRQ can't interrupt the IPI handlers or local APIC timer's interrupt handler). The end result is that you want to dynamically assign interrupt vectors to different things depending on what type of device is connected to it - for e.g. if IO APIC input #12 is an ethernet controller then you might assign a higher priority interrupt vector to it, and if IO APIC input #12 is a sound card you might want to assign a lower priority interrupt vector to it instead.

Software can also change a CPUs current priority (where the CPU's priority is the priority software set or the priority of the interrupt currently being handled, whichever is higher). This means that (for just one example) you could have a very high priority "real time" process, and when the scheduler gives this process CPU time it could adjust the CPU's priority to prevent low priority IRQs from interrupting while the very high priority "real time" process is running.

Also, for IRQs from an I/O APIC, you can use a special "send to lowest priority CPU" mode. In this case, if one CPU is running a very high priority "real time" process, another CPU is handling a high priority interrupt and a third CPU is doing nothing, then hardware can automatically send the IRQ to the CPU that is doing nothing.

Normally, you only need one IDT that's shared by all CPUs, where all IDT entries can be used by any CPU/s. In this case, for IRQs (from the IO APIC) the IO APIC's configuration determines which CPU/s actually receive the IRQ and some CPUs may not use some of the IDT entries. For example, if you tell the IO APIC to always send an IRQ to CPU #2, then CPU #1 never receives that IRQ and therefore doesn't actually use the IDT entry for that IRQ.

For massive systems, it might (in theory) be beneficial to be "clever". For example, you could arrange CPUs in groups and have a slightly different IDT for each group. In this case, you could configure the IO APIC so that one IRQ is sent to CPUs in the first group as interrupt 0x33, and a completely different IRQ is sent to a different group of CPUs as interrupt 0x33. The IDT for the first group of CPUs would use interrupt 0x33 for one IRQ handler and the IDT for the second group of CPUs would use interrupt 0x33 for a completely different IRQ handler. That way you can have more than 256 IRQs even though there's only 256 IDT entries (and reduce the chance of needing IRQ sharing, and maybe use interrupt priorities more efficiently).


Cheers,

Brendan

Your post solved some issues, but appeared others...

It would be possible to implement a system of "IRQ Poll"?

For example, I create a generic IDT containing handlers for each IRQ, and the same IDT is shared by all CPUs. So, a mouse IRQ occurs, which of the CPUs will be called? What is free? The highest priority?

Supposedly, the IOAPIC redirected the mouse IRQ to CPU0. While the CPU0 working at mouse IRQ, a keyboard IRQ occur? Another CPU is called? Or simply give the system broadcast to all processors?

I commented about this by the following fact, in my view I think it unnecessary to install several handlers, or direct a IRQ to specific CPU, according to the structure of my kernel (I thought of something like "unity is strength").

I know bit about MP. In MP systems can I have more than one IDT? On each CPU? More importantly, I need to initialize each CPU as the bootstrap CPU (GDT, IDT, PAGING)? So I can have a kernel running on each CPU (not that I go do it, the more curious...)?

Sorry about the amount of questions...

Hi,

arabasso wrote:It would be possible to implement a system of "IRQ Poll"?

Yes. For example you could disable IRQs (e.g. CLI) and poll the "interrupt received register" in the local APIC instead. You could also mask the IRQs everywhere and poll the individual devices instead. Of course the entire reason there's IRQs to begin with is that polling sucks, so...

arabasso wrote:For example, I create a generic IDT containing handlers for each IRQ, and the same IDT is shared by all CPUs. So, a mouse IRQ occurs, which of the CPUs will be called? What is free? The highest priority?

You tell the IO APIC which CPU (or which CPUs) it should send the mouse IRQ to when you configure the IO APIC.

arabasso wrote:Supposedly, the IOAPIC redirected the mouse IRQ to CPU0. While the CPU0 working at mouse IRQ, a keyboard IRQ occur? Another CPU is called? Or simply give the system broadcast to all processors?

You tell the IO APIC which CPU (or which CPUs) it should send the keyboard IRQ to when you configure the IO APIC.

arabasso wrote:I commented about this by the following fact, in my view I think it unnecessary to install several handlers, or direct a IRQ to specific CPU, according to the structure of my kernel (I thought of something like "unity is strength").

If you mean, configure everything so that all IRQs use the same interrupt vector, and have a common IRQ handler polls everything; then "unity is inefficient, slow and stupid". The idea of IRQs is to avoid the need for polling, and the idea of using different IRQs for different devices is to avoid the need to waste time trying to figure out which device needs attention.

arabasso wrote:I know bit about MP. In MP systems can I have more than one IDT? On each CPU?

It is possible to have a completely different IDT for each CPU. I don't think it's possible to think of a good reason to do that though.

arabasso wrote:More importantly, I need to initialize each CPU as the bootstrap CPU (GDT, IDT, PAGING)?

You can only have one CPU as the bootstrap CPU.

arabasso wrote:So I can have a kernel running on each CPU (not that I go do it, the more curious...)?

You can have the same kernel running on each CPU. This makes sense (for anything with cache coherency) in terms of efficiency and resource usage.

You can also have a different kernel running on each CPU. This makes sense for anything without cache coherency.


Cheers,

Brendan

arabasso wrote:More importantly, I need to initialize each CPU as the bootstrap CPU (GDT, IDT, PAGING)?

I guess what you mean by this question is: Does every CPU need the same initialization as needed by the bootstrap CPU?

Every CPU starts in real mode when it is started up by the OS, so you need to load a GDT and IDT, jump into protected mode, load CR3 with the base of a page directory... Just as you did with the bootstrap CPU. However, you will most likely use the same GDT, IDT and page directory that is also used by the bootstrap CPU, so you need to create these tables only once for the bootstrap CPU, and then load their addresses into every CPU's registers.

I think I spoke bad about the "IRQ poll".

I thought about setting up my kernel as follows: configure IOAPIC, so when one IRQ does, the IOAPIC even find a free CPU and call the corresponding handler. Thus, an IRQ could be handled by any CPU (which was free).

It was strange the way you asked, but who would actually manage it would be the APIC and not by software. Another thing I noticed in the posts, after all, how many have IOAPIC in the system? Only one? Or Each processor has its own?

1) If I had only one, I did not need to worry about choosing which CPU would handle certain IRQ, then the IO APIC would "push" the IRQs and each CPU that would be free "pop" and treating each one.

2) If each CPU has its own IOAPIC, and if I set all the CPU can receive any IRQ, so if a mouse IRQ happens there will be broadcast IRQ (the same request would be sent to all CPUs!?).

If a IOAPIC for each CPU, is set up (!!!without making code hacks, only program the IOAPIC!!!) as in scenario 1?

Hi,

arabasso wrote:I thought about setting up my kernel as follows: configure IOAPIC, so when one IRQ does, the IOAPIC even find a free CPU and call the corresponding handler. Thus, an IRQ could be handled by any CPU (which was free).

It was strange the way you asked, but who would actually manage it would be the APIC and not by software.

That sounds like "send to lowest priority CPU" to me; where the IO APIC is configured to send the IRQ to the CPU that currently has the lowest priority. This is usually a good way to do it.

You should probably read the chapter in Intel's manual that covers the local APIC. A lot of the concepts described there (e.g. physical vs. logical delivery, and the interrupt acceptance logic) help to understand the options you get for configuring IRQs in the IO APIC.

arabasso wrote:Another thing I noticed in the posts, after all, how many have IOAPIC in the system? Only one? Or Each processor has its own?

Intel's MultiProcessor specification tables and/or APCI tables will tell how many IO APICs are present and how many inputs (IRQs) each one supports. You can't make assumptions. In theory a computer with a single CPU could have 16 IO APICs, a large server with 64 CPUs could have one IO APIC, and in any case each IO APIC could support anything from 1 input/IRQ to 256 inputs/IRQs. In practice most computers only have one IO APIC and only larger servers (designed to handle lots of devices, typically with multiple PCI host controllers) have multiple IO APICs, and IO APICs typically have between 16 and 32 inputs (with 24 inputs/IRQs being very common).

The number of CPUs doesn't matter - it's mostly about the number of devices, and motherboard/chipset manufacturers trying to avoid the need for PCI IRQ sharing.

Also, it makes very little difference to an OS how many IO APICs there actually are. The OS should use "global IRQ numbers" for almost everything. For example, if an ethernet card is connected to "global IRQ #18" and you want to configure "global IRQ #18", then you call a "configure_global_IRQ()" function. The only place the number of IO APICs matters is inside that "configure_global_IRQ()" function - it doesn't matter for anything outside that function (doesn't matter for managing devices, doesn't matter for determining how each global interrupt should be configured, doesn't matter for in interrupt handlers, doesn't matter for sending the EOI, etc).

arabasso wrote:1) If I had only one, I did not need to worry about choosing which CPU would handle certain IRQ, then the IO APIC would "push" the IRQs and each CPU that would be free "pop" and treating each one.

2) If each CPU has its own IOAPIC, and if I set all the CPU can receive any IRQ, so if a mouse IRQ happens there will be broadcast IRQ (the same request would be sent to all CPUs!?).

If a IOAPIC for each CPU, is set up (!!!without making code hacks, only program the IOAPIC!!!) as in scenario 1?

I think you're missing something.

You do not want an IRQ to be broadcast to all CPUs, because that would mean all CPUs attempt to handle that IRQ at the same time. You want to send the IRQ to one CPU so that only one CPU attempts to handle the IRQ. This can be a specific CPU, or the lowest priority CPU in a group of CPUs, or the lowest priority CPU in all CPUs.

The "send to lowest priority CPU" thing is completely handled by hardware - the IO APIC knows the current priority of each CPU, and the IO APIC only sends the interrupt to one CPU.


Cheers,

Brendan