OSDev.org

I'm wondering if anyone has heard of this technique: using the local APIC to intentionally generate a self-interrupt -- specifically, as a technique to queue up some work for the kernel to do after any nested ISRs finish execution, but before continuing with normal thread execution (whether in kernel or user space).

In "Inside Windows NT", there's quite a bit of discussion about so-called "software interrupts" that are used to invoke the scheduler and run DPCs (deferred procedure calls). I don't think they mean the int nnn instruction, since these Dispatch/DPC-level interrupts seem to be asynchronous in nature and are subject to interrupt masking on the PIC. However, I've read up on the 8259 and it doesn't seem to have this self-interrupt facility. IIRC, the local APIC on more recent Intel processors does.

So, four questions -- Has anyone heard of this technique? Does it sound like a good/bad/ugly idea? Does it require a local APIC to do? And if so, when was the local APIC introduced to the PC architecture (with the 485, Pentium, P6...)?

Colonel Kernel wrote: So, four questions -- Has anyone heard of this technique? Does it sound like a good/bad/ugly idea? Does it require a local APIC to do? And if so, when was the local APIC introduced to the PC architecture (with the 485, Pentium, P6...)?

Yes, Good, Yes and 486 with external chip, Pentium builtin, numerous newer chips not built in (mainly cheap ones, notably the athlon model 1 too).

APIC is good

BTW, you can also easily send IPI's to other processors but not to you or to everybody, making for easy paging synchronising.

Hi,

Colonel Kernel wrote: So, four questions -- Has anyone heard of this technique? Does it sound like a good/bad/ugly idea? Does it require a local APIC to do? And if so, when was the local APIC introduced to the PC architecture (with the 485, Pentium, P6...)?

A) I've heard of it - it's described here:
http://www.osdever.net/?id=38

B) It is a good, bad and ugly idea, depending on how your OS handles IRQ's in general. It's good if it gives you a cleaner way to invoke the scheduler from within nested IRQ handlers. I can't think of any good reason for Microsoft's "deferred procedure calls" (other than bad IRQ handler design). In my OS the IRQ is handled by the kernel using interrupt gates (so interrupts are disabled). All the kernel does is send a message to any threads (device drivers) that have asked for it. In this case there's no need for self-interrupts (or IRQ priorities).

C) It does require a local APIC (which all Intel CPUs since the 80586/pentium have), but the local APIC must also be enabled. Most BIOSs for single CPU computers disable the local APIC, and once it has been disabled you can't enable it without rebooting. Computers/BIOSs for multi-CPU computers leave local APICs enabled - which may include single CPU BIOSs for newer motherboards that are designed to work with hyper-threading (Pentium IV).

D) AFAIK the local APIC was introduced with the 80286 or 80386 but it was a seperate chip outside the CPU (included on SMP motherboards) until the Pentium.


Cheers,

Brendan

Brendan wrote: B) It is a good, bad and ugly idea, depending on how your OS handles IRQ's in general. It's good if it gives you a cleaner way to invoke the scheduler from within nested IRQ handlers. I can't think of any good reason for Microsoft's "deferred procedure calls" (other than bad IRQ handler design). In my OS the IRQ is handled by the kernel using interrupt gates (so interrupts are disabled). All the kernel does is send a message to any threads (device drivers) that have asked for it. In this case there's no need for self-interrupts (or IRQ priorities).

Seems the only way to do it nicely in nested interrupt stuff, making it plainer to implement. Do you know any other way?

C) It does require a local APIC (which all Intel CPUs since the 80586/pentium have), but the local APIC must also be enabled. Most BIOSs for single CPU computers disable the local APIC, and once it has been disabled you can't enable it without rebooting. Computers/BIOSs for multi-CPU computers leave local APICs enabled - which may include single CPU BIOSs for newer motherboards that are designed to work with hyper-threading (Pentium IV).

aside from the not-enabled-part, think about a few that actually DONT have one? I believe a few early celerons didn't have one, at least one Athlon model doesn't have it, probably Via C3's don't, stuff like that. Always detect first. It's in both CPUID feature flags, bit 9.

D) AFAIK the local APIC was introduced with the 80286 or 80386 but it was a seperate chip outside the CPU (included on SMP motherboards) until the Pentium.

The 486 actually. 82489 if I recall correctly.

Hi,

Candy wrote: Seems the only way to do it nicely in nested interrupt stuff, making it plainer to implement. Do you know any other way?

Don't allow IRQs to be nested in the first place? If you do allow IRQs to be nested what are you going to do if there's no local APIC?

Candy wrote:

C) It does require a local APIC (which all Intel CPUs since the 80586/pentium have), but the local APIC must also be enabled. Most BIOSs for single CPU computers disable the local APIC, and once it has been disabled you can't enable it without rebooting. Computers/BIOSs for multi-CPU computers leave local APICs enabled - which may include single CPU BIOSs for newer motherboards that are designed to work with hyper-threading (Pentium IV).

aside from the not-enabled-part, think about a few that actually DONT have one? I believe a few early celerons didn't have one, at least one Athlon model doesn't have it, probably Via C3's don't, stuff like that. Always detect first. It's in both CPUID feature flags, bit 9.

I'm not too sure about which non-Intel chips do or don't have local APICs, which is why I wrote "all Intel CPUs since".. Are you sure about the early Celerons? I'm mostly going by this:

IA-32 Intel? Architecture Software Developer?s Manual Volume 3: System Programming Guide, Section 18.24. ADVANCED PROGRAMMABLE INTERRUPT CONTROLLER (APIC):

"The Advanced Programmable Interrupt Controller (APIC), referred to in this book as the local APIC, was introduced into the IA-32 processors with the Pentium processor (beginning with the 735/90 and 815/100 models) and is included in the Pentium 4, Intel Xeon, and P6 family processors. The features and functions of the local APIC are derived from the Intel 82489DX external APIC, which was used with the Intel486 and early Pentium processors. Additional refinements of the local APIC architecture were incorporated in the Pentium 4 and Intel Xeon processors."

Of course detecting it first using CPUID would be required in any case..

BTW - it seems you're right about 80486 being the first


Cheers,

Brendan

Thanks for the info/links!

The reason I'm looking into this is that I'm trying to understand the different ways kernels can deal with preemption. Meaning, a kernel thread doing something on behalf of a user thread gets preempted by a higher-priority thread. I'm not talking about nested interrupts here (although they nearly always come with the territory).

Brendan wrote: B) It is a good, bad and ugly idea, depending on how your OS handles IRQ's in general. It's good if it gives you a cleaner way to invoke the scheduler from within nested IRQ handlers. I can't think of any good reason for Microsoft's "deferred procedure calls" (other than bad IRQ handler design).

If this is bad design, then MS is not the only guilty party... AFAIK Linux does more or less the same thing ("bottom-half handlers" are much like DPCs IIRC).

I agree that the need for it is a bit yucky... I'm aiming for a microkernel design myself, but as I recently discovered, apparently some microkernels are preemptable too (still trying to wrap my brain around that one). So, I want to understand all the tradeoffs involved before making a decision. Requiring the local APIC to be present is one aspect of the tradeoff I guess...

Seems the only way to do it nicely in nested interrupt stuff, making it plainer to implement. Do you know any other way?

I can't think of any other way if you want to keep the notion of preemptible kernel threads. If you just want stacked interrupts and still call the scheduler at reasonable times, you could use a counter like Minix's k_reenter that gets incremented, decremented, and checked by the ISR stub code.

Don't allow IRQs to be nested in the first place?

Bad idea if low latency is important.

If you do allow IRQs to be nested what are you going to do if there's no local APIC?

See above... at least that's my guess. k_reenter is yucky.

Colonel Kernel wrote:

Seems the only way to do it nicely in nested interrupt stuff, making it plainer to implement. Do you know any other way?

I can't think of any other way if you want to keep the notion of preemptible kernel threads. If you just want stacked interrupts and still call the scheduler at reasonable times, you could use a counter like Minix's k_reenter that gets incremented, decremented, and checked by the ISR stub code.

Don't allow IRQs to be nested in the first place?

Bad idea if low latency is important.

If you do allow IRQs to be nested what are you going to do if there's no local APIC?

See above... at least that's my guess. k_reenter is yucky.

Using a temp var, something like that, makes your code yucky. The nicest design you can probably come up with is an external device that automatically tells the CPU that it should interrupt, at the right time. This is exactly the APIC method.

BTW, I'm going for APIC-procs only so no problems for me on this one ... Got a model sheet hanging above my work space for reference... all opterons/durons/athlons printed out, all feature flags...

Hi,

Candy wrote: BTW, I'm going for APIC-procs only so no problems for me on this one ... Got a model sheet hanging above my work space for reference... all opterons/durons/athlons printed out, all feature flags...

Hmm - I'm more interested in the local APIC timer.

Have you tried to enable the local APIC on single CPU machines (Pentium to Pentium III)?

I've tried to re-enable the local APIC on these machines by resetting the CPU using the CMOS restart code (CMOS 0x0F = 0x0A), triple fault, etc but failed.

Otherwise will your OS be for multiple CPU computers (and possibly pentium IVs) only?


Cheers,

Brendan

Brendan wrote: Hi,

Candy wrote: BTW, I'm going for APIC-procs only so no problems for me on this one ... Got a model sheet hanging above my work space for reference... all opterons/durons/athlons printed out, all feature flags...

Hmm - I'm more interested in the local APIC timer.

Have you tried to enable the local APIC on single CPU machines (Pentium to Pentium III)?

Otherwise will your OS be for multiple CPU computers (and possibly pentium IVs) only?

No not yet... btw, atm AMD only, don't feel like doing syscall and sysenter both right now...

Candy wrote:Using a temp var, something like that, makes your code yucky. The nicest design you can probably come up with is an external device that automatically tells the CPU that it should interrupt, at the right time. This is exactly the APIC method.

You could rig this up without an APIC using just your good old-fashioned 8259 PIC and some other programmable interrupt source, say the 8253 PIT. Start each interrupt handler by pushing the current PIC mask and then masking the PIT, end it by restoring the PIC state. When you want your software interrupt, set the PIT to generate an immediate interrupt. Because the PIC is masking it, nothing happens immediately, but after all the nested IRQ handlers unwind, the last one off will reenable the PIT interrupts when it restores the PIC mask, and that will immediately generate the desired interrupt, without having to wait for the next scheduler interrupt and without having to screw around anything like a k_reenter var.

Dreamsmith wrote: You could rig this up without an APIC using just your good old-fashioned 8259 PIC and some other programmable interrupt source, say the 8253 PIT.

How would you ensure that regular clock interrupts are still delivered at high priority in this case...?

Colonel Kernel wrote:How would you ensure that regular clock interrupts are still delivered at high priority in this case...?

I leave that as an exercise for the reader.

I used the PIT in my example because I don't use the PIT for anything else in my OS, so masking it doesn't interfere with my regular clock interrupts (which come from the RTC), and it's about the easiest thing you can count on being in any PC that can generate an interrupt on demand.

If you're using the PIT for some other purpose and can't change that, you'd need to find some other hardware interrupt you could subvert instead. All you need is one hardware interrupt that (a) you don't need to respond to immediately (if it's one you're not using at all, so much the better, but that's not necessary), and (b) can make the appropriate hardware generate an interrupt for you on demand.

<smacks forehead/>
Forgive my n00b-ness with implementation details... I thought that the PIT and the RTC were the same thing! :-[

Colonel Kernel wrote: <smacks forehead/>
Forgive my n00b-ness with implementation details... I thought that the PIT and the RTC were the same thing! :-[

Heh. That's understandable. A lot of people use the PIT for what I (and many others) use the RTC for -- generating periodic interrupts for task switching, keeping time, etc. The PIT was in every PC all the way back to the early stone age, whereas the RTC was introduced with the PC/AT (late bronze age ;D). The reason I prefer using the RTC is because the PIT is much more flexible -- I don't want to tie up as wonderfully versatile a chip at the PIT with a task as mundane as clock interrupts, not when I have the CMOS RTC just sitting there.