7 APIC questions

[Coconut9]

1st) What's the difference between the local APIC and the ???? APIC?
2nd) Which of them handles external interrupts (keyboard...) and witch have his registers into 0xFEE00000 address mapped?
3rd) How many a (for example 4 core) computer have?
4rd) Can a core send an interrupt to another?
5th) Can only a core handles external interrupts but the others can handle interrupts from the first?
6th) If I am handling an external interrupt (form PIT) and the user press a key the key (interrupt) will never go to the cpu right?
7th) If I am handling an internal interrupt (from program) and the user press a key the key (interrupt) will go to the cpu right when the IF flag goes true right?

[BenLunt]

I am guessing you are asking what is the difference between the Local APIC and the I/O APIC.

Each processor core will have a Local APIC. This is the hardware that handles the interrupt portion for the core. Each Local APIC has an ID and has hardware to send messages to other Local APICs. The I/O APIC is the hardware that handles the interrupt portion of the attached devices, in turn sending these interrupts to the Local APICs. All are on an APIC Bus, all connected to one another. Each processor/core capable of handling interrupts will have a Local APIC. Usually, there is only one I/O APIC having up to 24 interrupt vectors. However, there can be more than one I/O APIC.

The vector map in the I/O APIC is used to send the external interrupt to a specified Local APIC. For example, if you have two processor cores, each with one Local APIC, you can send the Timer interrupt to the first and the keyboard interrupt to the second.

If you only have one processor/core, you send each external interrupt to the same Local APIC. If two interrupts are pending, the first one goes through while the second waits until you have sent the EOI for the first one. Interrupts are not lost.

On a slightly different note, each Local APIC may and might have a timer attached. If you enable and use this timer, the Local APIC handles this interrupt locally never making it out to the I/O APIC. With this in mind, each processor/core can have its own timer, kind of like each processor/core having its own INT 0 capabilities.

There are other things you must know too. For example, the I/O APIC now will have vector redirection for ISA interrupts. The ISA's INT0 timer will most likely be redirected to vector 2. If you do not handle these redirections, when you watch for vector 0 to fire, you will be waiting a long time. See the ACPI or Multi-Processor specification for redirections.

The Local APICs also handle the processor initialization process with each processor starting out in real mode just like the first processor. Something that you must remember, each processor must be moved to protected mode or what ever mode you will be working with for that processor before it can handle interrupts.

I am sure I have missed something, and there are others here that can shed some more light on the subject, but this should get you started.

Ben

[Coconut9]

I am guessing you are asking what is the difference between the Local APIC and the I/O APIC.

Each processor core will have a Local APIC. This is the hardware that handles the interrupt portion for the core. Each Local APIC has an ID and has hardware to send messages to other Local APICs. The I/O APIC is the hardware that handles the interrupt portion of the attached devices, in turn sending these interrupts to the Local APICs. All are on an APIC Bus, all connected to one another. Each processor/core capable of handling interrupts will have a Local APIC. Usually, there is only one I/O APIC having up to 24 interrupt vectors. However, there can be more than one I/O APIC.

The vector map in the I/O APIC is used to send the external interrupt to a specified Local APIC. For example, if you have two processor cores, each with one Local APIC, you can send the Timer interrupt to the first and the keyboard interrupt to the second.

If you only have one processor/core, you send each external interrupt to the same Local APIC. If two interrupts are pending, the first one goes through while the second waits until you have sent the EOI for the first one. Interrupts are not lost.

On a slightly different note, each Local APIC may and might have a timer attached. If you enable and use this timer, the Local APIC handles this interrupt locally never making it out to the I/O APIC. With this in mind, each processor/core can have its own timer, kind of like each processor/core having its own INT 0 capabilities.

There are other things you must know too. For example, the I/O APIC now will have vector redirection for ISA interrupts. The ISA's INT0 timer will most likely be redirected to vector 2. If you do not handle these redirections, when you watch for vector 0 to fire, you will be waiting a long time. See the ACPI or Multi-Processor specification for redirections.

The Local APICs also handle the processor initialization process with each processor starting out in real mode just like the first processor. Something that you must remember, each processor must be moved to protected mode or what ever mode you will be working with for that processor before it can handle interrupts.

I am sure I have missed something, and there are others here that can shed some more light on the subject, but this should get you started.

Ben

Thanks so much you answered all my questions that have after I read the Intel's manual.
Can you tell my now which is the problem with this APIC timer init code (qemu crash):

Code: Select all

mov ebx,0xFEE003E0
mov [ebx],dword 0x3
mov ebx,0xFEE00380
mov [ebx],dword 0x100
mov ebx,0xFEE00320
mov [ebx],dword 0x20040

And 3 small question for the end at address 0xFEE00000 there exists local APIC registers right?
Also to send interrupt from an local APIC to another I will use the 0xFEE00300-0xFEE00310 (ICR) register?
If I send EOI but the IF flag isn't set the next interrupt will lost?

[BenLunt]

Thanks so much you answered all my questions that have after I read the Intel's manual.
Can you tell my now which is the problem with this APIC timer init code (qemu crash):

Code: Select all

mov ebx,0xFEE003E0
mov [ebx],dword 0x3
mov ebx,0xFEE00380
mov [ebx],dword 0x100
mov ebx,0xFEE00320
mov [ebx],dword 0x20040

It is best to use EQUATEs within your code. For example, what register is:

Code: Select all

mov  ebx, 0xFEE003E0

compared to:

Code: Select all

mov  ebx, (APIC_BASE + APIC_DCR)

Even more important, compare:

Code: Select all

mov [ebx],dword 0x20040

with:

Code: Select all

mov [ebx],dword APIC_PERIODIC | 0x40

The reason for the crash is probably because you haven't a valid ISR at vector 0x40 for that interrupt.

And 3 small question for the end at address 0xFEE00000 there exists local APIC registers right?

If you mean, there are APIC registers at 0xFEE00000, then yes.

Also to send interrupt from an local APIC to another I will use the 0xFEE00300-0xFEE00310 (ICR) register?

Yes, noting that the "message" (IPI) is not sent until the write to the low dword takes place.

If I send EOI but the IF flag isn't set the next interrupt will lost?

This is why you have an ISR and a IRR register. Check the In-service regsiter and the Interrupt Request register.
You also need to handle Spurious interrupts.

[simeonz]

I want to ask for followup to the above clarification, if I may. Couldn't the BIOS set the IA32_APIC_BASE msr to another address during initialization. Probably not the case for qemu and a lot of the actual configurations, but wouldn't the robust procedure involve reading the msr and using the base address stored there for cpus with cpuid family 6 and above.

[BenLunt]

I want to ask for followup to the above clarification, if I may. Couldn't the BIOS set the IA32_APIC_BASE msr to another address during initialization. Probably not the case for qemu and a lot of the actual configurations, but wouldn't the robust procedure involve reading the msr and using the base address stored there for cpus with cpuid family 6 and above.

Good point. However, not all machines have MSR's.

It is good to query the ACPI for the address(es), or the MSR, or even the Multi-Processor specification.

Ben

[LtG]

I want to ask for followup to the above clarification, if I may. Couldn't the BIOS set the IA32_APIC_BASE msr to another address during initialization. Probably not the case for qemu and a lot of the actual configurations, but wouldn't the robust procedure involve reading the msr and using the base address stored there for cpus with cpuid family 6 and above.

Good point. However, not all machines have MSR's.

It is good to query the ACPI for the address(es), or the MSR, or even the Multi-Processor specification.

Ben

Of course always verify, never assume.

Ben, did you have something specific in mind? Aren't they _CPU_ (not system) specific? Are there some contemporary x86 CPUs missing them?

[BenLunt]

I want to ask for followup to the above clarification, if I may. Couldn't the BIOS set the IA32_APIC_BASE msr to another address during initialization. Probably not the case for qemu and a lot of the actual configurations, but wouldn't the robust procedure involve reading the msr and using the base address stored there for cpus with cpuid family 6 and above.

Good point. However, not all machines have MSR's.

It is good to query the ACPI for the address(es), or the MSR, or even the Multi-Processor specification.

Ben

Of course always verify, never assume.

Ben, did you have something specific in mind? Aren't they _CPU_ (not system) specific? Are there some contemporary x86 CPUs missing them?

There is a bit in one of the CPUID return registers that states whether there is a MSR or not. I would have to look at my notes to see if this means all MSR's or just the APIC address MSR. That's all I meant.

[Brendan]

Hi,

There is a bit in one of the CPUID return registers that states whether there is a MSR or not. I would have to look at my notes to see if this means all MSR's or just the APIC address MSR. That's all I meant.

There's a bit in CPUID that says if the CPU supports MSRs at all (e.g. the RDMSR and WRMSR instructions); and there's a bit that says if the CPU supports x2APIC.

The MSR for the local APIC base address either didn't exist or wasn't used for 80486 and early Pentium (where the local APIC was a separate chip and not built into the CPU), and then became "standard for Intel for now" after that. Other CPU manufacturers (AMD, VIA, etc) and future Intel CPUs are free to do something completely different because software is not supposed to rely on the MSR and is supposed to use ACPI or MultiProcessor Spec tables instead (in other words, the MSR should only be used by firmware, because firmware is designed for a motherboard that only supports a specific small range of CPUs).

Also note that when there's more than 255 CPUs the firmware must use x2APIC for at least some CPUs (because not all "APIC IDs" can fit in 8 bits), and when there's less than 255 CPUs the firmware may still use x2APIC for some or all CPUs. When a CPU is using x2APIC the (memory mapped) "local xAPIC area" should not be touched (unknown consequences). This means that you need to parse ACPI's "APIC/MADT" table to determine if the OS should use x2APIC (and if the OS should switch CPUs to "x2APIC mode" if they're not using it already) before you use local APICs, and it's trivial to get the local APIC base address from ACPI while you're parsing this table.

If there's no ACPI tables (old computers); then you'd fall back to MultiProcessor specification tables and get the local APIC base address from there (and assume that x2APIC can't exist).

For both cases (ACPI and the MultiProcessor specification) there's a "local APIC version" field that you can check to determine if the local APIC is built into the CPU itself (which might or might not support the "local APIC base address" MSR), or if the local APIC is a separate external chip (which will not support the "local APIC base address" MSR). If the local APIC is a separate external chip then I treat the computer the same as "single CPU without local APIC" (it's not worth the hassle of supporting multi-CPU and local APIC for these old computers, even if you do support 80486 and early Pentium).


Cheers,

Brendan

[simeonz]

I will refer to Linux again, because it is a heavily used code base. Brendan's statements are correct, as far as I can see. Linux uses the following fallback strategy (from high to low precedence):

1. "Local APIC Address" field in the "Local APIC Address Override" structure in "Multiple APIC Description Table (MADT)" from ACPI
2. "Local Interrupt Controller Address" field in the "Multiple APIC Description Table (MADT)" header appendix from ACPI
3. "Address of Local APIC" field in the "MP Configuration Table Header" from MPS
4. IA32_APIC_BASE (a.k.a. APIC_BASE) msr for specific CPU families (family 6, some models in family 15, for "AuthenticAMD" and "GenuineIntel" products only)
5. 0xFEE00000 as default

The kernel assumes IA32_APIC_BASE presence for Intel and AMD with family 6 and above. According to this list, fragile as it may be, those are post-Pentium Pro and post-Athlon cpus (ignoring Cyrix and other vendors). Looking at the Athlon's programmer's manual (16.3.1 Local APIC Enable), the MSR has the same format as Intel's, so the assumption tentatively appears correct. If you need to use the IA32_APIC_BASE, you must verify you have CPUID using EFLAGS, execute request type 0 and gather the vendor id/string, execute CPU ID request type 1 if available (if not assume no MSR), and check the MSR feature bit and the familty id, which must be at least 6. The vendor string must be either "AuthenticAMD" or "GenuineIntel" as pointed above or the MSR cannot be relied upon.

I would also note that on Itanium, the ACPI points to the Processor Interrupt Block, not the APIC registers base. It doesn't matter much, because nothing works the same on Itanium and probably noone here wants to support it anyway.

Regarding the debugging, there is apparently an ongoing effort to expose the x86 MSRs through a monitor command of the qemu's gdb stub. However it is not in the master branch yet. There is a command "monitor info lapic" (and "monitor info ioapic" for that matter) which shows the configuration state of the lapic. So, you may be able to check its output after every instruction that manipulates the memory mapped registers and verify that the results reflect the modification.

Some references that I used (which probably duplicate the wiki):
ACPI 6.2 spec
MPS 1.4 spec
x86 Families and Models List
x86, x64 Instruction Latency, Memory Latency and CPUID dumps
(Some) unofficial Intel MSR list