APIC, SAPIC, xAPIC and x2APIC

[snooky]

Hi everybody!

My system is running fine in the smp environment. Thanks to your help about MP tables (especially thanks to Brendan!).

As I was implementing all the features I came across some different terms related to APICs. Until then, I thought there was the (one and only) APIC. Now there come the names "SAPIC" and "xAPIC" into play. I didn't find precise information about how APIC - SAPIC - xAPIC relate to each other.

In the IA32/Intel64 Manuals they are writing about xAPICs and apparently the "x" stands for "extended". But what is the extension in comparison to the "normal" APICs. Maybe I overlooked something, but I found no difference. Can someone please explain the extension compared to the "old"/"normal" APICs?

For the x2APICs, the successors of the xAPICs, the difference is clear, although I don't feel that the programming per MSRs is much more beneficial over accessing memory mapped registers. Why should it be so much better?

For "SAPIC", streamlined APIC, it's more simple. There is a Intel document that presents the SAPIC architecture and it says something like "SAPIC is very cool, better than the old APIC and you can program it like you ever did it". So the difference is the implementation (hardware) itself. Can someone please clarify what's "streamlined" there and where the advantage comes from?

Thank you in advance!
Snooky

[Brendan]

Hi,

In the IA32/Intel64 Manuals they are writing about xAPICs and apparently the "x" stands for "extended". But what is the extension in comparison to the "normal" APICs. Maybe I overlooked something, but I found no difference. Can someone please explain the extension compared to the "old"/"normal" APICs?

First there was the 82489DX (an external chip for 80486 CPUs) which had 8-bit APIC IDs and therefore supported up to 255 devices (where a device is a CPU or an I/O APIC). Then they introduced the "local APIC" (which was built into the CPU, for Pentium and P6) which reduced the APIC IDs to 4-bit and therefore only supported 15 devices. Next they introduced the xAPIC, which is probably called "extended" because the APIC IDs are 8-bits again and not 4-bits.

There's a few other differences with XAPIC - no support for cluster logical destination mode, no support for focus processors, and, some additional local APIC interrupt vectors for thermal monitoring and performance monitoring.

For the x2APICs, the successors of the xAPICs, the difference is clear, although I don't feel that the programming per MSRs is much more beneficial over accessing memory mapped registers. Why should it be so much better?

For programmers it doesn't make much difference, except when you try to use 64-bit local APIC registers. For example (even in long mode), if you want to write to the Interrupt Command Register you need to do it as 2 seperate 32-bit writes and you can't do it as a single 64-bit write. This means that you need to worry about an IRQ messing things up after you've written half of it. For MSRs, the WRMSR and RDMSR instructions can handle the full 64-bits atomically.

For hardware, it's a bit more complex. Every time the CPU reads from or writes to the physical address space, the CPU needs to do a lot of work (checking to see if the access hits the local APIC, checking if A20 is enabled or not, finding the address in the MTRRs, etc). To speed things up the CPU probably does these things in parallel, which makes it hard to skip unnecessary work. For e.g. if an access hits the local APIC there's no real need to look up the address in the MTRRs to see what sort of caching to use.

Using MSRs instead simplifies the work needed to be done for normal physical address space accesses (for e.g. no need to do "if( (size == 4) && ( (address & 3) == 0) && ( (address & 0xFFFFF000) == localAPICbase) { /* foo */ } else { /* bar */ }") and also simplifies the work needed to be done for local APIC accesses.

It'd make things much easier for CPU designers if MSRs were used for local APIC and nothing else. Unfortunately Intel can't really switch to "MSR only" for the local APIC without risking too much market share, because the new CPUs won't work with the old OSs. Instead they need a migration path, so the CPUs support "legacy xAPIC mode" and "x2APIC mode" now, and some time in the future (when all major OSs support x2APIC and Intel's sales won't be effected so much) the CPUs might only support "x2APIC mode".

Note: once "x2APIC mode" is enabled it can't be disabled by software, and if there's more than 255 CPUs the BIOS is meant to enable x2APIC mode before the OS boots. This means that any OS that doesn't support x2APIC will be useless for high-end servers in about 5 years, so most OSs will support x2APIC before then, and Intel will probably be able to get rid of "legacy local APIC mode" after that.

For "SAPIC", streamlined APIC, it's more simple. There is a Intel document that presents the SAPIC architecture and it says something like "SAPIC is very cool, better than the old APIC and you can program it like you ever did it". So the difference is the implementation (hardware) itself. Can someone please clarify what's "streamlined" there and where the advantage comes from?

I don't know much about SAPIC, but AFAIK it's for Itanium (and not for 80x86). However, the original Itanium had a "80x86 compatability mode" so it was possible to run a "slightly modified" 80x86 OS directly on an Itanium. Performance sucked though (later they ditched the 80x86 compatability mode and used emulation instead because emulation was faster). Sane people don't buy Itaniums to run 80x86 software, so IMHO there's not much point worrying about SAPIC (unless you port your OS to native Itanium code)...


Cheers,

Brendan

[snooky]

Thanks Brendan for your excellent explanations!

It would be nice if you could say something about AMD processors and their local APICs. Is there something like an xAPIC or x2APIC?

At least I found nothing that looks like a AMD's version of a x2APIC. Did I miss it or is there simply none or did AMD announce it?

For the xAPIC it's a bit different - I think that AMD uses local APICs that behave like xAPICs/are similar to xAPICs (if you switch them to 8 bit ID operation).

So long,
Snooky

[Brendan]

Hi,

It would be nice if you could say something about AMD processors and their local APICs. Is there something like an xAPIC or x2APIC?

In general, the problem with AMD is that the local APIC isn't in the normal manual, and you need a different "Kernel and BIOS developer guide" for each CPU. This is annoying - you can't find all the information for all their CPUs in the same place.

At least I found nothing that looks like a AMD's version of a x2APIC. Did I miss it or is there simply none or did AMD announce it?

I'm not sure if AMD support x2APIC yet - it might take them a while before they find time.

For the xAPIC it's a bit different - I think that AMD uses local APICs that behave like xAPICs/are similar to xAPICs (if you switch them to 8 bit ID operation).

Modern AMD CPUs do have a "local APIC mode" and an "xAPIC mode", but I'm not too sure which mode BIOSs use or which AMD CPUs do this. I'm fairly sure that none of AMD's CPUs ever supported the 82489DX (external APIC). Also, I'd expect some minor differences between AMD and Intel, especially with local APIC interrupts (the thermal management interrupt, the performance monitoring interrupt, etc).

I guess what I'm saying is that I haven't had time to read through each "Kernel and BIOS developer guide" yet - I've only glanced at some of them. This is partly because (until recently) I haven't had any computers with AMD CPUs that support SMP or local APICs.


Cheers,

Brendan