Confusion with Intel history (Local/IO APIC question)

[gusc]

I don't know it it's relevant, but I don't get the history of Intel architecture. They had a P6 arch starting with Pentium Pro (am I right?) and then came Pentium 4, which, as we all know, was a complete **** (even Intel knew that) and then came Core arch which was combination of P6 and those few good things from Nahlem, right? So, Intel's manual just goes around and says Pentium 4 and P6 architectures all around. Now the confusion for me is that - if I'm targeting Sandy/Ivy Bridge, does that mean that I have to point my attention to P6 architecture? As for example, is Local APIC and IO APIC connected through System Bus or APIC Bus? I don't know whether it's relevant or not, but I'd like to make it clear what is Intel referring to when they write P6 architecture.

Confusion #2 is with my OS and APIC's. I've my test environment broadened up a notch:

• Bochs (256Mb RAM, 2 CPUs)
• QEmu (256Mb RAM, 2 CPUs)
• VirtualBox (256Mb RAM, 2CPUs, VT-x and PAE enabled)
• VMWare Player (256Mb RAM, 2CPUs, VT-x enabled, I don't know about PAE though)

I'm enumerating all the APIC's through ACPI MADT and it shows different results in VirtualBox and VMWare - weren't they supposed to run a virtualization, thus pass through almost all of the system specs (like ACPI tables)? The thing is that VirtualBox shows 2 Local APICs, while VMWare shows something around 60!!!

And last but not least. There should be 1 Local APIC per core and a single IO APIC per system, right? But MADT has only a single address of Local APIC (same goes if I query a MSR). How do I get the other ones, or are they mapped per CPU? Funny enough, but enumerating them through ACPI it returns 2 entries for Local APIC, but these entries do not hold an address. I can just use them to enumerate how many CPUs there are.

Sorry for being a noob

PS. VMWare also has some bug with E820 memory map - it shows that I have 4GB of RAM instead of 256MB that I've configured.

[gusc]

I think I got #3 my self - I just remembered something about APIC ID's, Processor ID's and Interprocessor Interrupts. These ID's are used to send a messages with IPIs to other cores, thus poking and enabling them, right? And that's why there is only single Local APIC address available - it point's to current core that is executing this boot code.

[Opcode]

1) System Bus
2) Dunno that software
3) By default all LAPIC have the same memory mapped base address. This can however be overridden.

[gusc]

Just solved the mistery with VMWare Player and 60 Local APICs - it turned out that all of these Local APICs where flagged as disabled (and ACPI manual simply says - don't touch them).

1) System Bus

Let me rephrase the question - should I look at the P6 or Pentium 4 arch if I'm targeting Sandy/Ivy Bridge?

[Opcode]

You should look at the sandy/ivy bridge micro architecture if you're targeting sandy/ivy bridge architecture.

See Intel manual #3 section 10.2

[Combuster]

is Local APIC and IO APIC connected through System Bus or APIC Bus? I don't know whether it's relevant or not, but I'd like to make it clear what is Intel referring to when they write P6 architecture.

Both. Actual interrupt events are transferred over the APIC bus, whereas the processor configures the IO APIC over the system bus. The practical value of how a motherboard is wired should however not be your concern: you as a programmer won't notice anyway.

The local APIC is wired onto the processor core and there's no bus worth mentioning in between - in fact, when you address the local APIC you always get the one for the core you're executing from.

a single IO APIC per system

No, there can be any number. There must be at least one on a SMP system since the PIC can't distribute interrupts amongst cores. Beyond that, an IOAPIC serves a limited number of interrupt lines. Add a second, and you get more sources of interrupts that you're able to handle without needing to put devices on the same interrupt line - just as a second PIC was introduced in PCs because one wasn't possibly going to be enough.

if I'm targeting Sandy/Ivy Bridge, does that mean that I have to point my attention to P6 architecture?

The needed SMP setup code has seen no fundamental changes since the P5 architecture. You are free to look at the latest features and use them, but there's little advantage and a lot of complexity to gain there whereas in doing so you'd give up the ability to boot other cores or processors on all pentiums, atoms, opterons and athlons and then some. Are you ready to rewrite your entire code when you get your next computer?

[gusc]

You should look at the sandy/ivy bridge micro architecture if you're targeting sandy/ivy bridge architecture.

See Intel manual #3 section 10.2

OK, let's try this again, shall we? From the Intel's manual:

For the P6 family and Pentium processors, the I/O APIC and local APICs communicate through the 3-wire inter-
APIC bus (see Figure 10-3). Local APICs also use the APIC bus to send and receive IPIs. The APIC bus and its
messages are invisible to software and are not classed as architectural.
Beginning with the Pentium 4 and Intel Xeon processors, the I/O APIC and local APICs (using the xAPIC architec-
ture) communicate through the system bus (see Figure 10-2). The I/O APIC sends interrupt requests to the
processors on the system bus through bridge hardware that is part of the Intel chip set. The bridge hardware
generates the interrupt messages that go to the local APICs. IPIs between local APICs are transmitted directly on
the system bus.

Now my question was: Weather Sandy Bridge and Ivy Bridge (2nd and 3rd generation Intel Core CPUs) are based on P6 architecture or Nahlem (Pentium 4 nd Intel Xeon, mentioned above)?

[Owen]

The Pentium M and all descendant CPUs (Core, Core 2, Nehalem, Sandy Bridge, Ivy Bridge, Haswell) are all derived microarchitecturally form the P6, but not considered part of the P6 or Pentium "family". (The Pentium 4 had the NetBurst microarchitecture, for completeness).

What Intel say is absolutely accurate: in all Intel CPUs manufactured since the Pentium 4, interrupts travel via the system bus (which may be QPI on some CPUs, and possibly DMI on others)

None of this is relevant to you; it all works the same from a software perspective (except perhaps if you're a firmware developer)

Remember that two other companies manufacture x86 systems (The often forgotten VIA, and much less forgotten AMD)

[Opcode]

For the P6 family and Pentium processors, the I/O APIC and local APICs communicate through the 3-wire inter-
APIC bus (see Figure 10-3). Local APICs also use the APIC bus to send and receive IPIs. The APIC bus and its
messages are invisible to software and are not classed as architectural.
Beginning with the Pentium 4 and Intel Xeon processors the I/O APIC and local APICs (using the xAPIC architec-
ture) communicate through the system bus (see Figure 10-2). The I/O APIC sends interrupt requests to the
processors on the system bus through bridge hardware that is part of the Intel chip set. The bridge hardware
generates the interrupt messages that go to the local APICs. IPIs between local APICs are transmitted directly on
the system bus.

Now my question was: Weather Sandy Bridge and Ivy Bridge (2nd and 3rd generation Intel Core CPUs) are based on P6 architecture or Nahlem (Pentium 4 nd Intel Xeon, mentioned above)?

[Brendan]

Hi,

And last but not least. There should be 1 Local APIC per core and a single IO APIC per system, right?

No. For a system with hyper-threading there should be 1 local APIC per logical CPU (and typically 2 logical CPUs per core). For completeness, a computer consists of one or more NUMA domains; each NUMA domain has zero or more physical CPUs; each physical CPU has one or more cores; each core has 1 or more logical CPUs; and each logical CPU has a local APIC.

There can be multiple IO APICs. One of the computers here has 2 IO APICs, and I've seen specs for computers/servers with 4 IO APICs.

But MADT has only a single address of Local APIC (same goes if I query a MSR). How do I get the other ones, or are they mapped per CPU? Funny enough, but enumerating them through ACPI it returns 2 entries for Local APIC, but these entries do not hold an address. I can just use them to enumerate how many CPUs there are.

All local APICs are at the same physical address. A CPU can only access its own local APIC and can't access a different CPU's local APIC.


Cheers,

Brendan

[Opcode]

All local APICs are at the same physical address. A CPU can only access its own local APIC and can't access a different CPU's local APIC.

Moot point but I'm pretty sure you can "unchain" them and give each LAPIC a unique physical address; however I'd assume there is no "snooping" thus the second part of that statement is true. Not sure if there would be any be advantage of unchaining these but I believe the option is there. (In a bit of a hurry now, need to check the scope of the LAPIC base MSR--i.e. does it control the whole package?)

[Opcode]

Alright I'm back. Yeah I just checked it and yeah the MSR is unique. Besides now that I think about it if you're targetting Ivy/Sandy you don't even need to worry about MMIO, you can just access the LAPIC via MSR.

Oh and btw, congrats to Opcode on his first gold star!

[gusc]

Thanks guys, that cleared up some of confusion for me. It's still a little bit confusing with the P6/Nahlem thing and modern CPUs, but I'll assume that from now on I have to look at Nahlem+ in the manuals not P6.

[gusc]

Remember that two other companies manufacture x86 systems (The often forgotten VIA, and much less forgotten AMD)

I'm taking a step back here (from modern trends) - I'm targeting a single architecture to build a software (not even an OS) that runs directly on the hardware. Let's say it's a Bare Metal framework.

[dozniak]

Thanks guys, that cleared up some of confusion for me. It's still a little bit confusing with the P6/Nahlem thing and modern CPUs, but I'll assume that from now on I have to look at Nahlem+ in the manuals not P6.

Are you misspelling Nehalem on purpose? Please, stop.