Hi,
Candy wrote:
Yes, it's a good plan only to do MP/SMP. You will probably be able to use some form of SMP emulation that the opteron might have (something in the back of my head says this but I can't confirm nor deny), and you can probably get all memory as it's mapped by the BIOS to a linear segment
A NUMA/Opteron/64bit computer will work fine with a non-NUMA MP 32bit OS, the OS just won't take full advantage of the computers features. Access to memory won't be optimized and the OS wouldn't take advantage of the 64 bit registers (and additional registers). The 32 bit OS would still be able to access all physical memory though (PAE) and use all CPUs (MP/SMP).
No, it's a bad plan. Using those registers you can map memory everywhere you like it to be, offer advanced memory configuration and allow use of all memory without losing valuable megabytes on PCI mmapped IO. You can also create more optimized routing tables for the opterons and support weird configurations, and display the configuration in a more clear way.
My 32 bit kernel could optimize for NUMA memory accesses, but IMHO people with opteron servers will want a 64 bit OS anyway. On my OS linear memory isn't so valuable, as a device driver has an entire address space (3 Gb) for it's own private use. Access to the physical address space in 32 bit with PAE has the same restrictions as access in 64 bit.
Eventually I'll write a kernel for 64 bit. This kernel would support NUMA, but the NUMA support will be handled by the kernel (it'd be invisible to applications) and wouldn't effect the kernel API.
Therefore what I think I need (for the 32 bit kernel) is support for CPU affinity and CPU clustering. IMHO supporting CPU affinity is a good idea anyway, as it can make better use of CPU caches. CPU clustering is required for decent hyper-threading support - e.g. an SMP computer with 2 physical CPUs that contain 2 logical CPUs each would be considered 2 CPU clusters. In this case the CPU affinity support would try to make a thread run on the same CPU it ran last time, or if that's not practical it would try to make the thread run on a CPU in the same cluster (although this would depend on CPU load balancing). It's all about recycling shared caches, and making sure the kernel API is consistant across different kernel versions.
Also I fully expect AMD to add "hyper-threading" like technology to opterons, and wouldn't be surprised if Intel started using NUMA. This could easily lead to an 8 way motherboard with 32 logical CPUs (8 clusters of 4). Please note I'm talking about CPU clustering where all memory is directly accessable by all CPUs, not the CPU clustering done by mega-beasts (eg. Intel & IBM blade servers).
Cheers,
Brendan