Implementing SMP

[oked]

Hello

I would like to add SMP support to my kernel, i know what is required to know about spinlocks, but even after reading various information i still dont understand how to work with the apic and wake up and process all the other processors in the system.
and how exactly is hyper threading support should be programmed? is there any special restrictions i need to take into account?

if someone could please point me to a tutorial or even a piece of simple working code of how to easily implement SMP in a kernel (without regarding spinlocks) : that is:
initialize the APIC
intiialize other processors etc
use IPI to start schedule

how do I schedule a certain thread to a certain CPU?

'Multiprocessing Support for Hobby OSes Explained' is a really great article but is very unfinished and uninformative in many places, does any one know any other article like that with maybe more details and maybe more code?

Is there any place I can see some simple code of how to just make those things?

thanks!!!

[Brendan]

Hi,

I would like to add SMP support to my kernel, i know what is required to know about spinlocks, but even after reading various information i still dont understand how to work with the apic and wake up and process all the other processors in the system.

First, you need to detect which CPUs are present and get the local APIC ID for each one. To do this you'd look through Intel's MP Specification tables, or the ACPI equivelents.

For Intel's MP Specification, see:
http://www.intel.com/design/intarch/MANUALS/242016.htm

For ACPI Specifications, see:
http://www.acpi.info/

For both of these, there's tables in memory (or in ROM) that tell you things about the computer. You'd want to find CPU information, and (eventually) IO APIC/IRQ information.

I'd recommend starting by reading through Intel's MP specification from start to end (forget ACPI for now). Then read through the MP, APIC and initialization sections of your Intel System Programming Guide (Chapters 7, 8 and 9) skipping anything you already know.

Once you know the local APIC ID for each CPU, you'd use the boot processor (BSP) to send a special startup sequence to the other CPUs (the AP CPUs) using the BSP's local APIC. There's 2 different sequences, depending on if it's a newer CPU with a built in local APIC or an older CPU with an external local APIC (82489DX chip, used with 80486 and older CPUs). For the older CPUs it's just an "init IPI". For newer CPUs it's an "init IPI", a "startup IPI" and then another "startup IPI". For both of these there's timing requirements - see Intel's MP Specification for exact details.

For newer CPUs, this will make each AP CPU start running in real mode at the address you specified in the "startup IPI".

For older CPUs, the AP CPUs will start running the BIOS's boot code, so you need to set the reset code in the CMOS and the "warm start" vector at 0x0467. To be honest, I don't think I'm going to bother supporting multi-CPU 80486 or older computers anymore - it's not worth the hassle (I can't even find one on eBay to test my code on). This would make it much easier because you'd only need to worry about one AP startup method and you can skip the BIOS warm boot stuff.

[continued]

[Brendan]

[continued]

and how exactly is hyper threading support should be programmed? is there any special restrictions i need to take into account?

For hyper-threading, starting a logical CPU is done exactly the same as above. There are differences for detecting hyper-threading CPUs though. The MP specification tables will ignore them, so you'd need to use CPUID to see if it's supported before starting the logical CPUs. The ACPI tables will report them, but won't tell you if it's a logical CPU or a completely seperate CPU (use CPUID to find out).

Apart from this there are no special restrictions, but there are things you can do to improve performance - the PAUSE instruction in spinloops, making the scheduler balance load, etc. For example, if you've got a computer with 2 physical CPUs (chips) and hyper-threading (total of 4 logical CPUs) then it's better if one logical CPU on each chip is idle, rather than both logical CPUs on one chip idle while both logical CPUs on the other chip are in use.

if someone could please point me to a tutorial or even a piece of simple working code of how to easily implement SMP in a kernel (without regarding spinlocks) : that is:
initialize the APIC
intiialize other processors etc
use IPI to start schedule

There are no simple tutorials that cover everything - I mostly learnt from the MP Specification, ACPI and Intel system programmers manual, and a heap of trial and error. IIRC the tutorials that are available have errors, for example 'Multiprocessing Support for Hobby OSes Explained' assumes there's one IO APIC with 24 inputs where each input is always connected to the same thing (this is wrong - you have to detect how many IO APICs there are, how many inputs each has and what each input is connected to via. the MP Specification tables and/or the ACPI tables).

how do I schedule a certain thread to a certain CPU?

There's a large number of things that can effect scheduler design - my advice is to forget about the scheduler until you've got the other CPUs started and initialized (i.e. all CPUs running in protected mode with paging).

Is there any place I can see some simple code of how to just make those things?

Simple code for a relatively complex topic is rare. It'll probably take you a week to read through and understand the recommended material, which will give me a week to get some system topology stuff out the way. After this it's likely that we'll both be writing code to start AP CPUs at the same time, and my code will be constantly updated on my web site for you to read/watch. Otherwise, feel free to ask anything necessary ..


Cheers,

Brendan