Where to learn about multicore scheduling?

[vinnie]

I have written pre-emptive multitasking OSes for MCUs with single CPUs and this is my first time having a go at multicore ARM, AMD and Intel CPUs. I'm a bit confused by how you go about scheduling tasks for multicore CPUs etc. I could not see anything talking about it on the main page either. Before I spend money on buying some books, does anyone have a recommendation for particular books or other resources that helped them?

[vhaudiquet]

Are you looking for more theory, like scheduling algorithms, or how to actually send messages between processors and tell them to execute code ?

If you want theory, i don't think there is a lot in here, but almost all of the books i know about OS Development are talking about it.

If you want technical details, for x86 you'll have to use local APIC and IPIs (Inter Processor Interrupts), and ACPI to detect processors (the MADT), more details on the wiki :
https://wiki.osdev.org/APIC
https://wiki.osdev.org/ACPI
https://wiki.osdev.org/MADT

Those pages might help you :
https://wiki.osdev.org/Multiprocessing
https://wiki.osdev.org/SMP

[vinnie]

Thanks for those links, I'll read them again.

Not so much theory or technical, something that bridge the two a bit more. What I was hoping to see is a logical steps / roadmap to get from 0 to 100%. Your statement below has been very helpful in that regard:

... for x86 you'll have to use local APIC and IPIs (Inter Processor Interrupts), and ACPI to detect processors (the MADT) ...

Some of the questions I have are:

• Are there separate MMUs per Core or do you share the same MMU?
• If you implement a scheduler per core, do you run a separate process and stack for each?
• I always wrote pre-emptive OSes for MCUs such that the Kernel sits on-top of the scheduler and the Kernel is a process itself inside the scheduler. Is this a good way of doing it on modern CPUs?

I suspect many of these questions have no right or wrong answer, but I expect there are pros and cons to each and I want to understand that better while designing the OS.

[Octocontrabass]

Are there separate MMUs per Core or do you share the same MMU?

For the most part, you can treat each hardware thread as a fully independent CPU, with all the parts that a single-thread CPU would have, including MMU. (There are a handful of MSRs shared between hardware threads or cores, but you probably don't need to worry about those.)

• If you implement a scheduler per core, do you run a separate process and stack for each?
• I always wrote pre-emptive OSes for MCUs such that the Kernel sits on-top of the scheduler and the Kernel is a process itself inside the scheduler. Is this a good way of doing it on modern CPUs?

Either I'm not understanding you correctly or you're describing a system that would be considered very exotic outside a microcontroller.

[nullplan]

If you implement a scheduler per core, do you run a separate process and stack for each?

For stacks, a common model with desktop OSes is to have one kernel stack per task, as well as a few exception stacks per CPU. The exception stacks are for the critical exceptions Double Fault, Machine Check, and Non-Maskable Interrupt, because those can happen any time, including half-way through a stack switch. But they then get used very rarely.

In any case, this model supports quite a simple way of doing multi-tasking, where each task is just suspended on its own stack, in whatever state it wants. And switching tasks just means switching stacks.

The only alternative I have ever heard is to have one kernel stack per CPU. In that model you have to enumerate all the reasons why a task might be blocked, and switching tasks means figuring out what was going on in the new task before it got switched out. Also possible, but a bit harder.

But you always need separate CPUs to operate on separate stacks. Having them on the same stack can only lead to disaster.

I always wrote pre-emptive OSes for MCUs such that the Kernel sits on-top of the scheduler and the Kernel is a process itself inside the scheduler. Is this a good way of doing it on modern CPUs?

In that case we have a very different understanding of what a kernel is. To me, a kernel is a collection of functions. It does contain the system initialization function, yes, but that ends somewhere, and then the important ones are the system calls and the interrupt handling functions. Such a collection of functions cannot work as a coherent process, because it does not follow a coherent thread of execution.

Sure, a microkernel might move a lot of the interrupt processing to user space processes, but all that means is that the kernel still handles the interrupt, but then activates another process in response to it. The kernel has to handle all the interrupts, at least on the CPU level.