OSDev.org

How can an OS use the other cores on a multicore processor?

It's fairly simple. The OS detects the other cores, boots them and puts them into an operating state. How you deal with the cores is up to you.

The cores are detected by parsing either the ACPI or MP tables. In most computers, the ACPI tables will be more up-to-date. You search the tables for LAPIC entries, and send every usable LAPIC a specific sequence of interrupts; this wakes the core and tells it to start executing code at the physical address you specify as part of the interrupt. This code needs be located in the first MiB of memory, and should set up enough state to get to the necessary CPU mode.

Something to note is that it isn't just your code that has to be in the first MiB of memory. Your GDT needs to be there too - the IDT and paging tables could just stay where they are, since after you set up the GDT you can enable protected mode, giving you access to the first 4 GiB of memory. The paging tables use physical addresses, and the IDT could be loaded after the page tables, as long as interrupts stay disabled on the core you're booting until after a stable environment is set up.

After everything's booted, you have to make the cores communicate. This works in a similar manner to how you booted the core to start with - send it an interrupt. You'll generally have to send interrupts (because they're between cores they're often called inter-processor interrupts or IPIs) to invalidate page table entries, and deal with OS-specific things. I think Brendan posted a list somewhere, but I can't find it for some reason.

The main reason you'll want to boot other cores is so that your schedulers can handle multiple execution threads simultaneously. It's fairly common, I think, to have a list of processes per core and the ability to shift processes between cores (CPU affinity).

An idea I've been toying with for a few days is to detect if multiple cores are present, and boot them all, except for one, into long mode. The remaining one stays in protected mode. There could be a thread which is bound to that core, to handle v86 processes such as VBE mode enumeration. By sending a message to this process for it to execute code, it'd be possible to get the benefits of v86 mode combined with the benefits of long mode. The disadvantages though, are that caches could get flushed unnecessarily, and that you'd need to be extraordinarily careful with memory allocations. Allocating an address above the 4 GiB mark and giving it to a protected mode core would result in data corruption and/or page faults. You'd also need two sets of page tables - one for the protected mode core, another for the standard cores.

I'm not sure of whether this had been tried before, but it seems like a fairly obvious solution to the lack of v86 mode support in long mode. If any of the more knowledgeable people here could see any problems with it, I'd appreciate any feedback.

computafreak wrote:It's fairly simple. The OS detects the other cores, boots them and puts them into an operating state. How you deal with the cores is up to you.

The cores are detected by parsing either the ACPI or MP tables. In most computers, the ACPI tables will be more up-to-date. You search the tables for LAPIC entries, and send every usable LAPIC a specific sequence of interrupts; this wakes the core and tells it to start executing code at the physical address you specify as part of the interrupt. This code needs be located in the first MiB of memory, and should set up enough state to get to the necessary CPU mode.

Something to note is that it isn't just your code that has to be in the first MiB of memory. Your GDT needs to be there too - the IDT and paging tables could just stay where they are, since after you set up the GDT you can enable protected mode, giving you access to the first 4 GiB of memory. The paging tables use physical addresses, and the IDT could be loaded after the page tables, as long as interrupts stay disabled on the core you're booting until after a stable environment is set up.

After everything's booted, you have to make the cores communicate. This works in a similar manner to how you booted the core to start with - send it an interrupt. You'll generally have to send interrupts (because they're between cores they're often called inter-processor interrupts or IPIs) to invalidate page table entries, and deal with OS-specific things. I think Brendan posted a list somewhere, but I can't find it for some reason.

The main reason you'll want to boot other cores is so that your schedulers can handle multiple execution threads simultaneously. It's fairly common, I think, to have a list of processes per core and the ability to shift processes between cores (CPU affinity).

An idea I've been toying with for a few days is to detect if multiple cores are present, and boot them all, except for one, into long mode. The remaining one stays in protected mode. There could be a thread which is bound to that core, to handle v86 processes such as VBE mode enumeration. By sending a message to this process for it to execute code, it'd be possible to get the benefits of v86 mode combined with the benefits of long mode. The disadvantages though, are that caches could get flushed unnecessarily, and that you'd need to be extraordinarily careful with memory allocations. Allocating an address above the 4 GiB mark and giving it to a protected mode core would result in data corruption and/or page faults. You'd also need two sets of page tables - one for the protected mode core, another for the standard cores.

I'm not sure of whether this had been tried before, but it seems like a fairly obvious solution to the lack of v86 mode support in long mode. If any of the more knowledgeable people here could see any problems with it, I'd appreciate any feedback.

That sounds like one of the worst ideas i've heard in a long time. So, one of my cores is going to sit around doing mostly nothing until I want to change video resolutions? Why not just drop to real mode when I want to make the change? Or, use an x86 emulator to run the 16-bit code from long mode? I just don't see the point in leaving an idle cpu laying around just to handle mode changes. What happens when (if?) you get proper video drivers implemented? Then what will the lone real mode cpu do?

You could use VMX/SVM extensions to run stuff in real mode too, since that's supported by most processors these days. One of the first things I did with VMX was to create a VM86-based real-mode emulator (you must, actually, because most VMX implementations do not support real-mode directly). I used it to put the computer into 320x200x256 using BIOS mode 13h, while my OS operated in protected mode, and I wrote a little old school graphics demo to try it out.

You could allow any core to transition between protected and long mode as necessary, instead of having one core dedicated to 32-bit tasks. You would need to add support for v8086 tasks in your scheduler, and make sure no interrupts get sent to the core in questions while its not in long mode (no IPIs, no hardware interupts, etc.). However, this would be quite a tricky procedure and if you are doing it just for VESA its probably not worth it.

Ready4Dis wrote:

computafreak wrote:An idea I've been toying with for a few days is to detect if multiple cores are present, and boot them all, except for one, into long mode. The remaining one stays in protected mode. There could be a thread which is bound to that core, to handle v86 processes such as VBE mode enumeration. By sending a message to this process for it to execute code, it'd be possible to get the benefits of v86 mode combined with the benefits of long mode. The disadvantages though, are that caches could get flushed unnecessarily, and that you'd need to be extraordinarily careful with memory allocations. Allocating an address above the 4 GiB mark and giving it to a protected mode core would result in data corruption and/or page faults. You'd also need two sets of page tables - one for the protected mode core, another for the standard cores.

I'm not sure of whether this had been tried before, but it seems like a fairly obvious solution to the lack of v86 mode support in long mode. If any of the more knowledgeable people here could see any problems with it, I'd appreciate any feedback.

That sounds like one of the worst ideas i've heard in a long time. So, one of my cores is going to sit around doing mostly nothing until I want to change video resolutions? Why not just drop to real mode when I want to make the change? Or, use an x86 emulator to run the 16-bit code from long mode? I just don't see the point in leaving an idle cpu laying around just to handle mode changes. What happens when (if?) you get proper video drivers implemented? Then what will the lone real mode cpu do?

You'll note that I never said it was perfect, but that it was something I've been thinking about. There'd obviously be a massive performance penalty, and one of the CPU cores would be unusable. I simply threw an idea out there, so that if people were considering something similar, they'd be able to see the benefits and disadvantages. If you get proper video drivers implemented along with ACPI, there's very little need for v86 mode. The core could simply operate as any other.

I wouldn't drop to real mode either. You'd need two stubs in lower memory - one to run the BIOS code, another to boot an AP. You also lose the benefits of paging and such. x86 emulators could work, and that's a possibility which I hadn't thought about when typing my post.

In all fairness, the idea seems inefficient now, because the schedulers could execute real mode code using x86emu, or (should anyone decide to actually pursue the idea) move between long mode and protected mode. Personally, I dislike the second idea - it would overcomplicate the scheduler in my opinion. x86 emulators appear to be the best option for long mode calls to real mode functions, provided the emulator can support all the necessary opcodes.

Yes emulators can, and have been proven to work. Best part is, you can put the emulator as part of your Driver instead of in your kernel, so when you replace the driver with a driver for a specific video card, there is no remenants of real mode code in your kernel for any purpose. Anyways, it's probably the least instrusive way to do it, although getting an emulator working properly under your kernel may take a little work. (I have an almost complete emulator I wrote myself so i could keep the size down, and know that it'll work in my kernel since i wrote it with my kernel's system calls in mind). Anywayws, that's for another discussion, not about multicore's. Hopefully the OP got what he was looking for.

computafreak wrote:You'll note that I never said it was perfect, but that it was something I've been thinking about. There'd obviously be a massive performance penalty, and one of the CPU cores would be unusable. I simply threw an idea out there, so that if people were considering something similar, they'd be able to see the benefits and disadvantages.

I've been thinking the same thing. My code already treats the BSP as the BIOS server (so anything that needs to use the BIOS migrates there before running) in addition to running regular 32-bit code on it, and in my application there's never any use for more than four cores, and really most users need only one. So it already makes sense in this one case. Meanwhile in real life, as the number of cores starts getting ridiculous over the next few years, the idea of going to long mode and wasting one core on V86 doesn't seem so bad. Serves AMD right for disabling V86 in long mode!!! WHY...

John Wilson
D Bit

JohnWilson wrote:

computafreak wrote:You'll note that I never said it was perfect, but that it was something I've been thinking about. There'd obviously be a massive performance penalty, and one of the CPU cores would be unusable. I simply threw an idea out there, so that if people were considering something similar, they'd be able to see the benefits and disadvantages.

I've been thinking the same thing. My code already treats the BSP as the BIOS server (so anything that needs to use the BIOS migrates there before running) in addition to running regular 32-bit code on it, and in my application there's never any use for more than four cores, and really most users need only one. So it already makes sense in this one case. Meanwhile in real life, as the number of cores starts getting ridiculous over the next few years, the idea of going to long mode and wasting one core on V86 doesn't seem so bad. Serves AMD right for disabling V86 in long mode!!! WHY...

John Wilson
D Bit

Why? Really? I wish they got rid of 32-bit long mode and real mode along with it! Seriously, you miss V86 mode? I wish they would get rid of that crap already so people would stop having to use all these stupid hacks to get things working. I mean, do we really need support for something that was created 30 years ago? Seriously, why is it that people are so stuck in the past. Move on already. An x86 emulator today can run faster than the real mode PC's that real mode was designed for, so why waste the time supporting it in hardware, and keeping all these 'features' that are useless. If you don't want to update rom bios', detect if it's a real mode rom, and use an emulator. Otherwise, use a 32-bit or 64bit rom bios, same with bios. Why can't it be 32-bit, or 64-bit? Ok, it'd take a little more space, but it wouldn't be a lot, and it'd be a lot easier in general.

Hi,

Ready4Dis wrote:

JohnWilson wrote:I've been thinking the same thing. My code already treats the BSP as the BIOS server (so anything that needs to use the BIOS migrates there before running) in addition to running regular 32-bit code on it, and in my application there's never any use for more than four cores, and really most users need only one. So it already makes sense in this one case. Meanwhile in real life, as the number of cores starts getting ridiculous over the next few years, the idea of going to long mode and wasting one core on V86 doesn't seem so bad. Serves AMD right for disabling V86 in long mode!!! WHY...

Why? Really?

The other thing worth considering is UEFI.

Basically you want an OS that doesn't care what the firmware was; and many boot loaders (some for PC BIOS, one for EFI, etc) that work as an abstraction layer (to hide the firmware from the kernel and the rest of the OS).

If you design it right, there's no need for virtual8086 mode, no need for a real mode emulator and no need to waste a CPU.


Cheers,

Brendan

Thank you for the replies, I'll start working on APCI tables