OSDev.org

When system boots and it is in real-mode or even switched to protected-mode, but without any multi-tasking environment what is the type of execution on multi-core CPU? Could it be controlled what code each core to executed and I don't speak for interrupt handlers. I've read that it is possible to have two operating-system on Dual-core CPU, that share the same memory. So it could be possible to separate what is executed on each core.

http://en.wikipedia.org/wiki/Multi-core#Notes wrote:^ Two types of operating systems are able to utilize a dual-CPU multiprocessor: partitioned multiprocessing and symmetric multiprocessing (SMP). In a partitioned architecture, each CPU boots into separate segments of physical memory and operate independently; in an SMP OS, processors work in a shared space, executing threads within the OS independently.

So one could execute interrupt-handle and the other one - execute boot code. How could code may be controlled? And how far cores could be separated, e.g. is it possible only one of the core be in real-mode and the other one - in protected? (Probably not!)
I see that windows could set process affinity to limit the process to execute in only one core, could it be done the same thing in single-tasking real-mode, e.g. turning off one of the cores?

As a quick answer to your last question: "I see that windows could set process affinity to limit the process to execute in only one core, could it be done the same thing in single-tasking real-mode, e.g. turning off one of the cores?"

If you do nothing, ie. no code to specifically use mult cores, then it will only use one core.

Hi,

Masterkiller wrote:I've read that it is possible to have two operating-system on Dual-core CPU, that share the same memory.

That's not what the text you quoted from Wikipedia says.

What the Wikipedia is saying is that there's 2 types of operating systems. For the first type of OS ("partitioned") the same OS runs on each core but resources (memory, etc) are split between cores. For example, you might have a dual core CPU and 2 GiB of RAM, and the OS would have one copy of the kernel using 1 core and 1 GiB of RAM, and another copy of the same kernel using the other core and the other 1 GiB of RAM, and both copies of the kernel would act independently so that it looks a bit like 2 separate half sized computers (with the same OS running on both). This might be a good idea to remove scalability problems and for fault tolerance (e.g. if one kernel/core crashes, then the other kernel/core could still keep running).

For the second type of OS ("SMP") the resources are shared. This costs a little overhead because of unavoidable scalability issues, but sharing the resources can make a massive difference. For example, you shouldn't ever have one CPU completely overloaded while the other CPU is doing nothing, or have one CPU thrashing swap space while the other CPU has lots of free memory, which is what you'd get with a "partitioned" OS.


Cheers,

Brendan

captainwiggles wrote:If you do nothing, ie. no code to specifically use mult cores, then it will only use one core.

Sort of. If you don't set affinity, the process could move between cores as it is blocked, woken up, rescheduled, etc.

Brendan wrote:For example, you shouldn't ever have one CPU completely overloaded while the other CPU is doing nothing, or have one CPU thrashing swap space while the other CPU has lots of free memory, which is what you'd get with a "partitioned" OS.

Just to add to what Brendan said... That sort of "partitioning" is best implemented these days with virtualization. I personally don't see the point in partitioning without isolation, as IMO that is one of the key benefits of partitioning. Virtualization gives you that kind of isolation, while the scheme Brendan (and apparently Wikipedia) described does not.

LoseThos, my operating system, does master-slave multicore processing. One core is the master and it hands micro jobs to other cores. The cores, in fact all threads and processes, all have the same memory map--identity virtual=physical. Coordination and utilization is entirely up to the application. That is, the program allocates some uncached memory or uses locks to communicate between cores. I am targeting home computers where you want to primarily run one application at a time. I do all I/O on the master core and hand-off computing tasks to other cores.

10x to all, replays answers most of my question, but I still didn't get it. How code control another code on which CPU/core to execute.

Colonel Kernel wrote:

captainwiggles wrote:If you do nothing, ie. no code to specifically use mult cores, then it will only use one core.

Sort of. If you don't set affinity, the process could move between cores as it is blocked, woken up, rescheduled, etc.

The "process" is a set of "threads" that shares the same virtual memory, and thread is equal to task. So instead using hardware task-switching, I have idea to emulate it (you know it is not much, everyone here probably written software task-switching), but in this case CPU will do the code as in single-tasking system until timer interrupt fires and and it will not return to the current process, but will check all process/threads that are not idle and will move to the next one (or if all other thread/process or idle - back from the beggining else if no process to work currently - halt the processor). This may be not so perfect, but I will try it. And I was wondering how to tell the other core to start executing "other thread/process" (or actually code from different virtual address).

Masterkiller wrote:And I was wondering how to tell the other core to start executing "other thread/process" (or actually code from different virtual address).

The kernel runs on every core, so each core will reschedule itself as needed. You can either have them take threads from a shared run queue (not recommended as this would be a concurrency bottleneck), or have per-core run queues that get re-balanced periodically as system load changes.

If you want to let a processor know that it should re-schedule right away (e.g. -- a high-priority thread unblocks but should be run on a different processor), you can send it an IPI.

Colonel Kernel wrote:You can either have them take threads from a shared run queue (not recommended as this would be a concurrency bottleneck), or have per-core run queues that get re-balanced periodically as system load changes.

A single run queue shouldn't be too bad it you keep the task switches out of phase with each other

Dear Masterkiller,

I have the feeling you don't understand correctly how a multiprocessor system works in general. (If I'm wrong, please apologize.) So here are some notes from my point of view...

Note: For this posting I assume we are only talking about phyiscal processors (no SMT-tricks like HyperThreading) and symmetric multiprocessing (all processors are the same (no asymmetry); all memory, devices, processors, ... are accessible of every processor and for now different access times are negligible (which precludes NUMA issues)). Just keep things simple for now.

Disclaimer: There are many points about which could be discussed forever. It's just meant as a start point.

Masterkiller wrote:So one could execute interrupt-handle and the other one - execute boot code. [...] And how far cores could be separated, e.g. is it possible only one of the core be in real-mode and the other one - in protected? (Probably not!)

For example, assume a Intel Core 2 Duo. There are simply two processors on one chip. That's all.

It's much like you have a mainboard with two processor sockets and a single core-processor in each socket. But those mainboards are relatively expensive. The idea of "multicore" is to put all the processors (2, 3, 4, 6, 8, ...) on one single chip. Now, you don't need multiple sockets - only one, but you have multiple processors (assuming the chipset can deal with that).

That's something the industry warmly welcomes. Think about the fat server boxes with mainboards, which have 8 sockets. Before the "multicore era" there were only single core-processors in each of the sockets, thus providing 8 processors in 8 sockets. Now take 8 Xeon/Opteron Quadcores and put them into the sockets. Now the system has 8*4=32 processors in 8 sockets. Compare that with an unaffordable (and not commercially available?) 32 socket board with 32 single core-processors. The latter is, to say it provocatively, economical suicide.

[EDIT: For an example with 4 sockets with up to 6 cores each see IBM System x3850 M2. I tried to configure my own: a basic version with 4 Xeon 7460 (6 cores) is around $21,000.]

(There are more advantages and some issues to this approach, but this is beyond the scope for this post.)

To give a direct answer: Because there are physically independent processors, the code they execute is independent too. The cores can be in different operating modes. They all can have a totally different view of the environment (e.g. MTRRs set up differently or whatever...)

The only question is: Does it make sense? The answer to each situation is every time the same: It depends. It depends on what you are trying to achieve.

• Does it make sense to have a complete different view of the system from the individual CPUs (e.g. the aforementioned MTRRs)? - Well, it depends. (Most likely there will be no really useful case. In fact, it would be totally weird.)
• Does it make sense to have the CPUs in different operating modes? - It depends. Why not? There may be situations (like starting halted cores) where this is completely natural or even a necessity.
• Does it make sense to execute different processes on different cores? - It depends. If you want it to... (Virtually) all MP capable operating systems do it.
• Does it make sense to execute only one process at any time, but the process's different threads on different cores? - It depends. If you want it to... (Virtually) all MP capable operating systems do it.
• Does it make sense to let some cores execute interrupt handlers, while others execute "normal" processes/threads? - It depends. If you want it to... (Virtually) all MP capable operating systems do it.
• ...

It's up to the designers choice what he wants to do with the available processors. You may choose to support "this" but not "that".

And yes - theoretically you can execute two different operating systems on two cores in the same system. But consider that each of them thinks it is the only operating system executing. Both of them try to use memory, devices, processors, etc. at their own will, not thinking about there's someone that does interfering things. Without some tricks they will most likely trash data of one another. Not speaking about the issues about the states in the devices. Long story short: This will crash the machine - if not "coordinated" by something (that's the pointer to virtualization and partitioning and such).

Masterkiller wrote:And I was wondering how to tell the other core to start executing "other thread/process" (or actually code from different virtual address).

Guess what - it depends! It's up to your design. Just some *possible* ways to do it (disclaimer: I don't say they are good or bad):

• You could send an interrupt from one core to the other - an so called "inter-processor interrupt" (IPI) - using the Local APIC, which every processor has. The interrupt handler for this interrupt would look some data structures, of which it is aware where they are, and would schedule/dispatch another process/thread/whatever-you-call-it.
• You could use the timer every Local APIC has. So you have a timer for every processor and can let it trigger the scheduler on each individual processor independently.
• You could use the good old PIT and broadcast its interrupt requests to all processors (using IO APIC), which could trigger the schedular on all the cores.
• You could do cooperative scheduling and every time the task yields control to the operating system, you could check which task you want to run next. In this scenario you won't need IPIs or Local APIC timers at all.
• ...

It's all about your design. For MP systems you have many more degress of freedom compared to uniprocessor systems. You have a lot more choices to make and seemingly almost unbridled adjustment screws to turn. That's the point why it's so much harder to get it right or to at the very least not make the bollix of it.

Regards,
Thilo

Hyperdrive wrote: Dear Masterkiller,

I have the feeling you don't understand correctly how a multiprocessor system works in general. (If I'm wrong, please apologize.) So here are some notes from my point of view...

First 10x for the answer!
You are right. I've never code for multi-core before. But since my CPU is now dual-core I wish to start. I'm little "old thinking". While programming I assume that processor will execute instruction sequentially. I know that CPU actually execute them unordered but the design is internal to processor and unordered execution will never mess the code (unless flaw exist).
What I don't know is what BIOS execute before gives control to my code. I know that when my code starts, CPU is in real-mode, GateA20 (it is a part of memory controller, not CPU) is disabled, ax=0xAA55, dl=BIOS Number of boot device. That is enough to start writting boot code. I assume that CPU will execute step-by-step following the flow. For the Multi-processing it's easy. Every process is strictly isolated (so it seems to be only process in the system), it does not handle interrupts. Interruption by "something" (a device or OS) are completely transparent, so from itself every process seems to be executed in single-tasking system. (Process synchronization will be my nightmare).
So I understand everything that you say theoritically, but what happens in practise. On boot seems to have only one CPU (assuming others are off). Then you send IPI to LAPIC of other CPU. What mode is that CPU, how it start? How you schedule it. If it is just some device you could easily start it, do some work and stop it. But the point I don't get it - this device is processor, so you start it with some code, give some code to it to execute, and how code controls code. I know theoritically how to create IPI, but don't know what does Interrupt handle need to contain.

Application CPUs are in a halted state, equal to the state they were before being halted.
So, if the CPUs have not been started yet, they are in a bootup (INIT) state.

say you have a quad core CPU, so 4 logical (in this case it doesn't matter) processors.

you start with one CPU executing (the BSP). it will execute the really low-level grimy stuff like setting up the base environment and initializing devices (things where it would be more of a hassle than a benefit having to lock everything). Then when the time is right, the BSP tells the AP's (other CPU's) to wake up and start executing code at a specified vector in the startup IPI's. Those AP processors start in Real-Mode (16-bits) and with a usually undetermined register state (except for the ds/es IIRC). So those AP's need to do alot of the same stuff as the BSP like setting up paging (whether that means just copying the address of the existing tables or creating a whole new set), entering P/L-Mode, setting their control registers for various support (FPU/SSE/MMX/Paging) and setting up an IDT.

*then*, once they are up to speed on things, hardware like the IO-APIC can come into play and distribute interrupts to the various processors based on priority or physical/logical/cluster specification within the IO-APIC registers themselves.

I personally (due to my OS design) use those other AP processors in true parallel (no threading/tasking code that switches between them) as I'll have two or more processors executing non-sleep code at the same time. This is done vary carefully and there is still locking where needed and probably isn't a good design choice for a larger or more user-oriented design, but I just wanted to give another point of view of what these other AP's can be used for.

Theoretically I don't see a *code* problem with creating a multi-OS ... (erm OS?), as you can specify a *very* global memory location as the locking point so that the other CPU's can still know what is critical (or in use) memory between the OS's so that CPUs don't start trashing code that is in use elsewhere.

The problem with more than one OS operating on the same (x86/x86-64) motherboard at once is that some system resources cannot effecively be shared between kernels unless those kernels are specifically written to cooperate with oen another, in which case they aren't really seperate kernels are they? Some large mainframe systems can run more than one operating system because they have dedicated hardware support for partitioning system resources like memory and I/O. The average desktop system does not have these. In theory you could have a master kernel that executes in ring0 that handles this partitioning, and write the individual ODS's in ring1, but then they arent really OS's, just superuser applications.

abachler wrote:The problem with more than one OS operating on the same (x86/x86-64) motherboard at once is that some system resources cannot effecively be shared between kernels unless those kernels are specifically written to cooperate with oen another, in which case they aren't really seperate kernels are they? Some large mainframe systems can run more than one operating system because they have dedicated hardware support for partitioning system resources like memory and I/O. The average desktop system does not have these. In theory you could have a master kernel that executes in ring0 that handles this partitioning, and write the individual ODS's in ring1, but then they arent really OS's, just superuser applications.

Or you could use the processor hypervisor features and you basically end up with some sort of virtual machine/resource multiplexer.