x86 SMP with cpus on different processor modes

[tiago4orion]

It's possible to setup SMP on intel x86 processors and leave the BSP in real mode?

I have the following setup in mind:

- BSP in 16-bit real mode;
- 1 AP in 32-bit protected mode;
- Every other cpu off;
Why?
The project is a simple assembler IDE that runs in bare metal, and could be used to fast prototype of ideas talking directly to real hardware. The code is assembled in-memory, and then to "test" is just a jump to the assembled location. Will be used most to learn/teach OS design.

The use case is very simple (only need disk and VGA), but I need 4GB address space. Being able to use BIOS services could make it a trivial project. My initial plan was to use Unreal mode for it, but it turns out to be error prone.

My idea is use the AP processor to develop the simple OS (it's just editor TUI + assembler), and invoke the BSP processor to access disk. I'm aware that will require a mutual exclusion lock, but I never programmed a SMP and every reference online uses protected mode BSP.

It makes sense? What kind of problems I have in this design?

Thanks!

Project details: https://github.com/tiago4orion/EnzOS
Plan: https://github.com/tiago4orion/EnzOS/bl ... er/plan.md

[Brendan]

Hi,

It's possible to setup SMP on intel x86 processors and leave the BSP in real mode?

Yes (mostly). The BSP will need to switch to protected mode (or unreal mode) temporarily whenever it wants to start an AP CPU (or send an IPI to another CPU for any other reason).

My idea is use the AP processor to develop the simple OS (it's just editor TUI + assembler), and invoke the BSP processor to access disk. I'm aware that will require a mutual exclusion lock, but I never programmed a SMP and every reference online uses protected mode BSP.

You won't need a mutex if the BSP is the only CPU that can use BIOS functions and receive IRQs. What you will need is some sort of queue or FIFO buffer, so that the AP CPUs can put work on the queue for the BSP to find and do. For example, AP CPU might put "change video mode to something" on the queue (while the BSP is handling an IRQ or loading stuff from disk or something), and when the BSP finishes what it's doing it'd check the queue, find the "change video mode" and do that.

What kind of problems I have in this design?

The main problem you'll have is that if/when you get about 20% of the code done, you're going to decide to write something more "OS like" (e.g. with many CPUs all in protected mode, a scheduler and proper device drivers). For a simple example, while using the IDE (to edit source code) you'll want to run "make -j" (in the background) to test for compile-time errors in the code you're using the IDE to write, and then you're going to want to execute the compiled executable.


Cheers,

Brendan

[tiago4orion]

Thanks Brendan, very enlightening

The main problem you'll have is that if/when you get about 20% of the code done, you're going to decide to write something more "OS like" (e.g. with many CPUs all in protected mode, a scheduler and proper device drivers). For a simple example, while using the IDE (to edit source code) you'll want to run "make -j" (in the background) to test for compile-time errors in the code you're using the IDE to write, and then you're going to want to execute the compiled executable.

I understand that some useful features will cry for pmode, but only in the future and if my project reach this point I'm already very happy
I want to make something that could be useful very soon. A DOS-style (or Borland turbo C like) editor to boot from a usbdisk and give to user the opportunity to interact with real hardware very fast. The usbdisk could have a FAT partition with the OSDEV examples (tutorial to use BIOS services, examples to switch to protected mode, find the mp table, setup ACPI, and so on...). To run/modify basic examples like that, there's no need for sophisticated drivers.

Thanks again for the clarification!

[heat]

Thanks Brendan, very enlightening

The main problem you'll have is that if/when you get about 20% of the code done, you're going to decide to write something more "OS like" (e.g. with many CPUs all in protected mode, a scheduler and proper device drivers). For a simple example, while using the IDE (to edit source code) you'll want to run "make -j" (in the background) to test for compile-time errors in the code you're using the IDE to write, and then you're going to want to execute the compiled executable.

I understand that some useful features will cry for pmode, but only in the future and if my project reach this point I'm already very happy
I want to make something that could be useful very soon. A DOS-style (or Borland turbo C like) editor to boot from a usbdisk and give to user the opportunity to interact with real hardware very fast. The usbdisk could have a FAT partition with the OSDEV examples (tutorial to use BIOS services, examples to switch to protected mode, find the mp table, setup ACPI, and so on...). To run/modify basic examples like that, there's no need for sophisticated drivers.

Thanks again for the clarification!

ATA drivers aren't really sophisticated drivers, they are actually really simple. All you need to do is some simple out's and in's, and you're done. If you want DMA, it's just some more out's and in's. It's actually pretty simple.

[tiago4orion]

Thanks heat, I'll make some experiments before take a decision.

[Brendan]

Hi,

I want to make something that could be useful very soon. A DOS-style (or Borland turbo C like) editor to boot from a usbdisk and give to user the opportunity to interact with real hardware very fast.

The user won't have the ability to interact with real hardware - the real hardware would still be "owned" by the BIOS, so any hardware the user touches will break the BIOS, which will break the BSP's code, and break the IDE.

For example, one of the first things useful code would want to do is reconfigure the PIC chips (and/or mask IRQs in the PIC chips, and/or setup the IO APIC and MSI); which will break the BIOS. For a USB controller driver (e.g. for USB flash) you need to disable legacy emulation, which will break the BIOS. If you need any kind of timing (and most devices do need various time delays and/or time-outs) you'll need to take control of something (PIT, HPET, RTC or local APIC timer) and most of those options can/will break the BIOS.

Mostly; if you implement it you'll find that you can't use it for much because your hands are tied by the need to keep BIOS and BSP working; and (even for "barely usable" - e.g. no syntax highlighting, icons, etc in the IDE; with a poor file system, poor/slow disk IO, no networking, no way to view documentation in HTML or PDF form, etc) the time you spend implementing the IDE and everything it needs will be far greater than the time you spend using it.

The usbdisk could have a FAT partition with the OSDEV examples (tutorial to use BIOS services, examples to switch to protected mode, find the mp table, setup ACPI, and so on...). To run/modify basic examples like that, there's no need for sophisticated drivers.

To run basic examples, there's no need to have a bootable IDE and it's likely to be far better to boot them in something like Bochs or Qemu where you have a full/usable debugger. You won't be able to do anything with information from the (deprecated) MP spec table (it's only good for starting CPUs and finding out which IO APIC inputs PCI devices are connected to; and touching any of that will break your BSP and/or break the BIOS). For ACPI, almost everything will break the BIOS and can't be touched.

Also note that BIOS (and UEFI) are extremely bad for performance because they're designed for "single tasking, synchronous". What this means is that you can only do one BIOS (or UEFI) function at a time; and can't read from 3 different storage devices while receiving keyboard/mouse events while doing other things. Even if you were writing a real mode single-tasking OS, you'd still want to write your own drivers (instead of using BIOS functions) simply because the BIOS functions suck (e.g. prevent you from doing effective disk caching and/or pre-fetching, prevent you from using multiple different devices at the same time, etc).

Basically; for BIOS you get minor temporary/short-term convenience, but the price that you have to pay for this (massive limitations, poor performance and long-term inconvenience) is extremely high.


Cheers,

Brendan

[Schol-R-LEA]

@tiago4orion: would you mind giving some examples of the sorts of projects or examples you intend to use this for teaching, and how the CPUs being used for the examples will be arranged? As Brendan has already said, the plan as given is problematic because most of the subject you would want to cover in an OS course - scheduling, mutual exclusion primitives, memory allocation, low-level device programming for things like the PIC or the FDC and HDC - would either be infeasible to demonstrate because they would require you to override or replace the BIOS, or would be more easily demonstrated in some other manner in a sandbox environment running under a conventional parent OS.

In my experience, most OS courses (for a variety of reasons) don't actually discuss OS development at all, but rather OS design principles; the ones which do cover 'practical' OS programming usually use an existing toy OS such as Minix or NACHOS, and simply describe how they were written and the possible alternative design approaches. Why? Because the act of writing an OS, even as a whole-class group project, would exceed the time available for a single term, especially if the focus is on low-level topics - the design concepts behind them would get obscured, and the really important topics (which are most about creating appropriate resource-management abstractions) wouldn't even get touched.

[rdos]

Yes, of course, different cores can run in different modes. Actually, when an AP core is booted, it will startup in real mode, and then the OS will typically switch mode to protected mode or long mode.

Besides, when I run long mode applications in my OS, I switch processor mode to long mode while the scheduler runs the long mode application, and then switch back to protected mode when a 32-bit application is run. That works just fine.

[tiago4orion]

Hi,

The user won't have the ability to interact with real hardware - the real hardware would still be "owned" by the BIOS, so any hardware the user touches will break the BIOS, which will break the BSP's code, and break the IDE.

For example, one of the first things useful code would want to do is reconfigure the PIC chips (and/or mask IRQs in the PIC chips, and/or setup the IO APIC and MSI); which will break the BIOS. For a USB controller driver (e.g. for USB flash) you need to disable legacy emulation, which will break the BIOS. If you need any kind of timing (and most devices do need various time delays and/or time-outs) you'll need to take control of something (PIT, HPET, RTC or local APIC timer) and most of those options can/will break the BIOS.

I'm aware that depending on what the user make, it'll require a reboot of the machine to go back to the IDE. But in several cases there are alternatives to take control back after user code (if user follows some conventions), eg. trampoline code assembled after user code, explicit jump by user code to IDE recover memory address, invalid instruction, and so on;
But if user reconfigure the PIC, as you said, or make some other thing that break the BIOS, he'll need to reboot manually (if OS code get the chance to execute, it can detect if a reboot is needed and do so).

Mostly; if you implement it you'll find that you can't use it for much because your hands are tied by the need to keep BIOS and BSP working; and (even for "barely usable" - e.g. no syntax highlighting, icons, etc in the IDE; with a poor file system, poor/slow disk IO, no networking, no way to view documentation in HTML or PDF form, etc) the time you spend implementing the IDE and everything it needs will be far greater than the time you spend using it.

The idea of using BIOS was only to get some useful things very earlier (basic editor + assembler), but I agree is problematic.

Another idea is BSP rewrite the interrupt vector to common BIOS routines (int 10h, 13h, etc) be handled by my OS (with proper drivers in protected mode) instead of BIOS. I never tested it, and maybe the idea is broken, but the BSP ISR invocation could be handled by another pmode cpu with IPI message passing. But if it is doable, that way I can improve the user experience without majority of BIOS problems. Another advantage of this is lots of preemption points in the IDE (every time user issue a ISR). Brendan, what do you think?

Basically; for BIOS you get minor temporary/short-term convenience, but the price that you have to pay for this (massive limitations, poor performance and long-term inconvenience) is extremely high.

Yeah, I agree the cost of this limitations are high for an OS, but I'm in doubt for a simple IDE. Probably I'll get rid of the BIOS, but I'm still doing some experiments.

Thanks!

[tiago4orion]

@tiago4orion: would you mind giving some examples of the sorts of projects or examples you intend to use this for teaching, and how the CPUs being used for the examples will be arranged? As Brendan has already said, the plan as given is problematic because most of the subject you would want to cover in an OS course - scheduling, mutual exclusion primitives, memory allocation, low-level device programming for things like the PIC or the FDC and HDC - would either be infeasible to demonstrate because they would require you to override or replace the BIOS, or would be more easily demonstrated in some other manner in a sandbox environment running under a conventional parent OS.

I learned the basics of assembly using DEBUG (DOS program shipped with win95,win98, etc). Did you used it? It was amazing, and I miss it a lot today. For me, one of the best ways to teach because what you don't practice, you forget....

Take a look:
https://www.youtube.com/watch?v=s1uOGjt0YJk
http://thestarman.pcministry.com/asm/debug/debug2.htm

My initial idea was to create something like it to linux, but it turns out to require a complete VM.
If I finish this development environment some day, it'll have something like DEBUG built in.

In my experience, most OS courses (for a variety of reasons) don't actually discuss OS development at all, but rather OS design principles; the ones which do cover 'practical' OS programming usually use an existing toy OS such as Minix or NACHOS, and simply describe who they were written and the possible alternative design approaches. Why? Because the act of writing an OS, even as a whole-class group project, would exceed the time available for a single term, especially if the focus is on low-level topics - the design concepts behind them would get obscured, and the really important topics - which are most about creating appropriate resource-managment abstractions - wouldn't even get touched.

I see your point, and agree that for an academic course, this is the way to go. But I want something more realistic, where the student could take their own OS in a pendrive and improve it every day.

Cheers