"Smart" Boot Loader VS "Complicated" Kernel

[BASICFreak]

Hello again (thought you got rid of me didn't you )

Let me start by explaining (one of) the goals for my OS:
Since the start, I have wanted to develop a "cloud computing" system - many systems linked via Eth and/or RS232. But to have full support for each computer (CPU) I need both protected mode and long mode functionality (at once)

My test setup will be the following systems:
AMD K6 - 64MB
Intel P4 HT (no lm) - 4GB
Intel Core2 Quad - 8GB
AMD Phenom II x6 - 6GB
Intel Atom (ht, no lm) - 2GB

Which will give me 3x 32-bit processors, 2x HT (may avoid these cores all together), and 10x 64-bit processors - and a total of 20.06GB System Memory

So assuming I know how to communicate between the systems... (which as of yet I do not, but soon)

I have thought of two ways to make this work:
1) Have to osloader use CPUID to detect the presence of LM and insert the folder "x86_64/" or "x86/" to the file path respectfully and design two kernels (one for PM and one for LM)
2) Design my kernel to detect CPU functions and have both schedulers, IDTs, GDTs, etc in the same file

Now the issue with both ideas:
1) I will be forced to write two kernels one for x86 and one for x86_64
2) Unless I am mistaken, I will have to use Binary format to mix 32 and 64-bit code instead of the ELF format I am currently using (not to big of a deal) - but I may have to drop C all together as I'm unsure how LD and GCC will react with mixed code - though again no big deal, I'm moving all my C piece by piece to ASM already (and my skill in ASM is improving quickly).

So while method one seems the easiest, method two seems more efficient and interesting (IMO)


What are your thoughts? I'm torn between the two option listed, but have not considered any other (as I cannot see them)



Post Thought:
The kernel only handles Scheduling, Physical Memory Management, Paging, and IPC - plus the CPU (IDT, GDT, TSS, Etc.) all other hardware is controlled in Ring3 by "applications / drivers"

[Brendan]

Hi,

I have thought of two ways to make this work:
1) Have to osloader use CPUID to detect the presence of LM and insert the folder "x86_64/" or "x86/" to the file path respectfully and design two kernels (one for PM and one for LM)
2) Design my kernel to detect CPU functions and have both schedulers, IDTs, GDTs, etc in the same file

I use the first option. More specifically, my boot code does as much as possible (and is then discarded after), which means there's very little unnecessary initialisation code in the micro-kernel after boot. Typically the micro-kernel ends up between 64 KiB and 128 KiB; and the "sum of all pieces of boot code" ends up being several times larger than any micro-kernel.

Also note that there's 4 different case I worry about (some with sub-cases):

• Booting from BIOS (floppy, hard disk, CD-ROM, PXE/network)
• Booting from UEFI
• Booting directly from ROM (where the firmware is a custom boot loader)
• Booting from a previous instance of the OS (e.g. kexec, for fast kernel updates)

For this reason (and others) my boot code is split into multiple pieces; where the first piece ("boot loader") is different for each of the different cases (and sub-cases) mentioned above; but it starts a "boot abstraction layer" that is the same in all cases. It's the "boot abstraction layer" that decides which of the kernels to start; and there can be many kernels (e.g. plain 32-bit, 32-bit with PAE, or 64-bit; each for single CPU, SMP or NUMA; which can works out to 9 combinations).

1) I will be forced to write two kernels one for x86 and one for x86_64

For at least half of the code in a micro-kernel; you have to write different code anyway because of differences in how paging works, or differences in how APICs work, or differences in what gets saved/loaded during task switching, or differences in kernel API, or differences in interrupt handlers, or....

You can use conditional code; but that will mean that either your kernels are less efficient (because you're not taking advantages of features that exist in some cases but not in others) or your code is a pain to write/maintain because it's obfusticated with "#if USING_X86_64" mess. For a monolithic kernel it's completely different because there's far more generic code.

Of course I'm not saying you shouldn't use conditional code for some things. For example; you might have code for a 32-bit kernel that uses conditional code for "single-CPU vs. SMP vs. NUMA", and then have completely different code for a 64-bit kernel (that also uses conditional code for "single-CPU vs. SMP vs. NUMA"). All I'm really saying is "choose wisely" - some things cause significant differences (e.g. 32-bit vs. 64-bit) where conditional code can be more trouble than its worth, and some things don't cause much differences and conditional code is worthwhile.


Cheers,

Brendan

[jnc100]

1) Have to osloader use CPUID to detect the presence of LM and insert the folder "x86_64/" or "x86/" to the file path respectfully and design two kernels (one for PM and one for LM)

This is the classic way of doing things. You may even want different loaders for each architecture (as highlighted by Brendan), then each loader can be hardcoded to load a different kernel. You can easy share the same source code for platform-agnostic bits like the scheduler and IPC between the different kernels with some not-too-difficult Makefile (or alternative) directives.

Note that if you want to support UEFI in the future then it automatically executes a different kernel for you based upon the current architecture, and you are thus forced to create different kernels (or second stage loaders) anyway. If doing that, it doesn't then make sense to have the kernel make the same choice depending on architecture again. You can, obviously, then do different things within the kernel depending on different architecture 'sub-features' e.g. PAE, SSE support etc.

2) Unless I am mistaken, I will have to use Binary format to mix 32 and 64-bit code instead of the ELF format I am currently using (not to big of a deal) - but I may have to drop C all together as I'm unsure how LD and GCC will react with mixed code - though again no big deal, I'm moving all my C piece by piece to ASM already (and my skill in ASM is improving quickly).

This has been done before for user mode programs (I'm not sure about kernels though), most notably by Apple (see Fat binaries). There was also an effort to extend it to the ELF format, and some of the tools are already available (see here).

Regards,
John.

[BASICFreak]

@Brendan
@Jnc100

Thanks for the input, I think that I will do some in both

And if I ever get a computer with (U)EFI I may implement it, but till then nope. (went threw the discussion a few months back about updating BIOS - but my main test PC has a non re-programmable ROM and I have no blank ROMs nor the ROM writer)

B!

[tsdnz]

Hi, I am also developing a cloud operating system.

Let me start by explaining (one of) the goals for my OS:
Since the start, I have wanted to develop a "cloud computing" system - many systems linked via Eth and/or RS232. But to have full support for each computer (CPU) I need both protected mode and long mode functionality (at once)

My system only runs x64, long mode with 10 GB NIC's

I have written a couple of operating system and almost finished another, there will be a couple more operating systems needed before it can be put together.

I have just finished 5 x 18 hour days implementing my exception handler, it was great fun.
The exception handler is used in user space like this.

Code: Select all

while (true)
{
			Exception_Try(CatchMe)
			{
				Iterations++;
				
				asm volatile ("mov $0, %rax; idivq %rax;");
			}
			Exception_TryCompleted(CatchMe);
			Exception_Catch(CatchMe)
			{
				*t = Iterations;
			}
			Exception_End(CatchMe);
}

It returns any exception the kernel allows back to the user. The current exceptions are: Div0, Page-Fault, Float Point.
I was just thinking if I should allow the user to catch invalid-Opcode, Bound and decided no. But only a few line of code to add if I decide.
On a 1.9 GHz, machine it processes (*t is volatile in above code, Iterations is a register), around 5.5 million exceptions per second with the code above.
The exception handles kernel exceptions, user spaces exceptions, user space double exceptions (kill program), SYSCALL exceptions.

The OS grows automatically, if the user has paid, it works on a auction idea I have.
Code can move automatically from one core to another core or across computers to another core.
It all depends on the load, and a few other properties.

I have a couple of possible clients who will work with me once they see it going.
One of the requirements for one project is 200,000,000 active TCP/IP connections.
I am currently handling around 10,000,000 per second on a test OS I had running.
It can easily handle 200 million TCP/IP connections with out slowing down.
There will be plenty of these to allow for huge connections.
Lots more work to do, time I had an early night and dream of what is possible.

I asked a few weeks back if anyone was interested in helping with the OS I am working on.
I will get around to answering questions shortly.

http://forum.osdev.org/viewtopic.php?f=11&t=29706

[SpyderTL]

And if I ever get a computer with (U)EFI I may implement it, but till then nope. (went threw the discussion a few months back about updating BIOS - but my main test PC has a non re-programmable ROM and I have no blank ROMs nor the ROM writer)

VirtualBox supports UEFI booting, if you just need something to test on. (The console is a little slow, though.)

[BASICFreak]

Ok, I decided to share my plan (or what I have so far of it).

@Brendan - It's actually similar to what you described over the past week in other threads. (I know depends on my definition of similar )

Now, I hope I don't lose you - and this is clear. (I'm still practicing being able to describe whats in my head... )

NOTE: (E)UFI is not supported under these specs, yet.

So, we all know how a HDD boots - BIOS loads MBR, MBR loads VBR, and VBR loads something.

The file my (FAT) VBR loads is called "FATLOGIC" which is just a 16-bit real mode FAT HDD driver, and a wrapper to load the file "boot/osloader.bin". The beginning of the file has 3 bytes for a jump, a byte for the boot drive number, and 4 SEG:OFF pointers (2 are actually used ATM) then the wrapper and driver information.

These pointers are designated as fopen, fread, fwrite, and fclose - under every boot option, these pointers remain in the same place. And these are designed to be replaced by there 32/64 bit counterparts, in due time.

(Past here has not been fully tested)
Now, OSLoader has control over the system and its job is fairly simple; well, not really.
The OSLoader must first detect the CPU Vendor and decide if the CPU is supported by the OS (currently any reply other than "GenuineIntel" or "AuthenticAMD" is considered unsupported) and load the appropriate driver to it's proper location (for use later, after we are done in 16-bit real mode)

NOTE: The only reason I detect the vendor and load the file first is so I don't wast time on an unsupported system doing anything else.

Currently the only 16-bit modules that I know I will have are SVGA, and MEMORY (the Mem Map from INT 15h)

Once the 16-bit modules here are done, we then jump to the CPU initialization function located in the CPU Driver. Which will run through the CPU ID functions and detect the hardware and load the supported drivers/modules to initialize everything (A20, GDT, IDT, TLB, PM, LM, PSE, PAE, SSE, APIC,... you get it)

The very next thing before we actually initialize the CPU is to detect the Boot Medium and load a 32 or 64 bit driver into memory so we do not have to worry about 16-bit real mode interfaces once we have paging and or long mode activated. This also includes loading the file system driver for the next CPU mode.

NOTE: I'm actually unsure what I will have it do if it cannot find the driver for the Boot Medium yet...

So, we are finally ready to initialize the CPU's features - easy enough as the drivers are loaded in...

Then finally we get to the fun part, setting up a Page Frame Allocator and load the 3 files that make up the Kernel, which are literally 3 INT handlers (Memory, IPC, and Scheduling)

---------------------------------------------------

Anyways, I probably missed a-lot as this only actually scratches the surface of what I have in store.

My goal is to make the OSLoader and Kernel transparent, as in one of the same. All device driver loaded in OSLoader are all user-land processes, the Page Frame Allocator created during the OSLoader is the one the Kernel uses when allocating memory for processes / self, etc.

Now, the testing I have done over the past week shows this is possible; but, it also shows that it will take a fairly long time to complete and be extremely (again personal definitions) complicated. But if I were in a hurry or wanted something simple I wouldn't be OSDeving anyways (or at-least have stuck with my first design, which had inputs and outputs unlike the following ones where I just tested concepts)...


So; any comments, concerns, criticism, recommendations, or questions? ("need input, MORE input", Johnny Five)




Best regards,


B!

[tsdnz]

Hi,

Since the start, I have wanted to develop a "cloud computing" system - many systems linked via Eth and/or RS232. But to have full support for each computer (CPU) I need both protected mode and long mode functionality (at once)

Not sure why you want RS232?

What do you want your cloud system to do?
Is it designed for a specific task?
Does it need to support Linux or Windows?
Does it need virtualisation?

[embryo2]

Ok, I decided to share my plan (or what I have so far of it).

If the plan's goal is education then the plan is good. And even if the goal is a bit more complex then the plan is still good enough just because it's implementation will teach you a lot about the actual goal. In the end you'll just look at the result and understand - here is the place where it is possible to make things better. But until the implementation is close to completeness you have no way to look at and see the something, which you haven't defined clearly.

It is called iterative approach. You should make iterations and every iteration shows you another bit of the yet unknown picture.

But there's also a bit more efficient approach. And it has a problem. First - there should be a goal. It should be defined clearly. Next there should be a criteria. It defines the test for the goal's achievement progress. If the test is passed then the goal is achieved. Finally there is an optimization. You can assess a number of plans that make the situation when test is passed. And you can select the best performing plan. But again, you need to define what is the performance. Well, it's not a simple approach.

Anyway, the one thing you should understand is the fork on the way of every developer. Would the development be systemic or it wouldn't. I vote for the former, but it's much easier to work with the latter. And the choice is your.

Now about the plan (how to make it a bit more systemic).

The file my (FAT) VBR loads is called "FATLOGIC" which is just a 16-bit real mode FAT HDD driver, and a wrapper to load the file "boot/osloader.bin". The beginning of the file has 3 bytes for a jump, a byte for the boot drive number, and 4 SEG:OFF pointers (2 are actually used ATM) then the wrapper and driver information.

Here you try to separate some things from others. You see there's just one need now - to load more bytes. And you load them with the help of the separate entity (fatlogic). But is it optimal? May be. It depends on the criteria (which is still undefined). At least I see here "the driver", which actually performs the very simple task of loading some sectors from hard coded type of file system. Is there a need for such separate driver without really important functionality? First, it is the 16-bit code. Will it be used anytime later? Second, a good driver usually supports one kind of hardware, but supports it in a very efficient manner. Your driver seems to be very generic and it means it won't be efficient. Can we call a generic and inefficient thing "a driver"? If we wish, we can, but... May be it's a good idea to call it "just another stage of the bootstrapping process".

These pointers are designated as fopen, fread, fwrite, and fclose - under every boot option, these pointers remain in the same place. And these are designed to be replaced by there 32/64 bit counterparts, in due time.

Here it's probably the portability issue that was addressed with such solution. But it's interesting - do you plan to redirect calls like "fopen" to the "driver" discussed above? Most probably your code with the "fopen" calls will be compiled for 32 or even 64 bit mode, but the driver is for 16 bit mode. May be it worth to think about it.

The OSLoader must first detect the CPU Vendor and decide if the CPU is supported by the OS (currently any reply other than "GenuineIntel" or "AuthenticAMD" is considered unsupported) and load the appropriate driver to it's proper location (for use later, after we are done in 16-bit real mode)

Why not to support other vendors? Wouldn't it be better to check for the required processor features instead of the vendor?

Currently the only 16-bit modules that I know I will have are SVGA, and MEMORY (the Mem Map from INT 15h)

Here I see a bit different scale of things. Just a tiny bit, in fact. The one call of the INT 15h and the whole blob of the mathematics intermixed with a lot of low level hardware features for the SVGA to work properly. May be it's good idea to separate some things in the SVGA blob?

Once the 16-bit modules here are done, we then jump to the CPU initialization function located in the CPU Driver. Which will run through the CPU ID functions and detect the hardware and load the supported drivers/modules to initialize everything (A20, GDT, IDT, TLB, PM, LM, PSE, PAE, SSE, APIC,... you get it)

I see the "everything" as just a set of small functions like writeGDT or makePMSwitch. The separation of the functionality is a good idea, but if the number of separate functions is too big the mess will be insurmountable. May be it's a good idea to combine some functions in more manageable modules? Modules can be designed according the system goals. The goals can be - switch to protected mode, switch to long mode, installation of interrupt handlers (for the particular mode), initializing of paging, working with the interrupt controller and so on.

The very next thing before we actually initialize the CPU is to detect the Boot Medium and load a 32 or 64 bit driver into memory so we do not have to worry about 16-bit real mode interfaces once we have paging and or long mode activated. This also includes loading the file system driver for the next CPU mode.

Here you have many things in one place. I see the need for another separation of concerns here. The concerns are - CPU initialization, boot media detection, boot media read, 32 and 64 bit drivers (how many items here do you plan to create?), the mode switch (at least long mode), paging initialization, file system driver for the "next" CPU mode (is it a hardware thing or actually is the way to read blocks from disk formatted with FAT32 or NTFS or whatever?). So, the number of concerns looks a bit overextended for the such short phrase. But of course, you can do something about it!

NOTE: I'm actually unsure what I will have it do if it cannot find the driver for the Boot Medium yet...

Yes, there are also the process transition related concerns. But do you know the plan of the process? What should follow what? And why? I hope you know it in very deep details.

So, we are finally ready to initialize the CPU's features - easy enough as the drivers are loaded in...

May be it's really a bit easier if we already have the file system driver. Or memory map. Or SVGA support. May be.

Then finally we get to the fun part, setting up a Page Frame Allocator and load the 3 files that make up the Kernel, which are literally 3 INT handlers (Memory, IPC, and Scheduling)

And finally after we have the CPU initialized the only problem remains here is to make these 3 kernel files to do all the stuff a good kernel can do. It's really simple!

There's a bit of sarcasm in my post, so I should ask you not to be too angry. And after the emotions will go away may be it will be the good moment to think about some architecture. But it shouldn't be the OS architecture. I recommend you to start from something simple like bootloader architecture. If you succeed with the simple thing it's greatly increases the chances with bigger things like the YourOS.

[FallenAvatar]

The file my (FAT) VBR loads is called "FATLOGIC" which is just a 16-bit real mode FAT HDD driver, and a wrapper to load the file "boot/osloader.bin". The beginning of the file has 3 bytes for a jump, a byte for the boot drive number, and 4 SEG:OFF pointers (2 are actually used ATM) then the wrapper and driver information.

These pointers are designated as fopen, fread, fwrite, and fclose - under every boot option, these pointers remain in the same place. And these are designed to be replaced by there 32/64 bit counterparts, in due time.

You forgot about the BPB.

- Monk

[BASICFreak]

You forgot about the BPB.

You shouldn't make assumptions... The BPB is there, it's loaded at 0x7C03 (when the VBR is loaded) - and is accessed as an offset of BP (which is set to 7C00).
The BPB is a data structure not executable which is why I purposefully left it out of my post, but yes it does exist and is loaded and used.

The file my (FAT) VBR loads is called "FATLOGIC" which is just a 16-bit real mode FAT HDD driver, and a wrapper to load the file "boot/osloader.bin". The beginning of the file has 3 bytes for a jump, a byte for the boot drive number, and 4 SEG:OFF pointers (2 are actually used ATM) then the wrapper and driver information.

Here you try to separate some things from others. You see there's just one need now - to load more bytes. And you load them with the help of the separate entity (fatlogic). But is it optimal? May be. It depends on the criteria (which is still undefined). At least I see here "the driver", which actually performs the very simple task of loading some sectors from hard coded type of file system. Is there a need for such separate driver without really important functionality? First, it is the 16-bit code. Will it be used anytime later? Second, a good driver usually supports one kind of hardware, but supports it in a very efficient manner. Your driver seems to be very generic and it means it won't be efficient. Can we call a generic and inefficient thing "a driver"? If we wish, we can, but... May be it's a good idea to call it "just another stage of the bootstrapping process".

Currently the driver is generic, the key with this file is just to access the boot medium while still in 16-bit real mode. The goal is once the CPU has changed to 32/64 bit mode to have a less generic set of drivers (Medium and File System)
The current driver is read-only, though I am adding write functionality to it - allowing me to save a configuration script based on wither MAC (PXE) or CPU serial to allow skipping all the detection process.
- My driver seems to be very generic, hum..., it may be but you still have not seen it (not published yet, I have rewrote nearly all the code I have on GIT and haven't updated it - and won't until I get further into the design)

These pointers are designated as fopen, fread, fwrite, and fclose - under every boot option, these pointers remain in the same place. And these are designed to be replaced by there 32/64 bit counterparts, in due time.

Here it's probably the portability issue that was addressed with such solution. But it's interesting - do you plan to redirect calls like "fopen" to the "driver" discussed above? Most probably your code with the "fopen" calls will be compiled for 32 or even 64 bit mode, but the driver is for 16 bit mode. May be it worth to think about it.

As stated, this driver is only meant to be used during real mode 16-bit execution. It is to be replaced by the required User-Land Drivers for (in this case) HDD and FAT just before changing the execution mode.

The OSLoader must first detect the CPU Vendor and decide if the CPU is supported by the OS (currently any reply other than "GenuineIntel" or "AuthenticAMD" is considered unsupported) and load the appropriate driver to it's proper location (for use later, after we are done in 16-bit real mode)

Why not to support other vendors? Wouldn't it be better to check for the required processor features instead of the vendor?

I've thought about this, but at the same time a-lot of the CPUID functions are vendor specific and I may support others at a later point.

The very next thing before we actually initialize the CPU is to detect the Boot Medium and load a 32 or 64 bit driver into memory so we do not have to worry about 16-bit real mode interfaces once we have paging and or long mode activated. This also includes loading the file system driver for the next CPU mode.

Here you have many things in one place. I see the need for another separation of concerns here. The concerns are - CPU initialization, boot media detection, boot media read, 32 and 64 bit drivers (how many items here do you plan to create?), the mode switch (at least long mode), paging initialization, file system driver for the "next" CPU mode (is it a hardware thing or actually is the way to read blocks from disk formatted with FAT32 or NTFS or whatever?). So, the number of concerns looks a bit overextended for the such short phrase. But of course, you can do something about it!

Well, yes I plan on creating many [many] small modules - I figure linking a Kernel together would be more fun than building multiple Kernels for multiple systems. And yes, I realize it may not be the "best" or "most efficient" way of doing it, but...

NOTE: I'm actually unsure what I will have it do if it cannot find the driver for the Boot Medium yet...

Yes, there are also the process transition related concerns. But do you know the plan of the process? What should follow what? And why? I hope you know it in very deep details.

I know a large chunk of this, the only thing I have little idea about is the Multi-CPU specification - which is the actual reason for me rewriting code to start with. Now, I have been rethinking most of the process - so I may know less than I think I do, we will find out soon.

So, we are finally ready to initialize the CPU's features - easy enough as the drivers are loaded in...

May be it's really a bit easier if we already have the file system driver. Or memory map. Or SVGA support. May be.

Then finally we get to the fun part, setting up a Page Frame Allocator and load the 3 files that make up the Kernel, which are literally 3 INT handlers (Memory, IPC, and Scheduling)

And finally after we have the CPU initialized the only problem remains here is to make these 3 kernel files to do all the stuff a good kernel can do. It's really simple!

Exactly everything is simple . If I wanted simple I would be working on web applications right now, which may actually lead to monetary gain - instead I decide to do very strange low level things that will only benefit me in knowledge.

Now, my Goal is just education. I have no time constraints, I never plan on finishing this project - I may end up being done with this project, but it will never be finished (completed).

@embryo2 - I detected your sarcasm, can you detect mine... lol - And no, I don't take offense to sarcasm, most the time I actually enjoy it.

[BASICFreak]

I'm going to attempt to make this more clear...

The Kernel will technically consist of many files, Reloc ELFs or PIEs haven't fully decided yet, the job of the OSLoader is to initialize the environment and link the Kernel (and User Processes) from independent object files.

The only part of the Kernel to user sees is the 3 files (INT Handlers) mentioned earlier, and most - not all - of the other objects are user-land code.

The only reason for a 16-bit driver for File I/O is to load all the objects into memory - and very quickly this will be replaced by the VFS User Service, the HAL User Service, the Boot Medium User Driver, and the File System Plugin. But with a standard interface, the OSLoader doesn't have to worry about what Boot Medium or the File System that is on the medium - therefore the only difference between my PXE loader and FAT loader is the actual code communicating with the hardware (other than the PXE loader is one binary and FAT is 3 binaries [MBR, VBR, LOGIC])

The goal is to create a dynamic Kernel, one that can be built up "on the fly", all part should replaceable - preferably on a live system.

A system that loads the optimal pieces while not loading any unsupported features that wast space, or worse makes the system crash.

This is actually the most thought out design I have had so far, it is nowhere near complete or bullet proof - but I do see it becoming stable with a lot of work and maybe some select words...

Honestly, I guess I want to be different than [most] other designs.

And on the note of why RS232, why not. If it is an option why not allow it to be used. Not saying it would be quick, preferred, or optimal; but ~1MB/s is better than no communication... And I have a test system with no NIC but it has RS232, and I'm 99.97% sure I can use this for at-least something - not sure what though.


So; anyways, I do hope that helped - I'm still no good at explaining things (this post alone took me 30+ minutes to write), but at-least I'm trying to improve that . And I'm still sure I missed some very important details here.



Best regards,


B!

[embryo2]

Currently the driver is generic, the key with this file is just to access the boot medium while still in 16-bit real mode. The goal is once the CPU has changed to 32/64 bit mode to have a less generic set of drivers (Medium and File System)

It means there's no need for it after it has it's job done. And the job is just to find a few disk sector numbers and to load the data from the sectors. But the job can be done in many ways. The most interesting range can be described as "reliability quest". It can work only in emulator with only one set of settings, it's the first end of the range. And it can work on every imaginable system even if the system has no BIOS support for disk read operations or something even uglier. It's the second end of the range. For education it can be acceptable to implement something close to the simplest range end. But for education it is also beneficiary if you try to implement something close to the most complex end of the range. And if you succeed in implementing the second option then it actually can make a transformation from a beginner to some kind of experienced developer.

So, you can see there is the range. And it is applicable to everything you are planning now. It is acceptable to implement everything as close to the simplest end of the range as possible, but it won't make the experienced developer in the end.

And if you to decide about what end of the range you want to be close to then I recommend you to take it from the beginning - just try to implement a reliable staged (MBR, VBR, second stage) bootloader. The end result is close to the complex variant if it can survive as many boots on different machines as you can find. After you have it done it will be the time to say - I really can do something useful. And if you skip the complex solution then you can start to develop next part of the same system, as every beginner did implemented. But in the end they almost always can't say "I have done something useful".

But there's a problem. It is almost impossible to do all the stuff as good as possible. So, you need some compromise. And for you to be able to understand where is the optimal point of the compromise it is really very helpful to get at least something done as close to the complex end of the range as possible. The very obvious candidate here is the bootloader. But of course, you can make it simple and next try to create another component of the OS with the "close to complex" approach. It's up to you to decide what is more interesting to concentrate on. But there should be at least one thing which you can point at and say - it's really useful! It means efficient, reliable, convenient and something more.

Well, yes I plan on creating many [many] small modules
...
And yes, I realize it may not be the "best" or "most efficient" way of doing it

Here you can create "something useful" in the form of a good architecture. Nice architectural design really can be useful. But as with every complex thing it's not as simple as many people want it too be.

If I wanted simple I would be working on web applications right now, which may actually lead to monetary gain - instead I decide to do very strange low level things that will only benefit me in knowledge.

Hopefully the trade of the "strange for monetary" can really make you some progress. But beware of the endless loop of enhancements. In the "monetary" field people just stop you when it's not really great enhancement. And in the "strange" there's no people to stop you. So you can learn it yourself, but much after the time you would learn it in the "monetary" field. However, some limitations of the "monetary" do not allow to create something really exciting. Just beware of it all.

But with a standard interface, the OSLoader doesn't have to worry about what Boot Medium or the File System that is on the medium - therefore the only difference between my PXE loader and FAT loader is the actual code communicating with the hardware (other than the PXE loader is one binary and FAT is 3 binaries [MBR, VBR, LOGIC])

Here it seems you try to extend the boot process in the direction of flexibility. The flexibility requires you to know a lot about file systems, kinds of storage and things like network booting. It's also the good direction to help you to create "something useful". But here every kind of boot media (HDD, FDD, CD, network, USB, etc) still requires you to decide on the reliability point you are ready to stick with in the range from the simplest and up to the most complex. Basically, it requires you to repeat many steps but for different boot sources, hopefully at the same point in the complexity range. It means the flexibility requires you to have some knowledge about what complexity really is about. And the knowledge can be obtained after you have some "good" implementation of one boot variant. Next you can extend your knowledge while traveling around the "flexibility land".

The goal is to create a dynamic Kernel, one that can be built up "on the fly", all part should replaceable - preferably on a live system.

A system that loads the optimal pieces while not loading any unsupported features that wast space, or worse makes the system crash.

This is actually the most thought out design I have had so far, it is nowhere near complete or bullet proof - but I do see it becoming stable with a lot of work and maybe some select words...

Honestly, I guess I want to be different than [most] other designs.

I see the flight of your imagination, but before people fly they usually first prove they can do "something useful". Make a good bootloader or a good HDD driver or a good file system or whatever you like. But make it really good. And then you can find some better way toward the runway of your first flight.

[Kazinsal]

And on the note of why RS232, why not. If it is an option why not allow it to be used. Not saying it would be quick, preferred, or optimal; but ~1MB/s is better than no communication...

RS-232 is a lot slower than that. Recall that 1 MB/s is 8 Mb/s, or 8,000,000 bps. This is slightly lower than 10BASE-T ethernet, which runs at 10,000,000 bps on the wire. This is different than 10,000,000 bps of data, and includes overhead in encoding (Ethernet uses 8b/10b, where every 8 bits of data is encoded in 10 bits on the wire). Thus, in optimal conditions, Ethernet will be able to transmit 8,000,000 bps of data bits.

Back to RS-232. In theory, a 16550 UART can signal at up to 1.5Mb/s. That's 1,500,000 bps. Let's assume we're not using start, stop, or parity bits (we can check to see if our frame is good with a CRC like Ethernet uses, for example, and RS-232 doesn't require balanced signalling), and we're transmitting 8 bits per character on the line. This gives us 187,500 bytes per second of data on the wire. Pretty good for a serial line, but nowhere near even 10BASE-T.

Except we now have a problem. A 16550 UART needs a clock crystal running at 16x the desired baud rate in order to function. Most circuits with a 16550 will have a 1.8432 MHz crystal, which gives us a maximum baud rate of 1,843,200 / 16 = 115,200 bps, or 14,400 bytes per second on the wire at 8 data bits, no stop, no start, no parity. Some circuits on custom cards will have 7.3728 MHz crystals. You can get 460,800 bps from one of these, or 57.6 KB/s on the wire.

Also, RS-232 signals degrade rather rapidly over distance. You will not get more than a few metres out of 115,200 bps. With a really good cable, one source says you can get nine metres. For comparison, Category 5e cable for Ethernet is rated for up to 100 metres (more practically 90) at speeds up to Gigabit Ethernet.

So, RS-232 for 115.2 kbps at a distance of 5-10 metres, or Ethernet for upwards of 800 Mbps at a distance of 90-100 metres. There's other things to factor in here (like speed negotiation), but those are relatively minor compared to the complete lack of speed and distance even "modern" RS-232 has.

[BASICFreak]

@Kazinsal - I realized that, I just missed the typing of Mb/MB, I should have stated that some communication is better than none (like I was going to write). And I've used the UART at the ~1Mb on the system - it's more reliable at ~115Kb (actually just my RS232-USB kills the Kernel on windows at a higher speed, but just crashes the driver on *nix - What I get for using a cheap VEX Robotics USB/RS232 adapter...)

Again the RS232 will be (other than debug maybe) later and probably never used.

But that is simi-off topic, that is in the user-land applications like most other drivers.



Anyways, it doesn't seem anyone is totally against this idea - the only thing I really see is worry that I do not have the "skills" needed to do this? Which is totally opinion and you are entitled to it - and if I'm fully honest I am missing some, but I do not think it is to the extent you believe.

This week I have been busy, but I have the rest of today and half of tomorrow free so I will continue my testing and see just how interesting this will become.

The largest hurdle I see is the Multi-CPU initialization, and a proper scheduler for the multi-cpu system - as this is the only "new" thing I'm adding. Everything else is just updating (optimizations and improvements) as my current OSLoader was my first ASM project and my skill is much "better" now than it was then (and still improving).




Best regards,


B!