OSDev.org

Hey,

I'm starting to rewrite my kernel from scratch, and turn it into a micro-kernel. I figured that if i do that, i'll do it 100% 'by-the-book', and also
write my own bootloader.

And my question to you guys is, where does the job of the bootloader end? Does it only load the kernel into memory and transfer control to it?
Does it turn on protected mode, paging, and then transfer control to the kernel?
What does your bootloader do?

Thanks.

I use standard bootloader and don't waste time on it. This way I can focus on my OS instead of rewriting the damn bootloader every two weeks.

dozniak wrote:I use standard bootloader and don't waste time on it. This way I can focus on my OS instead of rewriting the damn bootloader every two weeks.

Why would you rewrite your bootloader every two weeks?

Hi,

Sourcer wrote:And my question to you guys is, where does the job of the bootloader end?

It depends a lot on the design of the OS. For example, having something like a "second stage" can change (reduce) the amount that a boot loader does (without changing "amount of things the sum of all boot code" does).

Sourcer wrote:Does it only load the kernel into memory and transfer control to it? Does it turn on protected mode, paging, and then transfer control to the kernel?

The absolute minimum that a boot loader could do is load something like a "second stage" (without loading any kernel at all - second stage can do that).

Note that for a micro-kernel you can't just load the kernel, because the kernel on its own has no device drivers and wouldn't be able to load device drivers from anywhere. You have to load some kind of "initial RAM disk" that contains things (drivers, file system code, etc) that the micro-kernel will need.

Sourcer wrote:What does your bootloader do?

Ignoring "fancy stuff"; my boot loaders:

• Load the rest of themselves into memory (if necessary)
• Setup a temporary/minimal physical memory manager (e.g. for the first ~640 KiB of memory on BIOS systems)
• Switch to protected mode and enable paging
• Load the BAL ("Boot Abstraction Layer") into virtual memory. Note: The BAL is designed to not care what the firmware is or what the boot device is.
• Use firmware to obtain "physical memory areas", convert them into the my format, and tell the BAL about each area (so that BAL can build a "physical address space map")
• Tell BAL to extended the temporary/minimal physical memory manager based on information in the "physical address space map" that the BAL constructed (so that the rest of boot can use up to 4 GiB of RAM).
• Load a "initial RAM disk" into virtual memory, and tell BAL about it (so that BAL can check it, and decompress it)
• Tell BAL a bunch of other information from the firmware (obtain ACPI tables, MP spec. tables, SMBIOS tables, get a "PCI configuration space access mechanism hint", etc
• Tell BAL about each "head" (monitor, display) that the boot loader can control, including all possible video modes that the firmware could set and the display's EDID.
• For each "head" that the boot loader told the BAL about; boot loader asks BAL which video mode it wants and boot loader sets that video mode. Note: BAL does some complex voodoo with EDID to determine best mode, etc; then starts a "boot module" to handle that video mode, and pre-initialises that boot module before telling boot loader to set the mode. This is intended to avoid a "boot module failed to initialise and the display doesn't work yet so there's no way to tell user what went wrong" problem.
• Finally, boot loader tells BAL it's finished (and that BAL can assume control of hardware - e.g. that "exitBootServices" has been called if firmware was UEFI); and BAL cleans up any memory the boot loader used (and starts nuking the daylights out of PCI devices, etc).

Note that I have another type of boot module that detects CPU features, etc. The BAL uses this information (plus information about the physical address space map) to auto-determine which type of kernel it should start; and the kernels themselves are in the "initial RAM disk" that the boot loader loaded. Once the BAL has determined what kind of kernel to start, it finds the corresponding "kernel setup module" and start it; and it's the "kernel setup module" that sets up the selected kernel (including changing CPU mode and paging to whatever the kernel wants, etc).

The "kernel setup module" also promotes itself and "boot modules" (e.g. those that were/are being used to handle video) so that they look like processes. This means that if anything goes wrong it's still possible for the "kernel setup module" (using the boot modules) to tell the user what happened (the micro-kernel is "micro", and therefore shouldn't and doesn't include any code for video at all). When all that's done, the "kernel setup module" starts other processes ("virtual file system", "device manager", etc) which in turn start more processes (device drivers, etc). Then "kernel setup module" terminates itself like any normal process would.


Cheers,

Brendan

Sourcer wrote:Hey,And my question to you guys is, where does the job of the bootloader end? Does it only load the kernel into memory and transfer control to it?
Does it turn on protected mode, paging, and then transfer control to the kernel?

It does whatever you want it to do.

What does your bootloader do?

I use GRUB. I'm interested in the mechanics of operating systems, but bootloaders are just boring. Might as well use the work that someone has already done for this task.

Sourcer wrote:And my question to you guys is, where does the job of the bootloader end?

The job of boot loader(s) is to setup a consistent initial environment for your kernel.
You either invent your own specification, or use someone else's like none or multiboot

Brendan wrote:Note that for a micro-kernel you can't just load the kernel, because the kernel on its own has no device drivers and wouldn't be able to load device drivers from anywhere. You have to load some kind of "initial RAM disk" that contains things (drivers, file system code, etc) that the micro-kernel will need.

You don't need a RAM disk file system for this. You can let the required modules piggy back on the kernel binary. I think this is how some microkernels are doing it (L4, Mach?). This is what I do at least. Also you need to start the file system servers before you actually read from the file system.

Hi,

OSwhatever wrote:

Brendan wrote:Note that for a micro-kernel you can't just load the kernel, because the kernel on its own has no device drivers and wouldn't be able to load device drivers from anywhere. You have to load some kind of "initial RAM disk" that contains things (drivers, file system code, etc) that the micro-kernel will need.

You don't need a RAM disk file system for this. You can let the required modules piggy back on the kernel binary. I think this is how some microkernels are doing it (L4, Mach?). This is what I do at least.

Even just concatenating multiple files and loading them into RAM is "a system of files that are in RAM".

OSwhatever wrote:Also you need to start the file system servers before you actually read from the file system.

No, you don't.

For mine, it's just a series of entries where each entry has a header (containing file name, etc) and the file's data; and boot code does a linear search to find the right entry for a file name. Later (when VFS is started) those entries are used to prime the VFS cache, so normal file IO can be used read the files (and will get a "VFS cache hit"). The VFS only needs file system servers to satisfy a "VFS cache miss" and for writes.

When file system server/s are started, the VFS checks if files in its cache are also in the file system/s, and if they're not it writes them to the file system (if possible). This creates the illusion that the VFS's cache was caching files that exist in the file system (even though it may have been caching files that didn't exist). This is also used for fault tolerance purposes (if you accidentally trash/wipe a normal file system's partition, the OS's critical files will be restored from the initial RAM disk); and used to simplify the OS installer - the OS installer only install files needed during boot (including the initial RAM disk) in a special "boot files" area (wherever makes sense for the boot loader - possibly on a TFTP server, possibly in UEFI system partition, ...); and doesn't create any actual file systems (but if you use the OS without any file systems for 10 days and then decide to create a file system for it, the OS will still copy critical files into that file system).


Cheers,

Brendan

My boot protocol looks surprisingly similar to Brendan's.

The differences are:

• My boot layer is loaded by GRUB. I don't have the time or motivation to write an own "fetch files from disk" loader.
• The boot layer prepares phyiscal memory management data structures for the kernel: It divides the system's RAM into regions with different properties (e.g. below 1 MiB, below 16 MiB, below 4 GiB) and builds buddy allocation tables for them. That way the kernel has physical MM available from the start which greatly simplifies the rest of the MM bootstrap code.
• I don't bother with graphics at boot. I have a driver for the graphics cards of my test machines and that suffices for now.

Once I support NUMA I'll move ACPI table parsing to the boot layer in order to be able to respect memory latencies.

Sourcer wrote:And my question to you guys is, where does the job of the bootloader end? Does it only load the kernel into memory and transfer control to it?
Does it turn on protected mode, paging, and then transfer control to the kernel?
What does your boot loader do?

My boot loader is the operating system.

Boot sector, second stage boot loader, third stage boot loader and forth stage operating system is one contiguous assembled (FASMg),
compiled (GCC) and linked named file which resides in the root directory of the bootable device file system.

The first 512 bytes of this file (the boot sector) and the following second stage boot loader (4096) bytes are additionally part of the
bootable image and contained within the 24 sector exFAT boot sector region or FAT32 reserved sectors (the two file systems I currently
support). The PC (or MBR loaded by the PC) loads the boot sector which loads the second stage boot loader which loads the third stage
boot loader from the named file in the root directory.

The entire file is not loaded. The beginning clusters which do not contain the third stage are skipped, the next cluster is buffer loaded
and the bytes up to the the third stage are skipped and the remaining are transferred immediately following the second stage in
memory. The remaining clusters are loaded in place up to the end of the third stage. Because the operating system is also a boot
loader, the forth stage will be loaded (the same way) only if required. No fixups or relocations are required and of course the second
stage has a rudimentary file system driver capable of loading the LFN file from the root directory.

The other benefit of this single file operating system is that the same file can be used in the formatting of other (exFAT or FAT32)
bootable devices as the same identical file is used for both file systems. Of course this not not preclude the operating system from
loading additional auxiliary files as usual.

A further benefit is that the forth stage shares the file system driver code in the second stage and does not need to include the same code.

I fully realize that this constitutes OSDev heresy.
I would however, be completely remiss if I also didn't mention here that the earth is not flat.

Heresy or not, what I'm more interested in is what you are using as a specification for exFAT?
My 2 target machines of 5, have bare NAND as a main persistent storage. One of the first thing I'll need is decent storage support. While at first, using eMMC and SD, the life is easier and it's quite well possible to begin with only FAT support, for NAND, it's not as simple.
Since I am a total noobass regarding file systems, I am just starting to look at what to chose for this capricious storage device.
Would exFAT be a good fit for a bare NAND? Even if so, I fear it's not exactly legal, to just implement it, right?
Then I'm looking at YAFFS2. But as it's almost always the case with open source, all the "documentation" on it, it's clumsy site passages of how cool it is. And then just a ton of the code, tuned into linux.

And to be on-topic, my bootloader is going to be an ordinary UEFI application OS loader when finished. It will not be bothered with multiboot OS selections, since it's a task of the UEFI Boot Manager. But probably it will incorporate selection of my OS version, - this is especially relevant for the development stage. It's good to have 1 loader which you can direct to boot your multiple OS development versions. The loader will reside in the system partition. OS files will be sitting in the boot partition. which are not the same. And if the boot partiotion won't be of supported by the FW type (other than FAT,) then my bootloader should have at least basic support of the boot partition type. Like in the paragraph above, I mentioned NAND storage. Boot loader shoud understand the type of FS chosen there. Be it YAFFS2 for example, it should be able to locate needed files there and load them.
Among those mentioned file it should load some registry hives into RAM. More, it should understand a little the registry organization structure to be able to read from there what it needs to load yet.
Then after reading from the registry part what it should load, it loads that (these are the kernel and drivers files).
It should prepare enough page tables for the kernel too.
It should transfer configuration information grabbed from the FW to the kernel (ACPI tables).
Then it should image load the kernel PE file, and its imports (hal.dll as a minimum). Thus it should load these as executable images, and dynamically link them.
Then transfer control to the kernel entry.
It's not written yet. But now I am doing almost the same with Sec.sys Dxe.exe pair. Where first is a UEFI loader and the second - UEFI kernel.

i think if you are into protected mode, the boot loader should set up the graphics, and also switch to protected mode for you. maybe you want a two stage boot loader for this.

For me, bootloader's job is:

1. load temporary GDT
2. put processor in protected mode
3. A20 gate
4. get kernel off of boot medium
5. get ramdisks off of boot medium
6. set video mode
7. get memory map
8. jump to kernel, passing off information about video, ramdisks, and memory

I use GRUB

zaval wrote:Heresy or not, what I'm more interested in is what you are using as a specification for exFAT?
My 2 target machines of 5, have bare NAND as a main persistent storage. One of the first thing I'll need is decent storage support. While at first, using eMMC and SD, the life is easier and it's quite well possible to begin with only FAT support, for NAND, it's not as simple.
Since I am a total noobass regarding file systems, I am just starting to look at what to chose for this capricious storage device.
Would exFAT be a good fit for a bare NAND? Even if so, I fear it's not exactly legal, to just implement it, right?
Then I'm looking at YAFFS2. But as it's almost always the case with open source, all the "documentation" on it, it's clumsy site passages of how cool it is. And then just a ton of the code, tuned into linux.

If you just want a simple file system, then the normal VFAT would suffice for most systems. VFAT is well documented and there are a lot of examples on the net. VFAT or exFAT might not be optimal for NAND and VFAT does a better job with eMMC with an on-board FTL layer.

I've discovered that creating your own filesystem not only give a lot of experience with them but could also be just as fast as implementing an existing one. You can often start very simple and then add features gradually. For raw NAND, log-structured file systems are usually a good fit like F2FS. Creating an own log-structured file system shouldn't be that hard and could also be quite fun as well as giving you total control. Developing file system can be done "offline", inside visual studio or similar so you will have all the debugging features of a normal program.

OSwhatever wrote: If you just want a simple file system, then the normal VFAT would suffice for most systems. VFAT is well documented and there are a lot of examples on the net. VFAT or exFAT might not be optimal for NAND and VFAT does a better job with eMMC with an on-board FTL layer.

Thanks. Yes, FAT is enough for the beginning. As a "dream" for farther stages, it would be very cool for me to get into NTFS of course, since my OS is (going to be) NT-like, but if not, then JFS2 or XFS. This is a wish list.) Too ambitious. My current target machine doesn't have neither SATA nor hdd anyway.

I've discovered that creating your own filesystem not only give a lot of experience with them but could also be just as fast as implementing an existing one. You can often start very simple and then add features gradually. For raw NAND, log-structured file systems are usually a good fit like F2FS. Creating an own log-structured file system shouldn't be that hard and could also be quite fun as well as giving you total control. Developing file system can be done "offline", inside visual studio or similar so you will have all the debugging features of a normal program.

Creating your own of course is always fun, but I think I've chosen a lot of bicycles to invent, adding there yet a FS, would be a real sign of goals infeasibility.
I think since I am a noob in FS design, it's better to choose something developed by professionals. F2FS makes compression of the content, right? I don't exactly like this and need. YAFFS2 looks a good fit for bare NAND. It's a log structured FS as well, doing wear leveling as a side product, which is good.