Generic bootloader

[neon]

Hello,

In my opinion a generic boot loader should leave paging disabled until required by the loader. This makes it simpler to work with firmware environments that can only execute in one CPU mode (like the BIOS). I like the idea of replacing multiboot though; multiboot 2 isn't properly implemented in GrUB last I seen.

I can share our design if interested. The aim of our design is to develop a generic boot loader framework capable of supporting multiple firmware types, architectures, boot interface specifications and easy to extend.

[egos]

I'm working for such boot loader. The general principles are:
- multitarget format support (Multiboot compliant, Linux kernel, Link kernel - it's main, boot sectors/boot records, maybe ELF and PE; the environment during booting module(s) should be based on the format of target module(s));
- own applet format support (with return to the kernel; with specific API);
- relocatable kernel (for supporting such formats that need (almost) all base memory);
- and so on.

[shikhin]

Hi,

Since I haven't replied since two days, I want to assure everyone that this project is NOT dead, and I'm currently thinking on the design (hampered due to what they call real life). The replies to each post, individually, shall soon follow - and so shall my design.

Regards,
Shikhin

[dozniak]

In my opinion; the first step would be to create a specification that clearly defines how the boot loader gets any information it needs from the "thing being booted", how the boot loader passes information to the "thing being booted", and how the boot loader passes control to the "thing being booted" (including the state of various pieces of hardware when this happens - e.g. if the "thing being booted" can assume A20 is enabled, if PCI devices are pre-configured, if interrupts and interrupt controllers are setup or not, etc).

Without some sort of specification that developers can use to determine what they'd need to implement, developers can't know what they'd need to implement..

One such specification exists, it is called "Multiboot specification". I recommend starting from there.

[dozniak]

Here's my prediction for Generic bootloader:
in 5 years you will laugh looking at the unfinished version 0.0.3 of the Generic bootloader and boot from UEFI+ or whatever they will name it.

[bluemoon]

Here's my prediction for Generic bootloader:
in 5 years you will laugh looking at the unfinished version 0.0.3 of the Generic bootloader and boot from UEFI+ or whatever they will name it.

Why? There is no conflict for UEFI and a boot loader (it can be a boot agent application under EFI framework).

[Griwes]

In my opinion; the first step would be to create a specification that clearly defines how the boot loader gets any information it needs from the "thing being booted", how the boot loader passes information to the "thing being booted", and how the boot loader passes control to the "thing being booted" (including the state of various pieces of hardware when this happens - e.g. if the "thing being booted" can assume A20 is enabled, if PCI devices are pre-configured, if interrupts and interrupt controllers are setup or not, etc).

Without some sort of specification that developers can use to determine what they'd need to implement, developers can't know what they'd need to implement..

One such specification exists, it is called "Multiboot specification". I recommend starting from there.

I call troll on this ^ one:

What I propose to do is to start a project for a bootloader (let's call it Generic bootloader, for a lack of any better name). The bootloader would obviously aim at beginners entering OSDev, who don't want to write any bootloaders themselves, but aren't happy with GRUB/multiboot.

Here's my prediction for Generic bootloader:
in 5 years you will laugh looking at the unfinished version 0.0.3 of the Generic bootloader and boot from UEFI+ or whatever they will name it.

Welcome to OSDev.org, where people tell you that you won't finish something, because other such thing already exists on daily basis!

Again, I call troll on this one.

[Brendan]

Hi,

Without some sort of specification that developers can use to determine what they'd need to implement, developers can't know what they'd need to implement..

One such specification exists, it is called "Multiboot specification". I recommend starting from there.

A simple boot loader that supports Multiboot could maybe solve half of the problems people have with GRUB - the extra/unnecessary complexity, the requirement for some sort of file system GRUB knows about, the hassle of porting tools (for installing and managing GRUB) to your OS, etc. For example, you could almost (but not quite) just concatenate the boot loader and a multiboot kernel and copy the result to a floppy or hard disk partition. You would need some way for the boot loader to determine where the end of the kernel is though, and you'd probably want some way to determine if there are modules and where they are too; but that could just be a simple list of "number of sectors" for each file.

There are also plenty of problems with Multiboot itself; so a simple boot loader (that isn't as complex/bloated/messy as GRUB) that supports Multiboot might not be what most people want.

Even in this case; starting from the "Multiboot specification" may be a good idea. For example, you could find things where Multiboot is lacking and extend/change/improve Multiboot. You might even be able to make these changes in a compatible way, depending on what those changes might be.


Cheers,

Brendan

[Brendan]

Hi,

In my opinion; the first step would be to list down some of the goals the project aims to fulfill. These are nonvolatile, and remain fixed throughout the project; rest all things are secondary (including the one you just said) and volatile. I would prefer drawing up the specifications with constant advice (and guidance) from whoever decides to participate.

That is a very good idea.

The first of these goals should be defining what the project itself is.

Traditionally, each OS has its own boot loader, and there's some sort of extra tool used to determine which OS (which boot loader) should be started. For example, you might have an MBR that displays a nice menu which chain-loads the selected OS's boot loader. For another example, you might have a UEFI menu or shell which starts the selected OS's "UEFI loader". One of the reasons why GRUB sucks is because it fails to separate these very different jobs - it's trying to be a "thing to decide which boot loader to start" (let's call them "boot managers" from now on) and also trying to be a boot loader for many very different cases, and it ends up being a complex mess (including problems like different OS's fighting over who controls GRUB's config and screwing each other up).

Note: I personally think that a boot manager should not be part of any OS; a boot manager should have its own installer and its own configuration; and end users should be able to replace any boot manager with any other boot manager *without* effecting any OSs boot loader. I also think that a good boot manager should provide some extra utilities, like support for boot password/s, a tool to allow partitions to be created and deleted, the ability to clone/backup partitions, etc. A boot loader should have none of these abilities (its job is to start one OS, nothing more).

For "boot managers"; you'd want one for PC BIOS and something completely different for UEFI.

For boot loaders themselves, I think "one boot loader" is a mistake. There are about 6 cases:

• Booting from floppy with PC BIOS (BPB, old "int 0x13" functions, no file system needed)
• Booting from "MBR partitioned" hard disk with PC BIOS (MBR partition table, "int 0x13 extensions", no file system needed)
• Booting from "GPT partitioned" hard disk with PC BIOS (GPT partition tables, "int 0x13 extensions", no file system needed)
• Booting from network with PC BIOS (PXE API, no file system needed)
• Booting from "no emulation El Torito" with PC BIOS (no partitions, "int 0x13 extensions" but with 2048-byte sectors, need ISO9660 file system)
• Booting with UEFI (everything different)

There is also the specification that determines how the boot loader passes control to the OS. This is a separate thing too. For example, you could write a specification like Multiboot; and 50 different people could write 50 different OSs that support your specification, and 20 different people could write 20 different boot loaders that support your specification; and anyone would be able to use any of those boot loaders with any of those OSs because they all comply with the same specification.

This gives us a total of at least 9 possibilities (2 boot managers, 6 boot loaders, plus at least 1 specification); where the project could be only one of these things, or some of them, or all of them.

However; for the 5 boot loaders for PC BIOS, some of the code has to be different (due to different boot devices), and some of the code will be the same in all of them (most but not all of the memory detection, all the video mode setup, etc). It makes sense to separate the "boot device dependant" code from the "boot device independant code". This might mean implementing the common ("boot device independant") code as a library so it can just be linked into every boot loader. Alternatively, it could mean splitting it into 2 independant binaries (a boot loader and a "stage 2") and defining a specification that describes how "boot loader" passes control to "stage 2". In both of these cases they could be considered independant pieces - e.g. 20 people could write 20 different boot loaders and 10 people could make 10 different "common PC BIOS code libraries" and everyone could mix and match; or 20 people could write 20 different boot loaders and 10 people could make 10 different "stage 2" binaries and everyone could mix and match.

This gives us a total of at least 10 possibilities; where the project itself could be just one of these things, or some of these things, or all of these things.

You could even split it into "n projects". For example, one project to develop and maintain the specification that determines how the boot loader passes control to the OS; plus one UEFI boot loader project; plus one "PC BIOS stage 2 and PC BIOS boot loaders" project; plus one "PC BIOS boot manager" project.


Cheers,

Brendan

[Griwes]

After thinking for a while, count me in. I have my own bootloader, and I need some future-kernel-specific things in it, but it can work just as indirection layer (bootloader -> "booter" as I call it -> kernel). And I still have few different bootloaders to write, then why not this one?

---------

EDIT: ok, let's make this post a little more constructive (note: when I write "the bootloader" below, I mean "every bootloader that satisfies the specification").

First, the bootloader should be file system agnostic, but not filesystem unaware (not that I'm arguing with that "no filesystem needed" from Brendan) - user (= operating system being booted, or it's developer; I'm going to keep using that word for them) should be able to put a filesystem driver, in format specified in specification (for example, with array of function addresses in first X bytes or something; it MUST be simple) in first (starting from Yth; we gotta get 2nd stage somewhere) Z sectors of disk/partition, and keep kernel (or application; who said that it must be kernel? again, let's stick with term "application" here; for more not really rational rationale look at naming paragraph) in some hardcoded location, like `boot/boot.gand` (why `.gand`? See naming paragraph). If those Z sectors aren't in filesystem driver format, they should be treated as application that is to be booted; that would be simple solution for both people wanting to be able to easily booted application, as well as to those wanting to recreate the partition every time they recompile their kernel.

Second, the boot sector should be filesystem aware; that is, I think that should be able to create per-filesystem (M/V)BR template file, that would allow the same code work (after, well, reassembling it) for different filesystems, that use different parts of (M/V)BR, as well as the legacy partition table. Just create template (putting some zeroes at places that shouldn't be overwritten to allow assembler to produce binary, but also creating some kind of 512 byte "mask", so HDD installer - assuming there is one - will be able to write boot sector without overwriting old data). Note: I have no idea how this is done in different OS installers, but I assume they use special filesystem drivers and preassembled boot sectors for supported filesystem; this solution *could* make it simpler and more generic.

Third, least important from those here - I talked about this on IRC before kinda like about a joke, but I think it's worth considering - name for the bootloader (or rather "the specification" and "the official specification implementation"). Following ELF and DWARF, I would like to propose GANDALF as name for the bootloader - `Generic, AdvaNceD Application Loader` (F could mean something like `(Filesystem aware)`). Here we get to that strange `.gand` extension - well, I hope it's self explanatory. We could even go with naming boot manager (to be written as another application for the bootloader!) like BALROG, where `LROG` at the end could mean `LoadeR Of Gandalf`; I still have to figure out how to use BA at the beginning. But again, that's not that important.

Finally, other features it would have to have, or could have - more important first:
- memory map sanitization - probably not necessary for UEFI (*probably*), but definitely recommended for ol' BIOS
- VBE mode setting (configurable at install)
- getting IOAPICs and LAPICs in known state (it's not much work, after all; by known state, I mean "disable everything you can")
- if it would already mess with APICs, maybe some basic ACPI parsing (like combining various entries of MADT, creating table that doesn't have variable-length entries, maybe - not much people would use it, but having that data parsed by write-once-use-all-the-time tool wouldn't hurt, especially when there is no SRAT and SLIT - applying SRAT/SLIT to that MADT structure and to memory map) - I mean, one table with variable length entries less to parse = profit!
- (?) AP booting (?)

-----

Back to naming - wouldn't you like to write about yourselves "member of Gandalf Development Team"?

[bluemoon]

I would like to add my opinion too.
Let's say we support two boot mechanism: BIOS and UEFI
For BIOS, there are few media types:

• HDD - Standard MBR will be loaded and chain to VBR_BIOS_HDD_{FS}
  FDD - Boot sector will be loaded, VBR_BIOS_FDD_{FS}
  iso9660 - Boot sector will be loaded, VBR_BIOS_CD_{FS}
  PXE - VBR_BIOS_PXE_{FS}

For UEFI, the similar versions apply, but instead of VBR it is the boot application.

So, there will be VBR_{FIRMWARE}_{MEDIA}_{FS}, many versions of VBR, it basically does:
1. provide API to identify booting mechanism, ie. BIOS, UEFI
2. provide API to access media (wrapper to the firmware functions)
3. provide API to access file (specific to the FS)

(I will not go into implementation detail here, but it can be highly modulized)
A specific version of VBR_{FIRMWARE}_{MEDIA}_{FS} will be picked up in the supported matrix, upon "format".

So, the VBRs or UEFI boot application will then load the next stage, by convention let's make it /boot/boot.bin
The boot.bin will see an uniform interface independent of boot mechanism (BIOS or UEFI)
It will make use of the API setup previously, and
1. handle the necessary business logic(memory map, video mode, CPUID, pull in configuration, etc),
2. load additional resources
3. setup consistent enviroment and machine state,
4. then boot the kernel.

[zhiayang]

I would like to add my opinion too.
Let's say we support two boot mechanism: BIOS and UEFI
-- SNIP --

So, there will be VBR_{FIRMWARE}_{MEDIA}_{FS}, many versions of VBR, it basically does:
1. provide API to identify booting mechanism, ie. BIOS, UEFI
2. provide API to access media (wrapper to the firmware functions)
3. provide API to access file (specific to the FS)

What do you mean by 'API'...? Preferably, the OS need only link/compile with one external header file, similar to GRUB (I must admit, it did well in that aspect,unlike UEFI). A pointer to a struct defined in the specification, perhaps?

[bluemoon]

I'm open to the actual implementation mechanism for API, just list here for example,
1. define entry points at specific location (eg. 7C00 - N*4) or random location (and pass pointer in register)
2. install interrupt handler
3. patch image of subsequent stages (run time linking)

For my own boot loader I used method 1.
Note that for method 3 it get redundent to patch multiple images(to tell them the API entry) if there are multiple stages.

[shikhin]

Hi.

Even in this case; starting from the "Multiboot specification" may be a good idea. For example, you could find things where Multiboot is lacking and extend/change/improve Multiboot. You might even be able to make these changes in a compatible way, depending on what those changes might be.

Starting from the multiboot specification seems like a good idea. What I have in mind (w/sh)ould be mostly very different from the specification, but I could at least improve upon it.

That is a very good idea.

The first of these goals should be defining what the project itself is.

Obviously; I just didn't realize how broad the term "bootloader" can be.

Traditionally, each OS has its own boot loader, and there's some sort of extra tool used to determine which OS (which boot loader) should be started. For example, you might have an MBR that displays a nice menu which chain-loads the selected OS's boot loader. For another example, you might have a UEFI menu or shell which starts the selected OS's "UEFI loader". One of the reasons why GRUB sucks is because it fails to separate these very different jobs - it's trying to be a "thing to decide which boot loader to start" (let's call them "boot managers" from now on) and also trying to be a boot loader for many very different cases, and it ends up being a complex mess (including problems like different OS's fighting over who controls GRUB's config and screwing each other up).

Note: I personally think that a boot manager should not be part of any OS; a boot manager should have its own installer and its own configuration; and end users should be able to replace any boot manager with any other boot manager *without* effecting any OSs boot loader. I also think that a good boot manager should provide some extra utilities, like support for boot password/s, a tool to allow partitions to be created and deleted, the ability to clone/backup partitions, etc. A boot loader should have none of these abilities (its job is to start one OS, nothing more).

For "boot managers"; you'd want one for PC BIOS and something completely different for UEFI.

A good boot manager seems to be an entire separate project of it's own. After some thought and contemplation, I think the project may have a separate boot manager (under the same umbrella) later. Currently, it just focuses on being a bootloader.

For boot loaders themselves, I think "one boot loader" is a mistake. There are about 6 cases:

• Booting from floppy with PC BIOS (BPB, old "int 0x13" functions, no file system needed)
• Booting from "MBR partitioned" hard disk with PC BIOS (MBR partition table, "int 0x13 extensions", no file system needed)
• Booting from "GPT partitioned" hard disk with PC BIOS (GPT partition tables, "int 0x13 extensions", no file system needed)
• Booting from network with PC BIOS (PXE API, no file system needed)
• Booting from "no emulation El Torito" with PC BIOS (no partitions, "int 0x13 extensions" but with 2048-byte sectors, need ISO9660 file system)
• Booting with UEFI (everything different)

My currently plan is to support all of the above, though I still have a doubt or two:

1. You only list "int 0x13 extensions" for both MBR & GPT partitioned hard disk - is it nice to assume every machine supports the extensions?
2. For floppy (with PC BIOS), MBR partitioned hard disk, and GPT partitioned hard disk, you say that no file system (support) is required. That almost seems impossible to be, given with the fact that several file systems don't allow for reserved sectors. Could you mind expanding?

There is also the specification that determines how the boot loader passes control to the OS. This is a separate thing too. For example, you could write a specification like Multiboot; and 50 different people could write 50 different OSs that support your specification, and 20 different people could write 20 different boot loaders that support your specification; and anyone would be able to use any of those boot loaders with any of those OSs because they all comply with the same specification.

Yes, it plans to be a specification along with an (hopefully ideal) implementation.

However; for the 5 boot loaders for PC BIOS, some of the code has to be different (due to different boot devices), and some of the code will be the same in all of them (most but not all of the memory detection, all the video mode setup, etc). It makes sense to separate the "boot device dependant" code from the "boot device independant code". This might mean implementing the common ("boot device independant") code as a library so it can just be linked into every boot loader. Alternatively, it could mean splitting it into 2 independant binaries (a boot loader and a "stage 2") and defining a specification that describes how "boot loader" passes control to "stage 2". In both of these cases they could be considered independant pieces - e.g. 20 people could write 20 different boot loaders and 10 people could make 10 different "common PC BIOS code libraries" and everyone could mix and match; or 20 people could write 20 different boot loaders and 10 people could make 10 different "stage 2" binaries and everyone could mix and match.

Yes, it plans to have a stage 2 (and a specification for it). I've mostly made up my mind on how it'd be - sorry for not posting here, I'd do that while replying to Griwes' & bluemoon's post.

Regards,
Shikhin

[shikhin]

Hi,

In my opinion a generic boot loader should leave paging disabled until required by the loader. This makes it simpler to work with firmware environments that can only execute in one CPU mode (like the BIOS). I like the idea of replacing multiboot though; multiboot 2 isn't properly implemented in GrUB last I seen.

I can share our design if interested. The aim of our design is to develop a generic boot loader framework capable of supporting multiple firmware types, architectures, boot interface specifications and easy to extend.

What I usually do (in my own project) is to "pretend" paging has been enabled. I create the page structures, and have functions for mapping (using the page structures directly, not using the self-mapping trick). However, I don't set the PE bit. The only thing I have to be sure is that I never access anything using the virtual address (since paging hasn't been enabled, I'd be accessing garbage), and everything works properly.

That also avoids the cost of transversing through the page structures, and also avoids the TLB misses.

Paging is only enabled just before jumping to the kernel.

I'm working for such boot loader. The general principles are:
- multitarget format support (Multiboot compliant, Linux kernel, Link kernel - it's main, boot sectors/boot records, maybe ELF and PE; the environment during booting module(s) should be based on the format of target module(s));
- own applet format support (with return to the kernel; with specific API);
- relocatable kernel (for supporting such formats that need (almost) all base memory);
- and so on.

Again, the goals match a lot. I hope you could give your guidance for this project too.

Regards,
Shikhin