Generic bootloader

[Brendan]

Hi,

You want to support different boot modes via modules - grub has them, you wan to support different filesystems support through modules and different stages - grub has them, and so on.

GRUB and multiboot has many serious design flaws. GRUB2 and multiboot2 fixes a small number of those flaws, and adds a lot more new flaws.

They're only useful for large OSs (Linux, FreeBSD, Solaris, etc; where GRUB developers customise the boot loader/manager to suit the OS) and small/limited OSs that are willing to accept the limitations. It's not a realistic option for anything in between.


Cheers,

Brendan

[dozniak]

GRUB and multiboot has many serious design flaws. GRUB2 and multiboot2 fixes a small number of those flaws, and adds a lot more new flaws.

I'm afraid that starting from scratch without even having a reference to already made mistakes just asks for making these same mistakes and some more for shure.

[Brynet-Inc]

GRUB and multiboot has many serious design flaws. GRUB2 and multiboot2 fixes a small number of those flaws, and adds a lot more new flaws.

I'm afraid that starting from scratch without even having a reference to already made mistakes just asks for making these same mistakes and some more for shure.

[bluemoon]

I'm afraid that starting from scratch without even having a reference to already made mistakes just asks for making these same mistakes and some more for shure.

I don't know why you have such an idea. I'm sure the opposite, everyone here understand what they want to make and perform necessary research. It's unfair to assume they has not done any research and thus naively thinks that they must doom to failure.

[bluemoon]

Sorry, it just seems way to obvious to me.
You want to support different boot modes via modules - grub has them, you wan to support different filesystems support through modules and different stages - grub has them, and so on.

These are not one would use to differentiate different products. modules/plugin, configurations, are just mechanism for extensions, just because they may use a similar mechanism to do some work does not automatically induce they are the same product - it's the work(or feature, goal, etc) that differentiate them.

you wan to support different filesystems support through modules

The specification is not ready yet, I'm not sure at the end how to support different file systems. It may be the VBR that handle the booting FS, then pull in drivers for other FS, I'm not sure.
Anyway, what do you expect if not (1)load FS module or (2) built the FS module into the boot loader - which in turn exceed 512B and require some kind of loading modules anyway?

different stages - grub has them, and so on.

Come on, it's either single stage or multiple stage, you can't have 2i +1 stage.

[dozniak]

Wouldn't "fixing" grub then work better?

[Kazinsal]

There's a difference between attempting to fix something that's mildly broken (think the chain on a bicycle) and attempting to fix something that's horribly mangled (think an entire bicycle that was run over by an 18-wheeler). One is a potentially valuable endeavour that would leave you with a nice, working bicycle. The other is fixing GRUB.

[Primis]

Count me in,
A bootloader shouldn't be larger than the kernel it's loading...
My only request is that we make this BSD-License Compatible.

[Brendan]

Hi,

you wan to support different filesystems support through modules

The specification is not ready yet, I'm not sure at the end how to support different file systems. It may be the VBR that handle the booting FS, then pull in drivers for other FS, I'm not sure.

For my boot loaders:

• Floppy boot loader uses "starting LBA of each file" where each file is in sequential sectors starting from its starting LBA. A floppy is too small to bother trying to store more than just the required files (boot code, kernel/s, boot image, etc) and there's only a small number of these required files. An actual file system (even just FAT) is a waste of time and space.
• For PXE/network boot; you have to use TFTP and therefore there's no file system code in the boot loader (the TFTP server can use any file system it likes and it doesn't matter).
• For "no emulation El Torito", the only sane choice is ISO9660. There are extensions to ISO9660 (Joliet, Rock Ridge) but these are compatible with ISO9660 and don't provide anything a boot loader would care about (e.g. you wouldn't want to mess with Unicode file names or *nix style file permissions).

That only leaves hard drives and "hard drive like devices". For these I keep changing my mind. One alternative is to have a special boot partition, which mostly just does the "starting LBA of each file stored in sequential sectors" thing.

The other alternative is to use "n reserved sectors" at the beginning of the partition to store things needed for boot; and mostly just do the "starting LBA of each file stored in sequential sectors" thing. For this case boot code doesn't need to care about the file system itself (it only uses those reserved sectors at the start of the partition), and I can guarantee it'll work fine for any native file system I design for my OS; which is all that matters to me.

Note: I refuse to allow my OS's boot code to be stored on something like FAT where anyone can change anything using any other OS; but do plan to support these file systems (for the purpose of exchanging files between different OSs). I have no intention of supporting file systems that are designed for other OS's security models (e.g. ext2/3/4, NTFS, etc); because I don't want my OS used to violate another OS's security and it's nearly impossible to honour another OSs security model (e.g. make sure that only user "foo" on my OS can access files belonging to user "foo" on the other OS).

Basically; for how I do things, the only file system any boot loader for my OS will ever need to care about is ISO9660.

I would suggest a similar approach for a generic boot loader. If you start messing about trying to support arbitrary file systems (including file systems that haven't been designed yet) then you'd probably have to support some sort of "file system modules" (including finding a suitable executable file format for those modules, defining APIs for the modules to use, etc); and then instead of figuring out where to load the kernel from you need to figure out where to load file system module/s from (which is the same problem you were trying to fix to begin with, just in a different place).


Cheers,

Brendan

[shikhin]

Hi,

I've been faced by the chicken&egg problem. If I try to write a specification, I think I need to reply & refer to all the posts. If I try to reply to all the posts, I keep on referring to the non-existent specification. I'll therefore draw up a rudimentary version of the specifications, over here.

Let's say we support two boot mechanism: BIOS and UEFI.

That's the plan.

For BIOS, there are few media types:

• HDD - Standard MBR will be loaded and chain to VBR_BIOS_HDD_{FS}
  FDD - Boot sector will be loaded, VBR_BIOS_FDD_{FS}
  iso9660 - Boot sector will be loaded, VBR_BIOS_CD_{FS}
  PXE - VBR_BIOS_PXE_{FS}

For UEFI, the similar versions apply, but instead of VBR it is the boot application.

So, there will be VBR_{FIRMWARE}_{MEDIA}_{FS}, many versions of VBR, it basically does:
1. provide API to identify booting mechanism, ie. BIOS, UEFI
2. provide API to access media (wrapper to the firmware functions)
3. provide API to access file (specific to the FS)

(I will not go into implementation detail here, but it can be highly modulized)
A specific version of VBR_{FIRMWARE}_{MEDIA}_{FS} will be picked up in the supported matrix, upon "format".

So, the VBRs or UEFI boot application will then load the next stage, by convention let's make it /boot/boot.bin
The boot.bin will see an uniform interface independent of boot mechanism (BIOS or UEFI)
It will make use of the API setup previously, and
1. handle the necessary business logic(memory map, video mode, CPUID, pull in configuration, etc),
2. load additional resources
3. setup consistent enviroment and machine state,
4. then boot the kernel.

I have a similarly based design, and, without any further ado, here it is:

There exists roughly two stages. The general idea is that the first stage (hence called "Stage 1") would be firmware-dependent and boot-device-dependent. The next stage (hence called "Stage 2") would be both firmware-independent and boot-device-independant.

Stage 1 would first start of by doing everything which is firmware-dependent - this includes enabling A20, getting a memory map, getting VBE information, getting EDID information, finding the ACPI & MPS tables, switching to an identifiable text mode, and anything else I haven't thought of. It would then expose an API which would allow Stage 2 to access all file related functions (Open, Read, Close) and other firmware related functions (SwitchVideoMode, anything else I haven't thought of).

Stage 2 would first parse the config file. It would then do everything common - it would clean (is sanitate a word?) the memory map, find the best video mode and switch to it, make some clean & neat table to store all the data. It would then read the kernel & modules, switch on paging (or perhaps long mode), and jump to the kernel.

That is mostly an overview. The actual implementation details would be something like the following:

1. Stage 1. The Stage 1 would have the purpose of doing all things firmware-specific and boot-device-specific. There'd be two versions of this:
   1. BIOS.
      • For all things NOT-hardisk, I'll split Stage 1 into two stages (d'oh!) - Stage 1 & Stage 1.5. Stage 1 should do everything boot-device-specific, while Stage 1.5 would do everything firmware-specific. Stage 1 would provide an API (most probably in some register, so that implementation can vary) which would allow the later stages to Open, Read & Close files. This API should be generic. The idea is that different Stage 1s can be used with the same Stage 1.5. The location for these would be as follows:
        
        • Floppy. For floppies, we'd support FAT 12/16/32 and the no-filesystem idea described by Brendan. In case a filesystem is present, FAT supports having some number of reserved sectors. All the stages (1, 1.5 & 2) should be stored in these reserved sectors. Since we can load the rest of the sectors for Stage 1 from these reserved sectors, the size of the Stage 1 isn't limited to 512 bytes.
        • PXE.For PXE, the location of the file on disk shouldn't really matter, and it's size isn't limited either.
        • CD.For CDs, if I recall correctly, ISO9660 doesn't put the bootsector at LBA 0, but somewhere around. Moreover, while the size of the boot file can be as large as you want, most firmwares support 2KiB best. The rest of the files would be in a BOOT directory.
        Stage 1 would be therefore filesystem (err, whatever) aware. It would load Stage 1.5, pass the API and jump to it. Stage 1.5 would do everything firmware specific (set A20 pin, get memory map, etc). These two stages would operate in real mode.
      • For all things harddisk, things start getting a bit confused (darn you to hell, filesystem designers). The most generic idea I could gather from all the posts, is still filesystem aware (can't do without it) and limited. The idea is that the MBR & VBR would both be custom. The MBR (512 bytes) would provide all hard disk specific functions (ReadSector, etc) and good enough error detection & reporting.
        
        It'd then pass control to the VBR (512 bytes) which would provide all filesystem related functions (Open, Read, Close), and would be different for each filesystem. These both, together, form the Stage 1. These would load Stage 1.5, which would do everything firmware related. Obviously, this is severely limited. Doing disk-access in 512 bytes, and filesystem-access in 512 bytes - I'm not too sure how plausible this is (especially to do it at a good level, not something for a 512-byte competition or something) Again, everything's in real mode, here.
   2. UEFI. For UEFI, Stage 1 would exist as a single binary. As far as I've heard, if it's a 32-bit PE, then it'd be loaded in 32-bit protected+paging mode. This is what we'd do, then (for reasons described later). This shouldn't be tough - again, I'm not entirely familiar with UEFI, but I think it provides a FAT partition in every case (even CDs), so it should be generic. If anyone has more knowledge on this, please share.
2. Stage 2. Stage 2 would do everything not related with the firmware. It'd be in 32-bit flat mode (no paging), which is why we prefer 32-bit protected+paging in the UEFI loader. Obviously, there'd be change in modes a lot - from 32-bit to real mode, or from no-paging to paging. These switches would be handled by lower modes (Stage 1), and higher modes shouldn't care about them. There'd be a minimalistic configuration file, since most of the information should be specified by some sort of a binary header in the kernel. A file is still needed for things like module list, etc. Please note that the Stage 2 would be same for UEFI & BIOS. It'd load the kernel, parse everything (and stuff), switch modes finally, and jump to the kernel.

I've missed a lot of stuff in there. I haven't covered GPT at all, since I lack intimate knowledge on it. I guess someone could assist. Stuff on UEFI is lacking, and I've assumed a lot for it. Again, this should be covered as we progress.

I'll now reply to the rest of the posts (and hopefully, it should be easier). Let the criticism flow!

Regards,
Shikhin

[shikhin]

Hi,

Yeah, it was a little chaotic ;D

Consider following file, to be included for every FS-specific (or MBR-specific, doesn't matter) bootsector:

Code: Select all

bits 16
org 0x7c00

%macro begin 0
begin:
    jmp 0x0:start
%endmacro

%macro code 0
start:
    mov ax, 0
    mov ds, ax
    mov es, ax 
    ; ...all the bootloader code
%endmacro

%macro fill 0
    times 510 - ($-$$) db 0
    dw 0xaa55
%endmacro

Then, an example of actual bootsector file would be:

Code: Select all

%include "gandboot.inc"
begin
db "Example Label   "
; ...whatever is required to be here
code
fill

I would have to think for a moment about generating those masks, though (like: original data - 0x93, 0x54, 0xff, 0xac; bootloader - 0x00, 0xac, 0xac, 0xef; mask - 0x00, 0xff, 0xff, 0x00 => 0x93, 0xac, 0xac, 0xac; hope you get the idea). The point in it is to ease creating bootsectors for exotic filesystems.

That... really shouldn't be required. The disk-specific code would be all same (one binary to rule 'em all), while the fs-specific code would be all different (so no need for any macros there). Unless I really missed your point again, and you meant something different...

Finally, other features it would have to have, or could have - more important first:
- memory map sanitization - probably not necessary for UEFI (*probably*), but definitely recommended for ol' BIOS
- VBE mode setting (configurable at install)
- getting IOAPICs and LAPICs in known state (it's not much work, after all; by known state, I mean "disable everything you can")
- if it would already mess with APICs, maybe some basic ACPI parsing (like combining various entries of MADT, creating table that doesn't have variable-length entries, maybe - not much people would use it, but having that data parsed by write-once-use-all-the-time tool wouldn't hurt, especially when there is no SRAT and SLIT - applying SRAT/SLIT to that MADT structure and to memory map) - I mean, one table with variable length entries less to parse = profit!
- (?) AP booting (?)

• Memory map sanitation is a must. I do it in my project too (though I'm not sure of the code quality; I wrote it a time ago, and never looked at it again).
• VBE mode setting should also be there (as an option). For this, I planned to have some set of controls in the config file, which would allow the OS to specify which types of modes it allows. For example, an OS could do something like the following in the config file:

  Code: Select all

  filter-vbe-ratio: 4,3; filter-vbe-max-resolution: 1024,786; filter-vbe-max-bpp: 32;

  That way, everyone can decide which modes they want to switch to, or not. It could perhaps also give a *specific* list of which modes it wants. The idea is open.
• No, I don't think I want to do anything with IOAPIC or LAPIC. The most I'd do is to get the PIT into a clean state (no "hangover" interrupts, all interrupts masked, cli). You may convince me otherwise.
• Again, same for ACPI parsing. What I plan to do, however, is to give the address of the RSDP & MPS main table to the kernel. That's necessarily... since UEFI does table-probing in a different firmware-dependent manner. You may convince me otherwise (for parsing ACPI).
• Again, I don't want to do AP booting. Each OS might take that differently, and having all those options in the configuration file... bah.

Fine, but: I would recommend against configuration files. Let the specification define bytes in 2nd stage that are responsible for given settings; that would remove the necessity to implement configuration file parsing in bootloader - at the cost of readability of settings, but creating small driver to change those settings based on some kind of CLI or GUI shouldn't be hard (granted you have driver for storage device). For example, in POSIXish environment, the driver could expose `/dev/bootloader`, allowing read/write access in some human-readable format.

Also, you meant PIC, not PIT, right? And to get truly clean state, you should also - IMHO - at least mask IOAPICs. Plus, I would drop MP support entirely; ACPI is there for really long time.

Those would be okay - but for things like a module list, you would need a configuration file.

Yes, PIC. We might look into IOAPIC. As for dropping MP support - I don't believe we should drop support for anything. We should try to support as archaic a system as possible, so that it can fit the demands for everyone (Want support for a 486? No worries, GANDALF can do that! or Want support for my latest 512-core machine, with Microsoft's new PUFU firmware? No worries, GANDALF can do that!).

If stages are broken down nicely, each stage is sufficiently small and don't need the macro to add extra dependency / complexity.
And I expect the amount of common code for each "small stage" is just a few instructions.

Exactly my point (please refer to design explained above).

/me skips replying to the next ~10 posts - those should be well answered by my previous design post.

It seems to me that you're designing GRUB here, folks.

No trolling, despite some weaklings attempting to pose it so.

Sorry, it just seems way to obvious to me.

You want to support different boot modes via modules - grub has them, you wan to support different filesystems support through modules and different stages - grub has them, and so on.

I'm afraid that starting from scratch without even having a reference to already made mistakes just asks for making these same mistakes and some more for shure.

I'm afraid we aren't designing GRUB here. Sorry, GNU.

Anyway, we do have some sort of a reference - multiboot. We know it's mistakes, and what it does correctly. We're learning from it, and we think we'll do everything much better.

Count me in,
A bootloader shouldn't be larger than the kernel it's loading...
My only request is that we make this BSD-License Compatible.

Code: Select all

            DO WHAT THE **** YOU WANT TO PUBLIC LICENSE
                    Version 2, December 2004

 Copyright (C) 2012 Gandalf Team <[email protected]>

 Everyone is permitted to copy and distribute verbatim or modified
 copies of this license document, and changing it is allowed as long
 as the name is changed.

            DO WHAT THE **** YOU WANT TO PUBLIC LICENSE
   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

  0. You just DO WHAT THE **** YOU WANT TO.

Anyone?

Regards,
Shikhin

[shikhin]

Hi,

For my boot loaders:

What I've found is that "my boot loader" != "generic bootloader".

• Floppy boot loader uses "starting LBA of each file" where each file is in sequential sectors starting from its starting LBA. A floppy is too small to bother trying to store more than just the required files (boot code, kernel/s, boot image, etc) and there's only a small number of these required files. An actual file system (even just FAT) is a waste of time and space.
• For PXE/network boot; you have to use TFTP and therefore there's no file system code in the boot loader (the TFTP server can use any file system it likes and it doesn't matter).
• For "no emulation El Torito", the only sane choice is ISO9660. There are extensions to ISO9660 (Joliet, Rock Ridge) but these are compatible with ISO9660 and don't provide anything a boot loader would care about (e.g. you wouldn't want to mess with Unicode file names or *nix style file permissions).

For PXE & "no emulation El Torito", our ideas match. On the other hand, for a floppy disk, one might or might not require a file system. I thus plan to support both (described above, in different post).

That only leaves hard drives and "hard drive like devices". For these I keep changing my mind. One alternative is to have a special boot partition, which mostly just does the "starting LBA of each file stored in sequential sectors" thing.

The other alternative is to use "n reserved sectors" at the beginning of the partition to store things needed for boot; and mostly just do the "starting LBA of each file stored in sequential sectors" thing. For this case boot code doesn't need to care about the file system itself (it only uses those reserved sectors at the start of the partition), and I can guarantee it'll work fine for any native file system I design for my OS; which is all that matters to me.

While I'd really love that idea (simple, and no hacky method), we're talking about a generic boot loader here.

I would suggest a similar approach for a generic boot loader. If you start messing about trying to support arbitrary file systems (including file systems that haven't been designed yet) then you'd probably have to support some sort of "file system modules" (including finding a suitable executable file format for those modules, defining APIs for the modules to use, etc); and then instead of figuring out where to load the kernel from you need to figure out where to load file system module/s from (which is the same problem you were trying to fix to begin with, just in a different place).

My idea roughly attempts to fix that, though it still lacks. I'm not too sure how to do it with something which doesn't care about filesystem (yet works with every filesystem). Perhaps the community can think up of something ingenious.

Regards,
Shikhin

[shikhin]

Hi,

As a quick update, I'll list down some of the things which we need to decide upon (apart from the design). I'll list down my preferences. If anyone has any other alternative, please list, and I'll surely consider it. Where there's a "??", we'll decide upon that later.

• Language choice. C and intel-style assembly.
• Code style. ??
• Tool choice. GCC, nasm and make.
• Revision control. Git & GitHub.
• License. WTFPL? However, IANAL so maybe new BSD might suit better? Surely not GPL, though. ??
• TABS vs SPACES! ??

Since there was no opposition to the name GANDALF (and it's really enticing), I think we'll go with it. #gandalf on freenode is ours to play with, and Gandalf-Team on GitHub as an organization is owned by us. If you want to be a member of that (you can either be an owner - full responsibility - or a contributor), please reply with your GitHub handle. The email gandalfgrey.team@gmail has also been registered by me, and once I've verified that you really want to participate, the password would be shared.

If there's ambiguity in anything else, please say so.

Let the game begin.

Regards,
Shikhin

[egos]

If you start messing about trying to support arbitrary file systems (including file systems that haven't been designed yet) then you'd probably have to support some sort of "file system modules" (including finding a suitable executable file format for those modules, defining APIs for the modules to use, etc); and then instead of figuring out where to load the kernel from you need to figure out where to load file system module/s from (which is the same problem you were trying to fix to begin with, just in a different place).

In my new stage 2 boot loader I use this way. Device and FS specific stage 1 loads stage 2 kernel and device and FS specific driver from root directory. I do not care about loading other drivers because my general aim is to organize choice between "multiple versions on single volume", not more. If you need to load something from other device/FS, put into boot menu a command to load other driver or to reload boot loader with other driver.

[shikhin]

Hi,

Another idea that everyone (including) egos seems to use. The VBR would just have enough code to load the filesystem driver, which would then, along with the MBR, form Stage 1. However, I'm not too sure if the VBR wouldn't really just become the filesystem driver (since it's got to have enough code to load the fs driver anyway). Is it worth it? What's your views?

Regards,
Shikhin