Generic bootloader

[shikhin]

Hi,

The design I use (and probably quite common) have multiple stages for boot loader:

Code: Select all

0. standard MBR
  - Load 512 bytes VBR, this is independent to OS, and can be replaced by any other standard MBR
1. VBR
  - This is specific for each type of file system, ie. I have FAT16.asm for FAT16, FAT32.asm for FAT32
  - This VBR provide the file loading API (at pre-defined entry point) to subsequent stages
  - This VBB then load /boot.bin from the file system
2. BOOT.BIN
  - Memory Map
  - Enable A20
  - Detect CPU
3. I also built this into boot.bin
  - Load 32/64bit kernel and initrd
  - Setup consistent environment (CPU registers state, 32-bit protected mode without paging, or compatibility mode with identity mapped region, etc)
  - pass control to kernel

So, to support more FS you basically write different VBRs with the file loading API. You may also need to extend the stages to support different media source (BIOS HDD service, or PXE, or EFI) in similar manner.

That looks like a good design. I have a nearly similar design in mind, and I'll soon post it (in response to your latest post). Please don't mind my slow replies; I just realized how buggy real life is. And it triple faults every night.

My boot-loader design is a little different.

I'm pretty sure everyone would have a differing boot-loader design (mine would not work with a generic loader, either). That's good for those who really want fine-tune control over how their kernel is put in power. For others, a generic design can be very well made - and that's what I'm aiming at.

...loads 14 sectors directly after the MBR...

While that would be neat if you'd have the entire hard disk at your leisure, but if the hard disk is already properly partitioned, then I guess that won't work properly (how would you gain the 14 sectors directly after the MBR without repartitioning, etc).

...a FAT file system...

I'd go for support for multiple file systems (or a boot-loader which works with them all, without caring for a file system).

As can be seen, this boot scenario cannot be supported by a generic boot-loader. If the boot-loader sets up paging, my interface will need to revert back to no paging and redo it from scratch. If it sets up a GDT or IDT, same thing, I'd have to replace those with my own GDT and IDT.

For paging, it's more cleaner to have an option in the configuration file. The default would obviously be a flat 32-bit environment (without paging), and long-mode and 32-bit paging can just be optional add-ons.

If the plan is to provide most of the initialization parts of an OS, which in the end will be a highly comforming OS implementation since it is impossible to do this in a common way for widely divergent OSes, then, count my out. OTOH, if the project aims at supporting more than GCC cross-compiled OS-images that initialize some *nix environment, then maybe.

Hrm. The project would surely not become an OS implementation. For the latter, I'd leave you to judge for yourself (the design would soon follow... d'oh, said that again, but I'll just reply to the backlog before the last three posts).

Most OSes don't have memory identity-mapped after boot, so why do you want that setup? The place of the kernel also is very different between OSes and operating modes.

I'd surely not set-up identity-mapped paging. As for the place of the kernel, it can either be specified using the vaddr field in the ELF header (if the kernel's format is ELF), or in the config file if it's some other format.

Regards,
Shikhin

[shikhin]

Hi,

So, for revision control, I use git (and hope everyone's happy with that)? GitHub with organizations would work pretty well for this project, too.

After thinking for a while, count me in.

Consider done.

First, the bootloader should be file system agnostic, but not filesystem unaware (not that I'm arguing with that "no filesystem needed" from Brendan) - user (= operating system being booted, or it's developer; I'm going to keep using that word for them) should be able to put a filesystem driver, in format specified in specification (for example, with array of function addresses in first X bytes or something; it MUST be simple) in first (starting from Yth; we gotta get 2nd stage somewhere) Z sectors of disk/partition, and keep kernel (or application; who said that it must be kernel? again, let's stick with term "application" here; for more not really rational rationale look at naming paragraph) in some hardcoded location, like `boot/boot.gand` (why `.gand`? See naming paragraph). If those Z sectors aren't in filesystem driver format, they should be treated as application that is to be booted; that would be simple solution for both people wanting to be able to easily booted application, as well as to those wanting to recreate the partition every time they recompile their kernel.

At first looks, it seems to be a nice solution. However, I still don't like the idea of putting anything in the first N sectors. As far as I know (though I may be wrong), ext2/3/4 don't support having any such reserved sectors at the start of the disk/partition. Moreover, entrusting the user to write a filesystem driver... is bad. Of course, we could have some pre-built drivers for each fs, and then put them in the first N sectors. But that'd be almost similar to bluemoon's VBR per filesystem design.

Second, the boot sector should be filesystem aware; that is, I think that should be able to create per-filesystem (M/V)BR template file, that would allow the same code work (after, well, reassembling it) for different filesystems, that use different parts of (M/V)BR, as well as the legacy partition table. Just create template (putting some zeroes at places that shouldn't be overwritten to allow assembler to produce binary, but also creating some kind of 512 byte "mask", so HDD installer - assuming there is one - will be able to write boot sector without overwriting old data). Note: I have no idea how this is done in different OS installers, but I assume they use special filesystem drivers and preassembled boot sectors for supported filesystem; this solution *could* make it simpler and more generic.

I'm sorry, I lost you here. Could you please rephrase the above paragraph, or something?

Third, least important from those here - I talked about this on IRC before kinda like about a joke, but I think it's worth considering - name for the bootloader (or rather "the specification" and "the official specification implementation"). Following ELF and DWARF, I would like to propose GANDALF as name for the bootloader - `Generic, AdvaNceD Application Loader` (F could mean something like `(Filesystem aware)`). Here we get to that strange `.gand` extension - well, I hope it's self explanatory. We could even go with naming boot manager (to be written as another application for the bootloader!) like BALROG, where `LROG` at the end could mean `LoadeR Of Gandalf`; I still have to figure out how to use BA at the beginning. But again, that's not that important.

Wouldn't you like to write about yourselves "member of Gandalf Development Team"?

I'm quote happy with GANDALF as the name for the bootloader (and the whole project). However... it isn't available as a name for an organization on github.

Finally, other features it would have to have, or could have - more important first:
- memory map sanitization - probably not necessary for UEFI (*probably*), but definitely recommended for ol' BIOS
- VBE mode setting (configurable at install)
- getting IOAPICs and LAPICs in known state (it's not much work, after all; by known state, I mean "disable everything you can")
- if it would already mess with APICs, maybe some basic ACPI parsing (like combining various entries of MADT, creating table that doesn't have variable-length entries, maybe - not much people would use it, but having that data parsed by write-once-use-all-the-time tool wouldn't hurt, especially when there is no SRAT and SLIT - applying SRAT/SLIT to that MADT structure and to memory map) - I mean, one table with variable length entries less to parse = profit!
- (?) AP booting (?)

• Memory map sanitation is a must. I do it in my project too (though I'm not sure of the code quality; I wrote it a time ago, and never looked at it again).
• VBE mode setting should also be there (as an option). For this, I planned to have some set of controls in the config file, which would allow the OS to specify which types of modes it allows. For example, an OS could do something like the following in the config file:

  Code: Select all

  filter-vbe-ratio: 4,3; filter-vbe-max-resolution: 1024,786; filter-vbe-max-bpp: 32;

  That way, everyone can decide which modes they want to switch to, or not. It could perhaps also give a *specific* list of which modes it wants. The idea is open.
• No, I don't think I want to do anything with IOAPIC or LAPIC. The most I'd do is to get the PIT into a clean state (no "hangover" interrupts, all interrupts masked, cli). You may convince me otherwise.
• Again, same for ACPI parsing. What I plan to do, however, is to give the address of the RSDP & MPS main table to the kernel. That's necessarily... since UEFI does table-probing in a different firmware-dependent manner. You may convince me otherwise (for parsing ACPI).
• Again, I don't want to do AP booting. Each OS might take that differently, and having all those options in the configuration file... bah.

Regards,
Shikhin

[rdos]

I don't like the idea of having configuration files for boot-loaders. Why not search the root on all available partitions for valid boot-images (if you use some type of signature this would work)? That's the method I use for RDOS boot-loader. It searches the partitions for valid RDOS images, and gives the user a list of possible boot choices. That way there is no need for configuration files.

About putting the image at fixed disc positions. If the boot-loader is installed in the MBR, a possible solution is to put the VBR directly after the MBR. That should not be used by any filesystem. Again, that is what RDOS bootloader does. It always installs itself in the MBR, and then uses 14 sectors directly after the MBR for second stage.

[bluemoon]

Why not search the root on all available partitions for valid boot-images.

As discussed above, this feature is provided by the boot manager and not the objective of this (these) boot loaders.

And I don't think it's good to put custom MBR (and there is no need anyway); a custome MBR introduce one more incompatible factor for other OS.

[shikhin]

Hi,

I don't like the idea of having configuration files for boot-loaders. Why not search the root on all available partitions for valid boot-images (if you use some type of signature this would work)? That's the method I use for RDOS boot-loader. It searches the partitions for valid RDOS images, and gives the user a list of possible boot choices. That way there is no need for configuration files.

That might work. However, for that, you'd be assuming that all the different kinds of kernels are same. They support all video modes, they want to be in 32-bit flat mode, etc. Without a configuration file, I don't see how generic this could be...

About putting the image at fixed disc positions. If the boot-loader is installed in the MBR, a possible solution is to put the VBR directly after the MBR. That should not be used by any filesystem. Again, that is what RDOS bootloader does. It always installs itself in the MBR, and then uses 14 sectors directly after the MBR for second stage.

What I meant to ask was (I'm not always clear with my questions): I don't think you can use your method of having 14 sectors after the MBR for an already partitioned disk, can you?

Regards,
Shikhin

[shikhin]

Hi,

Why not search the root on all available partitions for valid boot-images.

As discussed above, this feature is provided by the boot manager and not the objective of this (these) boot loaders.

And I don't think it's good to put custom MBR (and there is no need anyway); a custome MBR introduce one more incompatible factor for other OS.

I think rdos is implying that the bootloader searches for the specific boot image (using some signature) and boots it using a standard set of configurations. That won't be a boot manager... but I still don't like it.

(and yeah, no custom MBR)

Regards,
Shikhin

[bluemoon]

I don't think you can use your method of having 14 sectors after the MBR for an already partitioned disk, can you?

It depends on how far you want to compile to standard.
The MBR itself works on any LBA, however modern tools allocate partition on head boundary - so sector 2~62 is usually unused.

Having said that, if one in a zillion chance the user uses an ancient tool, or a custom OS tool that utilize all sectors, will you take the blame?

[shikhin]

Hi,

I don't think you can use your method of having 14 sectors after the MBR for an already partitioned disk, can you?

It depends on how far you want to compile to standard.
The MBR itself works on any LBA, however modern tools allocate partition on head boundary - so sector 2~62 is usually unused.

Having said that, if one in a zillion chance the user uses an ancient tool, or a custom OS tool that utilize all sectors, will you take the blame?

I want to compile to all standards 100%. Thus, I guess I won't take the risk (think of something else).

Regards,
Shikhin

[jnc100]

Surely some file system support is a must. Nearly all modern kernels require the boot loader to supply some sort of extra module that is loaded from the file system (e.g. an initial ramdisk), and therefore you need to be able to parse the file system to locate these and load them, or do you plan on hard coding the LBAs of these modules at install time and hoping the system doesn't move them?

In addition, if going down the direct-to-long mode route I'd suggest you simply load the kernel to whatever virtual address it is supposed to be loaded at, not identity map. Then provide the kernel with a list of used and free physical memory areas and also a list of used virtual memory areas. Unfortunately if we keep expanding the information to pass to the kernel in this way then eventually you will end up with something that looks a lot like GRUB 2.

Regards,
John.

[shikhin]

Hi,

Surely some file system support is a must. Nearly all modern kernels require the boot loader to supply some sort of extra module that is loaded from the file system (e.g. an initial ramdisk), and therefore you need to be able to parse the file system to locate these and load them, or do you plan on hard coding the LBAs of these modules at install time and hoping the system doesn't move them?

I think most people do that by having all those at a "reserved" area. I'd still have file system support (details soon to follow - oh no, said it again).

In addition, if going down the direct-to-long mode route I'd suggest you simply load the kernel to whatever virtual address it is supposed to be loaded at, not identity map. Then provide the kernel with a list of used and free physical memory areas and also a list of used virtual memory areas. Unfortunately if we keep expanding the information to pass to the kernel in this way then eventually you will end up with something that looks a lot like GRUB 2.

For physical memory - that's easy. I just add another type of entry to the memory map called "system area" which is what the kernel + modules use. For the virtual memory areas, I'd put the virtual addresses along the list of modules I pass to the kernel. That doesn't look overly convoluted to me.

Regards,
Shikhin

[rdos]

Hi,

I don't like the idea of having configuration files for boot-loaders. Why not search the root on all available partitions for valid boot-images (if you use some type of signature this would work)? That's the method I use for RDOS boot-loader. It searches the partitions for valid RDOS images, and gives the user a list of possible boot choices. That way there is no need for configuration files.

That might work. However, for that, you'd be assuming that all the different kinds of kernels are same. They support all video modes, they want to be in 32-bit flat mode, etc. Without a configuration file, I don't see how generic this could be...

That information is put in the image itself. It knows that it wants to be loaded in protected mode with paging disabled, or that it wants to be loaded in long mode with paging enabled. It is better to let the OS image supply this data rather than a configuration file, which eventually would be edited by an user that might not know the details of the OS image.

[jnc100]

Surely some file system support is a must. Nearly all modern kernels require the boot loader to supply some sort of extra module that is loaded from the file system (e.g. an initial ramdisk), and therefore you need to be able to parse the file system to locate these and load them, or do you plan on hard coding the LBAs of these modules at install time and hoping the system doesn't move them?

I think most people do that by having all those at a "reserved" area.

The initial ram disk on my linux box (stock ubuntu) is 14 MB in size - that's quite a large 'reserved area'.

For physical memory - that's easy. I just add another type of entry to the memory map called "system area" which is what the kernel + modules use. For the virtual memory areas, I'd put the virtual addresses along the list of modules I pass to the kernel. That doesn't look overly convoluted to me.

You're right, its not. Unfortunately when you try and support lots of different kernels, e.g. real mode, protected mode, long mode you realise you need to pass different information to each, then when you consider you boot from different mediums e.g. floppy/HD/CD/network etc you may want to pass information about those too (e.g. the bootp address and the tftp server in the case of network boot) and then when you add in 'optional' stuff like vbe modes your kernel information structure either becomes very large or you have a list of tagged items like Multiboot 2.

Regards,
John.

[shikhin]

Hi,

That information is put in the image itself. It knows that it wants to be loaded in protected mode with paging disabled, or that it wants to be loaded in long mode with paging enabled. It is better to let the OS image supply this data rather than a configuration file, which eventually would be edited by an user that might not know the details of the OS image.

So that when a monitor doesn't accept some sort of a VBE mode (and the OS can't detect that), the user won't be able to change the configurations (or do so via something more tough than just changing a file)?

The initial ram disk on my linux box (stock ubuntu) is 14 MB in size - that's quite a large 'reserved area'.

Agreed.

You're right, its not. Unfortunately when you try and support lots of different kernels, e.g. real mode, protected mode, long mode you realise you need to pass different information to each, then when you consider you boot from different mediums e.g. floppy/HD/CD/network etc you may want to pass information about those too (e.g. the bootp address and the tftp server in the case of network boot) and then when you add in 'optional' stuff like vbe modes your kernel information structure either becomes very large or you have a list of tagged items like Multiboot 2.

I see your problem - but I don't see any way to avoid the structure becoming too large.

(just for the record, real mode WON'T be supported, as far as I am concerned)

Regards,
Shikhin

[Griwes]

First, the bootloader should be file system agnostic, but not filesystem unaware (not that I'm arguing with that "no filesystem needed" from Brendan) - user (= operating system being booted, or it's developer; I'm going to keep using that word for them) should be able to put a filesystem driver, in format specified in specification (for example, with array of function addresses in first X bytes or something; it MUST be simple) in first (starting from Yth; we gotta get 2nd stage somewhere) Z sectors of disk/partition, and keep kernel (or application; who said that it must be kernel? again, let's stick with term "application" here; for more not really rational rationale look at naming paragraph) in some hardcoded location, like `boot/boot.gand` (why `.gand`? See naming paragraph). If those Z sectors aren't in filesystem driver format, they should be treated as application that is to be booted; that would be simple solution for both people wanting to be able to easily booted application, as well as to those wanting to recreate the partition every time they recompile their kernel.

At first looks, it seems to be a nice solution. However, I still don't like the idea of putting anything in the first N sectors. As far as I know (though I may be wrong), ext2/3/4 don't support having any such reserved sectors at the start of the disk/partition. Moreover, entrusting the user to write a filesystem driver... is bad. Of course, we could have some pre-built drivers for each fs, and then put them in the first N sectors. But that'd be almost similar to bluemoon's VBR per filesystem design.

The specification could as well state that "first sector of 2nd stage is X, first sector of driver/application is Y, their lengths are Z and W", and define what dwords/qwords (using NASM/YASM terms) define those values.

Second, the boot sector should be filesystem aware; that is, I think that should be able to create per-filesystem (M/V)BR template file, that would allow the same code work (after, well, reassembling it) for different filesystems, that use different parts of (M/V)BR, as well as the legacy partition table. Just create template (putting some zeroes at places that shouldn't be overwritten to allow assembler to produce binary, but also creating some kind of 512 byte "mask", so HDD installer - assuming there is one - will be able to write boot sector without overwriting old data). Note: I have no idea how this is done in different OS installers, but I assume they use special filesystem drivers and preassembled boot sectors for supported filesystem; this solution *could* make it simpler and more generic.

I'm sorry, I lost you here. Could you please rephrase the above paragraph, or something?

Yeah, it was a little chaotic ;D

Consider following file, to be included for every FS-specific (or MBR-specific, doesn't matter) bootsector:

Code: Select all

bits 16
org 0x7c00

%macro begin 0
begin:
    jmp 0x0:start
%endmacro

%macro code 0
start:
    mov ax, 0
    mov ds, ax
    mov es, ax 
    ; ...all the bootloader code
%endmacro

%macro fill 0
    times 510 - ($-$$) db 0
    dw 0xaa55
%endmacro

Then, an example of actual bootsector file would be:

Code: Select all

%include "gandboot.inc"
begin
db "Example Label   "
; ...whatever is required to be here
code
fill

I would have to think for a moment about generating those masks, though (like: original data - 0x93, 0x54, 0xff, 0xac; bootloader - 0x00, 0xac, 0xac, 0xef; mask - 0x00, 0xff, 0xff, 0x00 => 0x93, 0xac, 0xac, 0xac; hope you get the idea). The point in it is to ease creating bootsectors for exotic filesystems.

Finally, other features it would have to have, or could have - more important first:
- memory map sanitization - probably not necessary for UEFI (*probably*), but definitely recommended for ol' BIOS
- VBE mode setting (configurable at install)
- getting IOAPICs and LAPICs in known state (it's not much work, after all; by known state, I mean "disable everything you can")
- if it would already mess with APICs, maybe some basic ACPI parsing (like combining various entries of MADT, creating table that doesn't have variable-length entries, maybe - not much people would use it, but having that data parsed by write-once-use-all-the-time tool wouldn't hurt, especially when there is no SRAT and SLIT - applying SRAT/SLIT to that MADT structure and to memory map) - I mean, one table with variable length entries less to parse = profit!
- (?) AP booting (?)

• Memory map sanitation is a must. I do it in my project too (though I'm not sure of the code quality; I wrote it a time ago, and never looked at it again).
• VBE mode setting should also be there (as an option). For this, I planned to have some set of controls in the config file, which would allow the OS to specify which types of modes it allows. For example, an OS could do something like the following in the config file:

  Code: Select all

  filter-vbe-ratio: 4,3; filter-vbe-max-resolution: 1024,786; filter-vbe-max-bpp: 32;

  That way, everyone can decide which modes they want to switch to, or not. It could perhaps also give a *specific* list of which modes it wants. The idea is open.
• No, I don't think I want to do anything with IOAPIC or LAPIC. The most I'd do is to get the PIT into a clean state (no "hangover" interrupts, all interrupts masked, cli). You may convince me otherwise.
• Again, same for ACPI parsing. What I plan to do, however, is to give the address of the RSDP & MPS main table to the kernel. That's necessarily... since UEFI does table-probing in a different firmware-dependent manner. You may convince me otherwise (for parsing ACPI).
• Again, I don't want to do AP booting. Each OS might take that differently, and having all those options in the configuration file... bah.

Fine, but: I would recommend against configuration files. Let the specification define bytes in 2nd stage that are responsible for given settings; that would remove the necessity to implement configuration file parsing in bootloader - at the cost of readability of settings, but creating small driver to change those settings based on some kind of CLI or GUI shouldn't be hard (granted you have driver for storage device). For example, in POSIXish environment, the driver could expose `/dev/bootloader`, allowing read/write access in some human-readable format.

Also, you meant PIC, not PIT, right? And to get truly clean state, you should also - IMHO - at least mask IOAPICs. Plus, I would drop MP support entirely; ACPI is there for really long time.

[bluemoon]

%include "gandboot.inc"
begin
db "Example Label "
; ...whatever is required to be here
code
fill

If stages are broken down nicely, each stage is sufficiently small and don't need the macro to add extra dependency / complexity.
And I expect the amount of common code for each "small stage" is just a few instructions.