Generic bootloader

[Griwes]

As for the first paragraph: you are still mixing *bootloader* which isn't supposed to chainload with *boot manager* that is supposed to *only* chainload.

As for the second paragraph - it's so massive piece of bullcrap I don't even know how to respond it.

[rdos]

OK, I'll start a new thread on an user-friendly combined boot-loader and boot-manager since it is obvious that this is not the aim of this thread.

[b.zaar]

Ok, I think I see a good progression of this project to satisfy most of the people interested in contributing. Firstly the base system would be a simple boot loader, it can be installed to the MBR for a single OS install or it can be installed to the VBR of the OSes partition and be chain loaded from a non conforming (GANDALF boot standard) boot manager. The base system would include a simple library like PDClib. In this mode the boot loader loads only the single OS reading from a text configuration file for any boot options. The boot loader would configure the system to run in flat 32 bit pmode with a structure of the systems configuration in the GANDALF boot standard format. The boot loader requires executable file support (ELF/DWARF), as external modules, to be able to load the kernel. When the system is in a set mode it loads and executes it's first module or kernel file.

Then any boot loader extensions, scripting support, VBE support, Boot Manager support and conforming kernels would all be seen as modules to the boot loader. The boot loader remains resident and loads and runs the modules in order until it never returns to the boot loader as a boot manager or kernel has taken over control of the system. None of these extension are necessary to load any conforming kernel, they are there just to add functionality before the kernel is loaded and is controlling the PC. Any OS may choose not to include then in the primary installion of the boot loader.

[shikhin]

Hi,

Well, if people are opting for WTFPL and their kin, why not stick with the CC0 license - after all it is the license of PDCLib and you can't quite get more free than that.

Well, the WTFPL recommendation was... just a joke. CC0 looks nice too, but the sheer quantity of licenses available makes one wonder which to pick.

One other thing that can haunt people is the file format of the later stages (read: native windows compiler issues vs cross compiler issues vs possibly llvm). IMHO care has to be taken that unlike GRUB, compilation works out of the box on any host platform, even if it's just instructions to get the proper compiler.

Handling the file format of the later stages such that it causes no issues on different platforms (natively) might be tough. Just getting the proper compiler might be easier.

That'd be my $.02. I'd contribute code if I had time, intent, and an actual plan. Maybe I'll do an SFS driver sometime to promote some other community standard. But for now, I'll just be here sitting out the bureaucratic part .

Heh, thanks!

First I would like to offer scboot as a MBR/Boot sector that works on both HDD and FDD.

Writing our own MBR won't be tough (and it'd have the benefit of not relying on anything else).

For some of the following posts, Griwes answers them. For the rest, I agree with Brendan on the distinction between a boot manager and a boot loader.

Regards,
Shikhin

[shikhin]

Hi,

• Limit yourself. Small goals are easier than large goals.

Agreed.

• Don't take design suggestions from everybody.

This thread had a "get every bit of information from everyone" motive, but I do see the information overload issue. Anyway, I'm mostly trying to take in all the intelligent design suggestions (and leave out the rest).

• One great programmer is equal to a thousand average programmers when it comes to group work... Brooke's law comes in to play enormously in the early stages.

For that reason, I've thought up of the following:

• There is a GitHub organization for managing the members of this project.
• I've split the teams as two: Owners & Contributors. Owners get all the privileges, while contributors don't get administrative privileges.
• Currently, Owners are Griwes & I. Contributors are null. If you want to be either, please post a message here, and detailing some of your past works (to show your competence). I may not be the best one to curate this project... but I'll try.
• I'll try to limit Owners to mostly two more members, and Contributors to three more members, to keep the entire team short.
• Since I don't have that much of an experience with very large (large as in team) projects, I'll go with Griwes' suggestion: no one would commit directly to any repo in the organization. You'd fork & use pull requests, and have a branch per major feature.
• Since we don't have any forums for the project (if anyone is willing to donate a domain name for the project, we already have hosting space), we'd be using GitHub issues to think about any thing needed.

I'm more than happy to help out once we figure out how we're going to start collaborating on everything.

There.

Regards,
Shikhin

[shikhin]

Hi,

OK, I'll start a new thread on an user-friendly combined boot-loader and boot-manager since it is obvious that this is not the aim of this thread.

I'm sorry that this project wasn't able to satisfy your needs. I'd still try to ensure you that we WOULD write a boot-manager in the future, sometime. Just that it wouldn't be a part of the boot-loader (but a separate project).

Regards,
Shikhin

[Brendan]

Hi,

Ok, I think I see a good progression of this project to satisfy most of the people interested in contributing. Firstly the base system would be a simple boot loader, it can be installed to the MBR for a single OS install or it can be installed to the VBR of the OSes partition and be chain loaded from a non conforming (GANDALF boot standard) boot manager.

That sounds good to me.

The base system would include a simple library like PDClib.

A quick checklist:

• Does the boot loader need signals and signal handling?
• Does it need floating point (or complex numbers) for anything?
• Does it need setjmp/longjmp?
• Does fork()/exec() make any sense?
• Are time and date functions (and "sleep()") needed?
• How about localisation?
• Is support for wide characters needed?

This might only really leave file IO, heap and some string functions (strlen(), memcpy(), etc). For heap, UEFI firmware provides it (but it may need to be a little different as you may need to say what the memory is being allocated for) and for PC BIOS by the time you get through memory detection you'll probably have 90% of memory management done anyway (e.g. code to change the type of an arbitrary area in the memory map from one type into a different type - e.g. from "unknown" to "ACPI NVS", or from "usable RAM" to "allocated").

For file IO; directory handling can't work in some environments (e.g. when booting from network - TFTP simply doesn't support it) and writing to files can't work in some environments (CD-ROM). For simplicity, I wouldn't bother allowing directories to be read or created, and wouldn't support "stat()", and wouldn't support writing to files. I'd also use fixed file names, where the name/s of file/s are set when the boot loader is installed, because it's the only thing that makes sense when you can't read directories.

Of course you probably only need to read a maximum of 3 files - the OS's next piece of code (e.g. the kernel), an optional extra file (e.g. containing an initrd or boot image or something), and an optional something containing arbitrary data for the OS (e.g. kernel command line or whatever). Note: the OS's next piece of code might not be a kernel. For example, the initrd or boot image might contains 5 kernels, and the "OS's next piece of code" might be something to determine which of those kernels to start. The only thing a generic boot loader would need to know is that the "OS's next piece of code" is OS dependant and could be anything.

Now; if you're only loading a maximum of 3 files; excluding booting from network and CD-ROM, is it necessary to have file names when booting from hard drive or floppy? If people are willing to use something like "starting LBA and length" instead, then the need to support arbitrary file systems disappears (and a whole layer of pain/complexity disappears with it). This would mean that the files couldn't be fragmented (but that shouldn't be a problem and you could use "list of extents" instead if it is a problem). It would also mean that an OS would need to update the boot loader if the files are changed (but that is extremely easy due to "KISS"). What I'm suggesting here is that the advantages of supporting arbitrary file systems is "almost nothing" and the disadvantages (complexity/bloat) aren't justified.

I'd also point out (for the benefit of those people that simply assume something is good if Linux does it), that this is how the native boot loader for Linux (LILO) works. There's a utility (also called LILO) that installs the boot loader and creates a list of extents, and the boot loader simply loads sectors from that list of extents (with no knowledge of any file system needed). Note: GRUB is not the native boot loader for Linux - it's something that came alone much later, that tried to slaughter the boot loaders for many different OS's using a devastating weapon called "over engineering" (and sadly, succeeded in several cases).

So; getting back to C libraries, does the boot loader need anything more than "string.h", and is it worth bothering with a C library if a few loops are enough? Don't forget that the UEFI API is a pain (that doesn't use normal C calling conventions and requires special wrappers that are typically implemented in inline assembly), and that (for PC BIOS) most C compilers that people would want to use (GCC, clang/LLVM) don't support 16-bit/real mode code in the first place.

In this mode the boot loader loads only the single OS reading from a text configuration file for any boot options. The boot loader would configure the system to run in flat 32 bit pmode with a structure of the systems configuration in the GANDALF boot standard format. The boot loader requires executable file support (ELF/DWARF), as external modules, to be able to load the kernel. When the system is in a set mode it loads and executes it's first module or kernel file.

Is ELF necessary? Dynamic linking doesn't make any sense and I doubt anyone wants to use relocatable code, the section attributes will probably be ignored and you wouldn't want arbitrary sections at arbitrary addresses. I don't see any point requiring ELF for OS's that are hopefully not boring *nix clones (and therefore don't necessarily comply with the "UNIX System V" specifications, which includes the ELF file format). Just assume "flat binary" - it's easier and can work for ELF, PE, something designed for the OS itself, etc.

Then any boot loader extensions, scripting support, VBE support, Boot Manager support and conforming kernels would all be seen as modules to the boot loader. The boot loader remains resident and loads and runs the modules in order until it never returns to the boot loader as a boot manager or kernel has taken over control of the system. None of these extension are necessary to load any conforming kernel, they are there just to add functionality before the kernel is loaded and is controlling the PC. Any OS may choose not to include then in the primary installion of the boot loader.

I still don't think there's any need for any scripting languages. Slap some sort of data structure in the first module's "GANDALF header" and use fields in that data structure to control the boot loader. For example, you might have a few fields in this header that are used by the boot loader to determine which video modes are acceptable, a field for the entry point, and one more field in case I forgot something. If the OS needs a scripting language (for generating dynamic HTML pages or something?), then the OS's first module can be a script interpreter, or the OS project can fork the boot loader and add whatever features it likes.


Cheers,

Brendan

[rdos]

I think I'm starting to get totally lost on the objectives of this project.

First, it is claimed that this would not be a boot manager, and then there is talk about MBR and maybe it will become a boot manager later.

I mean, either it is a "local" boot loader located in some partition boot sector (in which case it can be chain loaded by some other boot manager), or it is a boot manager that can chain load. I cannot see that it can be both. And if you put it in the MBR, it MUST be a boot manager, since otherwise nothing else can be run.

Then there is multiboot, signatures and configuration files. Those are typically for boot managers, once again mixing up boot loaders with boot managers.

[bluemoon]

I feel the same as RDOS this time. I know it take time for the project to shape, until then I'll see how the goal and features become.

[shikhin]

Hi,

I think I'm starting to get totally lost on the objectives of this project.

First, it is claimed that this would not be a boot manager, and then there is talk about MBR and maybe it will become a boot manager later.

I mean, either it is a "local" boot loader located in some partition boot sector (in which case it can be chain loaded by some other boot manager), or it is a boot manager that can chain load. I cannot see that it can be both. And if you put it in the MBR, it MUST be a boot manager, since otherwise nothing else can be run.

Then there is multiboot, signatures and configuration files. Those are typically for boot managers, once again mixing up boot loaders with boot managers.

No, it's not supposed to become a boot manager later. GANDALF is and will remain a boot loader. What I meant was that we'll also write a boot manager (called BALROG) under the same umbrella sometime later.

Regards,
Shikhin

[Owen]

Ok, I think I see a good progression of this project to satisfy most of the people interested in contributing. Firstly the base system would be a simple boot loader, it can be installed to the MBR for a single OS install or it can be installed to the VBR of the OSes partition and be chain loaded from a non conforming (GANDALF boot standard) boot manager. The base system would include a simple library like PDClib. In this mode the boot loader loads only the single OS reading from a text configuration file for any boot options. The boot loader would configure the system to run in flat 32 bit pmode with a structure of the systems configuration in the GANDALF boot standard format. The boot loader requires executable file support (ELF/DWARF), as external modules, to be able to load the kernel. When the system is in a set mode it loads and executes it's first module or kernel file.

Then any boot loader extensions, scripting support, VBE support, Boot Manager support and conforming kernels would all be seen as modules to the boot loader. The boot loader remains resident and loads and runs the modules in order until it never returns to the boot loader as a boot manager or kernel has taken over control of the system. None of these extension are necessary to load any conforming kernel, they are there just to add functionality before the kernel is loaded and is controlling the PC. Any OS may choose not to include then in the primary installion of the boot loader.

The base system would include a simple library like PDClib.

A quick checklist:

• Does the boot loader need signals and signal handling?

The signal handling code in PDCLib consumes ~0.5kB of RAM/disk space

• Does fork()/exec() make any sense?

fork/exec are POSIX, not C. They are functions PDCLib will never contain

• Does it need floating point (or complex numbers) for anything?
• Does it need setjmp/longjmp?
• Are time and date functions (and "sleep()") needed?
• How about localisation?
• Is support for wide characters needed?

Excluding wide characters, localization, and all of these are features that you won't pay for if you don't use them through the magic of static linking. A valid implementation of many of them (e.g. the time functions) is to simply return an error saying they're not supported.

Localization is a pretty trivial matter (it only takes up a few extra bytes over not supporting it at all in the case where it doesn't get used - in the form of a pointer or two)

Wide characters are something I'll probably include the ability to compile out - but actually, yes, they're useful, if only for interacting with the Unicode portions of UEFI (e.g. the text output protocol)

This might only really leave file IO, heap and some string functions (strlen(), memcpy(), etc). For heap, UEFI firmware provides it (but it may need to be a little different as you may need to say what the memory is being allocated for) and for PC BIOS by the time you get through memory detection you'll probably have 90% of memory management done anyway (e.g. code to change the type of an arbitrary area in the memory map from one type into a different type - e.g. from "unknown" to "ACPI NVS", or from "usable RAM" to "allocated").

For heap support, on UEFI you could always choose not to use PDCLib's malloc and instead call through to the UEFI allocator; that is a fully supported thing to do.

Having a heap is useful for doing some buffering; plus, you never know what any dependent applications could need to do.

For file IO; directory handling can't work in some environments (e.g. when booting from network - TFTP simply doesn't support it) and writing to files can't work in some environments (CD-ROM). For simplicity, I wouldn't bother allowing directories to be read or created, and wouldn't support "stat()", and wouldn't support writing to files. I'd also use fixed file names, where the name/s of file/s are set when the boot loader is installed, because it's the only thing that makes sense when you can't read directories.

Of course you probably only need to read a maximum of 3 files - the OS's next piece of code (e.g. the kernel), an optional extra file (e.g. containing an initrd or boot image or something), and an optional something containing arbitrary data for the OS (e.g. kernel command line or whatever). Note: the OS's next piece of code might not be a kernel. For example, the initrd or boot image might contains 5 kernels, and the "OS's next piece of code" might be something to determine which of those kernels to start. The only thing a generic boot loader would need to know is that the "OS's next piece of code" is OS dependant and could be anything.

Now; if you're only loading a maximum of 3 files; excluding booting from network and CD-ROM, is it necessary to have file names when booting from hard drive or floppy? If people are willing to use something like "starting LBA and length" instead, then the need to support arbitrary file systems disappears (and a whole layer of pain/complexity disappears with it). This would mean that the files couldn't be fragmented (but that shouldn't be a problem and you could use "list of extents" instead if it is a problem). It would also mean that an OS would need to update the boot loader if the files are changed (but that is extremely easy due to "KISS"). What I'm suggesting here is that the advantages of supporting arbitrary file systems is "almost nothing" and the disadvantages (complexity/bloat) aren't justified.

I'd also point out (for the benefit of those people that simply assume something is good if Linux does it), that this is how the native boot loader for Linux (LILO) works. There's a utility (also called LILO) that installs the boot loader and creates a list of extents, and the boot loader simply loads sectors from that list of extents (with no knowledge of any file system needed). Note: GRUB is not the native boot loader for Linux - it's something that came alone much later, that tried to slaughter the boot loaders for many different OS's using a devastating weapon called "over engineering" (and sadly, succeeded in several cases).

stat and directory handling are POSIX. The C standard completely ignores the concept of directories; all the functions take a simple file name and the interpretation of that is system dependent.


Calling LILO the "native" boot loader of Linux is a little specious. Loadlin was at least contemporary with it; and both projects postdate the formation of the Linux project.

One could also easily get around the need to support millions of different file systems by just requiring a "boot partition" that is in a known, standard format (FAT?)

So; getting back to C libraries, does the boot loader need anything more than "string.h", and is it worth bothering with a C library if a few loops are enough? Don't forget that the UEFI API is a pain (that doesn't use normal C calling conventions and requires special wrappers that are typically implemented in inline assembly), and that (for PC BIOS) most C compilers that people would want to use (GCC, clang/LLVM) don't support 16-bit/real mode code in the first place.

On IA-32, ARM and IA-64, UEFI uses the "standard" calling convention

On AMD64, UEFI uses the "Microsoft" calling convention. There are two easy possibilities for getting around this:

1. Tell your compiler to use the MS ABI (i.e. GCC/Clang's -mabi=ms option)
2. Tell your compiler to use the MS ABI for the UEFI functions (i.e. the __attribute__((__ms_abi__)) option)

Is ELF necessary? Dynamic linking doesn't make any sense and I doubt anyone wants to use relocatable code, the section attributes will probably be ignored and you wouldn't want arbitrary sections at arbitrary addresses. I don't see any point requiring ELF for OS's that are hopefully not boring *nix clones (and therefore don't necessarily comply with the "UNIX System V" specifications, which includes the ELF file format). Just assume "flat binary" - it's easier and can work for ELF, PE, something designed for the OS itself, etc.

An ELF header is something as easy to embed in a flat binary as any other. You don't need much to make it loadable; the header and a segment table. Why reinvent the wheel, when there is already a perfectly good one?

Ok, I think I see a good progression of this project to satisfy most of the people interested in contributing. Firstly the base system would be a simple boot loader, it can be installed to the MBR for a single OS install or it can be installed to the VBR of the OSes partition and be chain loaded from a non conforming (GANDALF boot standard) boot manager. The base system would include a simple library like PDClib. In this mode the boot loader loads only the single OS reading from a text configuration file for any boot options. The boot loader would configure the system to run in flat 32 bit pmode with a structure of the systems configuration in the GANDALF boot standard format. The boot loader requires executable file support (ELF/DWARF), as external modules, to be able to load the kernel. When the system is in a set mode it loads and executes it's first module or kernel file.

Then any boot loader extensions, scripting support, VBE support, Boot Manager support and conforming kernels would all be seen as modules to the boot loader. The boot loader remains resident and loads and runs the modules in order until it never returns to the boot loader as a boot manager or kernel has taken over control of the system. None of these extension are necessary to load any conforming kernel, they are there just to add functionality before the kernel is loaded and is controlling the PC. Any OS may choose not to include then in the primary installion of the boot loader.

It would be nice to try and avoid any configuration files if possible - they make sense for a boot manager, not so much for a boot loader.

Let's simply load a designated file from a designated location as the first application and have it take over control? If it needs any modules, it can invoke them (as it would invoke any other application). The distinction between "applications" and "modules" can simply be that modules, to steal from DOS of all places, exit by "terminating and staying resident". They can register themselves with the core so that other applications can find them; perhaps also other applications can register to be notified when a new service is registered?

[Yoda]

Shikhin,
Did you look at my OS Boot Tools? It is ready made project for simple booting the small (and not only small) OS kernels independently of file system and partitioning scheme used. It does all mentioned for the first stage loading from many filesystems (including ISO9660) with the exception of PXE and UEFI. It also introduces the clear and standard interface between first stage bootloader and second stage / kernel. I suggest to adopt the proposed interface as a standard for the first stage.

For boot loaders themselves, I think "one boot loader" is a mistake. There are about 6 cases:

• Booting from floppy with PC BIOS (BPB, old "int 0x13" functions, no file system needed)
• Booting from "MBR partitioned" hard disk with PC BIOS (MBR partition table, "int 0x13 extensions", no file system needed)
• Booting from "GPT partitioned" hard disk with PC BIOS (GPT partition tables, "int 0x13 extensions", no file system needed)
• Booting from network with PC BIOS (PXE API, no file system needed)
• Booting from "no emulation El Torito" with PC BIOS (no partitions, "int 0x13 extensions" but with 2048-byte sectors, need ISO9660 file system)
• Booting with UEFI (everything different)

There are different leves of interactions in all these cases! UEFI means that you already use 32-bit environment, while others don't. Even MBR should interact with VBR to make booting process easy and reliable.

There is also the specification that determines how the boot loader passes control to the OS. This is a separate thing too. For example, you could write a specification like Multiboot; and 50 different people could write 50 different OSs that support your specification, and 20 different people could write 20 different boot loaders that support your specification; and anyone would be able to use any of those boot loaders with any of those OSs because they all comply with the same specification.

This gives us a total of at least 9 possibilities (2 boot managers, 6 boot loaders, plus at least 1 specification); where the project could be only one of these things, or some of them, or all of them.

That's not a problem providing that the interfaces are clearly specified and all components' implementations conform the standards. The main reason why I started my OS Boot Tools project is the complete absence of standards and comfortable tools for the first stage loading.

However, I still don't like the idea of putting anything in the first N sectors. As far as I know (though I may be wrong), ext2/3/4 don't support having any such reserved sectors at the start of the disk/partition.

Keeping something in fixed location is VERY bad idea making the installation of bootloader not user friendly and requires to take some serious precautions when handling filesystem. Most modern file systems reserve special area for boot code, ext2/3/4 is not an exclusion. The reserved space in ext2/3/4 is enough for loading the given file from the root directory.

The moral of the story?

• Limit yourself. Small goals are easier than large goals.
• Don't take design suggestions from everybody.
• Managing willpower and manpower is the hardest challenge.
• One great programmer is equal to a thousand average programmers when it comes to group work... Brooke's law comes in to play enormously in the early stages.

Exactly! You are right!

What do I propose for this project.
1. Break the whole project into weak coupled parts with clear interfaces. The first stage loading should be definitely separated into all-sufficient module/project. Besides, I see the set of second stage loaders and boot manager that should be made modular too.
2. Before making the same errors as other developers, inspect and discuss all flaws of other projects, without just mentioning "they are bad".
3. Get an extensive practice on booting, including odd hardware and rare circumstances.
4. Reduce enthusiastic noise full of overestimation that fills such projects and this topic.
5. Work! Work! Work! The makefile format and "tabs vs spaces" are not the things that worth spending the time while you don't have anything working. You'll be able to make cosmetic changes at any time.
6. You can't make steady specs until you get at least the raw working project. The flaws will become obvious only during using the almost ready project.

[Griwes]

Keeping something in fixed location is VERY bad idea making the installation of bootloader not user friendly and requires to take some serious precautions when handling filesystem. Most modern file systems reserve special area for boot code, ext2/3/4 is not an exclusion. The reserved space in ext2/3/4 is enough for loading the given file from the root directory.

http://www.nongnu.org/ext2-doc/ext2.html#EXT2-IMMUTABLE-FL - how is this not "user-friendly"?

And "fixed" means "set during installation/media preparation", not "fixed forever".

6. You can't make steady specs until you get at least the raw working project. The flaws will become obvious only during using the almost ready project.

You can't make bootloader and only then make specs; you have to work on both of them at the same time, for it to make sense. Also, you know, some of us have already written some bootloaders - it's not like we have no idea what want to do.

[Yoda]

how is this not "user-friendly"?

Because this requires special procedure every time you update second stage loader. This procedure may be risky in some circumstances or not be available at all if your OS fails booting. The example of risk is that installer fails recording the new version of loader for some reasons, but successfully deletes the old one. I already had in practice such problems several times with Linux. Again, not every filesystem provide means for "fixing" the location of file. Another requirement is special user API for getting the physical location of file. Most friendly procedure is just a simple file copying.

[Antti]

How is going to use this boot administrator? For newbies (like me), a system "run background" that is several orders of magnitude more complex than their own "OS" may make the whole OS development to lose its excitement. However, experts probably think this is not "professional" enough and do their own thing anyway (or use GRUB with its drawbacks).

Tip: skip this post and continue.