Generic bootloader

[AlfaOmega08]

Hi, just like b.zaar, I didn't read all of the pages. I just came on the forum after a long time, to discuss the need of a new bootloader, and whoa! I found this thread.

I'd love to help you with this project.

Thinking about style & language, I'm used to the one Griwes described. I would, however, prefer using tabs.
This way, each developer can configure his editor to make tabs look like 2/4/8 spaces at their wish. And you would still end up with consistent indenting.

Since this is a new and, afterall, experimental project, I would like to see some C++ advanced features. For example I would have a:

Code: Select all

class FileSystem
{
public:
    virtual File *Open(const char *fn) = 0;
    // [...]
};

class Fat32 : public FileSystem
{
public:
     // [...]
};

This might look crazy in such a low level environment, but you know C++ is almost 30 years old, and no mainstream bootoloader or OS really uses its powerful features.

Just... why not?

Edit:
Even though this won't solve the MBR problem (having different files for each combination of fs/device), once the main executable is booted, using C++ you may have the following neat code:

Code: Select all

Firmware *fw = new Bios;     /* Or new UEFI */
Device *dev = new CDDevice(fw);     /* Or new Disk, or PXE... */
Filesystem *fs = new Iso9660(dev);     /* Or new Ext2, Fat32, ... */
File *file = fs->Open("/boot/kernel");
ExecutableFormat *exec = new Elf(file);   /* ... */
exec->Load();
exec->Run();

[bluemoon]

If one of the goal is to provide a slim version of boot loader for hobby OS (verse bloat and huge GRUB), the Makefile should be quite trivial and work with gmake, cmake, etc.

make do not support variable alias(so each reference to a path wildcard require a re-scan) so I would not recommend it.

By the way, I see the trend of people excited to put everything into this project (and thus make it bloat and complex to use), I hope I'm wrong.

[Griwes]

The only issue (long term) could be make, but that's probably something to deal with as it comes

Use cmake from the start, then you won't have any make related problems.

CMake IS a problem. I would rather use shell scripts than CMake, to be honest.

This way, each developer can configure his editor to make tabs look like 2/4/8 spaces at their wish. And you would still end up with consistent indenting.

The problem with tabs is that they don't mix with spaces. Some day, you may want to use some spaces - and then your tabs will have killed you.

As for C++, there is no real need (oh, I didn't suppose I'd ever say that). You should use right tools for the right job; OO approach buys nothing here (raw array of function pointers sitting somewhere in memory is easy to handle using proper struct; you may not be able to do that using polymorphism, thanks to no standard way of handling vtables etc.).

[dozniak]

Makefile should be quite trivial and work with .., cmake

How do you picture that? Just curious.

[dozniak]

Use cmake from the start, then you won't have any make related problems.

CMake IS a problem. I would rather use shell scripts than CMake, to be honest.

Care to explain why? It looks a bit hypocritical after telling me that I do not provide enough explanations to just diss a portable tool able to generate all sorts of makefiles, VS projects, ninja scripts, Code::blocks projects etcetcetc, without any explanation.

So what exact problem IS cmake?

[bluemoon]

Makefile should be quite trivial and work with .., cmake

How do you picture that? Just curious.

That was a suggestion. I meant the Makefile should be keep simple, using baseline syntax, so you can always switch the make system with little effort.

[Owen]

A boot loader (that is not a "boot manager") should have no use for any scripting language.

Get a physical memory map, load some files, find some ACPI tables, setup a video mode, then pass control to the first loaded file. You'd need something to determine which video mode/s are supported (maybe some field's in the first file's header) and something to determine which files to load from where (e.g. "starting LBA" and "number of sectors" fields in the boot loader itself, that are set when the boot loader is installed).

Of course it's not quite that simple (e.g. you'd want to collect a bit more information than that, and have a standardised way to pass all the information to the first loaded file). However, if you start adding all sorts of unnecessary complexity (scripting languages? modules? a C library? Why not slap a Tetris clone in it in case the user gets bored? No, how about a web browser! Yeah, with flash; and java too so people can play Minecraft! Woot), then you'll end up with something like GRUB2 instead of something clean and efficient.


Cheers,

Brendan

A modern OS inherently needs a boot manager of some description. Whether it be to support multiple side by side kernels (as done by Linux), or resilient upgrades (e.g. where a "checkpoint" of the system state is taken before starting the upgrade, and if there is an issue with the upgrade one can always just boot from that checkpoint)

You argue that they're two separate jobs; on UEFI, this can easily be the case (because UEFI supports passing command line arguments to applications); however, when booted from the BIOS, there is clearly no clean way to pass the information. Besides; your proposal pretty much requires running the whole boot manager in real mode.

Maybe scripting is unnecessary; but at some point some configuration language probably will. This is all something that should probably be worked on once there are more solid foundations

---

I'm going to add to this:

I don't see a single loader being able to satisfy everyone's needs. I'd much rather see a simple "runtime" upon which said loader can be written.

There might be a BIOS chainloader application, a Multiboot loader application, a Linux loader application, plus new OSes could define their own. In that way, it's kind of an abstraction of what the various boot environments (BIOS, UEFI, bare metal, etc) provide.

The more familiar that runtime can be made, the better; thats why I propose the majority of it be the ISO C standard API: It's relatively slim and relatively complete.

On top of that could be added the necessary domain-specific functionality: an ELF loader, for example (for loading applications); perhaps routines for setting up page tables in a specific manner, then a final routine to enable it and jump to a specified address.

The runtime itself need not provide a boot menu. I'm going to suggest that the BIOS bootloader "backend" just load a specific file - but, because the BIOS doesn't provide a boot menu function of its own - that file be a boot manager.

The same boot manager could be used on bare metal systems as the platform boot manager. On UEFI, it can simply be ignored (because UEFI provides one already); instead, you'd just pass the application you want to run (and any arguments to it) straight to the runtime.

[JackScott]

Hi all,

I'd just like to mention group work in the context of Internet forums. I was one of the leaders of a previous group project of this forum (http://forum.osdev.org/viewtopic.php?t=16226) and we certainly bit off more than we could chew. We managed to get one issue out, then inspiration and willpower faded. To this day, only a single issue was released.

On the other hand, I think Pedigree (the operating system, not the beer or the dog food) is also a previous group project. By only taking in a few competent programmers to begin with and limiting design decisions to an even more select subset, they managed to pull off a group project that lasted at least two years and managed to produce 90% of the components of a usable operating system.

The moral of the story?

• Limit yourself. Small goals are easier than large goals.
• Don't take design suggestions from everybody.
• Managing willpower and manpower is the hardest challenge.
• One great programmer is equal to a thousand average programmers when it comes to group work... Brooke's law comes in to play enormously in the early stages.

[dozniak]

Makefile should be quite trivial and work with .., cmake

How do you picture that? Just curious.

That was a suggestion. I meant the Makefile should be keep simple, using baseline syntax, so you can always switch the make system with little effort.

Code: Select all

add_executable(gandalf source1.c source2.asm)

is rather hard to beat, ain't it?

[Cognition]

I agree with Brendan that it would be wise to avoid scope creep here and keep things simple. If it gets to the point where there's scripting and modules and so forth then you're probably better off just fixing grub really. It seemed like the main goal of this bootloader was to be small and simple vs feature rich. I'd agree that you really need some way to pass command line arguments, an initrd and switch between partitions or OS installs. If the goal is avoid a lot of GRUB's complexity and fluff though I'd definitely leave out things like bitmap menus and the ridiculous amount of scripting that grub2 currently uses.

I'm going to add to this:

I don't see a single loader being able to satisfy everyone's needs. I'd much rather see a simple "runtime" upon which said loader can be written.

There might be a BIOS chainloader application, a Multiboot loader application, a Linux loader application, plus new OSes could define their own. In that way, it's kind of an abstraction of what the various boot environments (BIOS, UEFI, bare metal, etc) provide.

I agree with this in theory, so long as the runtime is actually staged seperately from the bootmanager/loader itself. Simply having a predetermined boot environment that can pass control and whatever information it gathers to pre-kernel application should be sufficient. That said there's no reason that a good API couldn't be created for such applications. I think that would be the easiest and most flexible way to do things here in a manner that would keep complexity and size in the bootloader/bootmanager itself to a minimum. You should be able to do just about anything with an application and initrd image really. The image could contain an additional configuration file, kernel and the kernel's initrd, modules, etc. It would be simple enough to build a cross platform set of utilities to package such images and an API to interface with the bootloader and packages themselves. Keep the boot manager itself with a core set of functionality in the earlier stages of the bootloader and let OS developers or distributers customize the later stages on their own. It would be easy enough to build and host a subset of later stage apps that provide the functionality you listed with a standardized API/library that could be used for easy customization and extension.

[Owen]

I agree with this in theory, so long as the runtime is actually staged seperately from the bootmanager/loader itself. Simply having a predetermined boot environment that can pass control and whatever information it gathers to pre-kernel application should be sufficient. That said there's no reason that a good API couldn't be created for such applications. I think that would be the easiest and most flexible way to do things here in a manner that would keep complexity and size in the bootloader/bootmanager itself to a minimum. You should be able to do just about anything with an application and initrd image really. The image could contain an additional configuration file, kernel and the kernel's initrd, modules, etc. It would be simple enough to build a cross platform set of utilities to package such images and an API to interface with the bootloader and packages themselves. Keep the boot manager itself with a core set of functionality in the earlier stages of the bootloader and let OS developers or distributers customize the later stages on their own. It would be easy enough to build and host a subset of later stage apps that provide the functionality you listed with a standardized API/library that could be used for easy customization and extension.

I wrote up an overview as I envisage things. I've scaled things back a little from earlier thoughts; for example, I class a "command interpreter" as a "nice to have" thing that, besides which, would be easy to provide on top of the aforementioned runtime as an application.

The most complex part of the runtime, I see, is an executable loader; but that's pretty much a given, whatever we do.

[Brendan]

Hi,

I agree with Brendan that it would be wise to avoid scope creep here and keep things simple. If it gets to the point where there's scripting and modules and so forth then you're probably better off just fixing grub really. It seemed like the main goal of this bootloader was to be small and simple vs feature rich. I'd agree that you really need some way to pass command line arguments, an initrd and switch between partitions or OS installs. If the goal is avoid a lot of GRUB's complexity and fluff though I'd definitely leave out things like bitmap menus and the ridiculous amount of scripting that grub2 currently uses.

Exactly - if you try to do everything that anyone could ever want, it will be so bloated that nobody will want it (assuming it's possible to actually complete it).

If you start small and get the required functionality working (e.g. getting a memory map, setting up a video mode and loading "flat binary" file/s without any file system), then 75% of people will be happy because they'd be able to start working on their kernels with the minimum of hassle.

The other 25% can fork it and add extra/custom features specifically for their OS (e.g. support for their custom file system, ELF or PE or whatever support, scripting, encryption, etc); and they'll be happy too because they've got something clean, simple and reliable to build on and don't need to do a lot of "boilerplate" code. In fact, I'd encourage the "fork it if you want custom features" idea and call it a "boot kit" for this reason.

A boot loader (that is not a "boot manager") should have no use for any scripting language.

A modern OS inherently needs a boot manager of some description. Whether it be to support multiple side by side kernels (as done by Linux), or resilient upgrades (e.g. where a "checkpoint" of the system state is taken before starting the upgrade, and if there is an issue with the upgrade one can always just boot from that checkpoint)

Imagine something that displays a menu and allows the user to choose to boot Windows or Linux or BCOS; and then either chainloads Window's boot loader, chainloads GRUB or chainloads the BCOS boot loader. This is a boot manager. It might have features to allow the user to add new OSs to the list, might have password protection, might have extra tools for creating partitions, doing memory tests, etc. It might (should) come with it's own "stand-alone" installer. None of this has anything to do with any specific OS's boot loader.

Now imagine an OS's boot loader. That boot loader only ever needs to know how to boot one OS; and does not need to support chainloading, or the Linux boot protocol, or multiboot, or anything else intended for any other OS. The boot loader for that specific OS may have something to determine which kernel to boot (if that's something the specific OS needs), but deciding which kernel is very different to deciding which OS.

In addition, an OS should be able to easily change its own boot loader's configuration; but should not be able to touch the boot manager's configuration or any other OS's boot loader configuration. The idea of combining boot manager with boot loader is a very poor design that fails very easily for dual-boot systems (e.g. where multiple different OSs think they "own" the boot configuration and screw each other up) and is also a security disaster.

You argue that they're two separate jobs; on UEFI, this can easily be the case (because UEFI supports passing command line arguments to applications); however, when booted from the BIOS, there is clearly no clean way to pass the information. Besides; your proposal pretty much requires running the whole boot manager in real mode.

I don't understand why a well designed OS would need command line arguments (is it just because Linux is an ugly mess that expects end users to know more than kernel developers, and people make the mistake of assuming it's a good idea?). I also don't understand why a boot manager would need to run entirely in real mode, but I don't really care much because (hopefully) nobody here is talking about implementing a boot manager.


Cheers,

Brendan

[rdos]

I disagree that boot loader and boot manager should be different projects. If there are multiple OSes that can be booted, it doesn't work for the boot loader to just load one of these, and rely on that one to provide a boot-manager for the others. In any serious project that is not only installed permanently to load your own OS, you need a boot manager. I simply cannot see any reason why anybody would use a generic boot-loader to just load their own OS. Naturally, they would roll their own boot loader in such a case, especially since it can be tailor-made for the specific OS.

If I designed a multiple OS boot manager, I'd do it considerably different from Grub. Having a configuration file for boot alternatives is simply a bad idea. I would instead scan root partitions (and boot-sectors) for valid boot alternatives, and giving all valid alternatives automatically to the user. That would save a lot of problems for end-users that typically don't know the syntax of the boot scripting file, and in the case of Grub on Linux, have no idea how to recreate the scripts after modifications. Simply put, GRUB does an awful job as a boot-manager, and there is much that could be done better.

[bluemoon]

rdos I think you misunderstood.

I think most people agree that there is a need for proper boot manager, but we also think that boot manager is also a big project of its own, so it's an idea to keep it isolated from boot loader (and perhaps use the standard interface, ie. chainloading, to work with it).

Brendan has a good description for the main difference of their roles.

[Owen]

In addition, an OS should be able to easily change its own boot loader's configuration; but should not be able to touch the boot manager's configuration or any other OS's boot loader configuration. The idea of combining boot manager with boot loader is a very poor design that fails very easily for dual-boot systems (e.g. where multiple different OSs think they "own" the boot configuration and screw each other up) and is also a security disaster.

This is one of the reasons I emphasized every boot option being defined in it's own file; so that OSes don't accidentally trample each other. True, an OS could still screw everyone over by modifying the default= line in the boot manager configuration each time it starts up... but, then again, it could replace the MBR with it's own (with itself marked as primary) each boot...

I don't understand why a well designed OS would need command line arguments (is it just because Linux is an ugly mess that expects end users to know more than kernel developers, and people make the mistake of assuming it's a good idea?). I also don't understand why a boot manager would need to run entirely in real mode, but I don't really care much because (hopefully) nobody here is talking about implementing a boot manager

Command line arguments are a method, not an end goal. Specifically, in the "separate boot manager" scenario, there is no way for the boot manager to pass to the boot loader that it would like to load "bcos-17" instead of the newly installed "bcos-18" (because the latter has broken this machine, for example); or that you would like to disable the graphical boot splash (because the machine is hanging during boot and you want to diagnose why)