Generic bootloader

[bluemoon]

You should not rely on custom MBR as part of the boot sequence. AFAIK It will be overwritten by installing Windows, FreeBSD and whatnot.

For the API, due to the limited space I suggest to squeeze a bit out of it:
There is only open and read (ie. no close), an example interface (from my boot code) would then look like this:

Code: Select all

; HANDLE OpenFile(const char* filename);
; INPUT
;    si -> filename
; OUTPUT
;    eax = HANLDE of file, 0 if not found

; HANDLE ReadFile(HANDLE handle);
; INPUT
;    eax = HANLDE of file
;    di -> Data Buffer (must be big enough)
; OUTPUT
;    eax = HANLDE of file (May be changed), 0 if end of file
;    ecx = bytes read

Note  : Assume CS,DS,ES are zero
For example, on FAT, EAX is used as cluster number internally, and each read return a cluster-size data.

The usage would be:
handle = FileOpen(filename);
while ( (handle=FileRead(handle,buffer)) != 0 ) {
 copy buffer to high address
}

The above code is not suitable for generic boot loader without cleaning up, but should be enough for inspiration.


For SPACE vs TAB, this can never end, so I would suggest just let the code formatter do the job (manually or automatically)

[Griwes]

Hi,

As a quick update, I'll list down some of the things which we need to decide upon (apart from the design). I'll list down my preferences. If anyone has any other alternative, please list, and I'll surely consider it. Where there's a "??", we'll decide upon that later.

• Language choice. C and intel-style assembly.
• Code style. ??
• Tool choice. GCC, nasm and make.
• Revision control. Git & GitHub.
• License. WTFPL? However, IANAL so maybe new BSD might suit better? Surely not GPL, though. ??
• TABS vs SPACES! ??

For code style, I'd propose something along lines of this (up to 120 chars per line; that's how much is visible on Github without scrolling bar):

Code: Select all

void some_magic_function(int a, int * b)
{   <- you can quickly find the pair of {} even without editor highlighting them
    *b = a;

    if (a == 111 && some_global == 100) <- that's how we write in English, we shouldn't change that - spaces around operators for readability, no spaces after (/before ) for Englishness, space after if - if is not a function call
    {
        some_global++;
    }
}

For assembly:

Code: Select all

jmp     .boot
.boot:
    mov     eax, 1

(not much to be shown; just notice I made first column to be 8 char wide - it's easily configurable in every text editor to handle that; see below for tabs themselves).
Tools - I'd go for Clang, but they are mostly compatible. For assembler - YASM is more recent, let's go for it.
License - WTFPL is... well, I would prefer zlib, after all
Spaces. Simple reason - spaces mix well with spaces. Tabs do not.

As for other posts, like design ones, I'll look closely at them later; just woken up and wanted to comment on this one.

[shikhin]

Hi,

Some people were questioning the need to replace/rewrite something along the lines of multiboot(2). This should provide all your answers...

You should not rely on custom MBR as part of the boot sequence. AFAIK It will be overwritten by installing Windows, FreeBSD and whatnot.

That is a problem. We could squeeze in all disk-functions inside the VBR, but that'd be a disaster. IMHO, getting replaced by Windows isn't that much of a terror - but for some user, it might be...

Any solutions?

For the API, due to the limited space I suggest to squeeze a bit out of it:
There is only open and read (ie. no close), an example interface (from my boot code) would then look like this:

Code: Select all

; HANDLE OpenFile(const char* filename);
; INPUT
;    si -> filename
; OUTPUT
;    eax = HANLDE of file, 0 if not found

; HANDLE ReadFile(HANDLE handle);
; INPUT
;    eax = HANLDE of file
;    di -> Data Buffer (must be big enough)
; OUTPUT
;    eax = HANLDE of file (May be changed), 0 if end of file
;    ecx = bytes read

Note  : Assume CS,DS,ES are zero
For example, on FAT, EAX is used as cluster number internally, and each read return a cluster-size data.[/quote]

Seems good not to have any close(). Another way to simplify it might be to have a Open and a Close. The Open wouldn't bother with returning any handles - there'd be only one open file at once. Of course... that might be too simple. We can always change the implementation.

[quote="bluemoon"]For SPACE vs TAB, this can never end, so I would suggest just let the code formatter do the job (manually or automatically).[/quote]

Yes, but I'll just recommend spaces, internally (since that's what the code formatter would be doing, in any case).

[quote="Griwes"]For code style, I'd propose something along lines of this (up to 120 chars per line; that's how much is visible on Github without scrolling bar):
[code]
void some_magic_function(int a, int * b)
{   <- you can quickly find the pair of {} even without editor highlighting them
    *b = a;

    if (a == 111 && some_global == 100) <- that's how we write in English, we shouldn't change that - spaces around operators for readability, no spaces after (/before ) for Englishness, space after if - if is not a function call
    {
        some_global++;
    }
}

For assembly:

Code: Select all

jmp     .boot
.boot:
    mov     eax, 1

(not much to be shown; just notice I made first column to be 8 char wide - it's easily configurable in every text editor to handle that; see below for tabs themselves).
Tools - I'd go for Clang, but they are mostly compatible. For assembler - YASM is more recent, let's go for it.
License - WTFPL is... well, I would prefer zlib, after all
Spaces. Simple reason - spaces mix well with spaces. Tabs do not.

As for other posts, like design ones, I'll look closely at them later; just woken up and wanted to comment on this one.

For the code style, I'd want to recommend two changes. Firstly, there shouldn't be any limit as to the number of characters per line. No sane programmer would have any more than ~90 lines, unless it's absolutely necessary. Then also, I think one can make his/her own decisions. Therefore, no limit should be necessary.

Moreover, there should be no space preceding a (. I know it makes it more English compliant (if that's what you want to call it), but I think if(something) & foo(something) is better than the unnecessary whitespace.

For tools, I'd go for gcc. More widely supported. If YASM is roughly equivalent to NASM, then okay.

As for license, zlib seems good enough. And yeah, spaces, ftw!

Regards,
Shikhin

[Owen]

• Language choice. C and intel-style assembly.

Seems reasonable. As much as I prefer C++, C is likely to be less controversial

• Code style. ??

The One true brace variant of K&R style? Matching the style of the language standard seems like a reasonable approach.
4 space indents? (They seem the most common)

• Tool choice. GCC, nasm and make.

Or compatibles; in particular, Clang

For non-x86 platforms, GAS is pretty much the universal standard

The only issue (long term) could be make, but that's probably something to deal with as it comes

I may be biased, but suggest adopting PDCLib as the base C runtime library. Porting it is easy (And I'm more than happy to do that side of things, anyhow)

• Revision control. Git & GitHub.

I'm partial to Hg & Bitbucket (Hg has better branch tracking), but it's much of a muchness.

• License. WTFPL? However, IANAL so maybe new BSD might suit better? Surely not GPL, though. ??

For permissively licensed stuff, I'm partial to the ISC license, as a BSD-alike which is significantly shorter

• TABS vs SPACES! ??

SPACES! FOUR OF THEM!
[/list]

For contact, I'd much rather suggest a mailing list than a single E-Mail address. I have a mailman instance setup (at http://lists.e43.eu) and could quite easily set up one or more lists.

For issue tracking/wiki support: Github's issue tracker is functional... but not great. For PDCLib, I'm using a JIRA instance that Atlassian are graciously hosting for free. I'd be more than happy to setup a tracker & wiki there. JIRA can integrate pretty well with both GitHub and BitBucket. Relatedly, the Confluence wiki is quite simply fantastic.



File system drivers: To me, it makes sense for them to be C code, so they're portable. This also means that, on UEFI, the "file system driver" can be a simple pass through to the underlying UEFI FS APIs. It probably makes sense for the "FS driver interface" to be a slightly extended version of PDCLib's file ops backend structure (i.e. extended with directory functions/open)

[egos]

Removed.

[shikhin]

Hi,

• Code style. ??

The One true brace variant of K&R style? Matching the style of the language standard seems like a reasonable approach.
4 space indents? (They seem the most common)

One line per brace seems best to me. Also, compulsory braces for one line if() blocks.

• Tool choice. GCC, nasm and make.

Or compatibles; in particular, Clang.

For non-x86 platforms, GAS is pretty much the universal standard.

The only issue (long term) could be make, but that's probably something to deal with as it comes

Since we're only x86, YASM seems good. Compatibles like Clang are okay, and make is the most widely used thing I've found yet.

I may be biased, but suggest adopting PDCLib as the base C runtime library. Porting it is easy (And I'm more than happy to do that side of things, anyhow)

How's the status, again?

• License. WTFPL? However, IANAL so maybe new BSD might suit better? Surely not GPL, though. ??

For permissively licensed stuff, I'm partial to the ISC license, as a BSD-alike which is significantly shorter

Or zlib. Argh, so many options.

For contact, I'd much rather suggest a mailing list than a single E-Mail address. I have a mailman instance setup (at http://lists.e43.eu) and could quite easily set up one or more lists.

Sure, seems like a good option.

For issue tracking/wiki support: Github's issue tracker is functional... but not great. For PDCLib, I'm using a JIRA instance that Atlassian are graciously hosting for free. I'd be more than happy to setup a tracker & wiki there. JIRA can integrate pretty well with both GitHub and BitBucket. Relatedly, the Confluence wiki is quite simply fantastic.

I have no problem using them, though I find no problem with GitHub's wiki & issue tracker (and having everything at one place... makes sense).

File system drivers: To me, it makes sense for them to be C code, so they're portable. This also means that, on UEFI, the "file system driver" can be a simple pass through to the underlying UEFI FS APIs. It probably makes sense for the "FS driver interface" to be a slightly extended version of PDCLib's file ops backend structure (i.e. extended with directory functions/open)

More like a slightly limited version of that.

Anyway, couldn't help but notice all the references to PDCLib.

Regards,
Shikhin

[Owen]

Hi,

• Code style. ??

The One true brace variant of K&R style? Matching the style of the language standard seems like a reasonable approach.
4 space indents? (They seem the most common)

One line per brace seems best to me. Also, compulsory braces for one line if() blocks.

It's all a matter of taste; I've always favored K&R. Things like

Code: Select all

}
else
{

feel like gratuitous wastes of space, but, as with all matters of style... it's a matter of taste

Since we're only x86, YASM seems good. Compatibles like Clang are okay, and make is the most widely used thing I've found yet.

EFI is available on non-x86 platforms; and I see no reason to limit things to just x86.

PC-BIOS and UEFI are certainly worthy first platforms, but in the long run I'd like to see the capability to support others (e.g. ARM). That said, what I was saying is that GAS seems like the best option for those platforms.

I may be biased, but suggest adopting PDCLib as the base C runtime library. Porting it is easy (And I'm more than happy to do that side of things, anyhow)

How's the status, again?

Pretty much everything you'd need in a bootloader is implemented. If it isn't... well, file a ticket

For contact, I'd much rather suggest a mailing list than a single E-Mail address. I have a mailman instance setup (at http://lists.e43.eu) and could quite easily set up one or more lists.

Sure, seems like a good option.

For issue tracking/wiki support: Github's issue tracker is functional... but not great. For PDCLib, I'm using a JIRA instance that Atlassian are graciously hosting for free. I'd be more than happy to setup a tracker & wiki there. JIRA can integrate pretty well with both GitHub and BitBucket. Relatedly, the Confluence wiki is quite simply fantastic.

I have no problem using them, though I find no problem with GitHub's wiki & issue tracker (and having everything at one place... makes sense).

Main issue I have with GitHub's tracker is the lack of support for components and priorities, plus the very limited version system.

It can work... but it isn't the best

File system drivers: To me, it makes sense for them to be C code, so they're portable. This also means that, on UEFI, the "file system driver" can be a simple pass through to the underlying UEFI FS APIs. It probably makes sense for the "FS driver interface" to be a slightly extended version of PDCLib's file ops backend structure (i.e. extended with directory functions/open)

More like a slightly limited version of that.

That interface (the specialist - and completely optional - wide character functions aside) is about as minimal as it can practically get; one can't really trim it down much more than read, write, seek and close. For read only drivers, it would of course suffice to make the write function return false.

Anyway, couldn't help but notice all the references to PDCLib.

It's pretty much my main open source/OS Dev contribution these days; it's where I have the most practical experience with things.

[shikhin]

Hi,

It's all a matter of taste.

It's decided, then?

SEFI is available on non-x86 platforms; and I see no reason to limit things to just x86.

PC-BIOS and UEFI are certainly worthy first platforms, but in the long run I'd like to see the capability to support others (e.g. ARM). That said, what I was saying is that GAS seems like the best option for those platforms.

Noted.

That interface (the specialist - and completely optional - wide character functions aside) is about as minimal as it can practically get; one can't really trim it down much more than read, write, seek and close. For read only drivers, it would of course suffice to make the write function return false.

The point is... you wouldn't (rather shouldn't) need seek & write for a bootloader's fs driver. Even close can be trimmed out (though open is always needed).

It's pretty much my main open source/OS Dev contribution these days; it's where I have the most practical experience with things.

Afair, Solar handed it to you before he stepped down? How long has that been?

Regards,
Shikhin

[Combuster]

Well, if people are opting for WTFPL and their kin, why not stick with the CC0 license - after all it is the license of PDCLib and you can't quite get more free than that.

As for nasm/yasm - the main difference that's going to bite you is that nasm resolves path references relative to the working directory and yasm does that relative to the source code's path.


One other thing that can haunt people is the file format of the later stages (read: native windows compiler issues vs cross compiler issues vs possibly llvm). IMHO care has to be taken that unlike GRUB, compilation works out of the box on any host platform, even if it's just instructions to get the proper compiler.

That'd be my $.02. I'd contribute code if I had time, intent, and an actual plan. Maybe I'll do an SFS driver sometime to promote some other community standard. But for now, I'll just be here sitting out the bureaucratic part .

[Kazinsal]

I'm more than happy to help out once we figure out how we're going to start collaborating on everything.

I vote in favour of the OTBS, by the way!

[b.zaar]

Hi, I'd like to be a part of this project.

I haven't had time to read all 6 pages of the thread yet so I'm not sure if this has already been covered. First I would like to offer scboot as a MBR/Boot sector that works on both HDD and FDD.

I like the idea of a boot loader to use modules and a simple scripting language, 1 that already exists like BASIC or Javascript. Not strictly necessary but the modules could be zipped on the disk making the boot partition/directory smaller and the number of sectors read minimal, also network booting would require less packets.

Another idea I had a while ago would be to extend BIOS like DPMI used to for DOS. This way we could make an API that works with a similar interface for both BIOS legacy hardware and UEFI compatible hardware. A custom boot stage or kernel could then configure and use the system from PMODE using system calls to the boot loader. The boot loader would then look like a simple I/O library to the loaded kernel and would reside fully within the 1st megabyte. The kernel could use this library while running so the boot loader is permanently resident. This would ease early kernel development of newer operating system but could be ignored for mature operating systems.

[Owen]

I like the idea of a boot loader to use modules and a simple scripting language, 1 that already exists like BASIC or Javascript. Not strictly necessary but the modules could be zipped on the disk making the boot partition/directory smaller and the number of sectors read minimal, also network booting would require less packets.

An integrated command interpreter at least seems almost essential (for the purposes of user discovery/experimentation, plus recovery). My inclination would be towards something like Jim TCL, because that kills both the scripting and interpreter birds with one stone.

With that said, I'd like to see something which avoids the mess of scripts that Linux distros use to generate grub.conf. A good system, in my opinion, would be one where all the boot options are defined in separate files (how they are defined to be determined), and the only common file which the OSes have to deal with is a simple one which defines properties like "defaultboot" and "timeout". This should be a simple, well understood, easy to manipulate format

(Perhaps also "nextboot", so that they can perform specialized actions on next reboot; for example, booting into the OS after a successful install)

Another idea I had a while ago would be to extend BIOS like DPMI used to for DOS. This way we could make an API that works with a similar interface for both BIOS legacy hardware and UEFI compatible hardware. A custom boot stage or kernel could then configure and use the system from PMODE using system calls to the boot loader. The boot loader would then look like a simple I/O library to the loaded kernel and would reside fully within the 1st megabyte. The kernel could use this library while running so the boot loader is permanently resident. This would ease early kernel development of newer operating system but could be ignored for mature operating systems.

Every OS has its own needs - while a common boot loader can meet 95%+ of them, a common "OS abstraction layer probably can't.

More importantly, the number of services that UEFI provides after the ExitBootService call is very limited, and UEFI allows no control over where you are loaded (or even what physical RAM is free)

[dozniak]

The only issue (long term) could be make, but that's probably something to deal with as it comes

Use cmake from the start, then you won't have any make related problems.

[dozniak]

License:

BOOST license has a well-written rationale, explaining why it is better. Done by a lawyer. So it is seriously balanced between being as short as possible and as applicable legally as possible.

[Brendan]

Hi,

I like the idea of a boot loader to use modules and a simple scripting language, 1 that already exists like BASIC or Javascript. Not strictly necessary but the modules could be zipped on the disk making the boot partition/directory smaller and the number of sectors read minimal, also network booting would require less packets.

An integrated command interpreter at least seems almost essential (for the purposes of user discovery/experimentation, plus recovery). My inclination would be towards something like Jim TCL, because that kills both the scripting and interpreter birds with one stone.

A boot loader (that is not a "boot manager") should have no use for any scripting language.

Get a physical memory map, load some files, find some ACPI tables, setup a video mode, then pass control to the first loaded file. You'd need something to determine which video mode/s are supported (maybe some field's in the first file's header) and something to determine which files to load from where (e.g. "starting LBA" and "number of sectors" fields in the boot loader itself, that are set when the boot loader is installed).

Of course it's not quite that simple (e.g. you'd want to collect a bit more information than that, and have a standardised way to pass all the information to the first loaded file). However, if you start adding all sorts of unnecessary complexity (scripting languages? modules? a C library? Why not slap a Tetris clone in it in case the user gets bored? No, how about a web browser! Yeah, with flash; and java too so people can play Minecraft! Woot), then you'll end up with something like GRUB2 instead of something clean and efficient.


Cheers,

Brendan