How do I load my sector with if it's more than 512 bytes?

[LtG]

they also allow the boot loader proper (as opposed to the boot sector) to be anywhere in the file system, provided the boot sector can find at least enough of the boot loader to load and execute it - as seen with MS-DOS and FAT. So long as the FAT boot loader for an MS-DOS system can find the first entry of the FAT, it can find the start of IO.SYS - and that's all it needed to do. There are several other ways to do this, such as actually having other reserved sectors containing a second stage boot loader capable of actually parsing a FAT, so how you do it is a matter of choice.

I don't have much time, so I'm going to be brief.

The crux of the problem (for bootloaders) is that you have to store a pointer, in the bootloader itself, to the second (next) stage. In the FAT explanation you gave that would be a hardcoded "first (and second) entry in FAT". Personally I'm not a huge fan of that idea because now you are leaking bootloader stuff into the FS proper.

The FS should not (ever) care about bootloaders. It's a FS, and it should do its job the best it can, without caring about anyone or anything else.

The only "trick" that makes sense to me (because I haven't come up with anything better) is the simple fact that once you've installed your OS on a particular "partition" you've essentially also installed on a particular media. Knowing the particular media you can have media dependent bootloader and it can take advantage of "known facts". Which possibly include fixed starting location for bootloader sectors.

The simple fact is that bootloader runs 0.0001% of the time and 99.9999% of the time the FS proper is being used. Allowing the FS the most freedom to do its job is paramount. In practice for most (if not all) FS types this distinction might not make much of a difference, but at least I'm designing my own FS from the "perspective" of not caring at all about bootloader/bootsector and contain the bootstrap problem into the loader.

Hopefully the above makes some sense =)

[Schol-R-LEA]

The crux of the problem (for bootloaders) is that you have to store a pointer, in the bootloader itself, to the second (next) stage. In the FAT explanation you gave that would be a hardcoded "first (and second) entry in FAT". Personally I'm not a huge fan of that idea because now you are leaking bootloader stuff into the FS proper.

The FS should not (ever) care about bootloaders. It's a FS, and it should do its job the best it can, without caring about anyone or anything else.

While I don't disagree, in my explanation I was primarily focusing on how MS-DOS actually did this, rather than on judging if it was a good approach or not. The truth is, there really are no good approaches when it comes to the PC standard, because the PC standard was designed for a bygone era when booting from cassettes or floppy disks was the norm for personal computers (the original IBM PC had a built-in cassette port, and AFAICT the only type of disks the BIOS supported was 5 1/4" double-sided, double-density floppies with a 360KB formatted capacity), and no one thought twice about mixing things like that. It is only with the benefit of hindsight that this seem a bad idea - even in the mainframe world, this wasn't that uncommon an issue at the time.

It is unfortunate, I agree, but it is what we are stuck with when talking about PC compatible systems. For hard drives and more modern data storage such as SSDs, we have a few better options, paramount of which being to set aside a small, separate partition for the boot loader; AFAICT this is what Windows now does (the NTLDR partition), and it is also one of the options for GRUB. However, floppies don't have MBRs or partition tables.

The simple fact is that bootloader runs 0.0001% of the time and 99.9999% of the time the FS proper is being used. Allowing the FS the most freedom to do its job is paramount. In practice for most (if not all) FS types this distinction might not make much of a difference, but at least I'm designing my own FS from the "perspective" of not caring at all about bootloader/bootsector and contain the bootstrap problem into the loader.

That's the part that is frustrating to me and many others here, because writing a boot loader is not part of writing an operating system - it's a distraction from it. Too many newcomers assume that they need to write one, but the fact is that there is nothing to be gained, or learned, from doing it.

For an OS developer, writing your own boot loader is a purely a waste of time and effort.

Conversely, if writing a boot loader is your primary goal, you'd be better off studying the Multiboot Spec and how to implement that rather than working with floppy disk images and FAT file systems.

While I do intend to do just that, I don't see it as part of my OS work at all - Bereshith is a completely separate project only tangentially related to either Kether OS or the Kelephstis hypervisor.

I am willing to help the OP, despite this, but... well, I have no doubt at all that this is something that will be regretted as lost effort later.

The insistence on 'writing everything yourself' is false fire - it isn't possible, because you still need to use the BIOS for critical parts of the boot sequence, and by the time your code starts, a lot of other peoples' code had already run. And yes, a fair amount of some modern BIOSes - especially UEFI ones - is written in C, albeit with a compiler that can target real mode (which GCC can't).

My goal with VERBUM was to give enough information to show how the loader works, and dissuade others from writing one once I'd made it clear that, no, it is not something you need to do or even should be doing as part of an OS project. Embarrassing mistakes aside, it is clear that I failed at that.

TL;DR: Boot loader != operating system.

[simeonz]

The FS should not (ever) care about bootloaders. It's a FS, and it should do its job the best it can, without caring about anyone or anything else.

This assumes that all space allocation "outside" the FS is dealt with by partitioning, which is comparatively inflexible. Partitioning must either be used liberally to cover the worst case requirements or must not be used at all. In which case all space allocations must exist in the FS and be accessed through a complete FS driver implementation.

Let's assume that I want to store the kernel and the initial ram disk images somewhere where the boot code can find them. I will skip the stage1.5/stage2 aspects, not because they are irrelevant, but because they factor into that somewhat recursively.

• I can use a system partition of some kind that uses contiguous storage, which will require less effort for the boot loader. The OS must know how to maintain it after kernel updates (which is ok), and the user will need some kind of FS-like driver to access the data for direct troubleshooting and inspection. The space must be allocated liberally, as I said before, but this may not present an actual issue considering the usual kernel size. It may be possible to combine this with facility that rebalances the device space with the other partitions on the fly (using FS defragmentation and metadata fixup if necessary.)
• I can use some simpler older FS for the purposes of booting, like FAT. This will still require a dedicated partition and is similar to the situation above, except that I am reusing existing FS design. This saves me the effort of writing new OS driver code.
• I can have FS drivers inside my boot code that can read-only mount any filesystem that I support. This is the GRUB approach. I do not favor it, because it implements a restricted booting OS, which duplicates some of the main OS functionality and may work with inferior driver code. I also find this approach unprincipled, because loading the actual FS drivers is among the last things that the OS does. We use a ram disk to be able to load the important device drivers and initialize the hardware first, but we still have a separate read-only FS driver implemented in our bootloader. It doesn't make sense to me. Then there is the issue with handling journals in order to perform lookups from consistent state when we try to find the system files.
• Provide some means for allocating space externally inside the file system and expose allocation data in trivial to parse format. This will indeed blur the FS responsibilities, but allows sharing space more flexibly without the need for repartitioning. But I acknowledge that it is unclean, because the offered facilities may not offer portable solution to the booting problem.
• The above can instead be implemented by a logical volume remapping scheme such as LVM (i.e. such as the Linux kernel device mapper layer), which will abstract the space allocation details away from the FS, but will make the data inaccessible for hosts that cannot parse the volume structure. The boot code will be offered means to parse the location of a logical volume instead of partition, which can be grown without repartitioning.

Looking at this list, the first approach actually looks the simplest.

[Schol-R-LEA]

Actually, the simplest solution for most of us is to drop legacy BIOS entirely and use the built-in UEFI manager, which will let you go straight to p-mode or long-mode without anything else on the drive. However, while almost all newer motherboards from after, say, 2010 provide this option, not everyone here has a newer system.

Also, I can't see Intel dragging out support for real-mode much longer, even with desktop and server chips. They want to get rid of it, and so does almost everyone else. (Of course, the same is true of x86 itself, and Intel still have it around their necks like the proverbial albatross despite repeated attempts to put a stake through its heart - sorry for the mixed metaphor - so I may be wrong...)

IOW, writing your own MS-DOS style boot loader probably won't even be an option soon, something I am sure will elicit wailing and gnashing of teeth from the Tildes of the world.

For the rest of us, it will be a huge relief, assuming that we can still unlock the UEFI Secure Boot loader settings.

Or you could drop conventional PCs outright and go with OS dev targeting a system built on maker boards. I mention this only because my RPi3B should be arriving some time today, and I personally would rather be using system based on ARM, or (better still) MIPS, or RISC V (well, if any hard-silicon implementations are ever built), or (possibly best of all, though obviously no one knows yet) Mill, rather than x86, anyway. But that's just me. For the foreseeable future, I will still have my (pre-UEFI) Thinkpad, but given that I am already using Linux and intend to use that Pi for my first version of the Alexia kernel experiment, well...

[zaval]

Conversely, if writing a boot loader is your primary goal, you'd be better off studying the Multiboot Spec and how to implement that rather than working with floppy disk images and FAT file systems.

Speaking of time wastage, I'd say that messing around with a non-standard half-assed "Spec" as "Multiboot" would be a time wastage.
Developers are better off to align with real standards.
Floppies are fossiles, right, but FAT is not. It's like a crocodile which is more ancient than dinosaurs, still is well alive and flourishing. For boot loaders/firmwares its advantage is in its simplicity and efficiency. It would not be a time wastage to learn it and implement for ones wishing to write their own bootloaders. And of course, it's a very good FS starting point for OS developers, both because of its easiness and because it's incorprated in modern firmware and storage standards like UEFI, SD. It's totally not like throwbacking into 16-bit real mode x86 oddities, DOS, C64, "cooperative multitasking", etc. FAT is widely, very widely being used it makes it relevant.
As of writing their own bootloaders. Of course it's kind of true, that making it is like a wastage of time, from the OS development perspective. On the other hand, OS development self is the same.) If to be strictly practical. But it's all a "hobby" and "passion". So maybe if one wants, let him/her do it. It's his/her choice.

Actually, the simplest solution for most of us is to drop legacy BIOS entirely and use the built-in UEFI manager,

if I ever come to the x86 port, for my OS project, this will be the plan. Even though from 3 x86 machine i have now, only a laptop has it (marked as an experimental feature, 2011 made btw).
It's because, I believe, that most ethusiastic projects even being interesting for their creators only, still should look forward not past.

[LtG]

The truth is, there really are no good approaches when it comes to the PC standard, because the PC standard was designed for a bygone

While I agree with almost everything you said, this just isn't true. If you look at what I said from an even wider scope than the PC-platform, it's still true. I mean, from a computer perspective, you only have one option and that's "embedded starting sector" or any derivation thereof. The end result is essentially the same. The earliest part of the bootstrap needs to have hardcoded parts, the type of pointers those are (sector numbers or understanding a FS and "knowing" a file name, etc) isn't really relevant and can't be changed (AFAIK). So you still need the "pointer". Certainly you could try to make a "better" bootstrap system, but if you could, what would it be? And why couldn't you do that already now?

That's the part that is frustrating to me and many others here, because writing a boot loader is not part of writing an operating system - it's a distraction from it. Too many newcomers assume that they need to write one, but the fact is that there is nothing to be gained, or learned, from doing it.

For an OS developer, writing your own boot loader is a purely a waste of time and effort.

Conversely, if writing a boot loader is your primary goal, you'd be better off studying the Multiboot Spec and how to implement that rather than working with floppy disk images and FAT file systems.

While I do intend to do just that, I don't see it as part of my OS work at all - Bereshith is a completely separate project only tangentially related to either Kether OS or the Kelephstis hypervisor.

I am willing to help the OP, despite this, but... well, I have no doubt at all that this is something that will be regretted as lost effort later.

The insistence on 'writing everything yourself' is false fire - it isn't possible, because you still need to use the BIOS for critical parts of the boot sequence, and by the time your code starts, a lot of other peoples' code had already run.

[nitpick]
You _can_ write all of it yourself, but seriously I doubt anyone here _actually_ will.

In practice it depends how far you want to go, not just the platform (PC), "write" your own CPU, and of course your own language, which also means your own compiler, etc. It _could_ be done, but again, seriously?

I've seen some people here mention 10-20 years to do a "proper" OS, add to that a new language, compiler and CPU design...
[/nitpick]

So, yes, I agree, writing a bootloader is an absolute waste of time unless you want to replace GRUB. Bootloaders have nothing to do with os design or development and there's almost nothing (like 0.001% or less) to be learned from writing them. All you learn is a few BIOS routines (which a "real" OS (in prot-mode) can't use) and a few real mode things (which a "real" OS (in prot mode) can't use).

[LtG]

The FS should not (ever) care about bootloaders. It's a FS, and it should do its job the best it can, without caring about anyone or anything else.

This assumes that all space allocation "outside" the FS is dealt with by partitioning, which is comparatively inflexible. Partitioning must either be used liberally to cover the worst case requirements or must not be used at all. In which case all space allocations must exist in the FS and be accessed through a complete FS driver implementation.

Let's assume that I want to store the kernel and the initial ram disk images somewhere where the boot code can find them. I will skip the stage1.5/stage2 aspects, not because they are irrelevant, but because they factor into that somewhat recursively.

• I can use a system partition of some kind that uses contiguous storage, which will require less effort for the boot loader. The OS must know how to maintain it after kernel updates (which is ok), and the user will need some kind of FS-like driver to access the data for direct troubleshooting and inspection. The space must be allocated liberally, as I said before, but this may not present an actual issue considering the usual kernel size. It may be possible to combine this with facility that rebalances the device space with the other partitions on the fly (using FS defragmentation and metadata fixup if necessary.)
  .
  .
  .
  Looking at this list, the first approach actually looks the simplest.

As I said to Schol-R-LEA, I was talking in more abstract terms... But let's look at what I said and what you said..

Based on your last sentence, you seem to agree with me? That the allocation for bootstrap (boot stuff) should be outside of the "proper" FS?

But in even more abstract terms, I don't think you even have a choice. The MS-DOS way of using a "special" file which has to be in "special" location is essentially the exact same solution. The file ends up being inside the FS but has to be handler specially and thus isn't really within the FS. Or maybe rather it uses allocation space within the FS "proper" but isn't a normal file and the as such the files "meta-data" isn't within the FS "proper" because it needs special consideration.

My main point is that if you want something out of the FS you need FS "driver", if you don't want to include that in your bootstrap (because of 512B MBR/bootsector restriction or because you don't want bootstrap to be "tainted" (non-agnostic towards FS)) all you end up with is essentially creating a new "bootstrap FS" that does it for you. And that's fine, that's essentially what UEFI does.

I think if you look at all the options you presented they can all be boiled down to that. But if you do think one of them doesn't, please let me know. I'm still a bit pressed for time =/

[LtG]

Or you could drop conventional PCs outright and go with OS dev targeting a system built on maker boards. I mention this only because my RPi3B should be arriving some time today, and I personally would rather be using system based on ARM, or (better still) MIPS, or RISC V (well, if any hard-silicon implementations are ever built), or (possibly best of all, though obviously no one knows yet) Mill, rather than x86, anyway. But that's just me. For the foreseeable future, I will still have my (pre-UEFI) Thinkpad, but given that I am already using Linux and intend to use that Pi for my first version of the Alexia kernel experiment, well...

Nothing against non-conventional, but for desktop/server I can't really see why anyone would _prefer_ inferior products (Rpi, ARM, MIPS, RISC, etc)? It's not like their bootstrap is any better (it really can't be, you can get rid of some of the PC-platform peculiarities but you'll just end up with another set of them, and less well documented in most cases), and if you do "proper" coding you don't do it in assembly, so there's no issue there either.

They way I see it, an OS should not be written in C, instead we should all do what they did, create our own language to write our OS's. Of course due to practicality and due to C being "good enough" I do find it reasonable to use C (as I use it too). So the hardware level language is irrelevant for the most part, the OS should be an abstraction on C (or higher) level and at that point very few CPU's compete with x86. The only real competition is in really small form factor (Rpi etc) and in low power (mobiles phones/tablets). If you want performance (as in replacement for desktop/laptop/server) then x86 is still the only reasonable solution.

[simeonz]

Actually, the simplest solution for most of us is to drop legacy BIOS entirely and use the built-in UEFI manager, which will let you go straight to p-mode or long-mode without anything else on the drive.

That is a good prospect in the technical sense, and I personally agree that it solves the problem. However, there may be other considerations, like problems with secure boot. This is essentially non-technological issue. In that regard it is similar to other phenomena, like Windows driver signing and virtual secure mode, Intel software guard extensions, application stores even. All of these phenomena provide big companies with the ability to get early move on their competitors, etc. I know that this is the most paranoid way to think about it, but in relation to those aspects UEFI imprints a somewhat negative image. But this is kind of out of topic.

[LtG]

Conversely, if writing a boot loader is your primary goal, you'd be better off studying the Multiboot Spec and how to implement that rather than working with floppy disk images and FAT file systems.

Speaking of time wastage, I'd say that messing around with a non-standard half-assed "Spec" as "Multiboot" would be a time wastage.
Developers are better off to align with real standards.
Floppies are fossiles, right, but FAT is not. It's like a crocodile which is more ancient than dinosaurs, still is well alive and flourishing. For boot loaders/firmwares its advantage is in its simplicity and efficiency.

Multiboot is a real spec, whether you like it or note and whether or not it's "half-assed". Aligning with FAT provides no real benefits w.r.t. bootstrap.

Apart from "mobile" storage (floppies (seriously?) and usb storage) nobody uses FAT. I don't think I've used FAT in over a decade, even Windows prefers NTFS. And by FAT, are you implying that FAT12/16/32 are all the same?

FAT is a fossil, and unfortunately as bad as x86 at that, it refuses to die. While FOSS won't likely accept NTFS as well, FAT is still bad. We'd need something better in the middle, but there isn't anything so FAT survives, but not because it's good.

And if your plan is to do things "DOS" way and support FAT so you can use MS bootstrap, go for it, but it's just as arbitrary (and makes requirements of the OS) as GRUB, so I don't see any benefit. The way I see it, the second-last stage has to be your own, it gather data from the previous stage(s) and provides them to your OS/kernel in your format. So it's largely irrelevant whether you use GRUB/MS MBR/etc.

I find using GRUB is easiest, I still use my own next stage which does all the "dirty work"/"bootsrap" for my OS, so I can easily replace anytime I want =)

[simeonz]

My main point is that if you want something out of the FS you need FS "driver", if you don't want to include that in your bootstrap (because of 512B MBR/bootsector restriction or because you don't want bootstrap to be "tainted" (non-agnostic towards FS)) all you end up with is essentially creating a new "bootstrap FS" that does it for you. And that's fine, that's essentially what UEFI does.

Well, to paraphrase myself - either the bootloader must understand the FS storage format, the FS must understand the bootloader storage format, or the two must live completely independently (i.e. on different partitions). And from those options I am reluctant to make the bootloader understand the FS, because of code duplication and the difficulty of troubleshooting it. If I make multiple custom FSes with difficult to implement features (unlikely, but lets imagine for a while), would I like to write two sets drivers for them or just one. If I could unify the driver code somehow, to work in any execution environment, it would be a different story, but this will be probably difficult.

I suppose I trying to explore the argument that having more complexity in the FS and less complexity in the bootloader may be justified. But this is a design choice.

[azblue]

How do I load a kernel that is more THAN 512 BYTES?

Multiple people have answered this question multiple times, in multiple different ways, I provided you with code. If you're not just a troll, then you're going to have to explain specifically what it is you're not understanding with the answers you've already been given.

[neon]

Hello,

I would just recommend using an existing boot sector code for now. I know, you don't want to use someone else's code. You can use that code to load your actual boot loader ("stage 2") which you can then play with to load your kernel. I am saying this for your sanity -- in a boot sector, you only have about 350 bytes to work with. Besides, you can always go back and rewrite it later so its your own code. It is a lot easier to learn how to parse the file system and write test code when you don't have to worry about code size.

If you don't want to go this route, then you are pretty much on your own here. You have all the information presented about the file system already in this thread including sample code. You'll either get it or not -- there is no short cut. Working code and sample code isn't going to help you if you aren't willing to read what we provide you. If you have any specific questions or doubts about something that you are not sure about, feel free to ask. However repeating the same question over and over while ignoring (and rejecting some) answers isn't going to get you anywhere.

[Schol-R-LEA]

I think we need to re-focus. @stevewoods1986 has repeatedly stated what he wants to do, and while we might disagree, it does no one any good to rehash these topics here. I only replied to LtG and SimeonZ earlier because I felt my point needed to be made, but I was speaking to them, not Steve.

Seriously, we've spent over a page and a half arguing with each other while the OP wasn't even available to reply.

So, let's try this again. Steve, where do you stand with things now, what parts (if any) did our posts help with, what parts of either the posts or the wiki are unclear to you (I expect a lot of them, since, well, both are/were frequently unclear, period)?

And if you pardon my need for external validation, I was specifically wondering what, if anything, you thought of my explanation of x86 segmented memory here. I don't know if it was something you were actually having trouble with or not, but it was clear that I and most of the other posters were having some

Finally, have you set up an external source repo on some host such as GitHub or CloudForge? Aside from the obvious reasons for one, it would also make it easier for us to follow your progress if you gave us a link to one, rather than having to repeatedly post updates. Just a suggestion.

[stevewoods1986]

I think we need to re-focus. @stevewoods1986 has repeatedly stated what he wants to do, and while we might disagree, it does no one any good to rehash these topics here. I only replied to LtG and SimeonZ earlier because I felt my point needed to be made, but I was speaking to them, not Steve.

Seriously, we've spent over a page and a half arguing with each other while the OP wasn't even available to reply.

So, let's try this again. Steve, where do you stand with things now, what parts (if any) did our posts help with, what parts of either the posts or the wiki are unclear to you (I expect a lot of them, since, well, both are/were frequently unclear, period)?

And if you pardon my need for external validation, I was specifically wondering what, if anything, you thought of my explanation of x86 segmented memory here. I don't know if it was something you were actually having trouble with or not, but it was clear that I and most of the other posters were having some

Finally, have you set up an external source repo on some host such as GitHub or CloudForge? Aside from the obvious reasons for one, it would also make it easier for us to follow your progress if you gave us a link to one, rather than having to repeatedly post updates. Just a suggestion.

Well, the post suggesting C and GRUB. I think what Mike is saying could be worth a try. Now my next problem is integers