OSDev.org

Hello everybody.

I am in a process of creating my own bootloader, just for fun and because I've acquired some knowledge lately.
There is a thing that bothers me doe. How am I supposed to create a FAT 32 master boot record (aka MBR)? I am planning on using a virtual HDD. I need two primary partitions (each taking half of the space).
As far as I know I have 446 bytes for my stage 1 bootloader, 4 * 16 bytes (1 partition data chunk = 16 bytes) for the partitions and two extra bytes for the boot signature (0x55, 0xAA).
I think I have two distinct options:
1.Create it by myself, fill the first 446 bytes with executable code, figure out how partitions are coded, figure out where a BIOS parameter block resides, some magic.
2.Find a tool that can do all of the above but then how not to overwrite the previously created partition table??

What do you guys think? How would you do it? Do I really need the BPB, is it a waste of space?
I know that the stage one can easily be put on the hard drive but what about the second stage how am I supposed to put it inside a folder without being able to access the drive itself (inside Linux)?

Octacone wrote:How am I supposed to create a FAT 32 master boot record (aka MBR)?

You're not. You're supposed to have a generic (DOS-like) MBR, which then loads your FAT32 VBR. Doing things differently is a good way to break bad firmware.

Octacone wrote:How would you do it?

You can look at my code if it helps. As for actually getting the boot code on the disk, I do most of my development on Windows, so I use HxD for manually editing both disk images and physical disks.

Octacone wrote:Do I really need the BPB, is it a waste of space?

Your MBR should not have a BPB. Your VBR must have a BPB. Fortunately, FAT32 reserves some sectors for additional boot code, so even with the BPB you have more space than with FAT12 or FAT16 to find and load your second stage.

Octacone wrote:I know that the stage one can easily be put on the hard drive but what about the second stage how am I supposed to put it inside a folder without being able to access the drive itself (inside Linux)?

You can mount your disk image as a loop device in Linux. Once you do that, you can copy files to it just like any other disk.

Octocontrabass wrote:

Octacone wrote:How am I supposed to create a FAT 32 master boot record (aka MBR)?

You're not. You're supposed to have a generic (DOS-like) MBR, which then loads your FAT32 VBR. Doing things differently is a good way to break bad firmware.

Octacone wrote:How would you do it?

You can look at my code if it helps. As for actually getting the boot code on the disk, I do most of my development on Windows, so I use HxD for manually editing both disk images and physical disks.

Octacone wrote:Do I really need the BPB, is it a waste of space?

Your MBR should not have a BPB. Your VBR must have a BPB. Fortunately, FAT32 reserves some sectors for additional boot code, so even with the BPB you have more space than with FAT12 or FAT16 to find and load your second stage.

Octacone wrote:I know that the stage one can easily be put on the hard drive but what about the second stage how am I supposed to put it inside a folder without being able to access the drive itself (inside Linux)?

You can mount your disk image as a loop device in Linux. Once you do that, you can copy files to it just like any other disk.

Okay I think is am even more confused.
Where is a volume boot record actually located? Internet says at the beginning of the partition? But what partition? Am I supposed to execute it? Same applies when speaking of the BIOS Parameter Block.
There are so many images online and they aren't the same. Is this okay: (-> = then we have)
MBR(boot code + 4 partition entries (this thing tells us where is each partition located and how big it is) + 0xAA55) -> Partition Data 1 (BPB + VBR) -> Partition Data 2 (BPB + VBR)... -> Partition Data 4 (BPB + VBR).
So where does FAT put its "data"/"tables"? What does FAT create from the above? Why does Wiki tutorial on loop devices teach us to create a partition with fdisk and then we need to create a FAT 32 record again using mkdosfs?
Also why is there a tutorial that puts the BPB inside the MBR?

Octacone wrote:Where is a volume boot record actually located? Internet says at the beginning of the partition? But what partition?

The VBR is the first sector of every partition.

Only partitions you intend to boot need to have valid boot code in the VBR. Other partitions may have no boot code at all, or have code that displays an error message.

Octacone wrote:Am I supposed to execute it?

If the partition is marked active in the partition table, your MBR should load and execute the VBR. A more complex bootloader may also offer to boot other partitions aside from the one marked active. (For example, GRUB can do this.)

Octacone wrote:Same applies when speaking of the BIOS Parameter Block.

For FAT32 (and several other filesystems), the BPB is a required component of the VBR. Not all filesystems need a BPB, though some firmware will misbehave if the space normally used for the BPB contains something else.

Octacone wrote:So where does FAT put its "data"/"tables"?

The BPB contains several fields that are necessary to read and write a FAT filesystem. You can read Microsoft's FAT filesystem specification to see what's in the BPB and how it's used.

Octacone wrote:What does FAT create from the above?

Everything inside the partition is created for and used by the FAT filesystem.

Octacone wrote:Why does Wiki tutorial on loop devices teach us to create a partition with fdisk and then we need to create a FAT 32 record again using mkdosfs?

Because a partition can contain many different filesystems, and it would be impractical for a simple tool like fdisk to format the partitions it creates with every possible filesystem.

Octacone wrote:Also why is there a tutorial that puts the BPB inside the MBR?

Tutorials are often written by people with poor understanding of the topic. You shouldn't put too much trust into any tutorial related to OS development.

A MBR (Master Boot Record) is always at LBA 0, the first sector of the device, the sector the (Legacy) BIOS loads. As you know, this sector has a Partition Table of at least one and (usually) not more than four valid entries. These entries indicate how the device's media is divided into partitions. A Partition may be anywhere on the device. It is up to the partitioning utility to correctly and accurately partition the device so no partition overlaps and most/all media is used/partitioned.

The MBR can and usually does contain enough code and data to parse the Partition Table, looking for the active partition (first byte of entry), then loading the first sector of the found partition to address 0x07C00. Depending on whom you ask, there is either plenty of room or not near enough room in this first sector to accomplish this task. In past history, the first 63 sectors were usually used to boot a partition, given Boot Extenders, BIOS Extenders, etc.

A partition usually started at LBA 63, using most or all of the remaining media. If it was a large media device (couple hundred meg, which was large then), you may have one ore more partitions, using Extended Partition Entries.

Each of these partitions should be self contained. i.e.: If the partition starts at LBA 63 and ends at LBA 12345, the contained code/data/filesystem within it should not require any access to any LBA outside of these two limits.

If this partition is the active partition, the booted partition, it will have code within the first LBA of its partition (usually called an LSN (Linear Sector Number)), to continue loading the OS. This LSN 0 may contain a BPB, it may not, depending on the file system contained within this partition.

It is up to the MBR code to load this LSN 0 to 0x07C00, then the code at LSN 0 to load any remaining sectors (LSN 1 to LSN n) to load a second stage boot loader.

The problem comes when the code at LSN 0 doesn't know where it is located. For example, if LSN 0 is at LBA 63, when the code at LSN 0 sends a LBA value to the BIOS read service, how does it know this LBA value? This is where the BPB comes in. Within a FAT formatted BPB, there is a Hidden Sectors field that, in this case is 63, tells the code at LSN 0 where it is located. Not all file systems use this scheme. Ext2 for example.

To summarize, if you wish to write all of this from scratch, you will need a MBR code/data file that is 512 bytes exactly and most likely will never change. This MBR simply parses the Partition Table and loads the first sector of the found active partition. Period.

Then, you will need a first stage boot code file (called the VBR in previous posts) for each file system (and type) you want to support. This first stage code/data file can be 512 to n bytes in length, where as if it is more than one sector, your code must contain, within this first sector, all of the code and data to be able to load the remaining sectors of your first stage boot code. Then with all of the first stage boot code loaded, it will, if you wish, load a second stage loader.

Now, you can have the first stage boot code do all of the loading, but this is usually not the case.

As an example, my project has two different MBR/UEFI files. The first, the MBR code, is for legacy BIOSes and simply does what I explain above with a few extra perks. For example, it will parse extended partitions as well. The UEFI file, when booted from a legacy BIOS with a GPT at LBA 1, will parse the GPT and load the first sector of the found active partition. These two files seldom change and absolutely don't care what file system is on the partition it loaded, other than the System ID field in the MBR partition table entry.

I then have a first stage boot code/data file for each file system and their variants (FAT12, FAT16, FAT32) to load a single file from the said file system, called loader.sys. It is the job of this first stage code to simply parse the said file system and load the whole file. It also does a few other things needed for the loader.sys file to function correctly. Each of these source code/data files are (as expected) file system dependent and are all written in x86 assembly.

I then have a single loader.sys file that uses multiple source files, all but 5% written in C, to continue loading the OS. This loader.sys file's project files are written so that with a single #define, I can included or exclude a given file system. Therefore, I can compile this loader.sys file (the second stage boot code) to be included with any and all file systems (loader.sys > 90k) or compile it to be single file system dependent (loader.sys < 30k).

Does this give you an idea of what you can do? The most important thing you need to remember is that, as a hobbiest OS developer, you have full control after the POST passes control to address 0x07C00. You can do anything you want, formatting the hard drive (boot device) to any format you wish, loading the OS however you wish. The only thing you need to worry about is compatibility with firmware or other software/file systems.

Ben
http://www.fysnet.net/osdesign_book_series.htm

Thank you @Octocontrabass and @BenLunt so much. Now I know what to do. That was a really good explanation.
Ben looks like you have a beefy loader in there. I don't really need to care about GPT since I need to try this legacy stuff first.
The only thing I noticed is that I am supposed to load a VBR at 0x7C00 while my MBR is already loaded in there. Does that mean that I will have to do some relocation?
Why does fdisk only fill the first 512 bytes and nothing else? Even if I create two partitions there is no VBR or anything that would suggest me "oh that is a partition".
What changes does mkdosfs do to my HDD file?
One last question (the point of this topic actually), apart from the MBR code what else (not including the VBR code) am I supposed to make myself?
I don't know how to use those two because the Wiki tutorial is really outdated and fdisk doesn't show some stuff like it used to and thus I can't follow along.
Actually I would like to hear if there is a better method for creating an empty HDD file partitioning it and installing a FAT 32 filesystem on it.

Octacone wrote:The only thing I noticed is that I am supposed to load a VBR at 0x7C00 while my MBR is already loaded in there. Does that mean that I will have to do some relocation?

Depends on what you mean by relocation. You have to relocate your MBR code before you load the VBR to 0x07C00, yes. Have relocation elements within your code, in the sense of a linker, no?

Octacone wrote:Why does fdisk only fill the first 512 bytes and nothing else? Even if I create two partitions there is no VBR or anything that would suggest me "oh that is a partition".

FDISK only does the partitioning of the media, it doesn't have anything to do with the partition contents, only the locations of these partitions. It is up to you, or a formatting tool to do something with the partitions.

Octacone wrote:What changes does mkdosfs do to my HDD file?

What mkdosfs are you talking about? There is a Linux one, there is the one I use (http://www.fysnet.net/mtools.htm), there is the one I include with my book(s), there are many others.

Mine creates a hard drive image, skipping sectors for a MBR (if indicated), and then creates an empty FAT12, 16, or 32 file system within the partition, whether that be the whole image file, or one of the partitions you specify. The one included with my book will even add files to the image.

Octacone wrote:One last question (the point of this topic actually), apart from the MBR code what else (not including the VBR code) am I supposed to make myself? I don't know how to use those two because the Wiki tutorial is really outdated and fdisk doesn't show some stuff like it used to and thus I can't follow along. Actually I would like to hear if there is a better method for creating an empty HDD file partitioning it and installing a FAT 32 filesystem on it.

All of it or none of it. It all depends on what you wish to do, what you have time for, what you are interested in doing.

You can do it all, writing a MBR, writing a FDISK utility that will create the MBR, writing a VBR, writing a FORMAT utility that will create the VBR, writing the file system, writing a FORMAT utility that will format the file system, or you can use something like GRUB that will boot for you.

It all comes done to what you wish spending your time on. Some people want to start with the kernel and not even do any booting stuff. Great, get something like GRUB and an already formatted media device/image file and go for it. Some people like to start from the beginning, writing their own boot and loader code.

The sky is the limit, just make sure you use the correct tools to reach with.

Ben

I think that there is some confusion on the difference between the. MR and the VBR. You wouldn't be the first person here to have a problem with that; trust me, I have been there myself.

First, if you're MBR is loading to 07C00:0000, rather than 0700:0000 as it ought to, then either you're mistaken about which is the MBR and which is the VBR, or something is very wrong with your firmware. I am going to bet on the former.

EDIT never mind, I am the one who is confused. The MBR does load to 0000:7C000 (notes the correction about the segment and offset, BTW) I don't know why I was thinking it didn't.

You seem to be thinking of the MBR as the equivalent of the boot sector of a floppy disk, but that isn't exactly the case. The purpose of the MBR is not to boot the specific OS on the boot partition. Rather, it is to have a record of all the partitions on the drive, and provide the code that loads the VBR to 07C00:0000., as well as handling any anomalous quirks of the drive and/or the systems loaded on it.

Floppy disks don't have anything exactly like an MBR; since floppies can't be partitioned, they don't need one.

The VBR is the equivalent of the boot sector, and is the code which ought to be loaded to 07C00:0000.

Schol-R-LEA wrote:The VBR is the equivalent of the boot sector, and is the code which ought to be loaded to 07C00:0000.

Please double check your notes. The BIOS loads the first sector of the disk (whether it is a MBR, VBR, nothing, etc.) to physical address 0x07C00. In 16-bit real-mode terms, that is 07C0:0000, or 0000:7C00, or 0123:69D0, or 0321:49F0, or, well you get the idea.

The 16-bit segmented address of 7C00:0000 is physical address 0x7C000, getting close to the BIOS ROM area.

Ben

BenLunt wrote:

Schol-R-LEA wrote:The VBR is the equivalent of the boot sector, and is the code which ought to be loaded to 07C00:0000.

Please double check your notes. The BIOS loads the first sector of the disk (whether it is a MBR, VBR, nothing, etc.) to physical address 0x07C00. In 16-bit real-mode terms, that is 07C0:0000, or 0000:7C00, or 0123:69D0, or 0321:49F0, or, well you get the idea.

The 16-bit segmented address of 7C00:0000 is physical address 0x7C000, getting close to the BIOS ROM area.

Ben

Oh, hell, another mistake. I feel rather foolish about that. At least this one was probably a typo (too many zeros) rather than whatever the other thing was.

Does this look okay? There are supposed to be 2 partitions 4.9 GB each, FAT 32 filesystem. (I used this: mkdosfs -F32 HDD.img to create a filesystem)
Also what is that data at the beginning? Shouldn't that be empty?

Hi,

Octacone wrote:Does this look okay? There are supposed to be 2 partitions 4.9 GB each, FAT 32 filesystem. (I used this: mkdosfs -F32 HDD.img to create a filesystem)

No, it looks like there's no partition table at all (assuming that the offsets on the left hand side are correct and that the first image is the MBR, and assuming that it wasn't partitioned with GPT).

Octacone wrote:Also what is that data at the beginning? Shouldn't that be empty?

Ideally there should only be 2 cases:

• The 0xAA55 signature is not present, so that the BIOS knows the disk isn't bootable and can move on to the next device in its list of things to try to boot from
• The 0xAA55 signature is present, and the disk contains a boot loader and a bootable OS

Unfortunately Microsoft suck - when a disk is only used for data and shouldn't be bootable, they use a small piece of bootable code to display a "this disk isn't bootable" error message which does nothing useful. More specifically, it ruins the BIOS's ability to move on to the next device in its list of things to try to boot from, and annoyed everyone that accidentally left a "data only" floppy disk in their computer's floppy drive.

Of course open source developers will mindlessly clone Microsoft's stupidity without thinking about it (and Unix stupidity, and any other stupidity they can find).


Cheers,

Brendan

Brendan wrote:Hi,Unfortunately Microsoft suck - when a disk is only used for data and shouldn't be bootable, they use a small piece of bootable code to display a "this disk isn't bootable" error message which does nothing useful. More specifically, it ruins the BIOS's ability to move on to the next device in its list of things to try to boot from, and annoyed everyone that accidentally left a "data only" floppy disk in their computer's floppy drive.

Actually, it wasn't Mickeysoft's fault, not entirely at least - the IBM PC (as in the true-blue original Model 5150 - aptly named, given all the people it drove to insanity) couldn't support a second floppy drive, period, nor a hard drive (though some 'hard cards' were introduced which would have a thin hard disk - or in some cases, a bubble memory - on the card itself, which would run from their on-board extension ROMs and an MS-DOS .SYS driver, IIRC). It had one full height floppy drive, 4 inches (7 cm) tall, which filed the entirety of the single drive bay.

IIRC, if there was no disk in the drive, or the disk wasn't bootable, it did the same thing most of the home computers of the day did - it spun the drive up anyway, until you hit ESCape, at which point it dumped you into the ROM BASIC interpreter. The ROM BASIC on the IBM PC was smart enough to read a cassette drive, but not a disk drive, and since hardly anyone (in the US, at any rate) was still using cassette drives by the time the PC hit the market, well...

When the IBM PC/XT was introduced, coming in versions with either dual full-height floppy drives, or a full height 10MiB hard disk and a full-height floppy drive.

Unfortunately - again, if my memories of our already-outdated used XT are correct - the XT BIOS still had a ROM BASIC (it wasn't dropped until the AT), and still assumed that 'no bootable disk == wait to see if the put in a floppy or if they want to go to the BASIC interpreter'.

They had at least made it smart enough to boot from the hard disk if no floppy were present at all, but a data floppy would still dump you to BASIC. Worse still, in a dual-floppy system[1], it wouldn't boot from the B: drive come hell or high water - it simply didn't recognize it as a drive that could be booted from. This put MS in a sticky situation, as by then the ROM BASIC was already a thing of yesteryear, and they had to make do with it as they could.

The fact that some of the less well-behaved clones would do even stupider things if you booted the system with a data disk in the A: drive didn't really help matters any[2].

Their solution sucks, yes, and it sucks that it and so many other relics of the past are giving us headaches today, but as is so often the case, there is more than enough blame to go around.

1. Or the later installations with two after-market half-height floppies and a HDD - I did that for my father's, which was originally the full-height floppy model, though by the time I did this in 1990 the only MFM hard drive I could find that worked with it was a used 10MiB off of another XT which sounded like a sanding wheel when it spun up.
2. especially since not all of the machines MS-DOS ran on back in the day were actually PC-compatibles at all. The thrice-cursed DREC Painbow comes to mind - I learned Turbo Pascal on those miserable things in my first semester at SCSU, and everyone who used them were ecstatic when they finally chucked them the next semester (Spring 1987). There were problems even with some of the better ones, such as the unfortunate Olivetti/AT&T 8086-based model - it was actually significantly better then the IBM machines in some ways, but because it used passive backplane similar to the S-100 bus instead of a motherboard - a proprietary 16-bit one designed well before the AT bus was introduced - and had significant differences in the BIOS as well, compatibility for both hardware and software was piteously limited.

Octacone wrote:Does this look okay?

First, might I suggest that your hex dumps only list 16, 32, or possibly 64 bytes per line. It is much easier to see where something is, if it is aligned at 16 bytes.

By the names of the images, I am guessing that you have two partitions. Is one of them just after the other, give or take? Do you have the MBR before the first?

The first partition seems to be correct (a quick glance), but the second doesn't have the RRaA signature (again, a quick glance).

Octacone wrote:There are supposed to be 2 partitions 4.9 GB each, FAT 32 filesystem. (I used this: mkdosfs -F32 HDD.img to create a filesystem)
Also what is that data at the beginning? Shouldn't that be empty?

Again, what mkdosfs is that? A linux util?

What "data at the beginning"? The beginning will be the BPB, then sector 2 will be the FAT32 RRaA sig, etc.

Please be a little more detailed in your questions.

Ben

@Brendan
Oh that makes sense. It could be the tool I used.
Something is definitely overwritten.

@BenLunt sorry, I forgot to mention that everything was Linux related.
Actually that is a single HDD image. Those screenshots represent that HDD.
I will see if I can do anything about the way hexdumps are displayed.
The bytes I was talking about where those from 0x0 and further. I did not put them in there it is probably a misuse of mkdosfs. Mkdosfs doesn’t seem to care about fdisk partitioned virtual HDDs??

@Schol-R-LEA looks like you know a lot about old computers. It is always nice to learn some history along the way.