Getting Started (double entendre!)

[madeofstaples]

Haha, witty title (I thought so)

er, anyway, I have a lot to say/ask, so I'll divide things up:

Background
I've held back the urge to dive into OS development to let myself just read and absorb information, but now I think I'm ready to start.

I think I'll eventually just use Grub or something as my bootloader, however I've decided that I want to make one just for the understanding.

I've seen countless "Hello, World!" example bootloaders and have written a few small things to test ideas or answer things i wasn't sure about while reading, so I'm quite familiar with how everything works at this point - I used documentation on the keyboard controller to write something to enable A20 (the standard method and also one that supports the method by the HP Vectra chipset).

Some Design
I've chosen EXT2 as my main File System at least for now. I think I have EXT2 documentation burned on the back of my eyes. I created a drive image file, formatted it to EXT2, mounted it, played around, unmounted it. I then opened it up with a Hex Editor and found my way around - so I know that I fully understand how EXT2 works now, or at least enough to read.

I'm not going to worry about partition tables because, supposing I don't end up using Grub to load my kernel, I could use Grub to chainload my bootloader - so the bootloader is meant to be stored in the first 512 bytes of the partition (which, according to some documents I've read, is the main idea, though not necessarily followed).

However, when I look over what I think I need, I'm not sure I can get it to fit in 510 bytes!
So, make a second stage, right? yes, but supposing I want to do a double-stage bootloader, the first stage must...

• Figure out what physical drive it is on
  Figure out what partition it is on (if the drive is partitioned)
  Figure out if LBA48 is supported (worst-case scenario, the second stage is stored near the end of a huge drive! it could happen...)
  • If not, fall back to LBA28 (should be supported on most modern hardware and virtual machines, right?)
  Navigate the file system to find the data for the second stage
  Copy the data for the second stage to some part of memory and jump to it

Some Confusion
Now, from what I've been reading, BIOS calls won't necessarily work, right? Some implement the extensions to interrupt 0x13 that allow LBA mode, but usually that implementation is added by MS or IBM code, correct? So I can't rely on the BIOS.

I know how to check if LBA48 is supported, and I know how to read in LBA mode using PIO (both 28-bit and 48-bit).

I know how to navigate the file system, and I know how to copy data to a certain point in memory and run it.

I think I can figure out how to detect which drive, and which partition this bootloader was loaded from, but I can already see that being a good-sized chunk of code (at least when you consider the 510 byte limitation)!

Also, what if the bootloader happens to be on a CD or floppy disk? I haven't really found any clear documentation - can I access these devices using LBA28? Or is that yet another method that needs to be included in this small space?

A lot of bootloader examples say that the "drive number" is stored in DL. They say 0x00=Floppy, and 0x80=Hard Drive, right? But doesn't the BIOS check each drive until it finds a bootable one? What if that drive is the slave drive of the second IDE channel? DL=0x80 still? How do INT 0x13 calls know what you mean when you say 0x80 at this point? What about the CD drive? or a USB drive?

Also, I thought I read somewhere that another register would be set to the partition that was booted from. Is this true? I can't find the document to verify... If it is, what is the value of that register when the drive is not partitioned??

Also, many computers are coming with SATA drives rather than the ATA/133 drives. Is this a transparent change to the programmer? (oh please say yes)

Finally, can someone please point me in the direction of something that explains what is where in memory when the computer boots? I know the IDT is... somewhere, and then there's video memory... somewhere else... basically I need to know where i can load temporary values, and the second stage of my bootloader without messing something else up (also, it seems like the first 640 kbytes of memory are useable - why don't any examples seem to use that??)!

I figured I'd post all these questions and see what other people say before I start, so I don't head down the wrong path.


Thank you for any help!

[AJ]

Hi,

Not sure I can help with your disk questions because I went to GRUB ages ago, but I may be able to help with memory.

When you boot, there is no IDT, as this is a protected mode structure that you need to create. You have the real mode IVT which is located from 0x0000 - 0x0500. If you have switched to protected mode this can be overwritten, but only if you do not want to later use real-mode interrupts in v86 mode.

Then, from 0x0500 - 0x90000 (? - this varies from machine to machine), you have free RAM. From this point to 0x100000 (1MB) is the Bios Data Area (BDA) which you must not overwrite, even if you change to PMode.

At 0xB8000 is 80*25 text-mode video RAM. Each word contains a character code and a colour code. (There are other video areas at 0xB0000 and 0xA0000, I believe, but I have never used them).

Sometimes, there will be data such as VESA interfaces loaded higher in RAM.

In theory, when you boot, you should not assume anything about memory. Your boot loader is, of course, loaded at 0x7C00 - 0x7DFF. Have a look at bios function 15 http://www.ctyme.com/intr/rb-1741.htm for more information about this.

Although I know nothing about EXT2, if you need to execute more code than you have available in your own boot loader, you have 3 options:

1. Directly load the second sector of the disk, having formatted the disk so that the BPB shows more than one reserved sector.
2. Make your first stage boot loader simply read the file system and execute your second stage loader - nothing else (this is certainly possible in 510 bytes with FAT).
3. Use GRUB - you can still write a pre-kernel secondary bootloader which takes over from GRUB and does some system initialisation before your kernel starts up.

Hope some of this rambling helps!
Adam

[mathematician]

The BIOS passes the drive number to the program loaded onto the boot sector of a floppy in the dl register. On a hard disk the program loaded from the master boot record similarly gets the drive number passed to it in the dl register. The BIOS knows nothing about partitions, therefore the drive number in dl always refers to a physical drive. Whether on a floppy or hard disk, the BIOS always reads the first sector of track 0 on the physical disk.

As you say, there is precious little room for anything very much in a boot sector. If you disassemble an MBR you will find it using the classic CHS addressing scheme of int 13h fn 2. One solution to the LBA problem is to reserve the whole of track 0, head 0, for a program which will implement LBA extensions to the BIOS. It is the MBR's job to scan the partition table for the entry marked as active, and then load the boot sector for that partition; passing in DS:SI a pointer to the relevant partition table entry. Hard drives always have a partition table, even if there is only one partition on the disk. On a floppy DS:SI points to nothing in particular. The partition's boot loader must either load the second stage of a chain loader, or else load the kernel. In a multi-boot system it has to be the former.

The only real option is to reserve a fixed location on the partition for the second stage of a chain loader, or for a kernel file. In the available 512 bytes of the first stage loader there is no possibility of being able to parse the file system.

Since the BIOS only knows about physical drives, 80h always refers to the master on IDE cnannel 1, 81h to the slave on IDE channel 1, 82h to the master on IDE channel 2 and 83h to the slave on IDE channel 2. The programming interfaces for PATA an SATA drives are thankfully identical.

Grub is a chain loader which will load your kernel for you. Therefore if you use grub there is no need to worry about having to write your own loader. But if you do, there is nothing to say that the loaders on a floppy or hard disks have to be identical. When at some stage, probably well down the line, you write your disk formatting program it could determine which loader was the more appropriate.

In real mode the first kilobyte of memory is reserved for the interrupt vector table. Memory from 0x400 up to somewhere beyond 0x500 is reaerved by the BIOS. After being loaded at 7C00, the MBR usually relocates itself to 0x600, so it should be safe for your code to reuse the same memory. Memory from 0xA0000 up to the end of the first megabyte is reserved first for the video buffers and, after that, for BIOS extensions.
The reserved area of memory mentioned above are well known, so they are not reported by int 15h, fn 0xE820, but that function can be used to get a map of memory used by any other memory mapped devices that may be lurking. The protected mode IDT is anywhere you want to put it.

[AJ]

The only real option is to reserve a fixed location on the partition for the second stage of a chain loader, or for a kernel file. In the available 512 bytes of the first stage loader there is no possibility of being able to parse the file system.

I disagree.

Have a look at http://bos.asmhackers.net/ and download version 0.4. This contains a 512b boot loader which parses the FAT12 file system and loads the kernel. I am sure I have also seen this with FAT32. As for ext2 - I have no idea...

[mathematician]

The only real option is to reserve a fixed location on the partition for the second stage of a chain loader, or for a kernel file. In the available 512 bytes of the first stage loader there is no possibility of being able to parse the file system.

I disagree.

Have a look at http://bos.asmhackers.net/ and download version 0.4. This contains a 512b boot loader which parses the FAT12 file system and loads the kernel. I am sure I have also seen this with FAT32. As for ext2 - I have no idea...

I could only find version 1. A remark I would make is that disk_read assumes that you will not be reading beyond the end of a track; or at least that the BIOS is capable of multi-track operations. As it happens that will work if you assume the usual structure of a 1.44mb floppy. Elsewhere in his code, however, he tries to make his loader more general purpose than that. He also assumes that the filename will be upper case; a valid assumption in the case of an MS-DOS floppy, but not (for instance) in Windows. Another minor point is that he assumes a cluster size of one.

I suppose that no OS since MS-DOS has seriously tried to load an entire operating system from a floppy, But, in the days when operating systems did so, it was assumed that IO.SYS and MSDOS.SYS would be the first two entries in the root directory, and that they would occupy the sectors immediately following upon that root directory.

[madeofstaples]

Whew, I just wrote a ton. Feel free to skim over and answer parts of this post (thanks again) - I certainly took a few breaks while writing it. I'll divide it up again.

A lot of my questions this time are just out of curiosity...

In response to AJ

Memory

When you boot, there is no IDT, as this is a protected mode structure that you need to create. You have the real mode IVT which is located from 0x0000 - 0x0500. If you have switched to protected mode this can be overwritten, but only if you do not want to later use real-mode interrupts in v86 mode.

So if I want my OS to be able to emulate DOS to run old DOS programs, for example, I shouldn't touch that once I get into protected mode. BTW, I think I meant IVT, not IDT

Out of curiosity, if I disable interrupts (like, for enabling A20), supposing I have already loaded all the necessary code to do drive/FS access, since the drive accesses use PIO, I don't need to re-enable interrupts, right? (Well, even if they were calls to BIOS, those interrupts still work since they're non-maskable, I think).

If that's the case, these drive operations will load the kernel, and then I just have to switch to protected mode (at which point, you say, it's safe to mess with the IVT), which involves disabling interrupts anyway.

So I'm saying that it seems possible to engineer my code so that once I disable interrupts, I no longer need to re-enable them until protected mode. Can I mess with the IVT at that point (i.e, in real mode with no maskable interrupts - supposing I no longer need any non-maskable interrupts)? Or will that cause problems still? Just curious

Then, from 0x0500 - 0x90000 (? - this varies from machine to machine), you have free RAM. From this point to 0x100000 (1MB) is the Bios Data Area (BDA) which you must not overwrite, even if you change to PMode.

At 0xB8000 is 80*25 text-mode video RAM. Each word contains a character code and a colour code. (There are other video areas at 0xB0000 and 0xA0000, I believe, but I have never used them).

What is the function of the BDA? What happens if I overwrite it?

So video area is in the middle of the BIOS area? Hmm, I wonder what those other memory areas do... I guess I shouldn't worry about that until I start working on the kernel.

Sometimes, there will be data such as VESA interfaces loaded higher in RAM.

In theory, when you boot, you should not assume anything about memory. Your boot loader is, of course, loaded at 0x7C00 - 0x7DFF. Have a look at bios function 15 http://www.ctyme.com/intr/rb-1741.htm for more information about this.

I'm a little confused about this, two things:

1) If I cannot assume anything about memory, then how do I know, for example, that I can load a larger, second stage to that area you said is usually free RAM? (also, an aside, does anyone know the historical/backwards compatibility reasons for loading the bootsector to 0x7C00? Why not 0x0500, the start of the free RAM?)
2) The Ralph Brown page you linked, it says to set EAX, EBX, EDX, ECX etc... but I'm not in 32-bit protected mode yet! And I thought you could no longer use the BIOS interrupts from protected mode? Aren't these 32-bit registers??

Disks

Although I know nothing about EXT2, if you need to execute more code than you have available in your own boot loader, you have 3 options:

1. Directly load the second sector of the disk, having formatted the disk so that the BPB shows more than one reserved sector.
2. Make your first stage boot loader simply read the file system and execute your second stage loader - nothing else (this is certainly possible in 510 bytes with FAT).
3. Use GRUB - you can still write a pre-kernel secondary bootloader which takes over from GRUB and does some system initialisation before your kernel starts up.

Method #1: Sorry, BPB? Wikipedia gives me an idea of what this is, but I haven't yet seen this used, can you point me somewhere?

From context it seems like you're saying I can use the BPB to say "I'm going to reserve these first few blocks of the hard drive for boot code, so don't use it for filesystem data." however... it seems like (from wikipedia), that the BPB is a filesystem feature of FAT file systems and NTFS (and not EXT2), which makes sense because all the documentation of EXT2 says that the superblock starts at offset 1024 always.

That means I could safely use the second 512 bytes of the drive, but what if I want to support more filesystems? Is it a safe assumption that all filesystems will have some mechanism (i.e, a BPB or simply a specification) where I can reserve the first 1024 bytes? (ok maybe not all filesystems - surely older ones never thought we'd need more than 512 bytes for booting - but at least newer filesystems?)

Method #2: That's what I was originally thinking, but I highly doubt I can fit an EXT2 read implementation in 510 bytes (though I haven't tried yet, I'd hate to find out that I really can't the hard way).

Method #3: I think I will end up using Grub in the long run, but I'd like to take this challenge on for my own learning purposes

Hope some of this rambling helps!

Definitely! Thank you so much for taking the time to respond!

In response to mathematician

Disk

The BIOS passes the drive number to the program loaded onto the boot sector of a floppy in the dl register. On a hard disk the program loaded from the master boot record similarly gets the drive number passed to it in the dl register. The BIOS knows nothing about partitions, therefore the drive number in dl always refers to a physical drive. Whether on a floppy or hard disk, the BIOS always reads the first sector of track 0 on the physical disk.

As you say, there is precious little room for anything very much in a boot sector. If you disassemble an MBR you will find it using the classic CHS addressing scheme of int 13h fn 2. One solution to the LBA problem is to reserve the whole of track 0, head 0, for a program which will implement LBA extensions to the BIOS.

I was pretty sure about the first part of what you said, and now I'm certain (thanks).

Sorry, I haven't been paying much attention to CHS as I've been reading that most modern hardware should support LBA in some form... but from what I (think I) know, wouldn't that mean, I'd be asking for 512*(63 SectersPerTrack) / (1024 bytes/killobyte) = 31.5 killobytes?? That seems like plenty but what about the filesystem? The EXT2 superblock starts at offset 1024!

It is the MBR's job to scan the partition table for the entry marked as active, and then load the boot sector for that partition; passing in DS:SI a pointer to the relevant partition table entry.

Okay. I knew that about what the MBR is supposed to do, so what I'm focusing on right now is the second part of that (the bootsector of the partition - so I don't have to worry about the partition table and limiting myself farther to 446 bytes!)

DS:SI is a pointer to memory, right? so they point to the 16-byte data structure in memory, correct? And I should have my bootloader copy necessary information from that data structure before accessing more memory (so that I don't overwrite). Am I understanding that right?

Partition Tables

Hard drives always have a partition table, even if there is only one partition on the disk.

I'm not very familiar with the format of a partition table, so I'm going to pause right here to go educate myself...

All right, so in a system with a partitioned drive where my code is located on the first sector of a bootable partition, I don't need to worry - but you're saying that in a system with one disk with only one partition, the partition table still must exist... so that means either:

1) The partition table indicates that the first partition takes up the entire drive (starting at the very first sector, ending at the very last and thus the MBR of the drive = the bootsector of the partition) or
2) The partition table indicates that the first partition takes place somewhere after the first sector and ends at the last.

Which brings me to a few questions about partition tables:
Is method number 1, listed above, even possible? can the partition table include the sector that it's in, in a partition?

Method 2 makes more sense to me, because then there's a seperate place for the bootcode in the MBR and the bootloader in the first sector of the partition. Is that the idea?

I tried to come up with the values for the fields in the partition table for a 1GB hard drive with only one partition, and I got this far:

Code: Select all

partition 1:
0x80		// bootable
0x00		// first head
0x0000		// first cylinder and sector (C=0, S=0)
(arbitrary(?) byte) perhaps 0x83	// linux native partition system ID
0x8B		// ending head (decimal 139)
0x2083		// ending cylinder and sector (C=131, S=8)
0x????????	// "relative sector to start of partition" question... (see below)
0x00200000	// 1 GB = 1073741824 bytes / 512(bytes/sector)=2097152 sectors

Ok, first of all, what exactly does the "relative sector" field do? I can find plenty of examples of partition tables but this field seems obvious to everyone else (heh..). Maybe it's the wording that's throwing me off, but why would partition 1 need to define it's relative sector to the partition (itself???). At first when I saw "relative sector" I thought "offset", which is why I set the first CHS values to 0, thinking that putting an offset later would cause the partition not to overlay the MBR...

Which brings me to my next question: supposing, from the word "relative", that this field offsets the CHS values in any way - what happens as you approach the limit of the CHS values? Aren't they also limited by the hardware or does it convert to LBA?

How does it happen that I can partition a 100 GB hard drive into 50 GB partitions when the limit for CHS if 8.4 GB's?

Anyway, if it is the case that I can offset partitions, then I can see how the MBR can request all of head 0, track 0 without disturbing the filesystem...

Last question about partition tables: suppose, as it were, partition 1 takes up the entire drive. Do most OS's notice this and ignore the rest of the partition table (since, logically, there is no room for any more partitions)? I suppose this isn't very important since the MBR can't know anything about the partition table beforehand and therefore can't use it's space anyway... But I'm just curious, what would the partition table entries look like for partitions 2, 3, and 4?

On a floppy DS:SI points to nothing in particular. The partition's boot loader must either load the second stage of a chain loader, or else load the kernel. In a multi-boot system it has to be the former.

The only real option is to reserve a fixed location on the partition for the second stage of a chain loader, or for a kernel file. In the available 512 bytes of the first stage loader there is no possibility of being able to parse the file system.

Obviously, it's possible with a pretty simple file system, but I'm fairly certain it's not going to be possible with EXT2 (or if it is, it's going to be really difficult, especially as a first real bootloader). That being said, up until now I've been afraid of just loading a fixed location because I thought it'd interfere with the filesystem, but the answer to my partition questions might be the answer here as well.

Booting from different types of drives

Since the BIOS only knows about physical drives, 80h always refers to the master on IDE cnannel 1, 81h to the slave on IDE channel 1, 82h to the master on IDE channel 2 and 83h to the slave on IDE channel 2. The programming interfaces for PATA an SATA drives are thankfully identical.

Exactly what I was looking for, Thank you!

Grub is a chain loader which will load your kernel for you. Therefore if you use grub there is no need to worry about having to write your own loader. But if you do, there is nothing to say that the loaders on a floppy or hard disks have to be identical. When at some stage, probably well down the line, you write your disk formatting program it could determine which loader was the more appropriate.

I was toying around with that idea too, I thought it'd be really convenient to have one bootloader, however, I suppose even grub's setup probably detects the type of drive and writes accordingly. This makes a lot of sense - I think for now I'll just worry about the hard drive, and perhaps later the CD-ROM or USB drives (though, I'm not sure if qemu's BIOS will boot from an emulated USB drive? I haven't tried).

Memory

In real mode the first kilobyte of memory is reserved for the interrupt vector table. Memory from 0x400 up to somewhere beyond 0x500 is reserved by the BIOS.

Now I feel kind of bad asking this (I always thought it sounded kinda rude in debates), but does anyone have a source? AJ said something similar but seemed to indicate that 0x500 was the start of free RAM.

After being loaded at 7C00, the MBR usually relocates itself to 0x600, so it should be safe for your code to reuse the same memory.

Now this question is just out of pure interest, and I actually asked it above, but: Why is 0x7C00 the place to load every piece of bootcode (both the MBR and the bootsector of the partition)? Why isn't it standard to load the MBR to 0x600? and the bootsector to 0x800 or something? Surely that convention would save some code since the MBR wouldn't have to relocate itself?

Memory from 0xA0000 up to the end of the first megabyte is reserved first for the video buffers and, after that, for BIOS extensions.

Also out of interest (and also asked above, heh..), what exactly are the video buffers for that don't affect what you see on the screen?

The reserved area of memory mentioned above are well known, so they are not reported by int 15h, fn 0xE820, but that function can be used to get a map of memory used by any other memory mapped devices that may be lurking. The protected mode IDT is anywhere you want to put it.

Okay, this is another question I asked above - the int15h fn 0xE820 seems to use 32-bit registers, but I'm not in 32-bit mode? how's that?

Also, a memory map is obviously going to be dynamic and differ from system to system. Does this mean I must make the second stage position-independent code? Since I'm not sure I'll be able to load it to the same position every time? Anyone care to point me in the direction of a good tutorial on how to write position-independent code (or, which opcodes can I no longer use? or is that not the case?)

THANK YOU!
Thank you guys so much for your time, you've been a great help so far. I think the two biggest things keeping me from starting are my questions regarding partitions, and how I can use the BIOS call to get a memory map from real mode.

Oh, also:

Have a look at http://bos.asmhackers.net/ and download version 0.4. This contains a 512b boot loader which parses the FAT12 file system and loads the kernel. I am sure I have also seen this with FAT32. As for ext2 - I have no idea...

I think I can give some info here. I haven't looked into FAT very much. If I remember correctly, the design behind FAT file systems is kinda based on linked lists, right? It's probably a little more complicated than a simple linked list, I know, but that is the method of assigning parts of the drive to a file when it needs more space and it's a huge cause of fragmentation.

EXT2, on the other hand, is a surprisingly well designed file system, yet it's not too complicated. It's built up of "groups", which contain "blocks" (not in the LBA sense), which may contain several different structures. As such, to implement EXT2, it takes more code to locate data on a disk, but it takes less disk access (= faster, which doesn't really matter in the context of a bootloader, but it's a great choice for a default file system).

I get the impression that fitting a FAT32 read implementation is possible, but it's not extremely easy or trivial - it sounds like there are some tricks (albeit tricks that one could probably figure out on their own) to it. If that's the case, then I cannot fathom how an EXT2 read implementation would fit in the same space.

[mathematician]

Sorry, I haven't been paying much attention to CHS as I've been reading that most modern hardware should support LBA in some form... but from what I (think I) know, wouldn't that mean, I'd be asking for 512*(63 SectersPerTrack) / (1024 bytes/killobyte) = 31.5 killobytes?? That seems like plenty but what about the filesystem? The EXT2 superblock starts at offset 1024!

Conventionally the first partition begins with head 0, sector 1 (first sector), of track 1. However, that is only convention, so the partition table entry still has to say as much. That leaves you with track 0 to do what you like with.

Okay. I knew that about what the MBR is supposed to do, so what I'm focusing on right now is the second part of that (the bootsector of the partition - so I don't have to worry about the partition table and limiting myself farther to 446 bytes!)

Well the MBR will still only load sector 1 of the partition, but you can use the code in that sector to load as many additional sectors as you like.

DS:SI is a pointer to memory, right? so they point to the 16-byte data structure in memory, correct? And I should have my bootloader copy necessary information from that data structure before accessing more memory (so that I don't overwrite). Am I understanding that right?

DS:SI points to a 16 byte data structure. What you do with it is obviously up to you; move it to a place of safe keeping if you think that would be best.

All right, so in a system with a partitioned drive where my code is located on the first sector of a bootable partition, I don't need to worry - but you're saying that in a system with one disk with only one partition, the partition table still must exist... so that means either:

1) The partition table indicates that the first partition takes up the entire drive (starting at the very first sector, ending at the very last and thus the MBR of the drive = the bootsector of the partition) or
2) The partition table indicates that the first partition takes place somewhere after the first sector and ends at the last.

2 is correct. See above re location of first partition. The MBR and partition boot sector are two distinct entities, even when there is only one partition.

Ok, first of all, what exactly does the "relative sector" field do? I can find plenty of examples of partition tables but this field seems obvious to everyone else (heh..). Maybe it's the wording that's throwing me off, but why would partition 1 need to define it's relative sector to the partition (itself???). At first when I saw "relative sector" I thought "offset", which is why I set the first CHS values to 0, thinking that putting an offset later would cause the partition not to overlay the MBR...

I don't know why they called it the relative sector. It is actually the LBA address of the first sector in the partition

Last question about partition tables: suppose, as it were, partition 1 takes up the entire drive. Do most OS's notice this and ignore the rest of the partition table (since, logically, there is no room for any more partitions)? I suppose this isn't very important since the MBR can't know anything about the partition table beforehand and therefore can't use it's space anyway... But I'm just curious, what would the partition table entries look like for partitions 2, 3, and 4?

Partition type 0 indicates that a partition is either not present, or at least not used.

Obviously, it's possible with a pretty simple file system, but I'm fairly certain it's not going to be possible with EXT2 (or if it is, it's going to be really difficult, especially as a first real bootloader). That being said, up until now I've been afraid of just loading a fixed location because I thought it'd interfere with the filesystem, but the answer to my partition questions might be the answer here as well.

Well the thing about writing an operating system is that you can play the great dictator. Since they are the operating system files are the first files to be installed, you can plonk them down where you like, and then make directory and inode entries accordingly.

Now I feel kind of bad asking this (I always thought it sounded kinda rude in debates), but does anyone have a source? AJ said something similar but seemed to indicate that 0x500 was the start of free RAM.

I'm right (of course), but, unless you are counting on using the real mode bios to format a disk, there is probably nothing above 0x4FF which is of much interest to you. (If you were planning on formatting a disk you would need the data structure at 0x522.)

Now this question is just out of pure interest, and I actually asked it above, but: Why is 0x7C00 the place to load every piece of bootcode (both the MBR and the bootsector of the partition)? Why isn't it standard to load the MBR to 0x600? and the bootsector to 0x800 or something? Surely that convention would save some code since the MBR wouldn't have to relocate itself?

Just convention. If you wanted to know why 7C00 was chosen, you would have to ask the programmers who were working for IBM in 1981, and who wrote the first BIOS. The first PC was released with version 1 of MS-DOS, and that didn't even support hard disks, so the question of the MBR and partition boot sectors clashing at memory location 7C00 didn't even arise. When it did arise, relocating it to 0x600 was presumably thought to keep it out of the way of the BIOS data area, as well as any extensions to the same which might arise.

Also out of interest (and also asked above, heh..), what exactly are the video buffers for that don't affect what you see on the screen?

Depending upon the mode all the video buffers affect the screen in one way or another. When the display is being run in text mode the memory from 0xB8000 to approximately 0xBC000 contains four pages of text, and you can determine which page is actually displayed on the screen by programming hardware. By default the first page is displayed. The first graphics adaptor for the PC utilised the same area of memory, but from the EGA onwards the whole of the address space above 0xA0000 has been used.

Okay, this is another question I asked above - the int15h fn 0xE820 seems to use 32-bit registers, but I'm not in 32-bit mode? how's that?

This is a common misunderstanding. The early processors in the x86 family were 16 bit processors, so MS-DOS was a 16 bit operating system and all the applications available at the time were also 16 bit. When the first 32 bit processor, capable of protected mode, came out it caused no great excitement because both the operating system and apps available at the time could only run in real mode. From force of habit, and the fact that the OS could only make use of 16 bit registers, programmers continued to use only the 16 bit registers. However, there is absolutely no reason at all why you cannot use the 32 bit registers in real mode if you really want to.

Also, a memory map is obviously going to be dynamic and differ from system to system. Does this mean I must make the second stage position-independent code? Since I'm not sure I'll be able to load it to the same position every time? Anyone care to point me in the direction of a good tutorial on how to write position-independent code (or, which opcodes can I no longer use? or is that not the case?)

I suppose the answer to that might be to implement your default file system and a loader in the space below 0xA0000, and then to implement the rest of your operating system an the form of normal executable (and relocatable) files.

Oh, also:
AJ wrote:
Have a look at http://bos.asmhackers.net/ and download version 0.4. This contains a 512b boot loader which parses the FAT12 file system and loads the kernel. I am sure I have also seen this with FAT32. As for ext2 - I have no idea...

I think I can give some info here. I haven't looked into FAT very much. If I remember correctly, the design behind FAT file systems is kinda based on linked lists, right? It's probably a little more complicated than a simple linked list, I know, but that is the method of assigning parts of the drive to a file when it needs more space and it's a huge cause of fragmentation.

EXT2, on the other hand, is a surprisingly well designed file system, yet it's not too complicated. It's built up of "groups", which contain "blocks" (not in the LBA sense), which may contain several different structures. As such, to implement EXT2, it takes more code to locate data on a disk, but it takes less disk access (= faster, which doesn't really matter in the context of a bootloader, but it's a great choice for a default file system).

I get the impression that fitting a FAT32 read implementation is possible, but it's not extremely easy or trivial - it sounds like there are some tricks (albeit tricks that one could probably figure out on their own) to it. If that's the case, then I cannot fathom how an EXT2 read implementation would fit in the same space.

I will make a brief comment on that. FAT12 is pretty well universally used on floppies because anything more sophisticated would use up too much space on what is, after all, only a 1.44mb diskette.

[madeofstaples]

...

That clears a whole lot up, thanks!

I've come up with only a few more questions (I promise!)

The MBR and partition boot sector are two distinct entities, even when there is only one partition.

So, usually there's 30 or so killobytes that I can use before the first partition starts. Now, I'm not writing an MBR, but the partition boot sector, right? So, if I eventually decide to use Grub in the MBR, doesn't Grub use those 30 killobytes? so my partition boot sector still can't use that space, right?

Well the thing about writing an operating system is that you can play the great dictator. Since they are the operating system files are the first files to be installed, you can plonk them down where you like, and then make directory and inode entries accordingly.

To me that seems kinda like a dirty trick and here's why: Suppose a user of my OS accidentally does something stupid like oh.. say.. "rm -rf *" in the wrong directory as a user with the power to do harm (root or admin or whatever).

If a user somehow has the file system delete necessary boot files, and the user goes on to write more to the hard drive, the file system might overwrite where the bootsector is hard coded to load from. The only way to fix this at this point is re-installing the entire os!

Just convention. If you wanted to know why 7C00 was chosen, you would have to ask the programmers who were working for IBM in 1981, and who wrote the first BIOS. The first PC was released with version 1 of MS-DOS, and that didn't even support hard disks, so the question of the MBR and partition boot sectors clashing at memory location 7C00 didn't even arise. When it did arise, relocating it to 0x600 was presumably thought to keep it out of the way of the BIOS data area, as well as any extensions to the same which might arise.

Wouldn't it be more logical to have the MBR load the partition bootsector to 0x600? No relocation code, and bootsector code would just have to be assembled with an origin address 0x600? Whatever though, it doesn't matter that much.

This is a common misunderstanding. The early processors in the x86 family were 16 bit processors, so MS-DOS was a 16 bit operating system and all the applications available at the time were also 16 bit. When the first 32 bit processor, capable of protected mode, came out it caused no great excitement because both the operating system and apps available at the time could only run in real mode. From force of habit, and the fact that the OS could only make use of 16 bit registers, programmers continued to use only the 16 bit registers. However, there is absolutely no reason at all why you cannot use the 32 bit registers in real mode if you really want to.

Interesting, though after reading the Ralph Brown page carefully, at first it says this at the top:

ECX = size of buffer for result, in bytes (should be >= 20 bytes)

So, a minimum of 20 bytes
then in the notes farther down the page:

A maximum of 20 bytes will be transferred at one time, even if ECX is higher

so now 20 is a maximum?
Is this a typo?
I mean, it seems logical to use 20 (or multiples of 20), and given the discrepancy between some BIOS it's probably wisest to just use 20... but still, that contradiction is weird..

Last question about the Ralph Brown page:

(Table 00581)
Values for System Memory Map address type:
01h memory, available to OS
02h reserved, not available (e.g. system ROM, memory-mapped device)
03h ACPI Reclaim Memory (usable by OS after reading ACPI tables)
04h ACPI NVS Memory (OS is required to save this memory between NVS sessions)
other not defined yet -- treat as Reserved

I'm assuming my code should just pay attention to memory "available to OS", but the part I bolded, says it requires action of the OS? Is this just for Real Mode OS's? or do I need to pass this information on to the kernel?

I suppose the answer to that might be to implement your default file system and a loader in the space below 0xA0000, and then to implement the rest of your operating system an the form of normal executable (and relocatable) files.

So INT 0x15, function 0xE820, will never say that a region of memory below 0xA0000 is reserved?

I will make a brief comment on that. FAT12 is pretty well universally used on floppies because anything more sophisticated would use up too much space on what is, after all, only a 1.44mb diskette.

Good point

[Brendan]

Hi,

If a user somehow has the file system delete necessary boot files, and the user goes on to write more to the hard drive, the file system might overwrite where the bootsector is hard coded to load from. The only way to fix this at this point is re-installing the entire os!

If the file security in your OS doesn't work, then you're screwed anyway. Of course there's no real reason for the boot files to be in any file system (you could use a special installer that does direct disk access to change the boot files, and then just reserve space before the file system itself starts or have a special boot partition).

Wouldn't it be more logical to have the MBR load the partition bootsector to 0x600? No relocation code, and bootsector code would just have to be assembled with an origin address 0x600? Whatever though, it doesn't matter that much.

Once upon a time (a long time ago) hard disks didn't have partitions at all, and the OSs boot sector lived in the first sector of the disk and was loaded at 0x7C00.

If your OS installer modifies a few values in the boot loader when it installs the boot loader, and if the boot loader uses these values to find the second stage, then it's simple to write an OS that works on partitioned hard drives and "partitionless" hard drives.

Also, it's possible for an MBR to chainload another MBR which chainloads another MBR which loads the OS's boot sector.

Relocation is flexible, while having a different address for the boot sector means that the BIOS and any boot managers need to know if they're loading a boot manager or a boot sector before they actually load it.

Interesting, though after reading the Ralph Brown page carefully, at first it says this at the top:

ECX = size of buffer for result, in bytes (should be >= 20 bytes)

So, a minimum of 20 bytes
then in the notes farther down the page:

A maximum of 20 bytes will be transferred at one time, even if ECX is higher

so now 20 is a maximum?
Is this a typo?
I mean, it seems logical to use 20 (or multiples of 20), and given the discrepancy between some BIOS it's probably wisest to just use 20... but still, that contradiction is weird..

It's either a typo, or a comment that only applies to BIOSs that existed when Ralph updated his list last.

Basically, you must provide a buffer that is at least 20 bytes (at least enough space for the minimum/original amount of information). However, the BIOS function was designed so that it could be extended, so that if you tell the BIOS that your buffer is 40 bytes the BIOS might give you some extra information.

ACPI version 3.0 did extend this BIOS function to return an extra 4 bytes (a 32-bit set of attribute flags). A new OS would use a buffer that is 24 or more bytes, and would get 20 bytes on an older BIOS and 24 bytes on an ACPI 3.0 BIOS. An old OS that uses 20 byte buffer would get 20 bytes on an older BIOS and 20 bytes on an ACPI 3.0 BIOS.

Last question about the Ralph Brown page:

(Table 00581)
Values for System Memory Map address type:
01h memory, available to OS
02h reserved, not available (e.g. system ROM, memory-mapped device)
03h ACPI Reclaim Memory (usable by OS after reading ACPI tables)
04h ACPI NVS Memory (OS is required to save this memory between NVS sessions)
other not defined yet -- treat as Reserved

I'm assuming my code should just pay attention to memory "available to OS", but the part I bolded, says it requires action of the OS? Is this just for Real Mode OS's? or do I need to pass this information on to the kernel?

Ralph Brown's Interrupt list *is* getting old...

From ACPI 3.0:

Code: Select all

Value  Description
 4      ACPI NVS Memory. This range of addresses is in use or
        reserved by the system and must not be used by the OS.
        This range is required to be saved and restored across
        an NVS sleep.
 5      This range of addresses contains memory in which errors
        have been detected. This range must not be used by the OS.

For ACPI NVS memory, I assume this is so the BIOS doesn't forget what it's doing when the OS puts the computer into a deep sleep state (where RAM is saved to disk and the RAM chips lose power).

I suppose the answer to that might be to implement your default file system and a loader in the space below 0xA0000, and then to implement the rest of your operating system an the form of normal executable (and relocatable) files.

So INT 0x15, function 0xE820, will never say that a region of memory below 0xA0000 is reserved?

In almost all cases I've seen, function 0xE820 will say there's an unusable "system area" just below 0x000A0000 (which is the variable sized EBDA, and shouldn't be touched by the OS). If the OS assumes the area from 0x00000600 to 0x00080000 is usable RAM than it'll work correctly on 100% of 32-bit (and 64-bit) 80x86 computers. The closer your OS gets to 0x000A0000 without checking, the more chance there is that the EBDA will be too large and something won't work.

BTW you can use BIOS Int 0x12 to quickly check how much RAM there is below 0x000A0000 if you need to.


Cheers,

Brendan

[Solar]

In the available 512 bytes of the first stage loader there is no possibility of being able to parse the file system.

I disagree.

Have a look at http://bos.asmhackers.net/...

All this assuming you have full control over the hard disk, and limiting the choice of filesystem to whatever the MBR supports. No multiple OS installations...

[AJ]

Sorry I didn't mention that caveat.

I agree that writing your own bootloader to 'see what happens' is a very good idea to get you in to the idea of programming in an environment with no support libs (unless you cound the BIOS...) etc... Once you enter your pmode kernel, you will be more aware of what's going on 'behind the scenes'. however:

* If you plan to use GRUB in the longer term, make a boot loader for a FAT12 floppy disk first - you don't have to worry about partitions and you can parse the file system without much trouble in 512 bytes.
* A 'proper' bootloader can be like a mini-os (GRUB even has a cli). I assume that if you are wanting to develop an OS single-handed, you want to spend most of your time in the kernel. Unless you spend a *huge* amount of time on your bootloader, can you be sure it will be as flexible (supported media / file systems etc..) as GRUB?

If you want to write the successor to the other bootloaders out there - fantastic, but my advice would be if you are just writing the boot loader as an experiment, don't spend all of your dev time on it. You will have chances further down the line to become sick of device access and file systems

Oh - although I told you not to assume anything about RAM layout, this was, perhaps, a little unreasonable. You are fairly safe between 0x500 and 0x80000 (? not sure about this upper limit), and generaly in the space over 1MB (watch out for a hole in RAM from 15-16MB), but I would make it a priority to query the BIOS (or GRUB) for a system memory map as soon as you can in the boot process.

Also, you can use 32 bit registers in 16 bit real mode - the assembler will prefix the opcodes to let the CPU know this is happenning.

Cheers,
Adam

[mathematician]

A 'proper' bootloader can be like a mini-os ......can you be sure it will be as flexible (supported media / file systems etc..) as GRUB?

If it is a custom boot loader written specifically for your OS, it doesn't have to be.

[mathematician]

All this assuming you have full control over the hard disk, and limiting the choice of filesystem to whatever the MBR supports. No multiple OS installations...

During the installation process the OS could give me the option of either having it put in its own MBR, or of leaving the existing one in place. Operating systems, such as some varieties of Linux, which don't do that do not always leave me a very happy man. If you do put in your own master boot record, then either in the MBR itself, or else once your OS has started to load you could scan the partition table type codes for other known operating systems, and then offer the user a menu. If he elects for an OS other than your own you could then load its boot sector and jump to that.

[mathematician]

So, usually there's 30 or so killobytes that I can use before the first partition starts. Now, I'm not writing an MBR, but the partition boot sector, right? So, if I eventually decide to use Grub in the MBR, doesn't Grub use those 30 killobytes? so my partition boot sector still can't use that space, right?

If you are writing a partition boot sector it would be a bit of a cheek to try and use the free space on track 0. What you could do, however, is reserve some space somewhere on the partition. Remember that in writing your own OS you are the boss; so, even if you have a file system which is largely based upon ext2, it isn't necessary to copy it in its every detail.

[madeofstaples]

Wow, thank you everyone for your replies! here come mine

If a user somehow has the file system delete necessary boot files, and the user goes on to write more to the hard drive, the file system might overwrite where the bootsector is hard coded to load from. The only way to fix this at this point is re-installing the entire os!

If the file security in your OS doesn't work, then you're screwed anyway. Of course there's no real reason for the boot files to be in any file system (you could use a special installer that does direct disk access to change the boot files, and then just reserve space before the file system itself starts or have a special boot partition).

Well, my example wasn't file security not working, it was a user of the correct privileges accidentally removing the second-stage boot file.

While I can see your point, that the bootcode probably shouldn't be in the filesystem anyway, here are my qualms:
1) I'd like my OS to work with as many other pieces of software as possible - this means other OS's, other bootloaders, and other disk partitioners. Now, if I use so and so's fdisk program to create partitions, it seems (from what mathematician explains above) that I can assume they leave head 0 and track 0 all for the MBR. However, so and so's bootloaders for the MBR probably counts on this area too, so I probably can't reserve it for my OS, which means the only other place I could keep bootcode is on my partition.
2) It's extremely convenient. I can simply mount my disk image and place my bootcode file in the correct path and then run an emulation to test it.

I suppose having a special boot partition is a nice idea here, and I can see it as an option, but I don't think it should be mandatory. Then again, the more I think about it, it'd require a little more complex of an OS installer (it'd need some type of partitioning step, where it requires you to reserve x amount of space for a boot partition), but then I could work with the guarantee that I have a boot partition...

Wouldn't it be more logical to have the MBR load the partition bootsector to 0x600? No relocation code, and bootsector code would just have to be assembled with an origin address 0x600? Whatever though, it doesn't matter that much.

Once upon a time (a long time ago) hard disks didn't have partitions at all, and the OSs boot sector lived in the first sector of the disk and was loaded at 0x7C00.

If your OS installer modifies a few values in the boot loader when it installs the boot loader, and if the boot loader uses these values to find the second stage, then it's simple to write an OS that works on partitioned hard drives and "partitionless" hard drives.

Also, it's possible for an MBR to chainload another MBR which chainloads another MBR which loads the OS's boot sector.

Relocation is flexible, while having a different address for the boot sector means that the BIOS and any boot managers need to know if they're loading a boot manager or a boot sector before they actually load it.

Very good points that i didn't think of, and a very good answer to my question.

Basically, you must provide a buffer that is at least 20 bytes (at least enough space for the minimum/original amount of information). However, the BIOS function was designed so that it could be extended, so that if you tell the BIOS that your buffer is 40 bytes the BIOS might give you some extra information.

ACPI version 3.0 did extend this BIOS function to return an extra 4 bytes (a 32-bit set of attribute flags). A new OS would use a buffer that is 24 or more bytes, and would get 20 bytes on an older BIOS and 24 bytes on an ACPI 3.0 BIOS. An old OS that uses 20 byte buffer would get 20 bytes on an older BIOS and 20 bytes on an ACPI 3.0 BIOS.

Interesting, I will have to look up this documentation. I assume I'll need these extra 4 bytes to implement ACPI?

Last question about the Ralph Brown page:

(Table 00581)
Values for System Memory Map address type:
01h memory, available to OS
02h reserved, not available (e.g. system ROM, memory-mapped device)
03h ACPI Reclaim Memory (usable by OS after reading ACPI tables)
04h ACPI NVS Memory (OS is required to save this memory between NVS sessions)
other not defined yet -- treat as Reserved

I'm assuming my code should just pay attention to memory "available to OS", but the part I bolded, says it requires action of the OS? Is this just for Real Mode OS's? or do I need to pass this information on to the kernel?

Ralph Brown's Interrupt list *is* getting old...

From ACPI 3.0:

Code: Select all

Value  Description
 4      ACPI NVS Memory. This range of addresses is in use or
        reserved by the system and must not be used by the OS.
        This range is required to be saved and restored across
        an NVS sleep.
 5      This range of addresses contains memory in which errors
        have been detected. This range must not be used by the OS.

For ACPI NVS memory, I assume this is so the BIOS doesn't forget what it's doing when the OS puts the computer into a deep sleep state (where RAM is saved to disk and the RAM chips lose power).

So, I do need to pass that information on to the kernel if I want to have a "hibernate" feature, correct?

I suppose the answer to that might be to implement your default file system and a loader in the space below 0xA0000, and then to implement the rest of your operating system an the form of normal executable (and relocatable) files.

So INT 0x15, function 0xE820, will never say that a region of memory below 0xA0000 is reserved?

In almost all cases I've seen, function 0xE820 will say there's an unusable "system area" just below 0x000A0000 (which is the variable sized EBDA, and shouldn't be touched by the OS). If the OS assumes the area from 0x00000600 to 0x00080000 is usable RAM than it'll work correctly on 100% of 32-bit (and 64-bit) 80x86 computers. The closer your OS gets to 0x000A0000 without checking, the more chance there is that the EBDA will be too large and something won't work.

BTW you can use BIOS Int 0x12 to quickly check how much RAM there is below 0x000A0000 if you need to.

Okay, that's much better, if I can count on that address range then that should be plenty of space, so I don't really need to ask for a memory map from the BIOS except for when I load it to be passed to the kernel.

The Ralph Brown page says that Int 0x12 returns the amount of memory available starting at absolute address 0, but since 0x0 to about 0x600 are reserved, I should just make sure that the result of Int 0x12, minus 256 bytes, is large enough to load something at 0x600, or is that a bad assumption?

I agree that writing your own bootloader to 'see what happens' is a very good idea to get you in to the idea of programming in an environment with no support libs (unless you cound the BIOS...) etc... Once you enter your pmode kernel, you will be more aware of what's going on 'behind the scenes'. however:

* If you plan to use GRUB in the longer term, make a boot loader for a FAT12 floppy disk first - you don't have to worry about partitions and you can parse the file system without much trouble in 512 bytes.
* A 'proper' bootloader can be like a mini-os (GRUB even has a cli). I assume that if you are wanting to develop an OS single-handed, you want to spend most of your time in the kernel. Unless you spend a *huge* amount of time on your bootloader, can you be sure it will be as flexible (supported media / file systems etc..) as GRUB?

At this point, I'm thinking I just want to use GRUB to chainload my bootsector. Remember that I don't plan on writing the code for the MBR, but the bootsector of the partition.

I know that GRUB could just load my kernel, but I don't want to depend on GRUB.

Also, I would like to stay away from FAT (well, I'll probably support FAT32 for compatibility with windows and external hard drives, and FAT12 just for floppies eventually) because although it's simple, by design it fragments easily, and I already know EXT2 backwards and forwards.

Oh - although I told you not to assume anything about RAM layout, this was, perhaps, a little unreasonable. You are fairly safe between 0x500 and 0x80000 (? not sure about this upper limit), and generaly in the space over 1MB (watch out for a hole in RAM from 15-16MB), but I would make it a priority to query the BIOS (or GRUB) for a system memory map as soon as you can in the boot process.

From what Brendan said above, I think I won't need much space at all, so I think the BIOS call Int 0x12 should suffice until I get to the second stage of my bootloader, which will load the kernel, so it'll also load the memory map to pass to the kernel.

Why is there a hole from 15-16MB?

Also, you can use 32 bit registers in 16 bit real mode - the assembler will prefix the opcodes to let the CPU know this is happenning.

I really do learn something new every day, that's cool.

If you are writing a partition boot sector it would be a bit of a cheek to try and use the free space on track 0. What you could do, however, is reserve some space somewhere on the partition. Remember that in writing your own OS you are the boss; so, even if you have a file system which is largely based upon ext2, it isn't necessary to copy it in its every detail.

Okay, but this would mean destroying compatibility with linux or windows (using the ext2 driver). Instead, I think I have a better solution, given the assumption that most modern filesystems will have a 4kb data block size (right?)

So I have a partition bootloader, that in 510 bytes (512? do partition bootsectors need the boot signature? or does the partition simply need to be marked as bootable in the partition table?) must
1) Save partition info pointed to by DS:SI
2) use CHS BIOS calls to try to load head 0, track 0, sectors ??-63 (the end of the track, pretty much - with hope that the MBR doesn't need this space). If the data stored in this area has a certain signature, then run it (loaded it at 0x600)
3) If the signature did not exist it means the MBR program probably overwrote it - so now we should detect if LBA48 is enabled and fall back to LBA28 if it's not. Using the appropriate LBA implimentation, we should scan the raw contents of the partition looking for the signature. Given the 4kb block size this limits the second stage to 4 kb's. When we find the signature, we load the 4kb's to 0x600 and jump to it.
4) If the second stage cannot be found, display an error message.

The second stage has the advantages that the first stage already implements LBA read functions, and already checked if LBA48 was enabled, and has the partition info in memory. The second stage can use this information to:
1) Check if itself was loaded from the filesystem by examining the contents of memory of the bootsector. If it was, modify the bootsector to look for the second stage at the address it is located in the filesystem on the first try (so, the next time the computer restarts, it'll remember where the second stage is).
2) Implement EXT2
3) Enable A20
4) Load memory Map somewhere for the kernel to later access
5) Load the kernel (or third stage if necessary)
6) Enter protected mode set up the stack, and jump to the kernel

This way, the second stage will preferably be stored at the end of track 0, but just in case, a back-up is on the filesystem. I could later write a system utility that checks this and warns the user when the bootloader had to load from the filesystem rather than track 0, and give them the option to restore track 0 if this is in error (i.e, the user didn't change anything boot-wise).

Does anyone foresee any problems? Otherwise I think I'll start writing it