32-Bit Bootloader

[SpyderTL]

I have officially outgrown my own hand-rolled 16-bit Bootloader.

I'm copying data and functions from CD to memory using the INT 13h AH=42h method, which has worked well up until now. But I'm now running out of memory below 1MB, and things are starting to "collide" at boot time.

I can think of a few things that will let me save space below 1MB, which should give me a little more time, but I'm eventually going to need to be able to load my "Kernel" (as it were) to high memory.

I looked into unreal mode, but I don't think I want to go that route.

I could switch to 32-bit mode, and then start loading stuff from CD into memory, but the first problem that I'm going to run into is that I won't know which "device" that I am booting from.

Is there any way to figure out which device that we are booting from when switching from 16-bit (DL=0x80) to 32-bit mode (IDE Controller 0 Channel 1 - slave)?

How are you guys handling this issue?

[alexfru]

Is there any particular problem with first loading disk sectors below 1MB and then copying them above, e.g. load 64KB worth of sectors, copy, repeat until done? There's even a BIOS function to copy physical memory: Int 15/AH=87h. You could easily implement an equivalent yourself, if you'd like to.

[SpyderTL]

This is the first time I've ever heard about the INT 15h copy memory function. I'll give it a shot and see how it works out.

I'm assuming this function is fairly common on modern hardware (since the documentation differentiates between 286 and 386 functionality).

[alexfru]

i80286 and i80386 had different segment descriptors and task segments. What the BIOS here is asking you as the input is, if you look close at it, a partial GDT. Hence the difference between 286 and 386+.

[Brendan]

Hi,

This is the first time I've ever heard about the INT 15h copy memory function. I'll give it a shot and see how it works out.

I'm assuming this function is fairly common on modern hardware (since the documentation differentiates between 286 and 386 functionality).

Typically, the "int 0x15, ah=0x87" BIOS function is a pain in the neck and it's easier to just do it yourself.

Note that for copying memory, you only need to:

• Disable IRQs (CLI)
• (Optional) Load a "null IDT" (e.g. with limit=0) to ensure that a triple fault happens if an NMI occurs at exactly the wrong time (note: NMI typically signals hardware faults on old computers, so ignoring it is a bad idea)
• Set the "protected mode enabled" bit in CR0
• Do a "jmp" to flush the pipeline on old CPUs
• Load ES with a data segment from the GDT (typically "base = 0, limit = 4 GiB")
• If necessary, make sure the direction flag is sane (CLD)
• Do an "a32 es rep movsd" (to copy ECX*4 bytes from ES:ESI to ES:EDI). The "es" override prefix avoids the need to setup DS. The "a32" prefix is because you want 32-bit addressing (and 16-bit addressing is the default).
• (Optional) Load ES with a real mode compatible data segment from the GDT (e.g. with a 64 KIB limit). I wouldn't bother with this (it only effects the ES segment limit).
• Clear the "protected mode enabled" bit in CR0
• Do a "jmp" to flush the pipeline on old CPUs
• Load ES with a real mode value
• (Optional) Reload the BIOS's IDT
• Enable IRQs (STI)

You don't need a full switch to protected mode (no need to setup CS, SS, etc); and you will probably have a suitable data segment in the GDT anyway (used when you're finished with the BIOS and want to switch to protected mode before jumping to your kernel).


Cheers,

Brendan

[Octocontrabass]

Note that for copying memory, you only need to:

Those steps are almost identical to using unreal mode; the only difference is that with unreal mode, you do the copy after returning to "real" mode instead of before. (HIMEM.SYS takes it one step further by performing the switch in the GPF handler. This kind of optimization is probably unnecessary in a bootloader, but it's fun.)

[Combuster]

But unlike real mode, performing the copy from protected mode isn't a hack by anyone's definitions. And for quite a few implementations, both versions are essentially identical, save from the order of instructions.

[SpyderTL]

Typically, the "int 0x15, ah=0x87" BIOS function is a pain in the neck

Can you elaborate a bit? Calling INT 15h seems easier, based on what I'm seeing so far.

[Kazinsal]

Typically, the "int 0x15, ah=0x87" BIOS function is a pain in the neck

Can you elaborate a bit? Calling INT 15h seems easier, based on what I'm seeing so far.

The main problem with it is that it can only copy 64K at once, since it's been around since the AT/286 and was therefore built with a 24-bit address space of 64K segments in mind. Each call to the function you make requires the BIOS to perform the steps Brendan outlined for a 64K copy. This is inefficient.

Another key problem is that it's a BIOS function, and not one I can find a whole lot of information on the standardization and implementation of. Some BIOSes may not support it, and who knows what those which do support it clobber without telling you. The BIOS expects to have free reign of the system. It will do what it pleases.

Now, which is easier? Preparing for dozens of potentially dodgy BIOSes to potentially clobber various system structures in ways you can't always predict several times, or jumping into protected mode for a few dozen instructions to copy a block of memory once?

[SpyderTL]

That makes sense. Anyone know how Grub/Grub2 tackles this problem?

Edit: Also, anyone know if it's possible to figure out which device you booted from once you get to 32-bit mode?

[KemyLand]

Edit: Also, anyone know if it's possible to figure out which device you booted from once you get to 32-bit mode?

Just do it in real mode, save the info where you want (registers, memory?), then switch to PMode and you have the info

Anyway, with the best of all tricks, you'll only be able to store data in up to 8 registers, so you'll need to use memory/trick the stack to get this information. For the stack trick, you must in *first* place remember that you can use 32-bit GPR registers (E*X and E*I families) inside RMode.

Don't record too much information. Remember that once you are in PMode, you will have to win three tedious games: PCI, ATA PIO, (insert your driver's bugs here) to get a single bit from the HDD from here on. I'm really sure if all that BIOS stuff is somewhat helpful. If it is, it will only help a little with ATA PIO

Anyway, that for another post/wiki read. Just smash up the stack a little, then you have all your BIOS info fresh into PMode

[KemyLand]

Edit: Also, anyone know if it's possible to figure out which device you booted from once you get to 32-bit mode?

Just do it in real mode, save the info where you want (registers, memory?), then switch to PMode and you have the info

Anyway, with the best of all tricks, you'll only be able to store data in up to 8 registers, so you'll need to use memory/trick the stack to get this information. For the stack trick, you must in *first* place remember that you can use 32-bit GPR registers (E*X and E*I families) inside RMode.

Don't record too much information. Remember that once you are in PMode, you will have to win three tedious games: PCI, ATA PIO, (insert your driver's bugs here) to get a single bit from the HDD from here on. I'm really sure if all that BIOS stuff is somewhat helpful. If it is, it will only help a little with ATA PIO

Anyway, that's for another post/wiki read. Just smash up the stack a little, then you have all your BIOS info fresh into PMode. That's the way GRUB can pass you BIOS information

A a sidenote, it's good practice to make your bootloader a separate project, rather than what your OS (for now...) depends on. You'll rather advance little, even more when you could use GRUB's magic powers to get some info that's hard to get for your OS (i.e. memory map), so you can use cleaner methods in the kernel (not even a single return to RMode!). The BIOS has always been 16-bit mode thanks to IBM, so this is a real tricky land. I would recommend you to make your OS a Multiboot-complaint one. That way, you can use GRUB(2?) for now, then switch to your own bootloader when it's complaint and working enough. Look at "multiboot.h" and you will see all those nice features you can use, just go away from the video-mode setting ones.

Also, remember a bootloader isn't just a dumb stub that copies a flat binary from disk to RAM through INTs, it's a whole project by its own. Look at GRUB(2)/SYSLINUX/LILO and see their technics to go the way. It's just like the OSDev Wiki, its a collection of the ideas and information from programmers that went through the same way as you long ago.

[Brendan]

Hi,

That makes sense. Anyone know how Grub/Grub2 tackles this problem?

I think GRUB mostly runs in protected mode - when it wants to execute a BIOS function it switches back to real mode, calls the BIOS, then switches back to protected mode.

Edit: Also, anyone know if it's possible to figure out which device you booted from once you get to 32-bit mode?

There's multiple problems with (reliably) finding out what you booted from:

• It's possible for people to move their disks around (e.g. unplug it and plug it in somewhere else).
• It's possible for someone to copy a partition or disk (e.g. create a backup of it).
• It's possible that you booted from 2 or more disks (e.g. firmware does "software RAID").
• Your OS may not have suitable device drivers to access the boot device (e.g. SCSI controller).
• There's no way to determine how "BIOS device numbers" were mapped to actual devices. A lot of BIOSs have options change which disk is the "first" disk.
• For some cases (e.g. "hard disk emulation El Torito", booting from USB flash) the BIOS may deliberately lie (e.g. temporarily pretend that part of the CD is "device 0x80").
• Network boot (the boot device may not be a disk drive at all)
• There are tools out there that do strange things. For example, Etherboot will allow a computer to boot from network, download a disk image from network into RAM, then hook "int 0x13" and pretend that disk image is an actual disk drive.

For network boot (PXE) it's relatively easy for the boot loader to get the network card's MAC address, the DHCP server's IP address, and the TFTP server's IP address (and the computer's own IP address, which can be useful to avoid doing the slow DHCP thing a second time). If the boot loader doesn't get this information and tell the OS/kernel; then there may be 2 or more network cards with no way to tell which was used, and there's a (small) chance that the DHCP and TFTP server may have been changed or be offline by the time the OS asks again. Worst case might be 2 or more network cards where each is connected to a separate sub-net, where each separate sub-net has its own DHCP server and TFTP server. In that case, because the OS can't figure out which network card it can't figure out which DHCP server or which TFTP server either.

For booting from disk, the more information you've got the easier it is to guess right. The single biggest source of information is the BIOS's "IBM/MS INT 13 Extensions - GET DRIVE PARAMETERS" function (and if that isn't supported, then try the older "DISK - GET DRIVE PARAMETERS" function. However, (in case the boot device is an emulated device) you will need to try the "Bootable CD-ROM - TERMINATE DISK EMULATION" first - if the boot device wasn't emulated this will fail, and if it was an emulated device then this will tell you the device number for the real device.

If all that worked right, you should know which disk drive on which controller on which device on which bus. If it doesn't work right (e.g. "int 13 extensions" weren't supported) you might only have the number of heads, sectors per track, cylinders; and you'll need to hope that they make sense and that there's only one disk drive with those parameters.

Note: The most reliable fallback (for when information from the BIOS wasn't enough) that I can think of is for the boot loader to generate some sort of "unique" time-stamp and write that time-stamp to the boot device (using the BIOS) and give the OS that time-stamp. Then the OS can search for the disk/partition that has a matching time-stamp. This solves the "cloned disk" problem, and might also be useful for other reasons (e.g. when the OS is has booted it could write zeros to the area of the disk used for the unique time stamp, so that the boot loader can detect when the OS has failed to boot properly). Of course this won't work for read-only devices; and I'm not too sure that I'd want the boot loader to avoid write to the disk if it can't be avoided.

Mostly what I'm saying is that detecting which device you booted from is complicated, and needs to be a built-in part of the boot loader, and can't be done by the OS alone (without help from the boot loader and BIOS).


Cheers,

Brendan

[SpyderTL]

There's no way to determine how "BIOS device numbers" were mapped to actual devices. A lot of BIOSs have options change which disk is the "first" disk.

This is what I was afraid of.

For booting from disk, the more information you've got the easier it is to guess right. The single biggest source of information is the BIOS's "IBM/MS INT 13 Extensions - GET DRIVE PARAMETERS" function (and if that isn't supported, then try the older "DISK - GET DRIVE PARAMETERS" function. However, (in case the boot device is an emulated device) you will need to try the "Bootable CD-ROM - TERMINATE DISK EMULATION" first - if the boot device wasn't emulated this will fail, and if it was an emulated device then this will tell you the device number for the real device.

If all that worked right, you should know which disk drive on which controller on which device on which bus. If it doesn't work right (e.g. "int 13 extensions" weren't supported) you might only have the number of heads, sectors per track, cylinders; and you'll need to hope that they make sense and that there's only one disk drive with those parameters.

That is what I was looking for. I may try to use this, in the short term.

My problem is that I haven't finished my custom file system, so I've been using a simple "load everything in this list into memory" table until I implement a real file system. This is just now starting to come back to haunt me, as loading "everything" is now causing problems for my 16-bit boot loader.

What I should be doing is working on my file system and the logic to load things into memory, dynamically, at run time. But every time I start, I get a headache, and give up to work on cooler stuff..

So, for now, I've just removed some "components" that weren't being used to reduce the 16-bit memory usage. I'll just have to revisit this problem in a few weeks...

Thanks for all of the info, guys.

[Brendan]

Hi,

My problem is that I haven't finished my custom file system, so I've been using a simple "load everything in this list into memory" table until I implement a real file system. This is just now starting to come back to haunt me, as loading "everything" is now causing problems for my 16-bit boot loader.

What I should be doing is working on my file system and the logic to load things into memory, dynamically, at run time. But every time I start, I get a headache, and give up to work on cooler stuff..

What I do is, I cheat.

Instead of designing the native file system, I just say "the first N sectors of the partition are reserved for boot stuff", so I can design the file system around that later on.

Also, instead of loading many files in the boot loader, I just combine all of them into a single "boot image" file (basically just putting a header at the start of each file and then appending the whole lot together), and write code to find the individual files within that "boot image" so that it's like a simple (read only) RAM disk.

Of course then I get carried away and over-complicate it beyond recognition; but that's optional.


Cheers,

Brendan