OSDev.org

Hey all,

I've been pretty inactive for a while, but slowly getting back into my os coding. I know from my early works how much trouble I had with my boot loaders. The tutorials and information out there are pretty good at explaining the very basic concepts but there isn't much out there in the way of a comprehensive guide to all the things you should/could do as part of your boot-loaders.

My 2 stage loaders were designed with a few assumptions specific to my requirements in that I do not want to support partitions and I wanted to have a single unified boot-code that could be deployed to USB(floppy emulated or hdisk emulated) as well as bootable cd, proper hdisk and floppy disk and boot correctly from all of these scenarios. I didn't investigate network boot as it's not something I require.

I did do some investigations into EFI/UEFI and although it has some promising features it looks to me like it's still not widely adopted enough to be that useful (correct me if i'm wrong on this?)
I know that the MAC uses UEFI (but then they have exclusive control of their mainboard/platform).

I think there would be some value in compiling a full detailed guide with all our thoughts/findings/samples for what should go into a "modern" boot loader setup. I know a lot of ppl will be using GRUB and other ready-to-go setups, but this is after all all about rolling your own. We could cover many of the nuances of usb boot that I've found over time in home some mainboards require an FDD-emulated boot sector to have BPB and others not, as well as TPM modules. Much is already documented in sub tutorials and we could just reference those as needed like unreal mode/a20 gate/vbe/vga hw/pmode/longmode etc..

The end result should be something along the lines of.. "what are all the various steps and stages that need to be taken into account to boot a modern OS like win7/lux" but with the specifics left out like choice of FS (ie how to locate files under fat16/fat32 etc).
And we could (hopefully) compile a near-complete list of all the subtle nuances like fdd-emulation differences in bioses / 13h extended functions work sometimes/not others / can u use 13h ext. functions to load files of el-torito or should you load them use iso9660 spec. etc

Anyhow, it's just a thought and something I'd be happy to contribute all the code and notes i've compiled thus far into if anyone else is keen.

I like your idea. I have made a lot of boot loaders in the past, but I think the tutorial should be kept simple, after all it is for new developers.

I'm willing to help there, if everyone else thinks it is a good idea.

However, to state on the other side of the case, there are a lot of people who like theory tutorials, (me being one) because it gives the concepts to apply, and thus what you figure out is more your own code. But I think there are enough people who would appreciate this.

David

Not just good, but great idea!

I'm trying to make a more robust and practical CD/DVD-ROM bootloader now. It was hard to ever get started because nearly everyone uses floppies or a pre-existing bootloader. What I have so far definitely isn't tutorial worthy, but I'll be more than happy to share whatever I learn when its in better condition (which will be soon).

I think moving in this direction overall is a great idea. When you go buy Mac OS or Windows, you get a CD/DVD, not a floppy! Not being critical of anyone (as I have NO room to be), but I think CDs, DVDs and USB drives are much more elegant, and of course, grant you much more space. My computer doesn't even HAVE a floppy drive, lol. I think it's an all around more modern and robust solution for deployment, and can often be required for certain projects (like my [non-desktop] research project). I say, out with the 80s and in with the 90s! Well, at least some more tutorials and collaboration on alternative deployment options!...

Greetings,

johnsa wrote:I've been pretty inactive for a while, but slowly getting back into my os coding. I know from my early works how much trouble I had with my boot loaders. The tutorials and information out there are pretty good at explaining the very basic concepts but there isn't much out there in the way of a comprehensive guide to all the things you should/could do as part of your boot-loaders.

There are actually way too many things you could do. Here's a list of what my boot loader does:

Master Boot Record (512 bytes):

• Relocate itself to 0000:0600.
• Search for the active entry in the partition table (Note: test BYTE [si], 0x80 will tell you this, most MBRs I've seen use cmp instead).
• Check if the active entry uses LBA (unofficial) or CHS (Note: test BYTE [si], 0x01 will tell you this, but you might drop this support as it is unofficial).
• Get the disk geometry if it is a hard disk (Note: it should be possible to use a MBR on a floppy, but nobody really does this).
• Check for the availability of the extended read/write functions.
• Load the actual boot sector and jump to 0000:7C00 (Note: dl is reset to the disk drive and si to the start of the partition table).

Considerations:

• Support extended partitions (Note: the lazy way could simply be booting the extended partition's MBR).
• Support GPT and perhaps (U)EFI.
• Support your own partition table, as it is theoretically possible, however all other operating systems will not be able to support it.
• Set up VGA properly and display error messages and/or use the PC Speaker.

Part 1 (512 bytes; FD and HD):

• Nullify cs (a.k.a. jump to 0000:7C00).
• Set up the other segments and set up the stack at 0000:1000.
• Get the disk geometry if it is a hard disk.
• Check for the availability of the extended read/write functions.
• Load the second part and jump to 0000:1000 (Note: dl is set to the disk drive).

Some people just do everything in the bootsector: load the kernel, enable A20, set up the GDT, enable protected mode, execute the kernel, etc. Some other things you could is set up VGA properly or get the amount of RAM installed, however, you should keep in mind that you only have 512 bytes loaded for sure.

Part 1 (512 bytes; CD):

• Nullify cs (a.k.a. jump to 0000:7C00).
• Set up the other segments and set up the stack at 0000:1000.
• Check if the CD is booted as non-emulated.
• Relocate the second part and jump to 0000:1000 (Note: dl is reset to the disk drive; CD-ROM bootsectors are generally 2kB).

Part 2 (1.5K usually; filesystem-specific):

• Get the disk geometry if it is a hard disk.
• Check for the availability of the extended read/write functions (Note: actually optional for CDs).
• Parse the filesystem and search for the third part.
• Load the file off the disk and jump to 0000:1600 (Note: dl is set to the disk drive).

My boot loader is designed like this so other filesystem loaders could be concatenated with part 1.

Part 3 (a file on the disk):

• Test if the CPU is actually supported (80386+ and required instructions).
• Check if the system is a multi-processor system and what processors are on-board and also check their co-processor.
• Ensure that CPUID is enabled if it is available (Note: required for Cyrix and NexGen processors, if I remember well)
• Check for the availability of long mode, PAE, PSE36, etc.
• Set up a memory map using int 0x15.
• Ensure VGA is set up properly (or set up VBE).
• Load the 32-bit kernel or the 64-bit kernel (and additional modules) off the disk depending on whether long mode is properly supported or not (TODO: separate filesystems drivers to use in the boot loader) (Note: unreal mode, switching in between real and protected mode).
• Enter protected/long mode and execute the kernel (Note: includes setting up A20, GDT, etc.).

This ends up in supporting three types of disks: disks without partitions (FD, HD, USB, etc.), disks with partitions (HD and USB mainly, but also USB, CD, etc.) and CD's/DVD's.

Network booting isn't actually supported because I don't need it at the moment. All I know about it is that you could either use the PXE API or write your own drivers for each network device.

Boot loaders are really specific to your requirements, like I said before some people might just do everything in the 512 bytes they got and then just set up everything else in the kernel.

johnsa wrote:I did do some investigations into EFI/UEFI and although it has some promising features it looks to me like it's still not widely adopted enough to be that useful (correct me if i'm wrong on this?)
I know that the MAC uses UEFI (but then they have exclusive control of their mainboard/platform).

I only know some/most Itanium/Intel servers support it as well as the x86(-64) Macintosh platform.

johnsa wrote:I think there would be some value in compiling a full detailed guide with all our thoughts/findings/samples for what should go into a "modern" boot loader setup. I know a lot of ppl will be using GRUB and other ready-to-go setups, but this is after all all about rolling your own. We could cover many of the nuances of usb boot that I've found over time in home some mainboards require an FDD-emulated boot sector to have BPB and others not, as well as TPM modules. Much is already documented in sub tutorials and we could just reference those as needed like unreal mode/a20 gate/vbe/vga hw/pmode/longmode etc..

The end result should be something along the lines of.. "what are all the various steps and stages that need to be taken into account to boot a modern OS like win7/lux" but with the specifics left out like choice of FS (ie how to locate files under fat16/fat32 etc).
And we could (hopefully) compile a near-complete list of all the subtle nuances like fdd-emulation differences in bioses / 13h extended functions work sometimes/not others / can u use 13h ext. functions to load files of el-torito or should you load them use iso9660 spec. etc.

Also, some BIOSes support booting USB as HD or USB as CD as well as Zip100, LS120, etc., although Zip100 and such aren't really worthwhile to actually support. I am not sure if there actually BIOSes which support direct USB though. Most articles on the Wiki covering boot loading provide a lot of information already, but seem to missing some things here and there like perhaps USB-booting and proper A20 setup.


Regards,
Stephan J.R. van Schaik

To get started with here are the first 6 steps in my 1st stage loader. Shall we review this confirm accuracy and refine. Then We can update the doc. and move on.
Once it's all complete we add to wiki?

; 0000:7E00 - 9000:FFFF [ Free use memory ]

You forgot the EBDA - if you're paranoid you shouldn't load above 8000:0000 without checking. I haven't personally seen cases myself where 9000:7FFF as a limit doesn't work.

Nice spot! thanks, I'll add that in. Any other thoughts? These are purely my observations that I've seen to work sofar, but we all know how fickle pc hardware is especially with a complete lack of standards adoption
I'm not 100% sure if my thoughts around my BPB cheating/no partitions etc are legitimate and will work in all cases.. This is obviously purely for my design and others may want to stick to a traditional MBR and file-systems like fat16/32.

Just on the EBDA note, In my stage 2 loader I do the following:
giving the segment address of the ebda.

which we'll come to later (for now in stage 1 we just assume not to load over 8000:0000). I actually think I did have an issue with this once before on some machines, which oddly only turned up when i booted them from CD(fdd emulation) but from USB it worked. Obviously some data was getting over-written that affected something specific to the EBDA/emulation when trying to then read sectors via INT13 ext. This happen when loading kernel image in stage 2 and I moved it down to load at 7000:0000.

I also assume the EBDA to be max 4kb in size? correct? (Else it would run into video memory). Safe way to work would be to get the EBDA address , subtract from A0000 and not use anything in that range?

For my machines It always seemed to be located at the following address:
0009FC00 or segment of 0x9fc0:0000.

Perhaps we should make some quick notes about selection of these load addresses, is there a std. we should all use, like 7c00->move to 600h->load stage2 to 7c00->load kernel in blocks somewhere in mem and then transfer it to hi-mem(1mb)?

johnsa wrote: - Real Floppy Disk
- Real Hard Disk
- USB Flash Stick (under FDD emulation from BIOS)
- USB Flash Stick (under HDD emulation from BIOS)
- CDROM/DVDROM (either natively or via FDD image emulation)

Basically USB is supported via FD-emulation, HD-emulation and CD-emulation and CDs and DVDs are generally supported via FD-emulation and HD-emulation as for emulation. CDs and DVDs should also be supported natively by most BIOSes nowadays, not sure if the same counts for USB.

johnsa wrote:USB boot is achieved through emulation. The BIOS will either assume the USB Stick to be a floppy or hard-disk
and emulate accordingly both in terms of what is passed into the boot-loader in the DL register as well as the
INT 13h disk functions. It is also possible that USB sticks will appear as ZIP drives but this is something
we're not interested in and can ignore.

Zip100, LS-120 and LS-240 are assumably supported as floppy disks or hard disks.

johnsa wrote: 1) I don't support or believe in partitions. In 20 years and hundreds of machines I've never needed or used one.
My policy is if you need multiple logical volumes, install multiple physical volumes. My OS is not intended to
support multi-boot.

IMO a Wiki article should tend to be objective. However, it might be easier for a beginner to not support the partition table or GPT. What your OS is intended or not intended to support doesn't matter, the Wiki article is targeted at (new) OS developers.

johnsa wrote: 2) There is no manadatory requirement to store the OS loaders or kernel binary within the file-system (confirm?)
I chose to store these linearly at the beginning of whatever device is in question and the relevant file
system information comes directly after that. There maybe some pros/cons to this?

Basically, you could store the binary everywhere on the disk, as long as you know how to get it into memory.

johnsa wrote: 3) Because I don't use/need the BPB and my boot code is intended to be unified it will be present even in the
boot-sector of a hard-disk/cd etc.

You don't need the EBPB (FAT) though (assuming that this is present, as you mentioned it; I'm assuming it is not needed, but it may be by some BIOS, but that would be biased). BPB is simply required for compatibility, but may be dropped as well.

johnsa wrote: ; 0000:7E00 - 9000:FFFF [ Free use memory ]

If all that memory is actually available and if you aren't using the EBDA, like Combuster mentioned.

johnsa wrote: Some BIOSes will load the bootsector (512bytes) to 07c0:0000 while some will load it to 0000:7c00.
The flat binary assembler source will have been create to be at one of these by use of either:
org 0h (assuming the segment is 07c0)
or
org 7c00h (assuming the segment is 0 and the offset is 7c00)
Either way you need to ensure this is always fixed.

Virtually, a BIOS could load it to any segment:offset address as long as it represents 0x7C00, but that would be a BadIdea(tm).

johnsa wrote: The possible values the BIOS will give us are:
00h (First floppy)
01h (Second foppy)
80h (First hard disk)
81h (Second hard disk) and so on..

Basically, the BIOS could provide you any disk drive it wants to. If it is masked with 0x80, it's safe to assume that it is a hard disk, otherwise it's a floppy disk.

johnsa wrote: ;=============================================================================================================
STEP 4 (Transfer code location and jump)
;=============================================================================================================
Move the first stage loader to a safe location and continue execution.
The reason I previously selected to use 07c0:0000 instead of the offset model is that it makes this
step a bit easier. I can copy the entire boot-loader code (512 bytes) to a new segment address
and then use a label directly after the far jump as the same offset for that jump (as the labels
offsets relative to the segment remain unchanged, this applies to variables etc as well).
I chose 0060:0000 as the destination (this is the address used by DOS 1.x apparently) and from looking
at a memory map after boot this space should be safe.

xor si,si ; DS:SI = begin of original bootloader (7c00:0000).
mov di,600h ; ES:DI = destination (0000:0600).
mov cx,100h ; Copy 256 words (512 bytes).
rep movsw ; Copy bootloader.
db 0eah ; Hardcode force far jmp.
dw offset bootloader_exc0,0060h ; new code addr = (0060:bootloader_exc0).

That's only required for the MBR actually, unless this is your intention.

johnsa wrote:Just on the EBDA note, In my stage 2 loader I do the following:

Code: Select all

	;--------------------------------------------------------------------------------
	; Locate and Store BIOS EBDA Address.
	;--------------------------------------------------------------------------------
	mov ax,40h
	mov es,ax
	movzx eax,word ptr es:[0eh]
	shl eax,4
	mov fs:[BIOS_EBDA_ADDRESS],eax

giving the segment address of the ebda.

which we'll come to later (for now in stage 1 we just assume not to load over 8000:0000). I actually think I did have an issue with this once before on some machines, which oddly only turned up when i booted them from CD(fdd emulation) but from USB it worked. Obviously some data was getting over-written that affected something specific to the EBDA/emulation when trying to then read sectors via INT13 ext. This happen when loading kernel image in stage 2 and I moved it down to load at 7000:0000.

I also assume the EBDA to be max 4kb in size? correct? (Else it would run into video memory). Safe way to work would be to get the EBDA address , subtract from A0000 and not use anything in that range?

For my machines It always seemed to be located at the following address:
0009FC00 or segment of 0x9fc0:0000.

Perhaps we should make some quick notes about selection of these load addresses, is there a std. we should all use, like 7c00->move to 600h->load stage2 to 7c00->load kernel in blocks somewhere in mem and then transfer it to hi-mem(1mb)?

This might be of use.


Regards,
Stephan J.R. van Schaik.

Thanks for the detailed response!

I've made some updates regarding the boot emulation/devices, changed the wording and info to avoid being biased towards my design but keeping it simple without a file-system/mbr/partitions. These should be trivial to add if you have the file-system format and plan to do so - and it's all available on the wiki.

I'm leaving the EBPB/BPB bit in as I know some machines I have won't boot a USB at all without that there, even if it's not used. So as you say for compatability I'll keep it - and based on the agreement that it doesn't affect the loading of the
boot loader from any other device type it's of no harm (part of the goal to have a single source / unified loader).

Updated re the EBDA/wiki mem map.

Updated regarding the seg/ofs combinations but basically results in the same thing that you need to force it to a known (IE: compatible with the way you assembled the source).

Updated regarding the code-relocation only being necessary for MBR.

johnsa wrote:1) I don't support or believe in partitions. In 20 years and hundreds of machines I've never needed or used one. My policy is if you need multiple logical volumes, install multiple physical volumes. My OS is not intended to support multi-boot.

Objection.

1) Installing multiple physical volumes is usually not an option with laptops.

2) Whether or not partitions are supported is not up to the filesystem. A filesystem resides inside a partition.

3) If you do not support partitions, you are effectively asking people to completely wipe their system before they can give your OS a try. Freeing up a partition is easy, when compared to the task of backing up (and restoring) hundreds of gigs of data just for a peek-and-try.

3a) Most laptops have hidden partitions that hold recovery data. Not-so-experienced users might wipe out their fallback scenario by accident.

4) It's the task of the OS to enable users. You don't believe in partitions, I never owned a hard drive that had less than three of them (including at least two different OS installations). I never set up a system with less than two partitions (because seperating the OS and the data is a real nice win, especially for backup purposes). You can go ahead and use only a single partition / OS on your system, but not being able to use multiple partitions / OSs on my system is a showstopper for me (and many, many others).

5) I suspect you don't believe in partitions because you find them difficult to support.

Bottom line, anything that goes as "tutorial" must support partitions, IMNSHO.

I agree that many people like and use partitions. They're not an issue to support really, it's only a little bit of extra work to have an mbr and select a bootable partition. There's quite a lot of info and examples on how to do this.
I see this in two ways, for my OS.. I don't need them, for the tutorial it adds additionally complexity which once the basic "no-partition/no-fs" model is working for you adding the file-system/partition support is relatively easy and well
documented. My main goals here are:
1) the concept of unified boot-loader
2) getting your own boot loaders up and running as quickly as possible (you can expand later with your own requirements/features)
3) ensuring that these boot loaders work under all conditions and on as many systems as possible (if not all)

So many of the boot code sample i've seen are FD specific, too basic. And with most they don't deal with all the little quirks and subtleties. If one never took their boot-loader past these tutorial stages it's unlike that it would actually boot
on more than 50% of the machines out there while the linux/win guys manage to get everything(mostly) booted.
While we're not trying to compete with them (maybe some are) it would be nice to know that we're following the same steps to get to our
kernel (the real meat) up and running on all machines.

A quick question then in terms of native CDROM/DVDROM support.

Sectors are not 512byte (usually 2kb for CD?) so would the BIOS load a 2kb sector and expect the signature to be at the end of the sector or still at the end of the 512 byte block?
Would one use the normal int 13 read functions?
Would it then NOT supply a drive number of 00h or 80h but the "actual" drive number, like 81h etc?
How to check the CD is running natively and not under emulation?

johnsa wrote:3) ensuring that these boot loaders work under all conditions and on as many systems as possible (if not all)

All will not be possible, unless you treat x86(-64) as the only architecture around and even then it is still impossible as not all x86(-64) systems have a BIOS.

johnsa wrote:So many of the boot code sample i've seen are FD specific, too basic. And with most they don't deal with all the little quirks and subtleties. If one never took their boot-loader past these tutorial stages it's unlike that it would actually boot
on more than 50% of the machines out there while the linux/win guys manage to get everything(mostly) booted.

People who use Linux generally use GRUB as their boot loader or perhaps LILO, ELILO or ISOLINUX. The only general difference I've seen between most "basic" bootsectors and LILO is their Award BIOS bug fix, however, there might still be more that I haven't actually seen.

johnsa wrote:A quick question then in terms of native CDROM/DVDROM support.

Sectors are not 512byte (usually 2kb for CD?) so would the BIOS load a 2kb sector and expect the signature to be at the end of the sector or still at the end of the 512 byte block?

The bootsector should commonly be 2kB on a non-emulated CD and 512B on an emulated CD.

This contains some interesting information about CD sectors.

johnsa wrote:Would one use the normal int 13 read functions?

In my boot loader I check if the INT 0x13 extensions (INT 0x13; AX=0x4100) are available and if it is a hard disk, otherwise I'll just use the "normal" INT 0x13 functions.

johnsa wrote:Would it then NOT supply a drive number of 00h or 80h but the "actual" drive number, like 81h etc?

Not sure what you're exactly asking here.

johnsa wrote:How to check the CD is running natively and not under emulation?

The easy/lazy way is to put a signature at the end of the 2K sector so you can simply check if 0x83FC (assuming you're using a DWORD signature) is equal to the signature.


Regards,
Stephan J.R. van Schaik.

I would hope that unless it's itanium/UEFI it would have a BIOS and then as such our series of steps should work to boot it
All the non server x86-64 machines ive seen sofar are still BIOS, and most of the servers too.

In my boot loader I check if the INT 0x13 extensions (INT 0x13; AX=0x4100) are available and if it is a hard disk, otherwise I'll just use the "normal" INT 0x13 functions.

I do the same. (Although I've often noted funnies with this, where the extensions are reported but then sector reads fail.. but do work with normal int13) .
What I was referring to in terms of the non-emulated CD was if the normal int 13 functions still read 2kb sectors correctly, or do you/should you use the extensions if available and on a native CD?

The easy/lazy way is to put a signature at the end of the 2K sector so you can simply check if 0x83FC (assuming you're using a DWORD signature) is equal to the signature.

I like this..

johnsa wrote:I do the same. (Although I've often noted funnies with this, where the extensions are reported but then sector reads fail.. but do work with normal int13) .
What I was referring to in terms of the non-emulated CD was if the normal int 13 functions still read 2kb sectors correctly, or do you/should you use the extensions if available and on a native CD?

Basically you could assume that INT 0x13 extensions are always available when booting a CD natively as the El Torito standard was made when the INT 0x13 extensions already existed. You could however still check if the INT 0x13 extensions are still available anyway. The normal INT 0x13 functions should still work as the CHS limit is only 8GB. Also, you might want to write a floppy bootsector for booting the CD when the BIOS doesn't have proper support for it, although that is completely optional as most modern systems should support native CD booting.


Regards,
Stephan J.R. van Schaik.