PC booting -- so complex!

[eekee]

I'm not complaining. It's just that working with and writing about partitions caused me to reflect on it. For starters, the PC is an anomaly. 10 years after its launch, it was normal to have some non-volatile memory for operating system use. Plan 9 used it to store a security key. At some point, probably earlier than I think,[1] it became normal to have enough non-volatile memory on board to store a kernel. All the bootloader had to do was copy the kernel from slow ROM to fast RAM and jump to it. There's a great consequence of this: the kernel can then use its normal drivers to bring up everything else. When building a new architecture, you don't have to write 2 sets of drivers.

[1]: I thought it was by the end of the 90s, but then I remembered the Atari ST, launched in 1984 or 5, had its entire operating system in ROM. It's only rewriteable flash memory which took time to develop.

It might seem that keeping the kernel in onboard flash is bad for updates, particularly when you remember the limited lifespan of flash memory and the fact that if it's direct-addressed, (which it would be in this case,) you can't do wear leveling. This is solved with kexec. Of the systems I've seen, Plan 9 and the Sharp Zaurus would both load a kernel at boot which would then kexec the production kernel. I'm slightly ashamed to say that having encountered Plan 9 only on PCs, I thought its boot process strangely "impure", but now I realise it's just about only having to write drivers once.

So I guess I'm not talking about PC booting at all in this post. In fact, I have for years been thinking of loading Coreboot into my Thinkpad T440, turning its boot process into something more modern. I just didn't realise until now that it could simplify booting in a similar way to using a good bootloader.

[nullplan]

the kernel can then use its normal drivers to bring up everything else.

That is indeed what the Coreboot people are aiming at. They want to have a full Linux kernel in ROM, so that Coreboot only needs to enable RAM, decompress the kernel into the RAM, then run it. Then the boot kernel already has support for all the weird hardware and all the states it can be in, and it can load whatever actual kernel is requested directly, and then kexec into it. Obviously for Windows, they would have to modify kexec to chainload stuff.

That would also avoid the issue of having the bootloader have different capabilities from the OS. I had some fun building my most recent PC, which has an M.2 slot on the mainboard, but the BIOS has no capability to boot from it. So I have installed the boot loader on one of the SATA drives, but the OS itself is on the M.2 drive.

[xeyes]

Sorry if I didn't quite get your main point , but if this is the goal

When building a new architecture, you don't have to write 2 sets of drivers.

PC's FW (BIOS) design is, might I say, an almost ideal solution to the problem.

The idea of having all hardware details in FW and exposing a standard software defined interface between the FW and the kernel, is one of the greatest idea of abstraction I've ever seen. Although I'm not sure whether this idea was from CP/M or was a common thing among ROM based monitors back then.

Not only did this make the early OSes and booter programs easily portable across different hardware systems, the whole field of "OS development" is still benefitting from it:

• 1. Early OSes written against this software interface are not only portable between hardware systems back then, they are forward compatible and still highly portable even today, more than 40 years after they were written. For example, to implement a VM that can run a copy of CP/M as is, you only need to implement the well defined BIOS calls, no need to worry about all the details of the hardware components of that era.
  
  2. Even highly advanced OSes, such as Windows 7, still benefit from, and very much depend on this interface.
  
  3. IMO, the very philosophy of one hardware system paired with one specific FW which implements all needed drivers and expose a standard and open SW interface, that can support many different HW agnostic but standard interface compliant (aka can call BIOS calls) OSes, is what enabled OS dev as a hobby. Anyone can spend 3 min and print a character or even a string to 0xB8000 as a starting point, as well as debugging using the same when more complex debugging tools don't work.

The alternative could be much much worse:

• 1. Every OS would have to have all detailed knowledges (native drivers) of a hardware platform, to even attempt to boot.
  
  2. They'd be either outdated when the next generation of hardware arrives, or when the user decides to upgrade one of the hardware components. "forward compatible for 40 years" would be simply unimaginable.
  
  3. The very philosophy of one hardware system, paired with a dumb FW that implements nothing useful but a fancy splash screen, that supports only the OSes that already have native drivers for each and every hardware systems in existence, is something that can kill OS dev as a hobby right away.
  
  "OS dev as a hobby? Like your uncle Bob who spent 3 years mocking around his '*acintos* M* computer' and was finally able to print "hello world"?" If anyone thinks that he or she can do better than uncle Bob, try booting your own code on a latest iphone and see how that goes =)

writing about partitions ... so complex!

Partitions could of course become very complex, just as attempting to write native drivers for every hardward platforms out there could become very complex.

But in the "standard and simple" (BIOS) way, aren't partitions as easy as reading an array near offset 0x1b8 of the disk? IMO even the extended partitions could be safely ignored as 4 partitions seem plenty for a hobby kernel, if not, add another disk and now you have 8

[eekee]

@nullplan: Yes, exactly! Though I'm pretty sure the Coreboot people succeeded a long time ago.


@xeyes: Oof! Have you read any of the wiki pages dealing with the PC BIOS? Many attempt to dissuade you from having anything to do with it. The BIOS page doesn't directly discourage its use, but describes a lot of problems. For instance:

Unfortunately, RBIL does not clearly indicate which BIOS functions are "generic" (in some sense). That is, the ones that are always available, and that everyone uses. Partially this is because the "standard" BIOS functions grew over time, so if you go back far enough in time you can usually find a computer that does not support almost any specific BIOS function. But there is definitely a set that is commonly used in most current OSes.

It then lists functions, the majority of which would not have been available when CP/M was ported to the PC. Correspondingly, I expect many functions CP/M used are not likely to be available on later hardware.

The Real Mode OS Warning page is more direct, opening with this:

Using BIOS functions (and "avoiding" the need to write drivers) can seem easier for beginners that don't know any better, however it makes everything much much harder in the long run. This makes it an alluring trap for the unwary.

Sections of this page include BIOS Does Not Support Any Device Well, which explains that the PC BIOS does a poor job of supporting even hardware as ancient and simple as serial ports. BIOS disk support is synchronous, meaning you can't implement a write-back disk cache. I was thinking of using the BIOS for my first OS, but that one detail alone has dissuaded me. (On reflection, the serial port section alone would also be enough to dissuade me.) It's worth reading the entire page, I think.

If the BIOS was so important to Windows, how was Windows NT ported to Alpha? Windows 7 is a version of Windows NT; all 21st-century versions of Windows are except perhaps Windows ME. And isn't there now Windows for ARM? (I don't mean WinCE.)

But what about the concept of "a good BIOS"; firmware containing a generic set of drivers which could be used in any OS? Generic drivers come up occasionally, usually by someone new to OS dev. I've brought it up myself in the past. The answer is always that the design of the drivers depends on the design of the OS, so generic drivers would be of little use. At best, they'd degrade performance of operating systems which don't match their design.

I didn't mean partitions are complex, I meant booting a PC is complex. The MBR code has to move itself, then it has to load the PBR, then the PBR with very little space for code has to parse a filesystem and load the kernel, or if not the kernel then at least another bootloader stage. This latter part causes problems with filesystem and bootloader design. If you attempt to avoid the problem by hardcoding the location of the kernel into the bootloader, synthesizing the bootloader as LiLo did, then you risk running into the problem that the BIOS may present the disks in different order to the OS so the synthesized bootloader has the wrong numbers. And there's another problem in memory: some OSs require the PBR and all other bootloader stages to preserve the loaded MBR all the way to the kernel so it can know which partition it booted from. If you get chainloaded from an MBR designed for some other OS, how do you know where it moved itself to? You can get that from the DS:SI pointer, but what happens if you want your kernel at that location? Problems problems problems, NONE of which exist when dealing with bare hardware. I'm sure someone's going to reply to this with a standard address MBRs should copy themselves to, but that's another piece of knowlege to know on top of all the things you need to write actual drivers for your actual OS. It's no wonder the wiki attempts to dissuade people from writing their own bootloader.

As for targetting a phone, I guess we'll find out if I keep at it because I'm hoping to target my old Samsung S3 and S5 phones. The S5 had very good support from postmarketOS, so there's kernel source to read there. The S3 had less complete support but I think screen and USB worked.

[Octocontrabass]

Correspondingly, I expect many functions CP/M used are not likely to be available on later hardware.

I expect all functions CP/M-86 used are available on later hardware. After all, MS-DOS used those same functions.

I'm sure someone's going to reply to this with a standard address MBRs should copy themselves to, but that's another piece of knowlege to know on top of all the things you need to write actual drivers for your actual OS.

There is a "standard" address most MBRs will copy themselves to, but as far as I'm aware this is one aspect of the PC boot process that was never actually standardized. Of course, since there is no single document that collects all of the BIOS standards, it's possible I just haven't found the right standard yet.

[Ethin]

Doesn't UEFI solve these problems though? I mean, sure, it introduces new ones, but most of the problems involving partitions and FSes are pretty much solved by UEFI unless you use a custom FS in your boot partition.

[Octocontrabass]

UEFI definitely simplifies things. It automatically loads a relocatable PE executable from a FAT-formatted partition. If you want to load something from a custom filesystem, you can write your own filesystem driver without worrying about how to squeeze it into 512 bytes. All of the APIs you're likely to use in a bootloader are specified in a single document.

But good luck figuring out how to get the handles you need to use those APIs.

[Ethin]

UEFI definitely simplifies things. It automatically loads a relocatable PE executable from a FAT-formatted partition. If you want to load something from a custom filesystem, you can write your own filesystem driver without worrying about how to squeeze it into 512 bytes. All of the APIs you're likely to use in a bootloader are specified in a single document.

But good luck figuring out how to get the handles you need to use those APIs.

All of those APIs for driver dev are also specified in the UEFI spec as well. A UEFI driver doesn't use anything unique or special in terms of APIs; the drivers use the same protocols as anything else. The only time I can see this getting complicated is if you decide to (for whatever reason) write a driver that runs before any other protocols are available or something.

[vvaltchev]

UEFI definitely simplifies things. It automatically loads a relocatable PE executable from a FAT-formatted partition. If you want to load something from a custom filesystem, you can write your own filesystem driver without worrying about how to squeeze it into 512 bytes. All of the APIs you're likely to use in a bootloader are specified in a single document.

But good luck figuring out how to get the handles you need to use those APIs.

All of those APIs for driver dev are also specified in the UEFI spec as well. A UEFI driver doesn't use anything unique or special in terms of APIs; the drivers use the same protocols as anything else. The only time I can see this getting complicated is if you decide to (for whatever reason) write a driver that runs before any other protocols are available or something.

Maybe Octocontrabass just wanted to point out how, in UEFI in general (nothing specific to drivers), sometimes is tricky to figure out how to get a handle for something you want. It's typically a multi-step procedure that requires quite some digging in the spec, unless you've already done it before, of course.

[rdos]

UEFI definitely simplifies things. It automatically loads a relocatable PE executable from a FAT-formatted partition. If you want to load something from a custom filesystem, you can write your own filesystem driver without worrying about how to squeeze it into 512 bytes. All of the APIs you're likely to use in a bootloader are specified in a single document.

But good luck figuring out how to get the handles you need to use those APIs.

I don't think UEFI solves anything significant, other than introducing the possibility to lock-down PCs.

The boot code is not simpler, and requires a cross compiler that can emit the special PE format. The traditional PC boot sector can be written in assembly, and so only requires a x86 assembler (if you don't want to hand-code).

The idea that OS kernels are assumed to use PE format is not much of a help either. It just means I need to build an intermediate stub that must use PE format that jumps to the real OS image. With BIOS boot, I only need a second stage loader, which can be coded in assembly as well, and so I don't see how anything got easier.

That UEFI transfers control in protected mode or long mode is not of much help either. This only means that for long mode (a majority of today's UEFIs) I need a long mode stub that can switch to protected mode to be able to load the real OS kernel. Thus, I also need a long mode cross compiler which isn't exactly making things easier.

Next problem is to create anything that is visible. With BIOS and text-mode this is simply. You just write char + attribute in the video buffer starting at 0xB8000. For UEFI, you need to handle various display resolutions and color organization schemes. Characters must be rendered through a font.

That UEFI contains a lot of device drivers is of no use to the OS since these are only available up until control is released to the OS. Actually, BIOS is of more help here since it is possible to use BIOS inside the OS (even if generally not a good idea).

So, what exactly is better?

[Octocontrabass]

Maybe Octocontrabass just wanted to point out how, in UEFI in general (nothing specific to drivers), sometimes is tricky to figure out how to get a handle for something you want. It's typically a multi-step procedure that requires quite some digging in the spec, unless you've already done it before, of course.

Yep, that's exactly it.

I don't think UEFI solves anything significant, other than introducing the possibility to lock-down PCs.

It definitely solves the problem of digging through dozens of specification documents to find the one that covers what you need. But since you mentioned locking down PCs, I'd like to hear how you would harden firmware and bootloaders against rootkits.

The boot code is not simpler, and requires a cross compiler that can emit the special PE format. The traditional PC boot sector can be written in assembly, and so only requires a x86 assembler (if you don't want to hand-code).

The boot code may not be simpler, but the switch from assembly to C makes it much easier to write. Cross-compilers that target Windows can also target UEFI with some extra options, and it's easy to get a cross-compiler that targets Windows. And if you really want to write it in assembly, you only need a linker to go along with your assembler - and you might already be using a linker for your BIOS code anyway.

The idea that OS kernels are assumed to use PE format is not much of a help either. It just means I need to build an intermediate stub that must use PE format that jumps to the real OS image. With BIOS boot, I only need a second stage loader, which can be coded in assembly as well, and so I don't see how anything got easier.

UEFI wasn't designed to directly load an OS kernel. UEFI was designed to load a bootloader, and only the bootloader must be a PE binary. In fact, the latest version of the UEFI specification mentions the phrase "OS loader" over 100 times, but "OS kernel" shows up only once.

That UEFI transfers control in protected mode or long mode is not of much help either. This only means that for long mode (a majority of today's UEFIs) I need a long mode stub that can switch to protected mode to be able to load the real OS kernel. Thus, I also need a long mode cross compiler which isn't exactly making things easier.

UEFI was designed around the very reasonable assumption that anyone using a 64-bit CPU would want to boot a 64-bit OS.

Next problem is to create anything that is visible. With BIOS and text-mode this is simply. You just write char + attribute in the video buffer starting at 0xB8000. For UEFI, you need to handle various display resolutions and color organization schemes. Characters must be rendered through a font.

UEFI was designed around the very reasonable assumption that anyone writing an OS will know how to display text with a framebuffer.

That UEFI contains a lot of device drivers is of no use to the OS since these are only available up until control is released to the OS. Actually, BIOS is of more help here since it is possible to use BIOS inside the OS (even if generally not a good idea).

UEFI was designed around the very reasonable assumption that the OS will have its own drivers. The drivers in UEFI are meant to improve the boot process, allowing new features like booting an OS over the internet with HTTPS.

[rdos]

I don't think UEFI solves anything significant, other than introducing the possibility to lock-down PCs.

It definitely solves the problem of digging through dozens of specification documents to find the one that covers what you need. But since you mentioned locking down PCs, I'd like to hear how you would harden firmware and bootloaders against rootkits.

Finding the information about the layout of the boot sector, how to read the disc, get the physical memory map and potentially request keyboard input & write to screen will cover everything that a BIOS loader would need. All of this is in the BIOS manual, and it is far easier than doing the same thing with UEFI.

It's not true that all of the UEFI stuff required is in a single manual. For instance, the PE executable format is not there, and neither how to build a cross compiler that can create the UEFI image. You also need the correct header files, and those are not in the documentation either. It's more or less impossible to write an EFI boot loader in assembler since there are no include files.

The boot code is not simpler, and requires a cross compiler that can emit the special PE format. The traditional PC boot sector can be written in assembly, and so only requires a x86 assembler (if you don't want to hand-code).

The boot code may not be simpler, but the switch from assembly to C makes it much easier to write. Cross-compilers that target Windows can also target UEFI with some extra options, and it's easy to get a cross-compiler that targets Windows. And if you really want to write it in assembly, you only need a linker to go along with your assembler - and you might already be using a linker for your BIOS code anyway.

First, a reasonable OS project will need a fair bit of assembler anyway, so I'd say it is a requirement for an OS developper to know assembly for the target processor(s) of the OS. Thus, the claim that C is easier in the boot loader than assembler is a non-argument.

I use a linker in my BIOS code, but it writes it as a DOS executable, which is transformed to binary by a tool I've written myself. However, the main point that OpenWatcom can build it. OpenWatcom cannot build long mode UEFI executables, which is very inconvient.

Also, I need to build both a 32-bit and a 64-bit UEFI loader, and this doesn't work with typical Windows compilers. They can only build one of them. I use cygwin32 and cygwin64 to build the EFI loaders. Then I check the binaries into my SVN so I don't have to build them when I build my project from scratch.

[Octocontrabass]

Finding the information about the layout of the boot sector, how to read the disc, get the physical memory map and potentially request keyboard input & write to screen will cover everything that a BIOS loader would need. All of this is in the BIOS manual, and it is far easier than doing the same thing with UEFI.

Which BIOS manual?

It's not true that all of the UEFI stuff required is in a single manual. For instance, the PE executable format is not there,

Things that most bootloader developers won't need are located in other specifications and the UEFI specification references them. Most bootloader developers are going to use a linker that already knows how to create PE executables and won't need to read the PE specification.

and neither how to build a cross compiler that can create the UEFI image.

That sort of thing doesn't belong in a specification. (And you don't have to build it yourself. It's easy to find an already-built cross-compiler that targets Windows.)

You also need the correct header files, and those are not in the documentation either.

...Uh, yes they are. I've "written" my own UEFI headers by copy/pasting the documentation and fixing the typos. The compiler-specific bits are missing, but you can easily fill them in from the provided descriptions.

It's more or less impossible to write an EFI boot loader in assembler since there are no include files.

If you want to use a language other than C, you can translate the C definitions given in the UEFI specification. This is how there are Rust crates for UEFI.

First, a reasonable OS project will need a fair bit of assembler anyway, so I'd say it is a requirement for an OS developper to know assembly for the target processor(s) of the OS.

Yes. Where did I say otherwise?

I use a linker in my BIOS code, but it writes it as a DOS executable, which is transformed to binary by a tool I've written myself. However, the main point that OpenWatcom can build it. OpenWatcom cannot build long mode UEFI executables, which is very inconvient.

You shouldn't be surprised that obsolete tools don't work very well for modern systems.

Also, I need to build both a 32-bit and a 64-bit UEFI loader, and this doesn't work with typical Windows compilers. They can only build one of them. I use cygwin32 and cygwin64 to build the EFI loaders. Then I check the binaries into my SVN so I don't have to build them when I build my project from scratch.

Clang is a typical Windows compiler. I use it to build both 32-bit and 64-bit UEFI binaries.

[rdos]

Which BIOS manual?

You mean you don't have the IBM technical reference for personal computers from 1983?

It's a bit like not having the Texas Instruments TTL data book when building hardware.

[Octocontrabass]

You mean you don't have the IBM technical reference for personal computers from 1983?

Of course I have those manuals. According to the manuals from 1983, the only way to detect available memory is INT 0x12, which is limited to 640kB. Which manual do I use if I want to access more than 640kB?

But maybe you meant 1984. According to the manuals from 1984, INT 0x15 AH=0x88 can detect up to 64MB of additional memory. Which manual do I use if I want to access more than 64MB?