EFI and the future of BIOS

[Owen]

Though, unless I am mistaken, SMM occurs outside the normal execution environment, with its own memory. Or am I thinking of something else?

SMM is basically an odd version of Unreal Mode. There will often be some special "SM RAM", but it is mapped somewhere in the system address space. This is one of the reasons you should not modify any MTRRs set up by the BIOS; it is possible that the BIOS is doing something special with some regions of the address space.

[DavidCooper]

Go *back* to real-address mode to speak to BIOS? Why would anyone want to do that? Surely, by the time they are ready to switch to protected mode, one is ready to be autonomous, no? I cannot imagine going back. I mean, I would have to restore the IVT and the memory directly above it that BIOS uses.

Is it really worth spending years working out how to change screen mode without using the BIOS in order to avoid having to switch back to real mode to let the BIOS do it for you? No doubt it is worth it if you're trying to build a serious OS to rival Windows, and maybe it's worth doing it in a hobby OS if your main interest is in writing screen mode changing code, but the ususal advice is to set up a VESA mode and just stay in it. That doesn't seem like the best option to me - it only takes a fraction of a second on a simple OS to switch everything back to original settings, use the BIOS to change mode and then redo the setup the way you need it, and this allows you to choose a display that suits a specific application (some apps benefiting from a greater range of colours and higher resolution at the cost of speed while for others speed is much more important). Even if you have to undo masses of changes to use the BIOS and then make masses of changes back again it's still going to be a tiny cost - the clock may miss an update and be wrong for a second or two, but how many users are going to get upset about that?

Is it really such a problem to copy the IVT and BDA somewhere and put them back if you need to use the BIOS? I just leave them in place all the time, but even if I had to move them somewhere else and copy them back to use the BIOS there would be no visible delay. The other good reason for using the BIOS in a hobby OS is to load and store data via USB, and that again lets you concentrate on developing more interesting parts of your OS instead of getting bogged down in USB stuff from the outset. If you go down the alternative route of booting your OS from the internal hard drive, hardly anyone else will ever try out your OS because it will mess up their machine.

Being able to undo and redo everything is also a good exercise in itself, and it may aid you in tracking down some bugs if you're doing something incorrectly.

[Brendan]

Hi,

Go *back* to real-address mode to speak to BIOS? Why would anyone want to do that? Surely, by the time they are ready to switch to protected mode, one is ready to be autonomous, no? I cannot imagine going back. I mean, I would have to restore the IVT and the memory directly above it that BIOS uses.

Is it really worth spending years working out how to change screen mode without using the BIOS in order to avoid having to switch back to real mode to let the BIOS do it for you? No doubt it is worth it if you're trying to build a serious OS to rival Windows, and maybe it's worth doing it in a hobby OS if your main interest is in writing screen mode changing code, but the ususal advice is to set up a VESA mode and just stay in it. That doesn't seem like the best option to me - it only takes a fraction of a second on a simple OS to switch everything back to original settings, use the BIOS to change mode and then redo the setup the way you need it, and this allows you to choose a display that suits a specific application (some apps benefiting from a greater range of colours and higher resolution at the cost of speed while for others speed is much more important).

For a simple OS, I agree.

The problem is that as an OS developer tries to go beyond the "simple OS" stage it becomes increasingly more difficult to justify. Everything you touch (MSRs, local APIC, MTRRs, PIT, CMOS/RTC, HPET, USB, PCI device configuration for any/all devices, etc.) ends up being about 4 times as much work, because (for each piece of hardware) instead of just doing the initialisation once you have to be able to:

1. save the original device's state somehow (impossible for a range of different pieces of hardware)
2. do the initialisation (as you normally would)
3. save the OSs current state (a complete nightmare in a lot of cases, where devices may be in the middle of DMA transfers, or have queues of pending commands, etc)
4. restore the device's original state (that the BIOS expects)
5. restore the device's "pre-empted" state (that the OS expects)

To make it worse, for some things you have to restore the state/s in the correct order (and race conditions must be avoided). For example, you couldn't configure the PICs and disable IO APICs until all devices that were using MSI are reconfigured; and you couldn't restore the state of devices until the devices are taken out of any sleep states; and you couldn't restore a USB controller's state and then expect to be able to restore the state of USB devices attached to that controller. On top of that there's other CPUs to worry about - the CPU that got switched to real mode won't be able to respond to IPIs or anything, so all the other CPUs will probably grind to a halt trying (and failing) to do TLB shootdown or something. Basically you end up with a "stop the world" scenario where the entire OS and all of the hardware has to be stopped.

Then there's timing. It might take several seconds before "in progress" DMA transfers and bus mastering stuff completes, then you can start saving the state of everything; then start restoring the state of everything to BIOS defaults. Then there's the video mode switch itself (which is probably the single most expensive BIOS function), followed by restoring the state of everything again. It adds up to a massive problem for any hardware that is sensitive to latency - dropped packets for networking, lost keyboard and mouse data (and potentially screwed up keyboard/mouse state), annoying glitches in audio output (and worse if you're recording any sound), trashed writable CDs caused by buffer under-runs, dropped hot-plug or thermal sensor events, clock skew and/or missed timer IRQs, etc.

Then there's bugs. How are you going to test that all of the above works reliably on all different combinations of hardware under a range of different conditions? Quite simply you're not - it's going to be an untestable and unstable/glitchy mess, and the first time it fails is going to be the last time the unfortunate end-user will dare to use it.

Basically, it's one of those things that seems easy in the beginning, but becomes incredibly hard in the long run. I'd be tempted to suggest that people who suggest this method will never get past the "simple OS" stage, regardless of how hard they try.

Of course some people don't intend to go past the "simple OS" stage. For example, they might be writing a simple OS for educational purposes - e.g. because they want to learn how to design software with fatal flaws.


Cheers,

Brendan

[DavidCooper]

Hi,

The problem is that as an OS developer tries to go beyond the "simple OS" stage it becomes increasingly more difficult to justify.

Well, when you go beyond a simple OS you're soon going to want to do anything that involves USB directly instead of using the BIOS, so at that point you're ready to abandon the BIOS for anything other than changing screen mode, and if sufficient documentation is available to enable you to do that directly you might eventually abandon the BIOS for that too (assuming you consider it necessary to change it at all). There may be a stage in between, however, when it will still make more sense to "stop the world" for a little while to change screen mode rather than having to reboot the machine to do it in a much more disruptive way (or just put up with not being in the best screen mode for a particular task and enjoy watching the processor fail to update the screen at an adequate rate).

I'm also not sure just how much you would actually have to undo in order to use the BIOS to change screen mode, and it may vary on different machines. Is the BIOS really likely to interfere with other processors or even notice whether they're still actively processing data in the background? So long as they don't generate an interrupt that affects the processor being controlled by the BIOS, they may be completely invisible to it. I very much doubt that many BIOSes bother to check the state of every last piece of hardware on the machine every time they are asked to change screen mode just to see if they're still set up as it left them earlier.

... Basically you end up with a "stop the world" scenario where the entire OS and all of the hardware has to be stopped.

Stopping the world isn't necessarily going to be a problem, provided that you aren't doing something like writing a CD in the background, but if you want to run a specific program in a specific screen mode without anything else running in the background in order to get the best performance out of it, it may be possible to do that without having to reboot, and after you've finished with it you will then be able to go back into all the other programs you left open (but inactive) in the background. That functionality would be a real gain.

Then there's the video mode switch itself (which is probably the single most expensive BIOS function),

Surely it's pretty minor compared with loading or saving - just a fraction of a second?

followed by restoring the state of everything again. It adds up to a massive problem for any hardware that is sensitive to latency - dropped packets for networking, lost keyboard and mouse data (and potentially screwed up keyboard/mouse state), annoying glitches in audio output (and worse if you're recording any sound), trashed writable CDs caused by buffer under-runs,

You wouldn't do it at a time when anything like that was going to be disrupted - the OS would warn you clearly about anything that will be affected and suggest you wait until a point where no harm can be done to anything.

dropped hot-plug or thermal sensor events,

You'd be warned not to plug anything in or out. I don't know how you handle thermal sensor events, but if there's a risk of processors overheating while you're using the BIOS you can presumably stop them or slow them down as a precaution.

Then there's bugs. How are you going to test that all of the above works reliably on all different combinations of hardware under a range of different conditions? Quite simply you're not - it's going to be an untestable and unstable/glitchy mess, and the first time it fails is going to be the last time the unfortunate end-user will dare to use it.

If it causes a crash, the user may well never repeat the experiment, but just suppose it works absolutely fine every time on most machines - is it really so daft to provide the option? That option could then be disabled by the user for the future on that machine if there is a problem with it. I can see why you aren't keen to put in the work that would be needed for your OS to be able to do this kind of thing because "stopping the world" isn't in keeping with the kind of work a server needs to do, but I can't see anything that would necessarily result in bugs from programming your OS to be able to shut everything down and then start it all up again. If it's not possible for you to do that, then maybe you've built a bit of a mess which only works as well as it does by luck - even if it takes a while, it ought to be able to bring everything to a halt bit by bit without tripping over itself.

Basically, it's one of those things that seems easy in the beginning, but becomes incredibly hard in the long run. I'd be tempted to suggest that people who suggest this method will never get past the "simple OS" stage, regardless of how hard they try.

I suspect that they're more likely to succeed than if they shun the BIOS from the outset and set themselves a massive great boring slog of a task just to get their foot in the door. I initially tried to do everything without using the BIOS for anything beyond loading my boot sector, but it was a mistake - the BIOS is useful and will continue to be useful to me for a while yet. If you've got on top of USB and don't need to change screen mode (e.g. because your OS is designed for server use and the display is unimportant), then fine - you can stop using the BIOS moments after booting. If you manage to find sufficient documentation on changing the screen mode without using the BIOS and have the time to write all the code to do that, then again you may be free of the BIOS. However, I personally do not intend to spend years working on USB and screen-mode stuff at the moment - there are plenty of other areas of OS design which are higher priority, such as trying to build human-level A.I. into the heart of the system.

Of course some people don't intend to go past the "simple OS" stage. For example, they might be writing a simple OS for educational purposes - e.g. because they want to learn how to design software with fatal flaws.

Maybe so if their intention is to illustrate a fatal flaw. Mine is not. I need to be able to load and save data from time to time and there is very little harm at the moment in "stopping the world" for that, though it will clearly need to find a better way to do it later on. I also want users of my OS to be able to explore different screen modes, so I will continue to provide a route to doing that as my OS becomes more complex, and since changing screen mode isn't something you need to do every few seconds, there may never be any reason to bother doing it in any way other than via the BIOS. The reality is that there are perfectly good reasons for wanting to use the BIOS and it's up to each of us to make our own decisions about when to use or not use it. The ability of an OS to continue to use the BIOS does not need to restrict its complexity or capability. Even once my OS can save to USB devices directly rather than through the BIOS, it will still retain the BIOS method just in case the code for direct saving becomes corrupted (which might happen at some point while it's being modified) - it may on occasions enable important data to be saved that would otherwise be lost.

[gerryg400]

Stopping the world isn't necessarily going to be a problem, provided that you aren't doing something like writing a CD in the background,...

Even moderately mature OS's are always doing something. Setting aside interactions with the user there's cleaning caches, monitoring memory use, firing off timers that were started weeks ago to do a cron job, zeroing unused pages so that they are ready for their next allocation, shuffling things around in the scheduler to improve fairness, replying to ping messages from other machines, monitoring hardware to make sure it's connected and behaving correctly, etc. It never really ends. It can't end. These things aren't normally predicted, though I guess that would be possible.

It's not just what the core is currently doing that's important. There's the immediate future. Typically there will be a per-core scheduler full of threads to run, interrupts queued in local APICS, power saving considerations and so on. All this stuff needs to be taken care of.

The point is that to return to real mode reliably you would have to stop/handle all these things, and as Brendan pointed out that means stopping/handling them on all cores. Returning to real mode is not really the simple option. It just looks simple on the surface.

[guyfawkes]

[Off topic], I must add to this debate, I have not been coding my OS for very long, but as part of my learning stage, i used a 32bit OS that used the going back to realmode method, for testing bits of my OS.
It worked great, i did not get one crash or freeze, from the OS.
I could change vesa modes and vga modes, read/write to USB fobs.

I will be using it in my OS design.

[DavidCooper]

Even moderately mature OS's are always doing something. Setting aside interactions with the user there's cleaning caches, monitoring memory use, firing off timers that were started weeks ago to do a cron job, zeroing unused pages so that they are ready for their next allocation, shuffling things around in the scheduler to improve fairness, replying to ping messages from other machines, monitoring hardware to make sure it's connected and behaving correctly, etc. It never really ends. It can't end. These things aren't normally predicted, though I guess that would be possible.

Again that might be so for a server, but think about what Windows does when it hibernates: the state of the machine is saved in its whole multiplicity of aspects, and then it shuts down. I don't know if the BIOS is able to unhibernate in some manner that bypasses real mode, but I would have thought that everything will need to be set up from scratch regardless. So, we have a complete halt of everything that's going on in the machine (except the RTC), and then it's able to return to the same state and continue from where it left off to do everything it was doing before. Quite clearly it can be done.

It's not just what the core is currently doing that's important. There's the immediate future. Typically there will be a per-core scheduler full of threads to run, interrupts queued in local APICS, power saving considerations and so on. All this stuff needs to be taken care of.

I would assume that Windows puts everything like that on hold if it can. It certainly manages to do the whole job very quickly, so if you were doing the same kind of thing in order to change screen mode on an equally complex OS, you shouldn't have to wait more than half a minute to get it done, plus the same again to get it all back running again. That's a worst-case scenario - it should be a hell of a lot faster as you can leave the contents of memory unchanged if you aren't actually switching the machine off in the middle of the process and rebooting, but even if you did have to wait a whole minute just to change screen mode it could be well worth the wait on occasions given the improvement in the visual performance of the app you want to use which would gain from this switch.

The point is that to return to real mode reliably you would have to stop/handle all these things, and as Brendan pointed out that means stopping/handling them on all cores. Returning to real mode is not really the simple option. It just looks simple on the surface.

I don't doubt for a moment that it's hard to do, but if Windows can hack it, I think it's perfectly reasonable to think about aiming to do likewise.

[gerryg400]

Again that might be so for a server, but think about what Windows does when it hibernates: the state of the machine is saved in its whole multiplicity of aspects, and then it shuts down.

Quite true. In fact, reliably switching in and out of real mode is a task of possibly the same order of complexity as implementing hibenation. Windows is probably the only hobby OS that does it reliably.

In many cases drivers need to be hibernation- or sleep-capable. This would possible need to be extended so all your drivers are returning-to-real-mode-tempararily-capable.

[Owen]

...If you don't have graphics drivers for the machine you're running on, or are running a constrained set (e.g. Windows Safe Mode) which only includes a VESA driver... just resort to doing what everybody else does.

That is, invoke the VGA BIOS in a real mode emulator (be it hardware like v8086 or software like x8emu). Sure, the code can theoretically be batshit insane; in practice, since everyone else is running the VGA BIOS in an emulator, its perfectly safe to run the VGA BIOS in an emulator.

You don't need to worry about dropping interrupts on the floor either.

[SmD]

All of this mentioning of restoring the state to that which BIOS last knew brings me to another question: What about the stack? If modified, should it be restored to how BIOS left it? What about the other segments?

Earlier, when I spoke of BIOS trashing registers, I had the segment registers in mind, in addition to the other, normal registers which do not contain any sort of return values (all but AX for INT 12, for example).

[DavidCooper]

All of this mentioning of restoring the state to that which BIOS last knew brings me to another question: What about the stack? If modified, should it be restored to how BIOS left it? What about the other segments?

If an OS runs entirely in real mode, it is expected to be able to call on BIOS functions at any time no matter what state its stack is in, so all you have to do is make sure the stack is put back somewhere in low memory (the first 640KB).

Earlier, when I spoke of BIOS trashing registers, I had the segment registers in mind, in addition to the other, normal registers which do not contain any sort of return values (all but AX for INT 12, for example).

The BIOS may trash the contents of any of the normal registers, including segment registers. It shouldn't care what values are in those registers when you call a BIOS function, apart from those which are required to hold values relevant to that function, although one of my machines has a BIOS that crashes if BP and SP hold values that are miles apart. The trick is to assume that it will trash the lot and make sure you save and restore them all - it only takes a couple of dozen instructions to do the job, and it can be done as part of a standard routine for switching to real mode, calling the BIOS and switching back to protected mode - an address can be changed within that piece of code to make it call different real mode routines which request different BIOS services, so there's no need for duplication and it can all be extremely compact.

[Neolander]

The biggest problem which I have with UEFI is the whole notion of "You may read our specs, but to do anything with them you must join us". I don't agree with the notion that every single OS developer should be part of the secret UEFI society in order to survive

[Combuster]

We have reverse engineering for that

[Ready4Dis]

Sorry to bring up a month old topic, but I had to comment. First, owen, it is not the BIOS that creates the 2TB limit, it is the format of the MBR. It is completely posssible to switch over to GPT without using UEFI.

Secondly, you guys debating going back and forth to the bios for stuff... you forgot the third option, emulation. Use an x86 emulator to run the bios code from protected/long mode. It means you don't miss any interrupts, no other processes get halted, etc. Yes, the emulator takes a bit of space, but so does writing a proper video driver. I wouldn't use an emulator for things like USB disk access (unless you really have your emulator clock sources accurate), but for changing video modes, it's a simple method.

Dropping back to real mode from pmode is very easy, only a few lines of code, and it's the current method my OS uses until my emulator is working 100% (alhtough, there are a few open source ones that others use, I am writing my own). From long mode, it takes a few more hops (hence me changing over to using an emulator), but still isn't that bad.

UEFI is a very complex thing, it has a ton of functionality, which is also it's down side. If you intend to just use it for basic things, like reading a few sectors from a hard drive and jumping to your OS, then it's not much more complex than the bios (from your perspective). Benefit is, if you want to use it for loading data from a usb drive, or over a network, you don't have to write each and every driver, the UEFI driver should handle that for you to allow you to proceed with the important parts of your OS, and replace said sections when you want to start writing specific drivers.

[DavidCooper]

Secondly, you guys debating going back and forth to the bios for stuff... you forgot the third option, emulation. Use an x86 emulator to run the bios code from protected/long mode. It means you don't miss any interrupts, no other processes get halted, etc. Yes, the emulator takes a bit of space, but so does writing a proper video driver. I wouldn't use an emulator for things like USB disk access (unless you really have your emulator clock sources accurate), but for changing video modes, it's a simple method.

Dropping back to real mode from pmode is very easy, only a few lines of code, and it's the current method my OS uses until my emulator is working 100% (alhtough, there are a few open source ones that others use, I am writing my own). From long mode, it takes a few more hops (hence me changing over to using an emulator), but still isn't that bad.

At the time of this thread, I didn't know that it was safe to emulate the BIOS for screen-related stuff, but it came up soon after in another thread, so I am now planning to write an emulator in the near future for exactly that purpose. The speed of emulating code to change screen mode isn't at all important, but when changing the window into the screen memory or changing the first byte to display (for smooth scrolling in any direction) it wants to be reasonably fast, so I'd be interested to know if you've discovered how many instructions typically need to be emulated on each call for these functions to get a better idea of how important the speed issue actually might be - at the moment I don't know if it's worth putting in the effort to make a complex, fast emulator or if a simple, slow one would do the job without a noticeable delay. Have you tested your emulator to see how quickly it emulates code compared with running the same code directly, and how well do you think it'll compare with other emulators?

On the USB loading and saving point, my next target after getting the emulator written is to try to communicate with USB flash drives without using the BIOS, but it will doubtless be a long slog getting on top of the USB stuff, and I won't have a lot of time free to put into it either - it probably means my OS will effectively stop evolving for a year or more until I've got that one thing done.