I stopped worked on my little OS years ago for some personal reason but finally got better enough to want to return into that hobby.
But, as it was a long time ago, I needed to take that slowly to find my confidence again.
With my OS still in the freezer where I left it when I was falling deep in some bad place, I didn't have any project that where in that area of expertise to do some fun stuff.
So I decided to find a way to answer a question that I had for many years but never did a proper research on it:
How a kernel can recover from an unsupported boot process?
(if you don't want to read all the post, just go at the end of the URLs)
Most precisely, I wanted to see how a software would be able to keep running long enough to inform the user without causing any damage.
By unsupported boot process, I mean any boot process that wasn't expected and that put us in an unknown and isolated situation.
Unknown because we are outside our designed area of work and isolated as we have no clue on how and where to gain any help or informations.
Some of the scenario I wanted to answer as universally as possible:
- booted by a legacy bootloader in 16-bit real mode
- booted by a bootloader than can't parse our header or support our kernel but try to load it and execute it anyway
- booted by one of those long mode bootloader without any request from our part
- placed in any part of the RAM except the one we have been designed to be in
My goal was to find a way that work on x86/x86_64 and that is completely agnostic of it's environment except for some guaranteed stuff (like VGA video memory in the low memory area).
And it needed to be as universal as possible, capable of managing any situation it may encounter.
To be a little bit more creative, I fixed to end point at : displaying an error message to the user (bonus if it specify why we get there).
I also use one big assumption to make that challenge a bit simpler: if paging is enable, it's an identify mapping (most common without specific work between the OS image and the bootloader).
The solution I found use some binary tricks that are read differently by the same CPU depending of it's running mode. I put that trap code at 0x0, in a flat binary file.
giving ourself a virtual address was meaningless with the scenario it was designed to handle, and a flat binary file to have little to none element before the code itself.
It is capable to quickly (with only ~one instruction in 40 bits) separate the 16-bit execution flow from the other two. Letting us isolate that flow in specific code that will build on top of that to achieve a stable protected mode with known address environment. That was the easiest one to find as the two other were really tricky to force executing that kind of trick but still using only relative addresses and not trying to go anywhere outside our only somewhat known memory area. I finally find a solution that only require a simple JE/JNE instruction just after the trap code. And it setup the necessary flags for that jump with only 56 bits that the CPU will interpret as two or four instructions depending of it's mode. With that setup, both the 32-bit and 64-bit execution flows where isolated in specific part of the program.
If you only want the trap code, it work with only 96 bits in total (and three jumps). But I went farther by resetting the protected mode to a known state and dropped the long mode into a similar protected mode setup.
With those three executions flows isolated and working to build the same patched protected mode setup, I was able to use the same function for all of them to print the desired message to the user and halting the system
If you want to read more about the solution, I did a bigger and better explanation on the README.md file of the repository (with a TL;DR section that show directly the important assembly code).
If you want to see the code I wrote for that project you can find it in this repository.
The code is using the GNU Assembler (AT&T syntax) but the README is in NASM (intel syntax) because it seem more familiar for a lot of people and gitlab had syntax highlighting for it.
I was really happy with the final result and I think that it was both a really nice brain warmer and interesting exploration
Hope you find that as interesting as me or funny to explore. And yea, I may have done an overdose of assembly xD
