OSDev.org

Hey friends, I'm working on a challenge in a toy x86 OS that's run with qemu. The OS successfully transitions from real mode to protected mode, initializing the GDT, IDT, and remapping the PIC along the way. I'm trying to use an amalgam of inline assembly and compiled assembly programs to return to real mode and execute a program that relies on BIOS interrupts (0x10, 0x16, etc). I started with the the guide provided on the wiki for the transition in between modes and cross-referenced examples I found elsewhere, but I'm running into issues with the far jump instruction and am stuck in unreal mode as far as I can tell. I've gone through multiple iterations attempting this transition so I won't start by posting code, but this is the general process I've followed:

- copy my target program to 0x7c00 (tested in a boot sector initially to ensure valid code)
- disable interrupts
- reset cr0 without the paging or PE bits
- reload the IVT limits to 0x0 - 0x3ff
- clear the GDT registers
- remap the PIC
- set ESP to 0x7b00
- clear all registers (CS/DS/GS/SS/etc)
- far jump to 0x0000:0x7c00

When I trace this transition with gdb, I can verify that cr0, IVT, and GDT registers look like they do when I examine them prior to the real mode -> protected mode transition. However, the CS/DS/GS... registers still have limits at 0xffffffff instead of their initial 0xffff. When the code at 0x7c00 is executed, even though it was compiled with the bits 16 specifier, I can see that gdb is handling 32-bit addresses instead of the 16-bit addresses I expected. Lastly, whenever a syscall occurs, I can see that the segment isn't being processed correctly (the IVT address for 0x10 is 0xc0005663, but the memory location jumped to is 0x5663 instead of the expected 0xc5663 with proper segment:offset addressing. I also verified that there is valid handler code at 0xc5663). Consequently, I don't think the far jump is properly resetting the CS:IP limitations. Am I missing a critical part of the transition process or is there a standard way apart from a far jump to reset those segment limitations?

To properly return to real mode from 32-bit protected mode, you are going to have to go to 16-bit protected mode in the middle there. You will need a 16-bit code segment and also a 16-bit data segment in your GDT and load those into CS and SS while the CPU is still in protected mode.

Reason is that the CPU does not load the attributes of the segment registers in real mode, so they keep their last value. If you are in 32-bit PM, the code segment keeps the 32-bit attribute even in RM and there is no way to clear it. Same for the address size of the stack segment.

On the whole, doing this dance just to get back to 16-bit RM is so tedious and error prone that you are usually better off doing whatever it is you need done with your own drivers.

Okay, that makes sense (even though the process is super convoluted, haha). So does reloading the GDT with 16-bit code & data segment values and then far jumping reload the CS/SS registers, including their hidden segments?

(The challenge for this OS exists because this process is such a beast, so as much as I wish I could take another route, I'm stuck with this approach.)

chae010 wrote: ↑Wed Apr 16, 2025 12:14 pm Okay, that makes sense (even though the process is super convoluted, haha). So does reloading the GDT with 16-bit code & data segment values and then far jumping reload the CS/SS registers, including their hidden segments?

You can just add the 16-bit segments to the GDT you already have. It is perfectly happy to contain both 32-bit and 16-bit segments. Even 64-bit segments can be in there, if you want to have them. I would suggest using an instruction that allows you to load CS and SS at the same time, else you have a (small) section with mismatched code and stack segments. Luckily, such an instruction exists: iret. If you synthesize an iret frame, you can just jump to 16-bit protected mode very quickly.

chae010 wrote: ↑Wed Apr 16, 2025 8:47 amexecute a program that relies on BIOS interrupts (0x10, 0x16, etc)

Even if you have a BIOS (modern x86 PCs do not), this is a bad idea. Why can't you port some kind of virtual machine to your OS and run the program that way?

chae010 wrote: ↑Wed Apr 16, 2025 8:47 amLastly, whenever a syscall occurs, I can see that the segment isn't being processed correctly (the IVT address for 0x10 is 0xc0005663, but the memory location jumped to is 0x5663 instead of the expected 0xc5663 with proper segment:offset addressing.

Where are you seeing the wrong address? Last I checked, GDB assumes the CS/DS/ES/SS bases are always 0, so it'll only show you the offset portion of most addresses.

nullplan wrote: ↑Wed Apr 16, 2025 1:44 pm

chae010 wrote: ↑Wed Apr 16, 2025 12:14 pm Okay, that makes sense (even though the process is super convoluted, haha). So does reloading the GDT with 16-bit code & data segment values and then far jumping reload the CS/SS registers, including their hidden segments?

YI would suggest using an instruction that allows you to load CS and SS at the same time, else you have a (small) section with mismatched code and stack segments. Luckily, such an instruction exists: iret. If you synthesize an iret frame, you can just jump to 16-bit protected mode very quickly.

Ahh gotcha, thanks for the pointers. I'll give this a shot!

Octocontrabass wrote: ↑Wed Apr 16, 2025 9:09 pm

chae010 wrote: ↑Wed Apr 16, 2025 8:47 amexecute a program that relies on BIOS interrupts (0x10, 0x16, etc)

Even if you have a BIOS (modern x86 PCs do not), this is a bad idea. Why can't you port some kind of virtual machine to your OS and run the program that way?

chae010 wrote: ↑Wed Apr 16, 2025 8:47 amLastly, whenever a syscall occurs, I can see that the segment isn't being processed correctly (the IVT address for 0x10 is 0xc0005663, but the memory location jumped to is 0x5663 instead of the expected 0xc5663 with proper segment:offset addressing.

Where are you seeing the wrong address? Last I checked, GDB assumes the CS/DS/ES/SS bases are always 0, so it'll only show you the offset portion of most addresses.

I wholeheartedly agree that this is a bad idea, but it's not supposed to be easy or practical; it was purely an intellectual challenge. And regarding the GDB bases, I'm not actually sure about that. I thought I recalled seeing more than just address offsets but I'm rarely in asm view, so you are likely correct.