running 16bit real mode code in 32bit protected mode

[h0bby1]

the rest i don't care if it's made in 32 or 16 bit, any of them should work the same

The point is, that they don't work the same. The override prefix makes a 32 bit instruction out of a 16 bit opcode in RM16 and
a 16 bit instruction out of the same opcode (which is 32 bit) in PM32.
There is also the issue of the near jmp, which has a 16 bit offset in RM16 and a 32 bit offset in PM32. In this case, when run
in PM32 the binary code will eat the next opcode as well.

normally opcode should be almost identical from real mode to 16 bit pm

PM16 is a totally different story.

the code will never be run in pm32, but in pm16, due to 16 bit code segment selector used in the far call/software interupt

normally you can write some code in a 16 bit section with an assembler, and it shouldn't care at all if the cpu is in real mode or protected mode to assemble the file, the opcode are identical and interpreted identically in pm16 and real mode, even with prefixes, only the way memory seg/ofset address are interpreted

it's more tricky for all the call mecanism that push return address in the stack, because the format of the address pushed depend on many things, i watched the documentation of iret instruction in intel manual, it's behavior depend on many thing, cpu flags, type of code segment and override, it's behavior is rather complex

like say you would make a routine to switch to real mode, and call it from 32 bit code, the 'ret' instruction of the function would most likely fail after the switch to real mode, because address passed on the stack are expected to be 2 16 bits values, whereas in 32 bit mode, the value pushed on the stack are 32 bit value for seg/ofset of the return address, even if the segment part is always 16 bit, it's still pushed as a 32 bit value as well as the ofset, so the ret function would return to wherever and mostly likely create a page fault, unless it's a near call to an address that fit in 16 bits, in which case it should still work in both case, but it would leave the two upper bytes of the return address on the stack

interupts are easier because the idt already contain a selector, so the cpu is smart enougth to format the stack depending on the segment selector of the idt, so you don't have to care about the parameters of software interupt or exception , they are always formated according to the bit value of the destination selector, but far call and far jumps are more tricky


but there is no 'true' pm16 mode, except on 286, but on all 32 bit cpu, pm16 is just protected mode executed in a 16 bit code segment, and it can use the same kind of trick with prefix to use 32 bit version of opcode, it works the same than real mode, pm16 is not a special cpu mode or anything like this, it's just a 32 bit flag is present on code segment selector that don't need the prefix overide and exepect default size of 32 bit and probably various other thing, but pm code executed in 16 bit segment behave the same than real mode in respect to operand implicit size etc


before what i was doing is switching to real mode from a protected mode 16 bit code segment, and doing all the real mode code in the 16 bit segment, the 16 bit code can be either targeted at real mode or protected mode, it doesn't change anything, the cpu can be switched from protected mode to real mode back and forth within a 16 bit code segment, just need to be carefull that segment register contain valid values for the mode selected, and segment registers are always 16 bits anyway, so they can be passed between pm32 and pm16 without problem, only ofset has to be 16 bit 'normally' unless you use operand size overide prefix, which is done automatically in any case by most compilers and assemblers

the semantic of 'far call' can be confusing between real mode and protected mode, because in real mode 'far call' really mean often that the address is so far away that it need to change the segment register value to reach it, and so it mean jump to a '32 bit address', but in 32 bit mode, near jump are already 32 bit, and the far jump is more about 'make a segment selector switch' , but it can do a selector switch just to change the mode of execution or privilege value to point at the same address , so it's not even really the same behavior than in real mode, you could do a near jump in 32 bit protected mode to any address of the system, but the far jump allow to change the segment selector , and can switch the execution between 32 and 16 mode

[h0bby1]

the attached image show how i think it's supposed to work between all the different modes

there are differences as well in the format of the idt

the size of immediate values should be the same as the operand size

intel.png (5.88 KiB) Viewed 4105 times

according to this

the only thing a pm32 code need to care about to switch to 16 bit pm is the format of the far call return address, for it to return properly to the good 32 bit code segment when it's done, the return address must have a 16 bit ofset value and the code segment selector pushed as a 16 bit value

making a 16 bit far call from pm32 seem to be tricky , so it's why i push the value on the stack manually to have the return address set in a way that fit a '16 bit far ret', even if in the case it's an interupt handler, the format is the same, there is only the cpu flag register added to the stack as a 16 bit value for a 16 int/iret, and 32 bit cpu flag value for a 32 bit int/iret

the only thing a pm16 code should care about when switching to real mode are the value of the segment registers, which is normally taken care of by having segment selector of which value match the one a segment register would have in real mode to represent the address the segment selector point to, or any instruction of which behavior would depend on the value of cr0 register, and i don't think it change many thing except the way seg/ofset address pair are interpreted, and of course the expand down bit of the segment used for the stack must be clear for it to work in real mode, not sure if 32 bit flag change anything for data or stack segment

[b.zaar]

Hi, I had a similar idea to h0bby1 a long time ago but never got around to making anything. I finally had a chance to do some things over christmas/new years and this is the result. It's an 8086 16 bit emulator that "should" work in both 32 bit pmode and 64 bit long mode. It uses 16 bit protected mode segments and the debug exception to trap every instruction. It emulates segment changes on far calls, far jumps, interrupts, lds/les etc and lets the CPU run any other instructions.

I've only been able to test it on bochs and qemu at the moment but I'd like to know if it works for any one on real hardware they have and how fast it works.

It's still rough and it is part of a bigger project but should compile to a 10MB image for testing.

boot32.0.0.1.zip

I've attached the compiled 10mb image too.

[Owen]

Hi, I had a similar idea to h0bby1 a long time ago but never got around to making anything. I finally had a chance to do some things over christmas/new years and this is the result. It's an 8086 16 bit emulator that "should" work in both 32 bit pmode and 64 bit long mode. It uses 16 bit protected mode segments and the debug exception to trap every instruction. It emulates segment changes on far calls, far jumps, interrupts, lds/les etc and lets the CPU run any other instructions.

I've only been able to test it on bochs and qemu at the moment but I'd like to know if it works for any one on real hardware they have and how fast it works.

It's still rough and it is part of a bigger project but should compile to a 10MB image for testing.

boot32.0.0.1.zip

I've attached the compiled 10mb image too.

Thats an ingenious method. I'd long thought of how to use 16-bit PM to implement a real mode emulator, but never thought of using the debug trap exception.

If (for whatever reason) you needed more performance, I can see an easy way to attain it now too, via dynamic recompilation: On the first run, use the debug exception to trap every instruction execution. If its' a non-segment instruction, copy it to a buffer. If it is a segment instruction, copy a sequence which emulates it (perhaps a far call to a handler function or similar) to the buffer. You would then use page protection to detect any writes to the underlying code (indicating you need to discard the dynamically compiled buffer)

You might want to translate branches during the copy also (or at least reserve extra space for them - in case the branch target goes out of range, or the emulation sequences cause the code which fitted in one real mode segment to not fit one 16-bit pm segment)

[b.zaar]

If (for whatever reason) you needed more performance, I can see an easy way to attain it now too, via dynamic recompilation: On the first run, use the debug exception to trap every instruction execution. If its' a non-segment instruction, copy it to a buffer. If it is a segment instruction, copy a sequence which emulates it (perhaps a far call to a handler function or similar) to the buffer. You would then use page protection to detect any writes to the underlying code (indicating you need to discard the dynamically compiled buffer)

You might want to translate branches during the copy also (or at least reserve extra space for them - in case the branch target goes out of range, or the emulation sequences cause the code which fitted in one real mode segment to not fit one 16-bit pm segment)

This can be tricky to optimise. If the code runs through the first time without taking any branches and you have replaced that code with the far call then the next time through the same code section it branches you will not have the trap flag set to work on the new code.

If you really wanted to optimise it like this then it's probably better to use the source program as byte code running on a JIT compiler and spit out the code as a proper 32 bit compiled app.

Hmm... maybe that's my next challenge... BIOS byte code to 32 bit ;)

[Jezze]

I assume this is for executing BIOS calls? How would you go about integrating this into another operating system? Also it looked like it was all assembly, is it possible to add a C header file and use this as a library?

[b.zaar]

At the moment it's kind of proof of concept. I plan to use it in a boot loader. It would need to be modified to use as a C library but it's released under the bsd license so feel free to do anything with it. You would normally need to write you own vm86 monitor for an OS, think of this along similar lines.

[bubach]

http://forum.osdev.org/viewtopic.php?f=2&t=17573

[h0bby1]

i try not to use this trick too much, but it's mostly for executing bios routine so far, both for vesa and disk i/o, for now it seem to works well, and can executing bios routine from a multi tasking os without a switch to real mode, or change anything into the other interupt handler code, but i don't use them much, i just wanted to do this because bios can be a nice fallback for some things, and it's still some code made by manufacturer who can handle basics things on the hardware, and can be used as some kind of fallback drivers for some things maybe, i mostly plan to use it like that, or some bios routine can be usefull also to get some information about the hardware

i basically wrote an interupt handler for the pmode in the idt, it does a routing to the vector of the IVT, so that a call to int 10 or int 13 from pm mode will call this handler, and this handler will then fetch the address of the bios routine from the IVT, and do a far call to it using a 16 bit code segment selector, and trapping exception.

For now i just handle the mov to seg register and pop segment register function, and i assume the bios doesn't use pmode instruction like ldt/lgdt/lidt , and it works pretty well.

Just need to do some tweaking to have the value set to segment register in the rm16 code to match with an entry in the gdt that point at the same physical address (seg register value * 16), and struggled a bit like when the code can use pointer to a stack variable or something, the address set in the gdt entry need to match the value of segment registers for all to works well, in case the code has to recompute address from the seg:ofset register, as several combination of real mode seg:ofset address can point to the same physical address, but it can be safe for most cases.

i wondered a little if it could be possible to basically disassemble the whole bios area, and reassemble it in 32 bit mode, as once disassembled, the instruction set between the 16 bit mode and 32 bit are mostly similar, or to have something to convert 16 bit version of the opcode to 32 bit version, as i already have something a little bit like a debugger/disassembler in the exception handler code, but i'm not sure it would be very workable, or if it's better to just catch the problems on the fly with the exception

i just made some routine in assembler that are called from a C function, and it prepare the registers with value passed on the stack from the program and do the interupt call, need to do also a little trick to match memory pointer passed to the bios routine with the memory space of the program in protected mode, because the real mode code is only supposed to access memory in the first 1Mb.Need to pass bios code pointers to that area in 'real mode addressing', and then either translate the address for it to match with the regular pmode memory space, or copy the result toward another area.

I use a static area of some kilobyte in the first 1Mb of memory , and all the bios call will use pointer to that area if it has to send or recieve data, and i just have to translate the address to that area in 'real mode address space', and there is an assembler routine that return a pointer to that address, or it could copy it to an user supplied buffer, for now the address where data will be stored by those routine is implied and always at the same place, if you need to send data from a pointer to it, you have to copy it in that area before to do the bios call.

[freecrac]

Sorry, I didn't read your post to the very end but I just want to say that BIOSes often use temporary switch to protected mode, so trying to run BIOS code from 32-bit PM is likely to either fail (not ring0) or crash (ring0) your PM setup.

But when the BIOSes often switch to the protected mode, how the DOS based games of the 90th use the big-realmode?

Is there a list of bios routines with switching temporary to the protected mode?

Size prefix ignored (??)

Intel manual:

Instruction prefixes can be used to override the default operand size and address size of a code segment. These prefixes can be used in real-address mode as well as in protected mode and virtual-8086 mode. An operand-size or address-size prefix only changes the size for the duration of the instruction.

The following two instruction prefixes allow mixing of 32-bit and 16-bit operations within one segment:
•The operand-size prefix (66H)
•The address-size prefix (67H)

These prefixes reverse the default size selected by the D flag in the code-segment descriptor. For example, the processor can interpret the (MOV mem, reg) instruction in any of four ways:
•In a 32-bit code segment:
—Moves 32 bits from a 32-bit register to memory using a 32-bit effective address.
—If preceded by an operand-size prefix, moves 16 bits from a 16-bit register to memory using a 32-bit effective address.
—If preceded by an address-size prefix, moves 32 bits from a 32-bit register to memory using a 16-bit effective address.
—If preceded by both an address-size prefix and an operand-size prefix, moves 16 bits from a 16-bit register to memory using a 16-bit effective address.

•In a 16-bit code segment:
—Moves 16 bits from a 16-bit register to memory using a 16-bit effective address.
—If preceded by an operand-size prefix, moves 32 bits from a 32-bit register to memory using a 16-bit effective address.
—If preceded by an address-size prefix, moves 16 bits from a 16-bit register to memory using a 32-bit effective address.
—If preceded by both an address-size prefix and an operand-size prefix, moves 32 bits from a 32-bit register to memory using a 32-bit effective address.

Dirk

[h0bby1]

the code of the bios need to be executed on a 16 bit code segment in protected mode to have the correct opcode encoding