Interesting compatibility mode behavior

[doxrobot]

Hello.

My OS runs in long mode, I also make use of 32 and 16 bit compatibility modes for some older programs of mine. There is some interesting behavior that takes place on my test machine (AMD A8 quad core) that i'd like to document and see if this is normal behavior.. based on my assumptions and what I know, it isn't.

Here is a scenario:

from 32 bit compatibility mode (im using a flat memory model with these sub modes) I do a far transfer to a 16 bit code segment. The d/b flag of this executable segment is obviously clear. The far transfer is done at the same privilege level thus no stack switch takes place. The stack my 32 bit code uses prior to the far transfer is 0x14400, I do this purposely because I have my page tables set up in such a way that since 16 bit SP will use 4400, it's valid.

However interestingly enough, I use an operand size prefix on the far transfer to the 16 bit segment (thus converting it to a 16 bit operand size since we are still in a 32 bit segment) the old EIP and selector are pushed and aligned properly however they are pushed onto the 0x14400 stack, and the 16 bit code segment continues to use the full bit width of ESP instead of (0x4400) SP. It does this with no override prefixes. Pushes are normal though at 2 bytes. I don't understand why this happens.

I also got a little bored and tested it a bit more. If from a 32 bit code segment, I do a far transfer to a 16 bit segment, but use a 32 bit operand size in the far transfer, the 16 bit code segment will run using the full EIP instead of IP.

I guess i'm just wondering if this is by design, or if I should change my stack implementation.

Thanks

[sortie]

There is no sub 16-bit compatibility mode in long mode, you have to return to real protected mode before running 16-bit code.

Edit: It would seem I am wrong, carry on.

[Gigasoft]

There is no sub 16-bit compatibility mode in long mode, you have to return to real protected mode before running 16-bit code.

That is untrue and in direct contradiction with the manual.

Whether SP or ESP is used depends on the D/B bit of SS. EIP (when not running in 64 bit mode) is always 32 bits, but is truncated when executing a 16 bit transfer instruction.

[sortie]

Hmm, I must've been been listening too much to hearsay then. My bad! Was it that there was no real mode compatibility mode in long mode, but there was still protected 16-bit mode?

[doxrobot]

There is no sub 16-bit compatibility mode in long mode, you have to return to real protected mode before running 16-bit code.

That is untrue and in direct contradiction with the manual.

Whether SP or ESP is used depends on the D/B bit of SS. EIP (when not running in 64 bit mode) is always 32 bits, but is truncated when executing a 16 bit transfer instruction.

yes exactly. the far transfer should use the lower 16 bits of ESP in this case for the old selector/instruction pointer.

just very confused as to why this occurs. I don't have another machine to try and reproduce this on

[Brendan]

Hi,

However interestingly enough, I use an operand size prefix on the far transfer to the 16 bit segment (thus converting it to a 16 bit operand size since we are still in a 32 bit segment) the old EIP and selector are pushed and aligned properly however they are pushed onto the 0x14400 stack, and the 16 bit code segment continues to use the full bit width of ESP instead of (0x4400) SP. It does this with no override prefixes. Pushes are normal though at 2 bytes. I don't understand why this happens.

The CPU has a default operand size and a default address size. There's also an "operand size override prefix" and an "address size override prefix" to use whatever isn't the current default (note: I'm going to ignore REX prefixes to avoid unnecessary confusion, as they're only valid for 64-bit code).

For 32-bit code (default operand size and default address size is 32-bit):

• With no prefix; operands and addresses are 32-bit
• With operand size override prefix only; operands are 16-bit and addresses are 32-bit
• With address size override prefix only; operands are 32-bit and addresses are 16-bit
• With both operand and address size override prefix; operands are 16-bit and addresses are 16-bit

For a simple example, consider the difference between "mov eax,[esp]", "mov ax,[esp]", "mov eax,[esp]" and "mov ax,[sp]" - these are all the same instruction (just with different prefixes). The operand size prefix changes it from "eax" to "ax"; and the address size prefix changes it from "[esp]" to "[sp]".

For your case; the operand size is 16-bit (e.g. data pushed on the stack is 16-bit), and the address size is 32-bit (e.g. it uses ESP and not SP).


Cheers,

Brendan

[rdos]

Yes, as Brendan notes, which of SP/ESP that is used is determined by stack selector bitness, not by code selector bitness. That's also why 32-bit code could use 16-bit stack selectors and thus SP (which I used at an earlier point).

This behavior is required in order to support bitness switches with call/ret.

[doxrobot]

Brendan, rdos; thank you.

Assumption is indeed the mother of all screw ups. I guess I just assumed that while running a 16 bit code segment that regardless of d/b in SS descriptor cache that it would use a 16 bit stack. Thanks for the clarification.