Hi,
m wrote:So far I haven't come across the problem in walling's case.
Wallings case is 32-bit protected mode code being interrupted by a 16-bit protected mode IRQ handler, where the CPU switches to 16-bit and then saves a 16-bit return EIP on the stack when the IRQ occurs. In this case the highest 16-bits of EIP are lost, making return impossible (unless the highest 16-bits of the return address are zero anyway).
Also, as Walling is trying to run real mode BIOS IRQ handlers using 16-bit protected mode, the size of the IRET (and the stack mangling needed to make it work with a non-default size) aren't possible as the code is in ROM. This means he can't design the 32-bit code to keep a copy of the highest 16-bits of EIP in a temporary locatoin and restore those bits before the IRET.
Of course the CPU and BIOS designers never intended to allow running real mode code as 16-bit protected mode, so I'm guessing Walling will keep running into problems (e.g. with BIOS code and segment registers), and sooner or later realise that implementing work-arounds for these problems is harder than setting up virtual 8086 mode (which is designed specifically for this).
I also think that getting it to work reliably isn't Walling's reason for attempting it, and that it'd be both fun and educational to try to get it to work (regardless of whether it does or not).
BTW I'd be tempted to pre-fill the GDT with 16-bit read/write data segment descriptors, such that GDT entry 0x0008 has a base address of 0x00000080, GDT entry 0x0010 has a base address of 0x00000100, etc, all the way up to GDT entry 0xFFF8 with a base address of 0x000FFF80. Then I'd use an LDT full of 16-bit code segment descriptors.
That way if the BIOS does "mov ax,0x0040; mov ds,ax" it'd actually work correctly. If the BIOS does "mov ax,cs; mov ds, ax" it'd generate a general protection fault when the BIOS tries to use DS, and the general protection fault handler could detect that DS is a code segment in the LDT and change DS to the equivelent data segment in the GDT. It won't fix all segment related problems, but it'd improve your chances..
Cheers,
Brendan