OSDev.org

Hi,

I've recently tried to implement vm86 mode in my software task switching code. However, I've run into a problem. vm86 requires accurate segments (for CS and the like) before it can execute code and read data properly. These are imcompatible with pMode, so when I try to switch to a vm86 task, bochs complains that the new segment value (for 16 bit) is past the GDT limit, as heres my task switching code:

[code]
_ChangeTasks:

cli

push ds
push es
push fs
push gs

push eax
push ebx
push ecx
push edx
push esi
push edi
push ebp

mov eax,0x10
mov ds,eax
mov es,eax
mov fs,eax
mov gs,eax
mov ss,eax

mov ebx,[_currentTask]
shl ebx,3

lea esi,[_stackPointers]
mov [ds:ebx+esi],esp
mov [ds:ebx+esi+4],ss

mov ebx,[_nextTask]
mov [_currentTask],ebx
shl ebx,3

mov esp,[ds:ebx+esi]
mov ss,[ds:ebx+esi+4]

pop ebp
pop edi
pop esi
pop edx
pop ecx
pop ebx
pop eax

pop gs
pop fs
pop es
pop ds

sti

ret
[/code]

How can I prevent this, and has anyone implemented a vm86 task in a software task switcher?
Also, do I need a PL3 code and data selectors for switching to a vm86 task?

CloudNine

I think the problem here is that your vm86 task isn't actually switching into vm86 mode (otherwise, you wouldn't get a GDT limit error from bochs -- you can only get a GDT error in pmode).  The ChangeTask function above doesn't seem to load the new EFLAGS, so if you're going into it in pmode, you'll be in pmode all the way through it, but if those "pop gs/fs/es/ds" instructions are trying to pull vm86 segment offsets instead of pmode segment descriptors, you're hosed.

You'll need to either use a TSS, or something other than your typical task switching code to handle vm86 tasks.