[assembly] pmode - ret not working (solved)

[StefanWouters]

Hello everybody!
I am playing with protected mode. My bootloader jumps to pmode and starts the kernel in 32 bit mode.
After this, everything seems to be working correctly except for ret.
QEMU just reboots when I try to use ret to return to the previous location after a call.

Can someone point me into a general direction where to look? If you need to see some of my code, please ask for it. I'll post it here.

Thanks!

- Stefan

Edit: I forgot to mention that I am using assembly to write the operating system. (nasm for compilation)

[StefanWouters]

I'm very sorry. This message should probably have been posted in the OS Development section.

Could an admin please move this thread to the appropriate section on the forum and remove this reply?

[xenos]

Do i get this right: You are in real mode, call a function, switch to protected mode within this function, and then try to return? Well, that won't work. If you call a function in real mode, a 16 bit return address is pushed to the stack. If you perform a return in protected mode, a 32 bit return address is popped off the stack - and this won't match the 16 bit pushed when calling the function.

[StefanWouters]

Thank you for your reply, XenOS.

I'm sorry. This is not what I meant.

I went to pmode from the bootloader using a far jump.
After that I run some code from within the bootloader and make far jump to reach the kernel code.
Like this:

Code: Select all

[...]
call	os_init_gdt
mov		eax, cr0
or		al, 1
mov		cr0, eax
jmp		08h:pmode

BITS 32
pmode:
; Set data segments to ds (0x10)
mov		ax, 0x10
mov ds,ax
mov es,ax
mov fs,ax
mov gs,ax
mov ss,ax
mov		esp, 90000h				; stack esp
mov		ebp, 90000h				; stack ebp

cli

jmp		08h:02000h			                ; jump to the kernel (previously loaded into memory)

I then arrive in the kernel code, which was loaded from floppy into a location in memory.

Code: Select all

BITS 	32
jmp	os_main
[...]
os_main:

call    os_function   ; call a function
hlt                       ; crash!

os_function:
[...]                          ; here I am able to directly access video memory, I clear the screen and do some random painting
ret

The above code is working. In os_function I access the video memory and I'm able paint directly to the screen. I initialized VESA2.0 in the bootloader.
Once the code reaches ret, qemu reboots (I presume triple fault).
If I replace ret by hlt (which would've been executed on return anyway), I can see the amazingly boring painting I've made in the video memory and qemu just hangs on the hlt command.

I hope this clears up my problem a little bit.

Edit: By the way, I thought that maybe I made a mistake creating the stack in protected mode. However, in the graphics code I use pushes and pops and they work fine. Because of this, can I assume my stack is in order?

[xenos]

Perhaps it would be useful to see some debugger output from qemu. For example, you could add -d cpu_reset to the qemu command line, to obtain debugger output written to /tmp/qemu.log. It should tell you the register contents and perhaps also the reason for the triple fault, which might be very helpful.

Your code looks fine for me, but maybe you should check that your os_function doesn't mess up the stack. It sounds trivial, but you should check that you have an equal number of pop and push, don't pop the return address and replace it with something else etc.

[StefanWouters]

When I add the -d part to the qemu command line, qemu won't start at all.
I'm running qemu on 64 bit Windows 7. (not by choice, I'm using my office notebook for development)

Anyway, I have removed all the code except for the ret command from os_function like so:

Code: Select all

BITS    32
jmp   os_main
[...]
os_main:

call    os_function   ; call a function
hlt                       ; crash!

os_function:
ret

This does not work.
So, I've moved the call and ret to the bootloader, which now looks like this:

Code: Select all

BITS 32
pmode:
; Set data segments to ds (0x10)
mov      ax, 0x10
mov ds,ax
mov es,ax
mov fs,ax
mov gs,ax
mov ss,ax
mov      esp, 90000h            ; stack esp
mov      ebp, 90000h            ; stack ebp

cli

call os_function
    hlt   

os_function:
    ret

Now the code does exactly what I expect it to do.
So, somehow, after a jump out of the boot loader code causes the error.

Code: Select all

jmp        08h:02000h

Now, I would like to see debugger output. I'll try to figure out how to get debug info on reboot in qemu for Windows.
Until then, maybe the above information is useful.

I like a challenge, so if someone can point me into some direction of where to look, I'll try to figure it out myself.

[StefanWouters]

Solved!

I've copied the following code into the kernel:

Code: Select all

    mov        ax, 0x10                ; Set data segments to data selector (0x10)
    mov ds,ax
    mov es,ax
    mov fs,ax
    mov gs,ax
    mov ss,ax
    mov        esp, 90000h                ; Stack begins from 90000h
    mov        ebp, 90000h                ; Stack begins from 90000h

This is the same code that's in the boot loader.
Somehow, this helps... although I would like to understand why. Is my memory out of alignment after a far jump?