ASM Function Not Returning

[Matt]

I have yet another kernel (this one runs in virtual 8086 mode) that is mixed c and ASM.
a function called _SwitchBank0: which switches the video memory bank to 0.
When I try to call this function from my C file though, it never returns! Here is part of the asm code:

global _SwitchBank0
extern _drawstuff

...

main:

;Set Video Mode here (800x600x256)

call _drawstuff

loop:
jmp loop ;Hang

_SwitchBank0:
mov ax, 04F05h
mov dx, 0
xor bx, bx
V86_CALL 10h

mov ax, 04F05h
mov dx, 0
xor bx, bx
inc bx
V86_CALL 10h

ret

Now, most of the C file:

extern void SwitchBank0();

void drawstuff() {
SwitchBank0();
//draw some stuff
....

What happens is my kernel hangs at SwitchBank0. I am able to put the bank switching code right infront of 'call _drawstuff' in the ASM file and it works just fine, so I assume its a problem with the function. It doesn't return! Any ideas?

BTW, it draws just fine if I remove SwitchBank0() from my C file.

[glow=red,2,300][shadow=red,left,300]Thanks[/shadow][/glow]

[Pype.Clicker]

You should remember that C widely uses ebp register for its stack management. Any function that is called by a C program should first push ebp and pop ebp before returning. I know you don't spill ebp yourself, but can you guarantee that this "V86_CALL" code doesn't ...

[Matt]

hm... I tried pushing and popping ebp but it still doesn't return. Any other registers C uses that I should try?

[Matt]

Actually, its not returning even if I put 'jmp _SwitchBank0' in my ASM file right before I call _drawstuff! Could it be something to do with how my 'V86_CALL' function messes with registers? Here is the code for V86_CALL:

v86_call:
   regs   2            ;Pretend this is an interrupt.
                  ;EAX and the subroutine return
                  ;address are the 2 extra dwords
   mov   eax, esp         ;Use eax as frame pointer
   rol   esi, 16            ;Swap halves
   mov   [c_cs+eax], esi         ;Store cs
   shr   esi, 16            ;Zero extended
   mov   [c_eip+eax], esi      ;Store ip
   lea   esi, [v_lim+eax]      ;Other end of interrupt frame
   pop   dword [esi]         ;Save return address
   cli               ;No ints while esp0 invalid
   xchg   esi, [esp0]         ;Get old frame limit
                  ;Set new frame limit
   mov   [v_lim+4+eax], esi      ;Save old frame limit
   movzx   ebp, word [c_ss+esi-v_lim]   ;Get V86 ss
   movzx   esi, word [c_esp+esi-v_lim]   ;Get V86 sp
   shl   ebp, 1            ;(need ss*16 later)
   sub   esi, byte 6         ;Make room on V86 stack
   mov   word [ebp*8+esi], unwind_sw   ;Store return address
   mov   dword [ebp*8+esi+2], 0x3002FFFF ;Store return cs and flags
   shr   ebp, 1            ;Restore V86 ss
   xchg   ebp, [c_ss+eax]         ;Set ss, Get ebp
   xchg   esi, [c_esp+eax]      ;Set sp, Get esi
   pop   eax
   iretd               ; "CALL" V86 interrupt routine

[Matt]

thats strange... Now my computer resets when I try to switch banks more than once (manually)... I'm completely lost :-/

[Pype.Clicker]

Actually, its not returning even if I put 'jmp _SwitchBank0' in my ASM file right before I call _drawstuff! Could it be something to do with how my 'V86_CALL' function messes with registers? Here is the code for V86_CALL:

Code: Select all

v86_call:
   regs   2            ;Pretend this is an interrupt.
                  ;EAX and the subroutine return
                  ;address are the 2 extra dwords
   mov   eax, esp         ;Use eax as frame pointer
   rol   esi, 16            ;Swap halves
   mov   [c_cs+eax], esi         ;Store cs
   shr   esi, 16            ;Zero extended

hmm ... dangerous code around here:
- do ss and ds segments have identical bases ? if not, [esp] or [ebp] is related to ss while [eax] is related to ds ... just check, but a mov ebp,esp instead of mov eax,esp could be more intuitive ...

- what is the initial value of esi ??
- mov [c_cs + eax], esi is going to erase a place on your stack with your cs-based value. If c_cs is >0, this means you're overwriting some previous frame informations ... if c_cs is negative, then you're wildly using some space you did not allocate (with a sub esp, N) before, and thus the value is prone to overwriting on interrupt/function call ...

Code: Select all

   mov   [c_eip+eax], esi      ;Store ip
   lea   esi, [v_lim+eax]      ;Other end of interrupt frame
   pop   dword [esi]         ;Save return address

where is the corresponding push ?

Code: Select all

  cli               ;No ints while esp0 invalid
   xchg   esi, [esp0]         ;Get old frame limit
                  ;Set new frame limit
   mov   [v_lim+4+eax], esi      ;Save old frame limit
   movzx   ebp, word [c_ss+esi-v_lim]   ;Get V86 ss
   movzx   esi, word [c_esp+esi-v_lim]   ;Get V86 sp
   shl   ebp, 1            ;(need ss*16 later)
   sub   esi, byte 6         ;Make room on V86 stack
   mov   word [ebp*8+esi], unwind_sw   ;Store return address
   mov   dword [ebp*8+esi+2], 0x3002FFFF ;Store return cs and flags
   shr   ebp, 1            ;Restore V86 ss
   xchg   ebp, [c_ss+eax]         ;Set ss, Get ebp
   xchg   esi, [c_esp+eax]      ;Set sp, Get esi
   pop   eax
   iretd               ; "CALL" V86 interrupt routine

Obviously, your function *cannot* return because it's ended by some code that returns to V86 mode ... what you need here is an interrupt handler that will detect the end of the V86 call (i.e. because of a GPF - try to return to a HLT instruction, for instance: it's easy to implement and to detect) and will then restore the pmode task that called the V86 service, transfering the result state to the caller task ...

Now, this is far from being simple ... May the Source be with You

[Matt]

This is the code right below V86_CALL:
SEGMENT V86
unwind_sw:   int   46h            ;Get out of V86 mode
SEGMENT CODE

; The above instruction will bring us in here to clean up the stack and
; really return.
;_____________________________________________________________________________
service_46:
patch_vec 0x46, service_46, D_INT+D_DPL3   ;Patch vector to point here
   regs   0, eax
   mov   eax, ss            ;Set up segment registers
   mov   ds, eax
   mov   es, eax
   mov   eax, [v_lim+4+esp]      ;Get prior esp0
   mov   [esp0], eax         ;Restore it
   jmp   [v_lim+esp]         ;Return

I suppose thats used to return. Maybe I'd be better off in plain pmode using the vesa pmode interface for bank switching

[Pype.Clicker]

for sure if you're just concerned in bank switching, then VBE2/3 pmode interface is definitively what you need...

Also consider using /flat/ video modes (mapping all your video memory at some higher physical addresses, so that no bank switching is required anymore

[Berserk]

what is the above code all about??

Does it initialise v86 emulation?? & What is the part that says: "800x600x256" about? did u actually get this resolution??