Entering into VM8086 from assembler

[Klakap]

Good day!

Please, how I can enter to VM8086 mode completly from real mode(or in protected mode) in assembler? Wiki contiains only code for call from C code and I dont know how C pushing arguments into stack. Thank for answers!

[MichaelPetch]

I recommend looking at my Stackoverflow Question/Answer. The Q&A was inspired by a question previously asked of me by someone else. It is a discussion of the two primary mechanisms you may find useful in getting into v8086 mode. The code in the answers is all done as a bootloader that switches into protected mode and then enters v8086 mode. The code doesn't do anything useful beyond getting into protected mode and displaying something on the video display directly.

---
I have a small Multiboot based project that was supposed to be one part of an answer to another OSDev question that was asked a few months ago (but I never formalized). I intend to write an Stackoverflow answer about it. Of interest to you might be the enter_v86 function in pmutils.asm. The project is both NASM code and GCC code. Entry into v8086 mode is done in NASM. There is both 16-bit and 32-bit GCC code in the project. Files with .RMC instructions are source C files that are built with the `-m16` option and .RMH are headers specific to real mode code.

This code isn't a full v8086 monitor. It demonstrates handling the 16-bit IRET and INT n instructions only and uses UD2 instruction to exit the monitor. It works with Interrupts Flag off (IF=0) to simplify the demonstration code. The code uses an IO Privilege Level (IOPL=0). Extra work would be needed to use IOPL=3 with this code.

When an exception occurs in v8086 mode the stack frame looks like (each value occupies 4 bytes of space on the stack):

GS
FS
DS
ES
SS
ESP/SP
EFLAGS/FLAGS
CS
EIP/IP
Error Code (If applicable to an exception)

In order to enter v8086 mode the same stack layout is used by enter_v86 function and the VM bit in EFLAGS is set to 1 and IOPL=0. An IRET instruction is used for getting into v8086 mode as it had the ability to set the VM flag.

A version of enter_v86 (the code is assumed to be running in 32-bit protected mode already) that uses IOPL=0; hard codes the segment registers to 0; sets a real mode stack at 0x0000:8000; and doesn't provide a mechanism of coming out of v8086 mode and restoring protected mode state; and starts executing code at 0x0000:0x1000 in v8086 mode could look like:

Code: Select all

EFLAGS_VM_BIT          EQU 17      ; EFLAGS VM bit
EFLAGS_BIT1            EQU 1       ; EFLAGS bit 1 (reserved, always 1)
enter_v86:
    ; Set up initial v8086 stack frame
    xor eax, eax                ; EAX=0
    push eax                    ; Real mode GS=0
    push eax                    ; Real mode FS=0
    push eax                    ; Real mode DS=0
    push eax                    ; Real mode ES=0
    push 0x8000
    push 0xFFFE                 ; v8086 stack SS:SP (grows down from 0x8000:0xFFFE)
    push 1<<EFLAGS_VM_BIT | 1<<EFLAGS_BIT1
                                ; Set VM and BIT1 in flags and IOPL=0 (All other flags 0)
    push 0x0000                 ; Push entry point (segment)
    push 0x1000                 ; Push entry point (offset) start executing at 0x0000:0x1000

    mov ebx, eax                ; Set all the general purpose registers to 0
    mov ecx, eax                ;     EAX was set to 0 above
    mov edx, eax
    mov esi, eax
    mov edi, eax
    mov ebp, eax

    iret                        ; Transfer control to v8086 mode and our real mode code

To make a useful v8086 monitor you will need to set up a Task State Segment (TSS).

---
Please note: I don't know your intentions for using v8086 mode. You can't just enter v8086 mode and execute arbitrary real mode code and expect it to work without a proper monitor that traps and handles/emulates privileged and interrupt sensitive instructions. Without a proper v8086 monitor you'd be severely restricted in the type of code you could run. If you are intending to enter v8086 mode to run arbitrary real mode code and return to 32-bit protected mode you might be better off switching to regular real mode; run the real mode code; return to 32-bit protected mode.

Programming a v8086 monitor is not recommended for anyone who doesn't have a very good grasp of x86 instruction set; real mode; interrupt/exception handling; and the x86 architecture in general. IMHO Creating a v8086 monitor is not trivial and should not be taken on as a beginner level project.

[Klakap]

Thank you very much for explanation! It seems than VM8086 isnt good for me.