Dropping to real mode on VMWare/Virtualbox [solved]

bigorenski · Post by **bigorenski** » Fri Sep 26, 2025 5:11 pm

Hi. It's been a long time since my last post here... Hope you’re all doing well.

My OS is running in long mode, paging enabled, all cores enabled.
It's current driver model loads ELFs and runs them in ring0 as separate tasks.

I recently developed a 'bios driver' that will later be used by the VESA driver to change video modes. This driver drops to real mode and executes an interrupt, then returns to long mode.
This driver uses a 'thunk' compiled to a flat binary then converted to a C array, that is then copied to memory and executed.

In both Bochs and QEmu it works flawlessly (for now I'm just using it to set vga video mode 0x13).
On Virtualbox it doesn't do anything, doesn't put the machine in "guru meditation", nothing... Seems like the bios call never happened.
On VMWare it triple faults.

I've looked at logs from both but couldn't figure what happened. Logs are huge, and I'm not sure what to look for.

My question is:
Is dropping from long mode back to real mode simply unsupported by VirtualBox/VMware, or is there likely a problem with my implementation?

Relevant code portions:

Installation of the thunk:

Code: Select all

bool install_thunk_locked() {
    if (g_thunk_ready) {
        return true;
    }
    if (build_arch_x86_bioscall_thunk_o_len > kThunkCodeCapacity) {
        bochs_printf("[bioscall] thunk binary too large (%u bytes)\n",
                     static_cast<unsigned>(build_arch_x86_bioscall_thunk_o_len));
        return false;
    }

    auto* frame = thunk_frame();
    if (sizeof(*frame) > kThunkFrameCapacity) {
        bochs_printf("[bioscall] thunk frame too large (%u bytes)\n",
                     static_cast<unsigned>(sizeof(*frame)));
        return false;
    }

    auto* dest = reinterpret_cast<uint8_t*>(kThunkCodePhysical);
    for (size_t i = 0; i < build_arch_x86_bioscall_thunk_o_len; ++i) {
        dest[i] = build_arch_x86_bioscall_thunk_o[i];
    }

    memset(call_area(), 0, sizeof(bioscall_params));
    memset(frame, 0, sizeof(*frame));

    frame->gdt[0] = 0x0000000000000000ull;
    frame->gdt[1] = 0x00209A0000000000ull;
    frame->gdt[2] = 0x00CF92000000FFFFull;
    frame->gdt[3] = 0x00209A0000000000ull;
    frame->gdt[4] = 0x00CF92000000FFFFull; // 64-bit data segment with full limit for stack operations
    frame->gdt[5] = 0x00009A000000FFFFull;
    frame->gdt[6] = 0x000092000000FFFFull;
    frame->gdt[7] = 0x00CF9A000000FFFFull; // Dedicated 32-bit code segment for thunk transitions
    frame->gdt_descriptor.limit = static_cast<uint16_t>(sizeof(frame->gdt) - 1u);

    frame->frame_segment = static_cast<uint16_t>(kThunkFramePhysical >> 4);
    frame->frame_base32 = static_cast<uint32_t>(kThunkFramePhysical);
    frame->thunk_base32 = static_cast<uint32_t>(kThunkCodePhysical);
    frame->thunk_base64 = static_cast<uint64_t>(kThunkCodePhysical);
    frame->call_area_ptr = static_cast<uint16_t>(kCallAreaPhysical);

    g_thunk_ready = true;
    return true;
}

Call the thunk:

Code: Select all

    bioscall_thunk_frame* frame = thunk_frame();
    frame->call_area_ptr = static_cast<uint16_t>(kCallAreaPhysical);
    bioscall_thunk_fn thunk = thunk_entry();
    
    uint64_t result = thunk(area, frame); //here
    
    //while(1);
    //__asm__ __volatile__("xchg %bx, %bx");

    memcpy(params, area, sizeof(bioscall_params));
    spin_unlock();

    return result == 0 ? DRV_OPERATION_OK : DRV_OPERATION_FAILED;

Thunk:

Code: Select all

; BIOS real-mode thunk for bioscall driver
; This blob is copied to a low identity-mapped physical address by the driver.
; It accepts a pointer (RDI) to a bioscall_params structure located below 1MB.
; The thunk transitions from long mode to 16-bit real mode, invokes the BIOS
; interrupt encoded in the structure, captures the register results and
; returns to long mode.

BITS 64

DEFAULT REL

%define THUNK_PHYS_BASE            0x00007000
%define FRAME_PHYS_BASE            0x00005000
%define CALL_AREA_PHYS_BASE        0x00006000

%define CODE32_SEL 0x38
%define DATA32_SEL 0x10
%define CODE64_SEL 0x18
%define DATA64_SEL 0x20
%define CODE16_SEL 0x28
%define DATA16_SEL 0x30

%define BIOSCALL_OPT_DEFAULT_STACK 0x0001


; Offsets within bioscall_params (must match driver/ header definitions)
%define PARAM_INTERRUPT_OFFSET    0
%define PARAM_RESERVED_OFFSET     1
%define PARAM_OPTIONS_OFFSET      2
%define PARAM_REGS_IN_OFFSET      4
%define PARAM_REGS_OUT_OFFSET     (PARAM_REGS_IN_OFFSET + 28)

; Offsets within bioscall_registers
%define REGS_AX_OFFSET    0
%define REGS_BX_OFFSET    2
%define REGS_CX_OFFSET    4
%define REGS_DX_OFFSET    6
%define REGS_SI_OFFSET    8
%define REGS_DI_OFFSET    10
%define REGS_BP_OFFSET    12
%define REGS_SP_OFFSET    14
%define REGS_DS_OFFSET    16
%define REGS_ES_OFFSET    18
%define REGS_FS_OFFSET    20
%define REGS_GS_OFFSET    22
%define REGS_SS_OFFSET    24
%define REGS_FLAGS_OFFSET 26

; Offsets within bioscall_thunk_frame (must match header definition)
%define FRAME_SAVED_RSP           0
%define FRAME_SAVED_RBP           8
%define FRAME_SAVED_RFLAGS        16
%define FRAME_SAVED_RETURN_RIP    24
%define FRAME_SAVED_CR0           32
%define FRAME_SAVED_CR3           40
%define FRAME_SAVED_CR4           48
%define FRAME_SAVED_EFER_LOW      56
%define FRAME_SAVED_EFER_HIGH     60
%define FRAME_CALL_PARAMS_PTR64   64
%define FRAME_CALL_PARAMS_PTR32   72
%define FRAME_CALL_AREA_PTR       76
%define FRAME_FRAME_SEGMENT       78
%define FRAME_FRAME_BASE32        80
%define FRAME_THUNK_BASE64        84
%define FRAME_THUNK_BASE32        92
%define FRAME_RM_TEMP             100
%define FRAME_RM_FINAL_FLAGS      128
%define FRAME_PM32_STACK          132
%define FRAME_PM32_STACK_SIZE     512
%define FRAME_GDT                 644
%define FRAME_PM32_ENTRY_PTR      708
%define FRAME_PM32_RETURN_PTR     718
%define FRAME_LONG_MODE_RETURN_PTR 728
%define FRAME_SAVED_GDT_DESCRIPTOR 738
%define FRAME_GDT_DESCRIPTOR      750
%define FRAME_SAVED_IDTR          762
%define FRAME_RM_IDTR             772
%define FRAME_PM32_IDTR           782
%define FRAME_PM32_IDT            792
%define FRAME_PM32_IDT_ENTRIES    32
%define FRAME_SAVED_SEGMENTS      1048
%define FRAME_SAVED_CS            (FRAME_SAVED_SEGMENTS + 0)
%define FRAME_SAVED_DS            (FRAME_SAVED_SEGMENTS + 2)
%define FRAME_SAVED_ES            (FRAME_SAVED_SEGMENTS + 4)
%define FRAME_SAVED_FS            (FRAME_SAVED_SEGMENTS + 6)
%define FRAME_SAVED_GS            (FRAME_SAVED_SEGMENTS + 8)
%define FRAME_SAVED_SS            (FRAME_SAVED_SEGMENTS + 10)

%define FAR_PTR_OFFSET_OFFSET     0
%define FAR_PTR_SELECTOR_OFFSET   8

%define CR0_PG_BIT                (1 << 31)
%define CR0_PE_BIT                (1 << 0)

%define CR4_PGE_BIT               (1 << 7)
%define CR4_PCIDE_BIT             (1 << 17)
%define CR4_SMEP_BIT              (1 << 20)
%define CR4_SMAP_BIT              (1 << 21)
%define CR4_PKE_BIT               (1 << 22)
%define CR4_CET_BIT               (1 << 23)
%define CR4_CET_SHSTK_BIT         (1 << 24)
%define CR4_PAE_BIT               (1 << 5)
%define CR4_PSE_BIT               (1 << 4)
%define CR4_LA57_BIT              (1 << 12)

%define CR4_BITS_REQUIRE_PG_PRE   (CR4_PGE_BIT | CR4_PCIDE_BIT | CR4_SMEP_BIT | \
                                    CR4_SMAP_BIT | CR4_PKE_BIT | CR4_CET_BIT | \
                                    CR4_CET_SHSTK_BIT)
%define CR4_BITS_AFTER_LME_CLEAR  (CR4_PAE_BIT | CR4_PSE_BIT | CR4_LA57_BIT)

%define REGS_SIZE         28

%define DEFAULT_STACK_SP 0x7C00
%define DEFAULT_STACK_SS 0x0000

section .text

; Entry point called from long mode with SysV ABI. RDI points to bioscall_params.
; Returns 0 on success in RAX. Clobbers caller-save registers.

%define THUNK_SUCCESS 0

%define INT_DISPATCH_PATCH_ADDR \
    (THUNK_PHYS_BASE + (int_dispatch - bioscall_thunk_entry) + 1)

global bioscall_thunk_entry
bioscall_thunk_entry:
    push rbp
    mov rbp, rsp
    push rbx
    push r12
    push r13
    push r14
    push r15

    cli

    mov r12, rsi
    mov [frame_ptr64], r12

    mov [r12 + FRAME_SAVED_RSP], rsp
    mov [r12 + FRAME_SAVED_RBP], rbp

    pushfq
    pop qword [r12 + FRAME_SAVED_RFLAGS]

    mov rax, [rbp + 8]
    mov [r12 + FRAME_SAVED_RETURN_RIP], rax

    mov word [r12 + FRAME_SAVED_CS], cs
    mov word [r12 + FRAME_SAVED_DS], ds
    mov word [r12 + FRAME_SAVED_ES], es
    mov word [r12 + FRAME_SAVED_FS], fs
    mov word [r12 + FRAME_SAVED_GS], gs
    mov word [r12 + FRAME_SAVED_SS], ss

    ; Enforce a default stack if none was supplied
    movzx eax, word [rdi + PARAM_REGS_IN_OFFSET + REGS_SS_OFFSET]
    or ax, ax
    jne .have_ss
    mov word [rdi + PARAM_REGS_IN_OFFSET + REGS_SS_OFFSET], DEFAULT_STACK_SS
.have_ss:
    movzx eax, word [rdi + PARAM_REGS_IN_OFFSET + REGS_SP_OFFSET]
    or ax, ax
    jne .have_sp
    mov word [rdi + PARAM_REGS_IN_OFFSET + REGS_SP_OFFSET], DEFAULT_STACK_SP
.have_sp:

    ; Persist a 32-bit pointer for the 32-bit and 16-bit stages
    mov [r12 + FRAME_CALL_PARAMS_PTR64], rdi
    mov eax, edi
    mov [r12 + FRAME_CALL_PARAMS_PTR32], eax
    mov [r12 + FRAME_CALL_AREA_PTR], ax
    mov eax, esi
    mov [r12 + FRAME_FRAME_BASE32], eax
    mov edx, eax
    shr edx, 4
    mov [r12 + FRAME_FRAME_SEGMENT], dx

    ; Save control register state
    mov rax, cr0
    mov [r12 + FRAME_SAVED_CR0], rax
    mov rax, cr3
    mov [r12 + FRAME_SAVED_CR3], rax
    mov rax, cr4
    mov [r12 + FRAME_SAVED_CR4], rax
    mov ecx, 0xC0000080
    rdmsr
    mov [r12 + FRAME_SAVED_EFER_LOW], eax
    mov [r12 + FRAME_SAVED_EFER_HIGH], edx

    ; Record runtime base for debugging and load GDT descriptor
    lea rbx, [rel bioscall_thunk_entry]
    mov [r12 + FRAME_THUNK_BASE64], rbx

    sgdt [r12 + FRAME_SAVED_GDT_DESCRIPTOR]
    mov word [r12 + FRAME_SAVED_GDT_DESCRIPTOR + 10], 0

    lea rax, [r12 + FRAME_GDT]
    mov [r12 + FRAME_GDT_DESCRIPTOR + 2], rax
    mov word [r12 + FRAME_GDT_DESCRIPTOR + 10], 0

    lea rax, [rel pm32_entry]
    mov [r12 + FRAME_PM32_ENTRY_PTR + FAR_PTR_OFFSET_OFFSET], rax
    mov word [r12 + FRAME_PM32_ENTRY_PTR + FAR_PTR_SELECTOR_OFFSET], CODE32_SEL
    lea rax, [rel pm32_return]
    mov [r12 + FRAME_PM32_RETURN_PTR + FAR_PTR_OFFSET_OFFSET], rax
    mov word [r12 + FRAME_PM32_RETURN_PTR + FAR_PTR_SELECTOR_OFFSET], CODE32_SEL
    lea rax, [rel long_mode_return]
    mov [r12 + FRAME_LONG_MODE_RETURN_PTR + FAR_PTR_OFFSET_OFFSET], rax
    mov word [r12 + FRAME_LONG_MODE_RETURN_PTR + FAR_PTR_SELECTOR_OFFSET], CODE64_SEL

    sidt [r12 + FRAME_SAVED_IDTR]
    mov word [r12 + FRAME_RM_IDTR], 0x03FF
    mov qword [r12 + FRAME_RM_IDTR + 2], 0

    ; Prepare a minimal 32-bit IDT (all entries point to pm32 fault stub)
    mov word [r12 + FRAME_PM32_IDTR], FRAME_PM32_IDT_ENTRIES * 8 - 1
    lea rax, [r12 + FRAME_PM32_IDT]
    mov [r12 + FRAME_PM32_IDTR + 2], rax

    lea eax, [rel pm32_fault_stub]
    mov edx, eax
    shr eax, 16
    mov ecx, FRAME_PM32_IDT_ENTRIES
    lea rbx, [r12 + FRAME_PM32_IDT]
.fill_pm32_idt:
    mov word [rbx], dx
    mov word [rbx + 2], CODE32_SEL
    mov byte [rbx + 4], 0
    mov byte [rbx + 5], 0x8E
    mov word [rbx + 6], ax
    add rbx, 8
    loop .fill_pm32_idt

    lgdt [r12 + FRAME_GDT_DESCRIPTOR]

    ; Ensure segment registers use the thunk data segment before switching
    mov ax, DATA64_SEL
    mov ds, ax
    mov es, ax
    mov fs, ax
    mov gs, ax
    mov ss, ax

    ; Switch to a thunk-local stack before mode transitions
    lea rsp, [r12 + FRAME_PM32_STACK + FRAME_PM32_STACK_SIZE]

    ; Transfer to 32-bit protected mode portion
    jmp far [r12 + FRAME_PM32_ENTRY_PTR]

BITS 32

DEFAULT ABS

pm32_entry:
    mov ax, DATA32_SEL
    mov ds, ax
    mov es, ax
    mov fs, ax
    mov gs, ax
    mov ss, ax

    mov ebx, esi

    ; Clear paging-dependent CR4 features while PG is still set
    mov eax, [ebx + FRAME_SAVED_CR4]
    mov edx, eax
    and edx, ~CR4_BITS_REQUIRE_PG_PRE
    mov cr4, edx

    ; Drop paging
    mov eax, [ebx + FRAME_SAVED_CR0]
    and eax, 0x7FFFFFFF
    mov cr0, eax

    ; Disable long mode
    mov ecx, 0xC0000080
    rdmsr
    and eax, ~(1 << 8)
    wrmsr

    ; Clear remaining CR4 paging extensions
    mov eax, cr4
    and eax, ~CR4_BITS_AFTER_LME_CLEAR
    mov cr4, eax

    ; Drop into a 16-bit protected-mode segment (D=0) before disabling PE

    mov eax, [ebx + FRAME_THUNK_BASE32]
    add eax, pm16_entry - bioscall_thunk_entry
    push word CODE16_SEL
    push eax
    retf

BITS 16

pm16_entry:
    mov ax, DATA16_SEL
    mov ds, ax
    mov es, ax
    mov fs, ax
    mov gs, ax
    mov ss, ax

    mov eax, cr0
    and eax, 0xFFFFFFFE
    mov cr0, eax

    xor cx, cx
    mov dx, real_mode_entry - bioscall_thunk_entry
    add dx, THUNK_PHYS_BASE & 0xFFFF
    push word cx
    push word dx

    retf

BITS 32

DEFAULT ABS

pm32_return:
    mov ax, DATA32_SEL
    mov ds, ax
    mov es, ax
    mov fs, ax
    mov gs, ax
    mov ss, ax

    mov ebx, esi

    lidt [ebx + FRAME_SAVED_IDTR]

    ; Restore CR4 without PG-dependent bits
    mov eax, [ebx + FRAME_SAVED_CR4]
    mov edx, eax
    and edx, ~CR4_BITS_REQUIRE_PG_PRE
    mov cr4, edx

    ; Restore CR3 prior to enabling paging
    mov eax, [ebx + FRAME_SAVED_CR3]
    mov cr3, eax

    ; Restore EFER (re-enables LME)
    mov ecx, 0xC0000080
    mov eax, [ebx + FRAME_SAVED_EFER_LOW]
    mov edx, [ebx + FRAME_SAVED_EFER_HIGH]
    wrmsr

    ; Restore CR0
    mov eax, [ebx + FRAME_SAVED_CR0]
    mov cr0, eax

    ; Restore full CR4 state
    mov eax, [ebx + FRAME_SAVED_CR4]
    mov cr4, eax
    
    ; Build a 16:32 far pointer on the stack for the long-mode jump.
    ; The stored long_mode_return_ptr uses a 64-bit offset layout, so shuffle
    ; the pieces into the format expected by protected-mode far control transfers.
    mov eax, [ebx + FRAME_LONG_MODE_RETURN_PTR + FAR_PTR_OFFSET_OFFSET]
    mov cx, [ebx + FRAME_LONG_MODE_RETURN_PTR + FAR_PTR_SELECTOR_OFFSET]
    sub esp, 8
    mov [esp], eax
    mov [esp + 4], cx
    
    jmp far [esp]

pm32_fault_stub:
    cli
.pm32_fault_halt:
    hlt
    jmp .pm32_fault_halt

BITS 16

real_mode_entry:
    cli
    mov ax, (FRAME_PHYS_BASE >> 4)
    mov fs, ax
    
    lidt [fs:FRAME_RM_IDTR]
    mov ax, (CALL_AREA_PHYS_BASE >> 4)
    mov gs, ax
    mov ds, ax
    mov es, ax
    xor si, si

    mov al, [gs:si + PARAM_INTERRUPT_OFFSET] ;Overwrite 0x00 after 'int' instruction on 'int_dispatch:'
    mov byte [cs:INT_DISPATCH_PATCH_ADDR], al

    mov ax, [gs:si + PARAM_REGS_IN_OFFSET + REGS_AX_OFFSET]
    mov [fs:FRAME_RM_TEMP + REGS_AX_OFFSET], ax
    mov ax, [gs:si + PARAM_REGS_IN_OFFSET + REGS_BX_OFFSET]
    mov [fs:FRAME_RM_TEMP + REGS_BX_OFFSET], ax
    mov ax, [gs:si + PARAM_REGS_IN_OFFSET + REGS_CX_OFFSET]
    mov [fs:FRAME_RM_TEMP + REGS_CX_OFFSET], ax
    mov ax, [gs:si + PARAM_REGS_IN_OFFSET + REGS_DX_OFFSET]
    mov [fs:FRAME_RM_TEMP + REGS_DX_OFFSET], ax
    mov ax, [gs:si + PARAM_REGS_IN_OFFSET + REGS_SI_OFFSET]
    mov [fs:FRAME_RM_TEMP + REGS_SI_OFFSET], ax
    mov ax, [gs:si + PARAM_REGS_IN_OFFSET + REGS_DI_OFFSET]
    mov [fs:FRAME_RM_TEMP + REGS_DI_OFFSET], ax
    mov ax, [gs:si + PARAM_REGS_IN_OFFSET + REGS_BP_OFFSET]
    mov [fs:FRAME_RM_TEMP + REGS_BP_OFFSET], ax
    mov ax, [gs:si + PARAM_REGS_IN_OFFSET + REGS_SP_OFFSET]
    mov [fs:FRAME_RM_TEMP + REGS_SP_OFFSET], ax
    mov ax, [gs:si + PARAM_REGS_IN_OFFSET + REGS_DS_OFFSET]
    mov [fs:FRAME_RM_TEMP + REGS_DS_OFFSET], ax
    mov ax, [gs:si + PARAM_REGS_IN_OFFSET + REGS_ES_OFFSET]
    mov [fs:FRAME_RM_TEMP + REGS_ES_OFFSET], ax
    mov ax, [gs:si + PARAM_REGS_IN_OFFSET + REGS_FS_OFFSET]
    mov [fs:FRAME_RM_TEMP + REGS_FS_OFFSET], ax
    mov ax, [gs:si + PARAM_REGS_IN_OFFSET + REGS_GS_OFFSET]
    mov [fs:FRAME_RM_TEMP + REGS_GS_OFFSET], ax
    mov ax, [gs:si + PARAM_REGS_IN_OFFSET + REGS_SS_OFFSET]
    mov [fs:FRAME_RM_TEMP + REGS_SS_OFFSET], ax

    push word [gs:si + PARAM_REGS_IN_OFFSET + REGS_FLAGS_OFFSET]
    popf

    mov ax, [fs:FRAME_RM_TEMP + REGS_SS_OFFSET]
    mov ss, ax
    mov sp, [fs:FRAME_RM_TEMP + REGS_SP_OFFSET]

    mov bx, [fs:FRAME_RM_TEMP + REGS_BX_OFFSET]
    mov cx, [fs:FRAME_RM_TEMP + REGS_CX_OFFSET]
    mov dx, [fs:FRAME_RM_TEMP + REGS_DX_OFFSET]
    mov si, [fs:FRAME_RM_TEMP + REGS_SI_OFFSET]
    mov di, [fs:FRAME_RM_TEMP + REGS_DI_OFFSET]
    mov ax, [fs:FRAME_RM_TEMP + REGS_BP_OFFSET]
    mov bp, ax

    mov ax, [fs:FRAME_RM_TEMP + REGS_DS_OFFSET]
    mov ds, ax
    mov ax, [fs:FRAME_RM_TEMP + REGS_ES_OFFSET]
    mov es, ax
    mov ax, [fs:FRAME_RM_TEMP + REGS_AX_OFFSET]
int_dispatch:
    int 0x00
    

    pushf
    pop dx

    mov [fs:FRAME_RM_TEMP + REGS_AX_OFFSET], ax
    mov [fs:FRAME_RM_TEMP + REGS_BX_OFFSET], bx
    mov [fs:FRAME_RM_TEMP + REGS_CX_OFFSET], cx
    mov [fs:FRAME_RM_TEMP + REGS_DX_OFFSET], dx
    mov [fs:FRAME_RM_TEMP + REGS_SI_OFFSET], si
    mov [fs:FRAME_RM_TEMP + REGS_DI_OFFSET], di
    mov [fs:FRAME_RM_TEMP + REGS_BP_OFFSET], bp
    mov ax, ds
    mov [fs:FRAME_RM_TEMP + REGS_DS_OFFSET], ax
    mov ax, es
    mov [fs:FRAME_RM_TEMP + REGS_ES_OFFSET], ax
    mov ax, ss
    mov [fs:FRAME_RM_TEMP + REGS_SS_OFFSET], ax
    mov ax, sp
    mov [fs:FRAME_RM_TEMP + REGS_SP_OFFSET], ax
    mov [fs:FRAME_RM_FINAL_FLAGS], dx

    mov ax, (CALL_AREA_PHYS_BASE >> 4)
    mov ds, ax
    mov es, ax
    mov gs, ax
    xor si, si

    mov ax, [fs:FRAME_RM_TEMP + REGS_AX_OFFSET]
    mov [gs:si + PARAM_REGS_OUT_OFFSET + REGS_AX_OFFSET], ax
    mov ax, [fs:FRAME_RM_TEMP + REGS_BX_OFFSET]
    mov [gs:si + PARAM_REGS_OUT_OFFSET + REGS_BX_OFFSET], ax
    mov ax, [fs:FRAME_RM_TEMP + REGS_CX_OFFSET]
    mov [gs:si + PARAM_REGS_OUT_OFFSET + REGS_CX_OFFSET], ax
    mov ax, [fs:FRAME_RM_TEMP + REGS_DX_OFFSET]
    mov [gs:si + PARAM_REGS_OUT_OFFSET + REGS_DX_OFFSET], ax
    mov ax, [fs:FRAME_RM_TEMP + REGS_SI_OFFSET]
    mov [gs:si + PARAM_REGS_OUT_OFFSET + REGS_SI_OFFSET], ax
    mov ax, [fs:FRAME_RM_TEMP + REGS_DI_OFFSET]
    mov [gs:si + PARAM_REGS_OUT_OFFSET + REGS_DI_OFFSET], ax
    mov ax, [fs:FRAME_RM_TEMP + REGS_BP_OFFSET]
    mov [gs:si + PARAM_REGS_OUT_OFFSET + REGS_BP_OFFSET], ax
    mov ax, [fs:FRAME_RM_TEMP + REGS_SP_OFFSET]
    mov [gs:si + PARAM_REGS_OUT_OFFSET + REGS_SP_OFFSET], ax
    mov ax, [fs:FRAME_RM_TEMP + REGS_DS_OFFSET]
    mov [gs:si + PARAM_REGS_OUT_OFFSET + REGS_DS_OFFSET], ax
    mov ax, [fs:FRAME_RM_TEMP + REGS_ES_OFFSET]
    mov [gs:si + PARAM_REGS_OUT_OFFSET + REGS_ES_OFFSET], ax
    mov ax, [fs:FRAME_RM_TEMP + REGS_FS_OFFSET]
    mov [gs:si + PARAM_REGS_OUT_OFFSET + REGS_FS_OFFSET], ax
    mov ax, [fs:FRAME_RM_TEMP + REGS_GS_OFFSET]
    mov [gs:si + PARAM_REGS_OUT_OFFSET + REGS_GS_OFFSET], ax
    mov ax, [fs:FRAME_RM_TEMP + REGS_SS_OFFSET]
    mov [gs:si + PARAM_REGS_OUT_OFFSET + REGS_SS_OFFSET], ax
    mov ax, [fs:FRAME_RM_FINAL_FLAGS]
    mov [gs:si + PARAM_REGS_OUT_OFFSET + REGS_FLAGS_OFFSET], ax

    mov esi, [fs:FRAME_FRAME_BASE32]

    lidt [fs:FRAME_PM32_IDTR]

    xor ax, ax
    mov ds, ax
    mov es, ax
    mov gs, ax
    mov fs, ax

    cli

    mov eax, cr0
    or eax, 1                   ; Re-enter protected mode
    mov cr0, eax

    
    jmp CODE32_SEL:(pm32_return - bioscall_thunk_entry + THUNK_PHYS_BASE)

BITS 64

DEFAULT REL

long_mode_return:
    ;xchg bx, bx
    mov r12, [frame_ptr64]

    mov ax, DATA64_SEL
    mov ds, ax
    mov es, ax
    mov fs, ax
    mov gs, ax
    mov ss, ax

    lgdt [r12 + FRAME_SAVED_GDT_DESCRIPTOR]
    lidt [r12 + FRAME_SAVED_IDTR]

    mov ax, [r12 + FRAME_SAVED_DS]
    mov ds, ax
    mov ax, [r12 + FRAME_SAVED_ES]
    mov es, ax
    mov ax, [r12 + FRAME_SAVED_FS]
    mov fs, ax
    mov ax, [r12 + FRAME_SAVED_GS]
    mov gs, ax
    mov ax, [r12 + FRAME_SAVED_SS]
    mov ss, ax

    mov rsp, [r12 + FRAME_SAVED_RSP]
    mov rbp, [r12 + FRAME_SAVED_RBP]

    push qword [r12 + FRAME_SAVED_RFLAGS]
    popfq

    movzx edx, word [r12 + FRAME_SAVED_CS]
    mov r10, [r12 + FRAME_SAVED_CR3]
    mov r11, [r12 + FRAME_SAVED_RETURN_RIP]

    mov rax, THUNK_SUCCESS

    pop r15
    pop r14
    pop r13
    pop r12

    mov cr3, r10

    pop rbx
    pop rbp

    add rsp, 8
    push rdx
    push r11
    retfq

section .data
align 8

frame_ptr64         dq 0

Using the driver:

Code: Select all

Task* bioscall_driver = spawn_from_multiboot_module("BIOS_DRV", 0, TaskPrivilegeLevel::Kernel);
    if (!bioscall_driver) {
        bprint("Failed to spawn BIOS call driver\n");
    } else {
        Device* bioscall_device = DRIVER_MANAGER.wait_for_device("bioscall");
        if (!bioscall_device) {
            bprint("BIOS call device not available\n");
        } else {
            bioscall_params params{};
            params.interrupt = 0x10;
            params.options = BIOSCALL_OPTION_DEFAULT_STACK;
            params.in.ax = 0x0013; // VGA mode 13h (320x200x256)
            drv_operation_status status =
                DRIVER_MANAGER.ioctl_device(bioscall_device, BIOSCALL_IOCTL_EXECUTE, &params, nullptr);
            if (status != DRV_OPERATION_OK) {
                bprint("Failed to set VGA mode via BIOS call driver (status=%u)\n",
                      static_cast<unsigned>(status));
            } else {
                bprint("Set VGA mode 0x13 using BIOS call driver\n");
            }
        }
    }

bigorenski · Post by **bigorenski** » Fri Sep 26, 2025 5:55 pm

Updated the code to plot a line using the mode that was just set.

Bochs:

QEmu:

Virtualbox:

VMware:

I have the logs but they're too long to post here.
I'm used to debugging on bochs and qemu. Usually when my code works on them, it works on the other two(vbox, vmw).
What should I look for in the logs?

bigorenski · Post by **bigorenski** » Fri Sep 26, 2025 9:40 pm

Solved.

Changed that 16:64 far pointer(droping to pmode) to a 16:32 far pointer.
For some reason it didn't bother bochs/qemu.

Now it works on every vm and on real hardware.

Octocontrabass · Post by **Octocontrabass** » Fri Sep 26, 2025 11:06 pm

You found an incompatibility between Intel and AMD CPUs.

bigorenski · Post by **bigorenski** » Fri Sep 26, 2025 11:41 pm

Octocontrabass wrote: ↑Fri Sep 26, 2025 11:06 pm You found an incompatibility between Intel and AMD CPUs.

Damn

So it wasn't my fault after all
By the way, do you happen to know where I can find a reliable list of those incompatibilities?

Octocontrabass · Post by **Octocontrabass** » Sat Sep 27, 2025 5:06 pm

I don't know of any complete lists. The one I linked earlier only covers documented instruction encoding differences. Some more differences are mentioned on the wiki.

OSDev.org

Dropping to real mode on VMWare/Virtualbox [solved]

Dropping to real mode on VMWare/Virtualbox [solved]

Re: Dropping to real mode on VMWare/Virtualbox

Re: Dropping to real mode on VMWare/Virtualbox

Re: Dropping to real mode on VMWare/Virtualbox [solved]

Re: Dropping to real mode on VMWare/Virtualbox [solved]

Re: Dropping to real mode on VMWare/Virtualbox [solved]