What issue present in my code below which is not switching into user mode from kernel mode?

[gamingjam60]

I am building x86 architecture based 64 bit Operating System. Now My plan is to switch into user space(Lower Half Virtual Address) from kernel space(Higher Half Virtual address) by using below code which I have found from here:

Code: Select all

__attribute__((naked, noreturn))
void switch_to_user_mode_1(uint64_t stack_addr, uint64_t code_addr)
{
    asm volatile(" \
        push $0x23 \n\
        push %0 \n\
        push $0x202 \n\
        push $0x1B \n\
        push %1 \n\
        sti   \n\
        iretq \n\
        " :: "r"(stack_addr), "r"(code_addr));
}
```
I also add `sti` to start interrupt before iretq although iretq may store interrupt bit.

I have uheap_alloc function which allocate address at user space with user permission paging enabled in those locations.My System has interrupt system call 172 which print some message which can be called by `int 172`. I have checked it from kernel space it works.

To test and jump into a user space instruction I have written following assembly code:

Code: Select all

section .text

extern is_user_mode
extern print_dec

global user_stub 
global user_stub_end

user_stub:
    int 172
    call is_user_mode
    mov rdi, rax
    call print_dec

.hang:
    jmp .hang

user_stub_end: 

Now to switch into user space I am using

Code: Select all

uint64_t user_stub_size = ((uint64_t)&user_stub_end - (uint64_t)&user_stub);
uint64_t user_addr = (uint64_t) uheap_alloc(user_stub_size);    // Allocate a page in user space
if (!user_addr) return 0;
memcpy((void *)user_addr, (void *)&user_stub, user_stub_size);  // Copy function code

uint64_t stack_addr = ((uint64_t) uheap_alloc(STACK_SIZE)) + STACK_SIZE;

// I also made stack writable and non executable and user_addr readable and executable

switch_to_user_mode_1(stack_top_addr, code_addr);

I am expecting the interrupts(keyboard ,system call ) would work after switching into user mode but unfortunately after using `switch_to_user_mode_1` the system stalks without interrupt message.

Edit: GitHub



I do not understand what should I do to debug. I tried to debug by gdb to breaking the switch_to_user_mode_1 function after jumping into iretq showing some memory address with ? sign.

Someone said asm is better than c version of switch_to_user_mode

Code: Select all

section .text
global switch_to_user_mode  ; using switch_to_user_mode(uint64_t stack_addr, uint64_t code_addr) in c

switch_to_user_mode:
    cli                     ; Disable interrupts
    mov ax, 0x23            ; User-mode data segment (ring 3)
    mov ds, ax
    mov es, ax
    mov fs, ax
    mov gs, ax

    push 0x23               ; SS = user data segment
    push rdi                ; Stack address
    pushfq                  ; Push RFLAGS (Interrupts enabled) 0x202
    push 0x1B               ; CS = user code segment (ring 3)
    push rsi                ; Code address (Entry point)
    sti                     ; Enable interrupts

    iretq                  ; Return to user mode

The same problem present if I changed into `iretfq` from `iretq` the kernel mode present without switching into user mode!

[Octocontrabass]

using below code which I have found from here:

That code is broken. Only basic asm is allowed in naked functions.

I also add `sti` to start interrupt before iretq although iretq may store interrupt bit.

Remove it. IRETQ loads the interrupt flag from the stack.

Code: Select all

memcpy((void *)user_addr, (void *)&user_stub, user_stub_size);  // Copy function code

The CALL instruction encodes its operand relative to RIP. That means it won't work if you only copy the test function by itself without also copying the functions it calls. Either use absolute calls (like "mov rax, is_user_mode; call rax") or copy all of the code you want to use in user mode (assuming it's position-independent enough to work correctly at the new address).

I do not understand what should I do to debug. I tried to debug by gdb to breaking the switch_to_user_mode_1 function after jumping into iretq showing some memory address with ? sign.

Does the address correspond to the memory you allocated for user mode? If you disassemble the code at that address, does it look correct? If you run QEMU with "-d int", do you see any exceptions being logged?

[gamingjam60]

I made an user_programe.elf from

user_programe.asm

Code: Select all

; user_program.asm
[bits 32]
global user_main

section .text

user_main:
    mov eax, 0x12345678       ; Some dummy code
    int 172
hang:
    jmp hang                  ; Stay in user mode forever

user_linker_x86_64.ld

Code: Select all

ENTRY(user_main)

SECTIONS
{
    . = 0x400000;

    .text : {
        *(.text)
    }

    .rodata : {
        *(.rodata)
    }

    .data : {
        *(.data)
    }

    .bss : {
        *(.bss)
    }
}

The linker script said user_main function address is 0x400000

Now I am going to use this user_main function to switch into user space

Code: Select all

__attribute__((naked, noreturn))
void enter_user_mode(uint64_t rip, uint64_t rsp) {
    asm volatile (
        "cli\n"
        "mov $0x23, %%ax\n"
        "mov %%ax, %%ds\n"
        "mov %%ax, %%es\n"
        "mov %%ax, %%fs\n"
        "mov %%ax, %%gs\n"

        "pushq $0x23\n"       // user SS
        "pushq %[rsp]\n"      // user RSP
        "pushq 0x202\n"            
        "pushq $0x1B\n"       // user CS
        "pushq %[rip]\n"      // user RIP
        "iretq\n"
        :
        : [rip]"r"(rip), [rsp]"r"(rsp)
        : "memory"
    );
}

void init_user_mode(){
    uint64_t stack_base_addr = ((uint64_t) uheap_alloc(STACK_SIZE));
    for(uint64_t addr = stack_base_addr; addr < stack_base_addr + STACK_SIZE; addr += 0x1000){
        page_t *_page = get_page(addr, 0,  (pml4_t *)get_cr3_addr());
        _page->nx = 1;  // Making non-executable
        _page->rw = 1;  // Making it read-writable
    }
    
    uint64_t stack_top_addr = stack_base_addr + STACK_SIZE;   // Set stack top

    uint64_t code_addr = (uint64_t)&user_main;
    page_t *code_page = get_page(code_addr, 0, (pml4_t *)get_cr3_addr());
    code_page->rw = 0; // Making readable only
    code_page->nx = 0;  // Making executable
    
    flush_tlb_all();
    
    printf("Starting Switching to the user mode: code addr.- %x, stack addr.- %x\n", code_addr, stack_top_addr);
    enter_user_mode(0x400000, stack_top_addr);
}

Still stalking with output

Code: Select all

Starting Switching to the user mode: code addr.- 0xFFFFFFFF80012040, stack addr.- 0x5000

I was expecting code_addr's value would be 0x400000 but showing 0xFFFFFFFF80012040. What is happening in here?
I think I have some misconception in here. Can you help please.

[Octocontrabass]

I made an user_programe.elf

Why? It sounds like you're making changes at random instead of trying to understand the problem. Moving your user code to a separate binary isn't going to fix anything, but it will make things more difficult since now you need to load and parse that separate binary before you can run the user code.

[gamingjam60]

Yes my plan was to load elf file by limine bootloader and then parse the elf file But it not works. Even I come back into raw previous plan I changed into

Code: Select all

__attribute__((naked, noreturn))
void switch_to_user_mode(uint64_t stack_addr, uint64_t code_addr)
{
    asm volatile(
        "cli\n"                 // Disable Interrupt
        "mov $0x23, %%ax\n"     // User-mode data segment (ring 3)
        "mov %%ax, %%ds\n"
        "mov %%ax, %%es\n"
        "mov %%ax, %%fs\n"
        "mov %%ax, %%gs\n"
        "mov %%rsp, %%rax\n"    // Save Current stack pointer
        "pushq $0x23\n"         // Push USER_SS(DATA SEGMENT) Selector
        "pushq %[stack]\n"      // Push User Stack pointer
        "pushfq\n"              // Push RFLAGS
        "popq %%rax\n"          // Taking RFLAGS into RAX
        "orq $0x200, %%rax\n"   // Set IF flag to make automatic enabling Interrupt
        "pushq %%rax\n"         // Push Updated RFLAGS into the stack
        "pushq $0x1B\n"         // Push USER_CS(CODE SEGMENT)
        "pushq %[entry]\n"      // Push entry point address        
        "iretq\n"               // Interrupt Return to User Mode
        :
        : [stack] "r" (stack_addr), [entry] "r" (code_addr)
        : "rax"
    );
}


uint64_t create_user_function() {

    uint8_t *user_code = (uint8_t *) uheap_alloc(0x1000);

    // Example: simple infinite loop machine code
    uint8_t user_program[] = {
        0xeb, 0xfe  // Infinite loop: jmp $
    };

    memcpy(user_code, user_program, sizeof(user_program));

    return (uint64_t)user_code; // Return the user-accessible function pointer
}


void init_user_mode(){

    uint64_t code_addr = (uint64_t) create_user_function();
    page_t *code_page = get_page(code_addr, 0, (pml4_t *)get_cr3_addr());
    code_page->rw = 0; // Making readable only
    code_page->nx = 0;  // Making executable

    uint64_t stack_base_addr = ((uint64_t) uheap_alloc(STACK_SIZE));
    for(uint64_t addr = stack_base_addr; addr < stack_base_addr + STACK_SIZE; addr += 0x1000){
        page_t *_page = get_page(addr, 0,  (pml4_t *)get_cr3_addr());
        _page->nx = 1;  // Making non-executable
        _page->rw = 1;  // Making it read-writable
    }
    uint64_t stack_top_addr = stack_base_addr + STACK_SIZE;   // Set stack top
    
    flush_tlb_all();
    
    printf("Starting Switching to the user mode: code addr.- %x, stack addr.- %x\n", code_addr, stack_top_addr);
    switch_to_user_mode(stack_top_addr, code_addr);
}

Output:

Code: Select all

Starting Switching to the user mode: code addr.- 0x1000, stack addr.- 0x7000

But still it not working i.e. stalks the system and no interrupt working although using 0x202!

[gamingjam60]

Now showing following output :

Code: Select all

GDT Entry 0x0: Base=0x0 Limit=0x0 Access=0x0 Flags=0x0
GDT Entry 0x8: Base=0x0 Limit=0xFFFF Access=0x9A Flags=0xA
GDT Entry 0x10: Base=0x0 Limit=0xFFFF Access=0x93 Flags=0xA
GDT Entry 0x1B: Base=0x0 Limit=0xFFFF Access=0xFA Flags=0xA
GDT Entry 0x23: Base=0x0 Limit=0xFFFF Access=0xF2 Flags=0xA
GDT Entry 0x28: Base=0x8059A060 Limit=0x67 Access=0x8B Flags=0x0
Current stack address: 0xFFFF80007FF41FC0
Current rip address: 0xFFFFFFFF8000EF22
Current rflags address: 0x286
&user_programe: 0xFFFF80007FF41F86
user_code or user_programe is null
Starting Switching to the user mode: code addr.- 0x1000, stack addr.- 0x7000
General protection fault (pushes an error code)
recieved interrupt: 13
Error Code: 0x0
CS: 0x8, RIP : 0xFFFFFFFF8000EA61
Stack (rsp = 0xFFFF80007FF41F70) Contents(First 26) :
  [0xFFFF80007FF41F70] = 0x1000
  [0xFFFF80007FF41F78] = 0x1B
  [0xFFFF80007FF41F80] = 0x292
  [0xFFFF80007FF41F88] = 0x7000
  [0xFFFF80007FF41F90] = 0x23
  [0xFFFF80007FF41F98] = 0xFFFFFFFF8000EBE5
  [0xFFFF80007FF41FA0] = 0x7FF28030
  [0xFFFF80007FF41FA8] = 0x7000
  [0xFFFF80007FF41FB0] = 0x3000
  [0xFFFF80007FF41FB8] = 0x7FF28008
  [0xFFFF80007FF41FC0] = 0x1000
  [0xFFFF80007FF41FC8] = 0x7000
  [0xFFFF80007FF41FD0] = 0xFFFF80007FF41FF0
  [0xFFFF80007FF41FD8] = 0xFFFFFFFF800080A1
  [0xFFFF80007FF41FE0] = 0x0
  [0xFFFF80007FF41FE8] = 0xFEBD5000
  [0xFFFF80007FF41FF0] = 0x0
  [0xFFFF80007FF41FF8] = 0x0
  [0xFFFF80007FF42000] = 0x800000015CD00037
  [0xFFFF80007FF42008] = 0xFFFF
  [0xFFFF80007FF42010] = 0x0
  [0xFFFF80007FF42018] = 0x0
  [0xFFFF80007FF42020] = 0x0
  [0xFFFF80007FF42028] = 0x0
  [0xFFFF80007FF42030] = 0x0
  [0xFFFF80007FF42038] = 0x0
System Halted!

Here the stack contents show

+---------------+
| RIP | 0x1000 |
+------+--------+
| CS | 0x1B |
+------+--------+
| RFLAGS | 0x292 |
+------+--------+
| RSP | 0x7000 |
+------+--------+
| SS | 0x23 |
+------+--------+
| .... | .... |

But iret get interrupted so the current stack never changed but it is holding correct contents which are pushed by me by `switch_to_user_mode` function.

What is problem in here?

[Octocontrabass]

Code: Select all

__attribute__((naked, noreturn))

You still can't use extended asm inside a naked function. Only basic asm is allowed.

Code: Select all

        "pushfq\n"              // Push RFLAGS
        "popq %%rax\n"          // Taking RFLAGS into RAX
        "orq $0x200, %%rax\n"   // Set IF flag to make automatic enabling Interrupt
        "pushq %%rax\n"         // Push Updated RFLAGS into the stack

Why did you change this part? The previous code was fine. Stop changing things at random.

But still it not working i.e. stalks the system and no interrupt working although using 0x202!

Did you use your debugger to see what the CPU is doing when it gets stuck?

Now showing following output :

What did you change? Also, QEMU's interrupt log ("-d int") would be helpful.

[gamingjam60]

I came back into pure assembly

Code: Select all

switch_to_user_mode:
    cli
    mov ax, 0x23
    mov ds, ax
    mov es, ax
    mov fs, ax 
    mov gs, ax 

    push 0x23
    push rdi

    pushfq 
    pop rax
    or rax, 0x200
    push rax
    
    push 0x1B
    push rsi
    ; sti
    iretq
    

I use

Code: Select all

 switch_to_user_mode(0x1000, 0x7000)

to switch into user space code entry

Code: Select all

 
 uint8_t user_program[] = {
        0x31, 0xc9,                     // xor    %ecx,%ecx
        0xf7, 0xf1,                     // div    %ecx
        0xb8, 0x78, 0x56, 0x34, 0x12,   // mov $0x12345678,%eax
        0xcd, 0xac,                     // int $0xac
        0xeb, 0xfe                      // jmp 0x9
    };
  

Here I add division by zero at top but CPU is not display any interrupt message(It is functional at kernel space) also cpu not any other interrupt.I am setting IF(bit 9) by or with current flags 0x202 even I checked put sti just beforee iretq but still cpu not receive interrupt.

The debugger GDB is showing below after executing iretq

Code: Select all

0x1000 xor %ecx, %ecx
0x1002 div %ecx
0x1004 mov $0x1234567, %eax
0x1009 int $0xac
0x100b jmp 0x100b
0x100d add %al, (%rax)
0x100f add %al, (%rax)
0x1011 add %al, (%rax)
....

[gamingjam60]

Thankfully I have found the problem by suggestions made by Micheal Petch at stackoverflow
My TSS structure had problem as I added (align 16) which was making extra padding in that structure by gcc but changing into (packed) solves all problems. Now interrupt is working in user mode.