CPU restarts at Ring3 syscalls - for Ring0 works fine

Question about which tools to use, bugs, the best way to implement a function, etc should go here. Don't forget to see if your question is answered in the wiki first! When in doubt post here.
Post Reply
User avatar
Robert
Member
Member
Posts: 25
Joined: Wed Jan 13, 2021 8:49 am

CPU restarts at Ring3 syscalls - for Ring0 works fine

Post by Robert »

Hi!
I'm developing my own OS. Successfully started a task in user mode (CPL 3) but when I called a dummy system call it always restarted the CPU. The facts:

architecture: x86, simple 80386 instruction set I use. Test and debug in qemu.
the problem happens in protected mode, fully set GDT and IDT, no paging. Right before the syscall CPU is in a totally consistent state.
dummy syscall means an interrupt gate with DPL=3, resides in kernel (0x8 CS selector), type=0xee, executes cli hlt. Worked fine if I called it from kernel.
User space code is written in C++,with inline asm. Works fine without syscalls.
Eflags' IF is set.
Stacks are OK (both SS and ESP)
IDT vectors look like this: 0x0008090c 0x0010ee00 (DPL=3, S=0, P=1, type=e, selector=8, offset=0x10090c. All offsets are correct.
The restart happens at the 'int 0x30' instruction.
Do you have any idea, what I've missed?

Thanks in advance,
Robert
Octocontrabass
Member
Member
Posts: 5568
Joined: Mon Mar 25, 2013 7:01 pm

Re: CPU restarts at Ring3 syscalls - for Ring0 works fine

Post by Octocontrabass »

Robert wrote:Do you have any idea, what I've missed?
Did you set up exception handlers? They can help you troubleshoot issues like this.

Did you set up a TSS?
vvaltchev
Member
Member
Posts: 274
Joined: Fri May 11, 2018 6:51 am

Re: CPU restarts at Ring3 syscalls - for Ring0 works fine

Post by vvaltchev »

No sure what happened, but CPU restarts in case of a Triple_Fault.
Briefly, it means that a fault occurred while trying to execute an interrupt handler, a software exception triggered by your user code in this case, and that there was no double-fault handler or if a fault occurred while trying to execute the double fault handler itself.

One of the reasons for a fault to occur when the CPU is trying to execute an exception handler is being unable to write to the stack.
That's why Octocontrabass asked if you did setup a TSS. The CPU needs to know which ESP and SS to use in case an
interrupt occurs.
Tilck, a Tiny Linux-Compatible Kernel: https://github.com/vvaltchev/tilck
User avatar
Robert
Member
Member
Posts: 25
Joined: Wed Jan 13, 2021 8:49 am

Re: CPU restarts at Ring3 syscalls - for Ring0 works fine

Post by Robert »

Octocontrabass wrote:
Robert wrote:Do you have any idea, what I've missed?
Did you set up exception handlers? They can help you troubleshoot issues like this.

Did you set up a TSS?
TSS was the solution. Although I use sw task switching,kernel needed a TSS in GDT. Loaded by
LTR then everything worked fine.
Thanks for the help.

Problem solved, this topic became pointless.
Should I close it somehow?
Post Reply