Page 1 of 1

CPU restarts at Ring3 syscalls - for Ring0 works fine

Posted: Wed Jan 13, 2021 9:06 am
by Robert
Hi!
I'm developing my own OS. Successfully started a task in user mode (CPL 3) but when I called a dummy system call it always restarted the CPU. The facts:

architecture: x86, simple 80386 instruction set I use. Test and debug in qemu.
the problem happens in protected mode, fully set GDT and IDT, no paging. Right before the syscall CPU is in a totally consistent state.
dummy syscall means an interrupt gate with DPL=3, resides in kernel (0x8 CS selector), type=0xee, executes cli hlt. Worked fine if I called it from kernel.
User space code is written in C++,with inline asm. Works fine without syscalls.
Eflags' IF is set.
Stacks are OK (both SS and ESP)
IDT vectors look like this: 0x0008090c 0x0010ee00 (DPL=3, S=0, P=1, type=e, selector=8, offset=0x10090c. All offsets are correct.
The restart happens at the 'int 0x30' instruction.
Do you have any idea, what I've missed?

Thanks in advance,
Robert

Re: CPU restarts at Ring3 syscalls - for Ring0 works fine

Posted: Wed Jan 13, 2021 5:53 pm
by Octocontrabass
Robert wrote:Do you have any idea, what I've missed?
Did you set up exception handlers? They can help you troubleshoot issues like this.

Did you set up a TSS?

Re: CPU restarts at Ring3 syscalls - for Ring0 works fine

Posted: Wed Jan 13, 2021 8:52 pm
by vvaltchev
No sure what happened, but CPU restarts in case of a Triple_Fault.
Briefly, it means that a fault occurred while trying to execute an interrupt handler, a software exception triggered by your user code in this case, and that there was no double-fault handler or if a fault occurred while trying to execute the double fault handler itself.

One of the reasons for a fault to occur when the CPU is trying to execute an exception handler is being unable to write to the stack.
That's why Octocontrabass asked if you did setup a TSS. The CPU needs to know which ESP and SS to use in case an
interrupt occurs.

Re: CPU restarts at Ring3 syscalls - for Ring0 works fine

Posted: Thu Jan 14, 2021 8:19 am
by Robert
Octocontrabass wrote:
Robert wrote:Do you have any idea, what I've missed?
Did you set up exception handlers? They can help you troubleshoot issues like this.

Did you set up a TSS?
TSS was the solution. Although I use sw task switching,kernel needed a TSS in GDT. Loaded by
LTR then everything worked fine.
Thanks for the help.

Problem solved, this topic became pointless.
Should I close it somehow?