Yielding during a syscall/interrupt nesting

[max]

Hey guys,

I've reworked a lot of stuff in my kernel to ensure that higher-priority interrupts can occur while an interrupt is being handled. This works fine so far. But I'm having two cases for which I'm not sure why they are not working:

a)
* A process is doing some work and uses interrupt 0x80 to enter my syscall handler
* Context switch to kernel context, it processes that syscall
* The kernel now tries to yield itself by triggering another interrupt with "int 0x80" (for example, the processes tried to "read" but the device isn't ready and blocking, so I want another thread to do some work)

b)
* A process registers an IRQ handler, say for #1 to catch keyboard interrupts
* When I press a key, my interrupt handler is entered and I call the user-space function to handle the interrupt directly
* Now that interrupt handler might want to use a syscall; simply doing so with "int 0x80" to trigger it

For both cases, I get a general protection fault when my interrupt handler tries to set the FS/ES segments while continuing. I have a bit of trouble understanding why this would not work, why can I not trigger an interrupt while I'm in ring 0? For example the timer interrupt works fine, it interrupts my interrupt handler and after that, work is continued..

What would be the right approach to yield during a syscall? Not with another interrupt?

Thanks in advance!!

[Octocontrabass]

* The kernel now tries to yield itself by triggering another interrupt with "int 0x80" (for example, the processes tried to "read" but the device isn't ready and blocking, so I want another thread to do some work)

Why does yielding require an interrupt? Your scheduler probably should be capable of switching to another kernel stack through an ordinary function call, since the function's return address is itself stored on the stack. (I'm assuming you have one kernel stack per thread, since you didn't specify otherwise.)

* Now that interrupt handler might want to use a syscall; simply doing so with "int 0x80" to trigger it

Why can't it use an ordinary function call? Also, be careful how you call it, since you probably want to avoid doing things like yielding before EOI.

For both cases, I get a general protection fault when my interrupt handler tries to set the FS/ES segments while continuing.

This sounds like an unrelated bug. Nested interrupts should work as long as you have enough stack space.

What would be the right approach to yield during a syscall? Not with another interrupt?

Probably not with another interrupt. Usually it's an ordinary function call to your scheduler, since kernels usually have one kernel stack per thread. If you have a different number of kernel stacks, you might need something else.