Page 1 of 2

Ring 0 multithreading

Posted: Thu Jan 08, 2026 1:15 am
by protegee6155
Hello everyone,

I’ll start with some context. Over the past two weeks, I’ve been working on implementing ring 0 multithreading. My goal is to understand how kernel-level multitasking works internally before moving on to user-mode multitasking (and user mode in general). Unfortunately, I’ve reached a point where I seem to have hit a wall in the implementation.

My current multithreading design works as follows:
1) A timer interrupt fires. As a result, the CPU automatically pushes EFLAGS, CS, and EIP onto the current kernel stack. In addition, my interrupt stub saves all general-purpose registers and the data segment register onto the stack.
2) Control is transferred to a general interrupt handler, which in turn calls the timer interrupt handler. This handler invokes the scheduler.
3) The scheduler saves the current context (which was pushed onto the stack in step 1) into the current TCB (thread control block). It then selects the next TCB and replaces the context saved on the stack with the new thread’s context.
4) Control returns from the general interrupt handler. The interrupt stub restores the general-purpose registers and the data segment register (which were updated in step 3), and finally executes iret to resume execution.

Code: Select all

isr_common_stub:
    pusha                    // Pushes edi,esi,ebp,esp,ebx,edx,ecx,eax

    mov ax, ds               // Lower 16-bits of eax = ds.
    push eax                 // save the data segment descriptor

    mov ax, 0x10  // load the kernel data segment descriptor
    mov ds, ax
    mov es, ax
    mov fs, ax
    mov gs, ax

    call isr_stub_handler

    pop ebx        // reload the original data segment descriptor
    mov ds, bx
    mov es, bx
    mov fs, bx
    mov gs, bx

    popa                     // Pops edi,esi,ebp...
    add esp, 8     // Cleans up the pushed error code and pushed ISR number
    sti
    iret           // pops 5 things at once: CS, EIP, EFLAGS

Code: Select all

typedef struct cpu_status_struct {
    uint32_t ds;
    // Pushed by pusha.
    uint32_t edi, esi, ebp, esp, ebx, edx, ecx, eax;
    uint32_t int_no, err_code;
    // Pushed by the processor automatically.
    uint32_t eip, cs, eflags;
} cpu_status_t;
After several scheduling cycles, the system eventually triggers a page fault at what appears to be a random address. I suspect that this is caused by an incorrect restoration or modification of registers during step 4, but I haven’t been able to pinpoint the exact issue.

Does anyone have an idea what could cause this behavior or what I should focus on debugging?

Source code:
https://github.com/sDos280/MyOS

Re: Ring 0 multithreading

Posted: Thu Jan 08, 2026 1:35 am
by protegee6155
It seems that I have identified two separate problems in my current implementation.
Problem 1: In my original ISR, I had the following instruction:

Code: Select all

add esp, 8     // Cleans up the pushed error code and pushed ISR number
This assumes that the interrupt handler always pushes an error code and an interrupt number. However, after a context switch, ESP now points to the new thread’s stack, not the original one. As a result, this instruction removes two values that do not correspond to an error code or interrupt number, which corrupts the new thread’s stack.

To address this, I plan to use a timer-specific interrupt stub that does not push an error code or interrupt number, eliminating the need for add esp, 8.

Code: Select all

.global isr32
isr32:
    cli
    jmp isr_timer_stub

...

isr_timer_stub:
    pusha                    // Pushes edi, esi, ebp, esp, ebx, edx, ecx, eax

    mov ax, ds
    push eax                 // Save data segment

    mov ax, 0x10             // Load kernel data segment
    mov ds, ax
    mov es, ax
    mov fs, ax
    mov gs, ax

    call timer_interrupt_handler

    pop ebx                  // Restore original data segment
    mov ds, bx
    mov es, bx
    mov fs, bx
    mov gs, bx

    popa
    sti
    iret                     // Pops EIP, CS, EFLAGS
Problem 2: Incorrect stack state when executing iret
After popa, the stack pointer refers to the next thread’s stack, which means that iret will pop EIP, CS, and EFLAGS from data that does not correspond to a valid interrupt frame. This causes execution to resume with invalid state.

The difficulty I see here is that the CPU pushes EIP, CS, and EFLAGS before the general-purpose registers are saved, so I cannot simply move or adjust those first three values after switching stacks.

At the moment, I do not see a clean way to fix this issue. Is there a correct approach to handling this situation when implementing ring 0–only context switching?

If anyone has suggestions or can point out what I’m missing conceptually, I would really appreciate the help.

Re: Ring 0 multithreading

Posted: Thu Jan 08, 2026 10:44 pm
by Octocontrabass
protegee6155 wrote: Thu Jan 08, 2026 1:15 amI suspect that this is caused by an incorrect restoration or modification of registers during step 4, but I haven’t been able to pinpoint the exact issue.
This issue specifically is because all of your tasks are using the same stack. All of your tasks are using the same stack because IRET doesn't switch stacks in ring 0. I don't know for sure, but I suspect IRET doesn't switch stacks in ring 0 because you aren't supposed to use interrupts for task switching.

Anyway, the correct approach you're looking for is to switch tasks using a function that switches stacks. You call this function any time you want to switch tasks. Since the return address is stored on the stack, the function doesn't return to the original caller until some other task decides it's time to switch back.

The wiki has a pretty good explanation here, although the example code is very different from how I'd do it.

Re: Ring 0 multithreading

Posted: Fri Jan 09, 2026 11:19 am
by bunny
I haven't gotten to this point in my OS yet so I don't have much specific advice but I wanted to plug a good resource from my own grad experience -- https://web.eecs.utk.edu/~jplank/plank/ ... cture.html there is a kthread library I can share directly if you want it. The source code used to be open source but I think some links to it may be dead so it might be unintentionally closed for now. If you want to see it I can try to reach out for you. But hopefully this example of kernel threads helps in some way!

Re: Ring 0 multithreading

Posted: Fri Jan 09, 2026 12:31 pm
by Octocontrabass
I'm pretty sure using setjmp/longjmp that way is undefined behavior. It also breaks the context switch into two pieces, which makes it a lot easier to accidentally modify the saved context after saving it.

Re: Ring 0 multithreading

Posted: Sat Jan 10, 2026 2:43 pm
by protegee6155
Octocontrabass wrote: Thu Jan 08, 2026 10:44 pm The wiki has a pretty good explanation here, although the example code is very different from how I'd do it.
thanks! :D
I’m still struggling to understand how switch_to_task eventually returns execution to the correct place. Specifically, how and when is the next EIP placed onto a thread’s stack? Should that happen right before calling switch_to_task?

Consider the following scenario with only two threads, A and B:
1. Thread A is currently running.
2. A timer interrupt occurs. The CPU automatically pushes EFLAGS, CS, and EIP (and later the general-purpose registers) onto A’s stack.
3. The scheduler runs and saves A’s context into its TCB.
4. The scheduler selects thread B to run next.
5. The scheduler sets up B’s stack (by pushing to it B's eip) and calls switch_to_task (still in A's interrupt-thread execution flow).
6. Thread B starts running.
7. Later, another timer interrupt occurs. The CPU pushes EFLAGS, CS, and EIP (and registers) onto B’s stack.
8. The scheduler runs again and saves B’s context into its TCB.

My question is: how does thread B cause thread A to resume execution at the exact point where A last stopped (inside A’s interrupt/return-to-thread execution flow)?

In other words, how does the system know where to return in thread A after thread A previously switched to thread B?
At step 3, we save A’s execution state at the moment the timer interrupt occurred, not at the instruction immediately following the switch_to_task call. So how does execution eventually resume at the correct location, rather than “after switch_to_task”?

Re: Ring 0 multithreading

Posted: Sat Jan 10, 2026 3:05 pm
by nullplan
The way I'd do it is to save the context not in the task structure, but on the stack. In the task structure, you only save the top of stack. Switching tasks is then just switching stacks. So basically:
  1. Task A runs.
  2. Timer interrupt occurs. CPU and interrupt handler push context on stack.
  3. Interrupt handler runs scheduler after acknowledging the interrupt.
  4. Scheduler picks the next task, calls task switcher
  5. Task switcher pushes non-volatile registers, stores ESP into task A's task structure, loads ESP with the stack top from task B's structure, and restores registers.
On the C level, execution just continues from the point the task switcher was called. If that was inside an interrupt, execution should come to an IRET instruction before long.

To initialize a new task, you now have to write a structure on top of the new task's stack that looks like what the task switcher writes there, but with the return pointer set to the function that starts your actual thread, so when the task switcher executes RET, it actually starts your function. Although it will be advantageous to be able to call functions with a pointer argument, so maybe another assembler wrapper is needed.

To start a userspace application, in addition to the kernel context, you also write whatever context your userspace return functions expect and execute those.

Re: Ring 0 multithreading

Posted: Sat Jan 10, 2026 7:01 pm
by Octocontrabass
protegee6155 wrote: Sat Jan 10, 2026 2:43 pmI’m still struggling to understand how switch_to_task eventually returns execution to the correct place. Specifically, how and when is the next EIP placed onto a thread’s stack? Should that happen right before calling switch_to_task?
The next EIP is pushed on the thread's stack when the thread calls switch_to_task, same as any other function call.
protegee6155 wrote: Sat Jan 10, 2026 2:43 pmConsider the following scenario with only two threads, A and B:
Keep in mind interrupts aren't necessary for task switching. You can (and in the future maybe will) have threads that choose to switch tasks without an interrupt.
protegee6155 wrote: Sat Jan 10, 2026 2:43 pmMy question is: how does thread B cause thread A to resume execution at the exact point where A last stopped (inside A’s interrupt/return-to-thread execution flow)?
By switching to thread A's stack. Since the return address was pushed onto thread A's stack when thread A called switch_to_task, thread A will resume exactly where it left off when switch_to_task returns.
protegee6155 wrote: Sat Jan 10, 2026 2:43 pmSo how does execution eventually resume at the correct location, rather than “after switch_to_task”?
But the correct location is always “after switch_to_task".

Re: Ring 0 multithreading

Posted: Sun Jan 11, 2026 2:50 am
by protegee6155
Octocontrabass wrote: Sat Jan 10, 2026 7:01 pm The next EIP is pushed on the thread's stack when the thread calls switch_to_task, same as any other function call.

By switching to thread A's stack. Since the return address was pushed onto thread A's stack when thread A called switch_to_task, thread A will resume exactly where it left off when switch_to_task returns.
ok, i think i undestand why things will work this way, thanks!
nullplan wrote: Sat Jan 10, 2026 3:05 pm
  1. Task switcher pushes non-volatile registers, stores ESP into task A's task structure, loads ESP with the stack top from task B's structure, and restores registers.
ok. if my switch_to_task also expectes arguments would i also need to push those in order onto the stack?
nullplan wrote: Sat Jan 10, 2026 3:05 pm To initialize a new task, you now have to write a structure on top of the new task's stack that looks like what the task switcher writes there ...
may you please elaborate more on this part? is this related to the thing i have wrote higher about switch_to_task expecting (stack passed) arguments?

Re: Ring 0 multithreading

Posted: Sun Jan 11, 2026 9:59 am
by nullplan
protegee6155 wrote: Sun Jan 11, 2026 2:50 am may you please elaborate more on this part? is this related to the thing i have wrote higher about switch_to_task expecting (stack passed) arguments?
Well, yes, if switch_to_task expects arguments, you have to provide them, and your chosen ABI passes them on stack, and lets the caller clean them up (you are using the SysV ABI, right?).

So, in C, we first define the things I said. struct task contains a stack top, and switch_to_task takes arguments where to put the current stack top and where the new one is:

Code: Select all

struct task {
  ...
  uint32_t *stack_top;
};
void lowlevel_switch_to_task(uint32_t **oldstack, uint32_t *newstack);
I like to write in assembler only the things that absolutely must be in assembler, which is why I'd split up the low-level switching of the stacks and higher-level stuff like switching CR3, setting the "current task" variable you probably have somewhere.

With these definitions, the scheduler can be done like this:

Code: Select all

void schedule(void) {
  struct task *old = current;
  struct task *next = find_next_task();
  /* questions to ponder in future: Can next == old? Can next == NULL? */
  /* switching the "current" variable here is also hairy, because the stack isn't switched yet. May need improvement later. */
  current = next;
  lowlevel_switch_to_task(&old->stack_top, next->stack_top);
}
Now the actual task switcher: According to the ABI, you have to save EBX, EBP, ESI, EDI, and ESP. We can do this like this (AT&T syntax because I am more familiar with it):

Code: Select all

lowlevel_switch_to_task:
  movl 4(%esp), %ecx ; oldstack
  movl 8(%esp), %edx ; newstack
  pushl %ebp
  pushl %ebx
  pushl %esi
  pushl %edi
  movl %esp, (%ecx)
  movl %edx, %esp
  popl %edi
  popl %esi
  popl %ebx
  popl %ebp
  retl
Now, how to construct a new task? Just construct a new stack top like lowlevel_switch_to_task expects. As I said, it will be advantageous to have arguments for your threads. Since the above function by itself is incapable of invoking a C function with an argument, we will just have to make a new function for that.

Code: Select all

extern const char start_thread[]; /* why declare it as a function when you can't call it? */
struct task *task_create(void (*entry)(void *), void *arg, size_t stack_size) {
  stack_size = (stack_size + 15) & -16; /* stack alignment is 16 bytes */
  char *stack = kalloc(stack_size + sizeof (struct task));
  struct task *new = (void *)(stack + stack_size);
  uint32_t *ctx = (uint32_t *)new - 5;
  ctx[0] = (uint32_t) entry; // edi
  ctx[1] = (uint32_t) arg; // esi
  ctx[2] = 0; // ebx
  ctx[3] = 0; // ebp
  ctx[4] = (uint32_t) start_thread; // return pointer
  new->stack_top = ctx;
  return new;
}

Code: Select all

start_thread:
  ; when we get here, EDI = entry pointer and ESI = argument and EBP = 0, so we don't need to clear it
  andl $-16, %esp
  subl $12, %esp
  pushl %esi
  calll *%edi
  call exit_thread
  ud2
For ABI reasons, exit_thread must be called (not jumped to), even though it does not return. I'm putting a ud2 fail-safe there: If exit_thread() does manage to return somehow, I don't want the code to start executing uncharted territory, and with this instruction, it will fault instead.

As for how exiting a thread should work: Because you are still executing the thread, you cannot free its memory (you'd free your own stack). So you have to mark the thread as dead, then switch to the next task. And have some kind of maintenance task that frees unused task stacks when memory runs low.

Re: Ring 0 multithreading

Posted: Mon Jan 12, 2026 8:30 am
by protegee6155
Thanks for the response! It seems that I almost got it 😄.
That said, I still have some trouble understanding how the stack should be set up.

In your lowlevel_switch_to_task function, it looks like arguments are passed using the stack. However, in your start_thread function, it seems that arguments are expected to be passed using a different calling convention (a register-based one). At the same time, in your create_thread function, it again appears that arguments are passed via the stack. Could you explain why there is this difference?

Another thing I don’t understand is how the stack is initialized when a thread is first created. From what I can see, the stack is set up like this:

Code: Select all

(higher stack addresses)
- address of start_thread
- 0 (ebp)
- 0 (ebx)
- arg   (edi)
- entry (esi)
(lower stack addresses)
When start_thread is first executed (which I assume what actually begins is threading itself and not a thread start call — though I may be mistaken), the following code runs:

Code: Select all

andl $-16, %esp    // align stack to 16 bytes
subl $12, %esp     // move the stack 12 bytes lower (*)
pushl %esi         // set up the argument pointer for the entry function
calll *%edi        // jump to the entry point
Could you please explain the purpose of the instruction marked with (*)?

Re: Ring 0 multithreading

Posted: Mon Jan 12, 2026 11:36 am
by nullplan
protegee6155 wrote: Mon Jan 12, 2026 8:30 am However, in your start_thread function, it seems that arguments are expected to be passed using a different calling convention (a register-based one). At the same time, in your create_thread function, it again appears that arguments are passed via the stack. Could you explain why there is this difference?
start_thread is not a function that is callable from C code. It only exists as a landing pad for a new thread to drop into when it gets "restored" the first time using lowlevel_switch_to_task(). The task switcher doesn't know (because it has no need to know) that the next thread is new, it only switches the stack and restores the registers. But for a new thread, the side effect is that the values written to the stack before are now in registers, and execution starts at start_thread with those values already in the registers.
protegee6155 wrote: Mon Jan 12, 2026 8:30 am Another thing I don’t understand is how the stack is initialized when a thread is first created. From what I can see, the stack is set up like this:
That is correct, and then the task switcher will pop four of those values into registers and perform a return, thus popping the address of start_thread into EIP.
protegee6155 wrote: Mon Jan 12, 2026 8:30 am which I assume what actually begins is threading itself and not a thread start call — though I may be mistaken
No, what begins there is the new kernel thread. The only job start_thread has is to adapt the state the task switcher left us in to something the C compiler can use.
protegee6155 wrote: Mon Jan 12, 2026 8:30 am Could you please explain the purpose of the instruction marked with (*)?
Stack alignment. I am a stickler for alignment rules (because forgetting about them can bite you in the most inopportune moments), and the current (well, 20 years old by now) version of the i386 ABI says the stack has 16 bytes alignment. Specifically, that is 16 bytes alignment before the call instruction (which itself pushes another word). In other words, C compilers expect the stack to be 4 bytes below a 16-byte boundary when a function starts. So in this case, I allow the caller to start the thread with an arbitrary stack alignment, then align it to a 16-byte boundary. Since I want to provide one argument, I must write one word to the stack. The aligning "and" instruction may not have moved ESP at all, so I cannot write there. So I must push three pseudo-args, which I do with the sub instruction. Then we can push the actual argument and the resulting ESP is 16 bytes aligned, before the call instruction.

Re: Ring 0 multithreading

Posted: Tue Jan 13, 2026 10:15 am
by protegee6155
Thanks for all the help and explanations! I think I’ve managed to implement ring-0 multithreading!

If we’re still here 🙃 — how does multithreading between privilege rings work?

Is the correct approach to “just” use iret, and prepare the stack with the values iret expects when switching to a different ring (for example: EIP, CS, EFLAGS, and, when changing privilege level, also userESP and SS)?

Re: Ring 0 multithreading

Posted: Tue Jan 13, 2026 10:57 am
by bellezzasolo
protegee6155 wrote: Tue Jan 13, 2026 10:15 am Thanks for all the help and explanations! I think I’ve managed to implement ring-0 multithreading!

If we’re still here 🙃 — how does multithreading between privilege rings work?

Is the correct approach to “just” use iret, and prepare the stack with the values iret expects when switching to a different ring (for example: EIP, CS, EFLAGS, and, when changing privilege level, also userESP and SS)?
Try thinking about what happens before a context switch in user mode.

Either a syscall, or an interrupt fires, triggering a transition from ring 3 to ring 0.

So... context switch between ring 0, and when it returns, it will return to ring 3!

Re: Ring 0 multithreading

Posted: Tue Jan 13, 2026 1:09 pm
by protegee6155
Hmm, okay, that somewhat makes sense.
So in that case, should my interrupt-handling code—the part that executes iret at the end—also serve as my context-switch function?