Not about context switching

[Agola]

Hi,

I've asked a lot of question about context switching, but this is not about context switching. I'm having a conceptual problem about multitasking in my os.

I've *finally* finished context switching, and it works really good. This is the stack setup of tasks:

[args]
[eip (address of task)]
[eip (address of start_task)]
[eax]
[ebx]
[ecx]
[edx]
[ebp]
[esi]
[edi]

But what should "start_task" do? My implementation allocates the CPL=3 stack (if task->type isn't TASK_TYPE_KERNEL) of task, sets the IOPL and VM (if task->type is TASK_TYPE_V8086) and does an iret. (again if task->type isn't TASK_TYPE_KERNEL)

Also I noticed some problems:

start_task doesn't return to its caller, so return eip isn't in the stack as C expects. This means values of arguments in void start_task(uint32_t address_of_task, uint32_t args) will be wrong as address_of_task will be considered as return eip, address_of_task will point to args and args will point args + 4 in stack.

I fixed that rewriting start_task in asm, but same problem applies to tasks again.

What happens if I do that in a user program?

Code: Select all

void task1 (void* args)
{
    printf("Hello from task1! arg is : %X\n", (uint32_t)*args);
    exit();
}

int main(int argc, char** argv)
{
    printf("Hello from main, now I will create a task!\n");

    uint32_t arg = 0xDEADBEEF;
    pid_t pid = add_task(task1, (void*)&arg, PRIORITY_LOW, "task1"); //My os doesn't have fork, instead it uses its own task creating system
    
    return 0;
}

Again task1 expects eip on the stack and args points to args + 4.

If I load an elf file directly, there is no problem as _start is written in assembly. Problem occurs when I add a C function directly(without any assembly entry) as a task.

For example I have a "task monitor" task runs in kernel that autonomously manages tasks. In kernel_idle I create it with:

Code: Select all

void task_monitor(void* args)
{
     for (;;)
     {
         ...
     }
}

void kernel_idle()
{
    ...
    kadd_task(create_task_struct(task_monitor, TASK_TYPE_KERNEL, args), PRIORITY_MID, "Task Monitor");
    ...
}

Again args points to wrong address on stack because of there is no return eip on the stack. There isn't a stack frame on the stack also.

So tasks have that in the stack when they start running (after start_task):

[args]

But they should have these in the stack if I start a C function directly as a task:

[args]
[return eip]

or maybe:

[args]
[return eip]
[ebp (for stack frame)]

What should I do?

Thanks in advance.

[simeonz]

I wish I or someone else on the forum could help, but the question is a little unclear it seems. I understand the general context, but the specific question is a little fuzzy.

Let's think about it this way. The cpu execution is like a linked list. (I hope you are aware of this structure.) task0 runs until some event, such as interrupt, and some code switches the stack with mov into esp, does popa, pop, ret, whatever, and ends up in a different execution context. Say, the context of task1. task1 also runs until some event, and then the switch is performed again, and you end up into task2. Those mov, pop, ret, iret, are only facilities to overwrite the registers, perform far jumps, whatever is needed. They have nothing special about them. Take "ret" for example. This is not instruction for returning from subroutine. This is instruction that reads from the stack location, increments the stack pointer and jumps to the read address. You could use it for counting daisies if you wanted, if it suited you. All those instructions do is enable control flow and state transfers and thus form a rather complex execution chain for the cpu to run. Usually this execution chain mimics a data structure that the scheduler maintains. Lets say it is a linked list for simplicity and symmetry. So, the execution chain formed by switching the cpu context on designated events must mimic the node chain in the scheduler's linked list. Which means that creating a new task is simply a matter of filling the node data for a new element in this linked list with the starting context for that task, inserting said node into any position (such as the last, first, whatever), and letting the scheduler naturally reach that node after a number of events occur. Then, this new execution context will become part of the execution chain that the cpu alternates. You could technically kick the scheduler eagerly to switch into the just added element, but those are fine points and details that are not relevant for the proper operation.

I know I am answering too broadly, but I fail to decipher the exact source of your dilemma. You seem to have this fixed understanding about how every "ret" instruction must be associated with a call made by a "caller", which is not at all true. At least for OS development, such semantics are irrelevant. A ret instruction is simply a jump to address specified on the stack. The stack is simply memory pointed to by esp.

[Brendan]

Hi,

If I load an elf file directly, there is no problem as _start is written in assembly. Problem occurs when I add a C function directly(without any assembly entry) as a task.

Let's assume that:

• The kernel has no way of knowing which language any piece of user-space code was written in and no way of knowing what calling conventions/ABI it happens to use; and should never have any reason to care about these things.
• Every language has some kind of run-time support; which might be in the form of a library, or might be a virtual machine or interpreter.
• By necessity; a language's run-time support provides some abstractions (e.g. you call "write()" in a C standard library using one of the different calling conventions for C, and inside that library it does SYSCALL with an entirely different calling convention to ask kernel to write some data)
• The abstractions provided by a language's run-time always involve some assembly (e.g. it's impossible to generate "int 0x80" or "sysenter" or "syscall" instructions in plain C, so assembly is needed even if the calling conventions are forced to match somehow).

Now assume you have some sort of library for some sort of language, and that library has an assembly routine that:

• asks the kernel to start a task/thread, and deals with any errors that might be reported by the kernel in a language specific way (errno? throw an exception?)

Then (in the startup assembly for that task that the kernel started):

• sets up any language specific stuff (thread-local storage, signal handlers, ...) if the language has any of that
• calls a function pointer (one of the input args)
• if the called function returns:
  • Do any language specific task termination stuff ("atexit()", "pthread_join()", ...)
  • ask the kernel to terminate the task/thread

What I'm trying to say here is that the only problem I can see is that you've assumed it makes sense for the kernel to directly start a C function as a task.


Cheers,

Brendan

[Gigasoft]

I don't really see how this is a problem. If start_task doesn't return, then you just stick your favourite value in place of the return address.

As a possible source of inspiration, this is what happens in my OS:
- In general, the stack is set up so that execution begins at the requested function's address, the return address points to the ExitThread function (which, as you guessed it, exits the thread and causes its stack to be freed), and the requested number of parameters are copied onto the stack.
- For user mode threads, execution begins at a stub which allocates an user mode stack, creates a handle to the thread, sets up the user mode stack so that the user mode function will return to a ReturnFromUserMode function, and the requested number of parameters are copied onto the user mode stack. There is a CallUserMode function which causes a requested function to be called in user mode. After the user mode function returns, the user mode stack is freed and the handle is closed. The CallUserMode / ReturnFromUserMode mechanism allows multiple nested user mode calls.

[simeonz]

The previous posters have understood your question better. In light of what they said, I want to summarize and clarify a bit. As was already noted, your task1 function never returns. It has to call the "exit" system call instead, which is a non-returning system call that destroys the thread and delivers return code to the thread creator (optional). A kernel task would not perform the exit system call per se, but would directly mark itself as "dead", which would make it reclaimable by the kernel at later point (probably when the scheduler notices.) For example:

Code: Select all

void __attribute__((__noreturn__)) user_exit(int err)
{
  syscall_1(__NR_exit, err); //instead of return
}
void user_task1 (void* args)
{
  int err = do_fn_work();
  user_exit(err);
}
void __attribute__((__noreturn__)) kernel_exit(int err)
{
  struct task *t = get_current_task();
  
  t->code = err;
  t->state = TASK_STATE_DEAD;
  schedule(); //this deschedules the task, and would never return;
}
void kernel_task1 (void* args)
{
  int err = do_fn_work();
  kernel_exit(err);
}

In the kernel, the exit system call may do a billion things, but it would ultimately perform kernel_exit as well. The "__attribute__((__noreturn__))" tells the compiler that the function will never return to its caller, which basically means that the compiler knows not to perform "ret" in neither the function, nor its callers if this is the last function they call. The return address on the stack below user_task1 and kernel_task1 would be therefore irrelevant, if it wasn't for stack unwinding. Besides being used for printing stack traces and collecting profiling stack trace data, unwinding is used during exception handling if you are using C++. The unwinder will stop at null return address, so this is what you should put at the bottom of the stack.

As Brendan already noted, usually the thread function in userland is actually the thread routine of the thread library, which in turn calls the user specified routine after some initialization, and calls exit at the end. For this reason, the user specified routine can simply perform return with the thread return value, instead of calling exit.

[zesterer]

I don't know if this is what you're looking for, but here's my 2 cents:

In terms of kernel threads, my method is quite simple. When the scheduler encounters a thread in the "NEW" state (i.e: yet to be first run) it simply begins executing a piece of code that calls a function pointer (along with a few other things like setting up argc and argv as arguments to the thread function). After the thread function is finished, this piece of code then executes a function called thread_finish. This will mark the currently executing thread as being in the "DEAD" state, and then wait in an infinite loop for the next timer interrupt (later, I'll make it call a "YIELD" syscall instead rather than waiting). When the scheduler reaches it, it'll free the thread's stack memory and clean up anything else the thread was responsible for, and then remove it from the scheduler queue.

Here is the code I have that does that: https://github.com/zesterer/tupai/blob/ ... r.cpp#L143

Note: This doesn't apply to user-mode threading. That is more complex and will require a full-on context switch. However, the principle is similar.

Note 2: I don't yet have per-thread virtual memory. Adding it would not significantly change this code, other than the addition of a virtual memory switch when the scheduler decides to switch to a new thread.

[Agola]

Thanks everyone, and sorry because I couldn't explain my "question" actually.

Simply, I wanted to create tasks with pthread way in user-space (having it in kernel-space would good) :

Code: Select all

uint8_t data[0xDE, 0xAD, 0xBE, 0xEF];

void *thread(void *ptr)
{
     return ptr;
}

int main(int argc, char **argv)
{
    pthread_t thread1;

    pthread_create(&thread1, NULL, *thread, data);
    pthread_join(thread1,NULL);

    return 0;
}

But C functions expect return eip in the stack that they don't have as they won't return, so I can't access args, "args" points to a wrong address.

At least my multitasking implementation works good, but still there are many things I have to learn about osdeving.

Thanks in advance.