OSDev.org

Hi, I've been spending some time trying to decide how I wish for new processes to be created in my OS. Originally, I was trying to decide between a system like fork()/exec() and posix_spawn(). But instead of talking about those, i wanna talk about a new alternative I've come up with that might be able to solve the performance and complexity issues associated with fork()/exec() while being potentially equally powerful.

The idea is that you'd have a pair of system calls, let's call them create() and start(). create() would create an empty process that would inherit the file descriptors etc. of the parent process, just like fork(), but it would not copy the memory of the parent nor would it start executing, instead it would let the parent execute system calls as if it was the child, for example if the parent called dup2(), it would duplicate file descriptors in the child's file table instead of the parent's. start() would then load the child into memory and cause the child to start executing, like exec(), of course system calls would also return to their usual behavior.

Here is some pseudocode of what this might look like: In the end I'm not quite sure if this really is a good idea or not, perhaps there are some massive downsides I've missed, what do you people think?

Seems like you went to a whole lot of effort to reinvent clone(CLONE_VM|CLONE_VFORK). Basically, that is the way I am going to go. So still fork()+exec(), but the fork() replaced with something that does not copy memory and stops the parent thread. And that is also how most libcs implement posix_spawn() under Linux.

I basically combine CreateProcess with fork/exec. The application cannot see this, rather it's built into the kernel. There is one function to create a new process context in kernel space. The CreateProcess in kernel space then can use this function and combine it with creating a new thread in the process. The user mode equivalent of Windows CreateProcess then is implemented in the PE driver by using the CreateProcess syscall, and let the new kernel thread load the executable into memory and switch to user mode to start executing it. Fork is implemented in kernel mode by using create process function and copying the process space and setting writable pages to copy-on-write, and then creating a thread that returns back to user mode with a different CY flag compared to the caller. There is a bit more to it, but that is the basics.

I typically create new user mode processes with CreateProcess, but the command shell use fork/exec when a program is run without detaching it. This is due to CreateProcess creating a new console, while fork/exec use the original console.

While I was looking for information about what libcs BusyBox has been successfully linked against, I came across this entry in their FAQ about the vfork() system call. This might be of interest to you.

The gist of it is that fork()/exec() is very expensive if you don't have copy-on-write paging. vfork() will fork the process and put the parent to sleep. The child process continues executing in the same memory context as the parent until it calls exec() and a new one is created. The parent is not allowed to continue executing until the child calls exec(). I guess it's working for them, but you have to be very careful about what you do between fork() and exec().

I think what they're doing and what you're describing are almost the same. Their solution has the advantage that the same code will just fork()/exec() if it's compiled for an operating system that does have cheap fork()s. The advantage to yours is that it doesn't look like two threads executing in different processes.

lambduh wrote: ↑Tue Jun 18, 2024 8:27 amyou have to be very careful about what you do between fork() and exec().

vfork() is no longer in POSIX, and therefore I will not support it. However, back when it was, it was defined that the only thing the child was allowed to do was _exit() or exec(). No open() and close(), no pipe(), and definitely no setenv()! This is because parent and child are sharing a stack, so calling functions and expecting returns might clobber something.

clone() solves the issue by atomically moving the child to a new stack. That way, function calls are possible without hurting anything. This is why almost all Linux libcs implement posix_spawn() in terms of clone(CLONE_VM)

nullplan wrote: ↑Tue Jun 18, 2024 12:31 pm it was, it was defined that the only thing the child was allowed to do was _exit() or exec(). No open() and close(), no pipe(), and definitely no setenv()! This is because parent and child are sharing a stack, so calling functions and expecting returns might clobber something.

huh, in my imagination the child already had all the structs in the kernel so that it could modify its file descriptors. I guess I fundamentally did not understand how you're supposed to use vfork().

lambduh wrote: ↑Tue Jun 18, 2024 6:25 pm huh, in my imagination the child already had all the structs in the kernel so that it could modify its file descriptors. I guess I fundamentally did not understand how you're supposed to use vfork().

Oh, the kernel space is all set up. It's the user space that is broken by vfork(). That's also why things like open(), close(), and dup2() tended to work most of the time. And then it would break if someone changed the compiler settings too much.

lambduh wrote: ↑Tue Jun 18, 2024 6:25 pm

nullplan wrote: ↑Tue Jun 18, 2024 12:31 pm it was, it was defined that the only thing the child was allowed to do was _exit() or exec(). No open() and close(), no pipe(), and definitely no setenv()! This is because parent and child are sharing a stack, so calling functions and expecting returns might clobber something.

huh, in my imagination the child already had all the structs in the kernel so that it could modify its file descriptors. I guess I fundamentally did not understand how you're supposed to use vfork().

It's not just your imagination, this is what the authors of Unix intended. Some Plan 9 programs don't even check to see if they are the child process before making a bunch of function calls. fork() made sense in the original Unix, only one userspace program was in memory at once, the rest were swapped out. fork() would have changed the pid and some structures so the existing userspace would become a new program; it would be swapped to a different part of the disk. There would be no worries about clobbering the stack in such a design.

The restrictions @nullplan lists remind me of ucLinux; a port of Linux to mmu-less architectures. I'm pretty sure Linux had a fully functional fork() at the time. This was a long time ago, around the turn of the century, I may be mis-remembering.

eekee wrote: ↑Thu Jun 27, 2024 7:34 am It's not just your imagination, this is what the authors of Unix intended. Some Plan 9 programs don't even check to see if they are the child process before making a bunch of function calls. fork() made sense in the original Unix, only one userspace program was in memory at once, the rest were swapped out. fork() would have changed the pid and some structures so the existing userspace would become a new program; it would be swapped to a different part of the disk. There would be no worries about clobbering the stack in such a design.

Yeah, but that was fork(), not vfork(). In case of fork(), you have no problems, since child and parent are executing on different stacks, so they can do whatever, with the caveat that fork() children of multi-threaded programs must act as if they were in async-signal context, because some other thread may have held a lock.

This is changing, BTW; the next version of POSIX will add a function _Fork() that only does the forking thing, and still retain fork(), which will also call the pthread_atfork() handlers (so it is similar to exit() vs. _Exit()). musl has already taken the opportunity to redefine fork() to just take all implementation-internal locks before forking, so that the associated functionality is available in the child.

eekee wrote: ↑Thu Jun 27, 2024 7:34 am The restrictions @nullplan lists remind me of ucLinux; a port of Linux to mmu-less architectures. I'm pretty sure Linux had a fully functional fork() at the time. This was a long time ago, around the turn of the century, I may be mis-remembering.

Linux on architectures without MMU has traditionally not supported fork(), and would have that function fail with ENOSYS. So on those you only had vfork(). But these days you also have clone(), and I don't know how that works there. Possibly only with CLONE_VM.

Oh I see. So fork is still around and vfork is the optimized version where the child can do very little. Thanks.