Loading user space programs in virtual memory

[eino]

This is something I actually asked from NickJohnsson but I thought I should post it here ase well incase someone else ever wonders about the same thing.

Kernel creates a new process and a new virtual memory space for it. Is it the executable actually loaded to that virtual space or is it loaded somewhere else?

Well the answer maybe obvious, but I ask cause there's something I can't get my head around... Also are there any other approaches?

[Nessphoro]

Well of course, where else are you going to load it? The x86 architecture can only execute from RAM ( Besides that freaky bios init stuff). You could, I guess, not load it at all and use a sort of lazy-loading instead - loading the executable as it runs. Which might make launching apps instantaneous, but it will require more time later to load the additional stuff.

[bluemoon]

This is something I actually asked from NickJohnsson but I thought I should post it here ase well incase someone else ever wonders about the same thing.
Kernel creates a new process and a new virtual memory space for it. Is it the executable actually loaded to that virtual space or is it loaded somewhere else?
Well the answer maybe obvious, but I ask cause there's something I can't get my head around... Also are there any other approaches?

Sure. Instead of load the executable into memory, you may map the executable from cache (i.e. if the executable/dso already loaded recently).

[egos]

You could, I guess, not load it at all and use a sort of lazy-loading instead - loading the executable as it runs. Which might make launching apps instantaneous, but it will require more time later to load the additional stuff.

Yes. Just reserve memory (not necessarily RAM; RAM is necessary only for work set) for actual sections of executable module and link them with corresponding regions of address space. Make loading each page separately on demand.

[Brendan]

Hi,

Kernel creates a new process and a new virtual memory space for it. Is it the executable actually loaded to that virtual space or is it loaded somewhere else?

Typically the kernel creates a new "empty" (e.g. only containing the kernel itself) virtual address space, creates any data structures needed to track the process (e.g. "process information block" or something) and creates an initial thread for that process (which might begin running in the kernel, as there's nothing in user-space).

The next piece of code (running as a thread in the new process) might load *something* into that virtual address space. The "something" might be the executable file itself, or it might be some sort of executable file loader - e.g. something that parses the real executable file's headers, sets up (.text, .rodata, .data, .bss) sections to suit, does any dynamic linking, etc.

The "something" might even be a piece of software that determines what type of executable the real executable file is, and then launches something else to execute that file. For example, you might have a piece of software that looks at the executable file and sees if it's a script (then launches something like BASH to execute that script) or sees if it's an ELF file (and launches an ELF loader) or sees if it's Java byte-code (and launches a Java VM), or whatever else. This allows the kernel to execute a file without caring what the file actually is, and allows you to add support for completely different types of executable files without touching the kernel's code.

Of course this might even be recursive. For example, "something" might look at the file and realise it's a JPG picture and not an executable at all; and then figure out which executable to use for JPG pictures and ask the kernel to start that executable (and the kernel might start a second process to start the executable and open the JPG picture; and that second process might realise the "executable" is actually a WAV file and start a third process to play the WAV file; and the third process (a media player?) might play the WAV file which could be a recording of someone saying "JPG not supported!").


Cheers,

Brendan

[eino]

Brendan. That was an amazing answer!

[bewing]

You have to be a little careful, though. Brendan's *something* is an excellent idea, but it is also a perfect target for a malware trojan.

[JamesM]

<moved to correct subforum>

[linguofreak]

Kernel creates a new process and a new virtual memory space for it. Is it the executable actually loaded to that virtual space or is it loaded somewhere else?

On modern Unix-like OS's, the kernel actually doesn't create a new address space for a new process. It just duplicates the address space of the process that made the fork() call (fork() is the "create-new-process" call for Unix-likes), and marks everything "copy on write" in both processes. After the new process is created, both the new and the old process are running the same program from the same memory. Each process then takes different actions according to whether it is the old or new process. It is at this point that the new process usually calls exec(), which loads a new program into its address space.

The entire process looks like this:

Code: Select all

Program A ----> fork() -----+---->Program A                                                           Old Process
                            |
                            +---->Program A---> exec(Program B)---> Program B                         New Process

Note that while a new process is normally started like this, fork() and exec() can be used independently (for example, a program that just needs to start a new instance of itself only needs fork(), and a program that is done with its own work, but just needs to start up another program to do some clean-up work can just call exec() and start the new program without creating a new process).

[Owen]

Kernel creates a new process and a new virtual memory space for it. Is it the executable actually loaded to that virtual space or is it loaded somewhere else?

On modern Unix-like OS's, the kernel actually doesn't create a new address space for a new process. It just duplicates the address space of the process that made the fork() call (fork() is the "create-new-process" call for Unix-likes), and marks everything "copy on write" in both processes. After the new process is created, both the new and the old process are running the same program from the same memory. Each process then takes different actions according to whether it is the old or new process. It is at this point that the new process usually calls exec(), which loads a new program into its address space.

posix_spawn

[linguofreak]

posix_spawn

Yes, posix_spawn() exists, but, as stated in your link:

The posix_spawn() function is implementable as a library routine, but both posix_spawn() and posix_spawnp() are designed as kernel operations. Also, although they may be an efficient replacement for many fork()/ exec pairs, their goal is to provide useful process creation primitives for systems that have difficulty with fork(), not to provide drop-in replacements for fork()/ exec.

So, A) while POSIX does have a means of creating a process and a new address space for it all at once, it still isn't the primary method of spawning new processes under POSIX, being intended for use on platforms where fork() isn't feasible, and B) While it's intended to be a kernel operation, there's still no guarantee that it's not a library operation that calls fork() and exec().