Bootstrapping with dynamically loading libraries in a microkernel

[AndrewAPrice]

I'm exploring dynamically loading libraries in my microkernel.

Basically, I'm sick of everything linking against libc or my GUI framework (which links against Skia) blowing my binaries up to 10 MB.

When my OS is running, I use a service to parse the executable files, create an empty process, load it into memory, kick off the first thread. But, for bootstraping, I have a small ELF parser in my kernel for loading the inital set of services.

This works for statically linked binaries, but if I want to bootstrap the initial services that depend on dynamically linked libraries, this becomes more complicated, because the kernel has to resolve the dynamic linking for the first initial set of services.

Do I build two implementations of the dynamic linker - one for userspace and one for bootstrapping? Can my userspace service (that does the dynamic linking and loading after the OS is running) share the dynamically loaded libraries at bootstrapping time?

Some possible approaches:

• Build two implementations of the dynamic linker (one in the kernel for bootstrapping, one in userspace). Somehow allow the userspace implementation learn about the dynamically loaded by the kernel during bootstrapping.
• Statically link my userspace loader (which complicates my build system a little because I need to build two versions of several libraries - static and dynamic versions), and load that as my first service.

What do microkernels typically do?

[Octocontrabass]

Is there some reason why the ld-linux.so approach wouldn't work for you?

[nullplan]

I don't know about microkernels, but on Linux, dynamic loaders have been implemented in only two ways:

1. Do what glibc, uclibc, and dietlibc are doing and have separate loader and libc. This necessitates duplicating some parts of libc inside of the loader. The benefit is, if you can pull it off, you have a really small loader and it can, in theory, load other libc implementations as needed. Or:
2. Do what musl is doing and put all of libc and the dynamic loader into the same binary. That way, there is no code duplication and you can update the libc atomically with a single rename() call.

Oh, and on the kernel level, the difference is also minuscule: A dynamic ELF file contains a PT_INTERP segment, naming the dynamic linker. If the kernel encounters it while loading the ELF file, it also loads the dynamic linker into the same address space and initially jumps to its entry point. It also sets a couple of aux vectors differently, namely AT_BASE is set to the interpreter base and AT_ENTRY is set to the main executable entry point.

Additionally, and basically orthogonally to the above, if the ELF type is ET_DYN, the loader can load the file anywhere in address space. The kernel can use this for address space layout randomization.

[rdos]

I have a kernel side loader for user applications. Or, rather, several. In the past I have also supported DOS applications and 16-bit protected mode, although these are no longer actively used. All applications currently are PE format, but I have an experimental ELF loader too. I don't think loader code should be in the application.

For the kernel side, I don't support PE or ELF format for device drivers. They must use a special binary format with an RDOS header defining the code and data segments. This format is implemented in the OpenWatcom linker. I also don't build a huge kernel file, rather there is a mechanism for dynamic linking both for device drivers and user applications that is resolved by a kernel module. Therefore, each device driver can be linked as a "module" and then is loaded at boot time based on a configuration file.

Server modules (in the microkernel model) are just ordinary applications in PE format that are loaded in a bit of a special process.

I don't use DLLs a lot, and typically link applications with static libraries. It's not a big issue given that they are typically only a couple of MBs. The SSL server is the largest and is close to 5MB.

[AndrewAPrice]

Is there some reason why the ld-linux.so approach wouldn't work for you?

How does the ld-linux.so approach work in a microkernel?

[nullplan]

How does the ld-linux.so approach work in a microkernel?

I believe the biggest issue you have is your attempt to put the ELF support into a service. I don't think that works too well, since the service also has to be some kind of binary. I'd put it into the main kernel, since it is memory-management-adjacent.

[Octocontrabass]

How does the ld-linux.so approach work in a microkernel?

I don't think there's any difference. Processes do their own dynamic linking, the only support they get from the ELF loader is where it loads and executes ld-linux.so (or whatever) instead of attempting to load and execute the dynamic executable directly.