Bootstrapping with dynamically loading libraries in a microkernel

[AndrewAPrice]

I'm exploring dynamically loading libraries in my microkernel.

Basically, I'm sick of everything linking against libc or my GUI framework (which links against Skia) blowing my binaries up to 10 MB.

When my OS is running, I use a service to parse the executable files, create an empty process, load it into memory, kick off the first thread. But, for bootstraping, I have a small ELF parser in my kernel for loading the inital set of services.

This works for statically linked binaries, but if I want to bootstrap the initial services that depend on dynamically linked libraries, this becomes more complicated, because the kernel has to resolve the dynamic linking for the first initial set of services.

Do I build two implementations of the dynamic linker - one for userspace and one for bootstrapping? Can my userspace service (that does the dynamic linking and loading after the OS is running) share the dynamically loaded libraries at bootstrapping time?

Some possible approaches:

• Build two implementations of the dynamic linker (one in the kernel for bootstrapping, one in userspace). Somehow allow the userspace implementation learn about the dynamically loaded by the kernel during bootstrapping.
• Statically link my userspace loader (which complicates my build system a little because I need to build two versions of several libraries - static and dynamic versions), and load that as my first service.

What do microkernels typically do?

[Octocontrabass]

Is there some reason why the ld-linux.so approach wouldn't work for you?

[nullplan]

I don't know about microkernels, but on Linux, dynamic loaders have been implemented in only two ways:

1. Do what glibc, uclibc, and dietlibc are doing and have separate loader and libc. This necessitates duplicating some parts of libc inside of the loader. The benefit is, if you can pull it off, you have a really small loader and it can, in theory, load other libc implementations as needed. Or:
2. Do what musl is doing and put all of libc and the dynamic loader into the same binary. That way, there is no code duplication and you can update the libc atomically with a single rename() call.

Oh, and on the kernel level, the difference is also minuscule: A dynamic ELF file contains a PT_INTERP segment, naming the dynamic linker. If the kernel encounters it while loading the ELF file, it also loads the dynamic linker into the same address space and initially jumps to its entry point. It also sets a couple of aux vectors differently, namely AT_BASE is set to the interpreter base and AT_ENTRY is set to the main executable entry point.

Additionally, and basically orthogonally to the above, if the ELF type is ET_DYN, the loader can load the file anywhere in address space. The kernel can use this for address space layout randomization.

[rdos]

I have a kernel side loader for user applications. Or, rather, several. In the past I have also supported DOS applications and 16-bit protected mode, although these are no longer actively used. All applications currently are PE format, but I have an experimental ELF loader too. I don't think loader code should be in the application.

For the kernel side, I don't support PE or ELF format for device drivers. They must use a special binary format with an RDOS header defining the code and data segments. This format is implemented in the OpenWatcom linker. I also don't build a huge kernel file, rather there is a mechanism for dynamic linking both for device drivers and user applications that is resolved by a kernel module. Therefore, each device driver can be linked as a "module" and then is loaded at boot time based on a configuration file.

Server modules (in the microkernel model) are just ordinary applications in PE format that are loaded in a bit of a special process.

I don't use DLLs a lot, and typically link applications with static libraries. It's not a big issue given that they are typically only a couple of MBs. The SSL server is the largest and is close to 5MB.

[AndrewAPrice]

Is there some reason why the ld-linux.so approach wouldn't work for you?

How does the ld-linux.so approach work in a microkernel?

[nullplan]

How does the ld-linux.so approach work in a microkernel?

I believe the biggest issue you have is your attempt to put the ELF support into a service. I don't think that works too well, since the service also has to be some kind of binary. I'd put it into the main kernel, since it is memory-management-adjacent.

[Octocontrabass]

How does the ld-linux.so approach work in a microkernel?

I don't think there's any difference. Processes do their own dynamic linking, the only support they get from the ELF loader is where it loads and executes ld-linux.so (or whatever) instead of attempting to load and execute the dynamic executable directly.

[eekee]

A funny thing was going on with ld-linux.so in the 00s. It was described as just like an interpreter, and I believe there was some code in the kernel to load other 'interpreters' instead. There was talk of having Linux run Windows binaries by invoking Wine as the interpreter. Java code might already have been implemented that way.

What do I mean by "just like an interpreter"? #! scripts in Unix were traditionally handled by the shell, you couldn't just launch them with exec(). Linux changed so the kernel can recognize the #! line and invoke the interpreter with the script's filename. It can also recognize dynamically-linked ELF files and invoke ld-linux.so much like an interpreter, except I guess it may pass pointers to the already-mapped file.


From a more meta perspective, I've always intended to make the GUI into a set of services as QNX did in the 90s, but it seems like a whole other lot of design work and I'm starting to get fed up with the number of things I need to design. I'm tempted to give up on Kaph and just use a native Forth, write ad-hoc code and devlop guidelines ex post facto.

[nullplan]

There was talk of having Linux run Windows binaries by invoking Wine as the interpreter. Java code might already have been implemented that way.

That happened. It is now called binfmt_misc, and it allows you to add (actually prepend) new executable formats to the binary formats table. You give it a magic offset, length, and value, and the kernel now will run all binaries it is told to run that have the magic value in the magic place with the program given.

I have seen this used to run Windows programs with Wine or mono, and to run foreign-architecture ELF files with QEMU.

This of course opens the door to abuse, where a script file is executed, and its interpreter is a .NET EXE, and mono requires an interpreter. I think the stack can only go up to four layers before Linux just rejects it.

It can also recognize dynamically-linked ELF files and invoke ld-linux.so much like an interpreter, except I guess it may pass pointers to the already-mapped file.

Pretty much that. It recognizes ELF files in need of interpreting by the PT_INTERP program header. If found, it will continue to map the ELF file as it normally would, then map the interpreter, and prepare the aux header AT_BASE to point to the base of the interpreter and AT_PHDR to point at the program header of the main program.

I'm tempted to give up on Kaph and just use a native Forth, write ad-hoc code and devlop guidelines ex post facto.

That is the only sensible way to go. Only after you have experienced what works and what doesn't is it sensible to codify anything like guidelines.

[eekee]

binfmt_misc, of course! It's been so long.

Linux's 4-layers limit seems sensible. It might be configurable, not sure if it's a compile-time option or a sysctl, though my knowledge mostly predates sysctl so I guess it's compile-time.

It can also recognize dynamically-linked ELF files and invoke ld-linux.so much like an interpreter, except I guess it may pass pointers to the already-mapped file.

Pretty much that. It recognizes ELF files in need of interpreting by the PT_INTERP program header. If found, it will continue to map the ELF file as it normally would, then map the interpreter, and prepare the aux header AT_BASE to point to the base of the interpreter and AT_PHDR to point at the program header of the main program.

As it maps both of these, does at least 1 need to be position-independent? Oh but the 'interpreter' could relocate the ELF file, couldn't it? I didn't understand relocateability back then, so it didn't get stored in my memory.

I'm tempted to give up on Kaph and just use a native Forth, write ad-hoc code and devlop guidelines ex post facto.

That is the only sensible way to go. Only after you have experienced what works and what doesn't is it sensible to codify anything like guidelines.

That does make sense, thanks.