OSDev.org

Hi all,

In my 32 bit x86 OS I can load executables via ELF into user space at a fixed virtual address (usually 0x80000000). But now I want to start putting drivers in "kernel modules" and loading them at run-time and I want to use ELF for this too. The problem I'm having is that obviously kernel modules will not have their own virtual memory space so I can't just their load segments that at whatever address is specified in the elf program header.

So clearly they have to be relocated and this is where I am getting confused, because there seems to be more than one kind of "relocation" when it comes to ELF, and I'm not sure which applies to kernel modules. Should I be building my modules as relocatable files e.g. ld -r, or as dynamic libraries with position independent code? Does relocation really require that I use the section header (which are unused for my executable loader), or is the information in the program header table?

here is my current likely flawed elf code for user executables : https://github.com/pgrAm/JSD-OS/blob/ma ... rnel/elf.c. How would I go about adding support for relocation to this?

I'm interested in how you chose to implement kernel modules if you have them? And if they should dynamic libraries can someone refer me to some resources on how to load/relocate these?

few days ago I was wondering How does DOS loads drivers and TSRs while they can't be loaded at a fixed address (DOS has no virtual address so only one program at once can be loaded). I found they use self-relocation. the drivers relocate their code by self at runtime. this is for MZ flat binaries. You are using elf so surely you can easily relocate your code. elf is used in Linux, FreeBSD, Minix and others so you can take a look at their code to figure out how to do that.

For my OS, kernel modules are "relocatable executable" files, generated by passing the "-q" (aka "--emit-relocs") option to GNU ld ("-Xlinker -q" when linking via gcc). When loading, I read the relocation sections (so, yes, you do need to read the section headers) and perform the required relocations.

There is one "catch" however with this method, as the linker has already performed the relocations when creating the executable, so the only relocation you need to actually do anything with is "R_386_32" (at least for 32-bit x86) and then it's a simple case of adding the address you loaded the module at (you may need to subtract the default base address, but I link my modules with a base of 0 to avoid this) to the referred value.

PgrAm wrote:there seems to be more than one kind of "relocation" when it comes to ELF

Not really many kind; it's only one that has addend, and one that hasn't. Besides of that there's a record for code (functions), and a record for data. That's 4. You can ask the linker to only produce some of these. You don't have to support all, only the ones that your modules actually use (which in turn depends on your linker script).

PgrAm wrote:I'm interested in how you chose to implement kernel modules if you have them?

You can do that in many different ways. The simplest I think is to compile the modules as relocatable shared libraries, then designate an area for them in your kernel address-space. In your kernel you should have a list which keeps track of that area, aka which module loaded at which address and what is the next free address. Once the module is loaded, you have two options:
- use specific function names which you can add to a hook list in your kernel (like "irqhandler" or "opendevice" for example)
- call the entry point in each module which in turn calls register hooks calls to add its functions to the kernel's list.

PgrAm wrote:And if they should dynamic libraries can someone refer me to some resources on how to load/relocate these?

Our wiki page has some information on this, but it mainly focuses on loading shared libraries in user-space using the "needed" records in ELF.
In addition to that, this is what you need:
- you must create a list with addresses that you want to export to the modules. If you only allow your modules to call kernel functions, then this list can be static. If you allow your modules to call each other's functions (like usb-storage calling a function in usb-roothub), then you have to maintain this list dynamically as you load and unload the modules. When you add to this list, you'll save the module load address + offset in the module plus the symbol.
- on module load, you have to locate the relocation table(s). Some might have one table for both data and code, others might have two separated tables (depends on your linker script).
- iterate through that table, and replace the offset in the module with the offset in your export list + addend. So *(offset in table + load address) = offset in export list + addend in table.
- (optional) if you want dynamic export list, then iterate through the dynamic segment and add offsets of the newly loaded module to the list.

Here's a implementation to look at:
- locating RELA table
- replacing plt offsets
- replacing symbol offsets
This code does the aforementioned list with addresses the other way around: the relas array contains pairs of symbol and offset, but this is not the offset of the symbol, rather the offset where it's referenced. So when I load an ELF with an exported symbol, and I know where it's loaded in the memory, I iterate through this list and fill its references. This is faster, although care must be taken to load the ELF binaries in a specific order (this code loads shared libraries for an address-space, so I can know the order in advance; for you, as you might load and unload the modules in any order, so you'll need to keep a list of symbol and offset where is located rather than referenced.)

Another thing, this code assumes there's only one relocation table with both jump and data entries, because I wrote the linker script that way. It was due to a compatibility issue with gcc and Clang, this is the exception rather than the rule. You could use data relocation entries in a separate table too, this only depends on how you write your linker script.

(The code has comments about GNU ld and LLVM lld. Those are not relevant any more since I've convinced the LLVM developers to implement RELA tables the same way as GNU ld, and they did.)

Cheers,
bzt

mallard wrote: For my OS, kernel modules are "relocatable executable" files, generated by passing the "-q" (aka "--emit-relocs") option to GNU ld ("-Xlinker -q" when linking via gcc). When loading, I read the relocation sections (so, yes, you do need to read the section headers) and perform the required relocations.

There is one "catch" however with this method, as the linker has already performed the relocations when creating the executable, so the only relocation you need to actually do anything with is "R_386_32" (at least for 32-bit x86) and then it's a simple case of adding the address you loaded the module at (you may need to subtract the default base address, but I link my modules with a base of 0 to avoid this) to the referred value.

so you recreate the PE base relocation semantics. tell me, how you solve the ELF lack of import/export? I mean, if your drivers are not ELF "shared" objects, then they aren't PIC, and thus don't use GOT/PLT stuff for function linkage/binding. then the question arises, how you make all that function binding without a normal import/export mechanism? I have a problem of not supporting PE on mips and been thinking on how to resolve this using ELF on this platform without falling to that nasty PIC thing. as I saw it, it touches compiler internals, rather than just linker and that sucks. a call to the local function translates into "call MyFunction", whereas the call to imported function would be something like "call [IAT_BASE + MyFunction_IAT_OFFSET]".

drivers call kernel service functions, there are also interdriver communications (where one driver serves as a library for another one). this implies a highly dynamical import/export mechanism. with PE, it's more than easy, since that format is designed for modularity. but this ELF, how to do that with it? I am missing a lot in the ELF internals, but so far, intuitively I feel it is very against DLL. how did you make import/export in that context? how do you make the compiler understand that the call to the (exported) kernel function should be treated differently from local calls or calls resolved through GOT/PLT?

for the OP, maybe you try PE if your compiler, linker (and religious beliefs) allow. because what you described you want is done best with the PE. I wouldn't even think about ELF should I have a PE compiler for MIPS, which I don't want to drop no matter I have only one board with, that has long been dropped by the vendor.

Thanks for the code @bzt, I think reading it will clear up a lot for me. So basically I think I should implement the modules as shared libraries. compiling an linking (with clang/llvm), as such:
Then I will load segments from driver.sys to an address in kernel space, then modify the addresses in the relocation table to adjust for its new address, and then finally resolve references to my kernel symbol table. I'll see how this goes.

@zaval, my rationale for using elf are mostly that I was already using it for applications and it was widely supported by compilers/linkers, I'm certainly not married to it though.