ELF vs. raw binary for kernel image

[verticalegg]

Currently, I'm messing around with a basic kernel built for the RISC-V architecture. As RISC-V doesn't support GRUB/EFI natively, I'm using U-Boot as the current bootloader. Unfortunately, U-Boot's ELF support doesn't include relocation, I've had to link to a fixed location based on the board I'm compiling for. Due to this, I'm wondering whether it would be better to:

• link to a specific memory address based on board
• use a raw binary image to boot into so I can manage my own relative offsets
• load GRUB through U-Boot, then load a Multiboot image from GRUB

It's not a big deal as I can still test my code, but I would like to test code on different platforms and was wondering what you guys might think of this problem.

[Octocontrabass]

Does RISC-V position-independent code require any support from the loader? You might be able to build your kernel as a position-independent executable with no relocations.

Another option is to use two binaries: one is your kernel, and the other is a board-specific loader for your kernel. For example, you can have the loader set up paging so your kernel can run at its desired virtual address without any relocations.

[verticalegg]

Unfortunately, no matter what I try, the code throws an AMO access fault when loaded at an address different than the one in the linker script. This is after compiling with both PIC and PIE flags, as well as removing all references to addresses in the main program. I think it's because U-Boot just jumps to the start of where the header says the .text section says it is. As for the second option, I'd rather not implement my own loader - if I do go with another stage, I'm just going to use GRUB.

[Octocontrabass]

I'm not sure how much GRUB will help you here - last I checked, Multiboot2 doesn't provide relocation either, so your binary would still need to be linked to a board-specific address.

And I'm not suggesting you write a bunch of board-specific code, or even anything to load your kernel into memory from somewhere else. If you can get U-Boot (or GRUB, or whatever) to put your kernel into memory when it boots your loader, your loader only needs to parse a memory map and your kernel's headers to set up paging so your kernel can have a decent runtime environment. That's generic enough that you might only need separate linker scripts for each board's loader.

[nullplan]

ELF PIC requires loader support. In absence of such, you must perform the work yourself. musl's dynlinker does this, for example. You require a pointer to the image base, and one to the dynamic section. For the latter, you can almost always encode a dynamic position independent lookup of the symbol _DYNAMIC, typically as either a PC-relative lookup or a self-call trick. For example, in AMD64 you could do

Code: Select all

lea _DYNAMIC(%rip), %rsi

whereas in PowerPC it would be something like

Code: Select all

  bl 1f
   .long _DYNAMIC-.
1: mflr r4  #run-time address of the above into r4
  lwz r5,0(r4)  #offset into r5
  add r4,r4,r5 #run-time address of _DYNAMIC into r4

I don't know RISC-V, but these two approaches are typically sufficient.

Anyway, with the image base and the dynamic section in hand, you then look up the relocation tables, iterate over them, and process all the relative relocations by adding the image base to them. Since you are writing a kernel, you should not have any other relocations. But you can use "readelf -r" to verify the relocation type.

The code that does all of this must basically be the first thing after the _start label. Until it is done, you can pretty much only call other functions and refer to local (stack) variables. Global variables and constants only become available afterwards.

Code: Select all

void _dlstart_c(char *base, size_t *dyn, void (*next_stage)())
{
  size_t dynv[32];
  memset(dynv, 0, sizeof dynv);
  for (size_t i = 0; dyn[i]; i+= 2)
    if (dyn[i] < 32)
      dynv[dyn[i]] = dyn[i+1];
  size_t *rel = (void*)(base + dynv[DT_REL]);
  size_t rel_sz = dynv[DT_RELSZ];
  for (; rel_sz; rel_sz -= 2 * sizeof (size_t), rel += 2) {
    if (rel[1] == REL_RELATIVE) {
      size_t *rel_addr = (void*)(base + rel[0]);
      *rel_addr += (uintptr_t)base;
    }
  }
  rel = (void*)(base + dynv[DT_RELA]);
  rel_sz = dynv[DT_RELASZ];
  for (; rel_sz; rel_sz -= 3 * sizeof (size_t), rel += 3) {
    if (rel[1] == REL_RELATIVE) {
      size_t *rel_addr = (void*)(base + rel[0]);
      *rel_addr = (uintptr_t)base + rel[2];
    }
  }
  next_stage();
}

This code assumes you defined REL_RELATIVE to R_<arch>_RELATIVE from elf.h. It generally assumes you have elf.h available. You must provide the base address, dynamic address, and address of the next stage function from the external assembler code. That last one prevents the compiler from improperly inlining functions and moving initializations using relocations above the place where those relocations get filled in. The code is currently not passing any parameters to the next stage, but you will probably need to do that. It also assumes that the loader has already correctly mapped the ELF file into virtual memory.

Anyway, I have never found it necessary to have the kernel be a PIE, because all of virtual address space is available at the time you load the kernel. The only thing that might be taken is physical address space, but the proper way to abstract that is to use virtual memory.

[verticalegg]

Octocontrabass - you're right, I saw a patch but I think it's just a preparatory measure for if it ever does support relocation. As for the loader, I'm thinking a small UEFI program might be the best idea. Since UEFI has both FS support and memory allocation, it would be easy to build a somewhat dynamic loader, and since U-Boot has UEFI support it should be fairly portable. As far as paging, after exiting the UEFI boot environment it should be possible to remap virtual memory. If you have any other ideas please feel free to share - I'm not really that attached to the idea so it'd be nice to have some advice.

And nullplan - if I was able to get code running I would absolutely use your idea, but the real problem here is that U-Boot doesn't even get that far. Since the ELF header has an absolute start address, when linked to a different address than the load address we jump to somewhere completely random.

[nullplan]

And nullplan - if I was able to get code running I would absolutely use your idea, but the real problem here is that U-Boot doesn't even get that far. Since the ELF header has an absolute start address, when linked to a different address than the load address we jump to somewhere completely random.

D'oh. But that means U-Boot is wrong. If the ELF file declares type ET_DYN, then the start address is relative to the image base.

[verticalegg]

Okay- turns out I was being slightly stupid and managed to misconfigure my build script so it wasn't producing a PIE executable. However, now U-Boot just flat-out refuses to run it, so that doesn't change the fact that for a relocatable kernel I need another stage.

[Octocontrabass]

U-Boot has UEFI support

Well, that's excellent news. UEFI requires relocatable binaries, so you can write a single loader that will run on all of your boards. The only downside is that your loader will have to be a PE binary instead of ELF.

As far as paging, after exiting the UEFI boot environment it should be possible to remap virtual memory.

If you create the paging structures before you exit boot services, you can take advantage of UEFI's AllocatePages() to find free memory to store them instead of needing to parse the memory map yourself.

[verticalegg]

Oh, nice! I'll have to see how much stuff U-Boot implements for UEFI, since there was a disclaimer about not supporting everything, but that sounds like a plan for now.