UEFI Bare Bones

[xenos]

There are a few things I did not like about the UEFI Bare Bones article:

• It uses a Windows targeted toolchain, while basically all non-UEFI bare bones / tutorials here suggest using either a bare metal cross compiler or one specifically targeting the OS to be created.
• It pulls in code dependencies from gnu-efi.
• It is 64 bit only.

To have a "cleaner" environment and less dependencies, and have also a 32 bit version, I have worked out how to build an EFI application in a different way, which is closer to the other bare bones tutorials:

User:Xenos/UEFI Bare Bones

• It uses a bare metal toolchain, which is almost one-on-one taken from the GCC Cross-Compiler taken, and only a PE target for binutils is added.
• It uses no external code / libraries / headers, but comes with its own minimal efi.h header.
• It targets both 32 and 64 bit x86 systems.
• It includes instructions for running in VirtualBox.

Any comments and suggestions for improvements are welcome! Also I would like to add instructions for ARM as well, if I find a good target system.

[kzinti]

Your new version looks much better.

I've noticed that you didn't need to use any self-relocation code at startup, like gnu-efi does. I am curious why this is so... As far as I understand it ELF relocations are not understood by a PE loader. Does it have something to do with building the bootloader as a shared object (dynamic library)?

[xenos]

Thanks!

Yes, indeed, it is due to the use of position independent code with the -fpic switch (which is typical for a shared library).

[zaval]

There are a few things I did not like about the UEFI Bare Bones article:

It uses a Windows targeted toolchain, while basically all non-UEFI bare bones / tutorials here suggest using either a bare metal cross compiler or one specifically targeting the OS to be created.

Why this is "bad"? suggesting everywhere false advices of using linux as a "more developer friendly" environment is "good", but using a compiler, that is the most convenient one in targetting UEFI is "bad"? MSVC doesn't have all this stupidity of being either "bare bones/cross compiler" or being only able to compile for linux, it just compiles C/C++ code. it's multitargetting capable in other words. and UEFI is one of the targets. already! why this is less "clean", than jerking off with the gcc stiffness? you can target WinAPI, you can create a Windows driver, UEFI (both, the FW itself and its applications) and even your own OS with MSVC. without any hassle. it's not stubbornly tight with any C library or whatever.

I believe, the UEFI article should leave the note about MSVC being the most convenient compilation toolset for UEFI targetting. If, of course, the goal of the wiki is helping users and not pretending the one's preferences as such.

I've noticed that you didn't need to use any self-relocation code at startup, like gnu-efi does. I am curious why this is so... As far as I understand it ELF relocations are not understood by a PE loader. Does it have something to do with building the bootloader as a shared object (dynamic library)?
...
Yes, indeed, it is due to the use of position independent code with the -fpic switch (which is typical for a shared library).

guys, are you aware, that all this is not needed for PE/UEFI? shared libraries... why? UEFI loads your OS Loader and performs base relocations ANYWAY. your .so will make a bad service, since base relocated code is faster than PIC. just link you PE image at whatever image base address you want and UEFI will handle it.

if you are so inclined to use linux, then use a normal compiler tool, normal here means "clang". why you need to complicate yourself the task, moreover - advice this for others. you are producing these monsters, being PE outside but ELF .so inside, why? when everything you need is a PE generating compiler/linker. and UEFI headers.

yet once a proof of how all these ELF oriented toolchains are not appropriate and confusing for UEFI, and how they are "cleaner"... Instead of digging into UEFI itself or you loader logic, people mess around with absolutely unnecessary trickery with ELF. pic makes no sense in the PE environment. you only add the overhead and distract yourself. sigh. do whatever you want anyway, but please don't litter the wiki with the bias. writing a UEFI OS loader wiki should not be a "how to masturbate with ELF GCC to make PE UEFI image wiki".

[xenos]

It seems that you didn't get the point. Nobody said that it impossible to use a Windows targeted toolchain to write UEFI applications. Of course it is possible. The point is that it is not necessary to get a Windows targeted toolchain if one already has an ELF targeted toolchain anyway (maybe because of writing a non-EFI, multiboot kernel), and one can just keep using it to compile an UEFI application as well, instead of getting another one.

[zaval]

but you weren't talking about "adding", rather "replacing" with something "cleaner".
replacing a PE generating, UEFI targetting compiler toolset with an ELF oriented one is not cleaner. if one needs to use -fPIC for producing PE images, then this method, with all the honesty, shouldn't be even included in the wiki, since it's plainly nonsensical. just out of curiosity, can you tell me why you need here -fPIC?

[bzt]

Hi,

I like your page!

Two things I'd like to add: under "Creating disk image", you should link to here (as it describes how to create an image in more detail), and also a link to the ESP page would be great (considering it's mandatory for UEFI).

The other thing, I see not much point in 32 bit UEFI. There was a short time when it was supported, but Win made 64 bit mandatory a couple of years now. Same happened with Mac machines, so it is safe to assume 99% of UEFI machines support 64 bit. UEFI 32 is like 286 16 bit protected mode.

Cheers,
bzt

[kzinti]

I used to compile my UEFI bootloader using mingw (as per the current wiki page I suppose). It did work great and I agree with Zaval that it makes things easier.

Unfortunately it was impractical for me to keep using mingw:

1) I needed another compiler (mingw under Linux, MSVC under Windows) just to support UEFI. I much prefer having one toolchain that works for everything.
2) I couldn't figure out how to build mingw to match the latest version of GCC: mingw is out of sync with the latest releases of GCC which I wanted to use. Some C++ constructs would not compiler under mingw.
3) Having ONE compiler for all target is just a net win. Same bugs, same quirks, some behaviour everywhere.
4) One day I will want to have a native compiler toolchain in my OS. This cannot be MSVC. It won't be mingw (out of date compiler). I certainly do not want to have to port multiple compilers to my OS.
5) clang --> I did not investigate this, maybe it's worth looking at.
6) I much prefer dealing with ELF files than PE files. I can use the same tools for everyone and it just works.

IIRC, PE files typically handle being loaded at any address using relocations (just like ELF files do). But the format of these relocations is incompatible between PE and ELF. So the gnu-efi solution is simply to do the ELF relocations at the entry point. This is actually pretty easy to do:
https://github.com/kiznit/rainbow-os/bl ... .S#L40-L46
https://github.com/kiznit/rainbow-os/bl ... pp#L34-L87

Is this hacky? Sure is. Am I entirely happy with this? I am not. Does it work? yes. What's more, it solves all the concerns I had and listed at the top of this post.

Now XenOS's solution seems even better in that he was able to remove any relocation code from inside the bootloader by using Position-Independant Code. So effectively the produced ELF files doesn't need any relocations anymore. This seems less hacky and overall better than what I have right now.

I am not sure why you are being so aggressive about this. I do agree that we could update the wiki to add this new information and not erase the existing one. Ideally the wiki could explain how to use mingw, msvc, gcc or clang to build your UEFI bootloader, perhaps with pros and cons of each method.

For me, using the same toolchain (compiler, linker, tools) for all my binaries for all my target platforms trumps other concerns. MSVC is great but, I can't target ARM and AARCH64 with it. With GCC it's easy to create a cross compiler and target any architecture.

guys, are you aware, that all this is not needed for PE/UEFI? shared libraries... why? UEFI loads your OS Loader and performs base relocations ANYWAY. your .so will make a bad service, since base relocated code is faster than PIC. just link you PE image at whatever image base address you want and UEFI will handle it.

1) A PE UEFI executable is relocatable. There is no guarantee that the firmware will load it at the address you specified at link time. This works because PE files have relocations just like ELF files do, they just use a different format.
2) Performance of PIC vs non-PIC code here really doesn't matter. Who cares. It's a bootloader. Here things are I/O bound, CPU usage is a non-issue.

[kzinti]

The other thing, I see not much point in 32 bit UEFI. There was a short time when it was supported, but Win made 64 bit mandatory a couple of years now. Same happened with Mac machines, so it is safe to assume 99% of UEFI machines support 64 bit. UEFI 32 is like 286 16 bit protected mode.

IIRC the only 32 bits UEFI implementation that ever existed was for the very first Mac with UEFI support. The irony is that machine had a 64 bits processor. At some point I was looking to buy one of these machines just to test my bootloader. You can get one for cheap. But the machine is so old (way over 10 years) that it might actually be hard to find one that works. I concluded that it's not worth thinking about 32 bits UEFI. That said i still support it in my build system just so that I can ensure my code is "clean". The only place I run the 32 bits UEFI bootloader is in qemu.

[zaval]

MSVC is great but, I can't target ARM AARCH64 with it. With GCC it's easy to create a cross compiler and target any architecture.

You can. x86, x64, ARM32 and ARM64 are now all equally available archs with MSVC for any supported targets (WinAPI, drivers, UEFI, bare metal). and if you have some old installations, then Itanium as well. too bad, Intel ditched it.

IIRC, PE files typically handle being loaded at any address using relocations (just like ELF files do). But the format of these relocations is incompatible between PE and ELF. So the gnu-efi solution is simply to do the ELF relocations at the entry point. This is actually pretty easy to do:

ELF executables aren't relocatable. what ELF calls "relocations" are "object relocations", they get resolved at the link time, both in PE/COFF and ELF and are not present in the resulting executable. It's a concern for the linker.
PE executables and DLLs are relocatable - no matter at what address you linked them, they can be loaded at another one. And here, the mechanism of base relocations plays its role, the mechanism, that is imo superior to the PIC approach, since is more performant and much clear to understand. base relocations are very simple - it's an addendum that gets added to every absolute address reference. anywhere in data and code, when an absolute address is referenced, after loading the image, at its actual base address, the loader needs to fix those addresses, applying simple arithmetic. records for these places are being kept in the .reloc section, pointed by relocation directory entry. their types aren't numerous and one can figure them out in minutes.

if you need to get an ELF formatted image for something, you go with the appropriate tool and don't question why you should use it. why should it be different for another format? UEFI uses PE and thus - requires a PE capable toolset. I heard clang is doing well on that front, both for x86 and ARM, so you can do it on linux too. on Windows, there is just everything ready for you, just take it and make your OS loader instead of torturizing yourself with trying to fit an unsuitable thing.

[kzinti]

You can. x86, x64, ARM32 and ARM64 are now all equally available archs with MSVC for any supported targets (WinAPI, drivers, UEFI, bare metal). and if you have some old installations, then Itanium as well. too bad, Intel ditched it.

Ok, I didn't know that. I still can't port MSVC to my OS.

ELF executables aren't relocatable. what ELF calls "relocations" are "object relocations", they get resolved at the link time, both in PE/COFF and ELF and are not present in the resulting executable. It's a concern for the linker.

That's odd because I do have relocations in my non-PIC ELF code. It doesn't matter what you call them and whether or not they are the same in different binary formats. Sure they get resolved at link time to the preferred address, but I have no problem walking the "object relocation" list and adding an offset to them. In fact, PE and (non-PIC) ELF files work the exact same way here: they are linked to their preferred address and the loader is free to relocate them.

PE executables and DLLs are relocatable - no matter at what address you linked them, they can be loaded at another one. And here, the mechanism of base relocations plays its role, the mechanism, that is imo superior to the PIC approach, since is more performant and much clear to understand. base relocations are very simple - it's an addendum that gets added to every absolute address reference. anywhere in data and code, when an absolute address is referenced, after loading the image, at its actual base address, the loader needs to fix those addresses, applying simple arithmetic. records for these places are being kept in the .reloc section, pointed by relocation directory entry. their types aren't numerous and one can figure them out in minutes.

That's exactly what my code does right now using (non-PIC) ELF files. But instead of having the OS loader do it, I manually do it myself. So really, there is no practical difference here.

if you need to get an ELF formatted image for something, you go with the appropriate tool and don't question why you should use it. why should it be different for another format? UEFI uses PE and thus - requires a PE capable toolset. I heard clang is doing well on that front, both for x86 and ARM, so you can do it on linux too.

I respectfully disagree. You should always question what you are doing and whether or not the tools you are using will get you where you want to go. As I have expressed above, I have other practical concerns like one day having a native compiler running in my OS. Going with clang could be an option and I will consider it at some point (it's something I wanted to do anyways, I just didn't get to it).

on Windows, there is just everything ready for you, just take it and make your OS loader instead of torturizing yourself with trying to fit an unsuitable thing.

Implementing this wasn't torture, in fact I found it quite interesting and rewarding!

[zaval]

That's odd because I do have relocations in my non-PIC ELF code. It doesn't matter what you call them and whether or not they are the same in different binary formats. Sure they get resolved at link time to the preferred address, but I have no problem walking the "object relocation" list and adding an offset to them. In fact, PE and (non-PIC) ELF files work the exact same way here: they are linked to their preferred address and the loader is free to relocate them.

you can instruct the linker to keep info of relocations, but still it's not quite the same. by "resolved" relocations I meant those, that don't need load time fixing at all, say you place a variable X, compiler doesn't know where it ends up, so makes a relocation on it, but linker does know its final placement, - it resolves it PC-related (IP-related) and referencing X doesn't use absolute address at all. such things are the only real PIC. now, I ask, maybe I am wrong - what exactly relocations the ELF linker preserves - all or only those, that are subject to base relocations? because in PE, only the latter are bookkept, because it's a native method for the format, it knows what they are for. whereas for ELF, keeping relocations is something clueless, that is for "analyzing tools" and I fear it dumps everything that was relocations at the link time. this means, that you would need to through a lot of garbage, extracting only relocations of interest, but maybe I am wrong, since I ditched my attempts to try to convert ELF in PE a long time ago and forgot many things.

I have other practical concerns like one day having a native compiler running in my OS.

btw about this. I know this is one of the main points of those, who advocates for using gcc - like, hey, one day, your OS would be capable of running a compiler, self hosting, b1tches, so take gcc, it teaches you to have a million variants from the beginning!
I won't be arguing if this is a really good advice to listen to, but I don't see how using another compiler while you don't have yet such an established environment in your OS is preventing from having that future compiler. just don't second linux mistake and don't make code of your OS compiler bound.

[kzinti]

you can instruct the linker to keep info of relocations, but still it's not quite the same. by "resolved" relocations I meant those, that don't need load time fixing at all, say you place a variable X, compiler doesn't know where it ends up, so makes a relocation on it, but linker does know its final placement - it resolves it

So far I have the same understanding.

it resolves it PC-related (IP-related) and referencing X doesn't use absolute address at all.

That might be true for x86_64, but it certainly is not true for ia32 since that architecture doesn't have IP-relative addressing. It only has absolute addressing. For lot of details, see https://nullprogram.com/blog/2016/12/23/.

such things are the only real PIC. now, I ask, maybe I am wrong - what exactly relocations the ELF linker preserves - all or only those, that are subject to base relocations? because in PE, only the latter are bookkept

The exact same thing is happening with the (non-PIC) ELF files I am generating. They only contain "R_386_RELATIVE" or "R_X86_64_RELATIVE" relocations, which are basically relative offsets that need to be corrected to take the image base into account. Although they are called "relative" relocations, the instructions are actually using absolute addresses and this is why the offsets need to be corrected to account for the image base.

ia32 version: https://github.com/kiznit/rainbow-os/bl ... oc.cpp#L76
x86_64 version: https://github.com/kiznit/rainbow-os/bl ... oc.cpp#L76

Note that in both cases my linker script will link the ELF file with an image base of 0x00000000. This is not a requirement, it just makes it easier to debug things (relative addresses become offset into the memory image). The other reason to link at 0x00000000 is that it ensures my relocation code works properly since UEFI will not load my bootloader at that address (I verified this).

, because it's a native method for the format, it knows what they are for. whereas for ELF, keeping relocations is something clueless

I have no idea why you say that or what you mean by it. You seem to suggest that relocations in ELF files are a hack. They are not. They do work differently than for PE files, but they are not less hacky / clueless / useful. They are very well documented an easy to use. They are also used for the exact same reason relocations are used in PE files.

https://intezer.com/blog/elf/executable ... locations/

that is for "analyzing tools" and I fear it dumps everything that was relocations at the link time. this means, that you would need to through a lot of garbage, extracting only relocations of interest, but maybe I am wrong, since I ditched my attempts to try to convert ELF in PE a long time ago and forgot many things.

There isn't much to it. I have a ".reloc" section in my linker script:

Code: Select all

    .reloc ALIGN(4K) :
    {
        *(.reloc)
    }

and the only relocation types I get in my executable (they certainly are not discarded!) are R_386_NONE and R_386_RELATIVE. The former you can ignore and the later is trivial to handle.

btw about this. I know this is one of the main points of those, who advocates for using gcc - like, hey, one day, your OS would be capable of running a compiler, self hosting, b1tches, so take gcc, it teaches you to have a million variants from the beginning!
I won't be arguing if this is a really good advice to listen to, but I don't see how using another compiler while you don't have yet such an established environment in your OS is preventing from having that future compiler. just don't second linux mistake and don't make code of your OS compiler bound.

I agree that it doesn't prevent changing the compiler in the future. But I really don't want to spend time dealing with #ifdef GCC/MSVC/... and having to typedef every basic type I want to use because they might not be the same on all platforms. I also don't want to deal with compiler incompatibilities, compiler bugs, compiler supporting different version of C++, and so on. Basically, having the exact same compiler for all binaries just saves development time, a resource that is scarce on a one-man osdev project (I get the irony).

I did start this project using mingw for the UEFI bootloader as I used to think that using a non-PE compiler to generate PE files made no sense. In the end it was causing me more grief than it was helping me. This is when I switched to this method. I suppose YMMV.

[kzinti]

I was just going through my makefiles and noticed something interesting: I *am* using PIC and compiling my bootloader as a shared object ELF file. Yet I still see the need for self-relocation code. Something is not adding up here.

So apologies to Zaval and future readers, what I said in the previous post(s) is likely wrong in some specific areas. It sounds like normal ELF files (non-PIC) are indeed non-relocatable (otherwise I wouldn't have bothered with using PIC). That said, the performance impact really doesn't matter in a bootloader. If you are loading your kernel directly from the UEFI firmware, that might be a different story.

Here are the relevant bits:

https://github.com/kiznit/rainbow-os/bl ... mk#L10-L21

XenOS: are you sure you don't need self-relocation code? Your sample is very simple and I wonder if it is simply the case that it doesn't contain any relocations (or that your bootloader happens to be loaded at its preferred address). Otherwise I am not sure I understand what's different between our two implementations.

[xenos]

XenOS: are you sure you don't need self-relocation code? Your sample is very simple and I wonder if it is simply the case that it doesn't contain any relocations (or that your bootloader happens to be loaded at its preferred address). Otherwise I am not sure I understand what's different between our two implementations.

So far I have only worked out this little "Hello world" where indeed no relocations are needed, but PIC is needed, since the sections very likely get loaded at a different address from the preferred (in the linker script I put the start to the first page at 0x1000 - originally I had 0x0 there, but OVMF refused to load it). I guess I will see how it goes once I add more stuff.

I agree that using PIC and converting between ELF and PE/COFF, including compatibility issues with these two formats, is still far from optimal, and I still work on improving things. Ideally, I would like to use a bare metal, freestanding, PE targeted toolchain at first for UEFI bootloader / kernel building, and then a custom OS, hosted toolchain, which can build both applications for the custom OS and also the (in this case PE) kernel with -ffreestanding. So this is still work in progress, and the main point where I was struggling so far is the bare metal PE target, which is apparently not supported by all parts of the toolchain.

By "cleaner" I meant mostly that it needs less dependencies - like borrowing "parts of" gnu-efi in the original UEFI Bare Bones article. But also the MinGW targeted toolchain, which is designed for building Windows applications, comes with unnecessary parts for building just bare metal UEFI applications. Having a hosted Windows environment is simply not necessary, just as ELF kernels are (recommended to be) compiled with a bare metal ELF toolchain and not a Linux targeted one.

Sure, one can use MSVC if one has Windows and MSVC. But not everybody uses them - there are enough OS development folks who use different operating systems, different compilers (open source) and who would still like to do UEFI development, so directing everyone to just use MSVC is pointless here. The important point is to have different alternatives.