Announcing POSIX-UEFI

[vvaltchev]

If UEFI itself can't access paths inside a partition, then POSIX-UEFI can't either.

UEFI can open files at any given path inside FAT16/FAT32 partitions and you can choose which partition too, even if the interface for that is not so nice.

What do you mean "many cases"? So far GOP is the only one that I couldn't wrap up in POSIX calls.

Sorry, what is "GOP" ?

Everything needed by a basic boot loader covered.

Ehm, maybe. But I was thinking about the general case, not about a very basic bootloader. For example, I wrote a UEFI bootloader that take advantage of many UEFI features that are not covered.

What is your problem? By default you have to use that messed up UEFI interface for EVERYTHING or use those braindead library functions from EDK2 (which do not cover every UEFI protocol either btw). When you use POSIX-UEFI, you can spare yourself and use a much simpler interface in 99% of the time. And even for those 1% native UEFI calls POSIX-UEFI provides a simple implementation, a simple and fast build system (compared to EDK2's bloat). What is that you can't understand about this? If you are your own enemy, then go ahead and use that horrible EDK2. I've done that, I know exactly what I'm talking about. No sane people would ever think EDK2 is good or at least a bit usable.

Relax, man. There's no need to "heat up". I'm not attacking you nor your project. Actually, the only reason I'm spending time on this discussion is because I kind of liked your overall idea and I was trying to discuss ways to improve it, through constructive criticism. If you're against that, I'd be happy to stop.

GNU-EFI is much better, but it has its own pitfalls (the need of uefi_call_wrapper and doesn't compile with CLang).

I've been using only GNU-EFI, all the time. The EDK2 bloat is not worth to me either. But, I have to point out that GNU-EFI does not need uefi_call_wrapper: newer versions of GCC support directly the special MS-ABI attribute, but it works with older versions of GCC as well by using -maccumulate-outgoing-args. I've personally done it. uefi_call_wrapper() is needed only if you don't want or cannot use the MS ABI.

At the moment, I believe that the POSIX model cannot map well UEFI, roughly because UEFI is not a UNIX-like kernel.

Think it again. Apart from the ABI their API is not so different. UEFI mimics Windoze calls, and you can convert those into libc (anybody who has written a multiplatform application knows that). The only real difference is, that UEFI needs a huge load of helper functions to hide the actual protocol interfaces, while a true POSIX kernel doesn't, just a single libc with well-defined interface to wrap up syscalls.

I don't agree. The mapping is only partial. I've tried explaining this, no point in insisting further.

While some POSIX functions might be perfect for UEFI, in other cases you will inevitably end up with limitations.

For those rare cases you use the same interface that you'd have to use for EVERYTHING otherwise.

The problem is that, from MY point of view, those case are not so rare. The subset of UEFI you're wrapping is too small for me. Please, accept that.

Yes, we can! Here are the list of pseudo device files under POSIX-UEFI:
/dev/stdin - returns EFI_SIMPLE_TEXT_INPUT_PROTOCOL instance (ST->ConIn)
/dev/stdout - returns EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL instance (ST->ConOut)
/dev/stderr - returns EFI_SIMPLE_TEXT_OUTPUT_PROTOCOL instance (ST->StdErr)
/dev/serial - returns EFI_SERIAL_IO_PROTOCOL instance
/dev/diskN - returns EFI_BLOCK_IO_PROTOCOL instance
...anything else - returns EFI_SIMPLE_FILE_SYSTEM_PROTOCOL instance

That's great, actually! Do you have also /dev/diskNp0, /dev/diskNp1 etc. ?

We'd need maybe a different mechanism to enumerate all the disks and partitions. The same applies for video modes, network devices and other fancy UEFI features.

No, we don't. You're mixing up interface with implementation. Two totally different implementations might have the same interface. A good example fprintf. That's the best thing in UNIX and POSIX, everything is a file.

No, I'm not mixing anything. In UNIX everything is a file because we have kernel. We have /proc, /sys, /dev etc. That's why "everything is a file". If you don't have those special file systems, not everything can be a file.

Never intended to by strictly POSIX-compliant. That's not possible without a kernel.

OK, that's good to hear.

The point is, use interfaces that look like POSIX so that people don't have to learn and s*ck with that horrible GUID interface and learn nearly a hundred of helper functions from the supporting library (which is constantly changing btw). Instead programmers can use a few functions that they are already familiar with, so there's no need to learn everything from the gourd up.

That's exactly what I liked about your idea! The only thing that I've been trying to tell you all the time that you might need to add more functions, enumeration mechanisms etc. It still will be easier/more familiar than UEFI. I like the UNIX philosophy: simple problems have simple solutions; when you need something more advanced, you use a more advanced interface. That's why, your project is a good starting point, in my opinion.

One example, which is more obvious to you? Which one should an UEFI newbie choose?

a) find the system table, find boot services in it, and LocateProtocol method in it. Then use that method with some magic numbers to return a struct which in turn must be defined exactly the same way in your application as in the firmware, and which should contain a method called GetRNG() which has parameters that you have to learn

- or -

b) call the well-known rand() that you're already familiar with to do the same thing.

The newbie will choose option b), I totally agree. I believe that the whole discussion is about the fact that you're interested only in that case: the minimum set of functions for a trivial bootloader written (maybe) by a newbie. The case I believe instead is worth exploring as well is the general one: people with ANY degree of experience. Newbies will use the simple part of the interface and will do simple things. Experts will use everything and will be pleased to use a "uniform" interface instead of mixing UEFI and POSIX-like code.

Still, being YOUR project, obviously you can drive it in any direction you prefer. Nobody can ultimately force you to change your mind. The only thing other people like me can do is to offer some perspective and ideas. Your project seemed to me that can be evolved beyond a nice "helper tool for newbies" and that's why I've spent already a lot of time discussing with you. But, I don't plan continuing any further, unless you really ask for that; I believe I've done my best to share good ideas.

[nexos]

Sorry, what is "GOP" ?

The Grahpics Output Protocol. It is the equivalent of VBE on UEFI.

I've been using only GNU-EFI, all the time. The EDK2 bloat is not worth to me either. But, I have to point out that GNU-EFI does not need uefi_call_wrapper: newer versions of GCC support directly the special MS-ABI attribute, but it works with older versions of GCC as well by using -maccumulate-outgoing-args. I've personally done it. uefi_call_wrapper() is needed only if you don't want or cannot use the MS ABI

I find GNU-EFI to be one of the greatest hacks in history. I prefer using neither GNU-EFI or EDK2. I use MinGW with the EFI headers, and access everything natively.

[vvaltchev]

Sorry, what is "GOP" ?

The Grahpics Output Protocol. It is the equivalent of VBE on UEFI.

Ah, thanks.

I find GNU-EFI to be one of the greatest hacks in history. I prefer using neither GNU-EFI or EDK2. I use MinGW with the EFI headers, and access everything natively.

There's nothing really hacky with GNU-EFI. The problem is that gcc-linux-x86_64 cannot generate PE shared libraries (DLLs), while it supports MS_ABI etc.
We need the "objcopy" hack only because of the binary output format. GNU-EFI is not guilty for that. Any library will have the same problem, because the problem is in the toolchain, not in the library itself. I don't understand why this library gets attacked for that. With MinGW, you just skip the "objcopy hack", that's it because you can natively generate a DLL. That's it. There's nothing more "native" in the instructions, they're the same. What changes is the "packaging". Also, I'll repeat again: GNU-EFI does NOT need uefi_call_wrapper(). Just build with -DGNU_EFI_USE_MS_ABI. It feel that I already had the same discussion here.

[PeterX]

There's nothing really hacky with GNU-EFI. The problem is that gcc-linux-x86_64 cannot generate PE shared libraries (DLLs), while it supports MS_ABI etc.
We need the "objcopy" hack only because of the binary output format. GNU-EFI is not guilty for that. Any library will have the same problem, because the problem is in the toolchain, not in the library itself. I don't understand why this library gets attacked for that. With MinGW, you just skip the "objcopy hack", that's it because you can natively generate a DLL. That's it. There's nothing more "native" in the instructions, they're the same. What changes is the "packaging". Also, I'll repeat again: GNU-EFI does NOT need uefi_call_wrapper(). Just build with -DGNU_EFI_USE_MS_ABI. It feel that I already had the same discussion here.

Yes, you are right. But when we speak of GNU-EFI we implicitely mean also the toolchain, the cross-gcc mentioned by you. Mingw, Clang or similar don't need objcopy and the call wrapper. I currently use x86_64-w64-mingw32-gcc with the gnu-efi header files and it (finally) works. Again: You are right.

Greetings
Peter

[vvaltchev]

Yes, you are right. But when we speak of GNU-EFI we implicitely mean also the toolchain, the cross-gcc mentioned by you. Mingw, Clang or similar don't need objcopy and the call wrapper. I currently use x86_64-w64-mingw32-gcc with the gnu-efi header files and it (finally) works. Again: You are right.

Thanks, Peter

[bzt]

If UEFI itself can't access paths inside a partition, then POSIX-UEFI can't either.

UEFI can open files at any given path inside FAT16/FAT32 partitions and you can choose which partition too, even if the interface for that is not so nice.

So can POSIX-UEFI. Everything that's supported by UEFI, also supported by POSIX-UEFI (because it just passes the paths unchanged to the underlying protocol).

For example, I wrote a UEFI bootloader that take advantage of many UEFI features that are not covered.

Please forgive me if I take that with a grain of salt. Someone who wrote an UEFI bootloader that takes advantage of many UEFI features surely would know what GOP stands for. No offense. Could you name the interfaces you're missing?

I don't agree. The mapping is only partial. I've tried explaining this, no point in insisting further.

Let's just agreed on that GNU-EFI doesn't provide nice little functions calls for Block IO, Serial IO, nor for RNG. The fact is, POSIX-UEFI already wraps more protocols than the GNU-EFI library ever has.

You say that POSIX-UEFI doesn't cover enough interfaces, but your argument is invalid because GNU-EFI covers far less. Seriously, take a look at libefi.a's source. Mostly GUID variables, a few functions to get the command line arguments and cover up simple file system. That's all.

The problem is that, from MY point of view, those case are not so rare. The subset of UEFI you're wrapping is too small for me. Please, accept that.

I will, as soon as you name at least one interface you're missing. You are just saying you need so much things, but so far, you failed to even name one. We have clearified that partitions and block devices are not missing. I cannot do anything with you complaining if you don't say exactly what you're missing.

That's great, actually! Do you have also /dev/diskNp0, /dev/diskNp1 etc. ?

No. Why would anybody need those when UEFI provides mappings for the partitions natively? FS0:, FS1: etc. You said it yourself: "UEFI can open files at any given path inside FAT16/FAT32 partitions and you can choose which partition too".

No, I'm not mixing anything. In UNIX everything is a file because we have kernel. We have /proc, /sys, /dev etc. That's why "everything is a file". If you don't have those special file systems, not everything can be a file.

Yes, you are and really everything can be a file. Plan9 pushed that to the extreme. BTW what do you think, how does POSIX-UEFI masquare all those interfaces as a sinlge FILE* stream without a kernel? Because it's possible, I assure you.

The only thing that I've been trying to tell you all the time that you might need to add more functions

For what? I'm asking you from the beginning to please name exact interfaces you lack! I simply still don't know what you're missing.

Any library will have the same problem, because the problem is in the toolchain, not in the library itself.

And that's why POSIX-UEFI isn't just a library, but a build environment configurator as well. It ships a know-it-all Makefile for a good reason.

Literally the first sentence in the README:

Code: Select all

This is a very small build environment that helps you to develop for UEFI under Linux (and other POSIX systems).

And then in section "Distributing as Source"

Code: Select all

This is the preferred way, as it also provides a Makefile to set up your toolchain properly.

If you have an idea how to make this more obvious, I'm listening.

Cheers,
bzt

[vvaltchev]

Look, bzt. I'm not here to fight with people. I have been more than nice with you all the time, trying just to make some constructive criticism, while explicitly appreciating your idea, overall.

You obviously don't accept that and you don't really try to understand what am I saying. Now, what do you really want? To make me go through all my work and explain all the UEFI interfaces I've used and why? Sorry, forget about that. Doing that FOR YOU, with that attitude, it's simply not worth the effort: I've already made a few examples and that counted nothing for you. I'd probably have to explain in DETAILS why I needed to use each UEFI feature and spend days in this discussion. No, thank you. You don't have the right attitude to listen, for me to spend energies trying to teach something to you. It's totally non-sense. I have no point in fighting such a battle, really.

So, keep your believes and ideas and live a happy life. Sorry for commenting in this post at all. Apparently, I haven't understood yet that most people are never looking for a honest feedback with ideas, suggestions etc. People like you are just looking for a feedback like: "cool project, man!".

All the best

[nexos]

I really don't want to join the argument, so, just one thing to be said.
There are plenty of interfaces not covered by POSIX. No GOP wrapper. No memory wrapper. No SMP wrapper. No device detection. The list goes on. For this reason, you could, say, create a file /dev/gop, and use a ioctl or something to control GOP access. Accessing the UEFI interface alongside POSIX would be quite inconsistent.

[PeterX]

Accessing the UEFI interface alongside POSIX would be quite inconsistent.

Just my opinion: That would not be inconsistent, but rather perfectly ok. There are many situations where a lowlevel access is made beside a rather highlevel API. Be it a kernel in mainly C and Asm drivers. Or be it a game which uses sys/lib-calls for accessing files and doing graphics output "directly".

Greetings
Peter

[bzt]

Look, bzt. I'm not here to fight with people.

But it surely looks like it.

trying just to make some constructive criticism

Not by a long shot. A constructive criticism would be if you'd say "I miss the XY interface" because I use it for Z, and it would be great to have it. Then I could reply with one of the following three answers:
a) it is already covered, read the README (as it happened with your block IO example)
b) I'll find a way to cover that with POSIX API and I'll add XY interface to POSIX-UEFI
c) that interface is a niche or would require to add too much complexity, so wouldn't worth the effort.

I can't read minds, without knowing what you miss, I can't give you a precise answer, only vague ones like "neither GNU-EFI nor the EDK2 provides library for the GOP".

You obviously don't accept that and you don't really try to understand what am I saying.

That's because you aren't saying anything, only that "I miss a lot", and "I don't know, but this isn't enough", which is far from being constructive. Actually reminds me of my boy at the age of 5 when he was crying because he didn't know what he wanted. Until you tell me exactly what interface you're missing, so that we can discuss that interface in particular, there's really not much point in continuing this conversation with you.

Just my opinion: That would not be inconsistent, but rather perfectly ok.

Thanks, at least someone gets the point. It's much better if you can use a friendly API in 90% of the time and fallback to protocol interfaces in 10% than having to write everything using the ugly API all the time.

Cheers,
bzt

[nexos]

bzt: I get the point perfectly well and I think its a great project for some people. Me and @vvaltchev are just trying to give feedback and suggest improvements. If you don't like them, that's okay! Just please don't begin flaming simply because we have different opinions about this!

[nullplan]

Me and @vvaltchev are just trying to give feedback and suggest improvements. If you don't like them, that's okay! Just please don't begin flaming simply because we have different opinions about this!

But you did not! Jesus Christ! You did not suggest improvements, not concrete ones anyway, and anytime you were asked to manifest your suggestions in a more tangible way you shrunk away from the challenge and told bzt to calm down. As did vvaltchev. And then you tried to paint him as the toxic one. I can understand bzt's exasperation with you as that is precisely what I'm feeling as I read through the thread.

Again, and as simple as possible: What features are you missing? It is not enough to state that features are missing, you have to actually name them.

[vvaltchev]

It is not enough to state that features are missing, you have to actually name them.

Last attempt. The UEFI Spec 2.8B is ~2,500 pages long. Most of the UEFI features do not have a wrapper in that library. It doesn't make sense trying to list everything that is missing, even if both @nexos and I still tried to mention a few things, multiple times. Please check all the posts, again.

My feeling is that you guys are not really interested in a list of features. You (or just bzt) probably would like to question why I or everyone else would need every single XYZ feature in that list. Bzt implicitly (or explicitly) stated all the time that "you don't need anything else in a bootloader than what my library offers, 99% of the time". That's the problem. It's not about which features are missing, it's about the claim that most of the important features are already there. That's simply wrong and shows a very limited understanding of what UEFI offers. I cannot in this context write an essay explaining what UEFI offers, and what a real-world bootloader needs. The topic is too wide, so that's absurd. It's almost like explaining what's the job of a kernel. But I also won't try, because of way it has been asked to me (read below).

But that's not really my point here. The point here is that this should not become a fight. A guy announces a project and people share some feedback, in a nice and polite way. The project's owner is supposed to take some notes from that feedback. Eventually, she/he can ask for more details, but should never attack people for their opinion. In my case, I understood what bzt wanted to do from my 2nd comment, after this statement:

You could, but that's not what POSIX-UEFI is about. It does not want to redesign the interface, it just ships a small and sane POSIX build environment configurator for the host OS and provides simple and easy to use POSIX-like wrappers for the application. That's all.

So, I wrote:

I understand that now. I just got inspired by your idea and I started thinking about the more general idea of wrapping UEFI with a different interface.

and I ended the message with:

Finally, the 3rd approach, yours, is to just offer some POSIX functions without pretending to cover the whole UEFI.
I get that, just I wanted to offer some perspective about the other approaches and maybe make you at least consider the idea of not being strictly POSIX-compliant. You could offer in your library a set of functions designed for UEFI which are just inspired by POSIX, not strictly following it.

I expected to end the discussion there. Unfortunately, it didn't. Bzt couldn't just accept my different opinion as it is. He got "heated up" and started attacking me. That's a very bad and counter-productive attitude when you want to advertise your project. Because of that, the whole question about which features are missing doesn't feel genuine. It feels more like a way to challenge me, just to trying proving me wrong. That's pointless, I won't play that game. I don't need to prove anything. When it's obvious to me that the other person is way TOO FAR from my point of view to even BEGIN listening, it's better to just stop talking.

The only reason I've felt like writing this comment is that @nexos has been attacked because of me and, apparently, to some people like you, the whole discussion felt way different than I expected. I had to clarify better my position.

Said that, I'm not really angry at bzt or anybody else. Let's be civil. This forum is a unique place for having amazing discussions. I just hope that the amount of "stars" near our nicknames won't bias too much our perception about the experience of the other person.

Vlad

[eekee]

Um... smol question: Why is this called POSIX-UEFI? POSIX refers to specific standards which this project doesn't pretend to stick to, but Unix (these days) refers to the whole extended family of operating systems with this sort of interface without claiming compatibility with any specific POSIX standards. I just feel using POSIX for projects like this weakens the term when there's no need to do so.

[bzt]

But you did not! Jesus Christ! You did not suggest improvements, not concrete ones anyway, and anytime you were asked to manifest your suggestions in a more tangible way you shrunk away from the challenge and told bzt to calm down. As did vvaltchev.

Thanks!

Last attempt. The UEFI Spec 2.8B is ~2,500 pages long. Most of the UEFI features do not have a wrapper in that library.

Last attempt. Most of the UEFI features do not have a wrapper in GNU-EFI either. But I've already pointed this out to you. It is unreasonable to expect something from this library that neither GNU-EFI, nor the official EDK2 library can't provide.

Um... smol question: Why is this called POSIX-UEFI?

For two reasons:

First, it provides a very small, easy to use build environment on POSIX systems, like Linux. You just include its Makefile and that detects the toolchain and sets it up correctly (GNU / LLVM with all the nasty flags) to produce UEFI PE executable and provides a suitable crt0 for them. (Unlike GNU-EFI, which does not compile with CLang for example, and unlike EDK2 which is a huge bloated build environment, lot more difficult to use than including a Makefile.)

Second, it provides a POSIXish library (and a single header with ANSI C standard definitions) for the application, nearly identical to the POSIX standard, with only consistent differences. This ensures a much much much shorter learning curve than GNU-EFI or EDK2. Most notable difference is, that it must use wchar_t, not char, otherwise all the function names and arguments are the same, so for example you don't have to think about what opening a file is called, you can be sure it's "fopen", and that it has two arguments, the first being the file's name, the second the opening mode. Similarly, if you want to copy memory, you don't have to think about what funky name EDK2 gave to that function, or what's the ordering of the parameters, you can just use memcpy() as usual. This makes development for UEFI under Linux (and other POSIX systems) easier than ever before. Yes, it is not 100% POSIX, that never was a goal, making it similar so that developers can shorten the learning curve was.

Cheers,
bzt