i want to learn how to make an os in pure assembly

[nexos]

But the mem* functions can be like 10 lines of code each. My main beef with UEFI is its use of Unicode, which leads to some interesting things when a EFI loader shares the same codebase as a BIOS one

[Octocontrabass]

Maybe there's a switch in CLang that simply tells it to always use UEFI equivalents of these builtins, but I couldn't find this switch, that's for sure.

Are you sure you want that? SetMem() and CopyMem() are boot services, which means you must track them down in the boot services table before you can call them, and they're only valid before calling ExitBootServices(). ZeroMem() isn't part of UEFI at all, it's from a library provided by EDK II.

[kzinti]

(deleted)

[zaval]

It took me quite a while to figure out all the necessary CLang and lld switches. The most problematic part is, UEFI has renamed all the standard C functions

It's not C library functions. like at all, bzt.

like there's no memset, memcmp, memcpy etc. you have ZeroMem, CompareMem, CopyMem etc.

exactly, there is no memset, memcmp, memcpy etc. there is no standard C library in the UEFI environment.

but mem* still might be inserted into the binary by any C compiler even if the source doesn't reference them directly (depending on the code and optimization level of course). When such a builtin gets inlined, then no probs, you won't even notice. But under some circumstances sometimes only a libc call is emitted (even in the freestanding mode), and bam, you have a big problem, lots of linking errors.

and how this is a problem of UEFI or a proof, that it has been made with only MSVC in mind? if your compiler is so dumb, that still makes references to a library, it was explicitly commanded to not to, then this means your compiler is dumb and it failed to be "freestanding" even after being asked to be. there is no any "libc" in UEFI.

Yes, once you've figured out all the necessary switches, it's easy, but all problems seem easy once you know the solution... Maybe there's a switch in CLang that simply tells it to always use UEFI equivalents of these builtins, but I couldn't find this switch, that's for sure.

there should not be any "builtins" in this case, how is it that hard to get? you just call CopyMem() Boot Service as any other Boot Service, why on earth one would start to talk nonsense about gcc builtins?!! All the sane compiler should do here is just emit an indirect call to this function, PROVIDED by UEFI.

anyway, this statement of this user:

Correction: UEFI was written with purely MSVC in mind, not C in general

is one of those cases, where there is a need to put a huge @$$ font banner in red, suggested by the very same user, ironically, as a follow up to the statement, to not confuse newcomers, like this:
EXTREME_BULLSH1T!!!

I understand, that staring at CamelCased function names of UEFI, a MS crony Intel decorated UEFI in, causes bzt to feel non-stoppable overheat at his bottom parts, YET this is not an excuse to emit such WRONG and CONFUSING statements.

And of course, writing UEFI in assembly isn't any harder, than for BIOS. you just need to read carefully about calling conventions used and follow that. the why question is another one. but it's about personal preferences. it's doable, do it, if you want. there might be a few of tutorials, but again, remember, there is not much specific about that programming compared to the BIOS case or even assembly programming in the OS environment. read the UEFI spec and if you know the assembly in question, you are ready to go.



PS. personally, just for the sake of completeness, I compiled my UEFI OS Loader with clang too. For ARM32 and ARM64[1]. And it worked. heck, I even compiled the code for RISCV64! Just need to finish my ELF->PE conversion utility to try it out! yep, clang can't make PE targets for RISCV.

1 - There was one problem with 32 bit ARM, because clang linkers do not set proper MachineType for UEFI targets (which is 1C2), despite generate proper code - UEFI allows OSL to be compiled in Thumb code, but expects it to be "interworking" aware, that is, be aware, that UEFI services itself may be in ARM code. given, all the calls to UEFI go through the function pointer tables, the problem resolves itself, - 32 bit ARM's indirect call (through register) always takes care about changing the CPU mode - blx <reg> (branch, link and exchange (the mode)). but the linker in clang doesn't care about UEFI and supports only what's needed for Windows and because the latter is Thumb only, it sets the MachineType in 1C4 (Thumb only). Whereas it should be 1C2 (ARM or Thumb). MSVC's linker recognizes EFI_* subsystems and sets the subsystem properly, clang doesn't. irony, eh? seems, they at clang neglected UEFI completely. isn't it weird? like it's what nobody until now ever tried to use clang for UEFI OSL for 32 bit ARM? hardly so, so what's the reason of such a weird thing? well, the answer is probably that, all those who used clang for UEFI did it targetting ELF and then making manually ELF to PE conversion, with edk incl! either through some scripts or ugly trickery with headers. and noone even thought, wait the minute, why can't we try the already available PE targetting? ... anyway, a little patch for clang's linker is needed. or hexediting the resulting image. what I did. and it ran.

[nullplan]

I understand, that staring at CamelCased function names of UEFI, a MS crony Intel decorated UEFI in, causes bzt to feel non-stoppable overheat at his bottom parts, YET this is not an excuse to emit such WRONG and CONFUSING statements.

I find it funny that you are accusing bzt of overheating, yet you are the one with the most emotional reply yet. Maybe some introspection is in order.

I do find it fair to say that UEFI was designed with MSVC in mind (that's why the binary format is PE and the calling conventions are the MS ones, as that is the only thing MSVC is capable of generating, at least to my knowledge), and that clang works is just a happy little accident. Without definitive data, this is merely speculation, of course, but you have also offered no counter argument.

As for the mem functions... just write them yourself? It is not hard. That way you will not have to deal with boot services no longer being available after the exit.

[bzt]

But the mem* functions can be like 10 lines of code each.

Yes, but first you have to figure out why your Clang compiled code has those mem* calls in the first place and why are they causing linking errors. As I've said, all problems seem simple once you know the solution.

My main beef with UEFI is its use of Unicode, which leads to some interesting things when a EFI loader shares the same codebase as a BIOS one

Yep, I've workarounded that by using the "-fshort-wchar -Wno-builtin-declaration-mismatch -Wno-builtin-requires-header -Wno-incompatible-library-redeclaration" flags and defining those functions myself. Again, simple once you have figured out the necessary steps, but not trivial at all when you first bump into the problem.

As for the mem functions... just write them yourself? It is not hard. That way you will not have to deal with boot services no longer being available after the exit.

That's exactly what I did. Here mem* function prototypes are libc standard, but the prototypes of strcpy, strcat, strlen etc. use a "char_t" typedef, which might be resolved as
- "wchar_t" (normal UEFI-mode, requires some special CLang flags to work) or to
- "char" (with the UTF-8 wrapper added, and in this case no special CLang flags needed but instead you need an mbstowcs implementation for the conversion).

Cheers,
bzt

[Octocontrabass]

-fshort-wchar

You don't need to pass this to a cross-compiler; it's the default for UEFI (Windows) targets. You should be using a cross-compiler.

-Wno-builtin-declaration-mismatch -Wno-builtin-requires-header -Wno-incompatible-library-redeclaration

Why aren't you using "-ffreestanding"?

[bzt]

-fshort-wchar

You don't need to pass this to a cross-compiler; it's the default for UEFI (Windows) targets. You should be using a cross-compiler.

Yes, you need this, otherwise CLang and gcc uses 4 byte characters. At least that was the case not long ago, maybe they have fixed that. Obviously you can't use CLang in non-cross-compiler mode, because CLang is per definition a cross-compiler.

-Wno-builtin-declaration-mismatch -Wno-builtin-requires-header -Wno-incompatible-library-redeclaration

Why aren't you using "-ffreestanding"?

Stupid question. I am using -ffreestanding obviously. But these flags have absolutely nothing to do with freestanding mode. Both GNU gcc and CLang have builtins for many libc functions, strlen for example. These flags are required so that the compiler won't use its builtin versions, and that it won't report an error if you define strlen as "int strlen(wchar_t *str)" for example. You can't let the compilers to use their builtins (which is the default for freestanding mode too) because many of those builtins won't work in an UEFI environment. (Which leads us back to the problem where memcpy won't be inlined, rather a function call will be emitted, so you must implement mem* functions to avoid linking errors.)

Cheers,
bzt

[Octocontrabass]

Yes, you need this, otherwise CLang and gcc uses 4 byte characters. At least that was the case not long ago, maybe they have fixed that.

I've never seen that bug, so it must have been fixed a long time ago. I thought you didn't care about old compilers.

These flags are required so that the compiler won't use its builtin versions, and that it won't report an error if you define strlen as "int strlen(wchar_t *str)" for example.

Those things are already handled by "-ffreestanding".

[bzt]

I've never seen that bug, so it must have been fixed a long time ago.

So, let me get this straight, the bug doesn't exists, because you, Octocontrabass personally haven't seen it? Dude, you can't be serious... Just do a google, you'll find hundreds who have.

I thought you didn't care about old compilers.

And I thought you're a sane person. I wrote the Makefile 5 months ago, at the time those weren't "old" compilers, but the latest ones. Both CLang and gcc got patches in the last 5 months (I've run into one issue which was fixed in gcc in this time period), and that's most definitely not "long time ago". Not several years old like your compilers.

Those things are already handled by "-ffreestanding".

Citation needed! By checking the source of Clang I'm absolutely sure they aren't. The "-ffreestanding" only implies "-fno-builtin", but not the others, furthermore "-fno-builtin" only means do not inline builtin functions, a substituting call still can be emitted (with the standard prototype), see here, and here, and here, and here, and here, and here... man, just use your google-kung-fu, the internet is literally full of builtin related compiler bugs.

And independently to what you say, I've checked and those flags are needed (I can't even compile without "-Wno-incompatible-library-redeclaration" if I don't use the UTF8 -wrapper, because using wchar_t is an MSVC thing definitely not libc standard). I've updated my compilers about 2 months ago and my distro ships relatively recent versions (so they aren't that old), but I'll check again with the most cutting-edge versions too.

Cheers,
bzt

[Octocontrabass]

Just do a google, you'll find hundreds who have.

I found zero. Please show me the examples you found.

The "-ffreestanding" only implies "-fno-builtin", but not the others, furthermore "-fno-builtin" only means do not inline builtin functions, a substituting call still can be emitted (with the standard prototype), see here, and here, and here, and here, and here, and here... man, just use your google-kung-fu, the internet is literally full of builtin related compiler bugs.

How are any of those links related? The __builtin_[C standard library] functions require a C standard library, that's pretty clearly documented. If you use __builtin_strlen and you have an incompatible declaration of strlen, that's a bug in your code, not a bug in the compiler.

(I can't even compile without "-Wno-incompatible-library-redeclaration" if I don't use the UTF8 -wrapper, because using wchar_t is an MSVC thing definitely not libc standard).

You know that option only disables the warning, right? It doesn't change the generated code at all.

[Combuster]

because using wchar_t is an MSVC thing definitely not libc standard

That's sadly factually false:

The C and C++ standard libraries include a number of facilities for dealing with wide characters and strings composed of them. The wide characters are defined using datatype wchar_t, which in the original C90 standard was defined as

"an integral type whose range of values can represent distinct codes for all members of the largest extended character set specified among the supported locales" (ISO 9899:1990 §4.1.5)

Regarding this and related remarks, please be more careful designating or interpreting political opinion on visual studio as fact.

[bzt]

Just do a google, you'll find hundreds who have.

I found zero. Please show me the examples you found.

I did show you examples with links, actually. Check out my previous post.

because using wchar_t is an MSVC thing definitely not libc standard

That's sadly factually false:

No, it's not. Show me where does the C standard allow defining strcpy with wchar_t for example. It does not allow that, that's the very reason why the "-Wno-incompatible-library-redeclaration" CLang flag exists in the first place.

The C and C++ standard libraries include a number of facilities for dealing with wide characters and strings composed of them.

Which is completely irrelevant, as just because mbtowc and wctomb facilities exists, strcmp for example won't work with wchar_t.

Regarding this and related remarks, please be more careful designating or interpreting political opinion on visual studio as fact.

You are the one making a political opinion. All I've said was strictly technical, no politics at all.

Cheers,
bzt

[nullplan]

No, it's not. Show me where does the C standard allow defining strcpy with wchar_t for example.

bzt, you claimed wchar_t was an MSVC thing. It is objectively not. No, the C standard does not allow defining strcpy with wchar_t, but then that was neither MSVC's nor UEFI's doing, it was yours, wasn't it? In the C standard the function you are looking for is "wcscpy()".

It does not allow that, that's the very reason why the "-Wno-incompatible-library-redeclaration" CLang flag exists in the first place.

My car made a weird noise so I instructed the mechanics to put in sound deadening in the engine compartment so I don't have to listen to it.

The flag you reference is a warning flag. Specifically, it disables a warning. It does not do anything about the code generation. Now granted, there are some stupid warnings in compilers these days (GCC has a warning about snprintf() possibly truncating the output. Yes, that is the point.) but sometimes these warnings actually mean something. In this case they mean you missed -ffreestanding. (Yes, I just checked. Redefining strcpy() with wchar_t goes without warning as long as -ffreestanding is specified).

Which is completely irrelevant, as just because mbtowc and wctomb facilities exists, strcmp for example won't work with wchar_t.

No, but wcscmp() will. You can't use a hammer to drive a screw into the wall, so you can now either blame the manufacturer of the screws or just start using a bloody screw driver.

You are the one making a political opinion.

Where? Octo only quoted from the standard. I fail to see anything political in there, except perhaps his position that unnecessary politicization of technical topics is to be avoided.

All I've said was strictly technical, no politics at all.

Did you? I seem to remember you saying

Correction: UEFI was written with purely MSVC in mind, not C in general

Which is not technical in the slightest, unless you had authority to back that up. But you haven't presented any, yet. Unless you were in the room when those topics were discussed, this is pure speculation, borne chiefly out of your political opinions about the parties involved. Mind you, I agree that this is probably the case, but I don't hide my political view on the matter behind a veneer of technicality.

[Octocontrabass]

I did show you examples with links, actually. Check out my previous post.

I'm asking specifically for examples of GCC or Clang mistakenly using a 32-bit wchar_t when targeting the Windows ABI.

Show me where does the C standard allow defining strcpy with wchar_t for example.

Right here:

In a freestanding environment (in which C program execution may take place without any benefit of an operating system), the name and type of the function called at program startup are implementation-defined. Any library facilities available to a freestanding program, other than the minimal set required by clause 4, are implementation-defined.

The wording changes a bit depending on the draft you're looking at, but all of them say it's implementation-defined. GCC and Clang make no assumptions about C standard library functions in freestanding mode, aside from the four mem* functions, so you're free to redefine strcpy however you like.