article on C compiler and standard lib C related to osdev

[gerryg400]

there are still calling convention that are recognized by many if not all compilers, like __cdecl, __stdcall, __fastcall, and the cpu also has standard way to make calls

Recognised ? Maybe. But to then assume that that means binary compatibility is possibly dangerous. These calling conventions may be recognised by some compilers (remember that they are x86 32bit specific and so only apply to a very small subset of compilers) but the compilers aren't identical in their interpretation of the keywords.

According to wikipedia, fastcall on Borland's compiler is different from fastcall on Microsoft's compiler. Watcom's compiler ignores the fastcall keyword and implements its own fast call semantics. Even cdecl doesn't get treated the same by Microsoft and Borland compilers wrt some return types.

GCC does try to provide compatibility modes for the various compiler standards but it doesn't guarantee compliance.

Also, there are other things to consider like for example assumed stack alignment on function entry.

There is an article by Agner Fog somewhere about this I think.

[h0bby1]

there are still calling convention that are recognized by many if not all compilers, like __cdecl, __stdcall, __fastcall, and the cpu also has standard way to make calls

Recognised ? Maybe. But to then assume that that means binary compatibility is possibly dangerous. These calling conventions may be recognised by some compilers (remember that they are x86 32bit specific and so only apply to a very small subset of compilers) but the compilers aren't identical in their interpretation of the keywords.

According to wikipedia, fastcall on Borland's compiler is different from fastcall on Microsoft's compiler. Watcom's compiler ignores the fastcall keyword and implements its own fast call semantics. Even cdecl doesn't get treated the same by Microsoft and Borland compilers wrt some return types.

GCC does try to provide compatibility modes for the various compiler standards but it doesn't guarantee compliance.

Also, there are other things to consider like for example assumed stack alignment on function entry.

There is an article by Agner Fog somewhere about this I think.

Yes some of the convention , including also c++ method call can differ, fastcall is not always super compatible, but cdecl should be supported by most compiler for intel.If not then it cannot be used to build an executable for the platform. Any decent C compiler should at least support either stdcall or cdecl in a standard manner.

_stdcall is supposed to be older, more compatible, and even faster/lighter, so i guess for 100% compatiability, stdcall should be used

__cdecl is somehow safer if dealing with imported function, because most abi don't have a way to define the size of the arguments in the exported function definition, so if the caller layout and size of the argument on the stack is not what the function expect, with __cdecl as the caller is responsible for clearing the stack from the argument , it can still avoid some fatal error, argument will still be passed incorrectly, but the stack will be restored properly after the call with a __cdecl

With an stdcall, it's more intolerant to error as the stack is restored by the called function and the called function have no real way to know how the caller passed argument, but __stdcall (as the name indicate :p) is supposed to be more supported by even older/simpler compilers

But it's what i mention at the beginning of the article, that compiler might treat calling convention or other things differently, and using function attribute is still the way to make it more possibly compatible than if not using any. But i'm well aware it may not be possible in all case to achieve the exact behavior wanted from any compiler, hence also why libc and compiler are not all that independent from each other in general.

Nothing should be assumed (like stack aligment) that is not explicitly specified for the compiler used. there are way to make cross compilers stack alignment with macros.


like

Code: Select all

typedef unsigned long mem_ofset;
typedef void *mem_ptr;

#define declare_aligned_stack_16 (a,size) unsigned char __a[size+16]; mem_ptr a=((mem_ofset)(__a)&0xFFFFFFF0+16)

void my_func()
{
     declare_aligned_stack_16 (buffer,64);
}

(this would make warnings and not 64 bit compatible but it's the idea)

or specific code can be made in assembler to align the stack that has to be used instead of compiler specific functions

like

Code: Select all

_align_stack:
mov eax,[esp]



test esp,0x000000F
jz already_aligned
  and esp,0xFFFFFF0
  sub esp,16
already_aligned:


sub esp,4
mov [esp],eax

xor eax,eax
ret

need to restore the stack frame properly before returning from the calling function, and will mess up the address of stack variable in the function that are defined before the call, the compiler can take the specified alignment into account when he compile the code, so there are always many things that are more convenient when using compiler specific things, it's why they are here, but it also can create problem if you want to have code that can be safely compiled with different compilers on different systems

I have read all article from agner frog, they are great, and they talk about many of those issues, i'll probably put some quote from it in the article at some point. But agner frog more have optimization in mind than producing code that is compatible with all compilers. but he issues some of the things compilers might do and their relation with the C library.

[h0bby1]

you still can't install gcc on a plateform and using it to compile application that use the C library without installing those headers

h0bby1, which headers do you mean by 'those headers' ?

the headers of the C library, stddef.h, stdlib.h, string.h , stdio.h, that are in the package libc-dev on linux

[bluemoon]

I suggest you to actually read other's response and reasons, and double check the materials before defending your claims.
For example the calling convention issue there are tons of references, you can't just guess/claim it generally work, then restrict your scope to intel only, then perhaps further restrict the scope down to only the tools you used.

This is my kind advice.

[h0bby1]

I edited it slightly to change the term used from 'compiler specific' to 'build environment' specific, it's true it's more accurate, as technically, the same compiler can be used with different C library in different building configuration, and the C library depend more on the 'host' or 'building environment' rather than on the C compiler itself.

But still most if not all compiler need to have a C Library installed to be able to compile most C programs, and often the compiler, the c library, the linker, and other building tools come together in a bundle, and can be consider as part the process to transform a C program into an executable for a particular target.

Then the dependence is on this building chain rather than directly on the C compiler, but the C compiler is still the major component of this process, preprocessors or linker are much simpler and more inter compatible, and the C compiler/Library can be considered as what is specific to the build.

A particular building environment will most likely use a specific version of the C Library and C compiler that can be used to identity the characteristics specific to the build environment that matter to develop cross compiler C library and C programs. it can also include the cpu type, but making cross cpu C code is not covered here, it's assumed to target intel cpu.

I suggest you to actually read other's response and reasons, and double check the materials before defending your claims.
For example the calling convention issue there are tons of references, you can't just guess/claim it generally work, then restrict your scope to intel only, then perhaps further restrict the scope down to only the tools you used.

This is my kind advice.

yes i added some precision when needed when cpu specific things are involved, and i'll try cover some compilers behavior from what i have personally seen in general under linux and windows, and from other materials on different compilers and abi used, normally compiler document the way they handle function attribute, it's not just assumptions, it can need some specific declaration for each compiler based on preprocessor definitions.

Even the platform and cpu should be possible to detect using predefined preprocessor variables and macros , or some preprocessor definition can be required to be defined in the build to define which cpu it target.

Can also use different header files for each different cpu, selected with include path in the build settings. It's very unlikely a system will have to run executable compiled for two different cpu so application can safely use specific header based on cpu architecture, as it's most likely it's the lib made for the target cpu with the same header that will be present at runtime.


i will improve the preprocessor code for detection of compiler/cpu/abi and build configuration in os_def.h . i just put them like this for now, i'll change to the correct syntax and values latter


It's also hard to have an exhaustive list of all possibilities of compilers/libc/host os/target os/target cpu that can be present in general, in the absolute people should check how their compiler handle things for a particular cpu from the documentation, but this can be done also automatically with preproprocessor detections for all configuration of which behavior is known , if the behavior is not supported , need to add the definitions of all compiler specific directives manually in the build configuration.

Using the build environment default C headers won't solve anything, because C headers used could be different from two build.

Defining them explicitly need compiler specific directives to be selected , but it give more control over how the executable will be generated from the C program. Normally compilers document how they handle each attribute and calling conventions

Enough care should be used at least to compile the C Library ,function should be defined in a clear manner that is sure to be implemented correctly by the compiler used to compile it. the C code using the C Library must have a way to be built with compatible call, even if it require to select compiler specific code depending on detection of building configuration.

[h0bby1]

any compiler that can be used to build executable for windows or linux has to support correctly at least the __stdcall , as all system api use __stdcall in their function declaration, opengl, windows, X11 etc

(Actually, windows is worse because it doesn't even attempt versioning. And the smarter people know that so they bundle their own DLLs to waste disk space and hide the problem.)

Windows is maybe worst for direct versioning of binary file, but in the same time, lot of things are handled with COM.

The mecanism it uses make in sort you can instanciate a class implemented in the shared library to a virtual class interface that can be requested based on a version, it just suppose the com framework is compatible but the base COM api that is directly exported from the shared dll is very simple.

From there it doesn't need versioning on the library binary file, but only in the COM interface they export. you don't even have to assume anything about the location of the dll, it's registered in the registry base with an interface ID and a path on the hard drive, and any application requesting the version 1.3 of the interface ID 45454 can just use com to instanciate the class from any dll anywhere on the system that export this interface. Just need to know the id of the class and watching some doc to find out which system have which version of which interface installed by default. Normally system are retro compatible, so most of the time lower version are also implemented even in more recent system. DirectX often have several versions of some interfaces present on the system.

But the C library doesn't use a com like interface at all, and the C Library is always a special case because compiler works closely with them, and they are not exactly considered as a regular shared library in the build process.

[bluemoon]

any compiler that can be used to build executable for windows or linux has to support correctly at least the __stdcall , as all system api use __stdcall in their function declaration, opengl, windows, X11 etc

Not true, on 64-bit linux it use AMD x86_64 ABI for library call (and a custom kernel may use something different). I'm pretty sure Windows use different ABI for 64a and itanium.
(The rest of the article is skipped due to error.)

By the way, I get lost here, are we talking about compiler on those host environment or compiler intended for use to build an OS?

[h0bby1]

any compiler that can be used to build executable for windows or linux has to support correctly at least the __stdcall , as all system api use __stdcall in their function declaration, opengl, windows, X11 etc

Not true, on 64-bit linux it use AMD x86_64 ABI for library call (and a custom kernel may use something different). I'm pretty sure Windows use different ABI for 64a and itanium.
(The rest of the article is skipped due to error.)

yes, but they'll have to implement at least one function call method in a standard and documented manner that can be specified safely as calling convention with functions attribute.

i'm not sure how a compiler would interpret a call to a function declared with __stdcall if he compiles to 64 bit arch, it's something i need to check, but it's specially why specifying calling convention is important, like this, the compiler know how to deal with the function declaration and doesn't assume anything on the function attribute based on the default host configuration. If it doesn't support the requested calling convention, at least it will issue a warning , and different version can easily be made with different function attribute for different architectures.

If targeting 64 arch , arm, mips, ppc, 68k, or anything else, convention call need to be changed depending on the target configuration. And the C library to be recompiled for the specific target as well. As well as all the applications. So in that case it's not even really a problem because application will have to be compiled specifically for each cpu anyway. There is no way a binary application can be transfered from a cpu arch to another like that anyway, it's not only a matter of C library and calling conventions. it's not possible to build exe for different cpu with a with a single compiler/C library.

So yeah, it need a version of the C library for each target cpu, with at least specific convention call, but that's the least of the issue when needing to make C code that will compile without problem on any cpu. But each compiler for any cpu will still have a set of well defined calling convention that can be safely used to declare function prototype that it's rather certain the compiler will recognize and implement correctly. They depend on the cpu and the arch, but each cpu and arch will have at least one that you can more than safely assume that it will be correctly implemented.

If a compiler doesn't support the calling convention used for that particular architecture, it won't be able to produce executable for it, and using another C library in that case won't change much if the compiler doesn't support the calling convention.

gcc support the microsoft 64 bit abi with function attribute

http://gcc.gnu.org/onlinedocs/gcc/Funct ... butes.html

ms_abi/sysv_abi
On 32-bit and 64-bit (i?86|x86_64)-*-* targets, you can use an ABI attribute to indicate which calling convention should be used for a function. The ms_abi attribute tells the compiler to use the Microsoft ABI, while the sysv_abi attribute tells the compiler to use the ABI used on GNU/Linux and other systems. The default is to use the Microsoft ABI when targeting Windows. On all other systems, the default is the x86/AMD ABI.
Note, the ms_abi attribute for Microsoft Windows 64-bit targets currently requires the -maccumulate-outgoing-args option.

Just need to enable the correct function attributes in function declaration for each compiler. And preferably, choosing a calling convention well supported by the largest number of compilers for each cpu model.

By the way, I get lost here, are we talking about compiler on those host environment or compiler intended for use to build an OS?

Compilers on those host environment used to build an OS. And to build applications/Modules/Libraries/Drivers for the OS. And being able to build the kernel with a compiler on an host, the C library with another compiler on another host, and other libraries with other compilers on other hosts.

I don't even understand why 'host' couldn't build executable for different target with a single compiler. An host can use a compiler with a build configuration to produce exe for other system using the same cpu architecture.

I don't understand why it necessarily need a different compiler to build executable that target the same cpu architecture.

Each cpu architecture will have well defined C calling convention, and their own set of registers and functions, and will all have a set of common function in the C Library, but given a particular cpu architecture, i don't really see any reason why any compiler should not be able to produce executable for that system.

[dozniak]

lot of things are handled with COM.

So there needs to be an ABI for COM that has to be supported, right?

[Owen]

any compiler that can be used to build executable for windows or linux has to support correctly at least the __stdcall , as all system api use __stdcall in their function declaration

Very wrong

opengl

Only on Windows

windows

Partially (lots of Win32 is "cdecl", bits are "farcall", lots of C++ is "thiscall") - for i386. AMD64 and IA64 are completely different matters.

X11

Incredibly wrong.

"stdcall" is an invention of Microsoft. It is not standard; quite the contrary. It is i386 specific; for other architectures, there is one calling convention: that defined by the ABI (Exception: Windows for AMD64 and everything else for AMD64 implement different calling conventions - Microsoft have their own, and then there is the SystemV ABI calling convention specified by AMD which is vastly superior)

If you exclude Windows, and instead build a cross compiler for your target platform... you'll find calling convention issues go away, because everybody else defines a single calling convention that they use for everything unless they have very good reason.

And, anyway, Microsoft's compilers default to "cdecl" on i386.

[h0bby1]

lot of things are handled with COM.

So there needs to be an ABI for COM that has to be supported, right?

The base of COM is implemented via a few exported functions using regular dll export, like RegisterDLL/UnregisterDLL, and a few other functions, but the versioning is made via the registry base , the dll has to contain the code to register itself with an ID, and to get an instance of the interface for that particular ID.

But yes there is a minimal set of functions that use the system ABI. But the interface itself is defined independently of the ABI, you can use a COM interface in javascript, visual basic, C# , C++, the interface is defined using a specific language (IDL) and then the class definition for the particular language is generated from it. Only some base COM functions are directly using the regular ABI mecanism.

Application that use a COM interface don't need to be linked against the dll that implement the interface.

[h0bby1]

any compiler that can be used to build executable for windows or linux has to support correctly at least the __stdcall , as all system api use __stdcall in their function declaration

Very wrong

ok , let's reformulate this :

any compiler that can be used to build an executable that has to use a system API has to support correctly at least one calling convention for the cpu architecture it target.

any compiler that can be used to build an executable that will have to call system API on an x86 cpu has to support correctly a standard x86 calling convention.

any compiler that can be used to build an executable that will have to call system API on an x64 cpu has to support correctly a standard x64 calling convention.

opengl

Only on Windows

even on any platform, the compiler has to support a standard calling convention to be able to make use of any function defined in another module.

windows

Partially (lots of Win32 is "cdecl", bits are "farcall", lots of C++ is "thiscall") - for i386. AMD64 and IA64 are completely different matters.

yes, some are declared as cdecl, farcall, stdcall, thiscall for c++, with function attribute, and any compilers that has to build an executable that can call those functions will have to support this calling convention.

X11

Incredibly wrong.

any Library on any system on any cpu has to use some standard calling convention if they want other program to be able to call them. Then you can rely on the host compiler's default calling convention instead of defining the calling convention explicitly, but defining calling convention explicitly will still work if the host compiler's default calling convention is different from what you need.

"stdcall" is an invention of Microsoft. It is not standard; quite the contrary. It is i386 specific; for other architectures, there is one calling convention: that defined by the ABI (Exception: Windows for AMD64 and everything else for AMD64 implement different calling conventions - Microsoft have their own, and then there is the SystemV ABI calling convention specified by AMD which is vastly superior)

stdcall is standard for i386 cpu and most compilers who can compile an exe for i386. Even gcc-i386 recognize them.

For AMD64 there are also standard convention, there is one for SystemV and one specific to microsoft, but they are still standard convention that any compiler that want to produce executable able to call function from other module using this ABI will have to support in a standard manner, even if there are several standard.

In the case microsoft compilers doesn't seem to be able to support any other ABI for amd64 than the microsoft one, but gcc can support the two standard if the calling convention is specified in functions attributes. So even a program compiled with gcc under linux can build exe that use the microsoft standard for amd64 calling convention.

Which mean that if the ms x64 calling convention is specified explicitly in function declaration, an x64 executable can be compiled with gcc under linux that will be compatible with a library compiled under visual studio or any other compiler that support this calling convention.

If you exclude Windows, and instead build a cross compiler for your target platform... you'll find calling convention issues go away, because everybody else defines a single calling convention that they use for everything unless they have very good reason.

And, anyway, Microsoft's compilers default to "cdecl" on i386.

I don't want to exclude windows. Neither i want to exclude any host or compiler.

the default calling convention used can be changed from visual studio, and the default shouldn't matter if the calling convention is specified explicitly in the function declaration with function attributes.

The only case a cross compiler is necessary is when compiling for a different cpu arch, obviously you need a compiler that support the target cpu to compile an exe for it from any host, if this target cpu is different from the cpu from which you build the program, then yes you need a cross compiler, but otherwise there is no need for a cross compiler provided the compiler can produce an exe for the target cpu and support the calling convention used in the function declarations of the api as well as other things that i detail more in the article.

[gerryg400]

Just need to enable the correct function attributes in function declaration for each compiler. And preferably, choosing a calling convention well supported by the largest number of compilers for each cpu model.

Are you implying that it's possible to build an windows program or even a windows object file with a Linux x86-64 compiler ? Are you positive that the Linux compiler is able to produce PE files ?

[gerryg400]

you still can't install gcc on a plateform and using it to compile application that use the C library without installing those headers

h0bby1, which headers do you mean by 'those headers' ?

the headers of the C library, stddef.h, stdlib.h, string.h , stdio.h, that are in the package libc-dev on linux

This is not correct. gcc is completely separate from glibc. gcc does not rely on glibc. I have used gcc at work for many years and have never used glibc.

[Griwes]

the client C program use definition of the functions contained in the standard C header files

s/files/includes/, they are not required to be actual files.

malloc.h
This file contain declaration for functions related to memory allocation, such as malloc, calloc, realloc, and free.

No. `malloc`, `calloc`, `realloc` and `free` are declared in <stdlib.h>, and the C standard does not mention anything called "malloc.h". Again, standard includes are not required to be files.

RTTI (run time type information for c++ virtual classes)

Hardly "the C runtime", also it's part of core language, not of "runtime", whatever that means. There is no such thing as virtual classes in C++. Virtual calls don't rely on RTTI, only exception handling and dynamic_cast rely on RTTI.

The simplest is to declare functions with the same parameters and return type than the ones of the compiler, but with a different name. Using a suffix or prefix in the name of the function declaration is generally convenient, application will have to use this name instead of the standard C function name.

No, it's not the simplest. The simplest is to use what C and C++ standards guarantee to be available in a freestanding environment (minus things you'd want to disable yourself, like exceptions and RTTI in C++) and use a proper compiler targeting your target platform.

Each cpu architecture have their own set of calling convention, most intel compiler will support any of those in a standard manner:

"Standard"? What authority approved the standard you are talking about?

so this declaration in string.h from gcc C Library header file

It's not GCC C library, it's GNU C library.

would become this in build independent os_string.h

...which is retarded, because in sane setups you can just call it <string.h> and `strcpy`, because the compiler wouldn't find host's version anyway.

-----------

I just skimmed through the text on the wiki, and didn't even try to find brokenness of points about calling conventions, to avoid repeating what has been already said.

Bottom line: only use host compiler when compiling for the host. Each "target triple" consists of (usually) three elements:
- architecture
- OS
- environment (...which is a bit insane name for this, but let's keep it)

You are saying you should not need to use different compiler for the same architecture. It's bullshit. You do not need to use a different compiler for the same TARGET TRIPLE. Do not lie to the compiler. Obviously the ways to achieve target triple specific compilation is different for different compilers:
- for MSVC, as far as I know (but I have not researched the topic) - you are screwed
- for GCC - you build a cross compiler. The time it takes to build it is so short on modern hardware that "it takes time" is totally irrelevant.
- for Clang - you use `-target` switch. It's there since either 3.1 or 3.2, the mechanism used before was a bit more complicated, but there is no reason why anyone would use Clang <3.3 for osdev right now, so it's a non existent issue.

There's also problem of a *linker* for given target triple, and you do need to use a target-specific one. As soon as LLVM linker will become usable, I guess it will be just like Clang, so -target will suffice; for GNU ld, just build binutils for target triple. As simple as that.

----

has to support correctly a standard x64 calling convention

Again, what the heck is this "standard calling convention" and what widely accepted document is it described in?