Just to confirm: Is this how ELF files should be readed?

[simeonz]

Ok. How do I produce a DLL-like library with ELF? not PIC.

Answer. On x86 compile without "-fpic" and link with "-shared". This will leave the relocations which are necessary, i.e. those which were not solved by rip-relative displacements (jumps and calls vs variable addresses.) The result is of type ET_DYN, which means that it can be properly loaded at any address, provided the dynamic relocations are applied. On x64 again compile without "-fpic", but with "-mcmodel=large". This is necessary, because the default model is "small", meaning that the relocations assume that all objects will be placed in the first 2G. This is true for ET_EXEC, but not true for ET_DYN on x64, hence you need to inform the compiler (which is not aware whether you will link ET_DYN or ET_EXEC) with "mcmodel=large" that the relocations have to accommodate the larger address space or the final link will fail. Again, link with "-shared". This will leave only the important relocations, but since indirect rip-relative addressing is introduced in x64, most relocations get eliminated. If you think you get too many dynamic symbols, use the visibility flags and attributes of the compiler and linker in order to suppress interposition. (i.e. -fvisibility, -Bsymbolic)

And a side remark. In elf, the sections are more complex entities when compared to segments. The compiler needs to understand only how to generate section metadata. The loader needs to know how to read segment metadata. The link editor must know how to combine sections into segments. But the loader does not need to handle sections. Compared to segments, sections have names, groupings, COMDAT semantics. Segments don't need those things. Segments on the other hand support physical load addresses, for platforms that need them. Segments were created to simplify the loader, not to complicate it. But anyway, that is not important. I am not urging you to use any particular format. I am not a fanboy or hater. In my eyes, both windows and linux are serious accomplishments with numerous flaws. The choice is yours.

Edit: I corrected a mistake, in that I was advising to use -fvisibility, -Bsymbolic to reduce the got and plt indirections, aside from the fact that they reduce the exported symbols. But as was intended in the entire premise of the answer, on non-pic builds got and plt are not in play. Only the number of interposed symbols, and correspondingly, the number of relocations are reduced. In pic builds, those options do reduce the got and plt indirections and pre-apply some of the relocations as well.

[simeonz]

If this is the 'strings string strings' you were talking about, well, how would you store a label as anything other than a string? The only thing I can think of is some kind of hash, but then, how would you resolve a hash collision?

GUIDs would be nice, although one could speculate that they weren't yet in fashion when elf was conceived. For example, a GUID for the symbol, and milliseconds since 1970 in 64-bit field for version control.

[Schol-R-LEA]

If this is the 'strings string strings' you were talking about, well, how would you store a label as anything other than a string? The only thing I can think of is some kind of hash, but then, how would you resolve a hash collision?

GUIDs would be nice, although one could speculate that they weren't yet in fashion when elf was conceived. For example, a GUID for the symbol, and milliseconds since 1970 in 64-bit field for version control.

That might work for resolved locations (though a 16-byte GUID plus four bytes for the 64-bit time_t is often going to be longer than the label was

It won't work for an external symbol, though, at least not without having access to the symbol table of the external object file - at which point you may as well be linking them right then, assuming that it can be resolved at all.

Or did I misunderstand you point? (assuming you weren't just being sarcastic - I am pretty sure you were being mostly sarcastic, but I may be wrong.)

[simeonz]

That might work for resolved locations (though a 16-byte GUID plus four bytes for the 64-bit time_t is often going to be longer than the label was

Actually, it depends on the symbol. For C standard library symbols, the names are short. But the standard library has always had privileges in terms of the namespace allocation. Most other libraries either have to prefix their identifiers according to the library name and function, or use namespaces for C++, which after mangling with the type decoration can end up being considerably longer.

That being said, the size had nothing to do with it. For me, variable-length human-invented identifiers have no place in the world of globally distributed binaries. I am more inclined to deal with guid attributes in the language than to deal with name collisions at load time. The headers can be generated by tools, although one-time investment of effort is not going to destroy any schedule. But not least of all - we were talking about elf and not c/c++. Indeed elf is designed around c, but that is a hindrance in and of itself. Better development tools wouldn't use header files at all, and could use import databases that maintain the mapping, using datasets provided by the library vendor to populate the database originally.

It won't work for an external symbol, though, at least not without having access to the symbol table of the external object file - at which point you may as well be linking them right then, assuming that it can be resolved at all.

Actually, external symbols is exactly what I meant. I wouldn't dare to ask people to mess with guids for internal definitions. Unless the IDE or the toolchain provides the guid bindings automatically, in which case all the better. I am probably too influenced by COM. I really do consider human invented identification to be amateuristic technique when it comes to libraries. To me that is analogous to asking people to think of their own static ips before plugging into the internet.

Or did I misunderstand you point? (assuming you weren't just being sarcastic - I am pretty sure you were being mostly sarcastic, but I may be wrong.)

Unfortunately, no. I wasn't.

[Schol-R-LEA]

EDIT: I had contacted John Levine for permission to quote his book for the wiki, and unfortunately he made it clear in no uncertain terms that I should not. While I am quite certain that the excepts in this post would count as fair use, I do not wish to create ill-will with either him or his publishers, so I will be removing the quotes, though I am afraid that this will likely render the post unreadable.

The fact that the PE format doesn't separate the two doesn't mean it doesn't have that additional information that ELF puts in the program (segment) header. I may be wrong, but I am guessing that several of the parts of the PE header you have dismissed as 'I don't need to worry about that' are basically the same meta-data as in the ELF program header.

it's comparing apples and oranges. PE operates on sections as mapping entinties. No need of "segments". If you think .txt and .rdata and maybe even .idata (import) should be merged into one, just do it with the linker script - emit one output section for them - that's all, everything has been devised already. Segments are the same sections, just unnecessarily made as independent entities. but what linker sections do in ELF then? nothing. mess.

Fnark, I think I just figured out why this paragraph has been bothering me so much: you have the relationship backwards. Segments are the part you need at run time, comparable to the sections in a PE file¹; the 'sections' in an ELF file are only needed for linking, and are the majority of what strip clears (IIUC, C&CW).

(TBF, I got confused on this part, too. So far as I can tell, ELF sections aren't used at all at run time.)

BTW, this (IIUC) is also why the section headers are at the end of the file: the linker will arrange for the symbol tables, etc. for static meta-data to be at the end of the final executable file, right before the section header, meaning that all strip needs to do is step backwards through the section headers to find the first such section in the file, overwrite it with a dummy section header, reset the e_shoff, e_shentsize, e_shnum, and e_shstrndx fields² in the file header to reflect the new section header, and truncate the file after the dummy header.

To quote Levine: [...]

IIUC, while the OBJ and EXE/DLL formats in Windows are mostly the same, there are some things in the former not in the latter two, and vice versa. The executable PE files are the equivalent of a stripped ELF executable file, while the COFF OBJ temporary files have the equivalent of the ELF section headers but no segment (PE section) headers. DLLs use the same PE format as executables, with the difference marked by bit 13 the Characteristics set of flags (called f_flags in the OSDev wiki). Quoting Levine again: [...]

To go back to .bss for a moment, the PE (or rather, COFF)³ header calls that part the SizeOfUninitializedData field (as per Levine, who is apparently quoting the COFF documentation as well as Microsoft's; the header declaration shown in the OSdev wiki's page on COFF that header calls it bsize - same field, different name). To quote Levine yet again:
[...]

So again, I think Zaval's complaints here are misplaced.

Footnotes
1. According to Levine, Unix executable formats had been calling them 'segments' going all the way back to the NMAGIC version of a.out circa 1975. I gather that this applies to the (hoplessly inadequate) Unix COFF documentation, as well; apparently, it was Microsoft who changed it to 'sections' in 1993, when adapting the then ten-year-old Unix format for use in Windows NT, while at the same time making sure the format actually was documented sanely. [...]
2. Names as per Wikipedia; Levine calls them shdrpos, shdrent, phdrcnt, and strsec, while the OSDev wiki doesn't have an example C struct declaration for them (unlike with the PE page), but just gives them by their byte positions in the header and a description.
3.This passage seems relevant: [...]
Given the heavy influence of VMS on the NT design, I am not sure why they chose a Unix format instead of copying the VMS format - probably, it was because the goal was to have a Portable Executable format that could be used on the (now defunct) MIPS, Alpha, and SPARC versions of NT as well as the x86 version, and the VMS format was too deeply tied to the VAX architecture.

[Schol-R-LEA]

That might work for resolved locations (though a 16-byte GUID plus four bytes for the 64-bit time_t is often going to be longer than the label was

Actually, it depends on the symbol. For C standard library symbols, the names are short. But the standard library has always had privileges in terms of the namespace allocation. Most other libraries either have to prefix their identifiers according to the library name and function, or use namespaces for C++, which after mangling with the type decoration can end up being considerably longer.

I was being flippant; for such a system, size would be the least of one's worries, and in any case having a fixed-size field would probably save more memory than it would waste.

That being said, the size had nothing to do with it. For me, variable-length human-invented identifiers have no place in the world of globally distributed binaries. I am more inclined to deal with guid attributes in the language than to deal with name collisions at load time. The headers can be generated by tools, although one-time investment of effort is not going to destroy any schedule. But not least of all - we were talking about elf and not c/c++. Indeed elf is designed around c, but that is a hindrance in and of itself. Better development tools wouldn't use header files at all, and could use import databases that maintain the mapping, using datasets provided by the library vendor to populate the database originally.

It won't work for an external symbol, though, at least not without having access to the symbol table of the external object file - at which point you may as well be linking them right then, assuming that it can be resolved at all.

Actually, external symbols is exactly what I meant. I wouldn't dare to ask people to mess with guids for internal definitions. Unless the IDE or the toolchain provides the guid bindings automatically, in which case all the better. I am probably too influenced by COM. I really do consider human invented identification to be amateuristic technique when it comes to libraries. To me that is analogous to asking people to think of their own static ips before plugging into the internet.

While I do see your last point, and even agree with it, I am not certain how this is to be accomplished - either there would have to be a registrar of the bindings for all possible external references (which the compiler would need to be able to access even if the external element doesn't exist yet - or to put a different angle on it, the compiler could be put into in the position of creating a placeholder GUID for someone else's code, with no prior guarantee that the code will ever even be written, that the registrar will be accessible to a linker, or that the external element would match the signature assumed if it does), or a guaranteed way to regenerated exactly the same symbol identifier consistently without communication with the library, which is pretty much the opposite of what a GUID is meant to be (since the whole point of those is that there is only a vanishingly small chance of it ever being generated twice).

But perhaps I am missing what you have in mind. Please, feel free to elaborate. If you have a solution, I would be interested in hearing it, as it has bearing on some related problems I have for my own planned designs (which do indeed call for just such a universal distributed database of exported and imported code, just in a language context far removed from C or C++, and only bearing a distant similarity to even the more established branches of the Lisp family which are its closest relatives).

[simeonz]

While I do see your last point, and even agree with it, I am not certain how this is to be accomplished - either there would have to be a registrar of the bindings for all possible external references (which the compiler would need to be able to access even if the external element doesn't exist yet - or to put a different angle on it, the compiler could be put into in the position of creating a placeholder GUID for someone else's code, with no prior guarantee that the code will ever even be written, that the registrar will be accessible to a linker, or that the external element would match the signature assumed if it does), or a guaranteed way to regenerated exactly the same symbol identifier consistently without communication with the library, which is pretty much the opposite of what a GUID is meant to be (since the whole point of those is that there is only a vanishingly small chance of it ever being generated twice).

But perhaps I am missing what you have in mind. Please, feel free to elaborate. If you have a solution, I would be interested in hearing it, as it has bearing on some related problems I have for my own planned designs (which do indeed call for just such a universal distributed database of exported and imported code, just in a language context far removed from C or C++, and only bearing a distant similarity to even the more established branches of the Lisp family which are its closest relatives).

Actually, no. I wasn't thinking about a registrar. When I said database, I meant something else. But let me focus on the more down to earth usage first.

The standard workflow is for the library authors to create the header files along with the source files and build the library elfs/archives to be distributed. A client portion of the header files is supplied to the client code authors. Those, along with the rest of the project files, are used to build the client executable. This is what I consider to be the classic scenario. Of course, there are other possibilities. Such as the client code authors having no official header files, but being in possession of the library elfs/archives and knowing the exported functions' signatures. Or, the client code authors lacking even the library elfs/archives at the early development stages, but having information about the library's functions' signatures, provided on good faith by the library's vendors.

For the classic workflow, the changes are very small. In the final stages of development, before distribution, a GUID generator is used to make one unique id for each exported library routine/symbol. The client headers (which are also included by the library sources themselves, as is usual) are modified by appending an "__attribute__(GUID("guid-string"))" to the previously naked declarations. Now gcc knows enough to begin generating a .symbol.guids section in the output objects, which the linker cooks into the final elf/archive as exported (i.e. global, visible and defined) symbols metadata. Similarly, by including those headers, the client executable will be enriched with imported (i.e. global and undefined) symbols metadata. For the case of shared object libraries, the loader will now be able to bind the symbols using guids instead of strings. For static libraries, the link editor will be able to do the symbol resolution using guids.

For the case in which the signatures are known and the library itself is present, but there are no official headers, a tool from the toolchain can be used to list the GUID metadata. Think "objdump --guid" or something similar. Alternatively, a source code generator could be fed the library elf/archive directly and emit a "#pragma guid symbol = guid-string" header (analogous to "#pragma weak symbol1 = symbol2" currently available.)

For the case in which the signatures are known and the library and header files are both unavailable, the client source can be developed without guids, using plain identifiers. When the library is finalized, the guids or library files can be distributed to the clients, which can amend their projects. Or the authors, if they have indeed frozen the library interface at the preliminary stages, could generate the interface guids and provide them to the clients without the executable code.

Regarding my databases rant - I was obviously mostly drooling on futuristic concepts. Instead of headers, you can distribute dataset that provides the human readable name to guid mapping. In the more primitive case, a command line tool can then be used to import this dataset into any "namespace" of an identifier database. In the more advanced case, an IDE will enable you to fine-tune the import process. The database holds all mappings that the toolchain uses to resolve identifiers during the build. The attraction is in that the semantic and syntactical aspects become separated. The IDE can easily query and edit such a database, to track dependencies or to refactor the symbolic contents of a project without impacting the bindings beneath it. The internal references can be represented by a graph. However, the symbols at the boundary cannot be part of the graph itself, so they need to be keyed somehow. A guid key would solve that. There are a lot of details that need to be clarified even in theory. Such as a way to update import datasets without breaking the graph, etc. Also, this is more or less how IDEs work today, except that the code remains stored in inefficient human readable form and that the IDE's database is not reused by the toolchain. Newer programming languages are designed with the foresight of being machine parseable and this ameliorates the issue somewhat. Something of this kind has been tried before commercially (I cannot recall the product), but I don't think it caught on. Then again, it might have been just a timing issue. Or a financial one.

[Schol-R-LEA]

Regarding my databases rant - I was obviously mostly drooling on futuristic concepts. Instead of headers, you can distribute dataset that provides the human readable name to guid mapping.

That's hardly futuristic; languages such as Modula-2 and Ada were using definition files in the early 1980s, and Java classes bundle precisely the information you mean with the compiled bytecode (personally. I prefer them separate, but whatever). I don't know if D, Rust, Swift, or Go use compiled exports, but I am betting that at least one of them does (e.g., I am pretty sure 'crates' in Rust work similarly to Java jars).

The only reason it seems unusual is because C and C++ don't pre-compile export information, but instead rely on the simpler, and far less reliable, approach of automatic text insertion - something that was known to be a Bad Idea before C was even developed, but is so simple to implement so low overhead to run that it became the norm. C headers aren't really a separate thing from the code being compiled - its all sleight of hand. Worse, there really isn't any way to retrofit definition files onto C or C++ withouth breaking compatibility - you would end up with a different language, or at least a dialect (such as C++/CLI) which still retains the older, broken approach side-by-side with the new one.

This is a classic "Worse is Better" trade-off, where something known to be a terrible idea persists for decades because it happens to be easy to understand (on the surface, at least) and easy to do.

TL;DR - to get what you want, drop C and C++ in favor of, oh, just about any systems language developed since 1986 or so. Assuming that that is an option for you...

[zaval]

jeez, it's huge.
honestly now I am out of context for this avalanche dispute, I want all my free time I won of procrastination to dedicate to a slightly different topic, but the next task will be exactly my Pegen translator, so I promise to return here and try to add my opinions (turned to be so interesting to School-R-Lea ) and answer questions to me (in addition to what I wrote below). Too bad, School-R-Lea, you weren't as excited to participate in another question of mine - about ARM EL2 single VA space. Here, we probably should agree, that everyone has their own preferences and that's not something that's gonna change. I repeat, I want to support both formats. PE is native, for the OS and "the main" environment subsystem, Elf is not native - for POSIX subsystem. That's all.

simeonz, thank you for pinting me on that, I need to take a look at that. unfortunately, I am out of x86 now. But it might be interesting in relation to ARM/MIPS too.

School-R-Lea, just a short about "strings". No, I haven't confused elf itself with objdump output of it and no, I didn't mean "ELF" signature, nice joke! I clearly asked about exporting by ordinal, even showed a piece of code how it's made manually for PE. exported symbols, functions in this case, are exported by ordinal, no need to search for strings at the load time. Is there something similar on ELF?
and bss. I didn't say "uninitialized data" is not needed. I said, bss as a separate entity is not that needed. PE easily deals with space saving on uninitialized data without special bss. It's just easier and more elegant. Several times here this bss thing was mentioned as an example of Elf brilliance. On PE, as you can see, it's obtained even easier. Just VirtualSize of the data section accounts for all the data, whereas SizeOfRawData accounts only for initilialized data and thus - the size on disk. When loading this section, you just zero pad it, creating space for uninitialized data. That's all. See, there are no special overly devised entities. Finally, that "SizeOfUninitilaizedData" field you mentioned (yes, I'm aware of it) is a COFF legacy, barely needed.
and why I don't like PIC. because I think if the ISA gives a mix of PIC and not PIC addressing, then the format should be non PIC - to take advantage of all the possibilities ISA gives. And to not introduce perversions and performance penalties all those "overcomings" (for achieveing PIC code on a mix ISA) introduce.

[Schol-R-LEA]

Too bad, School-R-Lea, you weren't as excited to participate in another question of mine - about ARM EL2 single VA space.

That's mainly because I don't knoe enough about ARM yet to even understand the question.

But I will leave this as it is. I am still puzzled, but at this point, it isn't even a matter of agreeing to disagree - I have no love for ELF, specifically, I simply had the impression (probably wrongly) that you were basing your dislike of it on incorrect assumptions about it.

[simeonz]

That's hardly futuristic; languages such as Modula-2 and Ada were using definition files in the early 1980s, and Java classes bundle precisely the information you mean with the compiled bytecode (personally. I prefer them separate, but whatever). I don't know if D, Rust, Swift, or Go use compiled exports, but I am betting that at least one of them does (e.g., I am pretty sure 'crates' in Rust work similarly to Java jars).

Java and .Net use jars and assemblies to gather import information, but they are still fundamentally text based. It is true that they avoid the need for header files and that the languages were specifically designed to be easy to parse and refactor by the IDE. But they still use text as the primary medium that describes the program's technical and literary content. It is the program's primary storage format and the rest of the tools compensate the non-structured nature of the text information by keeping their own accelerating structures on the side. What I meant instead was a programming infrastructure built around storage in a graph format that separates the textual presentation from the structural information. IIRC, you had such ideas about your language design as well, so I expected that you should know what I mean here.

That being said - none of this is a practical necessity at the moment. It "doesn't solve an existing problem" as they say. It might make the IDEs faster, it would enable extra presentation features, better tooling, etc, but nothing disruptive on its own. Also, it wont harmonize with legacy technologies, like text-based version control, despite that in principle a graph representation offers better tracking capabilities in principle.

Edit: In retrospect, you may have meant that those languages bind by GUID, not identifiers, or that they use GUID when describing the referenced jars/assemblies. I am not sure how .NET references actually bind, so I may have misunderstood the statement. Also, I am not familiar with Modula-2 and Ada, and have no idea what import mechanism the rest of the languages use, so I restrained from commenting.

[simeonz]

and why I don't like PIC. because I think if the ISA gives a mix of PIC and not PIC addressing, then the format should be non PIC - to take advantage of all the possibilities ISA gives. And to not introduce perversions and performance penalties all those "overcomings" (for achieveing PIC code on a mix ISA) introduce.

Just a clarification here, because this is not very well explained by the documentation. If the elf symbols of a library are given protected or hidden visibility, which means that they cannot be interposed, the internal references to those symbols will rarely use got and plt indirections, even for PIC builds. Those mechanisms are primarily used for imported symbols. Msvc also uses IAT (i.e. PE's got) and function stubs (i.e. PE's plt), so the situation is mostly analogous. Sometimes gcc might use got for local symbols, despite that they cannot be interposed, but this is done when the alternative is to fix-up the code with relocations or to use slower code. But again, with proper visibility attributes (declaration features similar to __declspec(dllimport), __declspec(dllexport)), most got and plt indirections should be bypassed at link time.