Contest: make 32-bit PEs & ELFs easy bootable & relocateable

[NickJohnson]

... it is generally a bad idea to use a random compiler for OS development ...

Well, somebody could say to you: "... it is generally bad idea to use random OS ... stick with Windows ..." I suspect you wouldn't like that. You want to try something else, like build your own OS.

I think you misinterpreted the "random" part. He means that you can't just use a normal compiler for OS development: specifically, a compiler built for your host platform instead of your target platform. That's why you need a cross compiler.

[Combuster]

So, you're writing out a contest in order to get a desgin specifically for you?

What's wrong with making contest? Nobody forces you to participate. But if solution is posted anybody could use it since it will be ISC Licensed.

And nobody will because they have GRUB and don't need these other features.

Anyone who wants to load PEs or ELFs from their kernel needs to have some loader. It could reside inside the OS or be an external utility.

And most importantly: OS-specific.

... it is generally a bad idea to use a random compiler for OS development ...

Well, somebody could say to you: "... it is generally bad idea to use random OS ... stick with Windows ..." I suspect you wouldn't like that. You want to try something else, like build your own OS.

Do you honestly think it's a good idea to roll a dice and be forced to use the OS corresponding to the result? You'll make a choice based on several criteria. Not all OSes are suited to play games, not all compilers are suited for OS development.

Do you really think I shouldn't experiment with new ideas?

Apparently you consider a contest to be an experiment

What this all boils down to: you have much to learn about how things work, and why things work as they do. And people see you handing out assignments that are not based on reality. Wake up, Neo.

[technik3k]

I think you misinterpreted the "random" part. He means that you can't just use a normal compiler for OS development: specifically, a compiler built for your host platform instead of your target platform. That's why you need a cross compiler.

You are right, it may sound that way, but I did not mean really random. I want to compile on both Linux and Windows, have a choice of gcc and msvc compilers and be able to understand both ELF and PE formats. It is assumed that input executables will be x86 32-bit with base relocations included.

What this all boils down to: you have much to learn about how things work, and why things work as they do. And people see you handing out assignments that are not based on reality. Wake up, Neo.

Combuster, you seem to hint in your posts that you know the answers. Could you, please, point me to a compact open source code that I could easily adapt in loading relocatable PEs and ELFs for my project.

[Owen]

You seem to be trying to avoid compiler specificness, but that is going to happen anyway. For a trivial example, supporting both MSVC and GCC involves the following mess:

Code: Select all

// In a header
#ifdef __GNUC__
  #define PACKED __attribute__((packed))
#else
  #define PACKED
#endif

// In a source file
#ifdef __MSVC_VER
  #pragma pack(push)
  #pragma pack(1)
#endif
struct PACKED GDTR {
  uint16_t limit;
  void* base;
};
#ifdef __MSVC_VER
  #pragma pack(pop)
#endif

I don't know about you, but I think my choice would be to go "Urgh" and only support GCC-likes (So, GCC and Clang. And maybe ICC and Open64, *maybe*). And that excludes differences like inline assembly syntax, types (MSVC doesn't have stdint.h or stdbool.h, for example), naming conventions (PE: _name, ELF: name), and so on.

Your kernel is going to become specific to one executable format, and one compiler, very, very quickly. And implementing support for relocating relocatable ELF executables (Thats those compiled with ld -r, not ld -shared) is not particularly difficult.

And did I mention there is no reason you can't test this linker in user space? mmap in the test file, relocate it, invoke some code from it.

Adding a relocation stub to the beginning of the executable and storing the relocations in a custom format is a whole lot more onerous. Particularly as relocating a kernel module is going to involve knowing a lot about the kernel the binary is getting linked into (In order to mark .text and .rodata read only, and .rodata, .data and .bss non execute), and in order to link the loaded binary against kernel symbols. What you are asking for cannot be made generic.

[Combuster]

Combuster, you seem to hint in your posts that you know the answers. Could you, please, point me to a compact open source code that I could easily adapt in loading relocatable PEs and ELFs for my project.

*tries the Jedi trick yet again*

This is not what you are looking for.ambiguity intended

[technik3k]

You seem to be trying to avoid compiler specificness, but that is going to happen anyway. For a trivial example, supporting both MSVC and GCC involves the following mess:

Code: Select all

#ifdef __GNUC__
...
#ifdef __MSVC_VER
...

Actually, you gave a good example how this problem have been solved many times and is not really a big problem after all. One header is usually all it takes to solve compiler dependent issues with small projects like mine.

And did I mention there is no reason you can't test this linker in user space? mmap in the test file, relocate it, invoke some code from it.

Thanks, that's a great idea! Now I am going to use Win32 equivalent VirtualAlloc(NULL,size,MEM_COMMIT,PAGE_EXECUTE|PAGE_READWRITE) to test my code.

Adding a relocation stub to the beginning of the executable and storing the relocations in a custom format is a whole lot more onerous. Particularly as relocating a kernel module is going to involve knowing a lot about the kernel the binary is getting linked into (In order to mark .text and .rodata read only, and .rodata, .data and .bss non execute), and in order to link the loaded binary against kernel symbols. What you are asking for cannot be made generic.

That likely won't be a problem because I do not intend to use any paging, write protect anything, nor export any kernel symbols. One pointer to class factory with virtual functions (or system interrupt) is all that kernel needs to pass.

Combuster, you seem to hint in your posts that you know the answers. Could you, please, point me to a compact open source code that I could easily adapt in loading relocatable PEs and ELFs for my project.

*tries the Jedi trick yet again*

This is not what you are looking for.ambiguity intended

Combuster, this wasn't very helpful. Please, try again. Hint: you could have googled me this link: http://www.cultdeadcow.com/tools/pewrap.html

[JackScott]

No, you should have googled for that link, not Combuster. I agree with him, the link he gave you was much more useful.

[Combuster]

Could you, please, point me to a compact open source code that I could easily adapt in loading relocatable PEs and ELFs for my project.
(...)
Hint: you could have googled me this link: http://www.cultdeadcow.com/tools/pewrap.html

[Solar]

More popcorn, please!

@ technik3k: Let's play a quick Q&A session here. I'll even make it multiple choice as to make it easier.

Q: Do you believe that the people here do know what they're talking about?

A1: "Yes." -- Then, please, listen to what they say.

A2: "No." -- What are you doing here, then?

[gedd]

A2

PE is bad

[Owen]

You seem to be trying to avoid compiler specificness, but that is going to happen anyway. For a trivial example, supporting both MSVC and GCC involves the following mess:

Code: Select all

#ifdef __GNUC__
...
#ifdef __MSVC_VER
...

Actually, you gave a good example how this problem have been solved many times and is not really a big problem after all. One header is usually all it takes to solve compiler dependent issues with small projects like mine.

OK, so how do you portably pack structures without including #ifdefs for MSVC all over the place? At least with GCC you can #define __attributes...

[technik3k]

Could you, please, point me to a compact open source code that I could easily adapt in loading relocatable PEs and ELFs for my project.
(...)
Hint: you could have googled me this link: http://www.cultdeadcow.com/tools/pewrap.html

Thank you for the link, Combuster, it is a good start but doesn't do everything I need.

@ technik3k: Let's play a quick Q&A session here. I'll even make it multiple choice as to make it easier.

Q: Do you believe that the people here do know what they're talking about?

A1: "Yes." -- Then, please, listen to what they say.

A2: "No." -- What are you doing here, then?

You must be too much into binary system around here if you believe this question has only two answers. Some bits can carry more information then the others.

No, you should have googled for that link, not Combuster. I agree with him, the link he gave you was much more useful.

You guys are so cruel I am going to cry for the next 3 microseconds.

[technik3k]

OK, so how do you portably pack structures without including #ifdefs for MSVC all over the place? At least with GCC you can #define __attributes...

MSVC has similar, but not equivalent, keyword __declspec(align(n)), see: http://msdn.microsoft.com/en-us/library ... S.80).aspx

However, personally, I like to use bit fields for the situations you describe: http://msdn.microsoft.com/en-US/library ... S.80).aspx

[Owen]

Most compilers generate rather slow code for accessing bitfields, although that isn't important in this case.

As for __declspec(align(x)), that maps to GCC's __attribute__((aligned(x)), not __attribute__((packed)). Using alignment to specify structure packing requires messy structure definitions.

And you ignored the many other more major differences in my post.

[technik3k]

You seem to be trying to avoid compiler specificness, but that is going to happen anyway. For a trivial example, supporting both MSVC and GCC involves the following mess...

Owen, I never intended to argue that anyone should use multiple compilers instead of one. If your project needs features outside of the language specification you are going to encounter headwind anyway and smartest solution may be to settle on a single compiler and toolchain.

What you are asking for cannot be made generic.

I do not look for generic solution suitable for everybody. If this contest yields a utility that could handle both ELFs and PEs anybody could take only select parts of the code and adopt them to their OSes in a way it suits them. I wonder how many people reading this thread can load and relocatable executables in their OS (I would love to have such code).

And you ignored the many other more major differences in my post.
...
And that excludes differences like inline assembly syntax, types (MSVC doesn't have stdint.h or stdbool.h, for example), naming conventions (PE: _name, ELF: name), and so on.
...
In order to mark .text and .rodata read only, and .rodata, .data and .bss non execute, and in order to link the loaded binary against kernel symbols.

Since you define your own project goal you can choose anything you want to optimize for. It may be your time, code speed, size or ease of incorporating third party code with minimum modifications. I may choose one of the goals for my project to be compiler independent as much as possible and see how far it gets me.

For example, I may avoid specific headers, use compiler options like "gcc -fleading-underscore", or avoid dealing with export symbols altogether by implementing class factories and interfaces (COM) using C++ virtual functions. Similarly, no paging and flat address space means no need to worry about section access permissions. I intentionally decided to trade hardware protection for flexibility and will see to which degree it could be compensated in software.

The last few points, obviously, better suited for general architecture discussions and not this thread.