OSDev.org

Let's say I want to design and use a new executable format for my OS that has some special feature I cannot fit into ELF. On a scale from 1 to 10 (7 being the highest, of course), how hard would it be to make GCC create executables with that format? Has anyone here done such a thing?

NickJohnson wrote:Let's say I want to design and use a new executable format for my OS that has some special feature I cannot fit into ELF. On a scale from 1 to 10 (7 being the highest, of course), how hard would it be to make GCC create executables with that format? Has anyone here done such a thing?

I would bet it's more work on ld than gcc, but it would probably require work in both, and maybe if your especially unlucky, it may require modifying `as`...

Have you considered looking at an alternative compiler/linker that is a bit less huge?

I guess something like LLVM would be much nicer to modify - I've heard it is a lot smaller and more modular. I've been trying to get it to compile my kernel, but things keep breaking (probably because it's not a proper cross compiler). Has anyone worked with LLVM on a source level?

NickJohnson wrote:I guess something like LLVM would be much nicer to modify - I've heard it is a lot smaller and more modular. I've been trying to get it to compile my kernel, but things keep breaking (probably because it's not a proper cross compiler). Has anyone worked with LLVM on a source level?

Yes I have, what do you want to know?

NickJohnson wrote:On a scale from 1 to 10 (7 being the highest, of course)

How good are you at math?

Sorry I can't answer your question. I am planning the same thing in the future so I'm reading the replies with great interest.

I'd imagine it'll be a binutils issue. But I think it'll be certainly a lot easier than trying to port binutils and GCC to a new architecture..

It's pretty hard to say without knowing what makes your executable-format-to-be so special.

If it's not all that esoteric, you might get away with writing a new BFD back-end. (That's a part of binutils, not GCC at all.)

I agree with Solar, we probably need to know more specifics.

I've heard working with gcc is a pain. Depending on what you want, you might just want to do a binary to binary translation. Open the executable from gcc, grovel through it, shuffle stuff around, import some stuff from, then write out your new binary.

I'm asking this because I want to know whether I should invest energy in designing and writing a loader for a new executable format - if it's effectively impossible to modify an existing open source toolchain to support it, I would just try to work around ELF. So, I don't have an exact answer on how different the format would be, but I doubt it would be much different (the main differences would have to do with dynamic linking and relocation).

@Solar: That's really what I was looking for: it doesn't look too hard to write a BFD backend, and at least it's somewhat modular in design. I'll take a look at the actual code later today.

NickJohnson wrote:I'm asking this because I want to know whether I should invest energy in designing and writing a loader for a new executable format - if it's effectively impossible to modify an existing open source toolchain to support it, I would just try to work around ELF. So, I don't have an exact answer on how different the format would be, but I doubt it would be much different (the main differences would have to do with dynamic linking and relocation).

@Solar: That's really what I was looking for: it doesn't look too hard to write a BFD backend, and at least it's somewhat modular in design. I'll take a look at the actual code later today.

Also, if there is anything that you would have to modify in gcc, you may look instead at pcc. It is a very lightweight compiler that aims at replacing gcc(and soon may be able to as it already compiles most code) and it works with binutils...

You mean the Portable C Compiler?

You're not trying to generate assembly code for a different architecture, so there's no point looking at different compilers when it's a binutils issue and nothing to do with the compiler.

MessiahAndrw wrote:You're not trying to generate assembly code for a different architecture, so there's no point looking at different compilers when it's a binutils issue and nothing to do with the compiler.

Yeah, I realize that more clearly now. I really meant the GNU toolchain when I said GCC in the title: binutils and GCC appear pretty similar in code quality and organization, which is what is important.

The conponent you'll be looking to modify is libbfd. This is the common library gcc and binutips uses to access binary files. Search google for how to modify bfd. They have a very informative design document. I haven't actually implemented a new binary format but I have worked with binutils before(adding a new machine code backend) and I can tell you they're a real pleasure to work with. In all, it took me two days to fully complete my extension.