global vars, elf and bin

[Octacone]

Hi everyone, I have a problem with global variables and some linking decisions to make.

First of all when I declare:

Code: Select all

uint32_t test = 0xCAFEBEEF;

and try to print it, I get: 0xCAFEBEF0. Which is by itself strange. So I looked around and found an old topic about the same kind of issue I am experiencing.
The solution was to change from OUTPUT_FORMAT(binary) to OUTPUT_FORMAT(elf32-i386) which breaks everything for me.
I originally did not want my kernel to be formatted as .elf but it looks like I will be forced to do it, or maybe not?
Currently I have two compiled files called Kernel_C_Plus_Plus.o and Kernel_Assembly.o which are linked together and outputted as Kernel.sys.
Now my question is can I link those two (elf internally by definition) files together and let them be a flat binary? So I can load that file and directly jump to it, no parsing, no hustle (my current setup btw).
Or I'll be forced to switch to elf? If yes, is there a simple way to just jump to it, since it is already fully loaded into memory at 0x100000 (1MB)? No need for safety checks and dynamic relocation etc...

[Velko]

You can link it as ELF and then objcopy to flat binary.

Intermediate ELF image of your kernel can be very useful: you can disassemble, load debugging symbols, inspect it using various tools, etc.

[Octacone]

You can link it as ELF and then objcopy to flat binary.

Intermediate ELF image of your kernel can be very useful: you can disassemble, load debugging symbols, inspect it using various tools, etc.

I've seen people doing so, but how does my code know where all the sections are? Looks like that could be a massive problem. Something I didn't think of before.
Right now I am reading trough ELF specs and they are kind of confusing.
What exactly am I supposed to do with my elf image?
It is already fully loaded at 1 MB, I tried jumping to both 0x101000 and 0x100100 but that doesn't work.
As far as I could understand there are so called segments that I am supposed to load into memory? But where am I supposed to load them at? I want them (my kernel) at 1 MB and I specified that inside my linker file.
Does this sound sane:
1.Find where the program header table is.
2.It is like an array of program headers, number of them = phnum
3.Load them somewhere... mysterious to me (Edit: figured it out)
4.Jump to e->entry, but how can I jump there if that address contains Elf_32_Ehdr?

[MichaelPetch]

If you use objcopy to convert the ELF file to binary then you just load the entire file as is into memory. Maybe if you showed us your code and linker script (maybe put a project on github) with commands used to put it altogether. It sounds like you must have written your own bootloader? Did you consider just using grub? Or are you using that?

[iansjack]

The most likely explanation of your original problem is that there is a fault in the routine that you are using to print out the value of the variable. Check what it actually looks like in memory.

[Octacone]

If you use objcopy to convert the ELF file to binary then you just load the entire file as is into memory. Maybe if you showed us your code and linker script (maybe put a project on github) with commands used to put it altogether. It sounds like you must have written your own bootloader? Did you consider just using grub? Or are you using that?

I tried using objcopy and it works just "fine" (more on that later). Yes I have a custom bootloader.

The most likely explanation of your original problem is that there is a fault in the routine that you are using to print out the value of the variable. Check what it actually looks like in memory.

Looks fine in memory until a certain function gets called:
I'm such a genius... that was my mistake... I was incrementing it by 1 to test it lol...

Okay, so right now I can declare a variable and assign any number to it, except 0. When I make it zero it gives me some random junk. Since zero initialized values end up in .BSS I suspect there might be something wrong with that sections. I didn't forget to include it doe. Maybe it is not initialized or even worse it points to code.

Does anybody know if there are any significant differences/advantages that elf has over flat binaries (except debugging)?
Is writing an elf loader worth it (for kernel loading purposes ofc, not talking about userspace)?
Can anybody check if my understanding of ELF is correct, also I figured out where I should load those segments. (from the post above).

[iansjack]

Elf files can be considerably smaller than binaries. They include more information than a binary.

But, IMO, probably the biggest plus is that they ease the use of shared libraries, which any decent OS should aspire to for so many reasons.

[AJ]

+1 to iansjack

The most likely cause of your trouble with zero values is that the BSS is not being initialised. Whatever loader you are using should memset the entire BSS to zero.

Note also that in the ELF file you have program headers / section headers describing the amount of data in the file (which your loader should memcopy) and the amount of data in the loaded binary. If there is a difference between these two values, you should zero the "excess" bytes that are not contained in the ELF file on disk.

Cheers,
Adam

[Octacone]

Elf files can be considerably smaller than binaries. They include more information than a binary.

But, IMO, probably the biggest plus is that they ease the use of shared libraries, which any decent OS should aspire to for so many reasons.

What do you mean by smaller? My kernel is 20 KB larger when compiled as ELF, what is to be expected considering the additional tables.
Shared libraries sound like a nice idea, but I propose you're talking about userspace execution. But I am far away from being able to dream about that.

+1 to iansjack

The most likely cause of your trouble with zero values is that the BSS is not being initialised. Whatever loader you are using should memset the entire BSS to zero.

Note also that in the ELF file you have program headers / section headers describing the amount of data in the file (which your loader should memcopy) and the amount of data in the loaded binary. If there is a difference between these two values, you should zero the "excess" bytes that are not contained in the ELF file on disk.

Cheers,
Adam

I initialized the entire thing to 0 and it seems to be working. But there is some data after it, is that normal?
Nice to know, so I just need to discard everything that is not "mentioned" by those headers.

[iansjack]

I suspect that you are comparing an elf file that you haven't stripped unnecessary (for run time) information from and a binary. You should strip the elf file then compare them; this can reduce the size of the elf file by a large factor). ( https://www.computerhope.com/unix/strip.htm )

Why is a binary larger than an elf file (in general)? Take a simple example. Your code has a large table that is uninitialised, and so stored in the .bss section. The elf file just needs to record the length of the section, and your loader will set aside enough memory for it and zero that memory, but the binary needs to hold the whole table as a series of zeros (otherwise how would your loader know how long the .bss section is and how would its contents get zeroed?).

As for shared libraries; you may think they are a long way down the line - and indeed they may be. But when you get to that stage you will have to rewrite all your code to accommodate elf files rather than the binaries that you have been using. Why waste time doing this when simple elf files are easy to load? You will later have to expand your code to deal with relocation, but it's better than starting over. Putting off difficult concepts because there are easier alternatives just leads to more work in the long run. And the whole fun of OS development is getting to grips with difficult concepts. (Another example, IMO, is to use FAT rather than ext format for a filesystem - sooner or later you are going to want something better than FAT, so why not start as you mean to go on?)

[zaval]

elf files always are way bigger than plain binaries generated of them.
my initial code loader for arm, is 67 KB as elf and 2 KB as binary. and it doesn't have yet .bss. and without any alignment set.

speaking of which, don't even try to align sections on page size in elf by the way, this thing doesn't understand what file alignment and section alignment is, and literally puts those gaps in files, that makes them just huge. in fact sections in elf are just a fancy decor, because those geniuses thought that instead of using them, they will be better off to introduce an absolutely unnecessary entity, fully duplicating what sections could do - segments, turning the whole thing into pile of crap. elf format is pretty moronic honestly (first because of duplicating entities - sections and segments, second its approach to dynamic linking is braindead).

still, plain binaries are suitable only in some cases for loaders, that's all.

imo, the best approach to do dynamic linking is PE format.

[Octacone]

I suspect that you are comparing an elf file that you haven't stripped unnecessary (for run time) information from and a binary. You should strip the elf file then compare them; this can reduce the size of the elf file by a large factor). ( https://www.computerhope.com/unix/strip.htm )

Why is a binary larger than an elf file (in general)? Take a simple example. Your code has a large table that is uninitialised, and so stored in the .bss section. The elf file just needs to record the length of the section, and your loader will set aside enough memory for it and zero that memory, but the binary needs to hold the whole table as a series of zeros (otherwise how would your loader know how long the .bss section is and how would its contents get zeroed?).

As for shared libraries; you may think they are a long way down the line - and indeed they may be. But when you get to that stage you will have to rewrite all your code to accommodate elf files rather than the binaries that you have been using. Why waste time doing this when simple elf files are easy to load? You will later have to expand your code to deal with relocation, but it's better than starting over. Putting off difficult concepts because there are easier alternatives just leads to more work in the long run. And the whole fun of OS development is getting to grips with difficult concepts. (Another example, IMO, is to use FAT rather than ext format for a filesystem - sooner or later you are going to want something better than FAT, so why not start as you mean to go on?)

Are you sure that is true. Binaries do not contain .BSS sections. (I've read that on this forum multiple dozen of times) They will only hold it if you tell them to.
Well this is how I know how many bytes to zero:

Code: Select all

...other sections
...this is the last section:
.bss :
    {
        bss_start = .;
        *(.bss)
        *(COMMON)
        bss_end = .;
    }

and then BSS size = bss_end - bss_start.

My kernel code (ring 0) does not use any libraries. For example: if my IDT code needs a function called Memory_Set I would only need "#include Headers/Tools.h". The only situation in which I would need a library is when a third party code (loaded from the disk as a separate file) wants to use system calls. We might be misunderstanding each other a bit. So why would I want a library linked to my Kernel.elf? Explanations appreciated.

[Octacone]

elf files always are way bigger than plain binaries generated of them.
my initial code loader for arm, is 67 KB as elf and 2 KB as binary. and it doesn't have yet .bss. and without any alignment set.

speaking of which, don't even try to align sections on page size in elf by the way, this thing doesn't understand what file alignment and section alignment is, and literally puts those gaps in files, that makes them just huge. in fact sections in elf are just a fancy decor, because those geniuses thought that instead of using them, they will be better off to introduce an absolutely unnecessary entity, fully duplicating what sections could do - segments, turning the whole thing into pile of crap. elf format is pretty moronic honestly (first because of duplicating entities - sections and segments, second its approach to dynamic linking is braindead).

still, plain binaries are suitable only in some cases for loaders, that's all.

imo, the best approach to do dynamic linking is PE format.

67 KB vs 2 KB, that is the thing I was talking about. It wastes a lot of space.
So they are basically pre-aligned, that is both a plus (I don't have to) and a minus (space wasted).
Well I guess it is kind of outdated, but it is "simple", at least people say so. It is also very supported it seems.
Fortunately we can remove all the unnecessary sections (most of it lol).
I'll remember to take a look at PE when the times comes.

The other thing considering ELF kernels is physical memory restrictions.
My kernel is loaded at 1 MB and I have about 15 MB until I hit an ISA hole. I don't expect my kernel to grow that big (15 MB of code is a lot).
So I have a bootloader that loads a file called Kernel.elf at 1 MB but there is a problem since the file wants me to load its segments at 1 MB resulting in overwritten tables and data that I still would need in that moment to complete the loading process. I was thinking about moving it further away but that would limit my overall kernel size. (ISA hole remember?)

[iansjack]

If your binary doesn't contain the .bss, and doesn't contain any meta-data describing it, how does your loader know how much memory it needs to set aside? And how does it know what range needs to be zeroed?

Have you tried stripping your elf file yet?

[Octacone]

If your binary doesn't contain the .bss, and doesn't contain any meta-data describing it, how does your loader know how much memory it needs to set aside? And how does it know what range needs to be zeroed?

Have you tried stripping your elf file yet?

Did you see my last reply. I showed you how, look it up. Maybe you missed it.
Well yes, I only stripped the binary (.code + .data + .rodata + .bss) part.
Btw writing an ELF loader in assembly is a huge pain. I can do it but will take me a lot of time and debugging.