OSDev.org

Hey all,

I've started working on my image loader and investigating exe formats etc.
Obviously the industry standards like ELF64, PE+ exist.. what I'm trying to weigh up is whether it would be most advantageous to just stick with one of these or write a new one from scratch.
Pro's of going with an existing one is that I can use existing dev tools to create executables.. the biggest con is I had in mind a very different way of approaching dynamic libraries, executables and machine services which these formats wouldn't support. Perhaps it would be best to support both... a standard at first to get going, then look at a newer model afterwards.

That being said .. I considered what would be required to say for example use FASM to create a flat image, then be able to relocate that image.. it seems like it would extremely difficult, possibly full of potential problems without having relocation information generated at compile/asm time.. for example:

mov rdi,0
mov eax,[rdi]

The 0 could refer to a fixed memory address, or something which requires fixup.. and it would be difficult to tell in all circumstances.. perhaps I'm missing something? In any event it seems like this would a bad idea.

So that would leave only one option, use an existing exec format that FASM can output with fixup/relocation information... Which seems like a good route for now, so considering that my os is 64bit only, would ELF64 or PE+ be the better option?
(I realise i sort of answered my own question while writing this post.. but it serves as a sanity check and if anyone can find fault with my logic great

anyone ?

so basically it comes down to 3 choices

1) ELF64
2) PE+
3) Use flat binary images from FASM and somehow build relocation info for it

I use ELF (32/64) for my OS, since it is widely supported by compilers / linkers and highly portable across different architecures. It simply fits my needs

So ive decided not to use pe or elf...(feeling brave)
So the question now is.. assuming i'm using fasm for now, should i ouput as MS COFF 64, and write my own linker... or the more daring approach (if possible) write the image out as a flat binary, then have something similar to a linker/post-processor to allow it to be relocatable...?

It's a pity that ALL 64bit code couldn't be rip-relative and everything would be automatically relocatable.. sigh.. perfect world

So what exactly about ELF and PE makes them unsuitable for what you're doing? I've found that ELF has way more features than I need, although I haven't got much of an execution environment yet. Even a modification on one of those formats might be useful, because you could make some patched versions of GNU binutils instead of completely rewriting everything.

Of course, if you're just writing another executable format just to try it, more power to ya. I mean, nobody would write an OS if they didn't have that kind of attitude.

As for actually implementing the linker, I think it should be reasonably easy, although it technically wouldn't be flat binary if you set up some sort of relocation system, even temporarily. You would have to have some sort of symbol table as well, even if it is just one. Another pitfall would be the fact that code and data segments in the object files must stay the same relative to each other, because they are in one segment, which could make for some confusing relocation setups. You also couldn't easily set up read/write/execute flags to be interpreted by the loader without separate code and data. I'm no expert on the linking side though.

Well my main reason would be that... as you pointed out, the idea behind building an OS is to really "DIY". So it would be great to do everything yourself.
The one thing I was still checking out with ELF is that in some of the docs they say that relocation is made possible by the elf format using an IA32/64 register up, which if true is something I'd not want to have at all. PE I would avoid simply because I don't want any left over windows remnants or mz stubs etc, and to process them out would be almost the same as having my own linker in the first place (I think).

The one thing I was looking at doing with my OS, and I guess ELF wouldn't prohibit this in anyway is to try and unify the concept of a library, executable and a service/daemon.
For example you could type the name of a library from a shell and it would essentially run the init/entry-point code and load the library into the OS'es library manager.. the same with a service.. so that unlike windows there shouldn't be any extra effort in creating services and such.. it would almost bring back the concept of a TSR.. so both a shared lib and service would run the "app" part which would then terminate leaving the library behind or the service running..

In terms of ELF I'm thinking it wouldn't be a bad idea to implement it EVEN if I do land up doing my own thing as well.. it can't hurt to have different loaders. FASM directly generates a .o file (im running under Windows btw) so I haven't gotten a linker yet for elf format... What would be the best place to get one , GCC binutils for windows or something?

I think I see what your idea is. You have services implemented as a sort of dynamic library that can be loaded whenever a call to a specific service is made, and have code to set themselves up, right? That's pretty similar to a microkernel model, where services/servers lay dormant until they are called, but are normal programs. However, your way might be faster, especially if the libraries can be loaded alongside programs. Or maybe I have it all wrong... I'm not too familiar with Windows' concept of services.

I would recommend using GNU binutils if you're going to be linking ELFs, and especially if you want to make your own extensions, because it's open source. The object files FASM outputs could actually be any format... you need to make sure it outputs ELF objects before you can do ELF linking. I know that NASM has the -felf flag that makes it use ELF, but you'll have to read FASM's documentation. It's also not very hard to make an ELF loader for a kernel - mine's only about 100 lines of C.

Also, just because the point of a new OS is to do new stuff, it doesn't mean that standardized formats can't be really helpful, because there are usually many good, bug-free tools out there to manipulate them. For example, my initrd format is just a simple TAR archive. It's best to use something standard if it doesn't get in the way of the design you want to make. It also really helps in bootstrapping from another system, because you don't have to rewrite the toolchain or anything

Maybe you should look at the RDOFF2 file format. It is generated only by NASM from asm code. It is vesy simple to implemet. It have relocations and import/export symbols to simple header. In NASM you can generate code 32/64 bit.

johnsa wrote:It's a pity that ALL 64bit code couldn't be rip-relative and everything would be automatically relocatable.. sigh.. perfect world

Why can't it? All my code is position independent. Even the kernel itself is “linked” at 1000h, and then later paged up in high memory and simply jumped to.

I plan to implement libraries as a simple system call that copies the code into some given page-aligned address, with a reference-counted read-only code segment, a data segment, and a TLS, all a fixed distance from the library (and thus able to be rip-relative).

Programs will all be loaded at 2M and run their own initialization, and because the program itself specifies where the libraries shall be, no linking is needed, it simply loads a generated header file with the exported procedures, adds whatever offset is needed, calls the kernel to load it there, and uses it. Of course every program using the library must be reassembled if it changes, but that can easily be fixed by padding, being smart about where you place the exported routines so all changes happen at the end, or a version number, most likely a combination of all three. Libraries shared by the application and the library? Simply mapped both places.

Sounds like a good idea?

You could for a simple format, just load ebx with load address and then add it for thoughs address that need it.
You can even use this trick in fasm

Nick thats sort of the plan yes, in terms of implementing services as normal shared libraries/executables.

I've started working on a ELF loader, FASM can output straight to ELF executable format without a linker it seems.. which is fantastic

Amethana, I'm not sure how you got that right exactly, for example mov al,[aVariable] is RIP-relative... but mov al,[aVariable+rdi] isn't... which means to make all your code rip relative you'd have to pay really close attention all the time and avoid most of the helpful addressing modes?

On the ELF file side the format does seem to be really simple, i made a small test app with fasm and outputed to elf exe, then manually went through the file to validate it against the spec.. it seems 100% the only thing that i'm not 100% sure of is the following:... there is the file header, 2 program headers for code and data, then the code.. there doesn't seem to be any sections, and no relocation symbol tables or anything.. here is the breakdown:
the binary elf exe:

7F454C46020101000000000000000000
02003E0001000000B000000000000000
40000000000000000000000000000000
00000000400038000200400000000000
0100000005000000B000000000000000
B000000000000000B000000000000000
0D000000000000000D00000000000000
00100000000000000100000006000000
BD00000000000000BD10000000000000
BD100000000000000200000000000000
02000000000000000010000000000000
4831FF48FFC78A87BD100000C323


which leads to a manual break-down:

;elf_header:
; db 7fh,'ELF'
; db ELFCLASS64 ; EI_CLASS (1 for 32-bit, 2 for 64-bit objects)
; db ELFDATALSB ; EI_DATA encoding (1 little-endian, 2 big-endian)
; db 1 ; EI_VERSION (has the value EV_CURRENT, which is defined with the value 1)
; db ELFOSABI_SYSV ; EI_OSABI
; db 0 ; EI_ABIVERSION
; times 7 db 0
;
; dw ET_EXEC ; e_type Object file type (ET_EXEC 2 Executable file)
; dw 3Eh ; e_machine Machine type
; dd 1 ; e_version Object file version
; dq 0xb0 ; e_entry Entry point address
; dq 0 ; e_phoff Program header offset
; dq 0 ; e_shoff Section header offset
; dd 0 ; e_flags Processor-specific flags
; dw 0 ; e_ehsize ELF header size
; dw 0 ; e_phentsize Size of program header entry
; dw 2 ; e_phnum Number of program header entries
; dw 40h ; e_shentsize Size of section header entry
; dw 0 ; e_shnum Number of section header entries
; dw 0 ; e_shstrndx Section name string table indexehdrsize = $ - $$
;
;prog_headers:
; dd PT_LOAD ; Segment Type
; dd PF_R OR PF_X ; Segment Attrib. Read/Execute
; dq 0xb0 ; p_offset contains the offset, in bytes, of the segment from the beginning of the file.
; dq 0xb0 ; p_vaddr contains the virtual address of the segment in memory.
; dq 0xb0 ; p_paddr is reserved for systems with physical addressing.
; dq 0xa0 ; p_filesz contains the size, in bytes, of the file image of the segment.
; dq 0xa0 ; p_memsz contains the size, in bytes, of the memory image of the segment.
; dq 0x1000 ; p_align specifies the alignment constraint for the segment. Must be a power of two. The values of p_offset and p_vaddr must be congruent modulo the alignment.

; dd PT_LOAD ; Segment Type
; dd PF_R OR PF_W ; Segment Attrib. Read/Write
; dq 0xba ; p_offset contains the offset, in bytes, of the segment from the beginning of the file.
; dq 0x10ba ; p_vaddr contains the virtual address of the segment in memory.
; dq 0x10ba ; p_paddr is reserved for systems with physical addressing.
; dq 0x01 ; p_filesz contains the size, in bytes, of the file image of the segment.
; dq 0x01 ; p_memsz contains the size, in bytes, of the memory image of the segment.
; dq 0x1000 ; p_align specifies the alignment constraint for the segment. Must be a power of two. The values of p_offset and p_vaddr must be congruent modulo the alignment.

ELFCLASS32 = 1 ; 32bit object,executable
ELFCLASS64 = 2 ; 64bit object,executable

ELFDATALSB = 1 ; Little-Endian Data Format
ELFDATAMSB = 2 ; Big-Endian Data Format

ELFOSABI_SYSV = 0 ; System V ABI
ELFOSABI_HPUX = 1 ; HP-UX operating system
ELFOSABI_STANDALONE = 255 ; Standalone, embedded application

; ELF File Types
ET_NONE = 0 ; No file type
ET_REL = 1 ; Relocatable object file
ET_EXEC = 2 ; Executable file
ET_DYN = 3 ; Shared object file
ET_CORE = 4 ; Core file
ET_LOOS = 0xFE00 ; Environment-specific use
ET_HIOS = 0xFEFF
ET_LOPROC = 0xFF00 ; Processor-specific use
ET_HIPROC = 0xFFFF

; ELF Segment Types
PT_NULL = 0 ; Unused entry
PT_LOAD = 1 ; Loadable segment
PT_DYNAMIC = 2 ; Dynamic linking tables
PT_INTERP = 3 ; Program interpreter path name
PT_NOTE = 4 ; Note sections
PT_SHLIB = 5 ; Reserved
PT_PHDR = 6 ; Program header table
PT_LOOS = 0x60000000 ; Environment-specific use
PT_HIOS = 0x6FFFFFFF
PT_LOPROC = 0x70000000 ; Processor-specific use
PT_HIPROC = 0x7FFFFFFF

; ELF Segment Attributes
PF_X = 0x1 ; Execute permission
PF_W = 0x2 ; Write permission
PF_R = 0x4 ; Read permission
PF_MASKOS = 0x00FF0000 ; These flag bits are reserved for environment-specific use
PF_MASKPROC = 0xFF000000 ; These flag bits are reserved for processor-specific use

The confusing part then is:

4831FF(xor), 48FFC7(inc),8A87BD100000(mov),C3 (ret),2324 (data)
now the mov uses address: 0x000010bd .. which is the address of data byte 23 (first byte in the data segment) (in terms of addressing mode it'll add rdi).. but the point is now that value should be absolute, non-rip relative and has been calculated from the base of the segment with alignment 1000h .. now i could remove the header "at 0" part.. but I still don't get a table that would allow me to relocate/patch these address if I load to a different location other than 0?

johnsa wrote:Amethana, I'm not sure how you got that right exactly, for example mov al,[aVariable] is RIP-relative... but mov al,[aVariable+rdi] isn't... which means to make all your code rip relative you'd have to pay really close attention all the time and avoid most of the helpful addressing modes?

Just make extensive use of lea.

I see your point its sneaky, but then its restrictive in coding style.. too much so for me ...
I like things like:
mov eax,[array+rcx*4] .. without wasting the extra register..

I may be missing something, but I think ELF supports executing dynamic libraries as programs. I seem to recall running "/lib/libc.so.6" on a Linux system and then getting output on the console (just some stuff about the library version and compile flags). In principle, it should be possible to extend this to running arbitrary executables.

I think the reason you can run /lib/ld.so.* is that, because it is run before the loaded program, it has an entry point. If you just execute it, it starts at that entry point and must have a way of knowing that you directly executed it, so it just displays some info. I don't think it's really specific to ELF though, beyond the fact that it *has* an entry point.