OSDev.org

I'd like to know more about binary types like AOUT, ELF, and so on..
How do they differ ? Which is better ? Do I have to implement a new type for my new OS / is it possible ?

This was asked very recently here

you really shouldn't write your own binary format.
if you are using x86 use rdoff.
its very simple and it supports relocations and dynamic linking. there is a linker and example code for loading rdoff executables in nasm's source.
if you want more just use pe or elf. they are pretty good.
there are 64bit extensions for these as ELF64 and PE+

see http://www.wotsit.org/search.asp?s=binary
for specifications. the rdoff spec on wotsit is for the old version;
nasm sourcecode has rdoff documents but my advice would be reading example sourcecode.

writing a format is like writing an os altogether. ELF was the result of many months years of work. If you want some features, you could try and expand on an existing format (eg. Windows NE which basically expanded on DOS MZ).

Check the wiki for more information