Loading 64-bit executables in RDOS

Question about which tools to use, bugs, the best way to implement a function, etc should go here. Don't forget to see if your question is answered in the wiki first! When in doubt post here.
Post Reply
rdos
Member
Member
Posts: 3306
Joined: Wed Oct 01, 2008 1:55 pm

Loading 64-bit executables in RDOS

Post by rdos »

OK, so I suppose the 64-bit GCC executables are ok at this stage (at least from examining headers and disassembling them). Now it is time to load and execute them! :D

The first problem is how to fit them into a DOS/Windows centric environment, which allows chain-loading different types of executables. Actually, from command line, there are two different ways to load a new executable: 1) chain in same process 2) create new console and process and run them there (spawn). The first way will not be supported by 64-bit executables, primary because 64-bit needs a special 64-bit process, and secondary because everything below 4G in a 64-bit process is inaccessible from user-mode (that's why I needed the medium memory model).

Thus, the first thing would be to make the command-shell recognize 64-bit applications (this is done with a new syscall), and then always spawn them regardless of how they are invoked.

The next thing is to check for 64-bit executables in the spawn function (same syscall as above), and create a 64-bit process instead, and also skip a few steps there.

Edit: In segmented designs it is easy to get to a specific position simply by loading a fixed selector, and using it 0-based. The same doesn't work so well in flat mode, and often needs relocation. But for 64-bit an interesting alternative is to use the first linear page, especially since it is not accessible to applications anyway. So by loading the flat selector to ds, and using 0-based addresses, I can address the exact same locations from both protected mode and long mode without using index / base registers. I'll put the executable info at linear position 0, and let it reference pointers in the low 3G address space.
rdos
Member
Member
Posts: 3306
Joined: Wed Oct 01, 2008 1:55 pm

Re: Loading 64-bit executables in RDOS

Post by rdos »

Some progress. I've provided allocation functions for buffers between 1-3G. These are now used for loading relevant information from the ELF-executable. I've also loaded the program table, which contains two entries for code & data, located at the correct addresses (0x180E0000000 and 0x18120000000). I can decode the program table in long mode. All I basically need to do now is to reserve these address-spaces, and do an iretq to user-mode.

While the kernel-debugger has gotten some pretty nice functions for long mode, like being able to show memory contents for any 48-bit linear address, there still is a problem with the disassembly. I would need to eventually support the most common 64-bit prefixes and such in order to get meaningful disassembly.

Edit: Just skipping the REX attributes makes the disassembler a lot better.
rdos
Member
Member
Posts: 3306
Joined: Wed Oct 01, 2008 1:55 pm

Re: Loading 64-bit executables in RDOS

Post by rdos »

Now the most important 64-bit operations are supported by the kernel-debugger's built-in disassembler. It handles sib-bytes and R8-R15 correctly. Interesting also to note that R12 and R13 are a little special regarding addressing.

Edit: OK, I've executed the startup-code of the 64-bit application, and I'm about to execute the first syscall (which is not yet supported). The user stack is allocated in it's own area, and has 1G padding. The code is demand-loaded by page fault handler on first access.
Post Reply