Loading 64-bit executables in RDOS
Posted: Tue Dec 25, 2012 9:08 am
OK, so I suppose the 64-bit GCC executables are ok at this stage (at least from examining headers and disassembling them). Now it is time to load and execute them! 
The first problem is how to fit them into a DOS/Windows centric environment, which allows chain-loading different types of executables. Actually, from command line, there are two different ways to load a new executable: 1) chain in same process 2) create new console and process and run them there (spawn). The first way will not be supported by 64-bit executables, primary because 64-bit needs a special 64-bit process, and secondary because everything below 4G in a 64-bit process is inaccessible from user-mode (that's why I needed the medium memory model).
Thus, the first thing would be to make the command-shell recognize 64-bit applications (this is done with a new syscall), and then always spawn them regardless of how they are invoked.
The next thing is to check for 64-bit executables in the spawn function (same syscall as above), and create a 64-bit process instead, and also skip a few steps there.
Edit: In segmented designs it is easy to get to a specific position simply by loading a fixed selector, and using it 0-based. The same doesn't work so well in flat mode, and often needs relocation. But for 64-bit an interesting alternative is to use the first linear page, especially since it is not accessible to applications anyway. So by loading the flat selector to ds, and using 0-based addresses, I can address the exact same locations from both protected mode and long mode without using index / base registers. I'll put the executable info at linear position 0, and let it reference pointers in the low 3G address space.
The first problem is how to fit them into a DOS/Windows centric environment, which allows chain-loading different types of executables. Actually, from command line, there are two different ways to load a new executable: 1) chain in same process 2) create new console and process and run them there (spawn). The first way will not be supported by 64-bit executables, primary because 64-bit needs a special 64-bit process, and secondary because everything below 4G in a 64-bit process is inaccessible from user-mode (that's why I needed the medium memory model).
Thus, the first thing would be to make the command-shell recognize 64-bit applications (this is done with a new syscall), and then always spawn them regardless of how they are invoked.
The next thing is to check for 64-bit executables in the spawn function (same syscall as above), and create a 64-bit process instead, and also skip a few steps there.
Edit: In segmented designs it is easy to get to a specific position simply by loading a fixed selector, and using it 0-based. The same doesn't work so well in flat mode, and often needs relocation. But for 64-bit an interesting alternative is to use the first linear page, especially since it is not accessible to applications anyway. So by loading the flat selector to ds, and using 0-based addresses, I can address the exact same locations from both protected mode and long mode without using index / base registers. I'll put the executable info at linear position 0, and let it reference pointers in the low 3G address space.