Adding 64-bit support to RDOS

[rdos]

In long/64-bit mode, any stack fault in user mode is going to be reported as a page fault (being as its the only method of bounding the stack) as with any other addressing fault. In kernel mode, likewise, but being as the page fault handler would be unable to push anything to the stack, it would turn into a double fault.

Right, I didn't think about that. And using an IST for page fault handler would not be a good idea since its more or less valid for page fault handler to generate page faults (and thus reeenter itself, albeit with a different page fault address).

Erm, just make sure your kernel stacks are big enough?

There are situations where exception handlers (especially protection fault handler) would invoke themselves in a loop and use-up all kernel stack. I want to detect those and provide a usable way of analysing what went wrong.

Come to think of it, I must allocate guard pages for kernel stacks (above and below) in situations where a flat stack might be used, like when SYSENTER interface is used, and for long mode tasks. That would waste two pages of linear address space per task, but I cannot see any other way of solving that issue.

[bluemoon]

If you are worry on the address space usage, it may work to map different memory for kstack of different process/thread into same address range but I can imagine a difficult time for debugging.

Perhaps you are looking for one kernel stack per core design.

[rdos]

I've been fiddling with the 32-bit TSS in the design. Even though I removed much of the dependency on the 32-bit TSS a while back, and copied relevant fields in the syscalls for the debugger, it seems like there is still a dependency, and the TSS is mapped to the same area as the thread control block, and thus I cannot change the initial part, which I have to fix.

Edit: The 32-bit TSS now is more or less completely removed. It still exists for kernel stack and IO-permission bitmap, but it is not required for a thread, and it doesn't share anything with the thread control block.

The crash debugger now can list the 64-bit state when the core is running in long mode, and has some automatic detection of mode. There are some more work there as well, but at least I can now use the CrashGate syscall in long mode and get an interactive listing of machine state which allows inspection of memory content as well. The NMI IPI for long mode also seems to be functional.

[rdos]

If you are worry on the address space usage, it may work to map different memory for kstack of different process/thread into same address range but I can imagine a difficult time for debugging.

Perhaps you are looking for one kernel stack per core design.

I once had kernel stacks at a fixed location. There were some additional issues with debugging because of that design, but OTOH, those functions are pretty handy even after I left that idea. The main problem with the design was that it required a TLB-flush for each task switch.

I won't be using a kernel stack per core, as I will still have most of a functionality in a "monolithic" kernel.

This new project does offer lots of interesting new design ideas and possibilities, which make it an interesting educational challenge, in addition to a major feature update. But if I was to rewrite everything from scratch, the educational challenge would be minor, as most of the time would be reinventing the wheel, and only a little of it would be to implement new ideas. Researching how a microkernel could be implemented in RDOS also is an interesting design idea I'll research when I'm ready to write my first user-mode 64-bit code. I already have a synchronous IPC mechanism, but I'm not sure it is best for use in a microkernel design.

[rdos]

OK, time for the central (?) syscalls for switching between modes.

They are really simple, and shouldn't take much longer than ordinary CR3-reloads to execute:

First, the long mode driver needs to remember both protected mode and long mode IDT tables. This is done like this:

Code: Select all

prot_idt_size   DW ?
prot_idt_base   DD ? 

long_idt_size   DW 0FFFh
long_idt_base   DD IDT_LINEAR

The protected mode IDT address is peeked at device-driver startup-time, and saved with sidt fword ptr ds:prot_idt_size. The long mode IDT resides in the driver itself, and is also initialized at device-driver startup-time, and with some syscalls called from other drivers.

So, here are the procedures for switching between modes: (both use EAX for the new CR3)

Code: Select all

switch_to_long_mode   Proc far
    push eax
    push ebx
    push ecx
    push edx
    pushf
;
    mov ebx,eax
    cli
;
    mov eax,cr0
    and eax,7FFFFFFFh
    mov cr0,eax
;
    mov ecx,IA32_EFER
    rdmsr
    or eax,100h
    wrmsr
;
    mov cr3,ebx
;
    mov eax,cr0
    or eax,80000000h
    mov cr0,eax
;
    lidt fword ptr cs:long_idt_size
;
    popf
    pop edx
    pop ecx
    pop ebx
    pop eax
    ret
switch_to_long_mode  Endp

switch_to_protected_mode   Proc far
    push eax
    push ebx
    push ecx
    push edx
    pushf
;
    mov ebx,eax
    cli
;
    mov eax,cr0
    and eax,7FFFFFFFh
    mov cr0,eax
;
    mov ecx,IA32_EFER
    rdmsr
    and eax,0FFFFFEFFh   
    wrmsr
;
    mov cr3,ebx
;
    mov eax,cr0
    or eax,80000000h
    mov cr0,eax
;
    lidt fword ptr cs:prot_idt_size
;
    popf
    pop edx
    pop ecx
    pop ebx
    pop eax    
    ret
switch_to_protected_mode  Endp

Both the switch-procedures must execute in unity-mapped memory, since they disable paging as part of setting up the new mode.

Edit: Both procedures are tested with the crash-debugger, and works.

[bluemoon]

Are you sure not reloading GDT will not cause issue if a segment register is reloaded and the table is not in cache? This may be an issue unless GDTR contain linear address that is same in both protected mode and long mode.

They are really simple, and shouldn't take much longer than ordinary CR3-reloads to execute:

It depends on how you define much longer.

[rdos]

Are you sure not reloading GDT will not cause issue if a segment register is reloaded and the table is not in cache? This may be an issue unless GDTR contain linear address that is same in both protected mode and long mode.

There is no special GDT for long mode, and I don't use any special entries in the GDT for long mode apart from the special code selector. The selectors loaded into other segment registers don't matter, but typically I will load selector 0x20 which is a flat data selector. For the ss register, I'll load the same selector as is used by the SYSENTER interface (0x1A8). I've also decided not to use the long mode equivalents of LDT and TSS, but instead reload the TR and LDT in protected mode only, using the 32-bit protected mode versions of the TSS and LDT. The descriptors that differs are only in the IDT, and I have completely different IDTs between protected mode and long mode. Some of the descriptors (call gates for instance) are not valid in long mode, so I have to ensure that those are not used, which is actually simple since only 32-bit applications gets them patched into their code, and only if the SYSENTER interface is not used.

Besides, each core has its own GDTR, which is used to be able to access the core data area by loading a fixed descriptor, so using only one GDT for long mode would not work.

[rdos]

Now there is only one issue left before trying to introduce 64-bit threads to the scheduler. Since all calls to long mode must be from code using a flat stack, I need to change the stack reloaded by the scheduler to a flat stack. That would have no problems AFAIK, so should be straight forward. Debugging 64-bit threads also should be pretty easy, since I can plant CrashGate syscalls in the scheduler to get a freeze of core states that I can inspect.

Edit: The scheduler now works with a flat stack.

[rdos]

They are really simple, and shouldn't take much longer than ordinary CR3-reloads to execute:

It depends on how you define much longer.

A rough estimate is that it takes around 1-2us to execute on a modern, low-end CPU. That's because the paging enable/disable code in the previous boot, which did this operation once per 4k page, and which executed 1 million times per 4G, takes around a second to execute for 2G RAM, but is considerably faster on some machines (AMD Athlon II) and slower on others (Intel Atom).

[rdos]

In regards to LDT and TR registers I've made some experiments now.

First, a suitable way to handle LDTR in long mode with no 32-bit applications executing (I'll execute those in protected mode), is simply to load LDTR with zero. This works, and causes all LDT descriptors to generate general protection faults.

Second, it seems like the 64-bit TSS have the same positions for the IO-permission bitmap as the 32-bit TSS, which is a little odd. The Intel manual claims one has to load TR in long mode, but I hardly think they implemented checks in the processor to make sure the current base address was really loaded in long mode as opposed to protected mode. The fact that the IO-permission bitmap is at exactly the same position indicate they wanted to have some compability, which probably means it's not necesary to load TR in long mode. However, the final test would be to program an IST, and trigger it, but I think I wait with that test until I have the scheduler running my 64-bit thread properly.

[Owen]

The fact that the IO-permission bitmap is at exactly the same position indicate they wanted to have some compability, which probably means it's not necesary to load TR in long mode.

I presume you mean the IO permission bitmap offset entry?

I suspect that was simply left in the same place because it meant they didn't have to move it (in other words, it made the logic inside the CPU simpler). I see no other reason for the utterly bizzare layout of the long mode TSS (e.g. all the 64-bit values being 4 byte misaligned... Which I normally solve by intentionally "re-aligning" the TSS)

[rdos]

The scheduler now loads the 64-bit registers of the new 64-bit process, and jumps to the entry point. I can see that the core TR register is loaded, and that LDTR register is zero. A few of the process initializators also operate correctly, but then there is a page fault when initializing the system handle table. I suppose next will be to move the proper functionality to the 64-bit page fault handler, which shouldn't be that hard to do.

I might want to map the full page-tables into the last entry in the top level page tables (PML4) so I can access the full page tables from the long mode page fault handler, rather than using the below 4G mapping. I also plan to load long mode device-drivers (IOW, executable loaders) just above 4G. I'll reserve the first (and last) entry in the top level page table for kernel only. The rest of the address space would be available to 64-bit applications.

While the 32-bit page fault handler support two paging modes (ordinary and PAE), I currently see no reason why to code an abstraction layer into the long mode page fault handler. I might just as well use the page tables directly instead, and thus speed-up the page fault handler. The only thing the handler cannot do directly is to allocate physical memory, which it must call a 32-bit syscall to do.

[rdos]

The long mode page fault handler now can handle enough of the faults to be able to run the initializers for the the long mode process, so now the long mode process hits the startup break-point.

Next, I need to handle the breakpoint instruction the proper way, IOW save the 64-bit register state, put the thread in the debug-list, and schedule a new thread. Once that's ok, I should be able to view it in the kernel debugger.

BTW, the long mode page fault handler uses the full page table mapping at 0xFFFFFF80 00000000 and above. It's pretty easy to code the base addresses for the directory (0xFFFFFFFF C000000000), ptr directory (0xFFFFFFFF FFE00000) and PML4 (0xFFFFFFFF FFFFF000) when CR3 is mapped in the last PML4 entry.

[rdos]

Handling the breakpoint fault in long mode was really a breeze. Because no 32-bit or 16-bit protected applications, or DOS-extender applications will ever run in the long mode environment, I could just drop all the legacy interrupt chaining and reflection logic of DPMI. IOW, because of the design decision to run protected mode applications in protected mode rather than in compability mode, I can make a clean implementation of long mode interrupts & exceptions.

I'll also drop application chaining in long mode, and simply run one application per process instead, so I don't need chained contexts, which is also really messy in protected mode. Since long mode applications run in their own linear address area, they cannot coexist with protected mode applications anyway, so there is no need for the chaining logic.

[SparrowOS]

My breakpoints are owned by a task. When the task is run, I insert its breakpoints; when it is suspended, I remove them so the next task doesn't hit them. I have a special task which call-backs to refresh the screen and that screws-up your debugging because it's a different task from the task that owns the window. I started doing an IDE debugger, but it's worthless.