[Solved] Page fault in AMD SimNow! - possible caching issue?

[xenos]

This problem is driving me crazy for a few days already. My 64 bit kernel code runs perfectly fine on Bochs, QEMU and VirtualBox - but it crashes on AMD SimNow!. I use the latter because it provides the most accurate simulation of AMD CPUs, and I guess that's also the reason why it struggles with my code. So here's what I'm doing:

I have set up a PML4T at physical address 0x112000, so I have CR3 = 0x112000. I use a recursive mapping, so the last entry of the PML4T also points to 0x112000. (Actually it contains 0x112063 - present, r/w, dirty and accessed bits are set.) This means that the PML4T is mapped to 0xfffffffffffff000, the last page of virtual memory.

Now I would like to map a page from my kernel to virtual address 0xffff800000000000 - right at the beginning of the upper half of virtual memory. In order to do that, I need to create some lower-level paging structures for that address. So my kernel does the following (you may skip this - it's just to explain how I end up in the situation I'll describe in the next paragraph):

First, it notices that there is no page table for virtual address 0xffff800000000000. So it requests a free page from the page allocator, and gets one. This happens to be 0x0ff0f000 - somewhere close to the top of my 256MB RAM. It tries to map this page to 0xffffffc000000000 - this is where the page table for 0xffff800000000000 should appear by the recursive mapping I use. It finds that there is no page table for that address either, so it requests another free page, gets 0x0ff0e000 from the and tries to map it to 0xffffffffe0000000 - again, this address results from the recursive mapping, and it denotes both the page table for 0xffffffc000000000 and the page directory for 0xffff800000000000. Once again it turns out that there is no page table for 0xffffffffe0000000, so it requests yet another page, gets 0x0ff0d000, and tries to map it to 0xfffffffffff00000. Now finally, this mapping succeeds. The kernel adds an entry to virtual address 0xfffffffffffff800, which is mapped to physical 0x112800 - the middle of the PML4T. So finally all the other mappings should succeed and I end up with the following (this is what SimNow!'s debugger tells me as well):

Code: Select all

Virtual               Physical    Content
0xfffffffffffff000 => 0x00112000: 0x0000000000112063
0xfffffffffffff800 => 0x00112800: 0x000000000ff0d165
0xfffffffffff00000 => 0x0ff0d000: 0x000000000ff0e165
0xffffffffe0000000 => 0x0ff0e000: 0x000000000ff0f105

So, if you go through the page mappings, you should conclude that virtual 0xffffffc000000000 is mapped to physical 0x0ff0f000, right? And setting the first entry at 0xffffffc000000000 to some address should map this address to 0xffff800000000000? Well, on Bochs, QEMU and VirtualBox this is exactly what happens, and the mappings are exactly the same as here. (Except that the physical addresses are a bit different.) However, on SimNow! I get a page fault as soon as I try to write something to 0xffffffc000000000 - this is also the address I get reported in CR2. If I ask SimNow!'s debugger for a memory dump at virtual 0xffffffc000000000, it tells me that this page is not mapped.

So, if anybody has a clue what's going wrong, please let me know. I thought about a caching / TLB issue, so maybe I'm doing something wrong when it comes to TLB invalidation? However, I don't know how this can happen. I never access 0xffffffc000000000 before, so there should be no TLB entry. And even if there were a TLB entry, it should be cleared, because I do an "invlpg [rdx]" with rdx = 0xffffffc000000000 between the memory writes to 0xffffffffe0000000 and 0xffffffc000000000 - so that should clear the TLB entry for that address, right?

Now I'm really at wit's end. Does anybody have a suggestion what I should check next? Is there a way to dump the cache contents on AMD SimNow!? I searched the web, but I could not find any.

Edit: The paging code can be found in X86Pager.cpp and X86Pager.h.

[bluemoon]

First, it notices that there is no page table for virtual address 0xffff800000000000. So it requests a free page from the page allocator, and gets one. This happens to be 0x0ff0f000 - somewhere close to the top of my 256MB RAM. It tries to map this page to 0xffffffc000000000 - this is where the page table for 0xffff800000000000 should appear by the recursive mapping I use. It finds that there is no page table for that address either, so it requests another free page, gets 0x0ff0e000 from the and tries to map it to 0xffffffffe0000000 - again, this address results from the recursive mapping, and it denotes both the page table for 0xffffffc000000000 and the page directory for 0xffff800000000000. Once again it turns out that there is no page table for 0xffffffffe0000000, so it requests yet another page, gets 0x0ff0d000, and tries to map it to 0xfffffffffff00000. Now finally, this mapping succeeds. The kernel adds an entry to virtual address 0xfffffffffffff800, which is mapped to physical 0x112800 - the middle of the PML4T. So finally all the other mappings should succeed and I end up with the following (this is what SimNow!'s debugger tells me as well):

I'm not sure on PML4T, but for 32-bit 2-level recursive paging, I can just write to 0xFFC00000 + (address>>12)*4,
if PTE itself is not present, a #PF is generated and the PF handler allocate the page if address lies on the page table range and U/S flag indicates supervisor mode. It occasionally induce a few more INTs/IRETs but it on the other hand save you from unconditionally checking the directory for present, and should works on any level of paging.

[xenos]

I'm not sure on PML4T, but for 32-bit 2-level recursive paging, I can just write to 0xFFC00000 + (address>>12)*4,
if PTE itself is not present, a #PF is generated and the PF handler allocate the page if address lies on the page table range and U/S flag indicates supervisor mode. It occasionally induce a few more INTs/IRETs but it on the other hand save you from unconditionally checking the directory for present, and should works on any level of paging.

Good point, I should try that. It should work similarly for the 4-level x86_64 paging (with 4kB pages), by just writing to 0xFFFFFF8000000000 + ((address >> 12) & 0x0000000FFFFFFFFF) * 8, where the "&" cuts off the 16 bit "sign extension" for canonical x86_64 addresses (i.e. the upper 16 bits must be equal to bit 47). I just need to make sure that the #PF handler is re-entrant, since there might be up to 3 #PFs in a row (missing PT, missing PD, missing PDP - the PML4T is already present, of course).

However, I'm still wondering why the #PF happens at all, since I already mapped a page at the PT address. Thus, even though I like that idea, I'm not sure whether it solves the problem and I won't get another #PF after returning from the PF handler, which will then map another page, and so on, until I run out of memory. But anyway, I'll give it a try and see what happens.

EDIT: I just tried it and it works perfectly fine on Bochs, QEMU and VirtualBox. On SimNow!, I also get 3 #PFs for each missing level in the paging structures - followed by another #PF, just as before. But anyway, your suggestion helped me a lot, because now I saw that the error code for that last #PF is 0xB - indicating a "1" in one of the reserved bits. So now I know where to look for the problem! Thanks!

EDIT2: Indeed, that solved the problem! It turned out that I must set the "global" flag only for the lowest level PTE's, and that for any higher level this flag is reserved and must be 0. I changed my code a bit and now it works like a charm on SimNow! as well. Thanks a lot!