OSDev.org

...

ngx wrote:The problem gets even more strange when I understand that the second test fails not because I map something in the first one, but only when I map and then write/read from the mapped page(in the first test).

It sounds like you're not flushing stale TLB entries. You need a MOV to CR3 or an INVLPG to remove the stale TLB entry after you change the page tables.

Section 4.10.4 of Volume 3A of the Intel SDM explains it in great detail.

...

You should INVLPG every time you reduce access or change the address. So if Present changes from 1 to 0, RW changes from 1 to 0, Super changes from 0 to 1, or the address changes at all. Otherwise INVLPG is not necessary. Be aware that a stale TLB entry may cause a spurious page fault, or at least Linux has code for detecting and handling those. That is, you may get a page fault even though the access was permitted. In that case, just return. According to the manual it shouldn't happen, but I cannot imagine the Linux guys writing that special case just for fun.

Which to prefer depends. INVLPG is more targeted, MOV to CR3 is more of a shotgun approach. Normally you want to use INVLPG when just changing single mappings, but MOV to CR3 can be the better option if you don't know precisely how much you are changing. For example, my kernel is booting with PML4[0] == PML4[256], and PML4[256] maps all physical memory. So I need to remove all TLBs that came from PML4[0], but I really don't feel like using INVLPG on every 2MB page in physical memory. And at that time, the Global bit is not in use. So MOV to CR3 is just the easiest way to flush all possible TLBs from the identity mapping.

...

nullplan wrote:Be aware that a stale TLB entry may cause a spurious page fault, or at least Linux has code for detecting and handling those. That is, you may get a page fault even though the access was permitted. In that case, just return. According to the manual it shouldn't happen, but I cannot imagine the Linux guys writing that special case just for fun.

On a SMP system, that can always happen if another CPU concurrently mapped a page (assuming that you let threads share page tables). Maybe the Linux code is meant to cover this case?

The Intel SDM volume 3A section 4.10.4.3 explains how stale TLB entries can cause spurious page faults. The Linux code I found for handling spurious page faults has a comment directly referencing that section of the manual.

I recall reading somewhere that some obscure x86 CPUs can have spurious page faults caused by stale TLB entries that aren't possible on Intel or AMD CPUs, but I have no idea if it's true. There are probably errata that can cause spurious page faults too.