OSDev.org

rdos wrote:This is pretty simple in a single-processor environment. Just reload cr3 (or use invldpg) and you're done. Now that my SMP scheduler is stable, this seems like the next logical "target" in order to provide a stable environment. The simplest solution would be to just send IPIs to other cores as cr3 is reloaded, and let those IPIs reload cr3 in the target core. Would this be an adequate solution, or would it need to be more precise? My current solution only reloads cr3. It doesn't use the more precise invldpg, as I had problems with this before.

• Do TLB lookup, get TLB miss
• Do cache lookup for page directory entry, get cache miss
• Do "fetch from RAM" for page directory entry
• Wait until RAM responds
• Do cache lookup for page table entry, get cache miss
• Do "fetch from RAM" for page table entry
• Wait until RAM responds

1. Some CPUs (e.g. Cyrix) do remember if a page is "not present", and you do need to invalidate TLB entries when you change a page from "not present" to "present".
2. If you use the "self-reference trick" (e.g. page directory entry that points to the page directory itself) to create a "paging structure mapping", then; if you change a page directory entry you need for flush up 1024 TLB entries for the effected area *plus* 1 TLB in the "paging structure mapping" area.

gerryg400 wrote:In the case where you are flushing the TLB because some memory has been de-allocated, you may also need some sort of reply from the other cores to ensure that they have completed their TLB flush before your memory manager re-uses the recently freed pages.

Brendan wrote:For best performance on single-CPU (for "P6 or later"), you should be using "global" pages (where possible) so that TLB entries for pages that are the same in all virtual address spaces aren't flushed when you change CR3. In this case, you have to use INVLPG to flush individual "global" pages (as reloading CR3 won't flush them) and if you must flush all TLB entries (e.g. you changed a lot of page directory entries in kernel space and flushing everything is faster than doing INVLPG thousands of times) you should toggle the PGE flag in CR4.

Brendan wrote:The main way of avoiding IPIs is called "lazy TLB invalidation". If using stale TLB information will cause a page fault, then the page fault handler can check if the page fault was caused by a stale TLB entry and invalidate the TLB entry itself, and you don't need to send an IPI. This means you don't need to send an IPI to other CPU/s if you change a page from "not present" to "present", or from "supervisor" to "user", or from "read-only" to "read/write", or from "no-execute" to "executable" (or any combination of these). That roughly halves the number of IPIs. You do get some page faults instead, but only when the CPU actually does have the info in it's TLB and only if the TLB entry is used (so, the number of page faults is a lot less than the number of IPIs you would've received if you weren't using "lazy TLB invalidation").

rdos wrote:

gerryg400 wrote:In the case where you are flushing the TLB because some memory has been de-allocated, you may also need some sort of reply from the other cores to ensure that they have completed their TLB flush before your memory manager re-uses the recently freed pages.

That shouldn't be a problem, as ISRs and code that disables interrupts are not allowed to free memory. This is because memory allocation/free uses critical sections, which ISRs are not allowed to use.

gerryg400 wrote:

rdos wrote:

gerryg400 wrote:In the case where you are flushing the TLB because some memory has been de-allocated, you may also need some sort of reply from the other cores to ensure that they have completed their TLB flush before your memory manager re-uses the recently freed pages.

That shouldn't be a problem, as ISRs and code that disables interrupts are not allowed to free memory. This is because memory allocation/free uses critical sections, which ISRs are not allowed to use.

It doesn't just apply to ISR's or the kernel. Let's say that user 2 threads are running at the same time on different cores and that they are using the same address space. If one thread frees a page, you need to make sure that that page has been removed from both (actually all) TLB's before it is added to the free list. This requires some sort of synchronisation after the IPI.

   for each active core
        SendFlushIPI
   mov eax,cr3
   mov cr3,eax

; it should be safe to assume that any non-ISR based code would not use stale TLB entries on any core at this point

   for each active core
       if (core.cr3 == cr3)
           SendFlushIPI

   mov eax,cr3
   mov cr3,eax

Owen wrote:True. But it is a security or stability issue if an application accesses (successfully) memory that it has freed that has been re-allocated to someone else. Especially if that someone else is the kernel.

rdos wrote:

Owen wrote:True. But it is a security or stability issue if an application accesses (successfully) memory that it has freed that has been re-allocated to someone else. Especially if that someone else is the kernel.

Yes. Perhaps a way to solve this is to place just freed physical memory on a temporary "hot" list, and not make it available for re-allocation until all cores have scheduled (1ms).

Brendan wrote:Hi,

rdos wrote:

Owen wrote:True. But it is a security or stability issue if an application accesses (successfully) memory that it has freed that has been re-allocated to someone else. Especially if that someone else is the kernel.

Yes. Perhaps a way to solve this is to place just freed physical memory on a temporary "hot" list, and not make it available for re-allocation until all cores have scheduled (1ms).

I typically acquire a spinlock, set a "number of CPUs that need to handle the IPI" counter, send the IPI, then wait until the other CPUs have decreased the counter to zero before releasing the spinlock and allowing the originating CPU to continue. It's probably not the most elegant way, but it works.


Cheers,

Brendan

rdos wrote:Besides, having IOPL set to 0 (which disallows applications from manipulating interrupt-flag and thus hindering IRQs) should be enough to ensure freed pages cannot be accessed from another thread.

1. Process 1, thread 1, running on CPU#1 frees the page (and TLB entry is invalidated on CPU#1, and the IPI is sent to other CPUs)
2. Process 2, thread ?, running on CPU#2 allocates the page and stores your passwords in it
3. Process 1, thread 2, running on CPU#3 still has an old TLB entry for the page (because it hasn't invalidated yet), and reads the passwords from process 2
4. CPU #2 receives the IPI, but it's too late