Paging in an SMP environment

[yorknez]

Hello,

I am trying to make my own x86_64 OS from scratch and I am currently working on initializing the APs of the machine.

It is going well, but I have a question which I can't really find the answer for (perhaps I haven't looked hard enough): Is it recommended (or even a good idea) to share the paging structures between cores?

I think it would make my life ten times easier not having to do new paging structures for each core and I just realized that mapping the pages for my global allocator to each core would be a big pain.. At the same time, I think sharing the structures could lead to some problems, for example, sharing would mean that multiple cores can modify the Accessed and Dirty bits, which I'm not sure whether it's good or bad?

Thanks!

[nullplan]

The typical approach is to share all pages according to virtual space. So if two cores are executing threads of the same process, they actually share even the PML4, since they completely share the virtual space. Otherwise, all processes share the kernel half anyway, so naturally all cores share those pages.

The accessed and dirty bits are of course a concern if you actually use them. As far as I know, resetting those bits is akin to reducing access (because a CPU causing a write access on a non-dirty page needs to set the dirty bit again), so they require TLB invalidation on all cores. With user pages, this can more easily be accomplished by just telling the other CPUs to schedule in another process. With kernel pages, you need to perform a TLB shootdown if you do that (which is where you lock a global variable for the address to invalidate, then send an IPI to each CPU in turn for them to invalidate the address, and wait for all of them to return before unlocking that variable again). Obviously this is time consuming, so you want to minimize the number of times you do that. For example, having a malloc implementation that never frees pages means you never unmap those kernel pages again. And then unmapping is basically only done in the rare cases that modules get unloaded or something.

[yorknez]

My paging situation is not very advanced at the moment, I basically identity mapped the first 4 GiB of memory (this is done in my bootloader) so I can access stuff more easily (I will probably change that sometime soon) and I have at a high address the heap of the allocator. Also I don't think I will be using the Accessed and Dirty bits in the near future.

Given this, I take it that my APs can just use the same CR3 as the BSP? For now I don't have a scheduler or processes, I'm just trying to figure out how to make the APs work properly and then I will do scheduling with each process having it's own virtual space as you said.

P.S.: Sorry for the late reply, I didn't get notified yesterday

[Octocontrabass]

Given this, I take it that my APs can just use the same CR3 as the BSP?

Correct.

P.S.: Sorry for the late reply, I didn't get notified yesterday

Maybe creating the topic doesn't automatically subscribe you to it? You should get a notification for this post since I quoted you.

[yorknez]

Yes, it seems that I was not subscribed automatically. Also, getting quoted did notify me.

Anyway, thank you guys for the help!

[rdos]

I think you should prepare for handling TLB invalidations at an early stage. If you set pages to non-present, or change their mapping, then you need to invalidate the page on all cores. This can happing in the heap allocator, or at other places. If you don't implement this early on chances are big you will write code that will malfunction later.

You don't need a scheduler to handle this. You can setup an interrupt that when triggered will invalidate pages (or reload CR3), and then you trigger it when pages are changed.

[yorknez]

I think you should prepare for handling TLB invalidations at an early stage. If you set pages to non-present, or change their mapping, then you need to invalidate the page on all cores. This can happing in the heap allocator, or at other places. If you don't implement this early on chances are big you will write code that will malfunction later.

Yes, I will probably do this next, I've seen the "TLB shootdown" mentioned a lot when researching this topic.

You don't need a scheduler to handle this. You can setup an interrupt that when triggered will invalidate pages (or reload CR3), and then you trigger it when pages are changed.

I saw that one way is to do an IPI that tells all cores to use `invlpg` on a certain page, I think I'll go for this as reloading the CR3 invalidates the whole TLB.

[rdos]

I have a procedure in kernel that takes a linear address and number of pages to invalidate. It's implemented in the scheduler and keeps a list per core of pages that need to be invalidated on the next TLB interrupt. If too many pages are requested, the core will use load CR3 instead. There is also logic that avoids the TLB interrupt if the target address is in userspace, and the core is not running in that process. The procedure will also do the invalidation on the current core by internally invoking the TLB interrupt handling code.

Note that most of the code will need to be protected by spinlocks or similar.

I think to begin with you might just define the invalidation procedure and let it send the interrupt to other cores.