Moving to PAE-paging

[bluemoon]

You can mix 4K pages with 2M or 1G pages, but then it wouldn't be possible to map all structures at once with 1GB address space. (You can however still map them indirectly)

That is true but...

or you end up need 256G space to map the whole structure

...it is not the choice between the 1 GB and 256 GB. Maybe the amount of needed 4K pages, in addition to 2M pages, is not very significant. Then the size needed for page tables would be like 1.1 GB at most.

In that case you need a lookup table, which is not direct mapping we talking about.

[Antti]

OK, I should read the thread again. I admit that I do not understand what you mean.

I was thinking that if the choice was between the 4K or 2M pages, maybe it would be reasonable to use 4K pages, for example, to map memory area below 4 GB and use 2M pages above 4 GB. This would be "a static page table structure". Or something like that.

OK, I should read the thread again.

That would be wise.

[OdinVex]

So I can use 2MB pages and 4KB pages to fine-tune requests?

[Antti]

So I can use 2MB pages and 4KB pages to fine-tune requests?

At least I do not know what would prevent you from doing so.

[rdos]

Update: A large part of the "lazy" code that used the process page selector has now been made "well-behaved" and use syscalls to manipulate page tables instead. The remaining occurences of direct accesses probably are harder to remove as they are more "reasonable", so I'm now a lot closer to the goal of moving to PAE paging.

Edit: Basically all accesses to the process page selector has now been removed (or moved to the paging-sensitive module). There are 16 different procedures that needs to be ported to PAE-paging (probably a few more would be needed as the last adjustments are made).

The tricky thing could be to remove the references to the page directory, which I seem to have used at a few places (10-15 ín total).

[rdos]

The number of page table functions have grown to 25 now, but most are simple to port to PAE. I've also removed all direct references to page tables from the page fault handler, which was not that easy.

Basically, the paging setup code needs converting + perhaps 10-15 more references to kernel page tables.

I'm not sure if I'll try the switch on real hardware, or if I'll try to setup Bochs. This seems like a major switch that can cause a multitude of problems, but still I've debugged such before on real hardware.

[rdos]

All page-table code is now concentrated to a single module (ptable.asm). Except for a few exotic places that don't matter at the moment (crash-debuger and process termination). This was done with incremental changes, which proves this concept works for complex changes like page-table structure. Now I only need to write the PAE supporting functions, and test them (the latter will be the big problem as I more or less expect some problems that kills the boot).

The next step now is to setup identity-mapping in PAE mode (which I can use my pre-tasking panic handler to debug).

[Brendan]

Hi,

All page-table code is now concentrated to a single module (ptable.asm). Except for a few exotic places that don't matter at the moment (crash-debuger and process termination). This was done with incremental changes, which proves this concept works for complex changes like page-table structure.

You added SMP support, then you added ACPICA, then you added PAE, and then you're going to add 64-bit support. Do all of these changes and when you've finished work out how long it took to do all of these changes incrementally. Then, go back to the code you had before you started SMP support and use it as a reference while rewriting everything from scratch, and find out how long that would've taken. This is the only way of proving which method would've been the better/faster approach in the long run.

However, this still wouldn't really be a fair comparison - I can almost guarantee that if you'd started from scratch you would've also fixed a lot of other problems that are still in your "incremental" version. For an example, if I were starting RDOS from scratch I would've (intentionally) broken compatibility and ripped out all segmentation; and (especially for the 64-bit kernel) would've ended up with a much cleaner system than your "incremental change" approach ever will.


Cheers,

Brendan

[rdos]

I haven't put a lot of time into PAE. In fact, I haven't had time to do anything for almost a month when I last posted. I ancipate maybe I've used 10 hours on PAE this far, and maybe I'll need another 10 hours to be able to run RDOS with PAE paging. The only thing I've done is to write a rather complex device-driver for a new terminal hardware platform.

[rdos]

The first step in PAE-paging is now working. I can now setup PAE paging mode without the PC tripple-faulting. Didn't take a lot of time to do, but then I've not yet touched the PAE-specific interface, but just initialized page tables and physical memory.

[rdos]

Lots of progress with PAE-paging. I've now implemented & tested a little more than half of the functions, and the whole device-driver initialization now works with PAE paging. I've got one rather complex function left, process creation (and termination), which is a little tricky since it must also create / delete the 3:d paging level.

Edit: Now everything works until the first reload of CR3 by the scheduler.

[rdos]

Now the whole boot-process works, and the command shell is started with PAE-paging. However, there is still some bug(s) because certain operations seem to hang the entire OS (for instance, switching input focus doesn't work, neither does the remote debugger).

Edit: Still some minor issues, but now both the kernel debugger, remote debugger. and switching logic works, so the remaining issues should be relatively easy to fix.

[rdos]

Done. Everything seems to work after a final fix.

It took about 50 hours to implement and debug PAE-paging in RDOS.

[OdinVex]

These thoughts correct? Not sure if I grasp paging right. Been busy, haven't been able to come back to OS Dev for a while.

[ PML4 ][ PDP ][ PD ][ PT ] 512 * 8b = 4KB (4K Paging) Per Table

[ 0 ] [ 0 ] [0] [ 0-511 ] 16KB Overhead size to map 2MB
[ 0 ] [ 0 ] [ 0-511 ] [ (16KB * 512) ]: 8MB Overhead size to map 1GB
[ 0 ] [ 0-511 ] [ (8MB * 512) ] [ (16KB * 512) ]: 4GB Overhead size to map 512GB

[Combuster]

your math is a tad off:

4 tables = 16k to map 2MB aligned to 2MB at random locations
(3 + 512) * 4k = ~2MB to map 1GB.
(2 + 512 * 512) * 4k = ~1GB to map 512GB. However you should've used PSE by the time you get there, which will probably provide you with the ability to save over 99% of your overhead. Actually, mapping 1GB on the right processor may cost as little as 8k overhead.