Moving to PAE-paging

[OdinVex]

Would I be correct in assuming (64-Bit PAE Paging 4KB Pages) 8B X 512 X 512 X 512 to map 512GB? More efficient to map with 2MB Pages? Sorry if this is off-topic. Just want to be a footnote right now.

[rdos]

Making the assembly portions able to run in Long mode's compatibility submode. This would primarily involve removal of use of call gates and other system segment descriptors removed in long mode.

I expect this to work with no modifications. Call-gates are generated by fault-handlers on usage, and only for 32-bit applications. Most of the syscalls can eventually be supported by the sysenter method instead, which for most devices only require minor modifications (some require major, like if they use bp instead of ebp to access local variables). Calls in kernel are patched to far calls or push cs / near call, so thus don't use call gates. All syscalls are initially an invalid far call to selector 1-3.

Migrating the whole thing, over time, to C, or C++, or some other language in which things like pointer size don't get encoded into all my source code

I'm doing this gradually for more complex devices. A major undertaking is to eventually move VFS and filesystems to C. With that done, the code could easily be used both in 64-bit userland and as a 32-bit device-driver in kernel.

[rdos]

Would I be correct in assuming (64-Bit PAE Paging 4KB Pages) 8B X 512 X 512 X 512 to map 512GB? More efficient to map with 2MB Pages? Sorry if this is off-topic. Just want to be a footnote right now.

It would require 512G to map the page tables of entire address space (48 bits) into linear address space. However, since the 32-bit kernel doesn't need to access page tables for more than 4G, it is more convinient to do a mapping of only 4G which would require 4M with ordinary paging and 8M with PAE. Then there also needs to be the system page tables (that are global), which require the same amount of memory.

[rdos]

So you're saying that to add SMP support, all you did was replace CLI/STI with spinlocks; and you didn't redesign anything for scalability (e.g. change algorithms and data structures, to avoiding lock contention, reducing cache effects, etc)?

I would've expected that adding SMP support (if it was done properly - e.g. with attention to scalability) would've caused over 1/3rd of the kernel's files to have been modified on its own (but I guess you didn't do it properly, with no attention to scalability at all, and this is why so few files were changed).

It depends on how the existing multitasking functionality looks like. In a kernel that already takes advantage of multitasking in kernel, and uses kernel server threads, there shouldn't be major issues with switching to SMP. The goal in the transition was to be able to support the existing synchronization interface (which, IMHO, was well-thought-out) on SMP-systems. Another issue was code-patching, but that was also solved in a way that didn't require changes in drivers. The layout of the syscalls were changed, but since those are in an include-file, this change required no changes in drivers either.

[Brendan]

Hi,

I would've expected that adding SMP support (if it was done properly - e.g. with attention to scalability) would've caused over 1/3rd of the kernel's files to have been modified on its own (but I guess you didn't do it properly, with no attention to scalability at all, and this is why so few files were changed).

Theres a saying - great is the enemy of good.

Or - If I had infinite time, I'd implement a brilliantly scalable scheduler, have finely tuned lock granularity, and be using the optimal algorithms everywhere.

But I don't - and I mostly would like things to work. Therefore, I have a memory manager which is single threaded, my scheduler doesn't even attempt to minimise context switches, and I could probably make my memory barriers more precise and hence more efficient. But you know what? I'm fine with that.

But you're not using your OS as part of commercial products, the quality of your design and quality of your code is something that (currently) only effects you, and you're not being paid for your work.

This not just about segmentation alone. This is not just about SMP support alone. This is about systemic design and quality control problems. Go back through all of rdos' posts that provide details of parts of the OS and judge the quality of design for all the parts mentioned, then extrapolate that to parts that haven't been mentioned. Would you be comfortable telling companies/customers that this is an ideal foundation for their systems?

Now the "32-bit OS that was not written with portability or long mode in mind" has had support for 64-bit added to it in 8 hours. Do you think it only took 8 hours due to good engineering in the past, or do you think there's a tiny chance that it might have been a rushed hack that adds to the design and quality control problems?

I am convinced that (given the opportunity to make a clean start), rdos is capable of using hindsight to avoid some of the problems.

Because there were no existing 64-bit processes to worry about, support for 64-bit was a massive opportunity for rdos to make a clean break. This opportunity has been lost now. There are no more of these opportunities left (unless rdos decides to support ARM or another architecture, which isn't likely). The only thing left is excruciatingly slow and painful evolutionary change.

Rdos does deserve to be congratulated on his achievements. We should hold a wake to celebrate the demise of all hope.


Cheers,

Brendan

[rdos]

Now the "32-bit OS that was not written with portability or long mode in mind" has had support for 64-bit added to it in 8 hours.

Not so. I've used 8 hours to port the well-behaved code to use 64-bit physical addresses. There is probably another 8 hours or so before the less-well behaved code no longer directly uses the page tables. Then I estimate a few days to actually provide PAE-support. And with PAE-support, I definitely don't have 64-bit support added, more like a tiny bit of it.

I am convinced that (given the opportunity to make a clean start), rdos is capable of using hindsight to avoid some of the problems.

Absolutely, but I don't use the "start-over" method. I painfully (at least sometimes) gradually modify the code to make it cleaner. Having a clean API has been very important, and many changes in the past have targeted that problem.

In fact, if I had started the design by defining a new API, it would be very similar to the current, so why bother reinventing the wheel?

And I have no problem to modify code I wrote 10 or 20 years ago. It's almost as easy as if I wrote it yesterday.

Besides, Brendan, all the 64-bit code I'll be making eventually (after the PAE stuff is done), will be done from scratch.

I'm also pretty sure that I would not go by the standard design choices if I started from scratch. I would obviously have to leave-out segmentation, but I would not be likely to create a Linux cross-compiler and make a *nix clone. I would definitely want to reuse the idea behind the syscall-interface (which needs to be slightly redone for 64-bit code). Many of the virtual device interfaces would also be similar, although I might have coded some differently if I started from scratch. I would also keep the error code-less API.

[OdinVex]

Would I be correct in assuming (64-Bit PAE Paging 4KB Pages) 8B X 512 X 512 X 512 to map 512GB? More efficient to map with 2MB Pages? Sorry if this is off-topic. Just want to be a footnote right now.

It would require 512G to map the page tables of entire address space (48 bits) into linear address space. However, since the 32-bit kernel doesn't need to access page tables for more than 4G, it is more convinient to do a mapping of only 4G which would require 4M with ordinary paging and 8M with PAE. Then there also needs to be the system page tables (that are global), which require the same amount of memory.

Well I'm only writing for 64-Bit at the time being, want to know the maximum size required for only 512GB mapped at 4KB. Wonder if it'd be more efficient space-wise to utilize 2MB pages instead, or if they can both be used simultaneously. I appreciate the information. I thought it'd be 4K per PML4 table, 4K per PDP table, 4K per PD table, 4K per PT table (all 8B per entry, 512 entries each referencing 4K) so yeah 1G of tables (all combined in space) mapping 512G.

[bluemoon]

Current it sound less of concern on 64-bit mode to use 512GB or 1GB address space for page structures mapping. Using 2M page do have some benefit and drawback, but the save on 511GB address space is IMO considered negligible and do not account for the design choice.

However, to reduce the mapping to 1GB address space the system has to use all 2M page, which usually considered not flexible enough unless your system is specially tuned.

[Antti]

Has anyone used 1-Gbyte pages? I have not checked whether that feature is widely available or not in Long Mode.

Maybe there would be some use for such a big page size if doing Identity Page Mapping for some memory areas.

[OdinVex]

Current it sound less of concern on 64-bit mode to use 512GB or 1GB address space for page structures mapping. Using 2M page do have some benefit and drawback, but the save on 511GB address space is IMO considered negligible and do not account for the design choice.

However, to reduce the mapping to 1GB address space the system has to use all 2M page, which usually considered not flexible enough unless your system is specially tuned.

I don't understand what you meant at all. English isn't your first language?

Can anyone else shed light on my question and thought too, in hope of understanding?

[bluemoon]

I don't understand what you meant at all. English isn't your first language?

yeah and I lack of sleep yesterday, so let me rephrase it.

If you aim to use 1GB address space to map all the paging structures by using 2M page, you are restricted to use 2M page everywhere - ie. you can't mix with 4K page or you end up need 256G space to map the whole structure.

This lead to a drawback, the minimum block for allocation is 2M so you may waste a lot of memory for small allocations - if you are interested on the reason and pros&cons we can discuss it in other thread.

Most systems, if uses 2M or 1G page, support multiple-size allocation which can return 4K page.

[OdinVex]

Ah, so you can't mix-and-match, damn. In the end though, 512GB of memory can be mapped in 4KB pages using 1GB total?

[Antti]

If you aim to use 1GB address space to map all the paging structures by using 2M page, you are restricted to use 2M page everywhere - ie. you can't mix with 4K page or you end up need 256G space to map the whole structure.

What is preventing the use of 4K pages somewhere?

I would like to apologize in advance if I missed some obvious point you had here.

[bluemoon]

If you aim to use 1GB address space to map all the paging structures by using 2M page, you are restricted to use 2M page everywhere - ie. you can't mix with 4K page or you end up need 256G space to map the whole structure.

What is preventing the use of 4K pages somewhere?

I would like to apologize in advance if I missed some obvious point you had here.

You can mix 4K pages with 2M or 1G pages, but then it wouldn't be possible to map all structures at once with 1GB address space. (You can however still map them indirectly)

[Antti]

You can mix 4K pages with 2M or 1G pages, but then it wouldn't be possible to map all structures at once with 1GB address space. (You can however still map them indirectly)

That is true but...

or you end up need 256G space to map the whole structure

...it is not the choice between the 1 GB and 256 GB. Maybe the amount of needed 4K pages, in addition to 2M pages, is not very significant. Then the size needed for page tables would be like 1.1 GB at most.