Yet another paging question :P

[LaurensR1983]

Hi everybody...

So here's the situation: I'm about to implement paging. I understand the whole thing and most of my stuff is almost done. I only have one question though.

My kernel is physically loaded at 1MB, but I want to page it (or however you call it) at 2GB. So how does that affect other parts of the kernel? I can imagine it will cause some issues with the memory manager.

Like every memory block I allocate with malloc will make a small header at the beginning of each block, wich are connected through each other by using a heap structure. But should my memory manager map all that stuff in the new virtual memory space or the physical memory space? Should I create a way that when the kernel is doing it's stuff, the "normal" memory layout (1:1) is used instead of the new one? Malloc should only affect the pagetable directory of the process that is requesting the memory right?

Oh and one last question: should I turn paging on and off every time I want to point CR3 to another directory (by setting the flag in CR0) or can I just leave it turned on while changing CR3?

[Brendan]

Hi,

Like every memory block I allocate with malloc will make a small header at the beginning of each block, wich are connected through each other by using a heap structure. But should my memory manager map all that stuff in the new virtual memory space or the physical memory space? Should I create a way that when the kernel is doing it's stuff, the "normal" memory layout (1:1) is used instead of the new one? Malloc should only affect the pagetable directory of the process that is requesting the memory right?

Normally "malloc()" never sees anything except linear addresses, and doesn't use physical addresses for anything.

How other things work depends on what you want to do - for my OS, nothing is mapped 1:1 (everything is dynamically allocated), and the only things that care about physical addresses are the physical memory manager and device drivers that use bus mastering/DMA.

Oh and one last question: should I turn paging on and off every time I want to point CR3 to another directory (by setting the flag in CR0) or can I just leave it turned on while changing CR3?

Just change CR3 - you don't need to disable paging and enable it again.


Cheers,

Brendan

[LaurensR1983]

Hey thanks for your reply... it made things a bit clearer for me.

But how can the physical memory manager see the physical adresses if you have paging enabled? Should I temporarily change to a 1:1 map whenever the physical memory manager is doing it's work?

[JamesM]

There should be ONLY ONE time the memory management system needs to 'view' physical memory directly, and that's when copying memory from one VM space to another, such as with a fork() or when doing minor IPC. And also for copy-on-write stuff

All other times, the memory manager is only interested in what physical memory is available and not what is in the available physical memory. Because of this, it shouldn't have to ever view it.

Now, when copying memory from one VM space to another, You *could* use a different vm space which is identity mapped and copy it like that. That would be quite slow, as switching VM spaces flushes the CPU pipeline and its translation lookaside buffers.

The way I do it is I have a small asm snippet that:

Code: Select all

copyPagePhysical:
cli  ; disable interrupts, if we are interrupted here it would be 
catastrophic
mov ecx, [esp+4] ; get source frame start.
mov edx, [esp+8] ; get dest frame start.
push esi   ; We must save ESI so we don't trample anything
mov eax, cr0       ; disable paging
and eax, 0x7fffffff 
mov cr0, eax

mov esi, 0x1000  ; copy 4096 bytes

.loop:
mov eax, [ecx]    ; get source word
mov [edx], eax    ; store it at the dest word
add ecx, 4           ; go to the next source word
add edx, 4           ; go to the next dest word
sub esi, 4            ; 4 less bytes to copy
jnz .loop              ; if we still have stuff to copy, keep going

mov eax, cr0      ; enable paging again
or eax, 0x8000000
mov cr0, eax

pop esi
sti
ret

This may not be the most effective way of doing this, and I know that interrupts are disabled for all of the time. But I feel that this function gets called so infrequently it doesn't matter. It's not like I'm building a real-time system!!

JamesM

[LaurensR1983]

All other times, the memory manager is only interested in what physical memory is available and not what is in the available physical memory. Because of this, it shouldn't have to ever view it.

Hmmm never looked at it that way. Thanks for clearing that up.

Well I have finished my design for a basic paging implementation. I also have the minix book to help me out

Thanks for all the help so far!

[salil_bhagurkar]

Why does everyone want to shift their kernel to somewhere at the end of the RAM? Whats the point in doing so... I fail to find any reason...

[LaurensR1983]

It's not to the end of the RAM, but to the upper half of the adressable memory space

But anyway... You don't have to place the kernel in the upper memory part perse. The most important reason to use paging is to enable the usage of virtual memory (swap files etc).

[JamesM]

I use a lower-half kernel - I.e. my kernel is at 1MB where GRUB and God intended it to be...

[LaurensR1983]

JamesM: I'm pretty new to this stuff, but don't you have any problems if you want to use virtual 8086 mode or whatever? I read somewhere that could cause problems if you loaded your kernel at 1 Meg. But then again why would it be a problem, because you use the memory below 1 meg for that right?

[JamesM]

I don't want to switch to VM86 mode It's obsolete as of x86_64 anyway, it's just a hassle. I might as well just switch down to real mode for anything like that I need. The only thing I think I'll need real mode/vm86 for anyway is VESA mode setup.

JamesM

[Brendan]

Hi,

Now, when copying memory from one VM space to another, You *could* use a different vm space which is identity mapped and copy it like that. That would be quite slow, as switching VM spaces flushes the CPU pipeline and its translation lookaside buffers.

The way I do it is I have a small asm snippet that:

Ouch!

Disabling paging causes a full TLB flush each time you do it - it'd be much faster to temporarily map a page into address space, then copy data from one page to another without disabling paging at all. In general, if you think you need to disable paging for any reason you've messed something up and overlooked a faster way.

Next, think about "Copy On Write"...

JamesM: I'm pretty new to this stuff, but don't you have any problems if you want to use virtual 8086 mode or whatever? I read somewhere that could cause problems if you loadeded your kernel at 1 Meg. But then again why would it be a problem, because you use the memory below 1 meg for that right?

That depends...

If you use V86 as part of your kernel and nowhere else, then it can be easier to have the first 1 MB of the virtual address space as part of kernel space.

However, if you don't use V86 as part of your kernel and allow normal processes (virtualizers, X11, DOSemu, etc) to use V86 then it makes more sense to have the first 1 MB of the virtual address space as part of user space.

I use a lower-half kernel - I.e. my kernel is at 1MB where GRUB and God intended it to be...

GRUB loads code at 1 MB because it makes GRUB's code simpler.

Putting your kernel in the lower part of the virtual address space causes other problems besides V86 though. First, the first 16 MB is "precious" as it's usable for ISA DMA, 32-bit PCI devices, 64-bit PCI devices and normal RAM. Higher addresses are less "precious" as they can't be used for as many things.

Secondly, it causes problems for NUMA systems. Because most of the kernel uses RAM that's all in one NUMA domain you end up with an unbalanced system where code running on one set of CPUs (e.g. in one NUMA domain) peforms better than code running on other CPUs (in other NUMA domains) - it breaks the "all CPUs are equal" assumption that most people make. It's better to avoid this by selecting random pages for the kernel so that no CPUs are given and advantage or a disadvantage (so that all CPUs have equally "average" performance, rather than some fast and some slow). It's even better to specifically select certain pages from specific NUMA domains to suit what the kernel is using the pages for (so that all CPUs have equally "above average" performance).

For an example, the last version of my OS detected NUMA domains during boot, and had a seperate copy of the kernel's code for each NUMA domain. It looked like a single kernel because the same code was at the same virtual address, but it wasn't - all CPUs in the same NUMA domain used pages specifically chosen to suit that NUMA domain, so that all CPUs could access kernel code without penalties.

I did similar things for the kernel's data. Some of the kernel's data (things that aren't modified often) had a seperate copy for each NUMA domain (like the kernel's code). Some of the kernel's data used pages that were chosen to be fast for the CPU that accesses the data structure more often. For example, creating a process meant creating several pages of kernel data for the new process (page directory, process data page, thread data page, kernel stack, etc), which all used pages specifically suited to the NUMA domain the new process would be running on.

I guess what I'm saying is that choosing certain physical pages just because they happened to be convenient for people who wrote a boot loader doesn't seem very "optimal" to me. Going the next step and mapping those pages 1:1 makes even less sense....


Cheers,

Brendan