Paging..

[Warrior]

Can anyone give me a scenario where paging might come in useful? Maybe that can help me to understand it better.

I know it has to deal having virtual addresses point to physical ones but that's about it.

I still fail to see usefulness for this. Maybe I am just over analyzing this or maybe I'm totally off.

Enlighten me, please.

[AR]

Paging is used to create virtual memory spaces. It allows program A to run at 0x40000 and only see itself and the kernel whilst also allowing program B to run at 0x40000 and only see itself and the kernel. It allows you to isolate programs from one another basically. If you have ever heard of the Windows Paging File or Linux Swap Partition that is another benefit of paging (the program doesn't need to have all of itself in memory yet doesn't know that it isn't which allows the OS to transparently move parts of programs in and out without it noticing). You can do similar things with segmentation but that is unusual as no current compiler was designed to understand segmentation. However, unlike segmentation, the physical memory behind the pages does not need to be contiguous, it can be all over the place (eg. Virtual Page 0x40000 at Physical 0x120000, Virtual 0x41000 at Physical 0x161000, etc)

[Warrior]

So you can make a program appear alone in the OS and you can access multiple locations in memory at once by different programs.

It's looking pretty useful.

I'm going to read a bit into a basic paging tutorial so I can understand the terms (I know them but don't have them down yet) and then I'll be able to figure out what functions I need to provide to be able to page addresses on the fly.

[Solar]

Depends on what you mean, exactly, by saying "paging". I'll simply address all three benefits of a MMU (Memory Management Unit).

1: Virtual Addressing

By assigning virtual pages to physical memory, and having the MMU doing the mapping between virtual addresses and physical ones, any software can be written to start at e.g. 0x0000 0000, no matter where the physical memory actually is (which would also differ from machine to machine).

2: Memory Protection

By providing a seperate, non-overlapping virtual-to-physical mapping for each process, no process can harm another by randomly accessing the other process' memory. This adds a huge amount of stability and security to an OS.

3: Virtual Memory

If you have, say, 64 MByte of memory in your system, there are only so many processes you could have running concurrently. Fortunately, the MMU enables you to mark a page as "not resident" - should a virtual address from this page be accessed, the MMU would trigger a page fault. The idea being, if your physical memory runs low, you can take a number of RAM pages, mark them as "not resident", and save them to hard drive. If a page fault is triggered, "just" load them back into physical RAM and adjust the mapping to the new physical address.


Unless, of course, you meant segmentation vs. paging... which is another can of worms.

[Warrior]

For the third point:

You mark pages not in use as not resident (and save to HD) and when a page fault occurs you load them back (withought being non resident) ?

If the machine only has a set amount of memory how would this be useful -- is it possible to have more pages than you have ram?

I don't see the point of this.

[AR]

A page that is not resident in one process can then be reused to load a page into another process, this way the only memory limit is the size of the RAM + the size of the paging file/swap partition.

Each Process has a page directory, the page directory points to page tables, the page tables point to physical pages. The unused entries in the tables/directories are "not resident".

Each Process has up to 4GBs of virtual RAM, if the system only has 64MB then perhaps each process will only have about 1MB of that space actually in the real RAM, the rest is either free or paged out to disk. I'll try to break it down:

• Program requests more memory
• All RAM is full
• Kernel finds least used page currently in memory and writes it to disk
• Kernel loads the page for the current process into the spot just freed and alters the page tables accordingly (old process' page table has page marked 'not resident', new process recieves the pointer to the page)
• Process continues none the wiser

[Solar]

AR already answered it, but to reinforce:

...is it possible to have more pages than you have ram?

The limiting factors to the amount of pages you can have are:

1) the capacity of your MMU,
2) the capacity of your page directories (which have to reside in RAM too),
3) the capacity of your hard drive.

More physical RAM will make your computer faster (since hard drive access is much slower than RAM access), but it won't boost your memory capacity (much),

[Warrior]

That REALLY cleared it up. Thanks AR and Solar!

Edit: To make sure I know finally

Paging isn't defined by the amount of ram you have as it is
stored on a swap partition thus you have 4GB virtual no matter what.

Non Resident Pages on one process can be stored on the HD then reused in another process.

A page directory holds an array of page tables which each have pages pointing to physical memory?

I hope I got it.

[AR]

Pretty much, it also allows you to map the same page into two processes to create shared memory and so forth. There are also a few limitations with the other features of the x86, primarily that the GDT, IDT and interrupt handlers need to be mapped in every address space. This usually manifests itself as the following memory model:

• 1-2GB Program Code, ie. Code, Data, BSS, Heap, Stack starts at 2GB and expands down
• 2-3GB System Memory and Libraries(eg. DLLs)
• 3-4GB Kernel

Just to clarify, each process has 4GB of virtual RAM, the page directory is loaded into the CPU to map the address space, when you switch processes you switch the page directories so that a new 4GB of virtual RAM is created/loaded.

[Warrior]

So the entire kernel needs to be mapped to the processes memory?

I'm trying to do as much research as possible before coding my OS again, I don't want to enounter the previous pitfalls I encountered before.

[Pype.Clicker]

So the entire kernel needs to be mapped to the processes memory?

Yep, that's usually what people do. Note that system components might be running in dedicated address space, but one usually don't call them "kernel" when that's the case

[viral]

Advantages of paging :
================
- address translation: each task has the same virtual address
- address translation: turns fragmented physical addresses
into contiguous virtual addresses
- memory protection (buggy or malicious tasks can't harm each
other or the kernel)
- shared memory between tasks (a fast type of IPC, also
conserves memory when used for DLLs)
- demand-loading (prevents big load on CPU when a task first
starts running, conserves memory)
- memory-mapped files
- virtual memory swapping (lets system degrade gracefully when
memory required exceeds RAM size)

[Solar]

This usually manifests itself as the following memory model:

• 1-2GB Program Code, ie. Code, Data, BSS, Heap, Stack starts at 2GB and expands down
• 2-3GB System Memory and Libraries(eg. DLLs)
• 3-4GB Kernel

Before you copy that memory model verbatim, make yourself aware that having "only" 2 GB of addressable space can be a severe limitation for certain applications, like just about everything that has to do with caches (databases, proxies, that sort of things). For them, every MB more available to application data is godsend.

[Warrior]

Thanks for all your help and now with the Intel Programmers Doc in hand it's time for me to begin implementing paging into my kernel.

[Warrior]

Sorry for the double post!

So a memory allocator will need a physical and virutal memory
manager?

I know for paging you need to return key things like presence of pages and you need to be able to add page directories and add tables to those directories as well as setting some things in pages like thier presence, right?

I don't get what I need to do to initialize paging (after it's enabled)
since my thinking is that you add new directories/tables on the fly so theres no need to do anything else on startup?

So after I have this code to do what I specified above would I build a virtual memory manager ontop of that? What exactly would they provide? My guess is a way to map physical addresses to virtual ones and maybe some functions to wrap around my already existing ones in my code for management of pages and everything associated with them.

Now here's another gap, I don't see the association of a virtual mm with a physical one. I'd think a physical one wouldn't be just malloc() and free() but something that goes hand in hand with the virtual one and in the end a malloc() and free() implementation can be built ontop of them?

I'm just trying to make sense of all this, thanks for your time.

- Nelson