I'm really having trouble about understanding the paging...

[CrafterMan24]

Hello, OsDev forum...

I was making my os, and decided to adding Paging (lol it is really a required thing)

I read from BareBones tutorials and etc...

But i still can't understand how Paging works and how can i make my paging driver...

I don't want any code, i just want to learn paging... When i learn, i can code...

I'm trying to learn paging about 1-2 weeks, but i still can't understand Paging

I don't want use so much example code, i want to coding it myself

Thanks

[max]

Hey CrafterMan24,

please try to explain what exactly you don't understand, so we can try to help you better. Have you understood how the virtual to physical translation is done (how the calculation works)?

Greets,
Max

[CrafterMan24]

Hey CrafterMan24,

please try to explain what exactly you don't understand, so we can try to help you better. Have you understood how the virtual to physical translation is done (how the calculation works)?

Greets,
Max

I just understand page tables on page directory, enabling paging and loading page directory... All others... Umm. I understand nothing except them

[max]

I just understand page tables on page directory, enabling paging and loading page directory... All others... Umm. I understand nothing except them

Hm. So what do you think that these are used for?

When you load a page directory and paging is enabled, then this is what the MMU uses to translate any virtual address to a physical address. A virtual address is what executing code "sees" when accessing memory. A virtual address is translated to a physical address via the respective index in the paging directory and page table where the page is mapped into. If you place the address of a physical page (say, 0x1A841000) into page directory index 41, page table index 23, then the address will be:

Code: Select all

virtual address = (directory_index * 1024 * 0x1000) + (table_index * 0x1000);
virtual address = (41 * 1024 * 0x1000) + (23 * 0x1000);
virtual address = 0x0A417000;

When the executing code now accesses the memory at virtual 0x0A417000, it will actually write to the physical page 0x1A841000.

[SpyderTL]

A modern OS is responsible for managing all of the shared resources on the machine, and providing safe access to those resources to any applications running on that machine. Memory is just one of those resources.

One of the easiest ways to share resources between applications is to prevent "direct" access to those resources, and instead force the application to access "virtual" resources. That way, the application can assume that it is the only application running on the machine, and can access any resource it wants.

For memory, this can be done by programming the MMU to look up actual memory addresses in a table. The OS is responsible for keeping this table up to date.

So, now every application thinks that it can access 4GiB of memory (in 32-bit mode). But you may only have 1 GiB of actual memory. So, the OS can move some memory that is not being used to the hard drive by catching a "Page Fault" exception, which is thrown by the MMU if an address requested by an application is not listed in the table (or if it is listed, but has been flagged as "not available").

So, when this exception occurs, the OS is responsible for finding an area in actual memory that is not currently being used, and adding it to the table, giving it a new address that matches the address requested by the application. If the requested address is already in the table, then the OS may also need to move some memory from the hard drive back into memory.

Once all of this is done, the OS returns control back to the application, which attempts to access the memory again. All of this happens without the application even noticing. All the application sees is that it read some data from a memory address.

Hopefully, this clears things up a bit. It helps to think of each application running in its own virtual machine, and the OS just manages all of the virtual machines on one real machine, at least for me.

[CrafterMan24]

I just understand page tables on page directory, enabling paging and loading page directory... All others... Umm. I understand nothing except them

Hm. So what do you think that these are used for?

When you load a page directory and paging is enabled, then this is what the MMU uses to translate any virtual address to a physical address. A virtual address is what executing code "sees" when accessing memory. A virtual address is translated to a physical address via the respective index in the paging directory and page table where the page is mapped into. If you place the address of a physical page (say, 0x1A841000) into page directory index 41, page table index 23, then the address will be:

Code: Select all

virtual address = (directory_index * 1024 * 0x1000) + (table_index * 0x1000);
virtual address = (41 * 1024 * 0x1000) + (23 * 0x1000);
virtual address = 0x0A417000;

When the executing code now accesses the memory at virtual 0x0A417000, it will actually write to the physical page 0x1A841000.

Thanks, it really helped, i will start making my paging driver

[CrafterMan24]

A modern OS is responsible for managing all of the shared resources on the machine, and providing safe access to those resources to any applications running on that machine. Memory is just one of those resources.

One of the easiest ways to share resources between applications is to prevent "direct" access to those resources, and instead force the application to access "virtual" resources. That way, the application can assume that it is the only application running on the machine, and can access any resource it wants.

For memory, this can be done by programming the MMU to look up actual memory addresses in a table. The OS is responsible for keeping this table up to date.

So, now every application thinks that it can access 4GiB of memory (in 32-bit mode). But you may only have 1 GiB of actual memory. So, the OS can move some memory that is not being used to the hard drive by catching a "Page Fault" exception, which is thrown by the MMU if an address requested by an application is not listed in the table (or if it is listed, but has been flagged as "not available").

So, when this exception occurs, the OS is responsible for finding an area in actual memory that is not currently being used, and adding it to the table, giving it a new address that matches the address requested by the application. If the requested address is already in the table, then the OS may also need to move some memory from the hard drive back into memory.

Once all of this is done, the OS returns control back to the application, which attempts to access the memory again. All of this happens without the application even noticing. All the application sees is that it read some data from a memory address.

Hopefully, this clears things up a bit. It helps to think of each application running in its own virtual machine, and the OS just manages all of the virtual machines on one real machine, at least for me.

Wow really thanks, it really cleared thinks up

I found some cool info about paging from internet, i will try making paging driver now.

[embryo2]

All of this happens without the application even noticing.

Even if an application haven't noticed anything the user really notices that the application is almost dead. And (at least under Windows) if there's not enough memory then the whole OS goes "down". Very unpleasant situation.

I don't want to convince anybody not to use paging, but I want to remind that the paging is just one of possible memory management strategies and it's disadvantages should be considered very carefully. And as a result - alternate memory management strategies also should be explored.

[BASICFreak]

I don't want to convince anybody not to use paging, but I want to remind that the paging is just one of possible memory management strategies and it's disadvantages should be considered very carefully. And as a result - alternate memory management strategies also should be explored.

Well, the disadvantage of paging does NOT equal the disadvantage of not paging.

Without paging, you will never get into Long Mode, access >3GB of RAM, run multiple processes at once (yes segmentation exists on SOME processors so last is not 100% true) [just lost 70% of my systems resources, where's my 8GB of ram?]

With paging, you have to keep track of tables (1-4 layers depending on page size and type). [not much of a disadvantage IMO]

These are only the first things I thought of...

Even if an application haven't noticed anything the user really notices that the application is almost dead. And (at least under Windows) if there's not enough memory then the whole OS goes "down". Very unpleasant situation.

In my experience windows will not go down, paging to HDD will start and slow the system to a crawl. When the system is slow the app trying to use the pages on the HDD will be waiting for data and not responding the "I'm Alive" ping - and windows sees the program as crashed, if you touch it it closes if you don't it may stay open until its data arrives.

Or if its out of video memory you may have a full crash (I blame NVidia)



But, yes paging to HDD should be avoided! If you must, check speeds on disks and find the quickest one, and only cache low priority - hardly used - data

[wyj]

I don't want to convince anybody not to use paging, but I want to remind that the paging is just one of possible memory management strategies and it's disadvantages should be considered very carefully. And as a result - alternate memory management strategies also should be explored.

Well, the disadvantage of paging does NOT equal the disadvantage of not paging.

Without paging, you will never get into Long Mode, access >3GB of RAM, run multiple processes at once (yes segmentation exists on SOME processors so last is not 100% true) [just lost 70% of my systems resources, where's my 8GB of ram?]

With paging, you have to keep track of tables (1-4 layers depending on page size and type). [not much of a disadvantage IMO]

These are only the first things I thought of...

Even if an application haven't noticed anything the user really notices that the application is almost dead. And (at least under Windows) if there's not enough memory then the whole OS goes "down". Very unpleasant situation.

In my experience windows will not go down, paging to HDD will start and slow the system to a crawl. When the system is slow the app trying to use the pages on the HDD will be waiting for data and not responding the "I'm Alive" ping - and windows sees the program as crashed, if you touch it it closes if you don't it may stay open until its data arrives.

Or if its out of video memory you may have a full crash (I blame NVidia)



But, yes paging to HDD should be avoided! If you must, check speeds on disks and find the quickest one, and only cache low priority - hardly used - data

I think each process has its own PML4 , so cr3 must be changed when changing process, have I got it right?

even if we dont want paging we still have to prepare one PML4, one DirectPtr, one Directory ,since no ordinary cpu support 1GB page, as we must prepare GPT with GPT[0] and GPT[1] to bypass Segmentation, is it? or do we need more of them?
I dont really want to bypass paging because it's actually necessary to isolate ring0 code and ring3 processes, I'm just interesting in the possibility of embryo2's idea.

It seems that the "paging to HDD" which you are talking about means clear the PRESENT bit, and store it to disk, the same thing as the "swap" does?

AND very much thanks to your public-domain code because there are really rare references except the long and boring manual...while actually the volume 2 of it is much better.

[embryo2]

Well, the disadvantage of paging does NOT equal the disadvantage of not paging.

Yes, disadvantage is much greater.

Without paging, you will never get into Long Mode

First - it is the only one processor's family issue.
Second - the OS can take care of the quirks of the long mode.

access >3GB of RAM

OS can implement a managed environment where the actual address space, available to an application, can be very big.

run multiple processes at once (yes segmentation exists on SOME processors so last is not 100% true) [just lost 70% of my systems resources, where's my 8GB of ram?]

Haven't got your point. What is the problem? There are multiprocess OSes that use even real mode.

With paging, you have to keep track of tables (1-4 layers depending on page size and type). [not much of a disadvantage IMO]

With paging I need to find a memory that doesn't exist. So I have no option but to swap. And as a result OS goes down.

In my experience windows will not go down, paging to HDD will start and slow the system to a crawl.

Well, you can call it "crawl", but what I see is the half an hour period of unresponsiveness. Yes, it is possible to be patient and go forward one mouse click per minute, but I prefer to reboot the OS in 2 minutes and work much quicker.

[SpyderTL]

In general, I'd say that "swapping" is better than "crashing". I'd rather my machine be slow than not work when I'm using a lot of memory.

However, with that being said, one of the first things that I do on any Windows machine is disable paging.

I've had Windows complain that it was running low on memory on rare occasion, but I've never seen it crash because it was out of memory, even with paging disabled. Go figure...

[BASICFreak]

Either way OP is referencing paging (virtual memory / MMU) not swapping.

And until OP states otherwise I (and probably others) assume he is using an Intel 80x86 processor - as it is the most common here especially to a new comer.

So with the assumption of x86, my statements about using paging (MMU) and not - are still valid.

Embryo mentioned swapping, and windows, which is why I responded at all about that side - as it was not information the OP was looking for at-least at-the-moment.

@OP, if you are still having troubles let us know what and where and we will do our best to help.






@Embryo, sometimes you must wait patiently, let us say you are running a database server on windows XP with 2GB of ram - your database is a total of 8GB and if you restart your computer "because you rather wait 2 than xx minutes" you corrupt your entire database, while if you waited, it would have taken a few moments but the task would complete and your data would not be compromised (by one's own stupidity).

That being said I hate "swapping" and until I find an actual need I will avoid it at all costs.

[embryo2]

sometimes you must wait patiently

Yes, but I prefer to use OS where it's not so important to be patient. For example - Android just kills application of a lesser priority when application with high priority requires more memory. It requires some homework to be done by developers, but there's no viable wave of complaints about it. Developers just make their programs according to Google's recommendations and everything works just like magic.

The main point here - memory management strategies can be different.