Paging File Format or File System to handle it

[~]

I have been thinking that the reason why things like implementing paging and understanding the concept of higher-half is so difficult, is that there really isn't a clear idea for an existing standard and default paging map.

It look like without providing it, trying to implement it properly would be just like trying to invent a file format or a file system, so we would need to add default page mappings for Windows, Linux and other well-developed standard OS platforms.

If we were to draw a schematic showing where every paging region is and what it is (just like describing an image or an executable file format), how would the default paging map look for Windows or Linux, for being able to understand it at high level and also describing how to implement it at low level?

For example, when fully booting Linux or Windows, I imagine a paging memory structure like this:

- The very first, original goal of implementing paging would be having a very simple, raw kernel environment, and use it to allocate and deallocate memory easily and arbitrarily with simple API functions that do it and leave fragmentation to the kernel internals.

- Memory would in general be like a 4-Kilobyte disk.

- Just like a disk partition, but instead of having the partition structures at the start, we would keep them at the kernel.

- We would detect memory regions with BIOS service 0xE820 and by scanning PCI and their memory mappings through BARs (Base Address Registers).

- We would treat memory buffers, stacks, programs, paging and process structures, and other concrete first-level structures as files.

- It might be that hardware or software buffers start or end unaligned within the 4-Kilobyte pages, but we will always contain them like that and just have device drivers and programs get meaningful paged memory addresses.

- We could page everything, identity-map things that are vital, like kernel and system stacks and process/memory main structures, heaps, memory I/O (virtual paged address==physical address), and use meaningful and standard virtual paging addresses for everything else like device drivers, system or user programs.

- Probably we would only use a single page directory and apply the corresponding privilege ring levels to all pages.

- What use would a higher-half kernel have if we could use standarized paging addresses and make internal physical addresses, specially for the kernel itself, as standard as possible?

[onlyonemac]

The paging map is fairly undefined. Beyond the kernel, which is usually (but not necessarily) loaded at a fixed physical address, the paging maps are pretty much formed on a "where's the next free physical page" basis as the system is running. In terms of logical addresses, the kernel is pretty much always mapped at a fixed logical address (e.g. higher-half kernel) and code and data heaps for threads are usually mapped at fixed addresses (I can't think of any situation where this would be undesirable, nor where mapping things at arbitrary addresses wouldn't cause a lot of complications).

[iansjack]

I think that you are talking about memory maps rather than paging. Whether paging is involved or not is fairly irrelevant. Your post title rather confuses the matter by referring to paging files and file systems; the body of your post is about memory maps and memory mapping via the paging mechanism, which is a different topic.

Implementing paging is not really difficult as long as you read the reference on the data structures involved in it.

[~]

There isn't enough specification to understand paging easily.

There are just no full tutorials that show a concrete and well-standarized paging layout.

Thanks to good tutorials, I have learned to understand FAT filesystems, floppy programming, standard VGA programming, using Protected Mode, detecting hard disks and CD/DVD drives (IDE and SATA), GIF, BMP, EXE (Portable Executable) and TAR file formats, to the extent that there are such good tutorials.

But the same complication goes for multitasking. These two require extremely specific tutorials to the extent they develop, explain and implement a standard way to use them, so standard that they look like regular and easily-understandable file format mapping specifications.


I know that what makes it difficult to do a tutorial of this kind is that it might allow reserving memory freely without taking into account fragmentation or other complications, but it's only of interest for low level programs.

But if it isn't done, it will never stop being a unthinkably huge stopper for most developers until those tutorials, specifications and "file formats" are created.

[iansjack]

What have "file formats" got to do with paging?

Paging on the x86 processors is fully specified in the Intel Programmers' Manual. Also see http://wiki.osdev.org/Paging

[FallenAvatar]

There are just no full tutorials that show a concrete and well-standarized paging layout.

Please see http://wiki.osdev.org/Beginner_Mistakes ... al_on...3F

[gerryg400]

What's a paging layout ?

[onlyonemac]

There are just no full tutorials that show a concrete and well-standarized paging layout.

Paging layouts are not standardised. Different operating systems present different virtual address spaces to their userspace processes, and that's part of what makes one design distinct from another.