Memory managment broken

[WinExperements]

Hello! So i am trying to implement the FAT32 FS driver and got #PF when trying to allocate memory for the FAT, after some debugging i realize that returned address didn't mapped, so i map all detected memory at startup, but this not fix my problem, after checks i see that allocated address didn't mapped again, so after some reviews of code i am try to run it on real hardware(kernel works if i didn't mount FAT32), but i got triple fault and reboot. So my question is how i can correctly map the memory for propertly allocations when i allocates it?
There is how i am mapping the memory:

Code: Select all

static void x86_mapMem(int *dir) {
  int from = 0x10000;
  int to = pmml_getMemEnd();
  int cur = from;
  while(cur <= to) {
    int *table = vmm_allocTable(dir,cur);
    dir[PAGE_DIRECTORY_INDEX(cur)] = ((unsigned int)table) | 7;
    for (int i = 0; i < 1024; i++) {
      vmm_map(table,cur,cur);
      cur+=4096;
    }
    //printf("VMM: Cur: %x needed: %x\n",cur,to);
  }

[Octocontrabass]

You're calling vmm_map() with the apparent intention of filling all 1024 entries in the page table, but your start address does not correspond to the first entry in a page table.

Why have paging enabled in the first place if you're just going to identity map everything?

Have you tried using debugging tools like QEMU's "info tlb" and "info mem"?

[WinExperements]

Why have paging enabled in the first place if you're just going to identity map everything?

I am using paging for ELF loading, and okay how I correctly must call vmm_map to map allocated pages if it didn't mapped? Because my realization of mapping allocated pages gives me triple fault at startup

Have you tried using debugging tools like QEMU's "info tlb" and "info mem"?

Yeah, and I see that not all memory will mapped(the start of memory didn't mapped)

[WinExperements]

So what i should do for allocating pages with virtual memory, because my realization goes wrong?

[Octocontrabass]

I've already explained why the code you posted doesn't work. I'm not sure what else I can say about it. Have you tried stepping through it in a debugger so you can see exactly what it's doing?

[WinExperements]

Thank for replay, no I am not debug it, but I can do it now. So can you give example how correctly allocate specific count of pages, then map it?

[Octocontrabass]

I don't have any example code for that, and I'm not sure it would help you even if I did since I don't know what kind of data structures you're using to keep track of pages in your OS.

[WinExperements]

Okay, maybe I just rewrite the page allocation method

[WinExperements]

Hello! So i trying to rewrite the full OS using jamesmolloy tutorials, and stuck at page directory cloning.
My problem is that the tablesPhysical inside the page_directory_t is always zero.
Here the code:

Code: Select all

uint32_t phys;
    // Make a new page directory and obtain its physical address.
    page_directory_t *dir = (page_directory_t*)kmalloc_ap(sizeof(page_directory_t), &phys);
    // Ensure that it is blank.
    ASSERT(dir->tablesPhysical != 0);
    // Get the offset of tablesPhysical from the start of the page_directory_t structure.
    uint32_t offset = (uint32_t)dir->tablesPhysical - (uint32_t)dir;

    // Then the physical address of dir->tablesPhysical is:
    dir->physicalAddr = phys + offset; // Here is the problem

Maybe i don't understand some basic things? But because the address is 0, i got triple fault
EDIT: Okay i delete the problem line, and got this in bochs:

Code: Select all

00113650744e[CPU0  ] interrupt(): gate descriptor is not valid sys seg (vector=0x0e)
00113650744e[CPU0  ] interrupt(): gate descriptor is not valid sys seg (vector=0x08)

Okay if i debug it in the GDB, after changing the CR3 register it's jumps to 0x03
Okay tablesPhysical is 0, nothing changed when i remove the line

[nullplan]

So i trying to rewrite the full OS using jamesmolloy tutorials

Oh, WHY? That thing is known to be a buggy mess. Especially with the paging. For some odd reason, he thought it would be perfectly acceptable to keep the stack for each kernel task at the same virtual address and underlay them with different physical memory. This means your kernel tasks will not be able to share kernel stack pointers.

Save yourself the trouble, throw away the tutorials, and start reading the documentation until you understand it.

Code: Select all

uint32_t phys;
    // Make a new page directory and obtain its physical address.
    page_directory_t *dir = (page_directory_t*)kmalloc_ap(sizeof(page_directory_t), &phys);
    // Ensure that it is blank.
    ASSERT(dir->tablesPhysical != 0);
    // Get the offset of tablesPhysical from the start of the page_directory_t structure.
    uint32_t offset = (uint32_t)dir->tablesPhysical - (uint32_t)dir;

    // Then the physical address of dir->tablesPhysical is:
    dir->physicalAddr = phys + offset; // Here is the problem

Honestly, without context this makes little sense. Is kmalloc_ap() just a generic allocation function? If so, then the contents of "dir" are entirely uninitialized once they drop out of that function, so I have no idea what the assertion or the calculation afterwards are doing. If "tablesPhysical" is a physical address, then the calculation of "offset" is nonsensical too, since that is a difference between a physical and a virtual address. That is meaningless.

Okay if i debug it in the GDB, after changing the CR3 register it's jumps to 0x03

That means changing CR3 changed the interpretation of the current EIP value. So you failed at copying the page directory.

[WinExperements]

Save yourself the trouble, throw away the tutorials, and start reading the documentation until you understand it.

Okay, thank you for answer. Maybe I really need to read the documentation.
And yeah, I just use the kernel heap from this tutorial, everything else I rewrite using the documentation

[neon]

Hi,

I'd like to approach this a little differently as there is an important topic that you aren't going to find in documentation and as far as I know no tutorial really covers it: how to access and update page tables correctly. This effects the implementation of most of the core virtual memory management facilities. This gets complicated because paging structures store page frame numbers (PFN) not virtual addresses. I am aware of 2 methods: (1) recursive paging and (2) mapping all of physical memory into Kernel Space.

Option (1) is required if using 32 bit protected mode, optional for 64 bit. Option (2) is most used in 64 bit. I use option (1) but I think most here probably use option (2) that target 64 bit. Any of these two strategies will give you a virtual address to access any page table or directory. These are the only virtual addresses that you would use to read and write from for updating page tables. You wouldn't typically use a heap allocator to allocate paging structures since expanding the heap requires...paging structures.

Mapping Identity Space is trivial as you only need to update the PFN for the paging structures which can be accessed from one of the 2 methods noted earlier.

So what you have in the end is the following: (1) page frame allocator, (2) one of the above 2 design choices to access paging structures, (3) MmMapAddress et. al. that updates those paging structures, (4) a System/Kernel Space PTE Pool, and (5) a kernel heap allocator.

[Octocontrabass]

I am aware of 2 methods: (1) recursive paging and (2) mapping all of physical memory into Kernel Space.

There's a third option: mapping all of the page tables into the kernel's address space without using recursion. The advantage of this approach is that you can access any set of page tables at any time, like you would by mapping all physical memory, but without needing to actually map all physical memory. The disadvantage is that it's a lot more complex, so probably not the best approach for a beginner.

[neon]

Hi,

I am aware of 2 methods: (1) recursive paging and (2) mapping all of physical memory into Kernel Space.

There's a third option: mapping all of the page tables into the kernel's address space without using recursion. The advantage of this approach is that you can access any set of page tables at any time, like you would by mapping all physical memory, but without needing to actually map all physical memory. The disadvantage is that it's a lot more complex, so probably not the best approach for a beginner.

Honestly, while in theory it seems like a reasonable solution in practice I haven't seen this approach. Both recursive paging and mapping all of physical memory require almost no overhead (in terms of performance) but I feel that this third option may require a lot of extra micro-management that may add overhead. Of course, there may be a way to do it, just not sure. Recursive paging is automatic since you always know where the paging structures are (for at least the current process.) The second option ("mapping all physical memory") just requires adding the base to get the virtual address. I would be interested for examples of the third option though in practice.

Nonetheless, I just wanted to throw out a few more terms for the original poster to look into as I believe this is the most important starting point when working on memory managers. And its a shame that its not covered anywhere really well.

[nullplan]

Option (1) is required if using 32 bit protected mode, optional for 64 bit. Option (2) is most used in 64 bit.

I will note here that Linux on i386 uses option (2) for the most part. It linearly maps the start of physical memory to the 3GB line, and does so for the first 768 MB (assuming default configuration. You can change this if you really want to). So that leaves 256MB of kernel virtual memory untouched. In these first 768MB, it places the kernel image itself (which has to be loaded to the 1MB line), all of the things it has to know the physical address of without translation (so all the paging structures for all the tasks), and also everything that can't have its virtual address change after allocation. The final 256MB of virtual space are managed by an allocator, and are used for things like fixed I/O mappings (e.g. the LAPIC has its physical address just below 4GB, and must be accessed there). This block is called "high memory". When the allocator is called, it can be told whether it is OK to put an object there, and it will try to do so. But if memory runs out on high memory, it will start allocating low memory instead. This also handles the case of a PC with less than 768MB of memory, because a lot of low memory will be unused with just the things mentioned so far.

Since the mappings on the high 256MB are constantly switched out, a good balance between memory utilization and performance must be found.

In order to get started, you must decide on a scheme to write to your page tables without having a virtual address for them. With recursive mapping, you do that by giving them fixed virtual addresses, with a linear mapping you do that by giving them an address that is easily computed from the physical address. Once you have that started, I would suggest having your bootloader initialize the scheme for you, then just using it.