How do *you* manage virtual memory ?

[Pype.Clicker]

Hi fellow OS coderz ...

I'm currently extending my virtual memory manager, and i'm wondering whether i shouldn't restart it from scratch. In such case, advice from other experiences may be very valuable ...

So i'd like to ask you how you did address the problem of 'memory objects' allocation (how to reserve large chunk of memory for .text and .data sections, for user stacks, mapped files, shared libraries, etc.), or if you haven't implemented it yet, if you have hints on how your Reference Operating System (the one you're learning from if any) does it ...

So far, i was mainly creating lists inside of the page table (using free entries to store 'next free zone', 'zone size', etc. in the small 32 bits areas), but it has problems when you want to allocate objects that are larger than a page table

So i'm wondering about a technique that would rather use two binary trees (one sorting areas by their size so that you can quickly allocate new areas, and the other one sorting them by address range, so that you can quickly check if there should be a merge or not).

Also, i'd like to know how you planned to implement the 'properties' of memory regions, like whether it should zero new pages or not, on which file it should get it, if the page should become copy-on-write or just shared in case of a process fork and that kind of stuff.
On my side, i have so-called "memory objects" that handle messages coming from the kernel, and every region is mapped to a given memory object.

Looking forward for your hints.

[BI lazy]

hmmm ... I have one binary tree per process. This binary tree holds the allocated Virtual Memory - regardless whether it is already mapped in or not - the memory-service is responsible for this allocating stuff. The pager, whom the memory-service talks to, is responsible for the paging stuff - and, since it is a slimy little weasel that slips in and out of process adress spaces at need, - it is responsible for moving process images to their proper place at process start up - 0xdeadbeef hangs around at process start up and triggers a page fault. My memory manager acts on a Per Process base.

This way, I have split the memory management stuff between actual execution and management/policy. *shrugs* It's just micro kernel stuff.

well, anyway, blueIllusion is far away from being a high sophisticated modularized thing. But the first steps towards this idea grow out of my very dust.

hope it helps

stay safe

[kaos]

I'm still working out some of the ideas for the virtual memory manager. I have a vauge sort of plan that involves organizing a MemoryMap into Sections. A Section contains, among other things, a beginning and ending address, and a virtual pagetable, which is an array of pointers to VirtualPages. A VirtualPage may contain a pointer to a PhysicalPage, which is a structure that represents a physical page in memory. Alternately, a VirtualPage may contain a pointer to a File and an offset. A VirtualPage could contain flags that indicate that it is uninitialized data. Each VirtualPage would also have a list of referring Section/virtual address pairs in order to handle copy-on-write, etc.

When a pagefault occurs, the operating system would search the MemoryMap's Sections by the virtual address of the offending data. This search would probably just be a linear search since the number of sections is likely to be small. It would then lookup the VirtualPage by doing some simple arithmetic and indexing an array. From there it would determine the sort of action should be taken in order to resolve the pagefault.

There are still a lot of details to work out, and other people can undoubtedly tell you better methods of doing this. Take it for what it is worth.

[Candy]

I'm not yet managing my own virtual memory, but I'm setting up the stages for a clean rewrite. I'm thinking about making a list of memory pages that are available in kernel-space, and keeping a double list of pages (both the official page table entries and a reverse list having the virtual addresses in physical address order, and because it's a sparse list containing both the virtual and the physical address). I'll map in page by page for each process in his own memory map (located statically at 0xF0100000h) in his paged space, with the page directory table at 0xF0200000h.

MMAP'ed files and swap pages are used with a table having a 64-bit inode number, and 64-bit indexes into the memory, ordered by memory address bit.

Have to come back to you in a min...

[df]

i never thought much on vmm mostly as i never had a working disk driver i liked.. and doing good vmm is pointless if everything is in memory anyway

[Carnac]

I have a page level allocator which doubles for VM as well as physical page level allocation. Each heap is designated by a structure which basically defines the start address, the count of free pages in the heap, a bitmap demonstrating which pages are free/allocated, etc. So, basically, I have a pmemmap, and a vmemmap structure - ie, physical memory, and virtual memory.

In order to allocate a page, the corresponding id of the heap is passed to the page level allocator, ie, a pointer to pmemmap or vmemmap. The page level allocator then hunts through the structure looking for free pages in the bitmap, ie, a 0 bit. It also caches x number of free pages in a cache in the structure. Ie, when a page is freed, and the free cache isn't full, it adds it to the free cache for quick look up.

So, basically, the kernel manages 2 heaps - physical and virtual. When a request for memory from the kernel heap is made, the following occurs:

(a) vaddr = pag_Alloc (&vmemmap, pages)
(b) paddr = pag_AllocPage (&pmemmap);
(c) vm_Map () is called to set up a mapping of vaddr to paddr, where paddr is the address of a physical page, and vaddr is the address of the 'virtual' page.
(d) vaddr is bumped by a page size, and the code loops back to (b) until all the pages are mapped

The kernel also does < page size allocations via a kmalloc () interface, but I won't discuss that here for simplicity.

In order to handle segments for process data, text, etc, I have a heirarchy. The topmost being an 'as' structure, which basically holds the segments which are allocated to this process, ie, data, etc. 'as' points to the page directory for this process, as well as the 'seg' structures which are logical abstractions of segments, ie, they define the base address, access attributes, and contain a list of page structures defining the segment, ie, the page structures are abstractions of physical pages.

So it goes:

as
seg -> text
-> base, size
seg -> data
-> base, size
seg -> stack
-> base size

The only way of allocating memory in this model for processes, is to change the size of the data segment, and use higher level malloc () returns to manage it.

[Pype.Clicker]

thanks for all your (past and future) submissions ... i'll print this and read it with a teacup by my side and post my thought on it in a very soon future (as soon as my flat's mess will reach the 'low' threshold again )

[Pype.Clicker]

by reading those hints again and with my recent failure of integrating the 'binary trees' trick in the current framework, it looks like i'm going for a "redo-from-scratch" of the virtual memory... It will probably be designed around two 'main classes' :
* frames list (describing physical memory returned by a physical allocator service) and
* virtual regions (returned by a virtual allocator service)

from then, i think i'll have a 'mapper' service filling a region (or a sub-region) with a frame list, and report which sub-regions have a given property (like 'where the **** are my dirty pages ...)

hope it'll be cleaner that what i have for now ...